Compare commits

..

4 Commits

Author SHA1 Message Date
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
Daichi Narushima b2bb2ad0a0 Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names.

Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add reusable export, tick-window, and margin helpers for downstream tools.

Expose SQLite append/dedup export, recent tick retrieval, and minimum margin
summary through the SDK and CLI so projects like mteor can depend on mt5cli
instead of duplicating MT5 data plumbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version to 0.4.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for rate view resolution and SDK helpers.

Harden SQLite read-only connections, tighten view discovery, improve recent_ticks
fetch efficiency, default SQLite export to append, and expand tests and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix read-only SQLite URI construction on Windows.

Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 03:29:03 +09:00
Daichi Narushima 756faf747b Rename sqlite_history module to history (#17)
* Rename sqlite_history module to history.

Drop the sqlite-specific prefix now that history collection is the primary module name across SDK, tests, and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for history module rename.

Add a sqlite_history compatibility shim, clarify docs naming, and align the
module docstring with the collect-history SQLite scope.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove sqlite_history compatibility shim.

The rename to mt5cli.history is intentionally breaking; downstream code
should update imports rather than rely on a deprecated re-export path.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 02:40:25 +09:00
Daichi Narushima c4232bf44d Add incremental SQLite history SDK (#16)
* Add incremental SQLite history SDK for automated pipelines.

Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental history deals and stale rate view cleanup.

Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental deal filtering edge cases

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for incremental SQLite history.

Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document collect-history SQLite schema with ER diagram.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix account-event filtering and drop legacy rates resume.

Account events must follow only account_event_start, not per-symbol trade
cursors. Require normalized rates schema and fail fast when timeframe is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Validate normalized rates schema before incremental resume.

Require symbol, timeframe, and time on existing rates tables with clear
ValueError messages, and add regression tests for malformed schemas.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 01:28:22 +09:00
16 changed files with 5672 additions and 391 deletions
+72 -23
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history - **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values - **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration - **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation ## Installation
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands ## Commands
| Command | Description | | Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ | | ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date | | `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position | | `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range | | `latest-rates` | Export latest rates from a start position |
| `ticks-from` | Export ticks from a start date | | `rates-range` | Export rates for a date range |
| `ticks-range` | Export ticks for a date range | | `ticks-from` | Export ticks from a start date |
| `account-info` | Export account information | | `ticks-range` | Export ticks for a date range |
| `terminal-info` | Export terminal information | | `ticks-recent` | Export ticks from a recent trailing window |
| `version` | Export MetaTrader 5 version information | | `account-info` | Export account information |
| `last-error` | Export the last error information | | `terminal-info` | Export terminal information |
| `symbols` | Export symbol list | | `version` | Export MetaTrader 5 version information |
| `symbol-info` | Export symbol details | | `last-error` | Export the last error information |
| `symbol-info-tick` | Export the last tick for a symbol | | `symbols` | Export symbol list |
| `market-book` | Export market depth (order book) | | `symbol-info` | Export symbol details |
| `orders` | Export active orders | | `symbol-info-tick` | Export the last tick for a symbol |
| `positions` | Export open positions | | `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `history-orders` | Export historical orders | | `market-book` | Export market depth (order book) |
| `history-deals` | Export historical deals | | `orders` | Export active orders |
| `order-check` | Check funds sufficiency for a trade request | | `positions` | Export open positions |
| `order-send` | Send a trade request to the trade server (`--yes` required) | | `history-orders` | Export historical orders |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database | | `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`. Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -87,7 +93,50 @@ mt5cli -o history.db collect-history \
--timeframe M1 --flags ALL --if-exists append --with-views --timeframe M1 --flags ALL --if-exists append --with-views
``` ```
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing deals, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns and do not contribute to the weighted prices. History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
### Incremental history SDK
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, update_history, update_history_with_config
# Reuse an already-connected pdmt5 client (does not open/close MT5)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
update_history(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1", "H1"], # default: all fixed MT5 timeframes
lookback_hours=24,
create_rate_views=True,
with_views=True,
include_account_events=True,
)
finally:
client.shutdown()
# Standalone wrapper that opens and closes MT5 for you
update_history_with_config(
output="history.db",
symbols=["EURUSD"],
config=Mt5Config(login=12345),
)
```
- **`collect-history`**: explicit date-range export into SQLite.
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
## Requirements ## Requirements
+185
View File
@@ -0,0 +1,185 @@
# History Collection (SQLite)
::: mt5cli.history
## `collect-history` schema
The `collect-history` command (and the matching `collect_history` SDK function) writes
selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
- `symbol` is prepended on every table.
- `timeframe` is prepended on `rates` so appended runs at different bar sizes stay
distinguishable.
SQLite does not declare foreign keys. Rows are linked logically by `symbol`, time
windows, and (for deals) `position_id` / `order`. Duplicate rows are removed on
append using dataset-specific keys (for example `ticket` on history tables, or
`(symbol, timeframe, time)` on rates).
Optional views are created when `--with-views` is set and the `history-deals` dataset
was written.
### Entity-relationship diagram
Sample layout for a full collection with `--with-views`:
```mermaid
erDiagram
rates {
TEXT symbol "dedup key"
INTEGER timeframe "dedup key"
TEXT time "dedup key"
REAL open
REAL high
REAL low
REAL close
INTEGER tick_volume
INTEGER spread
INTEGER real_volume
}
ticks {
TEXT symbol "dedup key"
TEXT time "dedup key"
INTEGER time_msc "dedup key (preferred)"
REAL bid
REAL ask
REAL last
INTEGER volume
INTEGER flags
REAL volume_real
}
history_orders {
INTEGER ticket "dedup key"
TEXT symbol
TEXT time
INTEGER type
INTEGER state
REAL volume_initial
REAL price_open
REAL price_current
INTEGER magic
}
history_deals {
INTEGER ticket "dedup key"
INTEGER order
INTEGER position_id "groups position view"
TEXT symbol
TEXT time
INTEGER type "0/1 trade, else cash event"
INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
REAL volume
REAL price
REAL profit
REAL commission
REAL swap
REAL fee
}
cash_events {
INTEGER ticket
TEXT symbol
TEXT time
INTEGER type
REAL profit
}
positions_reconstructed {
INTEGER position_id
TEXT symbol
TEXT open_time
TEXT close_time
INTEGER direction
REAL volume_open
REAL volume_close
REAL volume_reversal
REAL open_price
REAL close_price
REAL total_profit
INTEGER reversal_count
INTEGER deals_count
}
rates ||--o{ history_deals : "symbol (logical)"
ticks ||--o{ history_deals : "symbol (logical)"
history_orders ||--o{ history_deals : "order ~ ticket (logical)"
history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
```
### Tables and views
| Object | Kind | Source | Notes |
| ------------------------- | ----- | -------------------- | ------------------------------------------------------------------------------------------- |
| `rates` | table | `copy_rates_range` | Indexed on `(symbol, timeframe, time)` when columns exist. |
| `ticks` | table | `copy_ticks_range` | Indexed on `(symbol, time)` when columns exist. |
| `history_orders` | table | `history_orders_get` | Fetched per `--symbol`, then concatenated. |
| `history_deals` | table | `history_deals_get` | Fetched per `--symbol`, then concatenated. Indexed on `(position_id, symbol)` when present. |
| `cash_events` | view | `history_deals` | Non-trade deal types (deposits, balance ops, etc.). Requires `type` column. |
| `positions_reconstructed` | view | `history_deals` | One row per closed `position_id`; volume-weighted prices and reversal stats. |
Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning
when required columns are missing.
### Incremental collection
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing legacy
naming schemes:
```python
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
```
Resolution rules:
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
are stored for the same symbol.
- When multiple naming candidates apply, prefers an existing managed
`rate_*__*` view from the candidate list.
- Falls back to single-timeframe naming when the database path is missing or
`rates` metadata is unavailable.
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
+24
View File
@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection. Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
### [History Collection (SQLite)](history.md)
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
## Architecture Overview ## Architecture Overview
The package follows a simple architecture built on top of pdmt5: The package follows a simple architecture built on top of pdmt5:
@@ -61,12 +65,18 @@ from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from mt5cli import ( from mt5cli import (
Dataset,
IfExists,
Mt5CliClient, Mt5CliClient,
collect_history, collect_history,
copy_rates_range, copy_rates_range,
detect_format, detect_format,
export_dataframe, export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
) )
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically # Fetch rates programmatically
rates = copy_rates_range( rates = copy_rates_range(
@@ -82,6 +92,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame # Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv") export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite # Collect history into SQLite
collect_history( collect_history(
Path("history.db"), Path("history.db"),
+45 -14
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history - **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values - **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration - **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation ## Installation
@@ -22,13 +23,23 @@ pip install mt5cli
## Programmatic usage / SDK usage ## Programmatic usage / SDK usage
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results. mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
```python ```python
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe from mt5cli import (
Mt5CliClient,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers # One-off fetch with module-level helpers
rates = copy_rates_range( rates = copy_rates_range(
@@ -39,10 +50,21 @@ rates = copy_rates_range(
) )
export_dataframe(rates, Path("rates.csv"), "csv") export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls # Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client: with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info() account = client.account_info()
positions = client.positions() positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command) # Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history( collect_history(
@@ -58,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API. Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start ## Quick Start
```bash ```bash
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- | | ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date | | `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position | | `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range | | `rates-range` | Export rates for a date range |
### Ticks ### Ticks
| Command | Description | | Command | Description |
| ------------- | ------------------------------ | | -------------- | ----------------------------------- |
| `ticks-from` | Export ticks from a start date | | `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range | | `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a trailing window |
### Information ### Information
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| `symbols` | Export symbol list | | `symbols` | Export symbol list |
| `symbol-info` | Export symbol details | | `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol | | `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume margin summary |
| `market-book` | Export market depth (order book) | | `market-book` | Export market depth (order book) |
### Trading ### Trading
| Command | Description | | Command | Description |
| ---------------- | ----------------------------------------------------------- | | ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders | | `orders` | Export active orders |
| `positions` | Export open positions | | `positions` | Export open positions |
| `history-orders` | Export historical orders | | `history-orders` | Export historical orders |
| `history-deals` | Export historical deals | | `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request | | `recent-history-deals` | Export historical deals from a trailing window |
| `order-send` | Send a trade request to the trade server (`--yes` required) | | `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`. Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -152,6 +181,8 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`. History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
See the [History schema diagram](api/history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
## Global Options ## Global Options
| Option | Description | | Option | Description |
+2
View File
@@ -24,6 +24,7 @@ theme:
features: features:
- content.code.annotate - content.code.annotate
- content.code.copy - content.code.copy
- content.code.mermaid
- navigation.indexes - navigation.indexes
- navigation.sections - navigation.sections
- navigation.tabs - navigation.tabs
@@ -57,6 +58,7 @@ nav:
- Overview: api/index.md - Overview: api/index.md
- CLI: api/cli.md - CLI: api/cli.md
- SDK: api/sdk.md - SDK: api/sdk.md
- History Collection (SQLite): api/history.md
- Utils: api/utils.md - Utils: api/utils.md
markdown_extensions: markdown_extensions:
+31 -1
View File
@@ -2,11 +2,13 @@
from importlib.metadata import version from importlib.metadata import version
from .history import load_rate_data, load_rate_data_from_connection
from .sdk import ( from .sdk import (
Mt5CliClient, Mt5CliClient,
account_info, account_info,
build_config, build_config,
collect_history, collect_history,
collect_latest_rates,
copy_rates_from, copy_rates_from,
copy_rates_from_pos, copy_rates_from_pos,
copy_rates_range, copy_rates_range,
@@ -15,26 +17,43 @@ from .sdk import (
history_deals, history_deals,
history_orders, history_orders,
last_error, last_error,
latest_rates,
market_book, market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders, orders,
positions, positions,
recent_history_deals,
recent_ticks,
symbol_info, symbol_info,
symbol_info_tick, symbol_info_tick,
symbols, symbols,
terminal_info, terminal_info,
update_history,
update_history_with_config,
) )
from .sdk import ( from .sdk import (
version as mt5_version, version as mt5_version,
) )
from .utils import detect_format, export_dataframe from .utils import (
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
)
__version__ = version(__package__) if __package__ else None __version__ = version(__package__) if __package__ else None
__all__ = [ __all__ = [
"Dataset",
"IfExists",
"Mt5CliClient", "Mt5CliClient",
"account_info", "account_info",
"build_config", "build_config",
"collect_history", "collect_history",
"collect_latest_rates",
"copy_rates_from", "copy_rates_from",
"copy_rates_from_pos", "copy_rates_from_pos",
"copy_rates_range", "copy_rates_range",
@@ -42,15 +61,26 @@ __all__ = [
"copy_ticks_range", "copy_ticks_range",
"detect_format", "detect_format",
"export_dataframe", "export_dataframe",
"export_dataframe_to_sqlite",
"history_deals", "history_deals",
"history_orders", "history_orders",
"last_error", "last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"market_book", "market_book",
"minimum_margins",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version", "mt5_version",
"orders", "orders",
"positions", "positions",
"recent_history_deals",
"recent_ticks",
"symbol_info", "symbol_info",
"symbol_info_tick", "symbol_info_tick",
"symbols", "symbols",
"terminal_info", "terminal_info",
"update_history",
"update_history_with_config",
] ]
+104
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
) )
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command() @app.command()
def rates_range( def rates_range(
ctx: typer.Context, ctx: typer.Context,
@@ -300,6 +325,44 @@ def ticks_range(
) )
@app.command()
def ticks_recent(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
seconds: Annotated[
float,
typer.Option(help="Lookback window in seconds."),
],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
count: Annotated[
int,
typer.Option(help="Maximum number of ticks to return."),
] = 10000,
flags: Annotated[
int,
typer.Option(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
),
)
@app.command() @app.command()
def account_info(ctx: typer.Context) -> None: def account_info(ctx: typer.Context) -> None:
"""Export account information.""" """Export account information."""
@@ -335,6 +398,16 @@ def symbol_info(
_execute_export(ctx, lambda: client.symbol_info(symbol)) _execute_export(ctx, lambda: client.symbol_info(symbol))
@app.command()
def minimum_margins(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
@app.command() @app.command()
def orders( def orders(
ctx: typer.Context, ctx: typer.Context,
@@ -427,6 +500,37 @@ def history_deals(
) )
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command() @app.command()
def version(ctx: typer.Context) -> None: def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information.""" """Export MetaTrader5 version information."""
+1537
View File
File diff suppressed because it is too large Load Diff
+626 -334
View File
File diff suppressed because it is too large Load Diff
+55 -10
View File
@@ -2,16 +2,18 @@
from __future__ import annotations from __future__ import annotations
import importlib
import json import json
import sqlite3
from datetime import UTC, datetime from datetime import UTC, datetime
from enum import StrEnum from enum import StrEnum
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard, cast from typing import TYPE_CHECKING, Any, TypeGuard
import click import click
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Sequence
import pandas as pd import pandas as pd
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -260,6 +262,50 @@ def detect_format(
raise ValueError(msg) raise ValueError(msg)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
table_name: str = "data",
*,
if_exists: IfExists = IfExists.APPEND,
index: bool = False,
index_label: str | None = None,
deduplicate_on: Sequence[str] | None = None,
) -> None:
"""Write a DataFrame to SQLite with configurable append and deduplication.
Args:
df: DataFrame to export.
output_path: SQLite database path.
table_name: Target table name.
if_exists: Conflict behavior when the table already exists.
index: Whether to write the DataFrame index as a column.
index_label: Column name for the index when ``index=True``.
deduplicate_on: Optional key columns to deduplicate after writing,
keeping the latest ``ROWID`` per key group. Deduplication scans the
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=index,
index_label=index_label,
)
if deduplicate_on:
from .history import drop_duplicates_in_table # noqa: PLC0415
drop_duplicates_in_table(
conn.cursor(),
table_name,
list(deduplicate_on),
keep="last",
)
conn.commit()
def export_dataframe( def export_dataframe(
df: pd.DataFrame, df: pd.DataFrame,
output_path: Path, output_path: Path,
@@ -289,14 +335,13 @@ def export_dataframe(
elif output_format == "parquet": elif output_format == "parquet":
df.to_parquet(output_path, index=False) df.to_parquet(output_path, index=False)
elif output_format == "sqlite3": elif output_format == "sqlite3":
sqlite3 = cast("Any", importlib.import_module("sqlite3")) export_dataframe_to_sqlite(
with sqlite3.connect(output_path) as conn: df,
df.to_sql( # type: ignore[reportUnknownMemberType] output_path,
table_name, table_name,
conn, if_exists=IfExists.REPLACE,
if_exists="replace", index=False,
index=False, )
)
else: else:
msg = f"Unsupported output format: {output_format}" msg = f"Unsupported output format: {output_format}"
raise ValueError(msg) raise ValueError(msg)
+2 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "mt5cli" name = "mt5cli"
version = "0.4.0" version = "0.5.0"
description = "Command-line tool for MetaTrader 5" description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}] authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}] maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -124,6 +124,7 @@ ignore = [
] ]
[tool.ruff.lint.per-file-ignores] [tool.ruff.lint.per-file-ignores]
"mt5cli/history.py" = ["TC003"]
"tests/**/*.py" = [ "tests/**/*.py" = [
"DOC201", # Missing return documentation "DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring "DOC501", # Raised exception missing from docstring
+176 -4
View File
@@ -6,7 +6,7 @@ import json
import logging import logging
import re import re
import sqlite3 import sqlite3
from datetime import UTC, datetime from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from unittest.mock import MagicMock from unittest.mock import MagicMock
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client) mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50, count=50,
) )
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range( def test_rates_range(
self, self,
tmp_path: Path, tmp_path: Path,
@@ -316,6 +351,65 @@ class TestCommands:
flags=2, flags=2,
) )
def test_ticks_recent(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test ticks-recent command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"ticks-recent",
"--symbol",
"EURUSD",
"--seconds",
"120",
"--date-to",
"2024-01-02",
"--count",
"500",
"--flags",
"ALL",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
def test_minimum_margins(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test minimum-margins command."""
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
mock_client.symbol_info.return_value = sym
mock_client.account_info.return_value = account
mock_client.symbol_info_tick.return_value = tick
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
mock_client.mt5.ORDER_TYPE_BUY = 0
mock_client.mt5.ORDER_TYPE_SELL = 1
output = tmp_path / "out.csv"
result = runner.invoke(
app,
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
)
assert result.exit_code == 0, result.output
mock_client.symbol_info.assert_called_once_with("EURUSD")
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
def test_orders( def test_orders(
self, self,
tmp_path: Path, tmp_path: Path,
@@ -392,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once() mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version( def test_version(
self, self,
tmp_path: Path, tmp_path: Path,
@@ -1089,7 +1261,7 @@ class TestCollectHistory:
assert all(row[0] not in {0, 1} for row in cash) assert all(row[0] not in {0, 1} for row in cash)
# Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed. # Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed.
# Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed. # Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed.
# Position 300 (open-only) and 400 (reversal-only) are excluded. # Position 400 (reversal-only with non-trade deal type) stays excluded.
assert set(positions) == {100, 200, 500, 600} assert set(positions) == {100, 200, 500, 600}
pos_100 = positions[100] pos_100 = positions[100]
tol = 1e-9 tol = 1e-9
@@ -1106,10 +1278,10 @@ class TestCollectHistory:
assert abs(pos_500[5] - 1.05) < tol assert abs(pos_500[5] - 1.05) < tol
pos_600 = positions[600] pos_600 = positions[600]
assert abs(pos_600[1] - 3.0) < tol assert abs(pos_600[1] - 3.0) < tol
assert abs(pos_600[2] - 3.0) < tol assert abs(pos_600[2] - 4.0) < tol # reversal + close volumes
assert abs(pos_600[3] - 1.0) < tol assert abs(pos_600[3] - 1.0) < tol
assert abs(pos_600[4] - 1.10) < tol assert abs(pos_600[4] - 1.10) < tol
assert abs(pos_600[5] - 1.40) < tol assert abs(pos_600[5] - 3.5475) < tol
assert pos_600[6] == 1 assert pos_600[6] == 1
def test_collect_history_filters_history_symbols_exactly( def test_collect_history_filters_history_symbols_exactly(
File diff suppressed because it is too large Load Diff
+787 -3
View File
@@ -4,9 +4,9 @@ from __future__ import annotations
import logging import logging
import sqlite3 import sqlite3
from datetime import UTC, datetime from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING from typing import TYPE_CHECKING, NamedTuple, cast
from unittest.mock import MagicMock from unittest.mock import MagicMock, call
import pandas as pd import pandas as pd
import pytest import pytest
@@ -15,12 +15,16 @@ from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING: if TYPE_CHECKING:
from pathlib import Path from pathlib import Path
from pdmt5 import Mt5DataClient
from mt5cli import sdk from mt5cli import sdk
from mt5cli.history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.sdk import ( from mt5cli.sdk import (
Mt5CliClient, Mt5CliClient,
account_info, account_info,
build_config, build_config,
collect_history, collect_history,
collect_latest_rates,
copy_rates_from, copy_rates_from,
copy_rates_from_pos, copy_rates_from_pos,
copy_rates_range, copy_rates_range,
@@ -29,17 +33,51 @@ from mt5cli.sdk import (
history_deals, history_deals,
history_orders, history_orders,
last_error, last_error,
latest_rates,
market_book, market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders, orders,
positions, positions,
recent_history_deals,
recent_ticks,
symbol_info, symbol_info,
symbol_info_tick, symbol_info_tick,
symbols, symbols,
terminal_info, terminal_info,
update_history,
update_history_with_config,
version, version,
) )
from mt5cli.utils import Dataset from mt5cli.utils import Dataset
class _TerminalInfo(NamedTuple):
connected: bool
path: str
class _AccountInfo(NamedTuple):
login: int
limits: dict[str, object]
class _MissingSummaryMethodClient:
def version(self) -> tuple[int, int, int]:
return (5, 0, 1)
def terminal_info(self) -> dict[str, bool]:
return {"connected": True}
def symbols_total(self) -> int:
return 42
class _NonCallableSummaryMethodClient:
version = (5, 0, 1)
_DEALS_FIXTURE: dict[str, list[object]] = { _DEALS_FIXTURE: dict[str, list[object]] = {
"ticket": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], "ticket": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
"position_id": [100, 100, 100, 0, 200, 200, 300, 400, 400, 500, 500, 600, 600, 600], "position_id": [100, 100, 100, 0, 200, 200, 300, 400, 400, 500, 500, 600, 600, 600],
@@ -227,6 +265,32 @@ class TestConnectionLifecycle:
client = Mt5CliClient() client = Mt5CliClient()
client.__exit__(None, None, None) client.__exit__(None, None, None)
def test_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test injected connected clients are not initialized or shut down."""
connected = MagicMock()
connected.account_info_as_df.return_value = pd.DataFrame({"a": [1]})
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
with Mt5CliClient.from_connected_client(connected) as client:
result = client.account_info()
assert result.to_dict("list") == {"a": [1]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
connected.account_info_as_df.assert_called_once()
after_exit = client.terminal_info()
assert after_exit.to_dict("list") == {"b": [2]}
connected.terminal_info_as_df.assert_called_once()
def test_constructor_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test constructor injection has the same non-owning lifecycle."""
connected = MagicMock()
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
client = Mt5CliClient(client=connected)
with client:
result = client.terminal_info()
assert result.to_dict("list") == {"b": [2]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
class TestModuleFunctions: class TestModuleFunctions:
"""Tests for module-level SDK wrappers.""" """Tests for module-level SDK wrappers."""
@@ -266,6 +330,7 @@ class TestModuleFunctions:
(last_error, (), "last_error_as_df"), (last_error, (), "last_error_as_df"),
(symbol_info_tick, ("EURUSD",), "symbol_info_tick_as_df"), (symbol_info_tick, ("EURUSD",), "symbol_info_tick_as_df"),
(market_book, ("EURUSD",), "market_book_get_as_df"), (market_book, ("EURUSD",), "market_book_get_as_df"),
(latest_rates, ("EURUSD", "M1", 10), "copy_rates_from_pos_as_df"),
], ],
) )
def test_module_functions_delegate( def test_module_functions_delegate(
@@ -349,6 +414,214 @@ class TestMt5CliClient:
assert isinstance(df, pd.DataFrame) assert isinstance(df, pd.DataFrame)
mock_client.copy_rates_range_as_df.assert_called_once() mock_client.copy_rates_range_as_df.assert_called_once()
def test_latest_rates_delegates_to_copy_rates_from_pos(
self,
mock_client: MagicMock,
) -> None:
"""Test latest_rates is a convenience wrapper for positional rates."""
Mt5CliClient().latest_rates("EURUSD", "M1", 5, start_pos=2)
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="EURUSD",
timeframe=1,
start_pos=2,
count=5,
)
def test_latest_rates_rejects_non_positive_count(self) -> None:
"""Test latest_rates validates count."""
with pytest.raises(ValueError, match="count must be positive"):
Mt5CliClient().latest_rates("EURUSD", "M1", 0)
def test_collect_latest_rates_returns_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test multi-target latest rate collection."""
result = collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
assert set(result) == {
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
}
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
def test_collect_latest_rates_uses_single_transient_connection(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test module helper opens one connection for all target pairs."""
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
mt5_data_client.assert_called_once()
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
mock_client.copy_rates_from_pos_as_df.assert_has_calls(
[
call(symbol="EURUSD", timeframe=1, start_pos=0, count=3),
call(symbol="EURUSD", timeframe=16385, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=1, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=16385, start_pos=0, count=3),
],
)
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
([], ["M1"], "At least one symbol"),
(["EURUSD"], [], "At least one timeframe"),
],
)
def test_collect_latest_rates_rejects_empty_inputs(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test multi-target latest rate input validation."""
with pytest.raises(ValueError, match=match):
Mt5CliClient().collect_latest_rates(symbols, timeframes, count=1)
def test_recent_history_deals_uses_trailing_window(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals calculates date_from from hours."""
result = recent_history_deals(
6,
date_to="2024-01-02T00:00:00+00:00",
group="*",
symbol="EURUSD",
)
assert isinstance(result, pd.DataFrame)
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group="*",
symbol="EURUSD",
ticket=None,
position=None,
)
def test_recent_history_deals_defaults_date_to_now(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals uses current UTC time when date_to is omitted."""
before = datetime.now(UTC)
recent_history_deals(1.0)
after = datetime.now(UTC)
call_kwargs = mock_client.history_deals_get_as_df.call_args.kwargs
assert before <= call_kwargs["date_to"] <= after
assert call_kwargs["date_from"] == call_kwargs["date_to"] - timedelta(hours=1)
def test_recent_history_deals_rejects_non_positive_hours(self) -> None:
"""Test recent_history_deals validates hours."""
with pytest.raises(ValueError, match="hours must be positive"):
Mt5CliClient().recent_history_deals(0)
def test_mt5_summary_returns_status_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary calls raw terminal/account status methods."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = {"connected": True}
mock_client.account_info.return_value = {"login": 123}
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True},
"account_info": {"login": 123},
"symbols_total": 42,
}
def test_mt5_summary_normalizes_namedtuple_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary returns structured plain Python values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True, "path": "terminal.exe"},
"account_info": {
"login": 123,
"limits": {"modes": ["netting", "hedging"], "servers": ["demo"]},
},
"symbols_total": 42,
}
def test_mt5_summary_as_df_stringifies_nested_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary_as_df returns export-safe tabular values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
result = mt5_summary_as_df()
assert len(result) == 1
assert result.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"path":"terminal.exe"}',
"account_info": (
'{"limits":{"modes":["netting","hedging"],'
'"servers":["demo"]},"login":123}'
),
"symbols_total": 42,
}
def test_mt5_summary_missing_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is missing."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _MissingSummaryMethodClient()),
)
with pytest.raises(
AttributeError,
match="MT5 client is missing required method: account_info",
):
client.mt5_summary()
def test_mt5_summary_non_callable_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is not callable."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _NonCallableSummaryMethodClient()),
)
with pytest.raises(
TypeError,
match="MT5 client attribute is not callable: version",
):
client.mt5_summary()
class TestCollectHistory: class TestCollectHistory:
"""Tests for collect_history SDK function.""" """Tests for collect_history SDK function."""
@@ -464,3 +737,514 @@ class TestCollectHistory:
} }
assert "cash_events" not in views assert "cash_events" not in views
assert "positions_reconstructed" not in views assert "positions_reconstructed" not in views
class TestUpdateHistory:
"""Tests for update_history SDK functions."""
@pytest.fixture
def connected_client(self) -> MagicMock:
"""Create a connected mock client without MT5 lifecycle patching."""
return MagicMock()
def test_update_history_appends_incrementally(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test sequential SQLite history updates use existing max timestamps."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
first_expected_start = datetime(2024, 1, 1, tzinfo=UTC)
second_expected_start = datetime(2024, 1, 1, 12, tzinfo=UTC)
rate_starts: list[datetime] = []
deal_starts: list[datetime] = []
def make_rates(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["timeframe"] == 1
assert kwargs["date_to"] == date_to
rate_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"open": [1.0 + len(rate_starts) / 10],
})
def make_deals(**kwargs: object) -> pd.DataFrame:
assert kwargs["date_to"] == date_to
deal_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"ticket": [10],
"position_id": [100],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
"entry": [0],
"volume": [1.0],
"price": [1.1],
"profit": [0.0],
})
connected_client.copy_rates_range_as_df.side_effect = make_rates
connected_client.history_deals_get_as_df.side_effect = make_deals
mocker.patch("mt5cli.sdk.Mt5DataClient")
output = tmp_path / "incremental-history.db"
for _ in range(2):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1"],
lookback_hours=24,
date_to=date_to,
with_views=True,
)
assert rate_starts == [first_expected_start, second_expected_start]
assert deal_starts == [first_expected_start, first_expected_start]
connected_client.initialize_and_login_mt5.assert_not_called()
connected_client.shutdown.assert_not_called()
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM rates").fetchone() == (1,)
assert conn.execute("SELECT open FROM rates").fetchone() == (1.2,)
assert conn.execute(
"SELECT COUNT(*) FROM history_deals",
).fetchone() == (1,)
assert conn.execute(
"SELECT name FROM sqlite_master WHERE name = 'cash_events'",
).fetchone() == ("cash_events",)
def test_update_history_rejects_invalid_inputs(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test validation errors for incremental history updates."""
output = tmp_path / "invalid-update.db"
with pytest.raises(ValueError, match="At least one symbol"):
update_history(
client=connected_client,
output=output,
symbols=[],
)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
lookback_hours=0,
)
with pytest.raises(ValueError, match="Invalid timeframe"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["BAD"],
)
with pytest.raises(ValueError, match="Invalid tick flags"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks},
flags="BAD",
)
def test_update_history_noops_for_empty_datasets(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 and SQLite writes."""
writer = mocker.patch("mt5cli.sdk.write_incremental_datasets")
connect = mocker.patch("mt5cli.sdk.sqlite3.connect")
update_history(
client=connected_client,
output=tmp_path / "empty-datasets.db",
symbols=["EURUSD"],
datasets=set(),
)
writer.assert_not_called()
connect.assert_not_called()
def test_update_history_uses_all_default_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test that timeframes=None writes rates for all default MT5 timeframes."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "default-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=None,
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert len(timeframes_written) == len(DEFAULT_HISTORY_TIMEFRAMES)
def test_update_history_uses_specified_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test explicit timeframes limit rate updates."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "specific-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1", "H1"],
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert timeframes_written == [1, 16385]
def test_update_history_updates_ticks_and_orders(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test incremental update writes selected ticks and orders datasets."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
expected_start = datetime(2024, 1, 1, tzinfo=UTC)
def make_ticks(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
assert kwargs["flags"] == 1
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"time_msc": [1_704_110_400_000],
"bid": [1.1],
})
def make_orders(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
return pd.DataFrame({
"ticket": [1],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
})
connected_client.copy_ticks_range_as_df.side_effect = make_ticks
connected_client.history_orders_get_as_df.side_effect = make_orders
output = tmp_path / "ticks-orders.db"
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks, Dataset.history_orders},
lookback_hours=24,
date_to=date_to,
)
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM ticks").fetchone() == (1,)
assert conn.execute(
"SELECT COUNT(*) FROM history_orders",
).fetchone() == (1,)
def test_update_history_with_config_opens_and_closes_connection(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history_with_config manages MT5 connection lifecycle."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "config-wrapper.db",
symbols=["EURUSD"],
datasets={Dataset.history_deals},
timeframes=["M1"],
flags="ALL",
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
deduplicate=False,
create_rate_views=False,
with_views=True,
include_account_events=False,
)
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
updater.assert_called_once()
assert updater.call_args.kwargs == {
"client": mock_client,
"output": tmp_path / "config-wrapper.db",
"symbols": ["EURUSD"],
"datasets": {Dataset.history_deals},
"timeframes": ["M1"],
"flags": "ALL",
"lookback_hours": 1,
"date_to": datetime(2024, 1, 1, tzinfo=UTC),
"deduplicate": False,
"create_rate_views": False,
"with_views": True,
"include_account_events": False,
}
def test_update_history_with_config_validates_before_connecting(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test invalid inputs fail before MT5 is initialized."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history_with_config(
output=tmp_path / "invalid-config.db",
symbols=["EURUSD"],
lookback_hours=0,
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
def test_update_history_with_config_noops_for_empty_datasets(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 initialization."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "empty-config.db",
symbols=["EURUSD"],
datasets=set(),
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
updater.assert_not_called()
def test_update_history_defaults_date_to_now(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history uses current UTC time when date_to is omitted."""
captured: dict[str, datetime] = {}
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
captured["end"] = args[7] # type: ignore[assignment]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
before = datetime.now(UTC)
update_history(
client=connected_client,
output=tmp_path / "now-default.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1"],
lookback_hours=12,
)
after = datetime.now(UTC)
assert before <= captured["end"] <= after
class TestRecentTicks:
"""Tests for recent_ticks helper."""
def test_recent_ticks_uses_explicit_date_to_window(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks fetches the requested trailing window."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [end],
"bid": [1.0],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=100,
flags="INFO",
config=build_config(login=123),
)
assert isinstance(result, pd.DataFrame)
client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
count=100,
flags=2,
)
client.copy_ticks_range_as_df.assert_not_called()
def test_recent_ticks_uses_latest_tick_when_date_to_omitted(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks anchors the window on the latest tick time."""
client = MagicMock()
tick = MagicMock()
tick.time = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.symbol_info_tick.return_value = tick
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [1, 2],
"bid": [1.0, 1.1],
})
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = Mt5CliClient().recent_ticks("EURUSD", 30, count=2, flags="ALL")
assert len(result) == 2
client.symbol_info_tick.assert_called_once_with("EURUSD")
client.copy_ticks_from_as_df.assert_called_once()
_, kwargs = client.copy_ticks_range_as_df.call_args
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_to"] == tick.time
assert kwargs["date_from"] == tick.time - timedelta(seconds=30)
assert kwargs["flags"] == 1
def test_recent_ticks_rejects_unsupported_tick_time(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks raises when the latest tick time is unsupported."""
client = MagicMock()
tick = MagicMock()
tick.time = object()
client.symbol_info_tick.return_value = tick
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
with pytest.raises(TypeError, match="Unsupported tick time value"):
Mt5CliClient().recent_ticks("EURUSD", 30)
@pytest.mark.parametrize(
"tick_time",
[
"2024-01-02T12:00:00+00:00",
1704196800,
],
)
def test_recent_ticks_coerces_string_and_unix_tick_times(
self,
mocker: MockerFixture,
tick_time: str | int,
) -> None:
"""Test recent_ticks accepts string and unix tick timestamps."""
client = MagicMock()
tick = MagicMock()
tick.time = tick_time
client.symbol_info_tick.return_value = tick
expected_end = (
datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
if isinstance(tick_time, str)
else datetime.fromtimestamp(tick_time, tz=UTC)
)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [expected_end],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
Mt5CliClient().recent_ticks("EURUSD", 30)
_, kwargs = client.copy_ticks_from_as_df.call_args
assert kwargs["date_from"] == expected_end - timedelta(seconds=30)
def test_recent_ticks_returns_full_frame_when_count_not_positive(
self,
mocker: MockerFixture,
) -> None:
"""Test non-positive count returns the full range without trimming."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=0,
config=build_config(login=123),
)
assert len(result) == 3
client.copy_ticks_from_as_df.assert_not_called()
client.copy_ticks_range_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
date_to=end,
flags=1,
)
class TestMinimumMargins:
"""Tests for minimum_margins helper."""
def test_minimum_margins_shape(
self,
mocker: MockerFixture,
) -> None:
"""Test minimum_margins returns the expected summary columns."""
client = MagicMock()
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
client.symbol_info.return_value = sym
client.account_info.return_value = account
client.symbol_info_tick.return_value = tick
client.order_calc_margin.side_effect = [12.5, 12.4]
client.mt5.ORDER_TYPE_BUY = 0
client.mt5.ORDER_TYPE_SELL = 1
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = minimum_margins("EURUSD", config=build_config(login=123))
pd.testing.assert_frame_equal(
result,
pd.DataFrame([
{
"symbol": "EURUSD",
"account_currency": "USD",
"volume_min": 0.01,
"buy_margin": 12.5,
"sell_margin": 12.4,
}
]),
)
client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
+108
View File
@@ -21,8 +21,10 @@ from mt5cli.utils import (
TIMEFRAME_MAP, TIMEFRAME_MAP,
TIMEFRAME_TYPE, TIMEFRAME_TYPE,
Dataset, Dataset,
IfExists,
detect_format, detect_format,
export_dataframe, export_dataframe,
export_dataframe_to_sqlite,
parse_datetime, parse_datetime,
parse_request, parse_request,
parse_tick_flags, parse_tick_flags,
@@ -130,6 +132,112 @@ class TestExportDataframe:
export_dataframe(sample_df, tmp_path / "out.txt", "xml") export_dataframe(sample_df, tmp_path / "out.txt", "xml")
class TestExportDataframeToSqlite:
"""Tests for export_dataframe_to_sqlite."""
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
"""Test append mode keeps prior rows in the SQLite table."""
output = tmp_path / "append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
"""Test deduplication keeps the latest ROWID for key columns."""
output = tmp_path / "dedup.db"
first = pd.DataFrame({
"symbol": ["EURUSD", "EURUSD"],
"time": ["2024-01-01", "2024-01-01"],
"bid": [1.0, 1.1],
})
second = pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
})
export_dataframe_to_sqlite(
first,
output,
"ticks",
if_exists=IfExists.REPLACE,
deduplicate_on=("symbol", "time"),
)
export_dataframe_to_sqlite(
second,
output,
"ticks",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "time"),
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, time, bid FROM ticks",
conn,
)
pd.testing.assert_frame_equal(
result.reset_index(drop=True),
pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
}),
)
def test_default_if_exists_appends_without_dropping_rows(
self,
tmp_path: Path,
) -> None:
"""Test the default append mode keeps prior rows."""
output = tmp_path / "default-append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items")
export_dataframe_to_sqlite(second, output, "items")
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_writes_index_with_label(self, tmp_path: Path) -> None:
"""Test optional index export with a custom label."""
output = tmp_path / "index.db"
frame = pd.DataFrame(
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
)
export_dataframe_to_sqlite(
frame,
output,
"margins",
if_exists=IfExists.REPLACE,
index=True,
index_label="symbol",
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, value FROM margins",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Parse helpers # Parse helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
Generated
+1 -1
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]] [[package]]
name = "mt5cli" name = "mt5cli"
version = "0.4.0" version = "0.5.0"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "click" }, { name = "click" },