Compare commits

..

4 Commits

Author SHA1 Message Date
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
Daichi Narushima b2bb2ad0a0 Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names.

Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add reusable export, tick-window, and margin helpers for downstream tools.

Expose SQLite append/dedup export, recent tick retrieval, and minimum margin
summary through the SDK and CLI so projects like mteor can depend on mt5cli
instead of duplicating MT5 data plumbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version to 0.4.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for rate view resolution and SDK helpers.

Harden SQLite read-only connections, tighten view discovery, improve recent_ticks
fetch efficiency, default SQLite export to append, and expand tests and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix read-only SQLite URI construction on Windows.

Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 03:29:03 +09:00
Daichi Narushima 756faf747b Rename sqlite_history module to history (#17)
* Rename sqlite_history module to history.

Drop the sqlite-specific prefix now that history collection is the primary module name across SDK, tests, and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for history module rename.

Add a sqlite_history compatibility shim, clarify docs naming, and align the
module docstring with the collect-history SQLite scope.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove sqlite_history compatibility shim.

The rename to mt5cli.history is intentionally breaking; downstream code
should update imports rather than rely on a deprecated re-export path.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 02:40:25 +09:00
Daichi Narushima c4232bf44d Add incremental SQLite history SDK (#16)
* Add incremental SQLite history SDK for automated pipelines.

Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental history deals and stale rate view cleanup.

Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental deal filtering edge cases

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for incremental SQLite history.

Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document collect-history SQLite schema with ER diagram.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix account-event filtering and drop legacy rates resume.

Account events must follow only account_event_start, not per-symbol trade
cursors. Require normalized rates schema and fail fast when timeframe is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Validate normalized rates schema before incremental resume.

Require symbol, timeframe, and time on existing rates tables with clear
ValueError messages, and add regression tests for malformed schemas.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 01:28:22 +09:00
16 changed files with 5672 additions and 391 deletions
+72 -23
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -87,7 +93,50 @@ mt5cli -o history.db collect-history \
--timeframe M1 --flags ALL --if-exists append --with-views
```
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing deals, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns and do not contribute to the weighted prices.
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
### Incremental history SDK
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, update_history, update_history_with_config
# Reuse an already-connected pdmt5 client (does not open/close MT5)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
update_history(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1", "H1"], # default: all fixed MT5 timeframes
lookback_hours=24,
create_rate_views=True,
with_views=True,
include_account_events=True,
)
finally:
client.shutdown()
# Standalone wrapper that opens and closes MT5 for you
update_history_with_config(
output="history.db",
symbols=["EURUSD"],
config=Mt5Config(login=12345),
)
```
- **`collect-history`**: explicit date-range export into SQLite.
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
## Requirements
+185
View File
@@ -0,0 +1,185 @@
# History Collection (SQLite)
::: mt5cli.history
## `collect-history` schema
The `collect-history` command (and the matching `collect_history` SDK function) writes
selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
- `symbol` is prepended on every table.
- `timeframe` is prepended on `rates` so appended runs at different bar sizes stay
distinguishable.
SQLite does not declare foreign keys. Rows are linked logically by `symbol`, time
windows, and (for deals) `position_id` / `order`. Duplicate rows are removed on
append using dataset-specific keys (for example `ticket` on history tables, or
`(symbol, timeframe, time)` on rates).
Optional views are created when `--with-views` is set and the `history-deals` dataset
was written.
### Entity-relationship diagram
Sample layout for a full collection with `--with-views`:
```mermaid
erDiagram
rates {
TEXT symbol "dedup key"
INTEGER timeframe "dedup key"
TEXT time "dedup key"
REAL open
REAL high
REAL low
REAL close
INTEGER tick_volume
INTEGER spread
INTEGER real_volume
}
ticks {
TEXT symbol "dedup key"
TEXT time "dedup key"
INTEGER time_msc "dedup key (preferred)"
REAL bid
REAL ask
REAL last
INTEGER volume
INTEGER flags
REAL volume_real
}
history_orders {
INTEGER ticket "dedup key"
TEXT symbol
TEXT time
INTEGER type
INTEGER state
REAL volume_initial
REAL price_open
REAL price_current
INTEGER magic
}
history_deals {
INTEGER ticket "dedup key"
INTEGER order
INTEGER position_id "groups position view"
TEXT symbol
TEXT time
INTEGER type "0/1 trade, else cash event"
INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
REAL volume
REAL price
REAL profit
REAL commission
REAL swap
REAL fee
}
cash_events {
INTEGER ticket
TEXT symbol
TEXT time
INTEGER type
REAL profit
}
positions_reconstructed {
INTEGER position_id
TEXT symbol
TEXT open_time
TEXT close_time
INTEGER direction
REAL volume_open
REAL volume_close
REAL volume_reversal
REAL open_price
REAL close_price
REAL total_profit
INTEGER reversal_count
INTEGER deals_count
}
rates ||--o{ history_deals : "symbol (logical)"
ticks ||--o{ history_deals : "symbol (logical)"
history_orders ||--o{ history_deals : "order ~ ticket (logical)"
history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
```
### Tables and views
| Object | Kind | Source | Notes |
| ------------------------- | ----- | -------------------- | ------------------------------------------------------------------------------------------- |
| `rates` | table | `copy_rates_range` | Indexed on `(symbol, timeframe, time)` when columns exist. |
| `ticks` | table | `copy_ticks_range` | Indexed on `(symbol, time)` when columns exist. |
| `history_orders` | table | `history_orders_get` | Fetched per `--symbol`, then concatenated. |
| `history_deals` | table | `history_deals_get` | Fetched per `--symbol`, then concatenated. Indexed on `(position_id, symbol)` when present. |
| `cash_events` | view | `history_deals` | Non-trade deal types (deposits, balance ops, etc.). Requires `type` column. |
| `positions_reconstructed` | view | `history_deals` | One row per closed `position_id`; volume-weighted prices and reversal stats. |
Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning
when required columns are missing.
### Incremental collection
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing legacy
naming schemes:
```python
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
```
Resolution rules:
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
are stored for the same symbol.
- When multiple naming candidates apply, prefers an existing managed
`rate_*__*` view from the candidate list.
- Falls back to single-timeframe naming when the database path is missing or
`rates` metadata is unavailable.
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
+24
View File
@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
### [History Collection (SQLite)](history.md)
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
## Architecture Overview
The package follows a simple architecture built on top of pdmt5:
@@ -61,12 +65,18 @@ from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
Dataset,
IfExists,
Mt5CliClient,
collect_history,
copy_rates_range,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically
rates = copy_rates_range(
@@ -82,6 +92,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite
collect_history(
Path("history.db"),
+45 -14
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -22,13 +23,23 @@ pip install mt5cli
## Programmatic usage / SDK usage
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results.
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe
from mt5cli import (
Mt5CliClient,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers
rates = copy_rates_range(
@@ -39,10 +50,21 @@ rates = copy_rates_range(
)
export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
@@ -58,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start
```bash
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range |
### Ticks
| Command | Description |
| ------------- | ------------------------------ |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| Command | Description |
| -------------- | ----------------------------------- |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a trailing window |
### Information
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume margin summary |
| `market-book` | Export market depth (order book) |
### Trading
| Command | Description |
| ---------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| Command | Description |
| ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -152,6 +181,8 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
See the [History schema diagram](api/history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
## Global Options
| Option | Description |
+2
View File
@@ -24,6 +24,7 @@ theme:
features:
- content.code.annotate
- content.code.copy
- content.code.mermaid
- navigation.indexes
- navigation.sections
- navigation.tabs
@@ -57,6 +58,7 @@ nav:
- Overview: api/index.md
- CLI: api/cli.md
- SDK: api/sdk.md
- History Collection (SQLite): api/history.md
- Utils: api/utils.md
markdown_extensions:
+31 -1
View File
@@ -2,11 +2,13 @@
from importlib.metadata import version
from .history import load_rate_data, load_rate_data_from_connection
from .sdk import (
Mt5CliClient,
account_info,
build_config,
collect_history,
collect_latest_rates,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -15,26 +17,43 @@ from .sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
symbol_info,
symbol_info_tick,
symbols,
terminal_info,
update_history,
update_history_with_config,
)
from .sdk import (
version as mt5_version,
)
from .utils import detect_format, export_dataframe
from .utils import (
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
)
__version__ = version(__package__) if __package__ else None
__all__ = [
"Dataset",
"IfExists",
"Mt5CliClient",
"account_info",
"build_config",
"collect_history",
"collect_latest_rates",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
@@ -42,15 +61,26 @@ __all__ = [
"copy_ticks_range",
"detect_format",
"export_dataframe",
"export_dataframe_to_sqlite",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"market_book",
"minimum_margins",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version",
"orders",
"positions",
"recent_history_deals",
"recent_ticks",
"symbol_info",
"symbol_info_tick",
"symbols",
"terminal_info",
"update_history",
"update_history_with_config",
]
+104
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
)
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command()
def rates_range(
ctx: typer.Context,
@@ -300,6 +325,44 @@ def ticks_range(
)
@app.command()
def ticks_recent(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
seconds: Annotated[
float,
typer.Option(help="Lookback window in seconds."),
],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
count: Annotated[
int,
typer.Option(help="Maximum number of ticks to return."),
] = 10000,
flags: Annotated[
int,
typer.Option(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
),
)
@app.command()
def account_info(ctx: typer.Context) -> None:
"""Export account information."""
@@ -335,6 +398,16 @@ def symbol_info(
_execute_export(ctx, lambda: client.symbol_info(symbol))
@app.command()
def minimum_margins(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
@app.command()
def orders(
ctx: typer.Context,
@@ -427,6 +500,37 @@ def history_deals(
)
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
+1537
View File
File diff suppressed because it is too large Load Diff
+626 -334
View File
File diff suppressed because it is too large Load Diff
+55 -10
View File
@@ -2,16 +2,18 @@
from __future__ import annotations
import importlib
import json
import sqlite3
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard, cast
from typing import TYPE_CHECKING, Any, TypeGuard
import click
if TYPE_CHECKING:
from collections.abc import Sequence
import pandas as pd
# ---------------------------------------------------------------------------
@@ -260,6 +262,50 @@ def detect_format(
raise ValueError(msg)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
table_name: str = "data",
*,
if_exists: IfExists = IfExists.APPEND,
index: bool = False,
index_label: str | None = None,
deduplicate_on: Sequence[str] | None = None,
) -> None:
"""Write a DataFrame to SQLite with configurable append and deduplication.
Args:
df: DataFrame to export.
output_path: SQLite database path.
table_name: Target table name.
if_exists: Conflict behavior when the table already exists.
index: Whether to write the DataFrame index as a column.
index_label: Column name for the index when ``index=True``.
deduplicate_on: Optional key columns to deduplicate after writing,
keeping the latest ``ROWID`` per key group. Deduplication scans the
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=index,
index_label=index_label,
)
if deduplicate_on:
from .history import drop_duplicates_in_table # noqa: PLC0415
drop_duplicates_in_table(
conn.cursor(),
table_name,
list(deduplicate_on),
keep="last",
)
conn.commit()
def export_dataframe(
df: pd.DataFrame,
output_path: Path,
@@ -289,14 +335,13 @@ def export_dataframe(
elif output_format == "parquet":
df.to_parquet(output_path, index=False)
elif output_format == "sqlite3":
sqlite3 = cast("Any", importlib.import_module("sqlite3"))
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists="replace",
index=False,
)
export_dataframe_to_sqlite(
df,
output_path,
table_name,
if_exists=IfExists.REPLACE,
index=False,
)
else:
msg = f"Unsupported output format: {output_format}"
raise ValueError(msg)
+2 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.0"
version = "0.5.0"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -124,6 +124,7 @@ ignore = [
]
[tool.ruff.lint.per-file-ignores]
"mt5cli/history.py" = ["TC003"]
"tests/**/*.py" = [
"DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring
+176 -4
View File
@@ -6,7 +6,7 @@ import json
import logging
import re
import sqlite3
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50,
)
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range(
self,
tmp_path: Path,
@@ -316,6 +351,65 @@ class TestCommands:
flags=2,
)
def test_ticks_recent(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test ticks-recent command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"ticks-recent",
"--symbol",
"EURUSD",
"--seconds",
"120",
"--date-to",
"2024-01-02",
"--count",
"500",
"--flags",
"ALL",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
def test_minimum_margins(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test minimum-margins command."""
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
mock_client.symbol_info.return_value = sym
mock_client.account_info.return_value = account
mock_client.symbol_info_tick.return_value = tick
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
mock_client.mt5.ORDER_TYPE_BUY = 0
mock_client.mt5.ORDER_TYPE_SELL = 1
output = tmp_path / "out.csv"
result = runner.invoke(
app,
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
)
assert result.exit_code == 0, result.output
mock_client.symbol_info.assert_called_once_with("EURUSD")
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
def test_orders(
self,
tmp_path: Path,
@@ -392,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version(
self,
tmp_path: Path,
@@ -1089,7 +1261,7 @@ class TestCollectHistory:
assert all(row[0] not in {0, 1} for row in cash)
# Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed.
# Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed.
# Position 300 (open-only) and 400 (reversal-only) are excluded.
# Position 400 (reversal-only with non-trade deal type) stays excluded.
assert set(positions) == {100, 200, 500, 600}
pos_100 = positions[100]
tol = 1e-9
@@ -1106,10 +1278,10 @@ class TestCollectHistory:
assert abs(pos_500[5] - 1.05) < tol
pos_600 = positions[600]
assert abs(pos_600[1] - 3.0) < tol
assert abs(pos_600[2] - 3.0) < tol
assert abs(pos_600[2] - 4.0) < tol # reversal + close volumes
assert abs(pos_600[3] - 1.0) < tol
assert abs(pos_600[4] - 1.10) < tol
assert abs(pos_600[5] - 1.40) < tol
assert abs(pos_600[5] - 3.5475) < tol
assert pos_600[6] == 1
def test_collect_history_filters_history_symbols_exactly(
File diff suppressed because it is too large Load Diff
+787 -3
View File
@@ -4,9 +4,9 @@ from __future__ import annotations
import logging
import sqlite3
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, NamedTuple, cast
from unittest.mock import MagicMock, call
import pandas as pd
import pytest
@@ -15,12 +15,16 @@ from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from pathlib import Path
from pdmt5 import Mt5DataClient
from mt5cli import sdk
from mt5cli.history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.sdk import (
Mt5CliClient,
account_info,
build_config,
collect_history,
collect_latest_rates,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -29,17 +33,51 @@ from mt5cli.sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
symbol_info,
symbol_info_tick,
symbols,
terminal_info,
update_history,
update_history_with_config,
version,
)
from mt5cli.utils import Dataset
class _TerminalInfo(NamedTuple):
connected: bool
path: str
class _AccountInfo(NamedTuple):
login: int
limits: dict[str, object]
class _MissingSummaryMethodClient:
def version(self) -> tuple[int, int, int]:
return (5, 0, 1)
def terminal_info(self) -> dict[str, bool]:
return {"connected": True}
def symbols_total(self) -> int:
return 42
class _NonCallableSummaryMethodClient:
version = (5, 0, 1)
_DEALS_FIXTURE: dict[str, list[object]] = {
"ticket": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
"position_id": [100, 100, 100, 0, 200, 200, 300, 400, 400, 500, 500, 600, 600, 600],
@@ -227,6 +265,32 @@ class TestConnectionLifecycle:
client = Mt5CliClient()
client.__exit__(None, None, None)
def test_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test injected connected clients are not initialized or shut down."""
connected = MagicMock()
connected.account_info_as_df.return_value = pd.DataFrame({"a": [1]})
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
with Mt5CliClient.from_connected_client(connected) as client:
result = client.account_info()
assert result.to_dict("list") == {"a": [1]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
connected.account_info_as_df.assert_called_once()
after_exit = client.terminal_info()
assert after_exit.to_dict("list") == {"b": [2]}
connected.terminal_info_as_df.assert_called_once()
def test_constructor_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test constructor injection has the same non-owning lifecycle."""
connected = MagicMock()
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
client = Mt5CliClient(client=connected)
with client:
result = client.terminal_info()
assert result.to_dict("list") == {"b": [2]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
class TestModuleFunctions:
"""Tests for module-level SDK wrappers."""
@@ -266,6 +330,7 @@ class TestModuleFunctions:
(last_error, (), "last_error_as_df"),
(symbol_info_tick, ("EURUSD",), "symbol_info_tick_as_df"),
(market_book, ("EURUSD",), "market_book_get_as_df"),
(latest_rates, ("EURUSD", "M1", 10), "copy_rates_from_pos_as_df"),
],
)
def test_module_functions_delegate(
@@ -349,6 +414,214 @@ class TestMt5CliClient:
assert isinstance(df, pd.DataFrame)
mock_client.copy_rates_range_as_df.assert_called_once()
def test_latest_rates_delegates_to_copy_rates_from_pos(
self,
mock_client: MagicMock,
) -> None:
"""Test latest_rates is a convenience wrapper for positional rates."""
Mt5CliClient().latest_rates("EURUSD", "M1", 5, start_pos=2)
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="EURUSD",
timeframe=1,
start_pos=2,
count=5,
)
def test_latest_rates_rejects_non_positive_count(self) -> None:
"""Test latest_rates validates count."""
with pytest.raises(ValueError, match="count must be positive"):
Mt5CliClient().latest_rates("EURUSD", "M1", 0)
def test_collect_latest_rates_returns_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test multi-target latest rate collection."""
result = collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
assert set(result) == {
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
}
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
def test_collect_latest_rates_uses_single_transient_connection(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test module helper opens one connection for all target pairs."""
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
mt5_data_client.assert_called_once()
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
mock_client.copy_rates_from_pos_as_df.assert_has_calls(
[
call(symbol="EURUSD", timeframe=1, start_pos=0, count=3),
call(symbol="EURUSD", timeframe=16385, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=1, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=16385, start_pos=0, count=3),
],
)
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
([], ["M1"], "At least one symbol"),
(["EURUSD"], [], "At least one timeframe"),
],
)
def test_collect_latest_rates_rejects_empty_inputs(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test multi-target latest rate input validation."""
with pytest.raises(ValueError, match=match):
Mt5CliClient().collect_latest_rates(symbols, timeframes, count=1)
def test_recent_history_deals_uses_trailing_window(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals calculates date_from from hours."""
result = recent_history_deals(
6,
date_to="2024-01-02T00:00:00+00:00",
group="*",
symbol="EURUSD",
)
assert isinstance(result, pd.DataFrame)
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group="*",
symbol="EURUSD",
ticket=None,
position=None,
)
def test_recent_history_deals_defaults_date_to_now(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals uses current UTC time when date_to is omitted."""
before = datetime.now(UTC)
recent_history_deals(1.0)
after = datetime.now(UTC)
call_kwargs = mock_client.history_deals_get_as_df.call_args.kwargs
assert before <= call_kwargs["date_to"] <= after
assert call_kwargs["date_from"] == call_kwargs["date_to"] - timedelta(hours=1)
def test_recent_history_deals_rejects_non_positive_hours(self) -> None:
"""Test recent_history_deals validates hours."""
with pytest.raises(ValueError, match="hours must be positive"):
Mt5CliClient().recent_history_deals(0)
def test_mt5_summary_returns_status_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary calls raw terminal/account status methods."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = {"connected": True}
mock_client.account_info.return_value = {"login": 123}
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True},
"account_info": {"login": 123},
"symbols_total": 42,
}
def test_mt5_summary_normalizes_namedtuple_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary returns structured plain Python values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True, "path": "terminal.exe"},
"account_info": {
"login": 123,
"limits": {"modes": ["netting", "hedging"], "servers": ["demo"]},
},
"symbols_total": 42,
}
def test_mt5_summary_as_df_stringifies_nested_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary_as_df returns export-safe tabular values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
result = mt5_summary_as_df()
assert len(result) == 1
assert result.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"path":"terminal.exe"}',
"account_info": (
'{"limits":{"modes":["netting","hedging"],'
'"servers":["demo"]},"login":123}'
),
"symbols_total": 42,
}
def test_mt5_summary_missing_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is missing."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _MissingSummaryMethodClient()),
)
with pytest.raises(
AttributeError,
match="MT5 client is missing required method: account_info",
):
client.mt5_summary()
def test_mt5_summary_non_callable_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is not callable."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _NonCallableSummaryMethodClient()),
)
with pytest.raises(
TypeError,
match="MT5 client attribute is not callable: version",
):
client.mt5_summary()
class TestCollectHistory:
"""Tests for collect_history SDK function."""
@@ -464,3 +737,514 @@ class TestCollectHistory:
}
assert "cash_events" not in views
assert "positions_reconstructed" not in views
class TestUpdateHistory:
"""Tests for update_history SDK functions."""
@pytest.fixture
def connected_client(self) -> MagicMock:
"""Create a connected mock client without MT5 lifecycle patching."""
return MagicMock()
def test_update_history_appends_incrementally(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test sequential SQLite history updates use existing max timestamps."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
first_expected_start = datetime(2024, 1, 1, tzinfo=UTC)
second_expected_start = datetime(2024, 1, 1, 12, tzinfo=UTC)
rate_starts: list[datetime] = []
deal_starts: list[datetime] = []
def make_rates(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["timeframe"] == 1
assert kwargs["date_to"] == date_to
rate_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"open": [1.0 + len(rate_starts) / 10],
})
def make_deals(**kwargs: object) -> pd.DataFrame:
assert kwargs["date_to"] == date_to
deal_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"ticket": [10],
"position_id": [100],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
"entry": [0],
"volume": [1.0],
"price": [1.1],
"profit": [0.0],
})
connected_client.copy_rates_range_as_df.side_effect = make_rates
connected_client.history_deals_get_as_df.side_effect = make_deals
mocker.patch("mt5cli.sdk.Mt5DataClient")
output = tmp_path / "incremental-history.db"
for _ in range(2):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1"],
lookback_hours=24,
date_to=date_to,
with_views=True,
)
assert rate_starts == [first_expected_start, second_expected_start]
assert deal_starts == [first_expected_start, first_expected_start]
connected_client.initialize_and_login_mt5.assert_not_called()
connected_client.shutdown.assert_not_called()
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM rates").fetchone() == (1,)
assert conn.execute("SELECT open FROM rates").fetchone() == (1.2,)
assert conn.execute(
"SELECT COUNT(*) FROM history_deals",
).fetchone() == (1,)
assert conn.execute(
"SELECT name FROM sqlite_master WHERE name = 'cash_events'",
).fetchone() == ("cash_events",)
def test_update_history_rejects_invalid_inputs(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test validation errors for incremental history updates."""
output = tmp_path / "invalid-update.db"
with pytest.raises(ValueError, match="At least one symbol"):
update_history(
client=connected_client,
output=output,
symbols=[],
)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
lookback_hours=0,
)
with pytest.raises(ValueError, match="Invalid timeframe"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["BAD"],
)
with pytest.raises(ValueError, match="Invalid tick flags"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks},
flags="BAD",
)
def test_update_history_noops_for_empty_datasets(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 and SQLite writes."""
writer = mocker.patch("mt5cli.sdk.write_incremental_datasets")
connect = mocker.patch("mt5cli.sdk.sqlite3.connect")
update_history(
client=connected_client,
output=tmp_path / "empty-datasets.db",
symbols=["EURUSD"],
datasets=set(),
)
writer.assert_not_called()
connect.assert_not_called()
def test_update_history_uses_all_default_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test that timeframes=None writes rates for all default MT5 timeframes."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "default-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=None,
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert len(timeframes_written) == len(DEFAULT_HISTORY_TIMEFRAMES)
def test_update_history_uses_specified_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test explicit timeframes limit rate updates."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "specific-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1", "H1"],
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert timeframes_written == [1, 16385]
def test_update_history_updates_ticks_and_orders(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test incremental update writes selected ticks and orders datasets."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
expected_start = datetime(2024, 1, 1, tzinfo=UTC)
def make_ticks(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
assert kwargs["flags"] == 1
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"time_msc": [1_704_110_400_000],
"bid": [1.1],
})
def make_orders(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
return pd.DataFrame({
"ticket": [1],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
})
connected_client.copy_ticks_range_as_df.side_effect = make_ticks
connected_client.history_orders_get_as_df.side_effect = make_orders
output = tmp_path / "ticks-orders.db"
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks, Dataset.history_orders},
lookback_hours=24,
date_to=date_to,
)
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM ticks").fetchone() == (1,)
assert conn.execute(
"SELECT COUNT(*) FROM history_orders",
).fetchone() == (1,)
def test_update_history_with_config_opens_and_closes_connection(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history_with_config manages MT5 connection lifecycle."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "config-wrapper.db",
symbols=["EURUSD"],
datasets={Dataset.history_deals},
timeframes=["M1"],
flags="ALL",
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
deduplicate=False,
create_rate_views=False,
with_views=True,
include_account_events=False,
)
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
updater.assert_called_once()
assert updater.call_args.kwargs == {
"client": mock_client,
"output": tmp_path / "config-wrapper.db",
"symbols": ["EURUSD"],
"datasets": {Dataset.history_deals},
"timeframes": ["M1"],
"flags": "ALL",
"lookback_hours": 1,
"date_to": datetime(2024, 1, 1, tzinfo=UTC),
"deduplicate": False,
"create_rate_views": False,
"with_views": True,
"include_account_events": False,
}
def test_update_history_with_config_validates_before_connecting(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test invalid inputs fail before MT5 is initialized."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history_with_config(
output=tmp_path / "invalid-config.db",
symbols=["EURUSD"],
lookback_hours=0,
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
def test_update_history_with_config_noops_for_empty_datasets(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 initialization."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "empty-config.db",
symbols=["EURUSD"],
datasets=set(),
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
updater.assert_not_called()
def test_update_history_defaults_date_to_now(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history uses current UTC time when date_to is omitted."""
captured: dict[str, datetime] = {}
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
captured["end"] = args[7] # type: ignore[assignment]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
before = datetime.now(UTC)
update_history(
client=connected_client,
output=tmp_path / "now-default.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1"],
lookback_hours=12,
)
after = datetime.now(UTC)
assert before <= captured["end"] <= after
class TestRecentTicks:
"""Tests for recent_ticks helper."""
def test_recent_ticks_uses_explicit_date_to_window(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks fetches the requested trailing window."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [end],
"bid": [1.0],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=100,
flags="INFO",
config=build_config(login=123),
)
assert isinstance(result, pd.DataFrame)
client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
count=100,
flags=2,
)
client.copy_ticks_range_as_df.assert_not_called()
def test_recent_ticks_uses_latest_tick_when_date_to_omitted(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks anchors the window on the latest tick time."""
client = MagicMock()
tick = MagicMock()
tick.time = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.symbol_info_tick.return_value = tick
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [1, 2],
"bid": [1.0, 1.1],
})
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = Mt5CliClient().recent_ticks("EURUSD", 30, count=2, flags="ALL")
assert len(result) == 2
client.symbol_info_tick.assert_called_once_with("EURUSD")
client.copy_ticks_from_as_df.assert_called_once()
_, kwargs = client.copy_ticks_range_as_df.call_args
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_to"] == tick.time
assert kwargs["date_from"] == tick.time - timedelta(seconds=30)
assert kwargs["flags"] == 1
def test_recent_ticks_rejects_unsupported_tick_time(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks raises when the latest tick time is unsupported."""
client = MagicMock()
tick = MagicMock()
tick.time = object()
client.symbol_info_tick.return_value = tick
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
with pytest.raises(TypeError, match="Unsupported tick time value"):
Mt5CliClient().recent_ticks("EURUSD", 30)
@pytest.mark.parametrize(
"tick_time",
[
"2024-01-02T12:00:00+00:00",
1704196800,
],
)
def test_recent_ticks_coerces_string_and_unix_tick_times(
self,
mocker: MockerFixture,
tick_time: str | int,
) -> None:
"""Test recent_ticks accepts string and unix tick timestamps."""
client = MagicMock()
tick = MagicMock()
tick.time = tick_time
client.symbol_info_tick.return_value = tick
expected_end = (
datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
if isinstance(tick_time, str)
else datetime.fromtimestamp(tick_time, tz=UTC)
)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [expected_end],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
Mt5CliClient().recent_ticks("EURUSD", 30)
_, kwargs = client.copy_ticks_from_as_df.call_args
assert kwargs["date_from"] == expected_end - timedelta(seconds=30)
def test_recent_ticks_returns_full_frame_when_count_not_positive(
self,
mocker: MockerFixture,
) -> None:
"""Test non-positive count returns the full range without trimming."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=0,
config=build_config(login=123),
)
assert len(result) == 3
client.copy_ticks_from_as_df.assert_not_called()
client.copy_ticks_range_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
date_to=end,
flags=1,
)
class TestMinimumMargins:
"""Tests for minimum_margins helper."""
def test_minimum_margins_shape(
self,
mocker: MockerFixture,
) -> None:
"""Test minimum_margins returns the expected summary columns."""
client = MagicMock()
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
client.symbol_info.return_value = sym
client.account_info.return_value = account
client.symbol_info_tick.return_value = tick
client.order_calc_margin.side_effect = [12.5, 12.4]
client.mt5.ORDER_TYPE_BUY = 0
client.mt5.ORDER_TYPE_SELL = 1
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = minimum_margins("EURUSD", config=build_config(login=123))
pd.testing.assert_frame_equal(
result,
pd.DataFrame([
{
"symbol": "EURUSD",
"account_currency": "USD",
"volume_min": 0.01,
"buy_margin": 12.5,
"sell_margin": 12.4,
}
]),
)
client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
+108
View File
@@ -21,8 +21,10 @@ from mt5cli.utils import (
TIMEFRAME_MAP,
TIMEFRAME_TYPE,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_request,
parse_tick_flags,
@@ -130,6 +132,112 @@ class TestExportDataframe:
export_dataframe(sample_df, tmp_path / "out.txt", "xml")
class TestExportDataframeToSqlite:
"""Tests for export_dataframe_to_sqlite."""
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
"""Test append mode keeps prior rows in the SQLite table."""
output = tmp_path / "append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
"""Test deduplication keeps the latest ROWID for key columns."""
output = tmp_path / "dedup.db"
first = pd.DataFrame({
"symbol": ["EURUSD", "EURUSD"],
"time": ["2024-01-01", "2024-01-01"],
"bid": [1.0, 1.1],
})
second = pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
})
export_dataframe_to_sqlite(
first,
output,
"ticks",
if_exists=IfExists.REPLACE,
deduplicate_on=("symbol", "time"),
)
export_dataframe_to_sqlite(
second,
output,
"ticks",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "time"),
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, time, bid FROM ticks",
conn,
)
pd.testing.assert_frame_equal(
result.reset_index(drop=True),
pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
}),
)
def test_default_if_exists_appends_without_dropping_rows(
self,
tmp_path: Path,
) -> None:
"""Test the default append mode keeps prior rows."""
output = tmp_path / "default-append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items")
export_dataframe_to_sqlite(second, output, "items")
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_writes_index_with_label(self, tmp_path: Path) -> None:
"""Test optional index export with a custom label."""
output = tmp_path / "index.db"
frame = pd.DataFrame(
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
)
export_dataframe_to_sqlite(
frame,
output,
"margins",
if_exists=IfExists.REPLACE,
index=True,
index_label="symbol",
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, value FROM margins",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
)
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
Generated
+1 -1
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.0"
version = "0.5.0"
source = { editable = "." }
dependencies = [
{ name = "click" },