Compare commits

...

6 Commits

Author SHA1 Message Date
Daichi Narushima 5b1d54bfe9 Add resilient multi-account orchestration helpers (#22)
* Add SDK orchestration helpers for resilient multi-account collection

- collect_latest_rates_for_accounts_with_retries(): exponential-backoff
  retries around collect_latest_rates_for_accounts(), retrying only
  Mt5TradingError/Mt5RuntimeError and re-raising on exhaustion.
- resolve_account_spec()/resolve_account_specs() and
  substitute_env_placeholders(): merge explicit overrides over AccountSpec
  fields and expand ${ENV_VAR} placeholders, raising ValueError on missing
  variables.
- ThrottledHistoryUpdater: monotonic-clock throttled wrapper around
  update_history() with should_update()/update() and opt-in suppress_errors.
- load_rate_series_by_granularity(): rate-series loader keyed by
  (symbol | None, granularity_name).
- Export new APIs, add unit tests (100% coverage), and document in README
  and docs/api.

* chore: bump version from 0.5.1 to 0.5.3 (#24)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: resolve leftover merge conflict markers in version files

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address PR review feedback on SDK orchestration helpers

- Use single-pass env substitution to avoid TOCTOU KeyError
- Apply backoff_base to all retry delays (backoff_base ** (attempt + 1))
- Preserve integer logins in resolve_account_spec; hide login in repr
- Fix docs examples (env ordering, while True loop, backoff comment)
- Parametrize suppress_errors tests for MT5 and SQLite errors

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-10 00:15:07 +09:00
Daichi Narushima ad9e513253 [codex] Guard dedup scopes by written columns (#23)
* Guard dedup scopes by written columns

* Address dedup scope review feedback

* Remove legacy dedup scope support

* Remove stale legacy descriptions

* chore: bump version from 0.5.1 to 0.5.2
2026-06-09 23:27:54 +09:00
Daichi Narushima 334f01b647 chore: bump version from 0.5.0 to 0.5.1 (#21) 2026-06-09 15:52:32 +09:00
Daichi Narushima 1b69e8f08e Add generic MT5 rate-loading SDK APIs for downstream reuse (#20) 2026-06-09 15:37:24 +09:00
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
Daichi Narushima b2bb2ad0a0 Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names.

Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add reusable export, tick-window, and margin helpers for downstream tools.

Expose SQLite append/dedup export, recent tick retrieval, and minimum margin
summary through the SDK and CLI so projects like mteor can depend on mt5cli
instead of duplicating MT5 data plumbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version to 0.4.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for rate view resolution and SDK helpers.

Harden SQLite read-only connections, tighten view discovery, improve recent_ticks
fetch efficiency, default SQLite export to append, and expand tests and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix read-only SQLite URI construction on Windows.

Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 03:29:03 +09:00
16 changed files with 4199 additions and 75 deletions
+38 -22
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -127,6 +133,16 @@ update_history_with_config(
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`/`sqlite3.Error` and let the caller decide logging.
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
## Requirements
+98
View File
@@ -129,3 +129,101 @@ when required columns are missing.
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing naming
schemes:
```python
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
```
Resolution rules:
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
are stored for the same symbol.
- When multiple naming candidates apply, prefers an existing managed
`rate_*__*` view from the candidate list.
- Falls back to single-timeframe naming when the database path is missing or
`rates` metadata is unavailable.
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
### Multi-series rate loading
For loading many rate series at once, build neutral `RateTarget` pairs and load
them from SQLite in one call. View names are resolved via the same
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
```python
from pathlib import Path
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
```
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
row-major order, normalizing timeframe names such as `"M1"` to their integer
values; set `allow_missing_symbol=True` to address series solely by
`explicit_tables` (targets carry `symbol=None`).
- `resolve_rate_tables()` maps targets to table or view names and validates that
any `explicit_tables` count matches the target count. Pass
`require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or managed view is missing. When
`explicit_tables` is provided, names are returned as-is and
`require_existing` is ignored.
- `load_rate_series_from_sqlite()` returns a mapping keyed by
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
requires existing managed `rate_*` compatibility views and raises
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
are rejected.
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
loads the series, and rekeys the result by granularity name to avoid
converting integer timeframes downstream:
```python
from mt5cli import load_rate_series_by_granularity
series = load_rate_series_by_granularity(
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
)
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
```
+20
View File
@@ -65,12 +65,18 @@ from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
Dataset,
IfExists,
Mt5CliClient,
collect_history,
copy_rates_range,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically
rates = copy_rates_range(
@@ -86,6 +92,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite
collect_history(
Path("history.db"),
+80
View File
@@ -1,3 +1,83 @@
# SDK Module
::: mt5cli.sdk
## Resilient multi-account orchestration
The SDK ships strategy-agnostic helpers for building long-running collectors on
top of the read-only client. None of them depend on a particular trading
application.
### Retrying transient rate collection
`collect_latest_rates_for_accounts_with_retries()` wraps
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
failure is re-raised once `retry_count` is exhausted.
```python
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
rates = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1", "H1"],
count=500,
retry_count=3,
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
)
```
### Resolving credentials and `${ENV_VAR}` placeholders
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
secrets out of plan/config files. A missing environment variable raises
`ValueError`.
```python
import os
from mt5cli import AccountSpec, resolve_account_specs
os.environ["MT5_LOGIN"] = "12345"
os.environ["MT5_PASSWORD"] = "secret"
accounts = [
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
]
resolved = resolve_account_specs(accounts, server="Broker-Demo")
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
```
### Throttled incremental history updates
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
between successful runs (using a monotonic clock), so an application loop can
call it every iteration without over-fetching.
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db",
datasets={Dataset.rates},
timeframes=["M1"],
interval_seconds=60, # <= 0 updates on every call
)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
while True:
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
# ... do other work; break when shutting down ...
finally:
client.shutdown()
```
By default `Mt5TradingError`, `Mt5RuntimeError`, and `sqlite3.Error` propagate so
the caller controls logging; pass `suppress_errors=True` to swallow them and
return `False` without advancing the throttle.
+43 -14
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -22,13 +23,23 @@ pip install mt5cli
## Programmatic usage / SDK usage
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results.
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe
from mt5cli import (
Mt5CliClient,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers
rates = copy_rates_range(
@@ -39,10 +50,21 @@ rates = copy_rates_range(
)
export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
@@ -58,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start
```bash
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range |
### Ticks
| Command | Description |
| ------------- | ------------------------------ |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| Command | Description |
| -------------- | ----------------------------------- |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a trailing window |
### Information
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume margin summary |
| `market-book` | Export market depth (order book) |
### Trading
| Command | Description |
| ---------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| Command | Description |
| ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`.
+76 -1
View File
@@ -2,11 +2,31 @@
from importlib.metadata import version
from .history import (
RateTarget,
build_rate_targets,
build_rate_view_name,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
)
from .sdk import (
AccountSpec,
Mt5CliClient,
ThrottledHistoryUpdater,
account_info,
build_config,
collect_history,
collect_latest_rates,
collect_latest_rates_for_accounts,
collect_latest_rates_for_accounts_with_retries,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -15,9 +35,19 @@ from .sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_session,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
resolve_account_spec,
resolve_account_specs,
substitute_env_placeholders,
symbol_info,
symbol_info_tick,
symbols,
@@ -28,17 +58,38 @@ from .sdk import (
from .sdk import (
version as mt5_version,
)
from .utils import Dataset, IfExists, detect_format, export_dataframe
from .utils import (
TICK_FLAG_MAP,
TIMEFRAME_MAP,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_tick_flags,
parse_timeframe,
)
__version__ = version(__package__) if __package__ else None
__all__ = [
"TICK_FLAG_MAP",
"TIMEFRAME_MAP",
"AccountSpec",
"Dataset",
"IfExists",
"Mt5CliClient",
"RateTarget",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"build_rate_targets",
"build_rate_view_name",
"collect_history",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
@@ -46,13 +97,37 @@ __all__ = [
"copy_ticks_range",
"detect_format",
"export_dataframe",
"export_dataframe_to_sqlite",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version",
"orders",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"positions",
"recent_history_deals",
"recent_ticks",
"resolve_account_spec",
"resolve_account_specs",
"resolve_history_datasets",
"resolve_history_tick_flags",
"resolve_history_timeframes",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
+104
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
)
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command()
def rates_range(
ctx: typer.Context,
@@ -300,6 +325,44 @@ def ticks_range(
)
@app.command()
def ticks_recent(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
seconds: Annotated[
float,
typer.Option(help="Lookback window in seconds."),
],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
count: Annotated[
int,
typer.Option(help="Maximum number of ticks to return."),
] = 10000,
flags: Annotated[
int,
typer.Option(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
),
)
@app.command()
def account_info(ctx: typer.Context) -> None:
"""Export account information."""
@@ -335,6 +398,16 @@ def symbol_info(
_execute_export(ctx, lambda: client.symbol_info(symbol))
@app.command()
def minimum_margins(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
@app.command()
def orders(
ctx: typer.Context,
@@ -427,6 +500,37 @@ def history_deals(
)
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
+676 -11
View File
@@ -4,8 +4,10 @@ from __future__ import annotations
import logging
import sqlite3
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Literal
from pathlib import Path
from typing import TYPE_CHECKING, Literal, cast
import pandas as pd
@@ -19,7 +21,7 @@ from .utils import (
)
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from collections.abc import Callable, Mapping, Sequence
from pdmt5 import Mt5DataClient
@@ -122,9 +124,641 @@ def build_rate_view_name(
return f"rate_{symbol}__{granularity}_{timeframe}"
SqliteConnOrPath = sqlite3.Connection | Path | str
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
value = identifier.strip()
if not value:
msg = f"SQLite {kind} name must not be empty."
raise ValueError(msg)
return value
def _open_history_connection(
conn_or_path: SqliteConnOrPath | None,
) -> tuple[sqlite3.Connection | None, bool]:
"""Open a read-only SQLite connection when given a path.
Returns:
A connection and whether the caller should close it. When ``conn_or_path``
is None or the path does not exist, returns ``(None, False)`` without
creating a database file.
"""
if conn_or_path is None:
return None, False
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
return None, False
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _open_existing_sqlite_database(
conn_or_path: SqliteConnOrPath,
) -> tuple[sqlite3.Connection, bool]:
"""Open a read-only SQLite database or reuse an existing connection.
Returns:
Tuple of connection and whether the caller should close it.
Raises:
ValueError: If the database path does not exist or is not a file.
"""
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
if not path.is_file():
msg = f"SQLite database path is not a file: {path}"
raise ValueError(msg)
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _validate_rate_load_request(table: str, count: int | None) -> str:
table_name = _require_non_empty_identifier(table, "table or view")
if count is not None and count <= 0:
msg = "count must be positive when provided."
raise ValueError(msg)
return table_name
def _ensure_rate_columns(columns: set[str], table: str) -> None:
if not columns:
msg = f"SQLite table or view not found: {table}"
raise ValueError(msg)
if "time" not in columns:
msg = f"SQLite table or view {table!r} must include a time column."
raise ValueError(msg)
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
msg = (
f"SQLite table or view {table!r} must include close, "
"or both ask and bid columns."
)
raise ValueError(msg)
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
parsed = frame["time"].map(parse_sqlite_timestamp)
if parsed.isna().any():
msg = f"SQLite table or view {table!r} contains unparsable time values."
raise ValueError(msg)
result = frame.drop(columns=["time"])
result.index = pd.DatetimeIndex(parsed, name="time")
return result.sort_index(kind="stable")
def load_rate_data_from_connection(
connection: sqlite3.Connection,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite table or view.
Args:
connection: Open SQLite connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
Raises:
ValueError: If inputs, schema, timestamps are invalid, or the table
or view contains no rows.
"""
table_name = _validate_rate_load_request(table, count)
columns = get_table_columns(connection, table_name)
_ensure_rate_columns(columns, table_name)
quoted_table = quote_sqlite_identifier(table_name)
if count is None:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
connection,
),
)
else:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
connection,
params=(count,),
),
)
if frame.empty:
msg = f"SQLite table or view {table_name!r} contains no rows."
raise ValueError(msg)
return _parse_rate_time_index(frame, table_name)
def load_rate_data(
conn_or_path: SqliteConnOrPath,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite database path or connection.
Args:
conn_or_path: SQLite database path or open connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
"""
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
return load_rate_data_from_connection(conn, table, count=count)
finally:
if should_close:
conn.close()
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
"""Return distinct timeframe counts per symbol from the normalized rates table."""
columns = get_table_columns(conn, Dataset.rates.table_name)
if not {"symbol", "timeframe"}.issubset(columns):
return None
rows = conn.execute(
"SELECT symbol, COUNT(DISTINCT timeframe) FROM rates GROUP BY symbol",
).fetchall()
return {str(symbol): int(count) for symbol, count in rows}
def _load_existing_rate_views(conn: sqlite3.Connection) -> set[str]:
"""Return mt5cli-managed ``rate_*__*`` compatibility view names."""
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'view' AND name GLOB 'rate_*__*'",
).fetchall()
return {str(row[0]) for row in rows}
def _rate_view_name_candidates(
*,
symbol: str,
granularity: str,
granularity_count: int,
timeframe: int,
) -> list[str]:
"""Return candidate view names in preference order."""
single = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=1,
timeframe=timeframe,
)
if granularity_count <= 1:
return [single]
multi = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=granularity_count,
timeframe=timeframe,
)
return [multi, single]
def _resolve_rate_view_name_from_context(
*,
symbol: str,
timeframe: int,
granularity_name: str,
timeframe_counts: dict[str, int] | None,
existing_views: set[str],
require_existing: bool = False,
) -> str:
"""Resolve one rate view name using preloaded SQLite metadata.
Returns:
Preferred mt5cli-managed rate compatibility view name.
Raises:
ValueError: If ``require_existing`` is True and no managed view exists.
"""
if timeframe_counts is None or symbol not in timeframe_counts:
candidates = [
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
),
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=2,
timeframe=timeframe,
),
]
else:
candidates = _rate_view_name_candidates(
symbol=symbol,
granularity=granularity_name,
granularity_count=timeframe_counts[symbol],
timeframe=timeframe,
)
for candidate in candidates:
if candidate in existing_views:
return candidate
if require_existing:
msg = (
f"No rate compatibility view exists for symbol {symbol!r} "
f"and granularity {granularity_name!r}; "
f"candidates: {', '.join(candidates)}."
)
raise ValueError(msg)
return candidates[0]
def resolve_rate_view_name(
conn_or_path: SqliteConnOrPath | None,
symbol: str,
granularity: str,
*,
require_existing: bool = False,
) -> str:
"""Resolve the mt5cli-managed rate compatibility view name.
Args:
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, the deterministic
default view name is returned without creating a database file.
symbol: Symbol stored in the normalized ``rates`` table.
granularity: Timeframe name (for example ``M1``) or integer string.
require_existing: When True, require the database and a managed view to exist.
Returns:
View name such as ``rate_EURUSD__1`` or ``rate_EURUSD__M1_1``.
Raises:
ValueError: If ``require_existing`` is True and the database or view is missing.
"""
timeframe = parse_timeframe(granularity)
granularity_name = resolve_granularity_name(timeframe)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
return build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
)
return _resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=granularity_name,
timeframe_counts=_load_rates_timeframe_counts(conn),
existing_views=_load_existing_rate_views(conn),
require_existing=require_existing,
)
finally:
if should_close and conn is not None:
conn.close()
def resolve_rate_view_names(
conn_or_path: SqliteConnOrPath | None,
symbols: Sequence[str],
granularities: Sequence[str],
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve rate compatibility view names for symbol and granularity pairs.
Args:
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, deterministic
default view names are returned without creating a database file.
symbols: Symbols stored in the normalized ``rates`` table.
granularities: Timeframe names (for example ``M1``) or integer strings.
require_existing: When True, require the database and managed views to exist.
Returns:
View names in row-major order: every ``granularity`` for the first
symbol, then every granularity for the next symbol, and so on.
"""
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
return [
resolve_rate_view_name(
conn_or_path,
symbol,
granularity,
require_existing=require_existing,
)
for symbol in symbols
for granularity in granularities
]
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for symbol in symbols:
for granularity in granularities:
timeframe = parse_timeframe(granularity)
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
@dataclass(frozen=True)
class RateTarget:
"""A single rate series identified by symbol and timeframe.
Attributes:
symbol: MT5 symbol name, or None when the rate series is addressed only
by an explicit table (for example a custom SQLite view).
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
"""
symbol: str | None
timeframe: int | str
def __post_init__(self) -> None:
"""Normalize accepted timeframe aliases to the stored integer value."""
if not isinstance(self.timeframe, int):
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
@property
def timeframe_int(self) -> int:
"""Return the timeframe as its integer MT5 value."""
return cast("int", self.timeframe)
def build_rate_targets(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
allow_missing_symbol: bool = False,
) -> list[RateTarget]:
"""Build rate targets for every symbol and timeframe combination.
Args:
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each timeframe instead of raising.
Returns:
Targets in row-major order: every timeframe for the first symbol, then
every timeframe for the next symbol, and so on.
Raises:
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
``allow_missing_symbol`` is False.
"""
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if not symbols:
if not allow_missing_symbol:
msg = "At least one symbol is required."
raise ValueError(msg)
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
return [
RateTarget(symbol=symbol, timeframe=tf)
for symbol in symbols
for tf in timeframes
]
def resolve_rate_tables(
conn_or_path: SqliteConnOrPath | None,
targets: Sequence[RateTarget],
explicit_tables: Sequence[str] | None = None,
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve SQLite table or view names for rate targets.
Args:
conn_or_path: SQLite database path or open connection. May be None when
``explicit_tables`` is provided, or when ``require_existing`` is
False and deterministic default view names are sufficient.
targets: Rate targets to resolve.
explicit_tables: Optional explicit table or view names. When provided,
they are used as-is and must match the number of targets.
require_existing: When True, require the database and managed views to
exist for each symbol target. Ignored when ``explicit_tables`` is
provided.
Returns:
Table or view names aligned with ``targets``.
Raises:
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
match the target count, a target without a symbol is resolved
without an explicit table, or ``require_existing`` is True and the
database or a managed view is missing.
"""
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is not None:
tables = list(explicit_tables)
if len(tables) != len(target_list):
msg = (
f"Expected {len(target_list)} explicit table(s) "
f"to match the targets, got {len(tables)}."
)
raise ValueError(msg)
return tables
if any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
timeframe_counts = None
existing_views: set[str] = set()
else:
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for target in target_list:
symbol = cast("str", target.symbol)
timeframe = target.timeframe_int
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget],
count: int,
explicit_tables: Sequence[str] | None = None,
) -> dict[tuple[str | None, int], pd.DataFrame]:
"""Load multiple rate series from a SQLite database.
Args:
conn_or_path: SQLite database path or open connection.
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair
must be unique.
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching targets.
When omitted, managed ``rate_*`` compatibility views must already
exist in the database.
Returns:
Mapping keyed by ``(symbol, timeframe_int)`` to each rate DataFrame.
Raises:
ValueError: If ``count`` is not positive, targets are empty, duplicate
``(symbol, timeframe_int)`` pairs are present, or table resolution
fails.
"""
if count <= 0:
msg = "count must be positive."
raise ValueError(msg)
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is None and any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
seen_keys: set[tuple[str | None, int]] = set()
for target in target_list:
key = (target.symbol, target.timeframe_int)
if key in seen_keys:
symbol_repr = repr(target.symbol)
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
raise ValueError(msg)
seen_keys.add(key)
tables = (
resolve_rate_tables(None, target_list, explicit_tables)
if explicit_tables is not None
else None
)
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
resolved_tables = tables or resolve_rate_tables(
conn,
target_list,
require_existing=True,
)
return {
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
conn,
table,
count=count,
)
for target, table in zip(target_list, resolved_tables, strict=True)
}
finally:
if should_close:
conn.close()
def load_rate_series_by_granularity(
conn_or_path: SqliteConnOrPath,
symbols: Sequence[str],
granularities: Sequence[int | str],
count: int,
*,
explicit_tables: Sequence[str] | None = None,
allow_missing_symbol: bool = False,
) -> dict[tuple[str | None, str], pd.DataFrame]:
"""Load rate series keyed by symbol and string granularity name.
Builds targets with :func:`build_rate_targets` and loads them with
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
name (for example ``M1``) instead of the integer timeframe to reduce
downstream boilerplate.
Args:
conn_or_path: SQLite database path or open connection.
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching the
built targets in row-major order. Required when symbols are omitted.
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each granularity instead of raising.
Returns:
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
table resolution fails, or duplicate targets are present.
"""
targets = build_rate_targets(
symbols,
granularities,
allow_missing_symbol=allow_missing_symbol,
)
series = load_rate_series_from_sqlite(
conn_or_path,
targets,
count,
explicit_tables=explicit_tables,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in series.items()
}
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
"""Return existing SQLite columns for a table."""
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
quoted_table = quote_sqlite_identifier(table)
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
return {str(row[1]) for row in rows}
@@ -404,7 +1038,20 @@ def drop_duplicates_in_table(
)
DedupScope = tuple[str, tuple[object, ...]]
@dataclass(frozen=True)
class DedupScope:
"""Scoped deduplication predicate and the columns it references.
Attributes:
where: SQL predicate appended to the duplicate-removal query.
params: Parameters bound to the scope predicate.
required_columns: Columns that must be present in the written table for
the scope to run.
"""
where: str
params: tuple[object, ...]
required_columns: frozenset[str]
def _record_dedup_scope(
@@ -412,17 +1059,25 @@ def _record_dedup_scope(
dataset: Dataset,
scope_where: str,
scope_params: tuple[object, ...],
required_columns: frozenset[str],
) -> None:
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
dedup_scopes.setdefault(dataset, []).append(
DedupScope(scope_where, scope_params, required_columns),
)
def deduplicate_history_tables(
conn: sqlite3.Connection,
written_columns: dict[Dataset, set[str]],
written_tables: set[Dataset],
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
) -> None:
"""Deduplicate appended history tables by stable identifiers."""
"""Deduplicate appended history tables by stable identifiers.
Scopes whose required columns are not present in the written table are
skipped. If all scopes for a dataset are skipped, the table receives one
unscoped deduplication pass instead.
"""
cursor = conn.cursor()
for dataset in written_tables:
columns = written_columns.get(dataset, set())
@@ -441,16 +1096,19 @@ def deduplicate_history_tables(
table,
)
continue
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
raw_scopes: Sequence[DedupScope] = (
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
)
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
if scopes:
for scope_where, scope_params in scopes:
for scope in scopes:
drop_duplicates_in_table(
cursor,
table,
list(keys),
keep="last",
scope_where=scope_where,
scope_params=scope_params,
scope_where=scope.where,
scope_params=scope.params,
)
continue
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
@@ -817,6 +1475,7 @@ def _write_incremental_rates(
Dataset.rates,
"symbol = ? AND timeframe = ? AND time >= ?",
(symbol, timeframe, start_date),
frozenset({"symbol", "timeframe", "time"}),
)
@@ -855,6 +1514,7 @@ def _write_incremental_ticks(
Dataset.ticks,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -893,6 +1553,7 @@ def _write_incremental_history_orders(
Dataset.history_orders,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -946,6 +1607,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_by_symbol[symbol, None]),
frozenset({"symbol", "time"}),
)
if "type" in columns:
_record_dedup_scope(
@@ -953,6 +1615,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
(account_event_start,),
frozenset({"type", "time"}),
)
if "type" not in columns and "symbol" in columns:
_record_dedup_scope(
@@ -960,6 +1623,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"(symbol IS NULL OR symbol = '') AND time >= ?",
(account_event_start,),
frozenset({"symbol", "time"}),
)
return
start_by_symbol = load_incremental_start_datetimes(
@@ -987,6 +1651,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
+864 -8
View File
File diff suppressed because it is too large Load Diff
+55 -10
View File
@@ -2,16 +2,18 @@
from __future__ import annotations
import importlib
import json
import sqlite3
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard, cast
from typing import TYPE_CHECKING, Any, TypeGuard
import click
if TYPE_CHECKING:
from collections.abc import Sequence
import pandas as pd
# ---------------------------------------------------------------------------
@@ -260,6 +262,50 @@ def detect_format(
raise ValueError(msg)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
table_name: str = "data",
*,
if_exists: IfExists = IfExists.APPEND,
index: bool = False,
index_label: str | None = None,
deduplicate_on: Sequence[str] | None = None,
) -> None:
"""Write a DataFrame to SQLite with configurable append and deduplication.
Args:
df: DataFrame to export.
output_path: SQLite database path.
table_name: Target table name.
if_exists: Conflict behavior when the table already exists.
index: Whether to write the DataFrame index as a column.
index_label: Column name for the index when ``index=True``.
deduplicate_on: Optional key columns to deduplicate after writing,
keeping the latest ``ROWID`` per key group. Deduplication scans the
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=index,
index_label=index_label,
)
if deduplicate_on:
from .history import drop_duplicates_in_table # noqa: PLC0415
drop_duplicates_in_table(
conn.cursor(),
table_name,
list(deduplicate_on),
keep="last",
)
conn.commit()
def export_dataframe(
df: pd.DataFrame,
output_path: Path,
@@ -289,14 +335,13 @@ def export_dataframe(
elif output_format == "parquet":
df.to_parquet(output_path, index=False)
elif output_format == "sqlite3":
sqlite3 = cast("Any", importlib.import_module("sqlite3"))
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists="replace",
index=False,
)
export_dataframe_to_sqlite(
df,
output_path,
table_name,
if_exists=IfExists.REPLACE,
index=False,
)
else:
msg = f"Unsupported output format: {output_format}"
raise ValueError(msg)
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.2"
version = "0.5.3"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
+173 -1
View File
@@ -6,7 +6,7 @@ import json
import logging
import re
import sqlite3
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50,
)
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range(
self,
tmp_path: Path,
@@ -316,6 +351,65 @@ class TestCommands:
flags=2,
)
def test_ticks_recent(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test ticks-recent command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"ticks-recent",
"--symbol",
"EURUSD",
"--seconds",
"120",
"--date-to",
"2024-01-02",
"--count",
"500",
"--flags",
"ALL",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
def test_minimum_margins(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test minimum-margins command."""
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
mock_client.symbol_info.return_value = sym
mock_client.account_info.return_value = account
mock_client.symbol_info_tick.return_value = tick
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
mock_client.mt5.ORDER_TYPE_BUY = 0
mock_client.mt5.ORDER_TYPE_SELL = 1
output = tmp_path / "out.csv"
result = runner.invoke(
app,
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
)
assert result.exit_code == 0, result.output
mock_client.symbol_info.assert_called_once_with("EURUSD")
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
def test_orders(
self,
tmp_path: Path,
@@ -392,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version(
self,
tmp_path: Path,
+930 -3
View File
File diff suppressed because it is too large Load Diff
+932 -3
View File
@@ -4,24 +4,32 @@ from __future__ import annotations
import logging
import sqlite3
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, NamedTuple, cast
from unittest.mock import MagicMock, call
import pandas as pd
import pytest
from pdmt5 import Mt5RuntimeError, Mt5TradingError
from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from pathlib import Path
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import sdk
from mt5cli.history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.sdk import (
AccountSpec,
Mt5CliClient,
ThrottledHistoryUpdater,
account_info,
build_config,
collect_history,
collect_latest_rates,
collect_latest_rates_for_accounts,
collect_latest_rates_for_accounts_with_retries,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -30,9 +38,19 @@ from mt5cli.sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_session,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
resolve_account_spec,
resolve_account_specs,
substitute_env_placeholders,
symbol_info,
symbol_info_tick,
symbols,
@@ -43,6 +61,32 @@ from mt5cli.sdk import (
)
from mt5cli.utils import Dataset
class _TerminalInfo(NamedTuple):
connected: bool
path: str
class _AccountInfo(NamedTuple):
login: int
limits: dict[str, object]
class _MissingSummaryMethodClient:
def version(self) -> tuple[int, int, int]:
return (5, 0, 1)
def terminal_info(self) -> dict[str, bool]:
return {"connected": True}
def symbols_total(self) -> int:
return 42
class _NonCallableSummaryMethodClient:
version = (5, 0, 1)
_DEALS_FIXTURE: dict[str, list[object]] = {
"ticket": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
"position_id": [100, 100, 100, 0, 200, 200, 300, 400, 400, 500, 500, 600, 600, 600],
@@ -230,6 +274,32 @@ class TestConnectionLifecycle:
client = Mt5CliClient()
client.__exit__(None, None, None)
def test_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test injected connected clients are not initialized or shut down."""
connected = MagicMock()
connected.account_info_as_df.return_value = pd.DataFrame({"a": [1]})
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
with Mt5CliClient.from_connected_client(connected) as client:
result = client.account_info()
assert result.to_dict("list") == {"a": [1]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
connected.account_info_as_df.assert_called_once()
after_exit = client.terminal_info()
assert after_exit.to_dict("list") == {"b": [2]}
connected.terminal_info_as_df.assert_called_once()
def test_constructor_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test constructor injection has the same non-owning lifecycle."""
connected = MagicMock()
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
client = Mt5CliClient(client=connected)
with client:
result = client.terminal_info()
assert result.to_dict("list") == {"b": [2]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
class TestModuleFunctions:
"""Tests for module-level SDK wrappers."""
@@ -269,6 +339,7 @@ class TestModuleFunctions:
(last_error, (), "last_error_as_df"),
(symbol_info_tick, ("EURUSD",), "symbol_info_tick_as_df"),
(market_book, ("EURUSD",), "market_book_get_as_df"),
(latest_rates, ("EURUSD", "M1", 10), "copy_rates_from_pos_as_df"),
],
)
def test_module_functions_delegate(
@@ -352,6 +423,214 @@ class TestMt5CliClient:
assert isinstance(df, pd.DataFrame)
mock_client.copy_rates_range_as_df.assert_called_once()
def test_latest_rates_delegates_to_copy_rates_from_pos(
self,
mock_client: MagicMock,
) -> None:
"""Test latest_rates is a convenience wrapper for positional rates."""
Mt5CliClient().latest_rates("EURUSD", "M1", 5, start_pos=2)
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="EURUSD",
timeframe=1,
start_pos=2,
count=5,
)
def test_latest_rates_rejects_non_positive_count(self) -> None:
"""Test latest_rates validates count."""
with pytest.raises(ValueError, match="count must be positive"):
Mt5CliClient().latest_rates("EURUSD", "M1", 0)
def test_collect_latest_rates_returns_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test multi-target latest rate collection."""
result = collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
assert set(result) == {
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
}
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
def test_collect_latest_rates_uses_single_transient_connection(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test module helper opens one connection for all target pairs."""
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
mt5_data_client.assert_called_once()
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
mock_client.copy_rates_from_pos_as_df.assert_has_calls(
[
call(symbol="EURUSD", timeframe=1, start_pos=0, count=3),
call(symbol="EURUSD", timeframe=16385, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=1, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=16385, start_pos=0, count=3),
],
)
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
([], ["M1"], "At least one symbol"),
(["EURUSD"], [], "At least one timeframe"),
],
)
def test_collect_latest_rates_rejects_empty_inputs(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test multi-target latest rate input validation."""
with pytest.raises(ValueError, match=match):
Mt5CliClient().collect_latest_rates(symbols, timeframes, count=1)
def test_recent_history_deals_uses_trailing_window(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals calculates date_from from hours."""
result = recent_history_deals(
6,
date_to="2024-01-02T00:00:00+00:00",
group="*",
symbol="EURUSD",
)
assert isinstance(result, pd.DataFrame)
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group="*",
symbol="EURUSD",
ticket=None,
position=None,
)
def test_recent_history_deals_defaults_date_to_now(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals uses current UTC time when date_to is omitted."""
before = datetime.now(UTC)
recent_history_deals(1.0)
after = datetime.now(UTC)
call_kwargs = mock_client.history_deals_get_as_df.call_args.kwargs
assert before <= call_kwargs["date_to"] <= after
assert call_kwargs["date_from"] == call_kwargs["date_to"] - timedelta(hours=1)
def test_recent_history_deals_rejects_non_positive_hours(self) -> None:
"""Test recent_history_deals validates hours."""
with pytest.raises(ValueError, match="hours must be positive"):
Mt5CliClient().recent_history_deals(0)
def test_mt5_summary_returns_status_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary calls raw terminal/account status methods."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = {"connected": True}
mock_client.account_info.return_value = {"login": 123}
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True},
"account_info": {"login": 123},
"symbols_total": 42,
}
def test_mt5_summary_normalizes_namedtuple_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary returns structured plain Python values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True, "path": "terminal.exe"},
"account_info": {
"login": 123,
"limits": {"modes": ["netting", "hedging"], "servers": ["demo"]},
},
"symbols_total": 42,
}
def test_mt5_summary_as_df_stringifies_nested_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary_as_df returns export-safe tabular values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
result = mt5_summary_as_df()
assert len(result) == 1
assert result.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"path":"terminal.exe"}',
"account_info": (
'{"limits":{"modes":["netting","hedging"],'
'"servers":["demo"]},"login":123}'
),
"symbols_total": 42,
}
def test_mt5_summary_missing_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is missing."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _MissingSummaryMethodClient()),
)
with pytest.raises(
AttributeError,
match="MT5 client is missing required method: account_info",
):
client.mt5_summary()
def test_mt5_summary_non_callable_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is not callable."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _NonCallableSummaryMethodClient()),
)
with pytest.raises(
TypeError,
match="MT5 client attribute is not callable: version",
):
client.mt5_summary()
class TestCollectHistory:
"""Tests for collect_history SDK function."""
@@ -808,3 +1087,653 @@ class TestUpdateHistory:
)
after = datetime.now(UTC)
assert before <= captured["end"] <= after
class TestRecentTicks:
"""Tests for recent_ticks helper."""
def test_recent_ticks_uses_explicit_date_to_window(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks fetches the requested trailing window."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [end],
"bid": [1.0],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=100,
flags="INFO",
config=build_config(login=123),
)
assert isinstance(result, pd.DataFrame)
client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
count=100,
flags=2,
)
client.copy_ticks_range_as_df.assert_not_called()
def test_recent_ticks_uses_latest_tick_when_date_to_omitted(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks anchors the window on the latest tick time."""
client = MagicMock()
tick = MagicMock()
tick.time = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.symbol_info_tick.return_value = tick
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [1, 2],
"bid": [1.0, 1.1],
})
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = Mt5CliClient().recent_ticks("EURUSD", 30, count=2, flags="ALL")
assert len(result) == 2
client.symbol_info_tick.assert_called_once_with("EURUSD")
client.copy_ticks_from_as_df.assert_called_once()
_, kwargs = client.copy_ticks_range_as_df.call_args
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_to"] == tick.time
assert kwargs["date_from"] == tick.time - timedelta(seconds=30)
assert kwargs["flags"] == 1
def test_recent_ticks_rejects_unsupported_tick_time(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks raises when the latest tick time is unsupported."""
client = MagicMock()
tick = MagicMock()
tick.time = object()
client.symbol_info_tick.return_value = tick
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
with pytest.raises(TypeError, match="Unsupported tick time value"):
Mt5CliClient().recent_ticks("EURUSD", 30)
@pytest.mark.parametrize(
"tick_time",
[
"2024-01-02T12:00:00+00:00",
1704196800,
],
)
def test_recent_ticks_coerces_string_and_unix_tick_times(
self,
mocker: MockerFixture,
tick_time: str | int,
) -> None:
"""Test recent_ticks accepts string and unix tick timestamps."""
client = MagicMock()
tick = MagicMock()
tick.time = tick_time
client.symbol_info_tick.return_value = tick
expected_end = (
datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
if isinstance(tick_time, str)
else datetime.fromtimestamp(tick_time, tz=UTC)
)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [expected_end],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
Mt5CliClient().recent_ticks("EURUSD", 30)
_, kwargs = client.copy_ticks_from_as_df.call_args
assert kwargs["date_from"] == expected_end - timedelta(seconds=30)
def test_recent_ticks_returns_full_frame_when_count_not_positive(
self,
mocker: MockerFixture,
) -> None:
"""Test non-positive count returns the full range without trimming."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=0,
config=build_config(login=123),
)
assert len(result) == 3
client.copy_ticks_from_as_df.assert_not_called()
client.copy_ticks_range_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
date_to=end,
flags=1,
)
class TestMinimumMargins:
"""Tests for minimum_margins helper."""
def test_minimum_margins_shape(
self,
mocker: MockerFixture,
) -> None:
"""Test minimum_margins returns the expected summary columns."""
client = MagicMock()
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
client.symbol_info.return_value = sym
client.account_info.return_value = account
client.symbol_info_tick.return_value = tick
client.order_calc_margin.side_effect = [12.5, 12.4]
client.mt5.ORDER_TYPE_BUY = 0
client.mt5.ORDER_TYPE_SELL = 1
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = minimum_margins("EURUSD", config=build_config(login=123))
pd.testing.assert_frame_equal(
result,
pd.DataFrame([
{
"symbol": "EURUSD",
"account_currency": "USD",
"volume_min": 0.01,
"buy_margin": 12.5,
"sell_margin": 12.4,
}
]),
)
client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
class TestMt5Session:
"""Tests for the mt5_session context manager."""
def test_yields_connected_client_and_shuts_down(
self,
mocker: MockerFixture,
) -> None:
"""Test mt5_session connects, yields a client wrapper, and shuts down."""
mock_client = MagicMock()
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
with mt5_session(build_config(path="/opt/mt5/terminal64.exe")) as client:
mock_client.initialize_and_login_mt5.assert_called_once()
assert isinstance(client, Mt5CliClient)
config = mt5_data_client.call_args.kwargs["config"]
assert config.path == "/opt/mt5/terminal64.exe"
mock_client.shutdown.assert_called_once()
def test_default_config_attaches_to_running_terminal(
self,
mocker: MockerFixture,
) -> None:
"""Test mt5_session builds a default config when none is supplied."""
mock_client = MagicMock()
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
with mt5_session():
pass
mt5_data_client.assert_called_once()
mock_client.shutdown.assert_called_once()
class TestAccountSpec:
"""Tests for account configuration helpers."""
def test_repr_omits_password(self) -> None:
"""Test AccountSpec repr does not expose plaintext passwords."""
spec = AccountSpec(symbols=["EURUSD"], login=123, password="secret")
assert "secret" not in repr(spec)
assert "password" not in repr(spec)
@pytest.mark.parametrize(
("login", "expected"),
[
(None, None),
(123, 123),
("", None),
(" ", None),
("456", 456),
],
)
def test_coerce_login(
self,
login: int | str | None,
expected: int | None,
) -> None:
"""Test login values are normalized for account configs."""
assert sdk._coerce_login(login) == expected # type: ignore[reportPrivateUsage]
def test_coerce_login_rejects_non_numeric_string(self) -> None:
"""Test non-numeric login strings raise ValueError."""
with pytest.raises(ValueError, match="invalid literal"):
sdk._coerce_login("abc") # type: ignore[reportPrivateUsage]
class TestCollectLatestRatesForAccounts:
"""Tests for collect_latest_rates_for_accounts."""
def test_merges_results_across_accounts(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test rates are collected and merged for each account group."""
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
accounts = [
AccountSpec(symbols=["EURUSD"], login="123"),
AccountSpec(symbols=["GBPUSD"], login=456),
]
result = collect_latest_rates_for_accounts(accounts, ["M1"], count=2)
assert set(result) == {("EURUSD", 1), ("GBPUSD", 1)}
assert mt5_data_client.call_count == 2
assert mock_client.initialize_and_login_mt5.call_count == 2
assert mock_client.shutdown.call_count == 2
def test_builds_config_from_account_and_base(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test account fields override base_config, empty login falls back."""
configs: list[object] = []
def _record_config(*, config: object) -> MagicMock:
configs.append(config)
return mock_client
mocker.patch("mt5cli.sdk.Mt5DataClient", side_effect=_record_config)
base = build_config(login=999, server="Base-Server", timeout=5000)
accounts = [
AccountSpec(symbols=["EURUSD"], login="", server="Acct-Server"),
]
collect_latest_rates_for_accounts(accounts, ["M1"], count=1, base_config=base)
assert len(configs) == 1
config = cast("Mt5Config", configs[0])
assert config.login == 999
assert config.server == "Acct-Server"
assert config.timeout == 5000
@pytest.mark.parametrize(
("accounts", "timeframes", "count", "match"),
[
([], ["M1"], 1, "At least one account"),
([AccountSpec(symbols=["EURUSD"])], [], 1, "At least one timeframe"),
(
[AccountSpec(symbols=[])],
["M1"],
1,
"Each account requires at least one symbol",
),
(
[AccountSpec(symbols=["EURUSD"])],
["M1"],
0,
"count must be positive",
),
],
)
def test_rejects_invalid_inputs(
self,
accounts: list[AccountSpec],
timeframes: list[str],
count: int,
match: str,
) -> None:
"""Test input validation for account-level rate collection."""
with pytest.raises(ValueError, match=match):
collect_latest_rates_for_accounts(accounts, timeframes, count)
def test_rejects_empty_symbols_before_connecting(
self,
mocker: MockerFixture,
) -> None:
"""Test all account symbols are validated before any MT5 connection."""
mt5_data_client = mocker.patch("mt5cli.sdk.Mt5DataClient")
accounts = [
AccountSpec(symbols=["EURUSD"], login=123),
AccountSpec(symbols=[], login=456),
]
with pytest.raises(
ValueError, match="Each account requires at least one symbol"
):
collect_latest_rates_for_accounts(accounts, ["M1"], count=1)
mt5_data_client.assert_not_called()
class TestCollectLatestRatesForAccountsWithRetries:
"""Tests for collect_latest_rates_for_accounts_with_retries."""
def test_returns_result_on_first_success(self, mocker: MockerFixture) -> None:
"""Test no retry happens when the first attempt succeeds."""
expected = {("EURUSD", 1): pd.DataFrame()}
wrapped = mocker.patch(
"mt5cli.sdk.collect_latest_rates_for_accounts",
return_value=expected,
)
sleep = mocker.patch("mt5cli.sdk.time.sleep")
accounts = [AccountSpec(symbols=["EURUSD"])]
result = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1"],
count=1,
retry_count=3,
)
assert result is expected
assert wrapped.call_count == 1
sleep.assert_not_called()
def test_retries_then_succeeds(self, mocker: MockerFixture) -> None:
"""Test transient MT5 errors are retried with exponential backoff."""
expected = {("EURUSD", 1): pd.DataFrame()}
wrapped = mocker.patch(
"mt5cli.sdk.collect_latest_rates_for_accounts",
side_effect=[
Mt5TradingError("boom"),
Mt5RuntimeError("boom"),
expected,
],
)
sleep = mocker.patch("mt5cli.sdk.time.sleep")
accounts = [AccountSpec(symbols=["EURUSD"])]
result = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1"],
count=1,
retry_count=2,
backoff_base=2,
)
assert result is expected
assert wrapped.call_count == 3
assert sleep.call_args_list == [call(2), call(4)]
def test_reraises_after_exhausting_retries(self, mocker: MockerFixture) -> None:
"""Test the final error is re-raised once retries are exhausted."""
wrapped = mocker.patch(
"mt5cli.sdk.collect_latest_rates_for_accounts",
side_effect=Mt5RuntimeError("boom"),
)
sleep = mocker.patch("mt5cli.sdk.time.sleep")
accounts = [AccountSpec(symbols=["EURUSD"])]
with pytest.raises(Mt5RuntimeError, match="boom"):
collect_latest_rates_for_accounts_with_retries(
accounts,
["M1"],
count=1,
retry_count=2,
)
assert wrapped.call_count == 3
assert sleep.call_count == 2
def test_does_not_retry_unrelated_errors(self, mocker: MockerFixture) -> None:
"""Test non-MT5 errors propagate without retrying."""
wrapped = mocker.patch(
"mt5cli.sdk.collect_latest_rates_for_accounts",
side_effect=ValueError("bad input"),
)
sleep = mocker.patch("mt5cli.sdk.time.sleep")
with pytest.raises(ValueError, match="bad input"):
collect_latest_rates_for_accounts_with_retries(
[AccountSpec(symbols=["EURUSD"])],
["M1"],
count=1,
retry_count=3,
)
assert wrapped.call_count == 1
sleep.assert_not_called()
class TestSubstituteEnvPlaceholders:
"""Tests for ${ENV_VAR} substitution."""
def test_substitutes_known_variables(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test placeholders are replaced with environment values."""
monkeypatch.setenv("MT5_LOGIN", "12345")
monkeypatch.setenv("MT5_SERVER", "Broker-Demo")
assert substitute_env_placeholders("${MT5_LOGIN}") == "12345"
assert substitute_env_placeholders("srv=${MT5_SERVER}!") == "srv=Broker-Demo!"
def test_returns_plain_strings_unchanged(self) -> None:
"""Test strings without placeholders are returned as-is."""
assert substitute_env_placeholders("plain") == "plain"
def test_raises_on_missing_variable(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test a missing environment variable raises a clear error."""
monkeypatch.delenv("MT5_MISSING", raising=False)
with pytest.raises(ValueError, match="'MT5_MISSING' is not set"):
substitute_env_placeholders("${MT5_MISSING}")
class TestResolveAccountSpec:
"""Tests for resolve_account_spec and resolve_account_specs."""
def test_substitutes_env_placeholders_in_account(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test account string fields resolve ${ENV_VAR} placeholders."""
monkeypatch.setenv("MT5_PASSWORD", "secret")
account = AccountSpec(
symbols=["EURUSD"],
login="${MT5_LOGIN}",
password="${MT5_PASSWORD}",
)
monkeypatch.setenv("MT5_LOGIN", "999")
resolved = resolve_account_spec(account)
assert resolved.login == "999"
assert resolved.password == "secret" # noqa: S105
assert resolved.symbols == ["EURUSD"]
def test_explicit_overrides_take_precedence(self) -> None:
"""Test explicit override values win over account fields."""
account = AccountSpec(symbols=["EURUSD"], login=111, server="Acct")
resolved = resolve_account_spec(
account,
login=222,
server="Override",
timeout=5000,
)
assert resolved.login == 222
assert resolved.server == "Override"
assert resolved.timeout == 5000
def test_resolves_string_login_override(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test string login overrides expand ${ENV_VAR} placeholders."""
monkeypatch.setenv("MT5_LOGIN", "777")
account = AccountSpec(symbols=["EURUSD"], login=111)
resolved = resolve_account_spec(account, login="${MT5_LOGIN}")
assert resolved.login == "777"
def test_preserves_integer_login_without_coercion(self) -> None:
"""Test integer logins remain integers after resolution."""
account = AccountSpec(symbols=["EURUSD"], login=111)
resolved = resolve_account_spec(account)
assert resolved.login == 111
assert isinstance(resolved.login, int)
def test_raises_on_missing_env_variable(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test missing environment variables raise ValueError."""
monkeypatch.delenv("MT5_NOPE", raising=False)
account = AccountSpec(symbols=["EURUSD"], server="${MT5_NOPE}")
with pytest.raises(ValueError, match="'MT5_NOPE' is not set"):
resolve_account_spec(account)
def test_resolve_account_specs_applies_to_all(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Test resolve_account_specs resolves every account in order."""
monkeypatch.setenv("MT5_SERVER", "Shared")
accounts = [
AccountSpec(symbols=["EURUSD"], server="${MT5_SERVER}"),
AccountSpec(symbols=["GBPUSD"], server="Fixed"),
]
resolved = resolve_account_specs(accounts, timeout=1000)
assert [a.server for a in resolved] == ["Shared", "Fixed"]
assert all(a.timeout == 1000 for a in resolved)
class TestThrottledHistoryUpdater:
"""Tests for the throttled incremental history updater."""
def test_updates_every_call_when_interval_non_positive(
self,
mocker: MockerFixture,
) -> None:
"""Test interval_seconds <= 0 updates on every call."""
update = mocker.patch("mt5cli.sdk.update_history")
client = MagicMock()
updater = ThrottledHistoryUpdater(output="history.db", interval_seconds=0)
assert updater.update(client, ["EURUSD"]) is True
assert updater.update(client, ["EURUSD"]) is True
assert update.call_count == 2
def test_throttles_within_interval(self, mocker: MockerFixture) -> None:
"""Test updates are skipped until the interval elapses."""
update = mocker.patch("mt5cli.sdk.update_history")
monotonic = mocker.patch("mt5cli.sdk.time.monotonic")
# Calls: set(t=100), check(t=105), check(t=200), set(t=200).
monotonic.side_effect = [100.0, 105.0, 200.0, 200.0]
client = MagicMock()
updater = ThrottledHistoryUpdater(output="history.db", interval_seconds=60)
assert updater.update(client, ["EURUSD"]) is True # first update at t=100
assert updater.update(client, ["EURUSD"]) is False # t=105, throttled
assert updater.update(client, ["EURUSD"]) is True # t=200, elapsed
assert update.call_count == 2
def test_update_passes_expected_arguments(
self,
mocker: MockerFixture,
) -> None:
"""Test update_history is called with the configured arguments."""
update = mocker.patch("mt5cli.sdk.update_history")
client = MagicMock()
updater = ThrottledHistoryUpdater(
output="history.db",
datasets={Dataset.rates},
timeframes=["M1", "H1"],
flags="INFO",
lookback_hours=12.0,
with_views=True,
include_account_events=False,
)
updater.update(client, ["EURUSD", "GBPUSD"])
update.assert_called_once_with(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"],
datasets={Dataset.rates},
timeframes=["M1", "H1"],
flags="INFO",
lookback_hours=12.0,
with_views=True,
include_account_events=False,
)
def test_propagates_errors_by_default(self, mocker: MockerFixture) -> None:
"""Test MT5/SQLite errors propagate and do not advance the throttle."""
mocker.patch(
"mt5cli.sdk.update_history",
side_effect=Mt5RuntimeError("boom"),
)
updater = ThrottledHistoryUpdater(output="history.db")
with pytest.raises(Mt5RuntimeError, match="boom"):
updater.update(MagicMock(), ["EURUSD"])
assert updater.last_update_monotonic is None
@pytest.mark.parametrize(
"error",
[
Mt5RuntimeError("boom"),
Mt5TradingError("trade failed"),
sqlite3.OperationalError("locked"),
],
)
def test_suppresses_errors_when_requested(
self,
mocker: MockerFixture,
error: Exception,
) -> None:
"""Test suppress_errors swallows recoverable errors and returns False."""
mocker.patch(
"mt5cli.sdk.update_history",
side_effect=error,
)
updater = ThrottledHistoryUpdater(
output="history.db",
suppress_errors=True,
)
assert updater.update(MagicMock(), ["EURUSD"]) is False
assert updater.last_update_monotonic is None
+108
View File
@@ -21,8 +21,10 @@ from mt5cli.utils import (
TIMEFRAME_MAP,
TIMEFRAME_TYPE,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_request,
parse_tick_flags,
@@ -130,6 +132,112 @@ class TestExportDataframe:
export_dataframe(sample_df, tmp_path / "out.txt", "xml")
class TestExportDataframeToSqlite:
"""Tests for export_dataframe_to_sqlite."""
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
"""Test append mode keeps prior rows in the SQLite table."""
output = tmp_path / "append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
"""Test deduplication keeps the latest ROWID for key columns."""
output = tmp_path / "dedup.db"
first = pd.DataFrame({
"symbol": ["EURUSD", "EURUSD"],
"time": ["2024-01-01", "2024-01-01"],
"bid": [1.0, 1.1],
})
second = pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
})
export_dataframe_to_sqlite(
first,
output,
"ticks",
if_exists=IfExists.REPLACE,
deduplicate_on=("symbol", "time"),
)
export_dataframe_to_sqlite(
second,
output,
"ticks",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "time"),
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, time, bid FROM ticks",
conn,
)
pd.testing.assert_frame_equal(
result.reset_index(drop=True),
pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
}),
)
def test_default_if_exists_appends_without_dropping_rows(
self,
tmp_path: Path,
) -> None:
"""Test the default append mode keeps prior rows."""
output = tmp_path / "default-append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items")
export_dataframe_to_sqlite(second, output, "items")
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_writes_index_with_label(self, tmp_path: Path) -> None:
"""Test optional index export with a custom label."""
output = tmp_path / "index.db"
frame = pd.DataFrame(
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
)
export_dataframe_to_sqlite(
frame,
output,
"margins",
if_exists=IfExists.REPLACE,
index=True,
index_label="symbol",
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, value FROM margins",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
)
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
Generated
+1 -1
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.2"
version = "0.5.3"
source = { editable = "." }
dependencies = [
{ name = "click" },