Compare commits

...

6 Commits

Author SHA1 Message Date
Daichi Narushima 18df96872b Add closed-bar rate helpers (v0.6.0) (#26)
* Add closed-bar rate helpers and bump version to 0.6.0.

Expose drop_forming_rate_bar and multi-account collectors so downstream apps no longer need count+1 fetches and manual bar trimming.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump pygments to 2.20.0 to fix CVE-2026-4539 ReDoS advisory.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback on closed-bar rate collection.

Validate count and start_pos before MT5 fetches, avoid redundant frame copies, clarify empty-series errors, and expand test coverage.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Include symbol and timeframe in empty closed-rate error messages.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 02:30:48 +09:00
Daichi Narushima 5b1d54bfe9 Add resilient multi-account orchestration helpers (#22)
* Add SDK orchestration helpers for resilient multi-account collection

- collect_latest_rates_for_accounts_with_retries(): exponential-backoff
  retries around collect_latest_rates_for_accounts(), retrying only
  Mt5TradingError/Mt5RuntimeError and re-raising on exhaustion.
- resolve_account_spec()/resolve_account_specs() and
  substitute_env_placeholders(): merge explicit overrides over AccountSpec
  fields and expand ${ENV_VAR} placeholders, raising ValueError on missing
  variables.
- ThrottledHistoryUpdater: monotonic-clock throttled wrapper around
  update_history() with should_update()/update() and opt-in suppress_errors.
- load_rate_series_by_granularity(): rate-series loader keyed by
  (symbol | None, granularity_name).
- Export new APIs, add unit tests (100% coverage), and document in README
  and docs/api.

* chore: bump version from 0.5.1 to 0.5.3 (#24)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: resolve leftover merge conflict markers in version files

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address PR review feedback on SDK orchestration helpers

- Use single-pass env substitution to avoid TOCTOU KeyError
- Apply backoff_base to all retry delays (backoff_base ** (attempt + 1))
- Preserve integer logins in resolve_account_spec; hide login in repr
- Fix docs examples (env ordering, while True loop, backoff comment)
- Parametrize suppress_errors tests for MT5 and SQLite errors

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-10 00:15:07 +09:00
Daichi Narushima ad9e513253 [codex] Guard dedup scopes by written columns (#23)
* Guard dedup scopes by written columns

* Address dedup scope review feedback

* Remove legacy dedup scope support

* Remove stale legacy descriptions

* chore: bump version from 0.5.1 to 0.5.2
2026-06-09 23:27:54 +09:00
Daichi Narushima 334f01b647 chore: bump version from 0.5.0 to 0.5.1 (#21) 2026-06-09 15:52:32 +09:00
Daichi Narushima 1b69e8f08e Add generic MT5 rate-loading SDK APIs for downstream reuse (#20) 2026-06-09 15:37:24 +09:00
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
13 changed files with 3414 additions and 70 deletions
+50 -25
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -50,30 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -129,7 +133,28 @@ update_history_with_config(
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
```python
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
```
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`/`sqlite3.Error` and let the caller decide logging.
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
+65 -2
View File
@@ -133,8 +133,8 @@ The `update_history` SDK path uses the same base tables and optional
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing legacy
naming schemes:
existing SQLite history database without creating files or guessing naming
schemes:
```python
from pathlib import Path
@@ -164,3 +164,66 @@ Resolution rules:
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
### Multi-series rate loading
For loading many rate series at once, build neutral `RateTarget` pairs and load
them from SQLite in one call. View names are resolved via the same
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
```python
from pathlib import Path
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
```
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
row-major order, normalizing timeframe names such as `"M1"` to their integer
values; set `allow_missing_symbol=True` to address series solely by
`explicit_tables` (targets carry `symbol=None`).
- `resolve_rate_tables()` maps targets to table or view names and validates that
any `explicit_tables` count matches the target count. Pass
`require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or managed view is missing. When
`explicit_tables` is provided, names are returned as-is and
`require_existing` is ignored.
- `load_rate_series_from_sqlite()` returns a mapping keyed by
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
requires existing managed `rate_*` compatibility views and raises
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
are rejected.
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
loads the series, and rekeys the result by granularity name to avoid
converting integer timeframes downstream:
```python
from mt5cli import load_rate_series_by_granularity
series = load_rate_series_by_granularity(
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
)
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
```
+100
View File
@@ -1,3 +1,103 @@
# SDK Module
::: mt5cli.sdk
## Resilient multi-account orchestration
The SDK ships strategy-agnostic helpers for building long-running collectors on
top of the read-only client. None of them depend on a particular trading
application.
### Retrying transient rate collection
`collect_latest_rates_for_accounts_with_retries()` wraps
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
failure is re-raised once `retry_count` is exhausted.
```python
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
rates = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1", "H1"],
count=500,
retry_count=3,
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
)
```
### Latest closed rate bars
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
row. `collect_latest_closed_rates_for_accounts()` fetches `count + 1` bars,
drops that row with `drop_forming_rate_bar()`, and validates each series is
non-empty. Use `collect_latest_closed_rates_by_granularity()` when callers
prefer keys such as `("EURUSD", "M1")` instead of integer timeframes.
```python
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
closed_m1 = rates["EURUSD", "M1"]
```
### Resolving credentials and `${ENV_VAR}` placeholders
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
secrets out of plan/config files. A missing environment variable raises
`ValueError`.
```python
import os
from mt5cli import AccountSpec, resolve_account_specs
os.environ["MT5_LOGIN"] = "12345"
os.environ["MT5_PASSWORD"] = "secret"
accounts = [
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
]
resolved = resolve_account_specs(accounts, server="Broker-Demo")
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
```
### Throttled incremental history updates
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
between successful runs (using a monotonic clock), so an application loop can
call it every iteration without over-fetching.
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db",
datasets={Dataset.rates},
timeframes=["M1"],
interval_seconds=60, # <= 0 updates on every call
)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
while True:
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
# ... do other work; break when shutting down ...
finally:
client.shutdown()
```
By default `Mt5TradingError`, `Mt5RuntimeError`, and `sqlite3.Error` propagate so
the caller controls logging; pass `suppress_errors=True` to swallow them and
return `False` without advancing the throttle.
+20 -9
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -34,6 +35,7 @@ from mt5cli import (
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
@@ -49,7 +51,8 @@ rates = copy_rates_range(
export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
@@ -59,6 +62,9 @@ margins = minimum_margins("EURUSD")
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
@@ -74,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start
```bash
@@ -104,6 +112,7 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range |
### Ticks
@@ -130,14 +139,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
### Trading
| Command | Description |
| ---------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| Command | Description |
| ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`.
+70
View File
@@ -2,11 +2,34 @@
from importlib.metadata import version
from .history import (
RateTarget,
build_rate_targets,
build_rate_view_name,
drop_forming_rate_bar,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
)
from .sdk import (
AccountSpec,
Mt5CliClient,
ThrottledHistoryUpdater,
account_info,
build_config,
collect_history,
collect_latest_closed_rates_by_granularity,
collect_latest_closed_rates_for_accounts,
collect_latest_rates,
collect_latest_rates_for_accounts,
collect_latest_rates_for_accounts_with_retries,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -15,11 +38,19 @@ from .sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_session,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
resolve_account_spec,
resolve_account_specs,
substitute_env_placeholders,
symbol_info,
symbol_info_tick,
symbols,
@@ -31,39 +62,78 @@ from .sdk import (
version as mt5_version,
)
from .utils import (
TICK_FLAG_MAP,
TIMEFRAME_MAP,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_tick_flags,
parse_timeframe,
)
__version__ = version(__package__) if __package__ else None
__all__ = [
"TICK_FLAG_MAP",
"TIMEFRAME_MAP",
"AccountSpec",
"Dataset",
"IfExists",
"Mt5CliClient",
"RateTarget",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"build_rate_targets",
"build_rate_view_name",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
"copy_ticks_from",
"copy_ticks_range",
"detect_format",
"drop_forming_rate_bar",
"export_dataframe",
"export_dataframe_to_sqlite",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version",
"orders",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"positions",
"recent_history_deals",
"recent_ticks",
"resolve_account_spec",
"resolve_account_specs",
"resolve_history_datasets",
"resolve_history_tick_flags",
"resolve_history_timeframes",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
+56
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
)
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command()
def rates_range(
ctx: typer.Context,
@@ -475,6 +500,37 @@ def history_deals(
)
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
+475 -18
View File
@@ -4,9 +4,10 @@ from __future__ import annotations
import logging
import sqlite3
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import TYPE_CHECKING, Literal
from typing import TYPE_CHECKING, Literal, cast
import pandas as pd
@@ -20,7 +21,7 @@ from .utils import (
)
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from collections.abc import Callable, Mapping, Sequence
from pdmt5 import Mt5DataClient
@@ -105,6 +106,23 @@ def resolve_granularity_name(timeframe: int) -> str:
return str(timeframe)
def drop_forming_rate_bar(df_rate: pd.DataFrame) -> pd.DataFrame:
"""Return closed bars from chronologically ordered MT5 rate data.
MetaTrader 5 ``copy_rates_from_pos(start_pos=0)`` includes the still-forming
current bar as the last row. Slice it off so downstream logic only sees
completed bars. Empty frames and single-row frames return empty results.
Args:
df_rate: Rate data ordered oldest-to-newest with the forming bar last.
Returns:
A new DataFrame with all rows except the last. Index and columns are
preserved. The input frame is not modified.
"""
return df_rate.iloc[:-1].copy()
def build_rate_view_name(
*,
symbol: str,
@@ -126,15 +144,26 @@ def build_rate_view_name(
SqliteConnOrPath = sqlite3.Connection | Path | str
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
value = identifier.strip()
if not value:
msg = f"SQLite {kind} name must not be empty."
raise ValueError(msg)
return value
def _open_history_connection(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
) -> tuple[sqlite3.Connection | None, bool]:
"""Open a read-only SQLite connection when given a path.
Returns:
A connection and whether the caller should close it. When the path does
not exist, returns ``(None, False)`` without creating a database file.
A connection and whether the caller should close it. When ``conn_or_path``
is None or the path does not exist, returns ``(None, False)`` without
creating a database file.
"""
if conn_or_path is None:
return None, False
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
@@ -144,6 +173,133 @@ def _open_history_connection(
return conn, True
def _open_existing_sqlite_database(
conn_or_path: SqliteConnOrPath,
) -> tuple[sqlite3.Connection, bool]:
"""Open a read-only SQLite database or reuse an existing connection.
Returns:
Tuple of connection and whether the caller should close it.
Raises:
ValueError: If the database path does not exist or is not a file.
"""
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
if not path.is_file():
msg = f"SQLite database path is not a file: {path}"
raise ValueError(msg)
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _validate_rate_load_request(table: str, count: int | None) -> str:
table_name = _require_non_empty_identifier(table, "table or view")
if count is not None and count <= 0:
msg = "count must be positive when provided."
raise ValueError(msg)
return table_name
def _ensure_rate_columns(columns: set[str], table: str) -> None:
if not columns:
msg = f"SQLite table or view not found: {table}"
raise ValueError(msg)
if "time" not in columns:
msg = f"SQLite table or view {table!r} must include a time column."
raise ValueError(msg)
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
msg = (
f"SQLite table or view {table!r} must include close, "
"or both ask and bid columns."
)
raise ValueError(msg)
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
parsed = frame["time"].map(parse_sqlite_timestamp)
if parsed.isna().any():
msg = f"SQLite table or view {table!r} contains unparsable time values."
raise ValueError(msg)
result = frame.drop(columns=["time"])
result.index = pd.DatetimeIndex(parsed, name="time")
return result.sort_index(kind="stable")
def load_rate_data_from_connection(
connection: sqlite3.Connection,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite table or view.
Args:
connection: Open SQLite connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
Raises:
ValueError: If inputs, schema, timestamps are invalid, or the table
or view contains no rows.
"""
table_name = _validate_rate_load_request(table, count)
columns = get_table_columns(connection, table_name)
_ensure_rate_columns(columns, table_name)
quoted_table = quote_sqlite_identifier(table_name)
if count is None:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
connection,
),
)
else:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
connection,
params=(count,),
),
)
if frame.empty:
msg = f"SQLite table or view {table_name!r} contains no rows."
raise ValueError(msg)
return _parse_rate_time_index(frame, table_name)
def load_rate_data(
conn_or_path: SqliteConnOrPath,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite database path or connection.
Args:
conn_or_path: SQLite database path or open connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
"""
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
return load_rate_data_from_connection(conn, table, count=count)
finally:
if should_close:
conn.close()
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
"""Return distinct timeframe counts per symbol from the normalized rates table."""
columns = get_table_columns(conn, Dataset.rates.table_name)
@@ -241,7 +397,7 @@ def _resolve_rate_view_name_from_context(
def resolve_rate_view_name(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
symbol: str,
granularity: str,
*,
@@ -250,7 +406,9 @@ def resolve_rate_view_name(
"""Resolve the mt5cli-managed rate compatibility view name.
Args:
conn_or_path: SQLite database path or open connection.
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, the deterministic
default view name is returned without creating a database file.
symbol: Symbol stored in the normalized ``rates`` table.
granularity: Timeframe name (for example ``M1``) or integer string.
require_existing: When True, require the database and a managed view to exist.
@@ -294,7 +452,7 @@ def resolve_rate_view_name(
def resolve_rate_view_names(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
symbols: Sequence[str],
granularities: Sequence[str],
*,
@@ -303,7 +461,9 @@ def resolve_rate_view_names(
"""Resolve rate compatibility view names for symbol and granularity pairs.
Args:
conn_or_path: SQLite database path or open connection.
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, deterministic
default view names are returned without creating a database file.
symbols: Symbols stored in the normalized ``rates`` table.
granularities: Timeframe names (for example ``M1``) or integer strings.
require_existing: When True, require the database and managed views to exist.
@@ -347,9 +507,275 @@ def resolve_rate_view_names(
conn.close()
@dataclass(frozen=True)
class RateTarget:
"""A single rate series identified by symbol and timeframe.
Attributes:
symbol: MT5 symbol name, or None when the rate series is addressed only
by an explicit table (for example a custom SQLite view).
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
"""
symbol: str | None
timeframe: int | str
def __post_init__(self) -> None:
"""Normalize accepted timeframe aliases to the stored integer value."""
if not isinstance(self.timeframe, int):
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
@property
def timeframe_int(self) -> int:
"""Return the timeframe as its integer MT5 value."""
return cast("int", self.timeframe)
def build_rate_targets(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
allow_missing_symbol: bool = False,
) -> list[RateTarget]:
"""Build rate targets for every symbol and timeframe combination.
Args:
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each timeframe instead of raising.
Returns:
Targets in row-major order: every timeframe for the first symbol, then
every timeframe for the next symbol, and so on.
Raises:
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
``allow_missing_symbol`` is False.
"""
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if not symbols:
if not allow_missing_symbol:
msg = "At least one symbol is required."
raise ValueError(msg)
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
return [
RateTarget(symbol=symbol, timeframe=tf)
for symbol in symbols
for tf in timeframes
]
def resolve_rate_tables(
conn_or_path: SqliteConnOrPath | None,
targets: Sequence[RateTarget],
explicit_tables: Sequence[str] | None = None,
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve SQLite table or view names for rate targets.
Args:
conn_or_path: SQLite database path or open connection. May be None when
``explicit_tables`` is provided, or when ``require_existing`` is
False and deterministic default view names are sufficient.
targets: Rate targets to resolve.
explicit_tables: Optional explicit table or view names. When provided,
they are used as-is and must match the number of targets.
require_existing: When True, require the database and managed views to
exist for each symbol target. Ignored when ``explicit_tables`` is
provided.
Returns:
Table or view names aligned with ``targets``.
Raises:
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
match the target count, a target without a symbol is resolved
without an explicit table, or ``require_existing`` is True and the
database or a managed view is missing.
"""
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is not None:
tables = list(explicit_tables)
if len(tables) != len(target_list):
msg = (
f"Expected {len(target_list)} explicit table(s) "
f"to match the targets, got {len(tables)}."
)
raise ValueError(msg)
return tables
if any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
timeframe_counts = None
existing_views: set[str] = set()
else:
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for target in target_list:
symbol = cast("str", target.symbol)
timeframe = target.timeframe_int
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget],
count: int,
explicit_tables: Sequence[str] | None = None,
) -> dict[tuple[str | None, int], pd.DataFrame]:
"""Load multiple rate series from a SQLite database.
Args:
conn_or_path: SQLite database path or open connection.
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair
must be unique.
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching targets.
When omitted, managed ``rate_*`` compatibility views must already
exist in the database.
Returns:
Mapping keyed by ``(symbol, timeframe_int)`` to each rate DataFrame.
Raises:
ValueError: If ``count`` is not positive, targets are empty, duplicate
``(symbol, timeframe_int)`` pairs are present, or table resolution
fails.
"""
if count <= 0:
msg = "count must be positive."
raise ValueError(msg)
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is None and any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
seen_keys: set[tuple[str | None, int]] = set()
for target in target_list:
key = (target.symbol, target.timeframe_int)
if key in seen_keys:
symbol_repr = repr(target.symbol)
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
raise ValueError(msg)
seen_keys.add(key)
tables = (
resolve_rate_tables(None, target_list, explicit_tables)
if explicit_tables is not None
else None
)
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
resolved_tables = tables or resolve_rate_tables(
conn,
target_list,
require_existing=True,
)
return {
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
conn,
table,
count=count,
)
for target, table in zip(target_list, resolved_tables, strict=True)
}
finally:
if should_close:
conn.close()
def load_rate_series_by_granularity(
conn_or_path: SqliteConnOrPath,
symbols: Sequence[str],
granularities: Sequence[int | str],
count: int,
*,
explicit_tables: Sequence[str] | None = None,
allow_missing_symbol: bool = False,
) -> dict[tuple[str | None, str], pd.DataFrame]:
"""Load rate series keyed by symbol and string granularity name.
Builds targets with :func:`build_rate_targets` and loads them with
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
name (for example ``M1``) instead of the integer timeframe to reduce
downstream boilerplate.
Args:
conn_or_path: SQLite database path or open connection.
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching the
built targets in row-major order. Required when symbols are omitted.
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each granularity instead of raising.
Returns:
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
table resolution fails, or duplicate targets are present.
"""
targets = build_rate_targets(
symbols,
granularities,
allow_missing_symbol=allow_missing_symbol,
)
series = load_rate_series_from_sqlite(
conn_or_path,
targets,
count,
explicit_tables=explicit_tables,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in series.items()
}
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
"""Return existing SQLite columns for a table."""
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
quoted_table = quote_sqlite_identifier(table)
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
return {str(row[1]) for row in rows}
@@ -629,7 +1055,20 @@ def drop_duplicates_in_table(
)
DedupScope = tuple[str, tuple[object, ...]]
@dataclass(frozen=True)
class DedupScope:
"""Scoped deduplication predicate and the columns it references.
Attributes:
where: SQL predicate appended to the duplicate-removal query.
params: Parameters bound to the scope predicate.
required_columns: Columns that must be present in the written table for
the scope to run.
"""
where: str
params: tuple[object, ...]
required_columns: frozenset[str]
def _record_dedup_scope(
@@ -637,17 +1076,25 @@ def _record_dedup_scope(
dataset: Dataset,
scope_where: str,
scope_params: tuple[object, ...],
required_columns: frozenset[str],
) -> None:
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
dedup_scopes.setdefault(dataset, []).append(
DedupScope(scope_where, scope_params, required_columns),
)
def deduplicate_history_tables(
conn: sqlite3.Connection,
written_columns: dict[Dataset, set[str]],
written_tables: set[Dataset],
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
) -> None:
"""Deduplicate appended history tables by stable identifiers."""
"""Deduplicate appended history tables by stable identifiers.
Scopes whose required columns are not present in the written table are
skipped. If all scopes for a dataset are skipped, the table receives one
unscoped deduplication pass instead.
"""
cursor = conn.cursor()
for dataset in written_tables:
columns = written_columns.get(dataset, set())
@@ -666,16 +1113,19 @@ def deduplicate_history_tables(
table,
)
continue
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
raw_scopes: Sequence[DedupScope] = (
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
)
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
if scopes:
for scope_where, scope_params in scopes:
for scope in scopes:
drop_duplicates_in_table(
cursor,
table,
list(keys),
keep="last",
scope_where=scope_where,
scope_params=scope_params,
scope_where=scope.where,
scope_params=scope.params,
)
continue
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
@@ -1042,6 +1492,7 @@ def _write_incremental_rates(
Dataset.rates,
"symbol = ? AND timeframe = ? AND time >= ?",
(symbol, timeframe, start_date),
frozenset({"symbol", "timeframe", "time"}),
)
@@ -1080,6 +1531,7 @@ def _write_incremental_ticks(
Dataset.ticks,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -1118,6 +1570,7 @@ def _write_incremental_history_orders(
Dataset.history_orders,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -1171,6 +1624,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_by_symbol[symbol, None]),
frozenset({"symbol", "time"}),
)
if "type" in columns:
_record_dedup_scope(
@@ -1178,6 +1632,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
(account_event_start,),
frozenset({"type", "time"}),
)
if "type" not in columns and "symbol" in columns:
_record_dedup_scope(
@@ -1185,6 +1640,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"(symbol IS NULL OR symbol = '') AND time >= ?",
(account_event_start,),
frozenset({"symbol", "time"}),
)
return
start_by_symbol = load_incremental_start_datetimes(
@@ -1212,6 +1668,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
+812 -6
View File
@@ -2,21 +2,27 @@
from __future__ import annotations
import json
import logging
import os
import re
import sqlite3
import time
from contextlib import contextmanager
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import TYPE_CHECKING, Self, TypeVar
from typing import TYPE_CHECKING, Self, TypeVar, cast
import pandas as pd
from pdmt5 import Mt5Config, Mt5DataClient
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError, Mt5TradingError
from .history import (
create_cash_events_view,
create_history_indexes,
create_positions_reconstructed_view,
drop_forming_rate_bar,
resolve_granularity_name,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
@@ -39,10 +45,17 @@ T = TypeVar("T")
logger = logging.getLogger(__name__)
__all__ = [
"AccountSpec",
"Mt5CliClient",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
@@ -51,11 +64,19 @@ __all__ = [
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"orders",
"positions",
"recent_history_deals",
"recent_ticks",
"resolve_account_spec",
"resolve_account_specs",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
@@ -78,6 +99,22 @@ def _coerce_tick_flags(flags: int | str) -> int:
return parse_tick_flags(flags)
def _plain_mt5_value(value: object) -> object:
asdict = getattr(value, "_asdict", None)
if callable(asdict):
return _plain_mt5_value(asdict())
if isinstance(value, dict):
typed_value = cast("dict[object, object]", value)
return {key: _plain_mt5_value(item) for key, item in typed_value.items()}
if isinstance(value, tuple):
typed_value = cast("tuple[object, ...]", value)
return [_plain_mt5_value(item) for item in typed_value]
if isinstance(value, list):
typed_value = cast("list[object]", value)
return [_plain_mt5_value(item) for item in typed_value]
return value
def _require_datetime(value: datetime | str) -> datetime:
if isinstance(value, datetime):
return value
@@ -90,6 +127,37 @@ def _coerce_datetime(value: datetime | str | None) -> datetime | None:
return parse_datetime(value)
def _require_positive(value: float, name: str) -> None:
if value <= 0:
msg = f"{name} must be positive."
raise ValueError(msg)
def _require_non_negative(value: int, name: str) -> None:
if value < 0:
msg = f"{name} must be non-negative."
raise ValueError(msg)
def _call_required_client_method(client: Mt5DataClient, name: str) -> object:
try:
method = getattr(client, name)
except AttributeError as exc:
msg = f"MT5 client is missing required method: {name}"
raise AttributeError(msg) from exc
if not callable(method):
msg = f"MT5 client attribute is not callable: {name}"
raise TypeError(msg)
return method()
def _mt5_summary_export_value(value: object) -> object:
plain_value = _plain_mt5_value(value)
if isinstance(plain_value, dict | list):
return json.dumps(plain_value, sort_keys=True, separators=(",", ":"))
return plain_value
def _coerce_tick_time(value: object) -> datetime:
if isinstance(value, datetime):
return value
@@ -230,6 +298,26 @@ def _run_with_client(
return fetch_fn(client)
@contextmanager
def mt5_session(config: Mt5Config | None = None) -> Iterator[Mt5CliClient]:
"""Open an MT5 terminal session and yield a connected client.
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` (when set),
logs in, yields a connected :class:`Mt5CliClient`, and always shuts the
terminal down on exit.
Args:
config: MT5 connection configuration. Defaults to an empty config that
attaches to a running terminal.
Yields:
Connected ``Mt5CliClient`` bound to the session.
"""
mt5_config = config or build_config()
with _connected_client(mt5_config) as client:
yield Mt5CliClient.from_connected_client(client)
class Mt5CliClient:
"""Programmatic client for read-only MetaTrader 5 data access."""
@@ -242,6 +330,7 @@ class Mt5CliClient:
server: str | None = None,
timeout: int | None = None,
config: Mt5Config | None = None,
client: Mt5DataClient | None = None,
) -> None:
"""Initialize the SDK client.
@@ -252,6 +341,8 @@ class Mt5CliClient:
server: Trading server name.
timeout: Connection timeout in milliseconds.
config: Optional pre-built ``Mt5Config`` (overrides other args).
client: Optional already-connected ``Mt5DataClient``. Injected
clients are reused as-is and are not initialized or shut down.
"""
self._config = config or build_config(
path=path,
@@ -260,7 +351,20 @@ class Mt5CliClient:
server=server,
timeout=timeout,
)
self._client: Mt5DataClient | None = None
self._client = client
self._owns_client = client is None
@classmethod
def from_connected_client(cls, client: Mt5DataClient) -> Self:
"""Bind to an already-connected ``Mt5DataClient`` without owning it.
The returned ``Mt5CliClient`` never initializes or shuts down the
injected client, including when used as a context manager.
Returns:
Client wrapper bound to the injected connection.
"""
return cls(client=client)
@property
def config(self) -> Mt5Config:
@@ -273,6 +377,8 @@ class Mt5CliClient:
Returns:
This client instance.
"""
if self._client is not None:
return self
client = Mt5DataClient(config=self._config)
try:
client.initialize_and_login_mt5()
@@ -280,6 +386,7 @@ class Mt5CliClient:
client.shutdown()
raise
self._client = client
self._owns_client = True # only set when this method created the client
return self
def __exit__(
@@ -289,15 +396,18 @@ class Mt5CliClient:
tb: object,
) -> None:
"""Shut down the persistent MT5 connection."""
if self._client is not None:
if self._client is not None and self._owns_client:
self._client.shutdown()
self._client = None
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
def _fetch_value(self, fetch_fn: Callable[[Mt5DataClient], T]) -> T:
if self._client is not None:
return fetch_fn(self._client)
return _run_with_client(self._config, fetch_fn)
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
return self._fetch_value(fetch_fn)
def copy_rates_from(
self,
symbol: str,
@@ -335,6 +445,54 @@ class Mt5CliClient:
),
)
def latest_rates(
self,
symbol: str,
timeframe: int | str,
count: int,
start_pos: int = 0,
) -> pd.DataFrame:
"""Return the latest rates from a bar position."""
_require_positive(count, "count")
return self.copy_rates_from_pos(symbol, timeframe, start_pos, count)
def collect_latest_rates(
self,
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
count: int,
start_pos: int = 0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Return latest rates for each symbol/timeframe pair.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``.
Raises:
ValueError: If ``count`` is not positive or inputs are empty.
"""
_require_positive(count, "count")
if not symbols:
msg = "At least one symbol is required."
raise ValueError(msg)
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
resolved_timeframes = [_coerce_timeframe(timeframe) for timeframe in timeframes]
return self._fetch_value(
lambda c: {
(symbol, timeframe): c.copy_rates_from_pos_as_df(
symbol=symbol,
timeframe=timeframe,
start_pos=start_pos,
count=count,
)
for symbol in symbols
for timeframe in resolved_timeframes
},
)
def copy_rates_range(
self,
symbol: str,
@@ -486,6 +644,24 @@ class Mt5CliClient:
),
)
def recent_history_deals(
self,
hours: float,
date_to: datetime | str | None = None,
group: str | None = None,
symbol: str | None = None,
) -> pd.DataFrame:
"""Return historical deals from a recent trailing window."""
_require_positive(hours, "hours")
end = _require_datetime(date_to) if date_to is not None else datetime.now(UTC)
start = end - timedelta(hours=hours)
return self.history_deals(
date_from=start,
date_to=end,
group=group,
symbol=symbol,
)
def version(self) -> pd.DataFrame:
"""Return MetaTrader5 version information."""
return self._fetch(lambda c: c.version_as_df())
@@ -553,6 +729,39 @@ class Mt5CliClient:
"""
return self._fetch(lambda c: _fetch_minimum_margins(c, symbol))
def mt5_summary(self) -> dict[str, object]:
"""Return a compact terminal/account status summary."""
def _summary(client: Mt5DataClient) -> dict[str, object]:
return {
"version": _plain_mt5_value(
_call_required_client_method(client, "version"),
),
"terminal_info": _plain_mt5_value(
_call_required_client_method(client, "terminal_info"),
),
"account_info": _plain_mt5_value(
_call_required_client_method(client, "account_info"),
),
"symbols_total": _plain_mt5_value(
_call_required_client_method(client, "symbols_total"),
),
}
return self._fetch_value(_summary)
def mt5_summary_as_df(self) -> pd.DataFrame:
"""Return an export-safe one-row terminal/account summary DataFrame."""
summary = self.mt5_summary()
return pd.DataFrame(
[
{
key: _mt5_summary_export_value(value)
for key, value in summary.items()
},
],
)
def _resolve_incremental_settings(
selected_datasets: set[Dataset],
@@ -769,6 +978,115 @@ def update_history_with_config( # noqa: PLR0913
)
class ThrottledHistoryUpdater:
"""Throttled incremental SQLite history updater for long-running apps.
Wraps :func:`update_history` with a minimum interval between successful
updates, so a tight application loop can call :meth:`update` every
iteration without re-fetching MT5 history more often than desired. Timing
uses a monotonic clock, so it is unaffected by wall-clock changes.
"""
def __init__(
self,
*,
output: Path | str,
datasets: set[Dataset] | None = None,
timeframes: Sequence[int | str] | None = None,
flags: int | str = "ALL",
lookback_hours: float = 24.0,
with_views: bool = False,
include_account_events: bool = True,
interval_seconds: float = 0.0,
suppress_errors: bool = False,
) -> None:
"""Initialize the throttled updater.
Args:
output: SQLite database path.
datasets: Datasets to include (defaults to all).
timeframes: Rate timeframes to update (defaults to all fixed MT5
timeframes).
flags: Tick copy flags as integer or name (e.g. ``ALL``).
lookback_hours: First-run lookback when a table has no prior rows.
with_views: Create ``cash_events`` and ``positions_reconstructed``
views.
include_account_events: Include account-level cash events.
interval_seconds: Minimum seconds between successful updates. Values
``<= 0`` update on every call.
suppress_errors: When True, ``Mt5TradingError``, ``Mt5RuntimeError``,
and ``sqlite3.Error`` raised during an update are swallowed and
:meth:`update` returns False without advancing the throttle. When
False (default), such errors propagate so callers control logging.
"""
self.output = output
self.datasets = datasets
self.timeframes = timeframes
self.flags = flags
self.lookback_hours = lookback_hours
self.with_views = with_views
self.include_account_events = include_account_events
self.interval_seconds = interval_seconds
self.suppress_errors = suppress_errors
self._last_update_monotonic: float | None = None
@property
def last_update_monotonic(self) -> float | None:
"""Return the monotonic timestamp of the last successful update."""
return self._last_update_monotonic
def should_update(self) -> bool:
"""Return whether enough time has elapsed to run another update.
Returns:
True when ``interval_seconds <= 0``, when no update has succeeded
yet, or when at least ``interval_seconds`` have elapsed since the
last successful update.
"""
if self.interval_seconds <= 0 or self._last_update_monotonic is None:
return True
return (time.monotonic() - self._last_update_monotonic) >= self.interval_seconds
def update(self, client: Mt5DataClient, symbols: Sequence[str]) -> bool:
"""Run a throttled incremental history update.
Args:
client: Connected MT5 data client.
symbols: Symbols to update.
Returns:
True if an update ran successfully, False if it was throttled or
(when ``suppress_errors`` is True) failed with a recoverable error.
Raises:
Mt5TradingError: If the update fails and ``suppress_errors`` is False.
Mt5RuntimeError: If the update fails and ``suppress_errors`` is False.
sqlite3.Error: If the SQLite write fails and ``suppress_errors`` is
False.
"""
if not self.should_update():
return False
try:
update_history(
client=client,
output=self.output,
symbols=symbols,
datasets=self.datasets,
timeframes=self.timeframes,
flags=self.flags,
lookback_hours=self.lookback_hours,
with_views=self.with_views,
include_account_events=self.include_account_events,
)
except (Mt5TradingError, Mt5RuntimeError, sqlite3.Error):
if self.suppress_errors:
logger.warning("Suppressed history update error", exc_info=True)
return False
raise
self._last_update_monotonic = time.monotonic()
return True
def collect_history(
output: Path,
symbols: list[str],
@@ -873,6 +1191,467 @@ def copy_rates_from_pos(
)
def latest_rates(
symbol: str,
timeframe: int | str,
count: int,
start_pos: int = 0,
*,
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return the latest rates from a bar position."""
return _make_client(config=config).latest_rates(
symbol,
timeframe,
count,
start_pos=start_pos,
)
def collect_latest_rates(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
count: int,
start_pos: int = 0,
config: Mt5Config | None = None,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Return latest rates for each symbol/timeframe pair."""
return _make_client(config=config).collect_latest_rates(
symbols,
timeframes,
count=count,
start_pos=start_pos,
)
@dataclass(frozen=True)
class AccountSpec:
"""Connection parameters and symbols for one MT5 account group.
Attributes:
symbols: Symbols to load latest rates for under this account.
login: Trading account login. String values are coerced to int when
non-empty.
password: Trading account password.
server: Trading server name.
path: Path to the MetaTrader5 terminal EXE file.
timeout: Connection timeout in milliseconds.
"""
symbols: Sequence[str]
login: int | str | None = field(default=None, repr=False)
password: str | None = field(default=None, repr=False)
server: str | None = None
path: str | None = None
timeout: int | None = None
_ENV_PLACEHOLDER_PATTERN = re.compile(r"\$\{(?P<name>[A-Za-z_][A-Za-z0-9_]*)\}")
def substitute_env_placeholders(value: str) -> str:
"""Replace ``${ENV_VAR}`` placeholders in a string with environment values.
Args:
value: String that may contain one or more ``${ENV_VAR}`` placeholders.
Returns:
The string with every placeholder replaced by its environment value.
Raises:
ValueError: If a referenced environment variable is not set.
"""
parts: list[str] = []
last_end = 0
for match in _ENV_PLACEHOLDER_PATTERN.finditer(value):
parts.append(value[last_end : match.start()])
name = match.group("name")
if name not in os.environ:
msg = f"Environment variable {name!r} is not set."
raise ValueError(msg)
parts.append(os.environ[name])
last_end = match.end()
parts.append(value[last_end:])
return "".join(parts)
def _resolve_field(override: str | None, account_value: str | None) -> str | None:
"""Resolve a string field from an override or account value with env subst.
Returns:
The explicit override when provided, otherwise the account value, with
any ``${ENV_VAR}`` placeholders substituted.
"""
value = override if override is not None else account_value
if value is None:
return None
return substitute_env_placeholders(value)
def _resolve_login(
override: int | str | None,
account_login: int | str | None,
) -> int | str | None:
"""Resolve a login from an override or account value with env substitution.
Returns:
The explicit override when provided, otherwise the account login.
Integer values are preserved; string values have ``${ENV_VAR}``
placeholders substituted.
"""
if override is not None:
if isinstance(override, int):
return override
return substitute_env_placeholders(override)
if account_login is None or isinstance(account_login, int):
return account_login
return substitute_env_placeholders(account_login)
def resolve_account_spec(
account: AccountSpec,
*,
login: int | str | None = None,
password: str | None = None,
server: str | None = None,
path: str | None = None,
timeout: int | None = None,
) -> AccountSpec:
"""Resolve an account's credentials from overrides and ``${ENV_VAR}`` values.
Explicit override arguments take precedence over the corresponding
:class:`AccountSpec` fields. The resolved string fields (``login``,
``password``, ``server``, ``path``) have any ``${ENV_VAR}`` placeholders
substituted from the environment.
Args:
account: Source account specification.
login: Optional explicit login override.
password: Optional explicit password override.
server: Optional explicit server override.
path: Optional explicit terminal path override.
timeout: Optional explicit connection timeout override.
Returns:
A new :class:`AccountSpec` with resolved credentials and the original
symbols preserved. Raises ``ValueError`` (via
:func:`substitute_env_placeholders`) if a referenced environment
variable is not set.
"""
return AccountSpec(
symbols=account.symbols,
login=_resolve_login(login, account.login),
password=_resolve_field(password, account.password),
server=_resolve_field(server, account.server),
path=_resolve_field(path, account.path),
timeout=timeout if timeout is not None else account.timeout,
)
def resolve_account_specs(
accounts: Sequence[AccountSpec],
*,
login: int | str | None = None,
password: str | None = None,
server: str | None = None,
path: str | None = None,
timeout: int | None = None,
) -> list[AccountSpec]:
"""Resolve credentials for multiple accounts.
Applies the same overrides and ``${ENV_VAR}`` substitution as
:func:`resolve_account_spec` to every account.
Args:
accounts: Source account specifications.
login: Optional explicit login override applied to each account.
password: Optional explicit password override applied to each account.
server: Optional explicit server override applied to each account.
path: Optional explicit terminal path override applied to each account.
timeout: Optional explicit timeout override applied to each account.
Returns:
Resolved account specifications in the original order. Raises
``ValueError`` (via :func:`substitute_env_placeholders`) if a referenced
environment variable is not set.
"""
return [
resolve_account_spec(
account,
login=login,
password=password,
server=server,
path=path,
timeout=timeout,
)
for account in accounts
]
def _coerce_login(login: int | str | None) -> int | None:
"""Coerce a login value to int, treating empty strings as unset.
Returns:
Integer login, or None when unset or an empty string.
"""
if login is None or isinstance(login, int):
return login
text = login.strip()
if not text:
return None
return int(text)
def _build_account_config(
account: AccountSpec,
base_config: Mt5Config | None,
) -> Mt5Config:
"""Build an ``Mt5Config`` for an account, falling back to ``base_config``.
Returns:
Merged MT5 configuration for the account.
"""
login = _coerce_login(account.login)
if login is None and base_config is not None:
login = base_config.login
return build_config(
path=account.path or (base_config.path if base_config else None),
login=login,
password=account.password or (base_config.password if base_config else None),
server=account.server or (base_config.server if base_config else None),
timeout=account.timeout
if account.timeout is not None
else (base_config.timeout if base_config else None),
)
def collect_latest_rates_for_accounts(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest rates across multiple MT5 account groups.
Each account is connected in turn, its symbols are read for every
timeframe, and the resulting frames are merged into a single mapping.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent bars to read per symbol/timeframe.
start_pos: Initial bar position offset.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``. When accounts share a
symbol/timeframe pair, the last account processed wins.
Raises:
ValueError: If ``accounts``, ``timeframes``, or any account's symbols are
empty, or ``count`` is not positive.
"""
account_list = list(accounts)
if not account_list:
msg = "At least one account is required."
raise ValueError(msg)
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if any(not account.symbols for account in account_list):
msg = "Each account requires at least one symbol."
raise ValueError(msg)
_require_positive(count, "count")
result: dict[tuple[str, int], pd.DataFrame] = {}
for account in account_list:
config = _build_account_config(account, base_config)
with Mt5CliClient(config=config) as client:
result.update(
client.collect_latest_rates(
account.symbols,
timeframes,
count=count,
start_pos=start_pos,
),
)
return result
def collect_latest_rates_for_accounts_with_retries(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest rates across accounts, retrying transient MT5 failures.
Wraps :func:`collect_latest_rates_for_accounts` with bounded exponential
backoff. Only ``pdmt5.Mt5TradingError`` and ``pdmt5.Mt5RuntimeError`` are
retried; other exceptions propagate immediately. The final failure is
re-raised once retries are exhausted.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent bars to read per symbol/timeframe.
start_pos: Initial bar position offset.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff. The delay before retry
attempt ``n`` (1-indexed) is ``backoff_base ** n`` seconds.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``. Propagates ``ValueError``
for invalid inputs (see :func:`collect_latest_rates_for_accounts`) and
re-raises the last ``pdmt5.Mt5TradingError`` or ``pdmt5.Mt5RuntimeError``
once retries are exhausted.
"""
attempts = max(retry_count, 0) + 1
def _collect() -> dict[tuple[str, int], pd.DataFrame]:
return collect_latest_rates_for_accounts(
accounts,
timeframes,
count,
start_pos=start_pos,
base_config=base_config,
)
for attempt in range(attempts - 1):
try:
return _collect()
except (Mt5TradingError, Mt5RuntimeError) as exc:
delay = backoff_base ** (attempt + 1)
logger.warning(
"Rate collection failed (attempt %d/%d): %s; retrying in %.1fs",
attempt + 1,
attempts,
exc,
delay,
)
time.sleep(delay)
return _collect()
def collect_latest_closed_rates_for_accounts(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest closed rate bars across multiple MT5 account groups.
When ``start_pos`` is ``0`` (the default), MetaTrader 5 includes the
still-forming current bar as the last row. This helper fetches
``count + 1`` bars, drops that bar with :func:`drop_forming_rate_bar`, and
validates that each resulting frame is non-empty. When ``start_pos`` is
greater than zero the forming bar is not in range, so only ``count`` bars
are fetched and no row is dropped.
Wraps :func:`collect_latest_rates_for_accounts_with_retries` for transient
MT5 error handling.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of closed bars to return per symbol/timeframe.
start_pos: Initial bar position offset passed to the underlying collector.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff between retry attempts.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``.
Raises:
ValueError: If inputs are invalid, or any series is empty (after
dropping the still-forming bar when ``start_pos`` is ``0``).
"""
_require_positive(count, "count")
_require_non_negative(start_pos, "start_pos")
fetch_count = count + 1 if start_pos == 0 else count
loaded = collect_latest_rates_for_accounts_with_retries(
accounts,
timeframes,
fetch_count,
start_pos=start_pos,
base_config=base_config,
retry_count=retry_count,
backoff_base=backoff_base,
)
result: dict[tuple[str, int], pd.DataFrame] = {}
for key, df_rate in loaded.items():
closed = drop_forming_rate_bar(df_rate) if start_pos == 0 else df_rate
if closed.empty:
symbol, timeframe = key
msg = f"Rate data is empty for {symbol!r} at timeframe {timeframe}."
raise ValueError(msg)
result[key] = closed
return result
def collect_latest_closed_rates_by_granularity(
accounts: Sequence[AccountSpec],
granularities: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, str], pd.DataFrame]:
"""Collect latest closed rate bars keyed by symbol and granularity name.
Thin wrapper around :func:`collect_latest_closed_rates_for_accounts` that
rekeys the result by granularity name (for example ``M1``) instead of the
integer timeframe.
Args:
accounts: Account groups to read. Each must define at least one symbol.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of closed bars to return per symbol/timeframe.
start_pos: Initial bar position offset passed to the underlying collector.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff between retry attempts.
Returns:
Mapping keyed by ``(symbol, granularity_name)``. Propagates
``ValueError`` from :func:`collect_latest_closed_rates_for_accounts`.
"""
loaded = collect_latest_closed_rates_for_accounts(
accounts,
granularities,
count,
start_pos=start_pos,
base_config=base_config,
retry_count=retry_count,
backoff_base=backoff_base,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in loaded.items()
}
def copy_rates_range(
symbol: str,
timeframe: int | str,
@@ -1024,6 +1803,23 @@ def history_deals(
)
def recent_history_deals(
hours: float,
date_to: datetime | str | None = None,
group: str | None = None,
symbol: str | None = None,
*,
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return historical deals from a recent trailing window."""
return _make_client(config=config).recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
)
def version(*, config: Mt5Config | None = None) -> pd.DataFrame:
"""Return MetaTrader5 version information."""
return _make_client(config=config).version()
@@ -1084,3 +1880,13 @@ def minimum_margins(
See ``Mt5CliClient.minimum_margins`` for return details.
"""
return _make_client(config=config).minimum_margins(symbol)
def mt5_summary(*, config: Mt5Config | None = None) -> dict[str, object]:
"""Return a compact terminal/account status summary."""
return _make_client(config=config).mt5_summary()
def mt5_summary_as_df(*, config: Mt5Config | None = None) -> pd.DataFrame:
"""Return an export-safe terminal/account status summary DataFrame."""
return _make_client(config=config).mt5_summary_as_df()
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.3"
version = "0.6.0"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
+113
View File
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50,
)
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range(
self,
tmp_path: Path,
@@ -451,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version(
self,
tmp_path: Path,
+694 -3
View File
@@ -10,14 +10,19 @@ from unittest.mock import MagicMock
import pandas as pd
import pytest
from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from pathlib import Path
from mt5cli import history
from mt5cli.history import (
DEFAULT_HISTORY_TIMEFRAMES,
DedupScope,
RateTarget,
append_dataframe,
augment_written_columns_from_sqlite,
build_rate_targets,
build_rate_view_name,
create_cash_events_view,
create_history_indexes,
@@ -25,12 +30,17 @@ from mt5cli.history import (
create_rate_compatibility_views,
deduplicate_history_tables,
drop_duplicates_in_table,
drop_forming_rate_bar,
filter_incremental_history_deals_frame,
filter_trade_history_frame,
get_history_deals_account_event_start_datetime,
get_incremental_start_datetime,
get_table_columns,
load_incremental_start_datetimes,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
parse_sqlite_timestamp,
quote_sqlite_identifier,
record_written_columns,
@@ -38,6 +48,7 @@ from mt5cli.history import (
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
write_collected_datasets,
@@ -58,6 +69,21 @@ class TestResolveRateViewName:
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
assert not db_path.exists()
def test_none_path_returns_default_name(self) -> None:
"""Test a None connection or path returns the deterministic default."""
assert resolve_rate_view_name(None, "EURUSD", "M1") == "rate_EURUSD__1"
assert resolve_rate_view_names(None, ["EURUSD"], ["M1", "H1"]) == [
"rate_EURUSD__1",
"rate_EURUSD__16385",
]
def test_none_path_with_require_existing_raises(self) -> None:
"""Test a None path under strict mode raises a clear error."""
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_name(None, "EURUSD", "M1", require_existing=True)
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_names(None, ["EURUSD"], ["M1"], require_existing=True)
def test_no_rates_table_falls_back_to_single_timeframe_name(
self,
tmp_path: Path,
@@ -338,6 +364,147 @@ class TestQuoteSqliteIdentifier:
assert quoted.endswith('"')
class TestLoadRateData:
"""Tests for SQLite rate-like table and view loading."""
def test_loads_close_rates_from_path_with_count(self, tmp_path: Path) -> None:
"""Test loading the latest close-based rates in ascending time order."""
db_path = tmp_path / "rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.executemany(
"INSERT INTO rates(time, close) VALUES (?, ?)",
[
("2024-01-01T00:00:00+00:00", 1.0),
("2024-01-01T00:02:00+00:00", 1.2),
("2024-01-01T00:01:00+00:00", 1.1),
],
)
frame = load_rate_data(db_path, "rates", count=2)
assert list(frame["close"]) == [1.1, 1.2]
assert isinstance(frame.index, pd.DatetimeIndex)
assert frame.index.name == "time"
assert frame.index.is_monotonic_increasing
def test_loads_ask_bid_tick_like_rates_from_connection(
self,
tmp_path: Path,
) -> None:
"""Test loading tick-like tables with bid and ask columns."""
db_path = tmp_path / "ticks.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(time TEXT, bid REAL, ask REAL)")
conn.execute(
"INSERT INTO ticks(time, bid, ask) VALUES (?, ?, ?)",
("2024-01-01T00:00:00+00:00", 1.0, 1.1),
)
frame = load_rate_data_from_connection(conn, "ticks")
path_frame = load_rate_data(conn, "ticks")
assert frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
assert path_frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
def test_loads_from_view(self, tmp_path: Path) -> None:
"""Test loading from a SQLite view."""
db_path = tmp_path / "view.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.execute(
"INSERT INTO rates(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
conn.execute("CREATE VIEW rate_view AS SELECT time, close FROM rates")
frame = load_rate_data_from_connection(conn, "rate_view")
assert list(frame["close"]) == [1.0]
def test_loads_quoted_identifier(self, tmp_path: Path) -> None:
"""Test table names are quoted safely."""
db_path = tmp_path / "quoted.db"
table = 'rate "quoted"'
quoted = quote_sqlite_identifier(table)
with sqlite3.connect(db_path) as conn:
conn.execute(f"CREATE TABLE {quoted}(time TEXT, close REAL)")
conn.execute(
f"INSERT INTO {quoted}(time, close) VALUES (?, ?)", # noqa: S608
("2024-01-01T00:00:00+00:00", 1.0),
)
frame = load_rate_data_from_connection(conn, table)
assert list(frame["close"]) == [1.0]
def test_rejects_missing_database_and_non_file(self, tmp_path: Path) -> None:
"""Test path validation for SQLite database inputs."""
with pytest.raises(ValueError, match="SQLite database not found"):
load_rate_data(tmp_path / "missing.db", "rates")
with pytest.raises(ValueError, match="not a file"):
load_rate_data(tmp_path, "rates")
@pytest.mark.parametrize(
("table", "count", "match"),
[
("", None, "must not be empty"),
("rates", 0, "count must be positive"),
("rates", -1, "count must be positive"),
],
)
def test_rejects_invalid_inputs(
self,
tmp_path: Path,
table: str,
count: int | None,
match: str,
) -> None:
"""Test request validation."""
db_path = tmp_path / "invalid-inputs.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
with pytest.raises(ValueError, match=match):
load_rate_data_from_connection(conn, table, count=count)
@pytest.mark.parametrize(
("ddl", "match"),
[
("CREATE TABLE rates(time TEXT, close REAL)", "contains no rows"),
("CREATE TABLE rates(close REAL)", "time column"),
("CREATE TABLE rates(time TEXT, open REAL)", "close, or both ask and bid"),
],
)
def test_rejects_invalid_tables(
self,
tmp_path: Path,
ddl: str,
match: str,
) -> None:
"""Test missing table, empty table, and invalid schemas."""
db_path = tmp_path / "invalid-tables.db"
with sqlite3.connect(db_path) as conn:
conn.execute(ddl)
with pytest.raises(ValueError, match=match):
load_rate_data_from_connection(conn, "rates")
with pytest.raises(ValueError, match="not found"):
load_rate_data_from_connection(conn, "missing")
def test_rejects_invalid_timestamp(self, tmp_path: Path) -> None:
"""Test unparsable timestamps fail clearly."""
db_path = tmp_path / "invalid-time.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.execute("INSERT INTO rates(time, close) VALUES (?, ?)", ("bad", 1.0))
with pytest.raises(ValueError, match="unparsable time"):
load_rate_data_from_connection(conn, "rates")
def test_loads_numeric_mt5_epoch_seconds(self, tmp_path: Path) -> None:
"""Test MT5-native integer timestamps are parsed as epoch seconds."""
db_path = tmp_path / "epoch-rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time INTEGER, close REAL)")
conn.execute(
"INSERT INTO rates(time, close) VALUES (?, ?)",
(1_704_067_200, 1.0),
)
frame = load_rate_data_from_connection(conn, "rates")
assert frame.index[0] == pd.Timestamp("2024-01-01", tz="UTC")
assert list(frame["close"]) == [1.0]
class TestResolveHistorySettings:
"""Tests for history dataset and timeframe resolution."""
@@ -368,6 +535,46 @@ class TestResolveHistorySettings:
assert resolve_granularity_name(1) == "M1"
class TestDropFormingRateBar:
"""Tests for drop_forming_rate_bar."""
def test_drops_still_forming_last_bar(self) -> None:
"""Test the still-forming last bar is removed."""
df_rate = pd.DataFrame(
{"time": [1, 2, 3], "close": [1.1, 1.2, 1.3]},
index=pd.Index(["a", "b", "c"], name="idx"),
)
result = drop_forming_rate_bar(df_rate)
pd.testing.assert_frame_equal(
result,
pd.DataFrame(
{"time": [1, 2], "close": [1.1, 1.2]},
index=pd.Index(["a", "b"], name="idx"),
),
)
assert df_rate.shape == (3, 2)
def test_returns_empty_frame_when_input_empty(self) -> None:
"""Test empty frames stay empty."""
df_rate = pd.DataFrame(columns=["time", "close"])
result = drop_forming_rate_bar(df_rate)
assert result.empty
assert list(result.columns) == ["time", "close"]
def test_returns_empty_frame_when_only_forming_bar_present(self) -> None:
"""Test a single-bar frame becomes empty after dropping the forming bar."""
df_rate = pd.DataFrame({"time": [1], "close": [1.1]})
result = drop_forming_rate_bar(df_rate)
assert result.empty
assert list(result.columns) == ["time", "close"]
class TestParseSqliteTimestamp:
"""Tests for parse_sqlite_timestamp."""
@@ -454,7 +661,7 @@ class TestIncrementalStart:
) -> None:
"""Test rates tables without timeframe fail fast during incremental resume."""
fallback = datetime(2024, 1, 1, tzinfo=UTC)
with sqlite3.connect(tmp_path / "legacy-rates.db") as conn:
with sqlite3.connect(tmp_path / "rates-without-timeframe.db") as conn:
conn.execute("CREATE TABLE rates(symbol TEXT, time TEXT, open REAL)")
conn.execute(
"INSERT INTO rates(symbol, time, open) VALUES (?, ?, ?)",
@@ -723,9 +930,10 @@ class TestDeduplication:
{Dataset.rates},
{
Dataset.rates: [
(
DedupScope(
"symbol = ? AND timeframe = ? AND time >= ?",
("EURUSD", 1, boundary),
frozenset({"symbol", "timeframe", "time"}),
),
],
},
@@ -738,6 +946,89 @@ class TestDeduplication:
("2024-01-02T00:00:00+00:00", 9.9),
]
def test_unusable_scope_falls_back_to_table_dedup(self, tmp_path: Path) -> None:
"""Test scopes with missing columns do not break stable-key dedup."""
boundary = datetime(2024, 1, 1, tzinfo=UTC)
with sqlite3.connect(tmp_path / "orders-without-time.db") as conn:
conn.execute(
"CREATE TABLE history_orders("
" ticket INTEGER, symbol TEXT, time_setup TEXT, type INTEGER)",
)
conn.executemany(
"INSERT INTO history_orders(ticket, symbol, time_setup, type)"
" VALUES (?, ?, ?, ?)",
[
(1, "EURUSD", "2024-01-01T00:00:00+00:00", 0),
(1, "EURUSD", "2024-01-01T00:00:01+00:00", 1),
],
)
deduplicate_history_tables(
conn,
{Dataset.history_orders: {"ticket", "symbol", "time_setup", "type"}},
{Dataset.history_orders},
{
Dataset.history_orders: [
DedupScope(
"symbol = ? AND time >= ?",
("EURUSD", boundary),
frozenset({"symbol", "time"}),
),
],
},
)
rows = conn.execute(
"SELECT ticket, time_setup, type FROM history_orders",
).fetchall()
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
def test_partially_unusable_scopes_only_run_usable_scopes(
self,
tmp_path: Path,
) -> None:
"""Test mixed scope filtering skips only scopes with missing columns."""
boundary = datetime(2024, 1, 2, tzinfo=UTC)
with sqlite3.connect(tmp_path / "partial-scope-filter.db") as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, open REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, open) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 2.0),
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 9.9),
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 100.0),
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 101.0),
],
)
deduplicate_history_tables(
conn,
{Dataset.rates: {"symbol", "timeframe", "time", "open"}},
{Dataset.rates},
{
Dataset.rates: [
DedupScope(
"symbol = ? AND timeframe = ? AND time >= ?",
("EURUSD", 1, boundary),
frozenset({"symbol", "timeframe", "time"}),
),
DedupScope(
"symbol = ? AND timeframe = ? AND broker = ?",
("USDJPY", 1, "demo"),
frozenset({"symbol", "timeframe", "broker"}),
),
],
},
)
rows = conn.execute(
"SELECT symbol, open FROM rates ORDER BY symbol, open",
).fetchall()
assert rows == [
("EURUSD", 9.9),
("USDJPY", 100.0),
("USDJPY", 101.0),
]
class TestRateCompatibilityViews:
"""Tests for rate compatibility view creation."""
@@ -1198,6 +1489,54 @@ class TestIncrementalIntegration:
"rate_EURUSD_M1__1",
}
def test_incremental_orders_without_time_deduplicate_by_ticket(
self,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test incremental history_orders without time deduplicate safely."""
def history_orders_get_as_df(**kwargs: object) -> pd.DataFrame:
if kwargs["symbol"] == "GBPUSD":
return pd.DataFrame()
return pd.DataFrame({
"ticket": [1, 1],
"symbol": ["EURUSD", "EURUSD"],
"time_setup": [
"2024-01-01T00:00:00+00:00",
"2024-01-01T00:00:01+00:00",
],
"type": [0, 1],
})
client = MagicMock()
client.history_orders_get_as_df.side_effect = history_orders_get_as_df
start = datetime(2024, 1, 1, tzinfo=UTC)
end = datetime(2024, 1, 2, tzinfo=UTC)
with (
sqlite3.connect(tmp_path / "incremental-orders-without-time.db") as conn,
caplog.at_level(logging.WARNING, logger="mt5cli.history"),
):
write_incremental_datasets(
conn,
client,
["EURUSD", "GBPUSD"],
{Dataset.history_orders},
[],
0,
start,
end,
deduplicate=True,
create_rate_views=False,
with_views=False,
include_account_events=False,
)
rows = conn.execute(
"SELECT ticket, time_setup, type FROM history_orders",
).fetchall()
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
assert "Skipping history_orders: dataset returned no columns" in caplog.text
def test_write_collected_datasets_and_edge_branches(
self,
tmp_path: Path,
@@ -1573,7 +1912,7 @@ class TestIncrementalHistoryDeals:
})
start = datetime(2024, 1, 1, tzinfo=UTC)
end = datetime(2024, 1, 3, tzinfo=UTC)
with sqlite3.connect(tmp_path / "legacy-deals.db") as conn:
with sqlite3.connect(tmp_path / "deals-without-type.db") as conn:
conn.execute(
"CREATE TABLE history_deals( ticket INTEGER, symbol TEXT, time TEXT)",
)
@@ -1772,3 +2111,355 @@ class TestWriteHelpers:
)
assert get_table_columns(conn, "rates") == {"time", "open"}
create_history_indexes(conn, written_columns)
class TestRateSourceHelpers:
"""Tests for generic rate-source SDK helpers."""
def test_rate_target_timeframe_int(self) -> None:
"""Test RateTarget resolves named and integer timeframes."""
target = RateTarget(symbol="EURUSD", timeframe="M1")
assert target.timeframe == 1
assert target.timeframe_int == 1
assert RateTarget(symbol="EURUSD", timeframe=16385).timeframe_int == 16385
def test_build_rate_targets_row_major(self) -> None:
"""Test targets are built in row-major symbol/timeframe order."""
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
assert [(t.symbol, t.timeframe) for t in targets] == [
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
]
def test_build_rate_targets_allows_missing_symbol(self) -> None:
"""Test missing symbols produce None-symbol targets when allowed."""
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
assert [(t.symbol, t.timeframe) for t in targets] == [
(None, 1),
(None, 16385),
]
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
(["EURUSD"], [], "At least one timeframe"),
([], ["M1"], "At least one symbol"),
],
)
def test_build_rate_targets_rejects_empty(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test target building input validation."""
with pytest.raises(ValueError, match=match):
build_rate_targets(symbols, timeframes)
def test_resolve_rate_tables_uses_explicit_tables(self) -> None:
"""Test explicit tables bypass view resolution when counts match."""
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
assert resolve_rate_tables(None, targets, ["t1", "t2"]) == ["t1", "t2"]
def test_resolve_rate_tables_rejects_mismatched_explicit_count(self) -> None:
"""Test explicit table count must match the number of targets."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="Expected 1 explicit table"):
resolve_rate_tables(None, targets, ["t1", "t2"])
def test_resolve_rate_tables_rejects_empty_targets(self) -> None:
"""Test resolving requires at least one target."""
with pytest.raises(ValueError, match="At least one rate target"):
resolve_rate_tables(None, [])
def test_resolve_rate_tables_requires_symbol_without_explicit(self) -> None:
"""Test None-symbol targets require explicit tables."""
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
with pytest.raises(ValueError, match="without a symbol"):
resolve_rate_tables(None, targets)
def test_resolve_rate_tables_resolves_view_names(self) -> None:
"""Test symbol targets resolve to default view names without a database."""
targets = build_rate_targets(["EURUSD"], ["M1", "H1"])
assert resolve_rate_tables(None, targets) == [
"rate_EURUSD__1",
"rate_EURUSD__16385",
]
def test_resolve_rate_tables_none_path_with_require_existing_raises(self) -> None:
"""Test strict mode rejects a missing database path."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_tables(None, targets, require_existing=True)
def test_resolve_rate_tables_missing_db_with_require_existing_raises(
self,
tmp_path: Path,
) -> None:
"""Test strict mode rejects a non-existing database path."""
db_path = tmp_path / "missing.db"
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_tables(db_path, targets, require_existing=True)
def test_resolve_rate_tables_missing_view_with_require_existing_raises(
self,
tmp_path: Path,
) -> None:
"""Test strict mode rejects databases without managed rate views."""
db_path = tmp_path / "no-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="No rate compatibility view exists"):
resolve_rate_tables(db_path, targets, require_existing=True)
def test_resolve_rate_tables_with_require_existing_resolves_views(
self,
tmp_path: Path,
) -> None:
"""Test strict mode resolves existing managed rate views."""
db_path = tmp_path / "strict-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
targets = build_rate_targets(["EURUSD"], ["M1"])
assert resolve_rate_tables(db_path, targets, require_existing=True) == [
"rate_EURUSD__1",
]
def test_resolve_rate_tables_batches_sqlite_metadata(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Test resolving multiple targets loads SQLite metadata once."""
db_path = tmp_path / "batch-rate-tables.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 16385, "2024-01-01T01:00:00+00:00", 1.1),
("GBPUSD", 1, "2024-01-01T00:00:00+00:00", 1.2),
],
)
create_rate_compatibility_views(conn)
counts_spy = mocker.spy(history, "_load_rates_timeframe_counts")
views_spy = mocker.spy(history, "_load_existing_rate_views")
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
assert resolve_rate_tables(db_path, targets) == [
"rate_EURUSD__M1_1",
"rate_EURUSD__H1_16385",
"rate_GBPUSD__1",
"rate_GBPUSD__16385",
]
assert counts_spy.call_count == 1
assert views_spy.call_count == 1
def test_load_rate_series_from_sqlite(self, tmp_path: Path) -> None:
"""Test loading multiple rate series keyed by symbol and timeframe."""
db_path = tmp_path / "series.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 1, "2024-01-01T00:01:00+00:00", 1.1),
],
)
create_rate_compatibility_views(conn)
targets = build_rate_targets(["EURUSD"], ["M1"])
result = load_rate_series_from_sqlite(db_path, targets, count=2)
assert set(result) == {("EURUSD", 1)}
assert len(result["EURUSD", 1]) == 2
def test_load_rate_series_by_granularity(self, tmp_path: Path) -> None:
"""Test loading rate series keyed by symbol and granularity name."""
db_path = tmp_path / "granularity.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 16385, "2024-01-01T00:00:00+00:00", 1.1),
],
)
create_rate_compatibility_views(conn)
result = load_rate_series_by_granularity(
db_path,
["EURUSD"],
["M1", "H1"],
count=1,
)
assert set(result) == {("EURUSD", "M1"), ("EURUSD", "H1")}
def test_load_rate_series_by_granularity_explicit_tables(
self,
tmp_path: Path,
) -> None:
"""Test explicit tables with None-symbol targets key by granularity."""
db_path = tmp_path / "granularity-explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
result = load_rate_series_by_granularity(
db_path,
[],
["M1"],
count=1,
explicit_tables=["custom_view"],
allow_missing_symbol=True,
)
assert set(result) == {(None, "M1")}
def test_load_rate_series_reuses_path_connection(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Test loading from a path opens SQLite once for resolve and reads."""
db_path = tmp_path / "single-open-series.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
connect_spy = mocker.spy(history.sqlite3, "connect")
result = load_rate_series_from_sqlite(
db_path,
build_rate_targets(["EURUSD"], ["M1"]),
count=1,
)
assert set(result) == {("EURUSD", 1)}
assert connect_spy.call_count == 1
def test_load_rate_series_with_explicit_tables(self, tmp_path: Path) -> None:
"""Test explicit tables and None-symbol targets load series."""
db_path = tmp_path / "explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
result = load_rate_series_from_sqlite(
db_path,
targets,
count=1,
explicit_tables=["custom_view"],
)
assert set(result) == {(None, 1)}
def test_load_rate_series_rejects_non_positive_count(self) -> None:
"""Test loading requires a positive count."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="count must be positive"):
load_rate_series_from_sqlite("unused.db", targets, count=0)
def test_load_rate_series_rejects_empty_targets(self) -> None:
"""Test loading requires at least one target before opening SQLite."""
with pytest.raises(ValueError, match="At least one rate target"):
load_rate_series_from_sqlite("unused.db", [], count=1)
def test_load_rate_series_requires_symbol_without_explicit_tables(self) -> None:
"""Test None-symbol targets require explicit tables before opening SQLite."""
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
with pytest.raises(ValueError, match="without a symbol"):
load_rate_series_from_sqlite("unused.db", targets, count=1)
def test_load_rate_series_requires_existing_managed_views(
self,
tmp_path: Path,
) -> None:
"""Test loading without explicit tables requires managed rate views."""
db_path = tmp_path / "no-managed-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="No rate compatibility view exists"):
load_rate_series_from_sqlite(db_path, targets, count=1)
def test_load_rate_series_rejects_duplicate_targets(self) -> None:
"""Test duplicate (symbol, timeframe) targets are rejected."""
targets = [
RateTarget("EURUSD", 1),
RateTarget("EURUSD", "M1"),
]
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
load_rate_series_from_sqlite("unused.db", targets, count=1)
def test_load_rate_series_rejects_duplicate_targets_with_explicit_tables(
self,
tmp_path: Path,
) -> None:
"""Test duplicate targets are rejected even with explicit tables."""
db_path = tmp_path / "duplicate-explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
targets = [
RateTarget("EURUSD", 1),
RateTarget("EURUSD", 1),
]
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
load_rate_series_from_sqlite(
db_path,
targets,
count=1,
explicit_tables=["custom_view", "custom_view"],
)
+954 -2
View File
File diff suppressed because it is too large Load Diff
Generated
+4 -4
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.3"
version = "0.6.0"
source = { editable = "." }
dependencies = [
{ name = "click" },
@@ -836,11 +836,11 @@ wheels = [
[[package]]
name = "pygments"
version = "2.19.2"
version = "2.20.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
]
[[package]]