Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 18df96872b | |||
| 5b1d54bfe9 | |||
| ad9e513253 | |||
| 334f01b647 | |||
| 1b69e8f08e | |||
| 9957b0a1de |
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
|
||||
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
|
||||
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
|
||||
- **Connection management**: Optional credentials, server, and timeout configuration
|
||||
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -50,30 +51,33 @@ python -m mt5cli -o account.csv account-info
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a recent trailing window |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
|
||||
| Command | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a recent trailing window |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a recent trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
|
||||
@@ -129,7 +133,28 @@ update_history_with_config(
|
||||
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
|
||||
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
|
||||
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
|
||||
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
|
||||
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
|
||||
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
|
||||
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
|
||||
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
|
||||
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
|
||||
|
||||
rates = collect_latest_closed_rates_by_granularity(
|
||||
[AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
)
|
||||
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
|
||||
```
|
||||
|
||||
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
|
||||
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`/`sqlite3.Error` and let the caller decide logging.
|
||||
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
|
||||
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
|
||||
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
|
||||
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
|
||||
|
||||
|
||||
+65
-2
@@ -133,8 +133,8 @@ The `update_history` SDK path uses the same base tables and optional
|
||||
### Rate view resolution
|
||||
|
||||
Downstream tools can resolve mt5cli-managed compatibility view names from an
|
||||
existing SQLite history database without creating files or guessing legacy
|
||||
naming schemes:
|
||||
existing SQLite history database without creating files or guessing naming
|
||||
schemes:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
@@ -164,3 +164,66 @@ Resolution rules:
|
||||
- Pass `require_existing=True` to raise `ValueError` instead of returning a
|
||||
best-guess name when the database or view is missing.
|
||||
- Accepts either a SQLite path or an open `sqlite3.Connection`.
|
||||
|
||||
### Rate data loading
|
||||
|
||||
Use `load_rate_data()` to load a table or view from a SQLite path, or
|
||||
`load_rate_data_from_connection()` when you already have a connection:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import load_rate_data
|
||||
from mt5cli.history import resolve_rate_view_name
|
||||
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
|
||||
rates = load_rate_data(Path("history.db"), view, count=1000)
|
||||
```
|
||||
|
||||
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
|
||||
validates that `time` exists, parses timestamps with pandas, and returns a
|
||||
DataFrame indexed by ascending `DatetimeIndex` named `time`.
|
||||
|
||||
### Multi-series rate loading
|
||||
|
||||
For loading many rate series at once, build neutral `RateTarget` pairs and load
|
||||
them from SQLite in one call. View names are resolved via the same
|
||||
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
|
||||
|
||||
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
|
||||
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
|
||||
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
|
||||
```
|
||||
|
||||
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
|
||||
row-major order, normalizing timeframe names such as `"M1"` to their integer
|
||||
values; set `allow_missing_symbol=True` to address series solely by
|
||||
`explicit_tables` (targets carry `symbol=None`).
|
||||
- `resolve_rate_tables()` maps targets to table or view names and validates that
|
||||
any `explicit_tables` count matches the target count. Pass
|
||||
`require_existing=True` to raise `ValueError` instead of returning a
|
||||
best-guess name when the database or managed view is missing. When
|
||||
`explicit_tables` is provided, names are returned as-is and
|
||||
`require_existing` is ignored.
|
||||
- `load_rate_series_from_sqlite()` returns a mapping keyed by
|
||||
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
|
||||
requires existing managed `rate_*` compatibility views and raises
|
||||
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
|
||||
are rejected.
|
||||
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
|
||||
loads the series, and rekeys the result by granularity name to avoid
|
||||
converting integer timeframes downstream:
|
||||
|
||||
```python
|
||||
from mt5cli import load_rate_series_by_granularity
|
||||
|
||||
series = load_rate_series_by_granularity(
|
||||
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
|
||||
)
|
||||
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
|
||||
```
|
||||
|
||||
+100
@@ -1,3 +1,103 @@
|
||||
# SDK Module
|
||||
|
||||
::: mt5cli.sdk
|
||||
|
||||
## Resilient multi-account orchestration
|
||||
|
||||
The SDK ships strategy-agnostic helpers for building long-running collectors on
|
||||
top of the read-only client. None of them depend on a particular trading
|
||||
application.
|
||||
|
||||
### Retrying transient rate collection
|
||||
|
||||
`collect_latest_rates_for_accounts_with_retries()` wraps
|
||||
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
|
||||
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
|
||||
failure is re-raised once `retry_count` is exhausted.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
|
||||
|
||||
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
|
||||
rates = collect_latest_rates_for_accounts_with_retries(
|
||||
accounts,
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
|
||||
)
|
||||
```
|
||||
|
||||
### Latest closed rate bars
|
||||
|
||||
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
|
||||
row. `collect_latest_closed_rates_for_accounts()` fetches `count + 1` bars,
|
||||
drops that row with `drop_forming_rate_bar()`, and validates each series is
|
||||
non-empty. Use `collect_latest_closed_rates_by_granularity()` when callers
|
||||
prefer keys such as `("EURUSD", "M1")` instead of integer timeframes.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
|
||||
|
||||
rates = collect_latest_closed_rates_by_granularity(
|
||||
[AccountSpec(symbols=["EURUSD"], login=12345)],
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
)
|
||||
closed_m1 = rates["EURUSD", "M1"]
|
||||
```
|
||||
|
||||
### Resolving credentials and `${ENV_VAR}` placeholders
|
||||
|
||||
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
|
||||
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
|
||||
secrets out of plan/config files. A missing environment variable raises
|
||||
`ValueError`.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mt5cli import AccountSpec, resolve_account_specs
|
||||
|
||||
os.environ["MT5_LOGIN"] = "12345"
|
||||
os.environ["MT5_PASSWORD"] = "secret"
|
||||
accounts = [
|
||||
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
|
||||
]
|
||||
|
||||
resolved = resolve_account_specs(accounts, server="Broker-Demo")
|
||||
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
|
||||
```
|
||||
|
||||
### Throttled incremental history updates
|
||||
|
||||
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
|
||||
between successful runs (using a monotonic clock), so an application loop can
|
||||
call it every iteration without over-fetching.
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
|
||||
from mt5cli import Dataset, ThrottledHistoryUpdater
|
||||
|
||||
updater = ThrottledHistoryUpdater(
|
||||
output="history.db",
|
||||
datasets={Dataset.rates},
|
||||
timeframes=["M1"],
|
||||
interval_seconds=60, # <= 0 updates on every call
|
||||
)
|
||||
|
||||
client = Mt5DataClient(config=Mt5Config(login=12345))
|
||||
client.initialize_and_login_mt5()
|
||||
try:
|
||||
while True:
|
||||
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
|
||||
# ... do other work; break when shutting down ...
|
||||
finally:
|
||||
client.shutdown()
|
||||
```
|
||||
|
||||
By default `Mt5TradingError`, `Mt5RuntimeError`, and `sqlite3.Error` propagate so
|
||||
the caller controls logging; pass `suppress_errors=True` to swallow them and
|
||||
return `False` without advancing the throttle.
|
||||
|
||||
+20
-9
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
|
||||
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
|
||||
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
|
||||
- **Connection management**: Optional credentials, server, and timeout configuration
|
||||
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -34,6 +35,7 @@ from mt5cli import (
|
||||
copy_rates_range,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
load_rate_data,
|
||||
minimum_margins,
|
||||
recent_ticks,
|
||||
)
|
||||
@@ -49,7 +51,8 @@ rates = copy_rates_range(
|
||||
export_dataframe(rates, Path("rates.csv"), "csv")
|
||||
|
||||
# Resolve SQLite rate compatibility views for downstream tools
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
|
||||
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
|
||||
|
||||
# Recent tick window and minimum margin summary
|
||||
ticks = recent_ticks("EURUSD", seconds=300)
|
||||
@@ -59,6 +62,9 @@ margins = minimum_margins("EURUSD")
|
||||
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
|
||||
account = client.account_info()
|
||||
positions = client.positions()
|
||||
latest = client.latest_rates("EURUSD", "M1", count=100)
|
||||
summary = client.mt5_summary()
|
||||
summary_table = client.mt5_summary_as_df()
|
||||
|
||||
# Bulk SQLite collection (same behavior as the collect-history CLI command)
|
||||
collect_history(
|
||||
@@ -74,6 +80,8 @@ collect_history(
|
||||
|
||||
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
|
||||
|
||||
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
@@ -104,6 +112,7 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
|
||||
| ---------------- | ---------------------------------- |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
|
||||
### Ticks
|
||||
@@ -130,14 +139,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
|
||||
|
||||
### Trading
|
||||
|
||||
| Command | Description |
|
||||
| ---------------- | ----------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| Command | Description |
|
||||
| ---------------------- | ----------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
|
||||
|
||||
@@ -2,11 +2,34 @@
|
||||
|
||||
from importlib.metadata import version
|
||||
|
||||
from .history import (
|
||||
RateTarget,
|
||||
build_rate_targets,
|
||||
build_rate_view_name,
|
||||
drop_forming_rate_bar,
|
||||
load_rate_data,
|
||||
load_rate_data_from_connection,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
resolve_history_datasets,
|
||||
resolve_history_tick_flags,
|
||||
resolve_history_timeframes,
|
||||
resolve_rate_tables,
|
||||
resolve_rate_view_name,
|
||||
resolve_rate_view_names,
|
||||
)
|
||||
from .sdk import (
|
||||
AccountSpec,
|
||||
Mt5CliClient,
|
||||
ThrottledHistoryUpdater,
|
||||
account_info,
|
||||
build_config,
|
||||
collect_history,
|
||||
collect_latest_closed_rates_by_granularity,
|
||||
collect_latest_closed_rates_for_accounts,
|
||||
collect_latest_rates,
|
||||
collect_latest_rates_for_accounts,
|
||||
collect_latest_rates_for_accounts_with_retries,
|
||||
copy_rates_from,
|
||||
copy_rates_from_pos,
|
||||
copy_rates_range,
|
||||
@@ -15,11 +38,19 @@ from .sdk import (
|
||||
history_deals,
|
||||
history_orders,
|
||||
last_error,
|
||||
latest_rates,
|
||||
market_book,
|
||||
minimum_margins,
|
||||
mt5_session,
|
||||
mt5_summary,
|
||||
mt5_summary_as_df,
|
||||
orders,
|
||||
positions,
|
||||
recent_history_deals,
|
||||
recent_ticks,
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
substitute_env_placeholders,
|
||||
symbol_info,
|
||||
symbol_info_tick,
|
||||
symbols,
|
||||
@@ -31,39 +62,78 @@ from .sdk import (
|
||||
version as mt5_version,
|
||||
)
|
||||
from .utils import (
|
||||
TICK_FLAG_MAP,
|
||||
TIMEFRAME_MAP,
|
||||
Dataset,
|
||||
IfExists,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
parse_datetime,
|
||||
parse_tick_flags,
|
||||
parse_timeframe,
|
||||
)
|
||||
|
||||
__version__ = version(__package__) if __package__ else None
|
||||
|
||||
__all__ = [
|
||||
"TICK_FLAG_MAP",
|
||||
"TIMEFRAME_MAP",
|
||||
"AccountSpec",
|
||||
"Dataset",
|
||||
"IfExists",
|
||||
"Mt5CliClient",
|
||||
"RateTarget",
|
||||
"ThrottledHistoryUpdater",
|
||||
"account_info",
|
||||
"build_config",
|
||||
"build_rate_targets",
|
||||
"build_rate_view_name",
|
||||
"collect_history",
|
||||
"collect_latest_closed_rates_by_granularity",
|
||||
"collect_latest_closed_rates_for_accounts",
|
||||
"collect_latest_rates",
|
||||
"collect_latest_rates_for_accounts",
|
||||
"collect_latest_rates_for_accounts_with_retries",
|
||||
"copy_rates_from",
|
||||
"copy_rates_from_pos",
|
||||
"copy_rates_range",
|
||||
"copy_ticks_from",
|
||||
"copy_ticks_range",
|
||||
"detect_format",
|
||||
"drop_forming_rate_bar",
|
||||
"export_dataframe",
|
||||
"export_dataframe_to_sqlite",
|
||||
"history_deals",
|
||||
"history_orders",
|
||||
"last_error",
|
||||
"latest_rates",
|
||||
"load_rate_data",
|
||||
"load_rate_data_from_connection",
|
||||
"load_rate_series_by_granularity",
|
||||
"load_rate_series_from_sqlite",
|
||||
"market_book",
|
||||
"minimum_margins",
|
||||
"mt5_session",
|
||||
"mt5_summary",
|
||||
"mt5_summary_as_df",
|
||||
"mt5_version",
|
||||
"orders",
|
||||
"parse_datetime",
|
||||
"parse_tick_flags",
|
||||
"parse_timeframe",
|
||||
"positions",
|
||||
"recent_history_deals",
|
||||
"recent_ticks",
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"resolve_history_datasets",
|
||||
"resolve_history_tick_flags",
|
||||
"resolve_history_timeframes",
|
||||
"resolve_rate_tables",
|
||||
"resolve_rate_view_name",
|
||||
"resolve_rate_view_names",
|
||||
"substitute_env_placeholders",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
|
||||
@@ -222,6 +222,31 @@ def rates_from_pos(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def latest_rates(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
timeframe: Annotated[
|
||||
int,
|
||||
typer.Option(
|
||||
click_type=TIMEFRAME_TYPE,
|
||||
help="Timeframe.",
|
||||
),
|
||||
],
|
||||
count: Annotated[int, typer.Option(help="Number of records.")],
|
||||
start_pos: Annotated[
|
||||
int,
|
||||
typer.Option(help="Start position (0 = current bar)."),
|
||||
] = 0,
|
||||
) -> None:
|
||||
"""Export latest rates from a start position."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(
|
||||
ctx,
|
||||
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def rates_range(
|
||||
ctx: typer.Context,
|
||||
@@ -475,6 +500,37 @@ def history_deals(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def recent_history_deals(
|
||||
ctx: typer.Context,
|
||||
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
|
||||
date_to: Annotated[
|
||||
datetime | None,
|
||||
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
|
||||
] = None,
|
||||
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
|
||||
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
|
||||
) -> None:
|
||||
"""Export historical deals from a recent trailing window."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(
|
||||
ctx,
|
||||
lambda: client.recent_history_deals(
|
||||
hours,
|
||||
date_to=date_to,
|
||||
group=group,
|
||||
symbol=symbol,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def mt5_summary(ctx: typer.Context) -> None:
|
||||
"""Export a compact terminal/account status summary."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(ctx, client.mt5_summary_as_df)
|
||||
|
||||
|
||||
@app.command()
|
||||
def version(ctx: typer.Context) -> None:
|
||||
"""Export MetaTrader5 version information."""
|
||||
|
||||
+475
-18
@@ -4,9 +4,10 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from typing import TYPE_CHECKING, Literal, cast
|
||||
|
||||
import pandas as pd
|
||||
|
||||
@@ -20,7 +21,7 @@ from .utils import (
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Sequence
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
|
||||
from pdmt5 import Mt5DataClient
|
||||
|
||||
@@ -105,6 +106,23 @@ def resolve_granularity_name(timeframe: int) -> str:
|
||||
return str(timeframe)
|
||||
|
||||
|
||||
def drop_forming_rate_bar(df_rate: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Return closed bars from chronologically ordered MT5 rate data.
|
||||
|
||||
MetaTrader 5 ``copy_rates_from_pos(start_pos=0)`` includes the still-forming
|
||||
current bar as the last row. Slice it off so downstream logic only sees
|
||||
completed bars. Empty frames and single-row frames return empty results.
|
||||
|
||||
Args:
|
||||
df_rate: Rate data ordered oldest-to-newest with the forming bar last.
|
||||
|
||||
Returns:
|
||||
A new DataFrame with all rows except the last. Index and columns are
|
||||
preserved. The input frame is not modified.
|
||||
"""
|
||||
return df_rate.iloc[:-1].copy()
|
||||
|
||||
|
||||
def build_rate_view_name(
|
||||
*,
|
||||
symbol: str,
|
||||
@@ -126,15 +144,26 @@ def build_rate_view_name(
|
||||
SqliteConnOrPath = sqlite3.Connection | Path | str
|
||||
|
||||
|
||||
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
|
||||
value = identifier.strip()
|
||||
if not value:
|
||||
msg = f"SQLite {kind} name must not be empty."
|
||||
raise ValueError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def _open_history_connection(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
) -> tuple[sqlite3.Connection | None, bool]:
|
||||
"""Open a read-only SQLite connection when given a path.
|
||||
|
||||
Returns:
|
||||
A connection and whether the caller should close it. When the path does
|
||||
not exist, returns ``(None, False)`` without creating a database file.
|
||||
A connection and whether the caller should close it. When ``conn_or_path``
|
||||
is None or the path does not exist, returns ``(None, False)`` without
|
||||
creating a database file.
|
||||
"""
|
||||
if conn_or_path is None:
|
||||
return None, False
|
||||
if isinstance(conn_or_path, sqlite3.Connection):
|
||||
return conn_or_path, False
|
||||
path = Path(conn_or_path)
|
||||
@@ -144,6 +173,133 @@ def _open_history_connection(
|
||||
return conn, True
|
||||
|
||||
|
||||
def _open_existing_sqlite_database(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
) -> tuple[sqlite3.Connection, bool]:
|
||||
"""Open a read-only SQLite database or reuse an existing connection.
|
||||
|
||||
Returns:
|
||||
Tuple of connection and whether the caller should close it.
|
||||
|
||||
Raises:
|
||||
ValueError: If the database path does not exist or is not a file.
|
||||
"""
|
||||
if isinstance(conn_or_path, sqlite3.Connection):
|
||||
return conn_or_path, False
|
||||
path = Path(conn_or_path)
|
||||
if not path.exists():
|
||||
msg = f"SQLite database not found: {path}"
|
||||
raise ValueError(msg)
|
||||
if not path.is_file():
|
||||
msg = f"SQLite database path is not a file: {path}"
|
||||
raise ValueError(msg)
|
||||
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
|
||||
return conn, True
|
||||
|
||||
|
||||
def _validate_rate_load_request(table: str, count: int | None) -> str:
|
||||
table_name = _require_non_empty_identifier(table, "table or view")
|
||||
if count is not None and count <= 0:
|
||||
msg = "count must be positive when provided."
|
||||
raise ValueError(msg)
|
||||
return table_name
|
||||
|
||||
|
||||
def _ensure_rate_columns(columns: set[str], table: str) -> None:
|
||||
if not columns:
|
||||
msg = f"SQLite table or view not found: {table}"
|
||||
raise ValueError(msg)
|
||||
if "time" not in columns:
|
||||
msg = f"SQLite table or view {table!r} must include a time column."
|
||||
raise ValueError(msg)
|
||||
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
|
||||
msg = (
|
||||
f"SQLite table or view {table!r} must include close, "
|
||||
"or both ask and bid columns."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
|
||||
parsed = frame["time"].map(parse_sqlite_timestamp)
|
||||
if parsed.isna().any():
|
||||
msg = f"SQLite table or view {table!r} contains unparsable time values."
|
||||
raise ValueError(msg)
|
||||
result = frame.drop(columns=["time"])
|
||||
result.index = pd.DatetimeIndex(parsed, name="time")
|
||||
return result.sort_index(kind="stable")
|
||||
|
||||
|
||||
def load_rate_data_from_connection(
|
||||
connection: sqlite3.Connection,
|
||||
table: str,
|
||||
count: int | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Load rate-like data from a SQLite table or view.
|
||||
|
||||
Args:
|
||||
connection: Open SQLite connection.
|
||||
table: Source table or view name.
|
||||
count: Optional number of most recent rows to load.
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by ascending ``time``.
|
||||
|
||||
Raises:
|
||||
ValueError: If inputs, schema, timestamps are invalid, or the table
|
||||
or view contains no rows.
|
||||
"""
|
||||
table_name = _validate_rate_load_request(table, count)
|
||||
columns = get_table_columns(connection, table_name)
|
||||
_ensure_rate_columns(columns, table_name)
|
||||
quoted_table = quote_sqlite_identifier(table_name)
|
||||
if count is None:
|
||||
frame = cast(
|
||||
"pd.DataFrame",
|
||||
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
|
||||
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
|
||||
connection,
|
||||
),
|
||||
)
|
||||
else:
|
||||
frame = cast(
|
||||
"pd.DataFrame",
|
||||
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
|
||||
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
|
||||
connection,
|
||||
params=(count,),
|
||||
),
|
||||
)
|
||||
if frame.empty:
|
||||
msg = f"SQLite table or view {table_name!r} contains no rows."
|
||||
raise ValueError(msg)
|
||||
return _parse_rate_time_index(frame, table_name)
|
||||
|
||||
|
||||
def load_rate_data(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
table: str,
|
||||
count: int | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Load rate-like data from a SQLite database path or connection.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
table: Source table or view name.
|
||||
count: Optional number of most recent rows to load.
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by ascending ``time``.
|
||||
|
||||
"""
|
||||
conn, should_close = _open_existing_sqlite_database(conn_or_path)
|
||||
try:
|
||||
return load_rate_data_from_connection(conn, table, count=count)
|
||||
finally:
|
||||
if should_close:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
|
||||
"""Return distinct timeframe counts per symbol from the normalized rates table."""
|
||||
columns = get_table_columns(conn, Dataset.rates.table_name)
|
||||
@@ -241,7 +397,7 @@ def _resolve_rate_view_name_from_context(
|
||||
|
||||
|
||||
def resolve_rate_view_name(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
symbol: str,
|
||||
granularity: str,
|
||||
*,
|
||||
@@ -250,7 +406,9 @@ def resolve_rate_view_name(
|
||||
"""Resolve the mt5cli-managed rate compatibility view name.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
conn_or_path: SQLite database path or open connection. When None or a
|
||||
non-existing path and ``require_existing`` is False, the deterministic
|
||||
default view name is returned without creating a database file.
|
||||
symbol: Symbol stored in the normalized ``rates`` table.
|
||||
granularity: Timeframe name (for example ``M1``) or integer string.
|
||||
require_existing: When True, require the database and a managed view to exist.
|
||||
@@ -294,7 +452,7 @@ def resolve_rate_view_name(
|
||||
|
||||
|
||||
def resolve_rate_view_names(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
symbols: Sequence[str],
|
||||
granularities: Sequence[str],
|
||||
*,
|
||||
@@ -303,7 +461,9 @@ def resolve_rate_view_names(
|
||||
"""Resolve rate compatibility view names for symbol and granularity pairs.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
conn_or_path: SQLite database path or open connection. When None or a
|
||||
non-existing path and ``require_existing`` is False, deterministic
|
||||
default view names are returned without creating a database file.
|
||||
symbols: Symbols stored in the normalized ``rates`` table.
|
||||
granularities: Timeframe names (for example ``M1``) or integer strings.
|
||||
require_existing: When True, require the database and managed views to exist.
|
||||
@@ -347,9 +507,275 @@ def resolve_rate_view_names(
|
||||
conn.close()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RateTarget:
|
||||
"""A single rate series identified by symbol and timeframe.
|
||||
|
||||
Attributes:
|
||||
symbol: MT5 symbol name, or None when the rate series is addressed only
|
||||
by an explicit table (for example a custom SQLite view).
|
||||
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
|
||||
"""
|
||||
|
||||
symbol: str | None
|
||||
timeframe: int | str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""Normalize accepted timeframe aliases to the stored integer value."""
|
||||
if not isinstance(self.timeframe, int):
|
||||
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
|
||||
|
||||
@property
|
||||
def timeframe_int(self) -> int:
|
||||
"""Return the timeframe as its integer MT5 value."""
|
||||
return cast("int", self.timeframe)
|
||||
|
||||
|
||||
def build_rate_targets(
|
||||
symbols: Sequence[str],
|
||||
timeframes: Sequence[int | str],
|
||||
*,
|
||||
allow_missing_symbol: bool = False,
|
||||
) -> list[RateTarget]:
|
||||
"""Build rate targets for every symbol and timeframe combination.
|
||||
|
||||
Args:
|
||||
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
|
||||
timeframes: MT5 timeframes as integers or names (for example ``M1``).
|
||||
allow_missing_symbol: When True and ``symbols`` is empty, build targets
|
||||
with ``symbol=None`` for each timeframe instead of raising.
|
||||
|
||||
Returns:
|
||||
Targets in row-major order: every timeframe for the first symbol, then
|
||||
every timeframe for the next symbol, and so on.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
|
||||
``allow_missing_symbol`` is False.
|
||||
"""
|
||||
if not timeframes:
|
||||
msg = "At least one timeframe is required."
|
||||
raise ValueError(msg)
|
||||
if not symbols:
|
||||
if not allow_missing_symbol:
|
||||
msg = "At least one symbol is required."
|
||||
raise ValueError(msg)
|
||||
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
|
||||
return [
|
||||
RateTarget(symbol=symbol, timeframe=tf)
|
||||
for symbol in symbols
|
||||
for tf in timeframes
|
||||
]
|
||||
|
||||
|
||||
def resolve_rate_tables(
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
targets: Sequence[RateTarget],
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
*,
|
||||
require_existing: bool = False,
|
||||
) -> list[str]:
|
||||
"""Resolve SQLite table or view names for rate targets.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection. May be None when
|
||||
``explicit_tables`` is provided, or when ``require_existing`` is
|
||||
False and deterministic default view names are sufficient.
|
||||
targets: Rate targets to resolve.
|
||||
explicit_tables: Optional explicit table or view names. When provided,
|
||||
they are used as-is and must match the number of targets.
|
||||
require_existing: When True, require the database and managed views to
|
||||
exist for each symbol target. Ignored when ``explicit_tables`` is
|
||||
provided.
|
||||
|
||||
Returns:
|
||||
Table or view names aligned with ``targets``.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
|
||||
match the target count, a target without a symbol is resolved
|
||||
without an explicit table, or ``require_existing`` is True and the
|
||||
database or a managed view is missing.
|
||||
"""
|
||||
target_list = list(targets)
|
||||
if not target_list:
|
||||
msg = "At least one rate target is required."
|
||||
raise ValueError(msg)
|
||||
if explicit_tables is not None:
|
||||
tables = list(explicit_tables)
|
||||
if len(tables) != len(target_list):
|
||||
msg = (
|
||||
f"Expected {len(target_list)} explicit table(s) "
|
||||
f"to match the targets, got {len(tables)}."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
return tables
|
||||
if any(target.symbol is None for target in target_list):
|
||||
msg = (
|
||||
"Cannot resolve a rate table for a target without a symbol; "
|
||||
"provide explicit_tables."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
conn, should_close = _open_history_connection(conn_or_path)
|
||||
try:
|
||||
if conn is None:
|
||||
if require_existing:
|
||||
path = (
|
||||
conn_or_path
|
||||
if isinstance(conn_or_path, (Path, str))
|
||||
else "database"
|
||||
)
|
||||
msg = f"SQLite database not found: {path}"
|
||||
raise ValueError(msg)
|
||||
timeframe_counts = None
|
||||
existing_views: set[str] = set()
|
||||
else:
|
||||
timeframe_counts = _load_rates_timeframe_counts(conn)
|
||||
existing_views = _load_existing_rate_views(conn)
|
||||
resolved: list[str] = []
|
||||
for target in target_list:
|
||||
symbol = cast("str", target.symbol)
|
||||
timeframe = target.timeframe_int
|
||||
resolved.append(
|
||||
_resolve_rate_view_name_from_context(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
granularity_name=resolve_granularity_name(timeframe),
|
||||
timeframe_counts=timeframe_counts,
|
||||
existing_views=existing_views,
|
||||
require_existing=require_existing,
|
||||
),
|
||||
)
|
||||
return resolved
|
||||
finally:
|
||||
if should_close and conn is not None:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_rate_series_from_sqlite(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
targets: Sequence[RateTarget],
|
||||
count: int,
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
) -> dict[tuple[str | None, int], pd.DataFrame]:
|
||||
"""Load multiple rate series from a SQLite database.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair
|
||||
must be unique.
|
||||
count: Number of most recent rows to load per series.
|
||||
explicit_tables: Optional explicit table or view names matching targets.
|
||||
When omitted, managed ``rate_*`` compatibility views must already
|
||||
exist in the database.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)`` to each rate DataFrame.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``count`` is not positive, targets are empty, duplicate
|
||||
``(symbol, timeframe_int)`` pairs are present, or table resolution
|
||||
fails.
|
||||
"""
|
||||
if count <= 0:
|
||||
msg = "count must be positive."
|
||||
raise ValueError(msg)
|
||||
target_list = list(targets)
|
||||
if not target_list:
|
||||
msg = "At least one rate target is required."
|
||||
raise ValueError(msg)
|
||||
if explicit_tables is None and any(target.symbol is None for target in target_list):
|
||||
msg = (
|
||||
"Cannot resolve a rate table for a target without a symbol; "
|
||||
"provide explicit_tables."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
seen_keys: set[tuple[str | None, int]] = set()
|
||||
for target in target_list:
|
||||
key = (target.symbol, target.timeframe_int)
|
||||
if key in seen_keys:
|
||||
symbol_repr = repr(target.symbol)
|
||||
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
|
||||
raise ValueError(msg)
|
||||
seen_keys.add(key)
|
||||
tables = (
|
||||
resolve_rate_tables(None, target_list, explicit_tables)
|
||||
if explicit_tables is not None
|
||||
else None
|
||||
)
|
||||
conn, should_close = _open_existing_sqlite_database(conn_or_path)
|
||||
try:
|
||||
resolved_tables = tables or resolve_rate_tables(
|
||||
conn,
|
||||
target_list,
|
||||
require_existing=True,
|
||||
)
|
||||
return {
|
||||
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
|
||||
conn,
|
||||
table,
|
||||
count=count,
|
||||
)
|
||||
for target, table in zip(target_list, resolved_tables, strict=True)
|
||||
}
|
||||
finally:
|
||||
if should_close:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_rate_series_by_granularity(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
symbols: Sequence[str],
|
||||
granularities: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
allow_missing_symbol: bool = False,
|
||||
) -> dict[tuple[str | None, str], pd.DataFrame]:
|
||||
"""Load rate series keyed by symbol and string granularity name.
|
||||
|
||||
Builds targets with :func:`build_rate_targets` and loads them with
|
||||
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
|
||||
name (for example ``M1``) instead of the integer timeframe to reduce
|
||||
downstream boilerplate.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
|
||||
granularities: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of most recent rows to load per series.
|
||||
explicit_tables: Optional explicit table or view names matching the
|
||||
built targets in row-major order. Required when symbols are omitted.
|
||||
allow_missing_symbol: When True and ``symbols`` is empty, build targets
|
||||
with ``symbol=None`` for each granularity instead of raising.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
|
||||
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
|
||||
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
|
||||
table resolution fails, or duplicate targets are present.
|
||||
"""
|
||||
targets = build_rate_targets(
|
||||
symbols,
|
||||
granularities,
|
||||
allow_missing_symbol=allow_missing_symbol,
|
||||
)
|
||||
series = load_rate_series_from_sqlite(
|
||||
conn_or_path,
|
||||
targets,
|
||||
count,
|
||||
explicit_tables=explicit_tables,
|
||||
)
|
||||
return {
|
||||
(symbol, resolve_granularity_name(timeframe)): frame
|
||||
for (symbol, timeframe), frame in series.items()
|
||||
}
|
||||
|
||||
|
||||
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
|
||||
"""Return existing SQLite columns for a table."""
|
||||
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
|
||||
quoted_table = quote_sqlite_identifier(table)
|
||||
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
|
||||
return {str(row[1]) for row in rows}
|
||||
|
||||
|
||||
@@ -629,7 +1055,20 @@ def drop_duplicates_in_table(
|
||||
)
|
||||
|
||||
|
||||
DedupScope = tuple[str, tuple[object, ...]]
|
||||
@dataclass(frozen=True)
|
||||
class DedupScope:
|
||||
"""Scoped deduplication predicate and the columns it references.
|
||||
|
||||
Attributes:
|
||||
where: SQL predicate appended to the duplicate-removal query.
|
||||
params: Parameters bound to the scope predicate.
|
||||
required_columns: Columns that must be present in the written table for
|
||||
the scope to run.
|
||||
"""
|
||||
|
||||
where: str
|
||||
params: tuple[object, ...]
|
||||
required_columns: frozenset[str]
|
||||
|
||||
|
||||
def _record_dedup_scope(
|
||||
@@ -637,17 +1076,25 @@ def _record_dedup_scope(
|
||||
dataset: Dataset,
|
||||
scope_where: str,
|
||||
scope_params: tuple[object, ...],
|
||||
required_columns: frozenset[str],
|
||||
) -> None:
|
||||
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
|
||||
dedup_scopes.setdefault(dataset, []).append(
|
||||
DedupScope(scope_where, scope_params, required_columns),
|
||||
)
|
||||
|
||||
|
||||
def deduplicate_history_tables(
|
||||
conn: sqlite3.Connection,
|
||||
written_columns: dict[Dataset, set[str]],
|
||||
written_tables: set[Dataset],
|
||||
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
|
||||
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
|
||||
) -> None:
|
||||
"""Deduplicate appended history tables by stable identifiers."""
|
||||
"""Deduplicate appended history tables by stable identifiers.
|
||||
|
||||
Scopes whose required columns are not present in the written table are
|
||||
skipped. If all scopes for a dataset are skipped, the table receives one
|
||||
unscoped deduplication pass instead.
|
||||
"""
|
||||
cursor = conn.cursor()
|
||||
for dataset in written_tables:
|
||||
columns = written_columns.get(dataset, set())
|
||||
@@ -666,16 +1113,19 @@ def deduplicate_history_tables(
|
||||
table,
|
||||
)
|
||||
continue
|
||||
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
|
||||
raw_scopes: Sequence[DedupScope] = (
|
||||
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
|
||||
)
|
||||
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
|
||||
if scopes:
|
||||
for scope_where, scope_params in scopes:
|
||||
for scope in scopes:
|
||||
drop_duplicates_in_table(
|
||||
cursor,
|
||||
table,
|
||||
list(keys),
|
||||
keep="last",
|
||||
scope_where=scope_where,
|
||||
scope_params=scope_params,
|
||||
scope_where=scope.where,
|
||||
scope_params=scope.params,
|
||||
)
|
||||
continue
|
||||
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
|
||||
@@ -1042,6 +1492,7 @@ def _write_incremental_rates(
|
||||
Dataset.rates,
|
||||
"symbol = ? AND timeframe = ? AND time >= ?",
|
||||
(symbol, timeframe, start_date),
|
||||
frozenset({"symbol", "timeframe", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -1080,6 +1531,7 @@ def _write_incremental_ticks(
|
||||
Dataset.ticks,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -1118,6 +1570,7 @@ def _write_incremental_history_orders(
|
||||
Dataset.history_orders,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -1171,6 +1624,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_by_symbol[symbol, None]),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
if "type" in columns:
|
||||
_record_dedup_scope(
|
||||
@@ -1178,6 +1632,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
|
||||
(account_event_start,),
|
||||
frozenset({"type", "time"}),
|
||||
)
|
||||
if "type" not in columns and "symbol" in columns:
|
||||
_record_dedup_scope(
|
||||
@@ -1185,6 +1640,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"(symbol IS NULL OR symbol = '') AND time >= ?",
|
||||
(account_event_start,),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
return
|
||||
start_by_symbol = load_incremental_start_datetimes(
|
||||
@@ -1212,6 +1668,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
|
||||
+812
-6
@@ -2,21 +2,27 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Self, TypeVar
|
||||
from typing import TYPE_CHECKING, Self, TypeVar, cast
|
||||
|
||||
import pandas as pd
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError, Mt5TradingError
|
||||
|
||||
from .history import (
|
||||
create_cash_events_view,
|
||||
create_history_indexes,
|
||||
create_positions_reconstructed_view,
|
||||
drop_forming_rate_bar,
|
||||
resolve_granularity_name,
|
||||
resolve_history_datasets,
|
||||
resolve_history_tick_flags,
|
||||
resolve_history_timeframes,
|
||||
@@ -39,10 +45,17 @@ T = TypeVar("T")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = [
|
||||
"AccountSpec",
|
||||
"Mt5CliClient",
|
||||
"ThrottledHistoryUpdater",
|
||||
"account_info",
|
||||
"build_config",
|
||||
"collect_history",
|
||||
"collect_latest_closed_rates_by_granularity",
|
||||
"collect_latest_closed_rates_for_accounts",
|
||||
"collect_latest_rates",
|
||||
"collect_latest_rates_for_accounts",
|
||||
"collect_latest_rates_for_accounts_with_retries",
|
||||
"copy_rates_from",
|
||||
"copy_rates_from_pos",
|
||||
"copy_rates_range",
|
||||
@@ -51,11 +64,19 @@ __all__ = [
|
||||
"history_deals",
|
||||
"history_orders",
|
||||
"last_error",
|
||||
"latest_rates",
|
||||
"market_book",
|
||||
"minimum_margins",
|
||||
"mt5_session",
|
||||
"mt5_summary",
|
||||
"mt5_summary_as_df",
|
||||
"orders",
|
||||
"positions",
|
||||
"recent_history_deals",
|
||||
"recent_ticks",
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"substitute_env_placeholders",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
@@ -78,6 +99,22 @@ def _coerce_tick_flags(flags: int | str) -> int:
|
||||
return parse_tick_flags(flags)
|
||||
|
||||
|
||||
def _plain_mt5_value(value: object) -> object:
|
||||
asdict = getattr(value, "_asdict", None)
|
||||
if callable(asdict):
|
||||
return _plain_mt5_value(asdict())
|
||||
if isinstance(value, dict):
|
||||
typed_value = cast("dict[object, object]", value)
|
||||
return {key: _plain_mt5_value(item) for key, item in typed_value.items()}
|
||||
if isinstance(value, tuple):
|
||||
typed_value = cast("tuple[object, ...]", value)
|
||||
return [_plain_mt5_value(item) for item in typed_value]
|
||||
if isinstance(value, list):
|
||||
typed_value = cast("list[object]", value)
|
||||
return [_plain_mt5_value(item) for item in typed_value]
|
||||
return value
|
||||
|
||||
|
||||
def _require_datetime(value: datetime | str) -> datetime:
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
@@ -90,6 +127,37 @@ def _coerce_datetime(value: datetime | str | None) -> datetime | None:
|
||||
return parse_datetime(value)
|
||||
|
||||
|
||||
def _require_positive(value: float, name: str) -> None:
|
||||
if value <= 0:
|
||||
msg = f"{name} must be positive."
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _require_non_negative(value: int, name: str) -> None:
|
||||
if value < 0:
|
||||
msg = f"{name} must be non-negative."
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _call_required_client_method(client: Mt5DataClient, name: str) -> object:
|
||||
try:
|
||||
method = getattr(client, name)
|
||||
except AttributeError as exc:
|
||||
msg = f"MT5 client is missing required method: {name}"
|
||||
raise AttributeError(msg) from exc
|
||||
if not callable(method):
|
||||
msg = f"MT5 client attribute is not callable: {name}"
|
||||
raise TypeError(msg)
|
||||
return method()
|
||||
|
||||
|
||||
def _mt5_summary_export_value(value: object) -> object:
|
||||
plain_value = _plain_mt5_value(value)
|
||||
if isinstance(plain_value, dict | list):
|
||||
return json.dumps(plain_value, sort_keys=True, separators=(",", ":"))
|
||||
return plain_value
|
||||
|
||||
|
||||
def _coerce_tick_time(value: object) -> datetime:
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
@@ -230,6 +298,26 @@ def _run_with_client(
|
||||
return fetch_fn(client)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def mt5_session(config: Mt5Config | None = None) -> Iterator[Mt5CliClient]:
|
||||
"""Open an MT5 terminal session and yield a connected client.
|
||||
|
||||
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` (when set),
|
||||
logs in, yields a connected :class:`Mt5CliClient`, and always shuts the
|
||||
terminal down on exit.
|
||||
|
||||
Args:
|
||||
config: MT5 connection configuration. Defaults to an empty config that
|
||||
attaches to a running terminal.
|
||||
|
||||
Yields:
|
||||
Connected ``Mt5CliClient`` bound to the session.
|
||||
"""
|
||||
mt5_config = config or build_config()
|
||||
with _connected_client(mt5_config) as client:
|
||||
yield Mt5CliClient.from_connected_client(client)
|
||||
|
||||
|
||||
class Mt5CliClient:
|
||||
"""Programmatic client for read-only MetaTrader 5 data access."""
|
||||
|
||||
@@ -242,6 +330,7 @@ class Mt5CliClient:
|
||||
server: str | None = None,
|
||||
timeout: int | None = None,
|
||||
config: Mt5Config | None = None,
|
||||
client: Mt5DataClient | None = None,
|
||||
) -> None:
|
||||
"""Initialize the SDK client.
|
||||
|
||||
@@ -252,6 +341,8 @@ class Mt5CliClient:
|
||||
server: Trading server name.
|
||||
timeout: Connection timeout in milliseconds.
|
||||
config: Optional pre-built ``Mt5Config`` (overrides other args).
|
||||
client: Optional already-connected ``Mt5DataClient``. Injected
|
||||
clients are reused as-is and are not initialized or shut down.
|
||||
"""
|
||||
self._config = config or build_config(
|
||||
path=path,
|
||||
@@ -260,7 +351,20 @@ class Mt5CliClient:
|
||||
server=server,
|
||||
timeout=timeout,
|
||||
)
|
||||
self._client: Mt5DataClient | None = None
|
||||
self._client = client
|
||||
self._owns_client = client is None
|
||||
|
||||
@classmethod
|
||||
def from_connected_client(cls, client: Mt5DataClient) -> Self:
|
||||
"""Bind to an already-connected ``Mt5DataClient`` without owning it.
|
||||
|
||||
The returned ``Mt5CliClient`` never initializes or shuts down the
|
||||
injected client, including when used as a context manager.
|
||||
|
||||
Returns:
|
||||
Client wrapper bound to the injected connection.
|
||||
"""
|
||||
return cls(client=client)
|
||||
|
||||
@property
|
||||
def config(self) -> Mt5Config:
|
||||
@@ -273,6 +377,8 @@ class Mt5CliClient:
|
||||
Returns:
|
||||
This client instance.
|
||||
"""
|
||||
if self._client is not None:
|
||||
return self
|
||||
client = Mt5DataClient(config=self._config)
|
||||
try:
|
||||
client.initialize_and_login_mt5()
|
||||
@@ -280,6 +386,7 @@ class Mt5CliClient:
|
||||
client.shutdown()
|
||||
raise
|
||||
self._client = client
|
||||
self._owns_client = True # only set when this method created the client
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
@@ -289,15 +396,18 @@ class Mt5CliClient:
|
||||
tb: object,
|
||||
) -> None:
|
||||
"""Shut down the persistent MT5 connection."""
|
||||
if self._client is not None:
|
||||
if self._client is not None and self._owns_client:
|
||||
self._client.shutdown()
|
||||
self._client = None
|
||||
|
||||
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
|
||||
def _fetch_value(self, fetch_fn: Callable[[Mt5DataClient], T]) -> T:
|
||||
if self._client is not None:
|
||||
return fetch_fn(self._client)
|
||||
return _run_with_client(self._config, fetch_fn)
|
||||
|
||||
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
|
||||
return self._fetch_value(fetch_fn)
|
||||
|
||||
def copy_rates_from(
|
||||
self,
|
||||
symbol: str,
|
||||
@@ -335,6 +445,54 @@ class Mt5CliClient:
|
||||
),
|
||||
)
|
||||
|
||||
def latest_rates(
|
||||
self,
|
||||
symbol: str,
|
||||
timeframe: int | str,
|
||||
count: int,
|
||||
start_pos: int = 0,
|
||||
) -> pd.DataFrame:
|
||||
"""Return the latest rates from a bar position."""
|
||||
_require_positive(count, "count")
|
||||
return self.copy_rates_from_pos(symbol, timeframe, start_pos, count)
|
||||
|
||||
def collect_latest_rates(
|
||||
self,
|
||||
symbols: Sequence[str],
|
||||
timeframes: Sequence[int | str],
|
||||
*,
|
||||
count: int,
|
||||
start_pos: int = 0,
|
||||
) -> dict[tuple[str, int], pd.DataFrame]:
|
||||
"""Return latest rates for each symbol/timeframe pair.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)``.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``count`` is not positive or inputs are empty.
|
||||
"""
|
||||
_require_positive(count, "count")
|
||||
if not symbols:
|
||||
msg = "At least one symbol is required."
|
||||
raise ValueError(msg)
|
||||
if not timeframes:
|
||||
msg = "At least one timeframe is required."
|
||||
raise ValueError(msg)
|
||||
resolved_timeframes = [_coerce_timeframe(timeframe) for timeframe in timeframes]
|
||||
return self._fetch_value(
|
||||
lambda c: {
|
||||
(symbol, timeframe): c.copy_rates_from_pos_as_df(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
start_pos=start_pos,
|
||||
count=count,
|
||||
)
|
||||
for symbol in symbols
|
||||
for timeframe in resolved_timeframes
|
||||
},
|
||||
)
|
||||
|
||||
def copy_rates_range(
|
||||
self,
|
||||
symbol: str,
|
||||
@@ -486,6 +644,24 @@ class Mt5CliClient:
|
||||
),
|
||||
)
|
||||
|
||||
def recent_history_deals(
|
||||
self,
|
||||
hours: float,
|
||||
date_to: datetime | str | None = None,
|
||||
group: str | None = None,
|
||||
symbol: str | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Return historical deals from a recent trailing window."""
|
||||
_require_positive(hours, "hours")
|
||||
end = _require_datetime(date_to) if date_to is not None else datetime.now(UTC)
|
||||
start = end - timedelta(hours=hours)
|
||||
return self.history_deals(
|
||||
date_from=start,
|
||||
date_to=end,
|
||||
group=group,
|
||||
symbol=symbol,
|
||||
)
|
||||
|
||||
def version(self) -> pd.DataFrame:
|
||||
"""Return MetaTrader5 version information."""
|
||||
return self._fetch(lambda c: c.version_as_df())
|
||||
@@ -553,6 +729,39 @@ class Mt5CliClient:
|
||||
"""
|
||||
return self._fetch(lambda c: _fetch_minimum_margins(c, symbol))
|
||||
|
||||
def mt5_summary(self) -> dict[str, object]:
|
||||
"""Return a compact terminal/account status summary."""
|
||||
|
||||
def _summary(client: Mt5DataClient) -> dict[str, object]:
|
||||
return {
|
||||
"version": _plain_mt5_value(
|
||||
_call_required_client_method(client, "version"),
|
||||
),
|
||||
"terminal_info": _plain_mt5_value(
|
||||
_call_required_client_method(client, "terminal_info"),
|
||||
),
|
||||
"account_info": _plain_mt5_value(
|
||||
_call_required_client_method(client, "account_info"),
|
||||
),
|
||||
"symbols_total": _plain_mt5_value(
|
||||
_call_required_client_method(client, "symbols_total"),
|
||||
),
|
||||
}
|
||||
|
||||
return self._fetch_value(_summary)
|
||||
|
||||
def mt5_summary_as_df(self) -> pd.DataFrame:
|
||||
"""Return an export-safe one-row terminal/account summary DataFrame."""
|
||||
summary = self.mt5_summary()
|
||||
return pd.DataFrame(
|
||||
[
|
||||
{
|
||||
key: _mt5_summary_export_value(value)
|
||||
for key, value in summary.items()
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _resolve_incremental_settings(
|
||||
selected_datasets: set[Dataset],
|
||||
@@ -769,6 +978,115 @@ def update_history_with_config( # noqa: PLR0913
|
||||
)
|
||||
|
||||
|
||||
class ThrottledHistoryUpdater:
|
||||
"""Throttled incremental SQLite history updater for long-running apps.
|
||||
|
||||
Wraps :func:`update_history` with a minimum interval between successful
|
||||
updates, so a tight application loop can call :meth:`update` every
|
||||
iteration without re-fetching MT5 history more often than desired. Timing
|
||||
uses a monotonic clock, so it is unaffected by wall-clock changes.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
output: Path | str,
|
||||
datasets: set[Dataset] | None = None,
|
||||
timeframes: Sequence[int | str] | None = None,
|
||||
flags: int | str = "ALL",
|
||||
lookback_hours: float = 24.0,
|
||||
with_views: bool = False,
|
||||
include_account_events: bool = True,
|
||||
interval_seconds: float = 0.0,
|
||||
suppress_errors: bool = False,
|
||||
) -> None:
|
||||
"""Initialize the throttled updater.
|
||||
|
||||
Args:
|
||||
output: SQLite database path.
|
||||
datasets: Datasets to include (defaults to all).
|
||||
timeframes: Rate timeframes to update (defaults to all fixed MT5
|
||||
timeframes).
|
||||
flags: Tick copy flags as integer or name (e.g. ``ALL``).
|
||||
lookback_hours: First-run lookback when a table has no prior rows.
|
||||
with_views: Create ``cash_events`` and ``positions_reconstructed``
|
||||
views.
|
||||
include_account_events: Include account-level cash events.
|
||||
interval_seconds: Minimum seconds between successful updates. Values
|
||||
``<= 0`` update on every call.
|
||||
suppress_errors: When True, ``Mt5TradingError``, ``Mt5RuntimeError``,
|
||||
and ``sqlite3.Error`` raised during an update are swallowed and
|
||||
:meth:`update` returns False without advancing the throttle. When
|
||||
False (default), such errors propagate so callers control logging.
|
||||
"""
|
||||
self.output = output
|
||||
self.datasets = datasets
|
||||
self.timeframes = timeframes
|
||||
self.flags = flags
|
||||
self.lookback_hours = lookback_hours
|
||||
self.with_views = with_views
|
||||
self.include_account_events = include_account_events
|
||||
self.interval_seconds = interval_seconds
|
||||
self.suppress_errors = suppress_errors
|
||||
self._last_update_monotonic: float | None = None
|
||||
|
||||
@property
|
||||
def last_update_monotonic(self) -> float | None:
|
||||
"""Return the monotonic timestamp of the last successful update."""
|
||||
return self._last_update_monotonic
|
||||
|
||||
def should_update(self) -> bool:
|
||||
"""Return whether enough time has elapsed to run another update.
|
||||
|
||||
Returns:
|
||||
True when ``interval_seconds <= 0``, when no update has succeeded
|
||||
yet, or when at least ``interval_seconds`` have elapsed since the
|
||||
last successful update.
|
||||
"""
|
||||
if self.interval_seconds <= 0 or self._last_update_monotonic is None:
|
||||
return True
|
||||
return (time.monotonic() - self._last_update_monotonic) >= self.interval_seconds
|
||||
|
||||
def update(self, client: Mt5DataClient, symbols: Sequence[str]) -> bool:
|
||||
"""Run a throttled incremental history update.
|
||||
|
||||
Args:
|
||||
client: Connected MT5 data client.
|
||||
symbols: Symbols to update.
|
||||
|
||||
Returns:
|
||||
True if an update ran successfully, False if it was throttled or
|
||||
(when ``suppress_errors`` is True) failed with a recoverable error.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If the update fails and ``suppress_errors`` is False.
|
||||
Mt5RuntimeError: If the update fails and ``suppress_errors`` is False.
|
||||
sqlite3.Error: If the SQLite write fails and ``suppress_errors`` is
|
||||
False.
|
||||
"""
|
||||
if not self.should_update():
|
||||
return False
|
||||
try:
|
||||
update_history(
|
||||
client=client,
|
||||
output=self.output,
|
||||
symbols=symbols,
|
||||
datasets=self.datasets,
|
||||
timeframes=self.timeframes,
|
||||
flags=self.flags,
|
||||
lookback_hours=self.lookback_hours,
|
||||
with_views=self.with_views,
|
||||
include_account_events=self.include_account_events,
|
||||
)
|
||||
except (Mt5TradingError, Mt5RuntimeError, sqlite3.Error):
|
||||
if self.suppress_errors:
|
||||
logger.warning("Suppressed history update error", exc_info=True)
|
||||
return False
|
||||
raise
|
||||
self._last_update_monotonic = time.monotonic()
|
||||
return True
|
||||
|
||||
|
||||
def collect_history(
|
||||
output: Path,
|
||||
symbols: list[str],
|
||||
@@ -873,6 +1191,467 @@ def copy_rates_from_pos(
|
||||
)
|
||||
|
||||
|
||||
def latest_rates(
|
||||
symbol: str,
|
||||
timeframe: int | str,
|
||||
count: int,
|
||||
start_pos: int = 0,
|
||||
*,
|
||||
config: Mt5Config | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Return the latest rates from a bar position."""
|
||||
return _make_client(config=config).latest_rates(
|
||||
symbol,
|
||||
timeframe,
|
||||
count,
|
||||
start_pos=start_pos,
|
||||
)
|
||||
|
||||
|
||||
def collect_latest_rates(
|
||||
symbols: Sequence[str],
|
||||
timeframes: Sequence[int | str],
|
||||
*,
|
||||
count: int,
|
||||
start_pos: int = 0,
|
||||
config: Mt5Config | None = None,
|
||||
) -> dict[tuple[str, int], pd.DataFrame]:
|
||||
"""Return latest rates for each symbol/timeframe pair."""
|
||||
return _make_client(config=config).collect_latest_rates(
|
||||
symbols,
|
||||
timeframes,
|
||||
count=count,
|
||||
start_pos=start_pos,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AccountSpec:
|
||||
"""Connection parameters and symbols for one MT5 account group.
|
||||
|
||||
Attributes:
|
||||
symbols: Symbols to load latest rates for under this account.
|
||||
login: Trading account login. String values are coerced to int when
|
||||
non-empty.
|
||||
password: Trading account password.
|
||||
server: Trading server name.
|
||||
path: Path to the MetaTrader5 terminal EXE file.
|
||||
timeout: Connection timeout in milliseconds.
|
||||
"""
|
||||
|
||||
symbols: Sequence[str]
|
||||
login: int | str | None = field(default=None, repr=False)
|
||||
password: str | None = field(default=None, repr=False)
|
||||
server: str | None = None
|
||||
path: str | None = None
|
||||
timeout: int | None = None
|
||||
|
||||
|
||||
_ENV_PLACEHOLDER_PATTERN = re.compile(r"\$\{(?P<name>[A-Za-z_][A-Za-z0-9_]*)\}")
|
||||
|
||||
|
||||
def substitute_env_placeholders(value: str) -> str:
|
||||
"""Replace ``${ENV_VAR}`` placeholders in a string with environment values.
|
||||
|
||||
Args:
|
||||
value: String that may contain one or more ``${ENV_VAR}`` placeholders.
|
||||
|
||||
Returns:
|
||||
The string with every placeholder replaced by its environment value.
|
||||
|
||||
Raises:
|
||||
ValueError: If a referenced environment variable is not set.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
last_end = 0
|
||||
for match in _ENV_PLACEHOLDER_PATTERN.finditer(value):
|
||||
parts.append(value[last_end : match.start()])
|
||||
name = match.group("name")
|
||||
if name not in os.environ:
|
||||
msg = f"Environment variable {name!r} is not set."
|
||||
raise ValueError(msg)
|
||||
parts.append(os.environ[name])
|
||||
last_end = match.end()
|
||||
parts.append(value[last_end:])
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _resolve_field(override: str | None, account_value: str | None) -> str | None:
|
||||
"""Resolve a string field from an override or account value with env subst.
|
||||
|
||||
Returns:
|
||||
The explicit override when provided, otherwise the account value, with
|
||||
any ``${ENV_VAR}`` placeholders substituted.
|
||||
"""
|
||||
value = override if override is not None else account_value
|
||||
if value is None:
|
||||
return None
|
||||
return substitute_env_placeholders(value)
|
||||
|
||||
|
||||
def _resolve_login(
|
||||
override: int | str | None,
|
||||
account_login: int | str | None,
|
||||
) -> int | str | None:
|
||||
"""Resolve a login from an override or account value with env substitution.
|
||||
|
||||
Returns:
|
||||
The explicit override when provided, otherwise the account login.
|
||||
Integer values are preserved; string values have ``${ENV_VAR}``
|
||||
placeholders substituted.
|
||||
"""
|
||||
if override is not None:
|
||||
if isinstance(override, int):
|
||||
return override
|
||||
return substitute_env_placeholders(override)
|
||||
if account_login is None or isinstance(account_login, int):
|
||||
return account_login
|
||||
return substitute_env_placeholders(account_login)
|
||||
|
||||
|
||||
def resolve_account_spec(
|
||||
account: AccountSpec,
|
||||
*,
|
||||
login: int | str | None = None,
|
||||
password: str | None = None,
|
||||
server: str | None = None,
|
||||
path: str | None = None,
|
||||
timeout: int | None = None,
|
||||
) -> AccountSpec:
|
||||
"""Resolve an account's credentials from overrides and ``${ENV_VAR}`` values.
|
||||
|
||||
Explicit override arguments take precedence over the corresponding
|
||||
:class:`AccountSpec` fields. The resolved string fields (``login``,
|
||||
``password``, ``server``, ``path``) have any ``${ENV_VAR}`` placeholders
|
||||
substituted from the environment.
|
||||
|
||||
Args:
|
||||
account: Source account specification.
|
||||
login: Optional explicit login override.
|
||||
password: Optional explicit password override.
|
||||
server: Optional explicit server override.
|
||||
path: Optional explicit terminal path override.
|
||||
timeout: Optional explicit connection timeout override.
|
||||
|
||||
Returns:
|
||||
A new :class:`AccountSpec` with resolved credentials and the original
|
||||
symbols preserved. Raises ``ValueError`` (via
|
||||
:func:`substitute_env_placeholders`) if a referenced environment
|
||||
variable is not set.
|
||||
"""
|
||||
return AccountSpec(
|
||||
symbols=account.symbols,
|
||||
login=_resolve_login(login, account.login),
|
||||
password=_resolve_field(password, account.password),
|
||||
server=_resolve_field(server, account.server),
|
||||
path=_resolve_field(path, account.path),
|
||||
timeout=timeout if timeout is not None else account.timeout,
|
||||
)
|
||||
|
||||
|
||||
def resolve_account_specs(
|
||||
accounts: Sequence[AccountSpec],
|
||||
*,
|
||||
login: int | str | None = None,
|
||||
password: str | None = None,
|
||||
server: str | None = None,
|
||||
path: str | None = None,
|
||||
timeout: int | None = None,
|
||||
) -> list[AccountSpec]:
|
||||
"""Resolve credentials for multiple accounts.
|
||||
|
||||
Applies the same overrides and ``${ENV_VAR}`` substitution as
|
||||
:func:`resolve_account_spec` to every account.
|
||||
|
||||
Args:
|
||||
accounts: Source account specifications.
|
||||
login: Optional explicit login override applied to each account.
|
||||
password: Optional explicit password override applied to each account.
|
||||
server: Optional explicit server override applied to each account.
|
||||
path: Optional explicit terminal path override applied to each account.
|
||||
timeout: Optional explicit timeout override applied to each account.
|
||||
|
||||
Returns:
|
||||
Resolved account specifications in the original order. Raises
|
||||
``ValueError`` (via :func:`substitute_env_placeholders`) if a referenced
|
||||
environment variable is not set.
|
||||
"""
|
||||
return [
|
||||
resolve_account_spec(
|
||||
account,
|
||||
login=login,
|
||||
password=password,
|
||||
server=server,
|
||||
path=path,
|
||||
timeout=timeout,
|
||||
)
|
||||
for account in accounts
|
||||
]
|
||||
|
||||
|
||||
def _coerce_login(login: int | str | None) -> int | None:
|
||||
"""Coerce a login value to int, treating empty strings as unset.
|
||||
|
||||
Returns:
|
||||
Integer login, or None when unset or an empty string.
|
||||
"""
|
||||
if login is None or isinstance(login, int):
|
||||
return login
|
||||
text = login.strip()
|
||||
if not text:
|
||||
return None
|
||||
return int(text)
|
||||
|
||||
|
||||
def _build_account_config(
|
||||
account: AccountSpec,
|
||||
base_config: Mt5Config | None,
|
||||
) -> Mt5Config:
|
||||
"""Build an ``Mt5Config`` for an account, falling back to ``base_config``.
|
||||
|
||||
Returns:
|
||||
Merged MT5 configuration for the account.
|
||||
"""
|
||||
login = _coerce_login(account.login)
|
||||
if login is None and base_config is not None:
|
||||
login = base_config.login
|
||||
return build_config(
|
||||
path=account.path or (base_config.path if base_config else None),
|
||||
login=login,
|
||||
password=account.password or (base_config.password if base_config else None),
|
||||
server=account.server or (base_config.server if base_config else None),
|
||||
timeout=account.timeout
|
||||
if account.timeout is not None
|
||||
else (base_config.timeout if base_config else None),
|
||||
)
|
||||
|
||||
|
||||
def collect_latest_rates_for_accounts(
|
||||
accounts: Sequence[AccountSpec],
|
||||
timeframes: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
start_pos: int = 0,
|
||||
base_config: Mt5Config | None = None,
|
||||
) -> dict[tuple[str, int], pd.DataFrame]:
|
||||
"""Collect latest rates across multiple MT5 account groups.
|
||||
|
||||
Each account is connected in turn, its symbols are read for every
|
||||
timeframe, and the resulting frames are merged into a single mapping.
|
||||
|
||||
Args:
|
||||
accounts: Account groups to read. Each must define at least one symbol.
|
||||
timeframes: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of most recent bars to read per symbol/timeframe.
|
||||
start_pos: Initial bar position offset.
|
||||
base_config: Optional base configuration whose fields fill any value not
|
||||
set on an individual account.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)``. When accounts share a
|
||||
symbol/timeframe pair, the last account processed wins.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``accounts``, ``timeframes``, or any account's symbols are
|
||||
empty, or ``count`` is not positive.
|
||||
"""
|
||||
account_list = list(accounts)
|
||||
if not account_list:
|
||||
msg = "At least one account is required."
|
||||
raise ValueError(msg)
|
||||
if not timeframes:
|
||||
msg = "At least one timeframe is required."
|
||||
raise ValueError(msg)
|
||||
if any(not account.symbols for account in account_list):
|
||||
msg = "Each account requires at least one symbol."
|
||||
raise ValueError(msg)
|
||||
_require_positive(count, "count")
|
||||
result: dict[tuple[str, int], pd.DataFrame] = {}
|
||||
for account in account_list:
|
||||
config = _build_account_config(account, base_config)
|
||||
with Mt5CliClient(config=config) as client:
|
||||
result.update(
|
||||
client.collect_latest_rates(
|
||||
account.symbols,
|
||||
timeframes,
|
||||
count=count,
|
||||
start_pos=start_pos,
|
||||
),
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def collect_latest_rates_for_accounts_with_retries(
|
||||
accounts: Sequence[AccountSpec],
|
||||
timeframes: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
start_pos: int = 0,
|
||||
base_config: Mt5Config | None = None,
|
||||
retry_count: int = 0,
|
||||
backoff_base: float = 2.0,
|
||||
) -> dict[tuple[str, int], pd.DataFrame]:
|
||||
"""Collect latest rates across accounts, retrying transient MT5 failures.
|
||||
|
||||
Wraps :func:`collect_latest_rates_for_accounts` with bounded exponential
|
||||
backoff. Only ``pdmt5.Mt5TradingError`` and ``pdmt5.Mt5RuntimeError`` are
|
||||
retried; other exceptions propagate immediately. The final failure is
|
||||
re-raised once retries are exhausted.
|
||||
|
||||
Args:
|
||||
accounts: Account groups to read. Each must define at least one symbol.
|
||||
timeframes: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of most recent bars to read per symbol/timeframe.
|
||||
start_pos: Initial bar position offset.
|
||||
base_config: Optional base configuration whose fields fill any value not
|
||||
set on an individual account.
|
||||
retry_count: Maximum number of retries after the first attempt. ``0``
|
||||
disables retries.
|
||||
backoff_base: Base for exponential backoff. The delay before retry
|
||||
attempt ``n`` (1-indexed) is ``backoff_base ** n`` seconds.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)``. Propagates ``ValueError``
|
||||
for invalid inputs (see :func:`collect_latest_rates_for_accounts`) and
|
||||
re-raises the last ``pdmt5.Mt5TradingError`` or ``pdmt5.Mt5RuntimeError``
|
||||
once retries are exhausted.
|
||||
"""
|
||||
attempts = max(retry_count, 0) + 1
|
||||
|
||||
def _collect() -> dict[tuple[str, int], pd.DataFrame]:
|
||||
return collect_latest_rates_for_accounts(
|
||||
accounts,
|
||||
timeframes,
|
||||
count,
|
||||
start_pos=start_pos,
|
||||
base_config=base_config,
|
||||
)
|
||||
|
||||
for attempt in range(attempts - 1):
|
||||
try:
|
||||
return _collect()
|
||||
except (Mt5TradingError, Mt5RuntimeError) as exc:
|
||||
delay = backoff_base ** (attempt + 1)
|
||||
logger.warning(
|
||||
"Rate collection failed (attempt %d/%d): %s; retrying in %.1fs",
|
||||
attempt + 1,
|
||||
attempts,
|
||||
exc,
|
||||
delay,
|
||||
)
|
||||
time.sleep(delay)
|
||||
return _collect()
|
||||
|
||||
|
||||
def collect_latest_closed_rates_for_accounts(
|
||||
accounts: Sequence[AccountSpec],
|
||||
timeframes: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
start_pos: int = 0,
|
||||
base_config: Mt5Config | None = None,
|
||||
retry_count: int = 0,
|
||||
backoff_base: float = 2.0,
|
||||
) -> dict[tuple[str, int], pd.DataFrame]:
|
||||
"""Collect latest closed rate bars across multiple MT5 account groups.
|
||||
|
||||
When ``start_pos`` is ``0`` (the default), MetaTrader 5 includes the
|
||||
still-forming current bar as the last row. This helper fetches
|
||||
``count + 1`` bars, drops that bar with :func:`drop_forming_rate_bar`, and
|
||||
validates that each resulting frame is non-empty. When ``start_pos`` is
|
||||
greater than zero the forming bar is not in range, so only ``count`` bars
|
||||
are fetched and no row is dropped.
|
||||
|
||||
Wraps :func:`collect_latest_rates_for_accounts_with_retries` for transient
|
||||
MT5 error handling.
|
||||
|
||||
Args:
|
||||
accounts: Account groups to read. Each must define at least one symbol.
|
||||
timeframes: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of closed bars to return per symbol/timeframe.
|
||||
start_pos: Initial bar position offset passed to the underlying collector.
|
||||
base_config: Optional base configuration whose fields fill any value not
|
||||
set on an individual account.
|
||||
retry_count: Maximum number of retries after the first attempt. ``0``
|
||||
disables retries.
|
||||
backoff_base: Base for exponential backoff between retry attempts.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)``.
|
||||
|
||||
Raises:
|
||||
ValueError: If inputs are invalid, or any series is empty (after
|
||||
dropping the still-forming bar when ``start_pos`` is ``0``).
|
||||
"""
|
||||
_require_positive(count, "count")
|
||||
_require_non_negative(start_pos, "start_pos")
|
||||
fetch_count = count + 1 if start_pos == 0 else count
|
||||
loaded = collect_latest_rates_for_accounts_with_retries(
|
||||
accounts,
|
||||
timeframes,
|
||||
fetch_count,
|
||||
start_pos=start_pos,
|
||||
base_config=base_config,
|
||||
retry_count=retry_count,
|
||||
backoff_base=backoff_base,
|
||||
)
|
||||
result: dict[tuple[str, int], pd.DataFrame] = {}
|
||||
for key, df_rate in loaded.items():
|
||||
closed = drop_forming_rate_bar(df_rate) if start_pos == 0 else df_rate
|
||||
if closed.empty:
|
||||
symbol, timeframe = key
|
||||
msg = f"Rate data is empty for {symbol!r} at timeframe {timeframe}."
|
||||
raise ValueError(msg)
|
||||
result[key] = closed
|
||||
return result
|
||||
|
||||
|
||||
def collect_latest_closed_rates_by_granularity(
|
||||
accounts: Sequence[AccountSpec],
|
||||
granularities: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
start_pos: int = 0,
|
||||
base_config: Mt5Config | None = None,
|
||||
retry_count: int = 0,
|
||||
backoff_base: float = 2.0,
|
||||
) -> dict[tuple[str, str], pd.DataFrame]:
|
||||
"""Collect latest closed rate bars keyed by symbol and granularity name.
|
||||
|
||||
Thin wrapper around :func:`collect_latest_closed_rates_for_accounts` that
|
||||
rekeys the result by granularity name (for example ``M1``) instead of the
|
||||
integer timeframe.
|
||||
|
||||
Args:
|
||||
accounts: Account groups to read. Each must define at least one symbol.
|
||||
granularities: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of closed bars to return per symbol/timeframe.
|
||||
start_pos: Initial bar position offset passed to the underlying collector.
|
||||
base_config: Optional base configuration whose fields fill any value not
|
||||
set on an individual account.
|
||||
retry_count: Maximum number of retries after the first attempt. ``0``
|
||||
disables retries.
|
||||
backoff_base: Base for exponential backoff between retry attempts.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, granularity_name)``. Propagates
|
||||
``ValueError`` from :func:`collect_latest_closed_rates_for_accounts`.
|
||||
"""
|
||||
loaded = collect_latest_closed_rates_for_accounts(
|
||||
accounts,
|
||||
granularities,
|
||||
count,
|
||||
start_pos=start_pos,
|
||||
base_config=base_config,
|
||||
retry_count=retry_count,
|
||||
backoff_base=backoff_base,
|
||||
)
|
||||
return {
|
||||
(symbol, resolve_granularity_name(timeframe)): frame
|
||||
for (symbol, timeframe), frame in loaded.items()
|
||||
}
|
||||
|
||||
|
||||
def copy_rates_range(
|
||||
symbol: str,
|
||||
timeframe: int | str,
|
||||
@@ -1024,6 +1803,23 @@ def history_deals(
|
||||
)
|
||||
|
||||
|
||||
def recent_history_deals(
|
||||
hours: float,
|
||||
date_to: datetime | str | None = None,
|
||||
group: str | None = None,
|
||||
symbol: str | None = None,
|
||||
*,
|
||||
config: Mt5Config | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Return historical deals from a recent trailing window."""
|
||||
return _make_client(config=config).recent_history_deals(
|
||||
hours,
|
||||
date_to=date_to,
|
||||
group=group,
|
||||
symbol=symbol,
|
||||
)
|
||||
|
||||
|
||||
def version(*, config: Mt5Config | None = None) -> pd.DataFrame:
|
||||
"""Return MetaTrader5 version information."""
|
||||
return _make_client(config=config).version()
|
||||
@@ -1084,3 +1880,13 @@ def minimum_margins(
|
||||
See ``Mt5CliClient.minimum_margins`` for return details.
|
||||
"""
|
||||
return _make_client(config=config).minimum_margins(symbol)
|
||||
|
||||
|
||||
def mt5_summary(*, config: Mt5Config | None = None) -> dict[str, object]:
|
||||
"""Return a compact terminal/account status summary."""
|
||||
return _make_client(config=config).mt5_summary()
|
||||
|
||||
|
||||
def mt5_summary_as_df(*, config: Mt5Config | None = None) -> pd.DataFrame:
|
||||
"""Return an export-safe terminal/account status summary DataFrame."""
|
||||
return _make_client(config=config).mt5_summary_as_df()
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "mt5cli"
|
||||
version = "0.4.3"
|
||||
version = "0.6.0"
|
||||
description = "Command-line tool for MetaTrader 5"
|
||||
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
|
||||
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
|
||||
client.market_book_get_as_df.return_value = sample_df
|
||||
client.order_check_as_df.return_value = sample_df
|
||||
client.order_send_as_df.return_value = sample_df
|
||||
client.version.return_value = (5, 0, 1)
|
||||
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
|
||||
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
|
||||
client.symbols_total.return_value = 42
|
||||
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
|
||||
return client
|
||||
|
||||
@@ -223,6 +227,37 @@ class TestCommands:
|
||||
count=50,
|
||||
)
|
||||
|
||||
def test_latest_rates(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test latest-rates command."""
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"latest-rates",
|
||||
"--symbol",
|
||||
"GBPUSD",
|
||||
"--timeframe",
|
||||
"H1",
|
||||
"--count",
|
||||
"50",
|
||||
"--start-pos",
|
||||
"2",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
|
||||
symbol="GBPUSD",
|
||||
timeframe=16385,
|
||||
start_pos=2,
|
||||
count=50,
|
||||
)
|
||||
|
||||
def test_rates_range(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
@@ -451,6 +486,84 @@ class TestCommands:
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.history_deals_get_as_df.assert_called_once()
|
||||
|
||||
def test_recent_history_deals(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test recent-history-deals command."""
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"recent-history-deals",
|
||||
"--hours",
|
||||
"6",
|
||||
"--date-to",
|
||||
"2024-01-02",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.history_deals_get_as_df.assert_called_once_with(
|
||||
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
|
||||
date_to=datetime(2024, 1, 2, tzinfo=UTC),
|
||||
group=None,
|
||||
symbol="EURUSD",
|
||||
ticket=None,
|
||||
position=None,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("filename", "reader"),
|
||||
[
|
||||
("summary.csv", "csv"),
|
||||
("summary.json", "json"),
|
||||
("summary.db", "sqlite3"),
|
||||
("summary.parquet", "parquet"),
|
||||
],
|
||||
)
|
||||
def test_mt5_summary_export_formats(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
filename: str,
|
||||
reader: str,
|
||||
) -> None:
|
||||
"""Test mt5-summary writes export-safe files for supported formats."""
|
||||
output = tmp_path / filename
|
||||
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert output.exists()
|
||||
mock_client.version.assert_called_once()
|
||||
mock_client.terminal_info.assert_called_once()
|
||||
mock_client.account_info.assert_called_once()
|
||||
mock_client.symbols_total.assert_called_once()
|
||||
if reader == "csv":
|
||||
frame = pd.read_csv(output)
|
||||
elif reader == "json":
|
||||
with output.open() as f:
|
||||
records = json.load(f)
|
||||
frame = pd.DataFrame(records)
|
||||
elif reader == "sqlite3":
|
||||
with sqlite3.connect(output) as conn:
|
||||
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT * FROM data",
|
||||
conn,
|
||||
)
|
||||
else:
|
||||
frame = pd.read_parquet(output)
|
||||
assert len(frame) == 1
|
||||
assert frame.iloc[0].to_dict() == {
|
||||
"version": "[5,0,1]",
|
||||
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
|
||||
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
|
||||
"symbols_total": 42,
|
||||
}
|
||||
|
||||
def test_version(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
|
||||
+694
-3
@@ -10,14 +10,19 @@ from unittest.mock import MagicMock
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from pytest_mock import MockerFixture # noqa: TC002
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import history
|
||||
from mt5cli.history import (
|
||||
DEFAULT_HISTORY_TIMEFRAMES,
|
||||
DedupScope,
|
||||
RateTarget,
|
||||
append_dataframe,
|
||||
augment_written_columns_from_sqlite,
|
||||
build_rate_targets,
|
||||
build_rate_view_name,
|
||||
create_cash_events_view,
|
||||
create_history_indexes,
|
||||
@@ -25,12 +30,17 @@ from mt5cli.history import (
|
||||
create_rate_compatibility_views,
|
||||
deduplicate_history_tables,
|
||||
drop_duplicates_in_table,
|
||||
drop_forming_rate_bar,
|
||||
filter_incremental_history_deals_frame,
|
||||
filter_trade_history_frame,
|
||||
get_history_deals_account_event_start_datetime,
|
||||
get_incremental_start_datetime,
|
||||
get_table_columns,
|
||||
load_incremental_start_datetimes,
|
||||
load_rate_data,
|
||||
load_rate_data_from_connection,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
parse_sqlite_timestamp,
|
||||
quote_sqlite_identifier,
|
||||
record_written_columns,
|
||||
@@ -38,6 +48,7 @@ from mt5cli.history import (
|
||||
resolve_history_datasets,
|
||||
resolve_history_tick_flags,
|
||||
resolve_history_timeframes,
|
||||
resolve_rate_tables,
|
||||
resolve_rate_view_name,
|
||||
resolve_rate_view_names,
|
||||
write_collected_datasets,
|
||||
@@ -58,6 +69,21 @@ class TestResolveRateViewName:
|
||||
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
|
||||
assert not db_path.exists()
|
||||
|
||||
def test_none_path_returns_default_name(self) -> None:
|
||||
"""Test a None connection or path returns the deterministic default."""
|
||||
assert resolve_rate_view_name(None, "EURUSD", "M1") == "rate_EURUSD__1"
|
||||
assert resolve_rate_view_names(None, ["EURUSD"], ["M1", "H1"]) == [
|
||||
"rate_EURUSD__1",
|
||||
"rate_EURUSD__16385",
|
||||
]
|
||||
|
||||
def test_none_path_with_require_existing_raises(self) -> None:
|
||||
"""Test a None path under strict mode raises a clear error."""
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_view_name(None, "EURUSD", "M1", require_existing=True)
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_view_names(None, ["EURUSD"], ["M1"], require_existing=True)
|
||||
|
||||
def test_no_rates_table_falls_back_to_single_timeframe_name(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
@@ -338,6 +364,147 @@ class TestQuoteSqliteIdentifier:
|
||||
assert quoted.endswith('"')
|
||||
|
||||
|
||||
class TestLoadRateData:
|
||||
"""Tests for SQLite rate-like table and view loading."""
|
||||
|
||||
def test_loads_close_rates_from_path_with_count(self, tmp_path: Path) -> None:
|
||||
"""Test loading the latest close-based rates in ascending time order."""
|
||||
db_path = tmp_path / "rates.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
|
||||
conn.executemany(
|
||||
"INSERT INTO rates(time, close) VALUES (?, ?)",
|
||||
[
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
("2024-01-01T00:02:00+00:00", 1.2),
|
||||
("2024-01-01T00:01:00+00:00", 1.1),
|
||||
],
|
||||
)
|
||||
frame = load_rate_data(db_path, "rates", count=2)
|
||||
assert list(frame["close"]) == [1.1, 1.2]
|
||||
assert isinstance(frame.index, pd.DatetimeIndex)
|
||||
assert frame.index.name == "time"
|
||||
assert frame.index.is_monotonic_increasing
|
||||
|
||||
def test_loads_ask_bid_tick_like_rates_from_connection(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test loading tick-like tables with bid and ask columns."""
|
||||
db_path = tmp_path / "ticks.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE ticks(time TEXT, bid REAL, ask REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO ticks(time, bid, ask) VALUES (?, ?, ?)",
|
||||
("2024-01-01T00:00:00+00:00", 1.0, 1.1),
|
||||
)
|
||||
frame = load_rate_data_from_connection(conn, "ticks")
|
||||
path_frame = load_rate_data(conn, "ticks")
|
||||
assert frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
|
||||
assert path_frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
|
||||
|
||||
def test_loads_from_view(self, tmp_path: Path) -> None:
|
||||
"""Test loading from a SQLite view."""
|
||||
db_path = tmp_path / "view.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO rates(time, close) VALUES (?, ?)",
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
conn.execute("CREATE VIEW rate_view AS SELECT time, close FROM rates")
|
||||
frame = load_rate_data_from_connection(conn, "rate_view")
|
||||
assert list(frame["close"]) == [1.0]
|
||||
|
||||
def test_loads_quoted_identifier(self, tmp_path: Path) -> None:
|
||||
"""Test table names are quoted safely."""
|
||||
db_path = tmp_path / "quoted.db"
|
||||
table = 'rate "quoted"'
|
||||
quoted = quote_sqlite_identifier(table)
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(f"CREATE TABLE {quoted}(time TEXT, close REAL)")
|
||||
conn.execute(
|
||||
f"INSERT INTO {quoted}(time, close) VALUES (?, ?)", # noqa: S608
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
frame = load_rate_data_from_connection(conn, table)
|
||||
assert list(frame["close"]) == [1.0]
|
||||
|
||||
def test_rejects_missing_database_and_non_file(self, tmp_path: Path) -> None:
|
||||
"""Test path validation for SQLite database inputs."""
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
load_rate_data(tmp_path / "missing.db", "rates")
|
||||
with pytest.raises(ValueError, match="not a file"):
|
||||
load_rate_data(tmp_path, "rates")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("table", "count", "match"),
|
||||
[
|
||||
("", None, "must not be empty"),
|
||||
("rates", 0, "count must be positive"),
|
||||
("rates", -1, "count must be positive"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_inputs(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
table: str,
|
||||
count: int | None,
|
||||
match: str,
|
||||
) -> None:
|
||||
"""Test request validation."""
|
||||
db_path = tmp_path / "invalid-inputs.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
|
||||
with pytest.raises(ValueError, match=match):
|
||||
load_rate_data_from_connection(conn, table, count=count)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("ddl", "match"),
|
||||
[
|
||||
("CREATE TABLE rates(time TEXT, close REAL)", "contains no rows"),
|
||||
("CREATE TABLE rates(close REAL)", "time column"),
|
||||
("CREATE TABLE rates(time TEXT, open REAL)", "close, or both ask and bid"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_tables(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
ddl: str,
|
||||
match: str,
|
||||
) -> None:
|
||||
"""Test missing table, empty table, and invalid schemas."""
|
||||
db_path = tmp_path / "invalid-tables.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(ddl)
|
||||
with pytest.raises(ValueError, match=match):
|
||||
load_rate_data_from_connection(conn, "rates")
|
||||
with pytest.raises(ValueError, match="not found"):
|
||||
load_rate_data_from_connection(conn, "missing")
|
||||
|
||||
def test_rejects_invalid_timestamp(self, tmp_path: Path) -> None:
|
||||
"""Test unparsable timestamps fail clearly."""
|
||||
db_path = tmp_path / "invalid-time.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
|
||||
conn.execute("INSERT INTO rates(time, close) VALUES (?, ?)", ("bad", 1.0))
|
||||
with pytest.raises(ValueError, match="unparsable time"):
|
||||
load_rate_data_from_connection(conn, "rates")
|
||||
|
||||
def test_loads_numeric_mt5_epoch_seconds(self, tmp_path: Path) -> None:
|
||||
"""Test MT5-native integer timestamps are parsed as epoch seconds."""
|
||||
db_path = tmp_path / "epoch-rates.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE rates(time INTEGER, close REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO rates(time, close) VALUES (?, ?)",
|
||||
(1_704_067_200, 1.0),
|
||||
)
|
||||
frame = load_rate_data_from_connection(conn, "rates")
|
||||
assert frame.index[0] == pd.Timestamp("2024-01-01", tz="UTC")
|
||||
assert list(frame["close"]) == [1.0]
|
||||
|
||||
|
||||
class TestResolveHistorySettings:
|
||||
"""Tests for history dataset and timeframe resolution."""
|
||||
|
||||
@@ -368,6 +535,46 @@ class TestResolveHistorySettings:
|
||||
assert resolve_granularity_name(1) == "M1"
|
||||
|
||||
|
||||
class TestDropFormingRateBar:
|
||||
"""Tests for drop_forming_rate_bar."""
|
||||
|
||||
def test_drops_still_forming_last_bar(self) -> None:
|
||||
"""Test the still-forming last bar is removed."""
|
||||
df_rate = pd.DataFrame(
|
||||
{"time": [1, 2, 3], "close": [1.1, 1.2, 1.3]},
|
||||
index=pd.Index(["a", "b", "c"], name="idx"),
|
||||
)
|
||||
|
||||
result = drop_forming_rate_bar(df_rate)
|
||||
|
||||
pd.testing.assert_frame_equal(
|
||||
result,
|
||||
pd.DataFrame(
|
||||
{"time": [1, 2], "close": [1.1, 1.2]},
|
||||
index=pd.Index(["a", "b"], name="idx"),
|
||||
),
|
||||
)
|
||||
assert df_rate.shape == (3, 2)
|
||||
|
||||
def test_returns_empty_frame_when_input_empty(self) -> None:
|
||||
"""Test empty frames stay empty."""
|
||||
df_rate = pd.DataFrame(columns=["time", "close"])
|
||||
|
||||
result = drop_forming_rate_bar(df_rate)
|
||||
|
||||
assert result.empty
|
||||
assert list(result.columns) == ["time", "close"]
|
||||
|
||||
def test_returns_empty_frame_when_only_forming_bar_present(self) -> None:
|
||||
"""Test a single-bar frame becomes empty after dropping the forming bar."""
|
||||
df_rate = pd.DataFrame({"time": [1], "close": [1.1]})
|
||||
|
||||
result = drop_forming_rate_bar(df_rate)
|
||||
|
||||
assert result.empty
|
||||
assert list(result.columns) == ["time", "close"]
|
||||
|
||||
|
||||
class TestParseSqliteTimestamp:
|
||||
"""Tests for parse_sqlite_timestamp."""
|
||||
|
||||
@@ -454,7 +661,7 @@ class TestIncrementalStart:
|
||||
) -> None:
|
||||
"""Test rates tables without timeframe fail fast during incremental resume."""
|
||||
fallback = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "legacy-rates.db") as conn:
|
||||
with sqlite3.connect(tmp_path / "rates-without-timeframe.db") as conn:
|
||||
conn.execute("CREATE TABLE rates(symbol TEXT, time TEXT, open REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, time, open) VALUES (?, ?, ?)",
|
||||
@@ -723,9 +930,10 @@ class TestDeduplication:
|
||||
{Dataset.rates},
|
||||
{
|
||||
Dataset.rates: [
|
||||
(
|
||||
DedupScope(
|
||||
"symbol = ? AND timeframe = ? AND time >= ?",
|
||||
("EURUSD", 1, boundary),
|
||||
frozenset({"symbol", "timeframe", "time"}),
|
||||
),
|
||||
],
|
||||
},
|
||||
@@ -738,6 +946,89 @@ class TestDeduplication:
|
||||
("2024-01-02T00:00:00+00:00", 9.9),
|
||||
]
|
||||
|
||||
def test_unusable_scope_falls_back_to_table_dedup(self, tmp_path: Path) -> None:
|
||||
"""Test scopes with missing columns do not break stable-key dedup."""
|
||||
boundary = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "orders-without-time.db") as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE history_orders("
|
||||
" ticket INTEGER, symbol TEXT, time_setup TEXT, type INTEGER)",
|
||||
)
|
||||
conn.executemany(
|
||||
"INSERT INTO history_orders(ticket, symbol, time_setup, type)"
|
||||
" VALUES (?, ?, ?, ?)",
|
||||
[
|
||||
(1, "EURUSD", "2024-01-01T00:00:00+00:00", 0),
|
||||
(1, "EURUSD", "2024-01-01T00:00:01+00:00", 1),
|
||||
],
|
||||
)
|
||||
deduplicate_history_tables(
|
||||
conn,
|
||||
{Dataset.history_orders: {"ticket", "symbol", "time_setup", "type"}},
|
||||
{Dataset.history_orders},
|
||||
{
|
||||
Dataset.history_orders: [
|
||||
DedupScope(
|
||||
"symbol = ? AND time >= ?",
|
||||
("EURUSD", boundary),
|
||||
frozenset({"symbol", "time"}),
|
||||
),
|
||||
],
|
||||
},
|
||||
)
|
||||
rows = conn.execute(
|
||||
"SELECT ticket, time_setup, type FROM history_orders",
|
||||
).fetchall()
|
||||
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
|
||||
|
||||
def test_partially_unusable_scopes_only_run_usable_scopes(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test mixed scope filtering skips only scopes with missing columns."""
|
||||
boundary = datetime(2024, 1, 2, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "partial-scope-filter.db") as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, open REAL)",
|
||||
)
|
||||
conn.executemany(
|
||||
"INSERT INTO rates(symbol, timeframe, time, open) VALUES (?, ?, ?, ?)",
|
||||
[
|
||||
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 2.0),
|
||||
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 9.9),
|
||||
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 100.0),
|
||||
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 101.0),
|
||||
],
|
||||
)
|
||||
deduplicate_history_tables(
|
||||
conn,
|
||||
{Dataset.rates: {"symbol", "timeframe", "time", "open"}},
|
||||
{Dataset.rates},
|
||||
{
|
||||
Dataset.rates: [
|
||||
DedupScope(
|
||||
"symbol = ? AND timeframe = ? AND time >= ?",
|
||||
("EURUSD", 1, boundary),
|
||||
frozenset({"symbol", "timeframe", "time"}),
|
||||
),
|
||||
DedupScope(
|
||||
"symbol = ? AND timeframe = ? AND broker = ?",
|
||||
("USDJPY", 1, "demo"),
|
||||
frozenset({"symbol", "timeframe", "broker"}),
|
||||
),
|
||||
],
|
||||
},
|
||||
)
|
||||
rows = conn.execute(
|
||||
"SELECT symbol, open FROM rates ORDER BY symbol, open",
|
||||
).fetchall()
|
||||
assert rows == [
|
||||
("EURUSD", 9.9),
|
||||
("USDJPY", 100.0),
|
||||
("USDJPY", 101.0),
|
||||
]
|
||||
|
||||
|
||||
class TestRateCompatibilityViews:
|
||||
"""Tests for rate compatibility view creation."""
|
||||
@@ -1198,6 +1489,54 @@ class TestIncrementalIntegration:
|
||||
"rate_EURUSD_M1__1",
|
||||
}
|
||||
|
||||
def test_incremental_orders_without_time_deduplicate_by_ticket(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Test incremental history_orders without time deduplicate safely."""
|
||||
|
||||
def history_orders_get_as_df(**kwargs: object) -> pd.DataFrame:
|
||||
if kwargs["symbol"] == "GBPUSD":
|
||||
return pd.DataFrame()
|
||||
return pd.DataFrame({
|
||||
"ticket": [1, 1],
|
||||
"symbol": ["EURUSD", "EURUSD"],
|
||||
"time_setup": [
|
||||
"2024-01-01T00:00:00+00:00",
|
||||
"2024-01-01T00:00:01+00:00",
|
||||
],
|
||||
"type": [0, 1],
|
||||
})
|
||||
|
||||
client = MagicMock()
|
||||
client.history_orders_get_as_df.side_effect = history_orders_get_as_df
|
||||
start = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
end = datetime(2024, 1, 2, tzinfo=UTC)
|
||||
with (
|
||||
sqlite3.connect(tmp_path / "incremental-orders-without-time.db") as conn,
|
||||
caplog.at_level(logging.WARNING, logger="mt5cli.history"),
|
||||
):
|
||||
write_incremental_datasets(
|
||||
conn,
|
||||
client,
|
||||
["EURUSD", "GBPUSD"],
|
||||
{Dataset.history_orders},
|
||||
[],
|
||||
0,
|
||||
start,
|
||||
end,
|
||||
deduplicate=True,
|
||||
create_rate_views=False,
|
||||
with_views=False,
|
||||
include_account_events=False,
|
||||
)
|
||||
rows = conn.execute(
|
||||
"SELECT ticket, time_setup, type FROM history_orders",
|
||||
).fetchall()
|
||||
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
|
||||
assert "Skipping history_orders: dataset returned no columns" in caplog.text
|
||||
|
||||
def test_write_collected_datasets_and_edge_branches(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
@@ -1573,7 +1912,7 @@ class TestIncrementalHistoryDeals:
|
||||
})
|
||||
start = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
end = datetime(2024, 1, 3, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "legacy-deals.db") as conn:
|
||||
with sqlite3.connect(tmp_path / "deals-without-type.db") as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE history_deals( ticket INTEGER, symbol TEXT, time TEXT)",
|
||||
)
|
||||
@@ -1772,3 +2111,355 @@ class TestWriteHelpers:
|
||||
)
|
||||
assert get_table_columns(conn, "rates") == {"time", "open"}
|
||||
create_history_indexes(conn, written_columns)
|
||||
|
||||
|
||||
class TestRateSourceHelpers:
|
||||
"""Tests for generic rate-source SDK helpers."""
|
||||
|
||||
def test_rate_target_timeframe_int(self) -> None:
|
||||
"""Test RateTarget resolves named and integer timeframes."""
|
||||
target = RateTarget(symbol="EURUSD", timeframe="M1")
|
||||
assert target.timeframe == 1
|
||||
assert target.timeframe_int == 1
|
||||
assert RateTarget(symbol="EURUSD", timeframe=16385).timeframe_int == 16385
|
||||
|
||||
def test_build_rate_targets_row_major(self) -> None:
|
||||
"""Test targets are built in row-major symbol/timeframe order."""
|
||||
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
|
||||
assert [(t.symbol, t.timeframe) for t in targets] == [
|
||||
("EURUSD", 1),
|
||||
("EURUSD", 16385),
|
||||
("GBPUSD", 1),
|
||||
("GBPUSD", 16385),
|
||||
]
|
||||
|
||||
def test_build_rate_targets_allows_missing_symbol(self) -> None:
|
||||
"""Test missing symbols produce None-symbol targets when allowed."""
|
||||
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
|
||||
assert [(t.symbol, t.timeframe) for t in targets] == [
|
||||
(None, 1),
|
||||
(None, 16385),
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("symbols", "timeframes", "match"),
|
||||
[
|
||||
(["EURUSD"], [], "At least one timeframe"),
|
||||
([], ["M1"], "At least one symbol"),
|
||||
],
|
||||
)
|
||||
def test_build_rate_targets_rejects_empty(
|
||||
self,
|
||||
symbols: list[str],
|
||||
timeframes: list[str],
|
||||
match: str,
|
||||
) -> None:
|
||||
"""Test target building input validation."""
|
||||
with pytest.raises(ValueError, match=match):
|
||||
build_rate_targets(symbols, timeframes)
|
||||
|
||||
def test_resolve_rate_tables_uses_explicit_tables(self) -> None:
|
||||
"""Test explicit tables bypass view resolution when counts match."""
|
||||
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
|
||||
assert resolve_rate_tables(None, targets, ["t1", "t2"]) == ["t1", "t2"]
|
||||
|
||||
def test_resolve_rate_tables_rejects_mismatched_explicit_count(self) -> None:
|
||||
"""Test explicit table count must match the number of targets."""
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="Expected 1 explicit table"):
|
||||
resolve_rate_tables(None, targets, ["t1", "t2"])
|
||||
|
||||
def test_resolve_rate_tables_rejects_empty_targets(self) -> None:
|
||||
"""Test resolving requires at least one target."""
|
||||
with pytest.raises(ValueError, match="At least one rate target"):
|
||||
resolve_rate_tables(None, [])
|
||||
|
||||
def test_resolve_rate_tables_requires_symbol_without_explicit(self) -> None:
|
||||
"""Test None-symbol targets require explicit tables."""
|
||||
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
|
||||
with pytest.raises(ValueError, match="without a symbol"):
|
||||
resolve_rate_tables(None, targets)
|
||||
|
||||
def test_resolve_rate_tables_resolves_view_names(self) -> None:
|
||||
"""Test symbol targets resolve to default view names without a database."""
|
||||
targets = build_rate_targets(["EURUSD"], ["M1", "H1"])
|
||||
assert resolve_rate_tables(None, targets) == [
|
||||
"rate_EURUSD__1",
|
||||
"rate_EURUSD__16385",
|
||||
]
|
||||
|
||||
def test_resolve_rate_tables_none_path_with_require_existing_raises(self) -> None:
|
||||
"""Test strict mode rejects a missing database path."""
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_tables(None, targets, require_existing=True)
|
||||
|
||||
def test_resolve_rate_tables_missing_db_with_require_existing_raises(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test strict mode rejects a non-existing database path."""
|
||||
db_path = tmp_path / "missing.db"
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_tables(db_path, targets, require_existing=True)
|
||||
|
||||
def test_resolve_rate_tables_missing_view_with_require_existing_raises(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test strict mode rejects databases without managed rate views."""
|
||||
db_path = tmp_path / "no-views.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="No rate compatibility view exists"):
|
||||
resolve_rate_tables(db_path, targets, require_existing=True)
|
||||
|
||||
def test_resolve_rate_tables_with_require_existing_resolves_views(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test strict mode resolves existing managed rate views."""
|
||||
db_path = tmp_path / "strict-views.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
assert resolve_rate_tables(db_path, targets, require_existing=True) == [
|
||||
"rate_EURUSD__1",
|
||||
]
|
||||
|
||||
def test_resolve_rate_tables_batches_sqlite_metadata(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test resolving multiple targets loads SQLite metadata once."""
|
||||
db_path = tmp_path / "batch-rate-tables.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.executemany(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
[
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
("EURUSD", 16385, "2024-01-01T01:00:00+00:00", 1.1),
|
||||
("GBPUSD", 1, "2024-01-01T00:00:00+00:00", 1.2),
|
||||
],
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
counts_spy = mocker.spy(history, "_load_rates_timeframe_counts")
|
||||
views_spy = mocker.spy(history, "_load_existing_rate_views")
|
||||
|
||||
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
|
||||
assert resolve_rate_tables(db_path, targets) == [
|
||||
"rate_EURUSD__M1_1",
|
||||
"rate_EURUSD__H1_16385",
|
||||
"rate_GBPUSD__1",
|
||||
"rate_GBPUSD__16385",
|
||||
]
|
||||
assert counts_spy.call_count == 1
|
||||
assert views_spy.call_count == 1
|
||||
|
||||
def test_load_rate_series_from_sqlite(self, tmp_path: Path) -> None:
|
||||
"""Test loading multiple rate series keyed by symbol and timeframe."""
|
||||
db_path = tmp_path / "series.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.executemany(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
[
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
("EURUSD", 1, "2024-01-01T00:01:00+00:00", 1.1),
|
||||
],
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
result = load_rate_series_from_sqlite(db_path, targets, count=2)
|
||||
assert set(result) == {("EURUSD", 1)}
|
||||
assert len(result["EURUSD", 1]) == 2
|
||||
|
||||
def test_load_rate_series_by_granularity(self, tmp_path: Path) -> None:
|
||||
"""Test loading rate series keyed by symbol and granularity name."""
|
||||
db_path = tmp_path / "granularity.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.executemany(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
[
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
("EURUSD", 16385, "2024-01-01T00:00:00+00:00", 1.1),
|
||||
],
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
|
||||
result = load_rate_series_by_granularity(
|
||||
db_path,
|
||||
["EURUSD"],
|
||||
["M1", "H1"],
|
||||
count=1,
|
||||
)
|
||||
|
||||
assert set(result) == {("EURUSD", "M1"), ("EURUSD", "H1")}
|
||||
|
||||
def test_load_rate_series_by_granularity_explicit_tables(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test explicit tables with None-symbol targets key by granularity."""
|
||||
db_path = tmp_path / "granularity-explicit.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
|
||||
result = load_rate_series_by_granularity(
|
||||
db_path,
|
||||
[],
|
||||
["M1"],
|
||||
count=1,
|
||||
explicit_tables=["custom_view"],
|
||||
allow_missing_symbol=True,
|
||||
)
|
||||
|
||||
assert set(result) == {(None, "M1")}
|
||||
|
||||
def test_load_rate_series_reuses_path_connection(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test loading from a path opens SQLite once for resolve and reads."""
|
||||
db_path = tmp_path / "single-open-series.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
connect_spy = mocker.spy(history.sqlite3, "connect")
|
||||
|
||||
result = load_rate_series_from_sqlite(
|
||||
db_path,
|
||||
build_rate_targets(["EURUSD"], ["M1"]),
|
||||
count=1,
|
||||
)
|
||||
|
||||
assert set(result) == {("EURUSD", 1)}
|
||||
assert connect_spy.call_count == 1
|
||||
|
||||
def test_load_rate_series_with_explicit_tables(self, tmp_path: Path) -> None:
|
||||
"""Test explicit tables and None-symbol targets load series."""
|
||||
db_path = tmp_path / "explicit.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
|
||||
result = load_rate_series_from_sqlite(
|
||||
db_path,
|
||||
targets,
|
||||
count=1,
|
||||
explicit_tables=["custom_view"],
|
||||
)
|
||||
assert set(result) == {(None, 1)}
|
||||
|
||||
def test_load_rate_series_rejects_non_positive_count(self) -> None:
|
||||
"""Test loading requires a positive count."""
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="count must be positive"):
|
||||
load_rate_series_from_sqlite("unused.db", targets, count=0)
|
||||
|
||||
def test_load_rate_series_rejects_empty_targets(self) -> None:
|
||||
"""Test loading requires at least one target before opening SQLite."""
|
||||
with pytest.raises(ValueError, match="At least one rate target"):
|
||||
load_rate_series_from_sqlite("unused.db", [], count=1)
|
||||
|
||||
def test_load_rate_series_requires_symbol_without_explicit_tables(self) -> None:
|
||||
"""Test None-symbol targets require explicit tables before opening SQLite."""
|
||||
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
|
||||
with pytest.raises(ValueError, match="without a symbol"):
|
||||
load_rate_series_from_sqlite("unused.db", targets, count=1)
|
||||
|
||||
def test_load_rate_series_requires_existing_managed_views(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test loading without explicit tables requires managed rate views."""
|
||||
db_path = tmp_path / "no-managed-views.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
targets = build_rate_targets(["EURUSD"], ["M1"])
|
||||
with pytest.raises(ValueError, match="No rate compatibility view exists"):
|
||||
load_rate_series_from_sqlite(db_path, targets, count=1)
|
||||
|
||||
def test_load_rate_series_rejects_duplicate_targets(self) -> None:
|
||||
"""Test duplicate (symbol, timeframe) targets are rejected."""
|
||||
targets = [
|
||||
RateTarget("EURUSD", 1),
|
||||
RateTarget("EURUSD", "M1"),
|
||||
]
|
||||
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
|
||||
load_rate_series_from_sqlite("unused.db", targets, count=1)
|
||||
|
||||
def test_load_rate_series_rejects_duplicate_targets_with_explicit_tables(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test duplicate targets are rejected even with explicit tables."""
|
||||
db_path = tmp_path / "duplicate-explicit.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
|
||||
("2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
targets = [
|
||||
RateTarget("EURUSD", 1),
|
||||
RateTarget("EURUSD", 1),
|
||||
]
|
||||
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
|
||||
load_rate_series_from_sqlite(
|
||||
db_path,
|
||||
targets,
|
||||
count=1,
|
||||
explicit_tables=["custom_view", "custom_view"],
|
||||
)
|
||||
|
||||
+954
-2
File diff suppressed because it is too large
Load Diff
@@ -487,7 +487,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "mt5cli"
|
||||
version = "0.4.3"
|
||||
version = "0.6.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
@@ -836,11 +836,11 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pygments"
|
||||
version = "2.19.2"
|
||||
version = "2.20.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user