Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d654b82f9d | |||
| b5e82e71c7 | |||
| 18df96872b | |||
| 5b1d54bfe9 | |||
| ad9e513253 | |||
| 334f01b647 | |||
| 1b69e8f08e | |||
| 9957b0a1de | |||
| b2bb2ad0a0 |
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
|
||||
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
|
||||
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
|
||||
- **Connection management**: Optional credentials, server, and timeout configuration
|
||||
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
|
||||
| Command | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a recent trailing window |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a recent trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
|
||||
@@ -127,6 +133,31 @@ update_history_with_config(
|
||||
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
|
||||
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
|
||||
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
|
||||
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
|
||||
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
|
||||
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
|
||||
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
|
||||
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
|
||||
|
||||
rates = collect_latest_closed_rates_by_granularity(
|
||||
[AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
)
|
||||
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
|
||||
```
|
||||
|
||||
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
|
||||
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`, `sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability errors for history API methods without advancing the throttle (other `AttributeError` / `TypeError` values always propagate).
|
||||
- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable `pdmt5.Mt5TradingClient` that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. The read-only `mt5_session()` / `Mt5CliClient` SDK is unchanged.
|
||||
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
|
||||
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
|
||||
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
|
||||
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -134,6 +165,63 @@ update_history_with_config(
|
||||
- Windows OS (MetaTrader 5 requirement)
|
||||
- MetaTrader 5 platform installed
|
||||
|
||||
### Migration note for mteor
|
||||
|
||||
Replace local MT5 lifecycle and trading helper code with mt5cli imports:
|
||||
|
||||
```python
|
||||
# Before (local mteor helpers)
|
||||
# with local_mt5_trading_session(config) as client:
|
||||
# side = local_detect_position_side(client, symbol)
|
||||
# sizing = local_calculate_margin_and_volume(client, symbol, unit_ratio, preserved_ratio)
|
||||
# limits = local_determine_order_limits(client, symbol, side, sl_ratio, tp_ratio)
|
||||
|
||||
# After (mt5cli shared layer)
|
||||
from pdmt5 import Mt5Config
|
||||
from mt5cli import (
|
||||
calculate_margin_and_volume,
|
||||
detect_position_side,
|
||||
determine_order_limits,
|
||||
mt5_trading_session,
|
||||
)
|
||||
|
||||
with mt5_trading_session(
|
||||
Mt5Config(path=terminal_path, login=login), retry_count=2
|
||||
) as client:
|
||||
side = detect_position_side(client, symbol)
|
||||
sizing = calculate_margin_and_volume(
|
||||
client, symbol, unit_margin_ratio=0.5, preserved_margin_ratio=0.2
|
||||
)
|
||||
if side is not None:
|
||||
limits = determine_order_limits(
|
||||
client,
|
||||
symbol,
|
||||
side,
|
||||
stop_loss_limit_ratio=0.01,
|
||||
take_profit_limit_ratio=0.02,
|
||||
)
|
||||
```
|
||||
|
||||
Throttled history updates use a separate read-only session:
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
|
||||
from mt5cli import ThrottledHistoryUpdater
|
||||
|
||||
updater = ThrottledHistoryUpdater(
|
||||
output="history.db", interval_seconds=60, suppress_errors=True
|
||||
)
|
||||
client = Mt5DataClient(config=Mt5Config(login=login))
|
||||
client.initialize_and_login_mt5()
|
||||
try:
|
||||
updater.update(client, ["EURUSD"])
|
||||
finally:
|
||||
client.shutdown()
|
||||
```
|
||||
|
||||
Read-only collectors can keep using `mt5_session()` and `Mt5CliClient` without changes.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
|
||||
@@ -129,3 +129,101 @@ when required columns are missing.
|
||||
The `update_history` SDK path uses the same base tables and optional
|
||||
`cash_events` / `positions_reconstructed` views. It additionally maintains
|
||||
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
|
||||
|
||||
### Rate view resolution
|
||||
|
||||
Downstream tools can resolve mt5cli-managed compatibility view names from an
|
||||
existing SQLite history database without creating files or guessing naming
|
||||
schemes:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
|
||||
|
||||
# Single symbol and granularity
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
|
||||
|
||||
# Batch resolution in row-major order
|
||||
views = resolve_rate_view_names(
|
||||
Path("history.db"),
|
||||
["EURUSD", "GBPUSD"],
|
||||
["M1", "H1"],
|
||||
)
|
||||
```
|
||||
|
||||
Resolution rules:
|
||||
|
||||
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
|
||||
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
|
||||
are stored for the same symbol.
|
||||
- When multiple naming candidates apply, prefers an existing managed
|
||||
`rate_*__*` view from the candidate list.
|
||||
- Falls back to single-timeframe naming when the database path is missing or
|
||||
`rates` metadata is unavailable.
|
||||
- Pass `require_existing=True` to raise `ValueError` instead of returning a
|
||||
best-guess name when the database or view is missing.
|
||||
- Accepts either a SQLite path or an open `sqlite3.Connection`.
|
||||
|
||||
### Rate data loading
|
||||
|
||||
Use `load_rate_data()` to load a table or view from a SQLite path, or
|
||||
`load_rate_data_from_connection()` when you already have a connection:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import load_rate_data
|
||||
from mt5cli.history import resolve_rate_view_name
|
||||
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
|
||||
rates = load_rate_data(Path("history.db"), view, count=1000)
|
||||
```
|
||||
|
||||
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
|
||||
validates that `time` exists, parses timestamps with pandas, and returns a
|
||||
DataFrame indexed by ascending `DatetimeIndex` named `time`.
|
||||
|
||||
### Multi-series rate loading
|
||||
|
||||
For loading many rate series at once, build neutral `RateTarget` pairs and load
|
||||
them from SQLite in one call. View names are resolved via the same
|
||||
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
|
||||
|
||||
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
|
||||
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
|
||||
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
|
||||
```
|
||||
|
||||
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
|
||||
row-major order, normalizing timeframe names such as `"M1"` to their integer
|
||||
values; set `allow_missing_symbol=True` to address series solely by
|
||||
`explicit_tables` (targets carry `symbol=None`).
|
||||
- `resolve_rate_tables()` maps targets to table or view names and validates that
|
||||
any `explicit_tables` count matches the target count. Pass
|
||||
`require_existing=True` to raise `ValueError` instead of returning a
|
||||
best-guess name when the database or managed view is missing. When
|
||||
`explicit_tables` is provided, names are returned as-is and
|
||||
`require_existing` is ignored.
|
||||
- `load_rate_series_from_sqlite()` returns a mapping keyed by
|
||||
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
|
||||
requires existing managed `rate_*` compatibility views and raises
|
||||
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
|
||||
are rejected.
|
||||
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
|
||||
loads the series, and rekeys the result by granularity name to avoid
|
||||
converting integer timeframes downstream:
|
||||
|
||||
```python
|
||||
from mt5cli import load_rate_series_by_granularity
|
||||
|
||||
series = load_rate_series_by_granularity(
|
||||
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
|
||||
)
|
||||
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
|
||||
```
|
||||
|
||||
+27
-2
@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
|
||||
|
||||
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
|
||||
|
||||
### [Trading](trading.md)
|
||||
|
||||
Trading-capable session management and operational helpers built on `pdmt5.Mt5TradingClient`. Complements the read-only SDK without changing existing `Mt5CliClient` behavior.
|
||||
|
||||
### [History Collection (SQLite)](history.md)
|
||||
|
||||
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
|
||||
@@ -28,8 +32,9 @@ The package follows a simple architecture built on top of pdmt5:
|
||||
|
||||
1. **CLI Layer** (`cli.py`): Typer application with subcommands that delegate to the SDK and export results.
|
||||
2. **SDK Layer** (`sdk.py`): Read-only data access functions, `Mt5CliClient`, and `collect_history` orchestration.
|
||||
3. **Utils Layer** (`utils.py`): Constants, enums, custom Click parameter types, parsing helpers, and format detection/export utilities.
|
||||
4. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient` and `Mt5Config` from the pdmt5 package for all MetaTrader 5 data access.
|
||||
3. **Trading Layer** (`trading.py`): Trading-capable sessions and operational helpers on `Mt5TradingClient`.
|
||||
4. **Utils Layer** (`utils.py`): Constants, enums, custom Click parameter types, parsing helpers, and format detection/export utilities.
|
||||
5. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient`, `Mt5TradingClient`, and `Mt5Config` from the pdmt5 package for MetaTrader 5 access.
|
||||
|
||||
## Usage Guidelines
|
||||
|
||||
@@ -65,12 +70,18 @@ from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import (
|
||||
Dataset,
|
||||
IfExists,
|
||||
Mt5CliClient,
|
||||
collect_history,
|
||||
copy_rates_range,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
minimum_margins,
|
||||
recent_ticks,
|
||||
)
|
||||
from mt5cli.history import resolve_rate_view_name
|
||||
|
||||
# Fetch rates programmatically
|
||||
rates = copy_rates_range(
|
||||
@@ -86,6 +97,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
|
||||
# Export a DataFrame
|
||||
export_dataframe(rates, Path("output.csv"), "csv")
|
||||
|
||||
# Append to SQLite with deduplication
|
||||
export_dataframe_to_sqlite(
|
||||
rates,
|
||||
Path("history.db"),
|
||||
"rates",
|
||||
if_exists=IfExists.APPEND,
|
||||
deduplicate_on=("symbol", "timeframe", "time"),
|
||||
)
|
||||
|
||||
# Resolve rate compatibility views and fetch recent ticks
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
|
||||
ticks = recent_ticks("EURUSD", seconds=300)
|
||||
margins = minimum_margins("EURUSD")
|
||||
|
||||
# Collect history into SQLite
|
||||
collect_history(
|
||||
Path("history.db"),
|
||||
|
||||
+111
@@ -1,3 +1,114 @@
|
||||
# SDK Module
|
||||
|
||||
::: mt5cli.sdk
|
||||
|
||||
## Resilient multi-account orchestration
|
||||
|
||||
The SDK ships strategy-agnostic helpers for building long-running collectors on
|
||||
top of the read-only client. None of them depend on a particular trading
|
||||
application.
|
||||
|
||||
### Retrying transient rate collection
|
||||
|
||||
`collect_latest_rates_for_accounts_with_retries()` wraps
|
||||
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
|
||||
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
|
||||
failure is re-raised once `retry_count` is exhausted.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
|
||||
|
||||
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
|
||||
rates = collect_latest_rates_for_accounts_with_retries(
|
||||
accounts,
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
|
||||
)
|
||||
```
|
||||
|
||||
### Latest closed rate bars
|
||||
|
||||
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
|
||||
row. `collect_latest_closed_rates_for_accounts()` fetches `count + 1` bars,
|
||||
drops that row with `drop_forming_rate_bar()`, and validates each series is
|
||||
non-empty. Use `collect_latest_closed_rates_by_granularity()` when callers
|
||||
prefer keys such as `("EURUSD", "M1")` instead of integer timeframes.
|
||||
|
||||
```python
|
||||
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
|
||||
|
||||
rates = collect_latest_closed_rates_by_granularity(
|
||||
[AccountSpec(symbols=["EURUSD"], login=12345)],
|
||||
["M1", "H1"],
|
||||
count=500,
|
||||
retry_count=3,
|
||||
)
|
||||
closed_m1 = rates["EURUSD", "M1"]
|
||||
```
|
||||
|
||||
### Resolving credentials and `${ENV_VAR}` placeholders
|
||||
|
||||
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
|
||||
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
|
||||
secrets out of plan/config files. A missing environment variable raises
|
||||
`ValueError`.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mt5cli import AccountSpec, resolve_account_specs
|
||||
|
||||
os.environ["MT5_LOGIN"] = "12345"
|
||||
os.environ["MT5_PASSWORD"] = "secret"
|
||||
accounts = [
|
||||
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
|
||||
]
|
||||
|
||||
resolved = resolve_account_specs(accounts, server="Broker-Demo")
|
||||
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
|
||||
```
|
||||
|
||||
### Throttled incremental history updates
|
||||
|
||||
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
|
||||
between successful runs (using a monotonic clock), so an application loop can
|
||||
call it every iteration without over-fetching.
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
|
||||
from mt5cli import Dataset, ThrottledHistoryUpdater
|
||||
|
||||
updater = ThrottledHistoryUpdater(
|
||||
output="history.db",
|
||||
datasets={Dataset.rates},
|
||||
timeframes=["M1"],
|
||||
interval_seconds=60, # <= 0 updates on every call
|
||||
)
|
||||
|
||||
client = Mt5DataClient(config=Mt5Config(login=12345))
|
||||
client.initialize_and_login_mt5()
|
||||
try:
|
||||
while True:
|
||||
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
|
||||
# ... do other work; break when shutting down ...
|
||||
finally:
|
||||
client.shutdown()
|
||||
```
|
||||
|
||||
By default recoverable errors (`Mt5TradingError`, `Mt5RuntimeError`,
|
||||
`sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability
|
||||
`AttributeError` / `TypeError` for history API methods) propagate so the caller
|
||||
controls logging; pass `suppress_errors=True` to swallow them and return
|
||||
`False` without advancing the throttle. Other `AttributeError` / `TypeError`
|
||||
values always propagate. Input validation (`_resolve_update_history_request`)
|
||||
runs before any MT5 or SQLite calls, but when `suppress_errors=True` the
|
||||
resulting `ValueError` is suppressed along with other recoverable errors.
|
||||
|
||||
## Trading-capable sessions
|
||||
|
||||
For order placement and trading calculations, use the dedicated
|
||||
[Trading module](trading.md). The read-only `Mt5CliClient` and `mt5_session()`
|
||||
helpers in this module are unchanged.
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Trading Module
|
||||
|
||||
::: mt5cli.trading
|
||||
|
||||
## Trading-capable MT5 sessions
|
||||
|
||||
`mt5_trading_session()` complements the read-only `mt5_session()` helper in
|
||||
`sdk.py`. It yields a connected `pdmt5.Mt5TradingClient`, uses
|
||||
`Mt5Config.path` to launch the terminal when configured, and always calls
|
||||
`shutdown()` on exit.
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5Config
|
||||
|
||||
from mt5cli import mt5_trading_session
|
||||
|
||||
with mt5_trading_session(
|
||||
Mt5Config(path=r"C:\Program Files\MetaTrader 5\terminal64.exe", login=12345),
|
||||
retry_count=2,
|
||||
) as client:
|
||||
positions = client.positions_get_as_df(symbol="EURUSD")
|
||||
```
|
||||
|
||||
The read-only `Mt5CliClient` / `mt5_session()` API is unchanged.
|
||||
|
||||
## Operational trading helpers
|
||||
|
||||
These helpers are strategy-agnostic and do not depend on signal detection,
|
||||
betting logic, or scheduling code in downstream applications.
|
||||
|
||||
```python
|
||||
from mt5cli import (
|
||||
calculate_margin_and_volume,
|
||||
detect_position_side,
|
||||
determine_order_limits,
|
||||
)
|
||||
|
||||
side = detect_position_side(client, "EURUSD")
|
||||
sizing = calculate_margin_and_volume(
|
||||
client,
|
||||
"EURUSD",
|
||||
unit_margin_ratio=0.5,
|
||||
preserved_margin_ratio=0.2,
|
||||
)
|
||||
limits = determine_order_limits(
|
||||
client,
|
||||
"EURUSD",
|
||||
side="long",
|
||||
stop_loss_limit_ratio=0.01,
|
||||
take_profit_limit_ratio=0.02,
|
||||
)
|
||||
```
|
||||
|
||||
Protective ratios must satisfy `0 <= ratio < 1`; `0` omits that level.
|
||||
`calculate_margin_and_volume()` clamps negative `margin_free` to `0.0`
|
||||
before sizing.
|
||||
|
||||
## Migration from mteor-local helpers
|
||||
|
||||
| mteor-local concern | mt5cli replacement |
|
||||
| -------------------------------------------------------- | ----------------------------------------------- |
|
||||
| Manual terminal spawn/kill around trading code | `mt5_trading_session()` |
|
||||
| Local position-side detection | `detect_position_side()` |
|
||||
| Local margin/volume sizing | `calculate_margin_and_volume()` |
|
||||
| Local SL/TP price derivation | `determine_order_limits()` |
|
||||
| Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)` |
|
||||
|
||||
Keep read-only data collection on `mt5_session()` / `Mt5CliClient`; use
|
||||
`mt5_trading_session()` only where order placement or trading calculations are
|
||||
required.
|
||||
+43
-14
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
|
||||
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
|
||||
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
|
||||
- **Connection management**: Optional credentials, server, and timeout configuration
|
||||
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -22,13 +23,23 @@ pip install mt5cli
|
||||
|
||||
## Programmatic usage / SDK usage
|
||||
|
||||
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results.
|
||||
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
|
||||
|
||||
```python
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe
|
||||
from mt5cli import (
|
||||
Mt5CliClient,
|
||||
collect_history,
|
||||
copy_rates_range,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
load_rate_data,
|
||||
minimum_margins,
|
||||
recent_ticks,
|
||||
)
|
||||
from mt5cli.history import resolve_rate_view_name
|
||||
|
||||
# One-off fetch with module-level helpers
|
||||
rates = copy_rates_range(
|
||||
@@ -39,10 +50,21 @@ rates = copy_rates_range(
|
||||
)
|
||||
export_dataframe(rates, Path("rates.csv"), "csv")
|
||||
|
||||
# Resolve SQLite rate compatibility views for downstream tools
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
|
||||
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
|
||||
|
||||
# Recent tick window and minimum margin summary
|
||||
ticks = recent_ticks("EURUSD", seconds=300)
|
||||
margins = minimum_margins("EURUSD")
|
||||
|
||||
# Reuse one MT5 connection for multiple calls
|
||||
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
|
||||
account = client.account_info()
|
||||
positions = client.positions()
|
||||
latest = client.latest_rates("EURUSD", "M1", count=100)
|
||||
summary = client.mt5_summary()
|
||||
summary_table = client.mt5_summary_as_df()
|
||||
|
||||
# Bulk SQLite collection (same behavior as the collect-history CLI command)
|
||||
collect_history(
|
||||
@@ -58,6 +80,8 @@ collect_history(
|
||||
|
||||
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
|
||||
|
||||
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
|
||||
| ---------------- | ---------------------------------- |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
|
||||
### Ticks
|
||||
|
||||
| Command | Description |
|
||||
| ------------- | ------------------------------ |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| Command | Description |
|
||||
| -------------- | ----------------------------------- |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a trailing window |
|
||||
|
||||
### Information
|
||||
|
||||
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume margin summary |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
|
||||
### Trading
|
||||
|
||||
| Command | Description |
|
||||
| ---------------- | ----------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| Command | Description |
|
||||
| ---------------------- | ----------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
|
||||
|
||||
@@ -58,6 +58,7 @@ nav:
|
||||
- Overview: api/index.md
|
||||
- CLI: api/cli.md
|
||||
- SDK: api/sdk.md
|
||||
- Trading: api/trading.md
|
||||
- History Collection (SQLite): api/history.md
|
||||
- Utils: api/utils.md
|
||||
|
||||
|
||||
+92
-1
@@ -2,11 +2,34 @@
|
||||
|
||||
from importlib.metadata import version
|
||||
|
||||
from .history import (
|
||||
RateTarget,
|
||||
build_rate_targets,
|
||||
build_rate_view_name,
|
||||
drop_forming_rate_bar,
|
||||
load_rate_data,
|
||||
load_rate_data_from_connection,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
resolve_history_datasets,
|
||||
resolve_history_tick_flags,
|
||||
resolve_history_timeframes,
|
||||
resolve_rate_tables,
|
||||
resolve_rate_view_name,
|
||||
resolve_rate_view_names,
|
||||
)
|
||||
from .sdk import (
|
||||
AccountSpec,
|
||||
Mt5CliClient,
|
||||
ThrottledHistoryUpdater,
|
||||
account_info,
|
||||
build_config,
|
||||
collect_history,
|
||||
collect_latest_closed_rates_by_granularity,
|
||||
collect_latest_closed_rates_for_accounts,
|
||||
collect_latest_rates,
|
||||
collect_latest_rates_for_accounts,
|
||||
collect_latest_rates_for_accounts_with_retries,
|
||||
copy_rates_from,
|
||||
copy_rates_from_pos,
|
||||
copy_rates_range,
|
||||
@@ -15,9 +38,19 @@ from .sdk import (
|
||||
history_deals,
|
||||
history_orders,
|
||||
last_error,
|
||||
latest_rates,
|
||||
market_book,
|
||||
minimum_margins,
|
||||
mt5_session,
|
||||
mt5_summary,
|
||||
mt5_summary_as_df,
|
||||
orders,
|
||||
positions,
|
||||
recent_history_deals,
|
||||
recent_ticks,
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
substitute_env_placeholders,
|
||||
symbol_info,
|
||||
symbol_info_tick,
|
||||
symbols,
|
||||
@@ -28,31 +61,89 @@ from .sdk import (
|
||||
from .sdk import (
|
||||
version as mt5_version,
|
||||
)
|
||||
from .utils import Dataset, IfExists, detect_format, export_dataframe
|
||||
from .trading import (
|
||||
calculate_margin_and_volume,
|
||||
detect_position_side,
|
||||
determine_order_limits,
|
||||
mt5_trading_session,
|
||||
)
|
||||
from .utils import (
|
||||
TICK_FLAG_MAP,
|
||||
TIMEFRAME_MAP,
|
||||
Dataset,
|
||||
IfExists,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
parse_datetime,
|
||||
parse_tick_flags,
|
||||
parse_timeframe,
|
||||
)
|
||||
|
||||
__version__ = version(__package__) if __package__ else None
|
||||
|
||||
__all__ = [
|
||||
"TICK_FLAG_MAP",
|
||||
"TIMEFRAME_MAP",
|
||||
"AccountSpec",
|
||||
"Dataset",
|
||||
"IfExists",
|
||||
"Mt5CliClient",
|
||||
"RateTarget",
|
||||
"ThrottledHistoryUpdater",
|
||||
"account_info",
|
||||
"build_config",
|
||||
"build_rate_targets",
|
||||
"build_rate_view_name",
|
||||
"calculate_margin_and_volume",
|
||||
"collect_history",
|
||||
"collect_latest_closed_rates_by_granularity",
|
||||
"collect_latest_closed_rates_for_accounts",
|
||||
"collect_latest_rates",
|
||||
"collect_latest_rates_for_accounts",
|
||||
"collect_latest_rates_for_accounts_with_retries",
|
||||
"copy_rates_from",
|
||||
"copy_rates_from_pos",
|
||||
"copy_rates_range",
|
||||
"copy_ticks_from",
|
||||
"copy_ticks_range",
|
||||
"detect_format",
|
||||
"detect_position_side",
|
||||
"determine_order_limits",
|
||||
"drop_forming_rate_bar",
|
||||
"export_dataframe",
|
||||
"export_dataframe_to_sqlite",
|
||||
"history_deals",
|
||||
"history_orders",
|
||||
"last_error",
|
||||
"latest_rates",
|
||||
"load_rate_data",
|
||||
"load_rate_data_from_connection",
|
||||
"load_rate_series_by_granularity",
|
||||
"load_rate_series_from_sqlite",
|
||||
"market_book",
|
||||
"minimum_margins",
|
||||
"mt5_session",
|
||||
"mt5_summary",
|
||||
"mt5_summary_as_df",
|
||||
"mt5_trading_session",
|
||||
"mt5_version",
|
||||
"orders",
|
||||
"parse_datetime",
|
||||
"parse_tick_flags",
|
||||
"parse_timeframe",
|
||||
"positions",
|
||||
"recent_history_deals",
|
||||
"recent_ticks",
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"resolve_history_datasets",
|
||||
"resolve_history_tick_flags",
|
||||
"resolve_history_timeframes",
|
||||
"resolve_rate_tables",
|
||||
"resolve_rate_view_name",
|
||||
"resolve_rate_view_names",
|
||||
"substitute_env_placeholders",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
|
||||
+104
@@ -222,6 +222,31 @@ def rates_from_pos(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def latest_rates(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
timeframe: Annotated[
|
||||
int,
|
||||
typer.Option(
|
||||
click_type=TIMEFRAME_TYPE,
|
||||
help="Timeframe.",
|
||||
),
|
||||
],
|
||||
count: Annotated[int, typer.Option(help="Number of records.")],
|
||||
start_pos: Annotated[
|
||||
int,
|
||||
typer.Option(help="Start position (0 = current bar)."),
|
||||
] = 0,
|
||||
) -> None:
|
||||
"""Export latest rates from a start position."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(
|
||||
ctx,
|
||||
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def rates_range(
|
||||
ctx: typer.Context,
|
||||
@@ -300,6 +325,44 @@ def ticks_range(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def ticks_recent(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
seconds: Annotated[
|
||||
float,
|
||||
typer.Option(help="Lookback window in seconds."),
|
||||
],
|
||||
date_to: Annotated[
|
||||
datetime | None,
|
||||
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
|
||||
] = None,
|
||||
count: Annotated[
|
||||
int,
|
||||
typer.Option(help="Maximum number of ticks to return."),
|
||||
] = 10000,
|
||||
flags: Annotated[
|
||||
int,
|
||||
typer.Option(
|
||||
click_type=TICK_FLAGS_TYPE,
|
||||
help="Tick flags (ALL, INFO, TRADE, or integer).",
|
||||
),
|
||||
] = 1,
|
||||
) -> None:
|
||||
"""Export ticks from a recent time window."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(
|
||||
ctx,
|
||||
lambda: client.recent_ticks(
|
||||
symbol,
|
||||
seconds,
|
||||
date_to=date_to,
|
||||
count=count,
|
||||
flags=flags,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def account_info(ctx: typer.Context) -> None:
|
||||
"""Export account information."""
|
||||
@@ -335,6 +398,16 @@ def symbol_info(
|
||||
_execute_export(ctx, lambda: client.symbol_info(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
def minimum_margins(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
) -> None:
|
||||
"""Export minimum-volume buy and sell margin requirements."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(ctx, lambda: client.minimum_margins(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
def orders(
|
||||
ctx: typer.Context,
|
||||
@@ -427,6 +500,37 @@ def history_deals(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def recent_history_deals(
|
||||
ctx: typer.Context,
|
||||
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
|
||||
date_to: Annotated[
|
||||
datetime | None,
|
||||
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
|
||||
] = None,
|
||||
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
|
||||
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
|
||||
) -> None:
|
||||
"""Export historical deals from a recent trailing window."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(
|
||||
ctx,
|
||||
lambda: client.recent_history_deals(
|
||||
hours,
|
||||
date_to=date_to,
|
||||
group=group,
|
||||
symbol=symbol,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def mt5_summary(ctx: typer.Context) -> None:
|
||||
"""Export a compact terminal/account status summary."""
|
||||
client = _sdk_client(ctx)
|
||||
_execute_export(ctx, client.mt5_summary_as_df)
|
||||
|
||||
|
||||
@app.command()
|
||||
def version(ctx: typer.Context) -> None:
|
||||
"""Export MetaTrader5 version information."""
|
||||
|
||||
+693
-11
@@ -4,8 +4,10 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Literal, cast
|
||||
|
||||
import pandas as pd
|
||||
|
||||
@@ -19,7 +21,7 @@ from .utils import (
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Sequence
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
|
||||
from pdmt5 import Mt5DataClient
|
||||
|
||||
@@ -104,6 +106,23 @@ def resolve_granularity_name(timeframe: int) -> str:
|
||||
return str(timeframe)
|
||||
|
||||
|
||||
def drop_forming_rate_bar(df_rate: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Return closed bars from chronologically ordered MT5 rate data.
|
||||
|
||||
MetaTrader 5 ``copy_rates_from_pos(start_pos=0)`` includes the still-forming
|
||||
current bar as the last row. Slice it off so downstream logic only sees
|
||||
completed bars. Empty frames and single-row frames return empty results.
|
||||
|
||||
Args:
|
||||
df_rate: Rate data ordered oldest-to-newest with the forming bar last.
|
||||
|
||||
Returns:
|
||||
A new DataFrame with all rows except the last. Index and columns are
|
||||
preserved. The input frame is not modified.
|
||||
"""
|
||||
return df_rate.iloc[:-1].copy()
|
||||
|
||||
|
||||
def build_rate_view_name(
|
||||
*,
|
||||
symbol: str,
|
||||
@@ -122,9 +141,641 @@ def build_rate_view_name(
|
||||
return f"rate_{symbol}__{granularity}_{timeframe}"
|
||||
|
||||
|
||||
SqliteConnOrPath = sqlite3.Connection | Path | str
|
||||
|
||||
|
||||
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
|
||||
value = identifier.strip()
|
||||
if not value:
|
||||
msg = f"SQLite {kind} name must not be empty."
|
||||
raise ValueError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def _open_history_connection(
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
) -> tuple[sqlite3.Connection | None, bool]:
|
||||
"""Open a read-only SQLite connection when given a path.
|
||||
|
||||
Returns:
|
||||
A connection and whether the caller should close it. When ``conn_or_path``
|
||||
is None or the path does not exist, returns ``(None, False)`` without
|
||||
creating a database file.
|
||||
"""
|
||||
if conn_or_path is None:
|
||||
return None, False
|
||||
if isinstance(conn_or_path, sqlite3.Connection):
|
||||
return conn_or_path, False
|
||||
path = Path(conn_or_path)
|
||||
if not path.exists():
|
||||
return None, False
|
||||
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
|
||||
return conn, True
|
||||
|
||||
|
||||
def _open_existing_sqlite_database(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
) -> tuple[sqlite3.Connection, bool]:
|
||||
"""Open a read-only SQLite database or reuse an existing connection.
|
||||
|
||||
Returns:
|
||||
Tuple of connection and whether the caller should close it.
|
||||
|
||||
Raises:
|
||||
ValueError: If the database path does not exist or is not a file.
|
||||
"""
|
||||
if isinstance(conn_or_path, sqlite3.Connection):
|
||||
return conn_or_path, False
|
||||
path = Path(conn_or_path)
|
||||
if not path.exists():
|
||||
msg = f"SQLite database not found: {path}"
|
||||
raise ValueError(msg)
|
||||
if not path.is_file():
|
||||
msg = f"SQLite database path is not a file: {path}"
|
||||
raise ValueError(msg)
|
||||
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
|
||||
return conn, True
|
||||
|
||||
|
||||
def _validate_rate_load_request(table: str, count: int | None) -> str:
|
||||
table_name = _require_non_empty_identifier(table, "table or view")
|
||||
if count is not None and count <= 0:
|
||||
msg = "count must be positive when provided."
|
||||
raise ValueError(msg)
|
||||
return table_name
|
||||
|
||||
|
||||
def _ensure_rate_columns(columns: set[str], table: str) -> None:
|
||||
if not columns:
|
||||
msg = f"SQLite table or view not found: {table}"
|
||||
raise ValueError(msg)
|
||||
if "time" not in columns:
|
||||
msg = f"SQLite table or view {table!r} must include a time column."
|
||||
raise ValueError(msg)
|
||||
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
|
||||
msg = (
|
||||
f"SQLite table or view {table!r} must include close, "
|
||||
"or both ask and bid columns."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
|
||||
parsed = frame["time"].map(parse_sqlite_timestamp)
|
||||
if parsed.isna().any():
|
||||
msg = f"SQLite table or view {table!r} contains unparsable time values."
|
||||
raise ValueError(msg)
|
||||
result = frame.drop(columns=["time"])
|
||||
result.index = pd.DatetimeIndex(parsed, name="time")
|
||||
return result.sort_index(kind="stable")
|
||||
|
||||
|
||||
def load_rate_data_from_connection(
|
||||
connection: sqlite3.Connection,
|
||||
table: str,
|
||||
count: int | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Load rate-like data from a SQLite table or view.
|
||||
|
||||
Args:
|
||||
connection: Open SQLite connection.
|
||||
table: Source table or view name.
|
||||
count: Optional number of most recent rows to load.
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by ascending ``time``.
|
||||
|
||||
Raises:
|
||||
ValueError: If inputs, schema, timestamps are invalid, or the table
|
||||
or view contains no rows.
|
||||
"""
|
||||
table_name = _validate_rate_load_request(table, count)
|
||||
columns = get_table_columns(connection, table_name)
|
||||
_ensure_rate_columns(columns, table_name)
|
||||
quoted_table = quote_sqlite_identifier(table_name)
|
||||
if count is None:
|
||||
frame = cast(
|
||||
"pd.DataFrame",
|
||||
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
|
||||
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
|
||||
connection,
|
||||
),
|
||||
)
|
||||
else:
|
||||
frame = cast(
|
||||
"pd.DataFrame",
|
||||
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
|
||||
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
|
||||
connection,
|
||||
params=(count,),
|
||||
),
|
||||
)
|
||||
if frame.empty:
|
||||
msg = f"SQLite table or view {table_name!r} contains no rows."
|
||||
raise ValueError(msg)
|
||||
return _parse_rate_time_index(frame, table_name)
|
||||
|
||||
|
||||
def load_rate_data(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
table: str,
|
||||
count: int | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Load rate-like data from a SQLite database path or connection.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
table: Source table or view name.
|
||||
count: Optional number of most recent rows to load.
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by ascending ``time``.
|
||||
|
||||
"""
|
||||
conn, should_close = _open_existing_sqlite_database(conn_or_path)
|
||||
try:
|
||||
return load_rate_data_from_connection(conn, table, count=count)
|
||||
finally:
|
||||
if should_close:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
|
||||
"""Return distinct timeframe counts per symbol from the normalized rates table."""
|
||||
columns = get_table_columns(conn, Dataset.rates.table_name)
|
||||
if not {"symbol", "timeframe"}.issubset(columns):
|
||||
return None
|
||||
rows = conn.execute(
|
||||
"SELECT symbol, COUNT(DISTINCT timeframe) FROM rates GROUP BY symbol",
|
||||
).fetchall()
|
||||
return {str(symbol): int(count) for symbol, count in rows}
|
||||
|
||||
|
||||
def _load_existing_rate_views(conn: sqlite3.Connection) -> set[str]:
|
||||
"""Return mt5cli-managed ``rate_*__*`` compatibility view names."""
|
||||
rows = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'view' AND name GLOB 'rate_*__*'",
|
||||
).fetchall()
|
||||
return {str(row[0]) for row in rows}
|
||||
|
||||
|
||||
def _rate_view_name_candidates(
|
||||
*,
|
||||
symbol: str,
|
||||
granularity: str,
|
||||
granularity_count: int,
|
||||
timeframe: int,
|
||||
) -> list[str]:
|
||||
"""Return candidate view names in preference order."""
|
||||
single = build_rate_view_name(
|
||||
symbol=symbol,
|
||||
granularity=granularity,
|
||||
granularity_count=1,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
if granularity_count <= 1:
|
||||
return [single]
|
||||
multi = build_rate_view_name(
|
||||
symbol=symbol,
|
||||
granularity=granularity,
|
||||
granularity_count=granularity_count,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
return [multi, single]
|
||||
|
||||
|
||||
def _resolve_rate_view_name_from_context(
|
||||
*,
|
||||
symbol: str,
|
||||
timeframe: int,
|
||||
granularity_name: str,
|
||||
timeframe_counts: dict[str, int] | None,
|
||||
existing_views: set[str],
|
||||
require_existing: bool = False,
|
||||
) -> str:
|
||||
"""Resolve one rate view name using preloaded SQLite metadata.
|
||||
|
||||
Returns:
|
||||
Preferred mt5cli-managed rate compatibility view name.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``require_existing`` is True and no managed view exists.
|
||||
"""
|
||||
if timeframe_counts is None or symbol not in timeframe_counts:
|
||||
candidates = [
|
||||
build_rate_view_name(
|
||||
symbol=symbol,
|
||||
granularity=granularity_name,
|
||||
granularity_count=1,
|
||||
timeframe=timeframe,
|
||||
),
|
||||
build_rate_view_name(
|
||||
symbol=symbol,
|
||||
granularity=granularity_name,
|
||||
granularity_count=2,
|
||||
timeframe=timeframe,
|
||||
),
|
||||
]
|
||||
else:
|
||||
candidates = _rate_view_name_candidates(
|
||||
symbol=symbol,
|
||||
granularity=granularity_name,
|
||||
granularity_count=timeframe_counts[symbol],
|
||||
timeframe=timeframe,
|
||||
)
|
||||
for candidate in candidates:
|
||||
if candidate in existing_views:
|
||||
return candidate
|
||||
if require_existing:
|
||||
msg = (
|
||||
f"No rate compatibility view exists for symbol {symbol!r} "
|
||||
f"and granularity {granularity_name!r}; "
|
||||
f"candidates: {', '.join(candidates)}."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
return candidates[0]
|
||||
|
||||
|
||||
def resolve_rate_view_name(
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
symbol: str,
|
||||
granularity: str,
|
||||
*,
|
||||
require_existing: bool = False,
|
||||
) -> str:
|
||||
"""Resolve the mt5cli-managed rate compatibility view name.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection. When None or a
|
||||
non-existing path and ``require_existing`` is False, the deterministic
|
||||
default view name is returned without creating a database file.
|
||||
symbol: Symbol stored in the normalized ``rates`` table.
|
||||
granularity: Timeframe name (for example ``M1``) or integer string.
|
||||
require_existing: When True, require the database and a managed view to exist.
|
||||
|
||||
Returns:
|
||||
View name such as ``rate_EURUSD__1`` or ``rate_EURUSD__M1_1``.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``require_existing`` is True and the database or view is missing.
|
||||
"""
|
||||
timeframe = parse_timeframe(granularity)
|
||||
granularity_name = resolve_granularity_name(timeframe)
|
||||
conn, should_close = _open_history_connection(conn_or_path)
|
||||
try:
|
||||
if conn is None:
|
||||
if require_existing:
|
||||
path = (
|
||||
conn_or_path
|
||||
if isinstance(conn_or_path, (Path, str))
|
||||
else "database"
|
||||
)
|
||||
msg = f"SQLite database not found: {path}"
|
||||
raise ValueError(msg)
|
||||
return build_rate_view_name(
|
||||
symbol=symbol,
|
||||
granularity=granularity_name,
|
||||
granularity_count=1,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
return _resolve_rate_view_name_from_context(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
granularity_name=granularity_name,
|
||||
timeframe_counts=_load_rates_timeframe_counts(conn),
|
||||
existing_views=_load_existing_rate_views(conn),
|
||||
require_existing=require_existing,
|
||||
)
|
||||
finally:
|
||||
if should_close and conn is not None:
|
||||
conn.close()
|
||||
|
||||
|
||||
def resolve_rate_view_names(
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
symbols: Sequence[str],
|
||||
granularities: Sequence[str],
|
||||
*,
|
||||
require_existing: bool = False,
|
||||
) -> list[str]:
|
||||
"""Resolve rate compatibility view names for symbol and granularity pairs.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection. When None or a
|
||||
non-existing path and ``require_existing`` is False, deterministic
|
||||
default view names are returned without creating a database file.
|
||||
symbols: Symbols stored in the normalized ``rates`` table.
|
||||
granularities: Timeframe names (for example ``M1``) or integer strings.
|
||||
require_existing: When True, require the database and managed views to exist.
|
||||
|
||||
Returns:
|
||||
View names in row-major order: every ``granularity`` for the first
|
||||
symbol, then every granularity for the next symbol, and so on.
|
||||
"""
|
||||
conn, should_close = _open_history_connection(conn_or_path)
|
||||
try:
|
||||
if conn is None:
|
||||
return [
|
||||
resolve_rate_view_name(
|
||||
conn_or_path,
|
||||
symbol,
|
||||
granularity,
|
||||
require_existing=require_existing,
|
||||
)
|
||||
for symbol in symbols
|
||||
for granularity in granularities
|
||||
]
|
||||
timeframe_counts = _load_rates_timeframe_counts(conn)
|
||||
existing_views = _load_existing_rate_views(conn)
|
||||
resolved: list[str] = []
|
||||
for symbol in symbols:
|
||||
for granularity in granularities:
|
||||
timeframe = parse_timeframe(granularity)
|
||||
resolved.append(
|
||||
_resolve_rate_view_name_from_context(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
granularity_name=resolve_granularity_name(timeframe),
|
||||
timeframe_counts=timeframe_counts,
|
||||
existing_views=existing_views,
|
||||
require_existing=require_existing,
|
||||
),
|
||||
)
|
||||
return resolved
|
||||
finally:
|
||||
if should_close and conn is not None:
|
||||
conn.close()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RateTarget:
|
||||
"""A single rate series identified by symbol and timeframe.
|
||||
|
||||
Attributes:
|
||||
symbol: MT5 symbol name, or None when the rate series is addressed only
|
||||
by an explicit table (for example a custom SQLite view).
|
||||
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
|
||||
"""
|
||||
|
||||
symbol: str | None
|
||||
timeframe: int | str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""Normalize accepted timeframe aliases to the stored integer value."""
|
||||
if not isinstance(self.timeframe, int):
|
||||
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
|
||||
|
||||
@property
|
||||
def timeframe_int(self) -> int:
|
||||
"""Return the timeframe as its integer MT5 value."""
|
||||
return cast("int", self.timeframe)
|
||||
|
||||
|
||||
def build_rate_targets(
|
||||
symbols: Sequence[str],
|
||||
timeframes: Sequence[int | str],
|
||||
*,
|
||||
allow_missing_symbol: bool = False,
|
||||
) -> list[RateTarget]:
|
||||
"""Build rate targets for every symbol and timeframe combination.
|
||||
|
||||
Args:
|
||||
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
|
||||
timeframes: MT5 timeframes as integers or names (for example ``M1``).
|
||||
allow_missing_symbol: When True and ``symbols`` is empty, build targets
|
||||
with ``symbol=None`` for each timeframe instead of raising.
|
||||
|
||||
Returns:
|
||||
Targets in row-major order: every timeframe for the first symbol, then
|
||||
every timeframe for the next symbol, and so on.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
|
||||
``allow_missing_symbol`` is False.
|
||||
"""
|
||||
if not timeframes:
|
||||
msg = "At least one timeframe is required."
|
||||
raise ValueError(msg)
|
||||
if not symbols:
|
||||
if not allow_missing_symbol:
|
||||
msg = "At least one symbol is required."
|
||||
raise ValueError(msg)
|
||||
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
|
||||
return [
|
||||
RateTarget(symbol=symbol, timeframe=tf)
|
||||
for symbol in symbols
|
||||
for tf in timeframes
|
||||
]
|
||||
|
||||
|
||||
def resolve_rate_tables(
|
||||
conn_or_path: SqliteConnOrPath | None,
|
||||
targets: Sequence[RateTarget],
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
*,
|
||||
require_existing: bool = False,
|
||||
) -> list[str]:
|
||||
"""Resolve SQLite table or view names for rate targets.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection. May be None when
|
||||
``explicit_tables`` is provided, or when ``require_existing`` is
|
||||
False and deterministic default view names are sufficient.
|
||||
targets: Rate targets to resolve.
|
||||
explicit_tables: Optional explicit table or view names. When provided,
|
||||
they are used as-is and must match the number of targets.
|
||||
require_existing: When True, require the database and managed views to
|
||||
exist for each symbol target. Ignored when ``explicit_tables`` is
|
||||
provided.
|
||||
|
||||
Returns:
|
||||
Table or view names aligned with ``targets``.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
|
||||
match the target count, a target without a symbol is resolved
|
||||
without an explicit table, or ``require_existing`` is True and the
|
||||
database or a managed view is missing.
|
||||
"""
|
||||
target_list = list(targets)
|
||||
if not target_list:
|
||||
msg = "At least one rate target is required."
|
||||
raise ValueError(msg)
|
||||
if explicit_tables is not None:
|
||||
tables = list(explicit_tables)
|
||||
if len(tables) != len(target_list):
|
||||
msg = (
|
||||
f"Expected {len(target_list)} explicit table(s) "
|
||||
f"to match the targets, got {len(tables)}."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
return tables
|
||||
if any(target.symbol is None for target in target_list):
|
||||
msg = (
|
||||
"Cannot resolve a rate table for a target without a symbol; "
|
||||
"provide explicit_tables."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
conn, should_close = _open_history_connection(conn_or_path)
|
||||
try:
|
||||
if conn is None:
|
||||
if require_existing:
|
||||
path = (
|
||||
conn_or_path
|
||||
if isinstance(conn_or_path, (Path, str))
|
||||
else "database"
|
||||
)
|
||||
msg = f"SQLite database not found: {path}"
|
||||
raise ValueError(msg)
|
||||
timeframe_counts = None
|
||||
existing_views: set[str] = set()
|
||||
else:
|
||||
timeframe_counts = _load_rates_timeframe_counts(conn)
|
||||
existing_views = _load_existing_rate_views(conn)
|
||||
resolved: list[str] = []
|
||||
for target in target_list:
|
||||
symbol = cast("str", target.symbol)
|
||||
timeframe = target.timeframe_int
|
||||
resolved.append(
|
||||
_resolve_rate_view_name_from_context(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
granularity_name=resolve_granularity_name(timeframe),
|
||||
timeframe_counts=timeframe_counts,
|
||||
existing_views=existing_views,
|
||||
require_existing=require_existing,
|
||||
),
|
||||
)
|
||||
return resolved
|
||||
finally:
|
||||
if should_close and conn is not None:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_rate_series_from_sqlite(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
targets: Sequence[RateTarget],
|
||||
count: int,
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
) -> dict[tuple[str | None, int], pd.DataFrame]:
|
||||
"""Load multiple rate series from a SQLite database.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair
|
||||
must be unique.
|
||||
count: Number of most recent rows to load per series.
|
||||
explicit_tables: Optional explicit table or view names matching targets.
|
||||
When omitted, managed ``rate_*`` compatibility views must already
|
||||
exist in the database.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol, timeframe_int)`` to each rate DataFrame.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``count`` is not positive, targets are empty, duplicate
|
||||
``(symbol, timeframe_int)`` pairs are present, or table resolution
|
||||
fails.
|
||||
"""
|
||||
if count <= 0:
|
||||
msg = "count must be positive."
|
||||
raise ValueError(msg)
|
||||
target_list = list(targets)
|
||||
if not target_list:
|
||||
msg = "At least one rate target is required."
|
||||
raise ValueError(msg)
|
||||
if explicit_tables is None and any(target.symbol is None for target in target_list):
|
||||
msg = (
|
||||
"Cannot resolve a rate table for a target without a symbol; "
|
||||
"provide explicit_tables."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
seen_keys: set[tuple[str | None, int]] = set()
|
||||
for target in target_list:
|
||||
key = (target.symbol, target.timeframe_int)
|
||||
if key in seen_keys:
|
||||
symbol_repr = repr(target.symbol)
|
||||
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
|
||||
raise ValueError(msg)
|
||||
seen_keys.add(key)
|
||||
tables = (
|
||||
resolve_rate_tables(None, target_list, explicit_tables)
|
||||
if explicit_tables is not None
|
||||
else None
|
||||
)
|
||||
conn, should_close = _open_existing_sqlite_database(conn_or_path)
|
||||
try:
|
||||
resolved_tables = tables or resolve_rate_tables(
|
||||
conn,
|
||||
target_list,
|
||||
require_existing=True,
|
||||
)
|
||||
return {
|
||||
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
|
||||
conn,
|
||||
table,
|
||||
count=count,
|
||||
)
|
||||
for target, table in zip(target_list, resolved_tables, strict=True)
|
||||
}
|
||||
finally:
|
||||
if should_close:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_rate_series_by_granularity(
|
||||
conn_or_path: SqliteConnOrPath,
|
||||
symbols: Sequence[str],
|
||||
granularities: Sequence[int | str],
|
||||
count: int,
|
||||
*,
|
||||
explicit_tables: Sequence[str] | None = None,
|
||||
allow_missing_symbol: bool = False,
|
||||
) -> dict[tuple[str | None, str], pd.DataFrame]:
|
||||
"""Load rate series keyed by symbol and string granularity name.
|
||||
|
||||
Builds targets with :func:`build_rate_targets` and loads them with
|
||||
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
|
||||
name (for example ``M1``) instead of the integer timeframe to reduce
|
||||
downstream boilerplate.
|
||||
|
||||
Args:
|
||||
conn_or_path: SQLite database path or open connection.
|
||||
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
|
||||
granularities: MT5 timeframes as integers or names (for example ``M1``).
|
||||
count: Number of most recent rows to load per series.
|
||||
explicit_tables: Optional explicit table or view names matching the
|
||||
built targets in row-major order. Required when symbols are omitted.
|
||||
allow_missing_symbol: When True and ``symbols`` is empty, build targets
|
||||
with ``symbol=None`` for each granularity instead of raising.
|
||||
|
||||
Returns:
|
||||
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
|
||||
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
|
||||
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
|
||||
table resolution fails, or duplicate targets are present.
|
||||
"""
|
||||
targets = build_rate_targets(
|
||||
symbols,
|
||||
granularities,
|
||||
allow_missing_symbol=allow_missing_symbol,
|
||||
)
|
||||
series = load_rate_series_from_sqlite(
|
||||
conn_or_path,
|
||||
targets,
|
||||
count,
|
||||
explicit_tables=explicit_tables,
|
||||
)
|
||||
return {
|
||||
(symbol, resolve_granularity_name(timeframe)): frame
|
||||
for (symbol, timeframe), frame in series.items()
|
||||
}
|
||||
|
||||
|
||||
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
|
||||
"""Return existing SQLite columns for a table."""
|
||||
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
|
||||
quoted_table = quote_sqlite_identifier(table)
|
||||
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
|
||||
return {str(row[1]) for row in rows}
|
||||
|
||||
|
||||
@@ -404,7 +1055,20 @@ def drop_duplicates_in_table(
|
||||
)
|
||||
|
||||
|
||||
DedupScope = tuple[str, tuple[object, ...]]
|
||||
@dataclass(frozen=True)
|
||||
class DedupScope:
|
||||
"""Scoped deduplication predicate and the columns it references.
|
||||
|
||||
Attributes:
|
||||
where: SQL predicate appended to the duplicate-removal query.
|
||||
params: Parameters bound to the scope predicate.
|
||||
required_columns: Columns that must be present in the written table for
|
||||
the scope to run.
|
||||
"""
|
||||
|
||||
where: str
|
||||
params: tuple[object, ...]
|
||||
required_columns: frozenset[str]
|
||||
|
||||
|
||||
def _record_dedup_scope(
|
||||
@@ -412,17 +1076,25 @@ def _record_dedup_scope(
|
||||
dataset: Dataset,
|
||||
scope_where: str,
|
||||
scope_params: tuple[object, ...],
|
||||
required_columns: frozenset[str],
|
||||
) -> None:
|
||||
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
|
||||
dedup_scopes.setdefault(dataset, []).append(
|
||||
DedupScope(scope_where, scope_params, required_columns),
|
||||
)
|
||||
|
||||
|
||||
def deduplicate_history_tables(
|
||||
conn: sqlite3.Connection,
|
||||
written_columns: dict[Dataset, set[str]],
|
||||
written_tables: set[Dataset],
|
||||
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
|
||||
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
|
||||
) -> None:
|
||||
"""Deduplicate appended history tables by stable identifiers."""
|
||||
"""Deduplicate appended history tables by stable identifiers.
|
||||
|
||||
Scopes whose required columns are not present in the written table are
|
||||
skipped. If all scopes for a dataset are skipped, the table receives one
|
||||
unscoped deduplication pass instead.
|
||||
"""
|
||||
cursor = conn.cursor()
|
||||
for dataset in written_tables:
|
||||
columns = written_columns.get(dataset, set())
|
||||
@@ -441,16 +1113,19 @@ def deduplicate_history_tables(
|
||||
table,
|
||||
)
|
||||
continue
|
||||
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
|
||||
raw_scopes: Sequence[DedupScope] = (
|
||||
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
|
||||
)
|
||||
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
|
||||
if scopes:
|
||||
for scope_where, scope_params in scopes:
|
||||
for scope in scopes:
|
||||
drop_duplicates_in_table(
|
||||
cursor,
|
||||
table,
|
||||
list(keys),
|
||||
keep="last",
|
||||
scope_where=scope_where,
|
||||
scope_params=scope_params,
|
||||
scope_where=scope.where,
|
||||
scope_params=scope.params,
|
||||
)
|
||||
continue
|
||||
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
|
||||
@@ -817,6 +1492,7 @@ def _write_incremental_rates(
|
||||
Dataset.rates,
|
||||
"symbol = ? AND timeframe = ? AND time >= ?",
|
||||
(symbol, timeframe, start_date),
|
||||
frozenset({"symbol", "timeframe", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -855,6 +1531,7 @@ def _write_incremental_ticks(
|
||||
Dataset.ticks,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -893,6 +1570,7 @@ def _write_incremental_history_orders(
|
||||
Dataset.history_orders,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
@@ -946,6 +1624,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_by_symbol[symbol, None]),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
if "type" in columns:
|
||||
_record_dedup_scope(
|
||||
@@ -953,6 +1632,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
|
||||
(account_event_start,),
|
||||
frozenset({"type", "time"}),
|
||||
)
|
||||
if "type" not in columns and "symbol" in columns:
|
||||
_record_dedup_scope(
|
||||
@@ -960,6 +1640,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"(symbol IS NULL OR symbol = '') AND time >= ?",
|
||||
(account_event_start,),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
return
|
||||
start_by_symbol = load_incremental_start_datetimes(
|
||||
@@ -987,6 +1668,7 @@ def _write_incremental_history_deals(
|
||||
Dataset.history_deals,
|
||||
"symbol = ? AND time >= ?",
|
||||
(symbol, start_date),
|
||||
frozenset({"symbol", "time"}),
|
||||
)
|
||||
|
||||
|
||||
|
||||
+1059
-8
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,210 @@
|
||||
"""Trading-capable MetaTrader 5 session helpers and operational utilities."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
|
||||
from pdmt5 import Mt5Config, Mt5TradingClient
|
||||
|
||||
from .sdk import build_config
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
import pandas as pd
|
||||
|
||||
PositionSide = Literal["long", "short"]
|
||||
OrderSide = Literal["long", "short"]
|
||||
|
||||
__all__ = [
|
||||
"OrderSide",
|
||||
"PositionSide",
|
||||
"calculate_margin_and_volume",
|
||||
"detect_position_side",
|
||||
"determine_order_limits",
|
||||
"mt5_trading_session",
|
||||
]
|
||||
|
||||
|
||||
def _require_unit_ratio(value: float, name: str) -> None:
|
||||
if not 0.0 <= value <= 1.0:
|
||||
msg = f"{name} must be between 0 and 1 inclusive."
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _require_protective_ratio(value: float, name: str) -> None:
|
||||
if not 0.0 <= value < 1.0:
|
||||
msg = f"{name} must be at least 0 and less than 1."
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _sum_position_volume(positions: pd.DataFrame, position_type: object) -> float:
|
||||
matched = positions.loc[positions["type"] == position_type, "volume"]
|
||||
if matched.empty:
|
||||
return 0.0
|
||||
return float(matched.to_numpy(dtype=float).sum())
|
||||
|
||||
|
||||
def _normalize_order_side(side: str) -> OrderSide:
|
||||
normalized = side.lower()
|
||||
if normalized in {"long", "buy"}:
|
||||
return "long"
|
||||
if normalized in {"short", "sell"}:
|
||||
return "short"
|
||||
msg = (
|
||||
f"Unsupported order side: {side!r}. Expected 'long', 'short', 'buy', or 'sell'."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def detect_position_side(
|
||||
client: Mt5TradingClient,
|
||||
symbol: str,
|
||||
) -> PositionSide | None:
|
||||
"""Detect the net open position side for a symbol.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
symbol: Symbol to inspect.
|
||||
|
||||
Returns:
|
||||
``"long"`` when net buy volume exceeds sell volume, ``"short"`` when
|
||||
net sell volume exceeds buy volume, or ``None`` when no positions exist
|
||||
or buy/sell volumes are exactly balanced.
|
||||
"""
|
||||
positions = client.positions_get_as_df(symbol=symbol)
|
||||
if positions.empty:
|
||||
return None
|
||||
|
||||
buy_type = client.mt5.POSITION_TYPE_BUY
|
||||
sell_type = client.mt5.POSITION_TYPE_SELL
|
||||
buy_volume = _sum_position_volume(positions, buy_type)
|
||||
sell_volume = _sum_position_volume(positions, sell_type)
|
||||
net_volume = buy_volume - sell_volume
|
||||
if net_volume > 0:
|
||||
return "long"
|
||||
if net_volume < 0:
|
||||
return "short"
|
||||
return None
|
||||
|
||||
|
||||
def calculate_margin_and_volume(
|
||||
client: Mt5TradingClient,
|
||||
symbol: str,
|
||||
unit_margin_ratio: float,
|
||||
preserved_margin_ratio: float,
|
||||
) -> dict[str, float]:
|
||||
"""Calculate tradable margin and volumes from account free margin.
|
||||
|
||||
Applies ``preserved_margin_ratio`` to keep a reserve off ``margin_free``,
|
||||
then allocates ``unit_margin_ratio`` of the remainder as the margin budget
|
||||
for volume sizing on both buy and sell sides.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
symbol: Symbol used for minimum-lot margin and volume calculations.
|
||||
unit_margin_ratio: Fraction of post-reserve margin to allocate per unit.
|
||||
preserved_margin_ratio: Fraction of ``margin_free`` to preserve.
|
||||
|
||||
Returns:
|
||||
Dictionary with ``margin_free``, ``available_margin``, ``trade_margin``,
|
||||
``buy_volume``, and ``sell_volume``. Negative ``margin_free`` values are
|
||||
clamped to ``0.0`` before sizing.
|
||||
"""
|
||||
_require_unit_ratio(unit_margin_ratio, "unit_margin_ratio")
|
||||
_require_unit_ratio(preserved_margin_ratio, "preserved_margin_ratio")
|
||||
|
||||
account = client.account_info_as_dict()
|
||||
margin_free = max(0.0, float(account.get("margin_free") or 0.0))
|
||||
available_margin = margin_free * (1.0 - preserved_margin_ratio)
|
||||
trade_margin = available_margin * unit_margin_ratio
|
||||
buy_volume = client.calculate_volume_by_margin(symbol, trade_margin, "BUY")
|
||||
sell_volume = client.calculate_volume_by_margin(symbol, trade_margin, "SELL")
|
||||
return {
|
||||
"margin_free": margin_free,
|
||||
"available_margin": available_margin,
|
||||
"trade_margin": trade_margin,
|
||||
"buy_volume": buy_volume,
|
||||
"sell_volume": sell_volume,
|
||||
}
|
||||
|
||||
|
||||
def determine_order_limits(
|
||||
client: Mt5TradingClient,
|
||||
symbol: str,
|
||||
side: OrderSide | str,
|
||||
stop_loss_limit_ratio: float,
|
||||
take_profit_limit_ratio: float,
|
||||
) -> dict[str, float | None]:
|
||||
"""Derive entry and protective order prices from current market quotes.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
symbol: Symbol used for the quote lookup.
|
||||
side: Position side as ``"long"``/``"short"`` (``"buy"``/``"sell"``
|
||||
aliases are accepted).
|
||||
stop_loss_limit_ratio: Relative distance from entry for stop loss in
|
||||
``[0, 1)``. A value of ``0`` omits the stop loss.
|
||||
take_profit_limit_ratio: Relative distance from entry for take profit in
|
||||
``[0, 1)``. A value of ``0`` omits the take profit.
|
||||
|
||||
Returns:
|
||||
Dictionary with ``entry``, ``stop_loss``, and ``take_profit`` keys.
|
||||
Omitted protective levels are returned as ``None``.
|
||||
"""
|
||||
_require_protective_ratio(stop_loss_limit_ratio, "stop_loss_limit_ratio")
|
||||
_require_protective_ratio(take_profit_limit_ratio, "take_profit_limit_ratio")
|
||||
normalized_side = _normalize_order_side(side)
|
||||
tick = client.symbol_info_tick_as_dict(symbol=symbol)
|
||||
entry = float(tick["ask"] if normalized_side == "long" else tick["bid"])
|
||||
|
||||
stop_loss: float | None = None
|
||||
if stop_loss_limit_ratio > 0:
|
||||
if normalized_side == "long":
|
||||
stop_loss = entry * (1.0 - stop_loss_limit_ratio)
|
||||
else:
|
||||
stop_loss = entry * (1.0 + stop_loss_limit_ratio)
|
||||
|
||||
take_profit: float | None = None
|
||||
if take_profit_limit_ratio > 0:
|
||||
if normalized_side == "long":
|
||||
take_profit = entry * (1.0 + take_profit_limit_ratio)
|
||||
else:
|
||||
take_profit = entry * (1.0 - take_profit_limit_ratio)
|
||||
|
||||
return {
|
||||
"entry": entry,
|
||||
"stop_loss": stop_loss,
|
||||
"take_profit": take_profit,
|
||||
}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def mt5_trading_session(
|
||||
config: Mt5Config | None = None,
|
||||
retry_count: int = 0,
|
||||
) -> Iterator[Mt5TradingClient]:
|
||||
"""Open a trading-capable MT5 session and always shut down safely.
|
||||
|
||||
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` when set,
|
||||
initializes and logs in via ``initialize_and_login_mt5()``, yields a
|
||||
connected :class:`~pdmt5.Mt5TradingClient`, and calls ``shutdown()`` on
|
||||
exit even when an error is raised inside the context.
|
||||
|
||||
Args:
|
||||
config: MT5 connection configuration. Defaults to an empty config that
|
||||
attaches to a running terminal.
|
||||
retry_count: Number of initialization retries passed to
|
||||
``Mt5TradingClient``.
|
||||
|
||||
Yields:
|
||||
Connected ``Mt5TradingClient`` bound to the session.
|
||||
"""
|
||||
mt5_config = config or build_config()
|
||||
client = Mt5TradingClient(config=mt5_config, retry_count=retry_count)
|
||||
try:
|
||||
client.initialize_and_login_mt5()
|
||||
yield client
|
||||
finally:
|
||||
client.shutdown()
|
||||
+55
-10
@@ -2,16 +2,18 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, TypeGuard, cast
|
||||
from typing import TYPE_CHECKING, Any, TypeGuard
|
||||
|
||||
import click
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
import pandas as pd
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -260,6 +262,50 @@ def detect_format(
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def export_dataframe_to_sqlite(
|
||||
df: pd.DataFrame,
|
||||
output_path: Path,
|
||||
table_name: str = "data",
|
||||
*,
|
||||
if_exists: IfExists = IfExists.APPEND,
|
||||
index: bool = False,
|
||||
index_label: str | None = None,
|
||||
deduplicate_on: Sequence[str] | None = None,
|
||||
) -> None:
|
||||
"""Write a DataFrame to SQLite with configurable append and deduplication.
|
||||
|
||||
Args:
|
||||
df: DataFrame to export.
|
||||
output_path: SQLite database path.
|
||||
table_name: Target table name.
|
||||
if_exists: Conflict behavior when the table already exists.
|
||||
index: Whether to write the DataFrame index as a column.
|
||||
index_label: Column name for the index when ``index=True``.
|
||||
deduplicate_on: Optional key columns to deduplicate after writing,
|
||||
keeping the latest ``ROWID`` per key group. Deduplication scans the
|
||||
full table, so repeated appends cost O(table size); index the key
|
||||
columns when appending frequently.
|
||||
"""
|
||||
with sqlite3.connect(output_path) as conn:
|
||||
df.to_sql( # type: ignore[reportUnknownMemberType]
|
||||
table_name,
|
||||
conn,
|
||||
if_exists=if_exists.value,
|
||||
index=index,
|
||||
index_label=index_label,
|
||||
)
|
||||
if deduplicate_on:
|
||||
from .history import drop_duplicates_in_table # noqa: PLC0415
|
||||
|
||||
drop_duplicates_in_table(
|
||||
conn.cursor(),
|
||||
table_name,
|
||||
list(deduplicate_on),
|
||||
keep="last",
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def export_dataframe(
|
||||
df: pd.DataFrame,
|
||||
output_path: Path,
|
||||
@@ -289,14 +335,13 @@ def export_dataframe(
|
||||
elif output_format == "parquet":
|
||||
df.to_parquet(output_path, index=False)
|
||||
elif output_format == "sqlite3":
|
||||
sqlite3 = cast("Any", importlib.import_module("sqlite3"))
|
||||
with sqlite3.connect(output_path) as conn:
|
||||
df.to_sql( # type: ignore[reportUnknownMemberType]
|
||||
table_name,
|
||||
conn,
|
||||
if_exists="replace",
|
||||
index=False,
|
||||
)
|
||||
export_dataframe_to_sqlite(
|
||||
df,
|
||||
output_path,
|
||||
table_name,
|
||||
if_exists=IfExists.REPLACE,
|
||||
index=False,
|
||||
)
|
||||
else:
|
||||
msg = f"Unsupported output format: {output_format}"
|
||||
raise ValueError(msg)
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "mt5cli"
|
||||
version = "0.4.2"
|
||||
version = "0.6.1"
|
||||
description = "Command-line tool for MetaTrader 5"
|
||||
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
|
||||
+173
-1
@@ -6,7 +6,7 @@ import json
|
||||
import logging
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
|
||||
client.market_book_get_as_df.return_value = sample_df
|
||||
client.order_check_as_df.return_value = sample_df
|
||||
client.order_send_as_df.return_value = sample_df
|
||||
client.version.return_value = (5, 0, 1)
|
||||
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
|
||||
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
|
||||
client.symbols_total.return_value = 42
|
||||
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
|
||||
return client
|
||||
|
||||
@@ -223,6 +227,37 @@ class TestCommands:
|
||||
count=50,
|
||||
)
|
||||
|
||||
def test_latest_rates(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test latest-rates command."""
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"latest-rates",
|
||||
"--symbol",
|
||||
"GBPUSD",
|
||||
"--timeframe",
|
||||
"H1",
|
||||
"--count",
|
||||
"50",
|
||||
"--start-pos",
|
||||
"2",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
|
||||
symbol="GBPUSD",
|
||||
timeframe=16385,
|
||||
start_pos=2,
|
||||
count=50,
|
||||
)
|
||||
|
||||
def test_rates_range(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
@@ -316,6 +351,65 @@ class TestCommands:
|
||||
flags=2,
|
||||
)
|
||||
|
||||
def test_ticks_recent(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test ticks-recent command."""
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"ticks-recent",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
"--seconds",
|
||||
"120",
|
||||
"--date-to",
|
||||
"2024-01-02",
|
||||
"--count",
|
||||
"500",
|
||||
"--flags",
|
||||
"ALL",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.copy_ticks_from_as_df.assert_called_once_with(
|
||||
symbol="EURUSD",
|
||||
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
|
||||
count=500,
|
||||
flags=1,
|
||||
)
|
||||
mock_client.copy_ticks_range_as_df.assert_not_called()
|
||||
|
||||
def test_minimum_margins(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test minimum-margins command."""
|
||||
sym = MagicMock(volume_min=0.01)
|
||||
account = MagicMock(currency="USD")
|
||||
tick = MagicMock(ask=1.1010, bid=1.1000)
|
||||
mock_client.symbol_info.return_value = sym
|
||||
mock_client.account_info.return_value = account
|
||||
mock_client.symbol_info_tick.return_value = tick
|
||||
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
|
||||
mock_client.mt5.ORDER_TYPE_BUY = 0
|
||||
mock_client.mt5.ORDER_TYPE_SELL = 1
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.symbol_info.assert_called_once_with("EURUSD")
|
||||
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
|
||||
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
|
||||
|
||||
def test_orders(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
@@ -392,6 +486,84 @@ class TestCommands:
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.history_deals_get_as_df.assert_called_once()
|
||||
|
||||
def test_recent_history_deals(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test recent-history-deals command."""
|
||||
output = tmp_path / "out.csv"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"recent-history-deals",
|
||||
"--hours",
|
||||
"6",
|
||||
"--date-to",
|
||||
"2024-01-02",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.history_deals_get_as_df.assert_called_once_with(
|
||||
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
|
||||
date_to=datetime(2024, 1, 2, tzinfo=UTC),
|
||||
group=None,
|
||||
symbol="EURUSD",
|
||||
ticket=None,
|
||||
position=None,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("filename", "reader"),
|
||||
[
|
||||
("summary.csv", "csv"),
|
||||
("summary.json", "json"),
|
||||
("summary.db", "sqlite3"),
|
||||
("summary.parquet", "parquet"),
|
||||
],
|
||||
)
|
||||
def test_mt5_summary_export_formats(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
filename: str,
|
||||
reader: str,
|
||||
) -> None:
|
||||
"""Test mt5-summary writes export-safe files for supported formats."""
|
||||
output = tmp_path / filename
|
||||
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert output.exists()
|
||||
mock_client.version.assert_called_once()
|
||||
mock_client.terminal_info.assert_called_once()
|
||||
mock_client.account_info.assert_called_once()
|
||||
mock_client.symbols_total.assert_called_once()
|
||||
if reader == "csv":
|
||||
frame = pd.read_csv(output)
|
||||
elif reader == "json":
|
||||
with output.open() as f:
|
||||
records = json.load(f)
|
||||
frame = pd.DataFrame(records)
|
||||
elif reader == "sqlite3":
|
||||
with sqlite3.connect(output) as conn:
|
||||
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT * FROM data",
|
||||
conn,
|
||||
)
|
||||
else:
|
||||
frame = pd.read_parquet(output)
|
||||
assert len(frame) == 1
|
||||
assert frame.iloc[0].to_dict() == {
|
||||
"version": "[5,0,1]",
|
||||
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
|
||||
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
|
||||
"symbols_total": 42,
|
||||
}
|
||||
|
||||
def test_version(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
|
||||
+971
-3
File diff suppressed because it is too large
Load Diff
+1266
-5
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,356 @@
|
||||
"""Tests for trading session helpers and operational utilities."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from pdmt5 import Mt5RuntimeError
|
||||
from pytest_mock import MockerFixture # noqa: TC002
|
||||
|
||||
from mt5cli.sdk import build_config
|
||||
from mt5cli.trading import (
|
||||
calculate_margin_and_volume,
|
||||
detect_position_side,
|
||||
determine_order_limits,
|
||||
mt5_trading_session,
|
||||
)
|
||||
|
||||
|
||||
class TestDetectPositionSide:
|
||||
"""Tests for detect_position_side."""
|
||||
|
||||
def test_returns_none_when_no_positions(self) -> None:
|
||||
"""Test None is returned when no open positions exist."""
|
||||
client = MagicMock()
|
||||
client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
|
||||
assert detect_position_side(client, "EURUSD") is None
|
||||
|
||||
def test_returns_long_for_net_buy_volume(self) -> None:
|
||||
"""Test long is returned when buy volume exceeds sell volume."""
|
||||
client = MagicMock()
|
||||
client.mt5.POSITION_TYPE_BUY = 0
|
||||
client.mt5.POSITION_TYPE_SELL = 1
|
||||
client.positions_get_as_df.return_value = pd.DataFrame(
|
||||
{
|
||||
"type": [0, 0, 1],
|
||||
"volume": [0.2, 0.1, 0.05],
|
||||
},
|
||||
)
|
||||
|
||||
assert detect_position_side(client, "EURUSD") == "long"
|
||||
|
||||
def test_returns_short_for_net_sell_volume(self) -> None:
|
||||
"""Test short is returned when sell volume exceeds buy volume."""
|
||||
client = MagicMock()
|
||||
client.mt5.POSITION_TYPE_BUY = 0
|
||||
client.mt5.POSITION_TYPE_SELL = 1
|
||||
client.positions_get_as_df.return_value = pd.DataFrame(
|
||||
{
|
||||
"type": [1, 1],
|
||||
"volume": [0.3, 0.1],
|
||||
},
|
||||
)
|
||||
|
||||
assert detect_position_side(client, "EURUSD") == "short"
|
||||
|
||||
def test_returns_none_for_balanced_hedged_positions(self) -> None:
|
||||
"""Test None is returned when buy and sell volumes net to zero."""
|
||||
client = MagicMock()
|
||||
client.mt5.POSITION_TYPE_BUY = 0
|
||||
client.mt5.POSITION_TYPE_SELL = 1
|
||||
client.positions_get_as_df.return_value = pd.DataFrame(
|
||||
{
|
||||
"type": [0, 1],
|
||||
"volume": [0.2, 0.2],
|
||||
},
|
||||
)
|
||||
|
||||
assert detect_position_side(client, "EURUSD") is None
|
||||
|
||||
|
||||
class TestCalculateMarginAndVolume:
|
||||
"""Tests for calculate_margin_and_volume."""
|
||||
|
||||
def test_calculates_margin_budget_and_volumes(self) -> None:
|
||||
"""Test margin budget and buy/sell volumes are derived from ratios."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_dict.return_value = {"margin_free": 1000.0}
|
||||
client.calculate_volume_by_margin.side_effect = [0.3, 0.2]
|
||||
|
||||
result = calculate_margin_and_volume(
|
||||
client,
|
||||
"EURUSD",
|
||||
unit_margin_ratio=0.5,
|
||||
preserved_margin_ratio=0.2,
|
||||
)
|
||||
|
||||
assert result == {
|
||||
"margin_free": 1000.0,
|
||||
"available_margin": 800.0,
|
||||
"trade_margin": 400.0,
|
||||
"buy_volume": 0.3,
|
||||
"sell_volume": 0.2,
|
||||
}
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "BUY")
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "SELL")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("account_dict", "expected_margin_free"),
|
||||
[
|
||||
({"margin_free": 0.0}, 0.0),
|
||||
({}, 0.0),
|
||||
({"margin_free": None}, 0.0),
|
||||
],
|
||||
)
|
||||
def test_zero_or_missing_margin_free(
|
||||
self,
|
||||
account_dict: dict[str, float | None],
|
||||
expected_margin_free: float,
|
||||
) -> None:
|
||||
"""Test missing or zero margin_free yields zero trade margin."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_dict.return_value = account_dict
|
||||
client.calculate_volume_by_margin.return_value = 0.0
|
||||
|
||||
result = calculate_margin_and_volume(
|
||||
client,
|
||||
"EURUSD",
|
||||
unit_margin_ratio=0.5,
|
||||
preserved_margin_ratio=0.2,
|
||||
)
|
||||
|
||||
assert result["margin_free"] == expected_margin_free
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
|
||||
|
||||
def test_clamps_negative_margin_free_to_zero(self) -> None:
|
||||
"""Test negative margin_free is clamped to zero before sizing."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_dict.return_value = {"margin_free": -500.0}
|
||||
client.calculate_volume_by_margin.return_value = 0.0
|
||||
|
||||
result = calculate_margin_and_volume(
|
||||
client,
|
||||
"EURUSD",
|
||||
unit_margin_ratio=0.5,
|
||||
preserved_margin_ratio=0.2,
|
||||
)
|
||||
|
||||
expected_margin_free = 0.0
|
||||
assert result["margin_free"] == expected_margin_free
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
|
||||
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("unit_ratio", "preserved_ratio"),
|
||||
[
|
||||
(-0.1, 0.0),
|
||||
(1.1, 0.0),
|
||||
(0.5, -0.1),
|
||||
(0.5, 1.1),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_ratios(
|
||||
self,
|
||||
unit_ratio: float,
|
||||
preserved_ratio: float,
|
||||
) -> None:
|
||||
"""Test invalid ratio values raise ValueError."""
|
||||
with pytest.raises(ValueError, match="must be between 0 and 1"):
|
||||
calculate_margin_and_volume(
|
||||
MagicMock(),
|
||||
"EURUSD",
|
||||
unit_margin_ratio=unit_ratio,
|
||||
preserved_margin_ratio=preserved_ratio,
|
||||
)
|
||||
|
||||
|
||||
class TestDetermineOrderLimits:
|
||||
"""Tests for determine_order_limits."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("side", "expected_entry_key"),
|
||||
[
|
||||
("long", "ask"),
|
||||
("short", "bid"),
|
||||
("buy", "ask"),
|
||||
("sell", "bid"),
|
||||
],
|
||||
)
|
||||
def test_uses_expected_quote_for_entry(
|
||||
self,
|
||||
side: str,
|
||||
expected_entry_key: str,
|
||||
) -> None:
|
||||
"""Test entry price is taken from ask for long/buy and bid for short/sell."""
|
||||
client = MagicMock()
|
||||
client.symbol_info_tick_as_dict.return_value = {"ask": 1.1010, "bid": 1.1000}
|
||||
|
||||
result = determine_order_limits(
|
||||
client,
|
||||
"EURUSD",
|
||||
side,
|
||||
stop_loss_limit_ratio=0.0,
|
||||
take_profit_limit_ratio=0.0,
|
||||
)
|
||||
|
||||
assert (
|
||||
result["entry"]
|
||||
== client.symbol_info_tick_as_dict.return_value[expected_entry_key]
|
||||
)
|
||||
assert result["stop_loss"] is None
|
||||
assert result["take_profit"] is None
|
||||
|
||||
def test_calculates_long_protective_levels(self) -> None:
|
||||
"""Test long stop loss and take profit are placed below/above entry."""
|
||||
client = MagicMock()
|
||||
client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
|
||||
|
||||
result = determine_order_limits(
|
||||
client,
|
||||
"EURUSD",
|
||||
"long",
|
||||
stop_loss_limit_ratio=0.02,
|
||||
take_profit_limit_ratio=0.03,
|
||||
)
|
||||
|
||||
assert result == {
|
||||
"entry": 100.0,
|
||||
"stop_loss": 98.0,
|
||||
"take_profit": 103.0,
|
||||
}
|
||||
|
||||
def test_calculates_short_protective_levels(self) -> None:
|
||||
"""Test short stop loss and take profit are placed above/below entry."""
|
||||
client = MagicMock()
|
||||
client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
|
||||
|
||||
result = determine_order_limits(
|
||||
client,
|
||||
"EURUSD",
|
||||
"short",
|
||||
stop_loss_limit_ratio=0.02,
|
||||
take_profit_limit_ratio=0.03,
|
||||
)
|
||||
|
||||
assert result == {
|
||||
"entry": 99.0,
|
||||
"stop_loss": 100.98,
|
||||
"take_profit": 96.03,
|
||||
}
|
||||
|
||||
def test_rejects_unknown_side(self) -> None:
|
||||
"""Test unsupported side values raise ValueError."""
|
||||
with pytest.raises(ValueError, match="Unsupported order side"):
|
||||
determine_order_limits(
|
||||
MagicMock(),
|
||||
"EURUSD",
|
||||
"flat",
|
||||
stop_loss_limit_ratio=0.01,
|
||||
take_profit_limit_ratio=0.01,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("stop_loss_ratio", "take_profit_ratio"),
|
||||
[
|
||||
(-0.05, 0.01),
|
||||
(0.01, 2.0),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_protective_ratios(
|
||||
self,
|
||||
stop_loss_ratio: float,
|
||||
take_profit_ratio: float,
|
||||
) -> None:
|
||||
"""Test out-of-range protective ratios raise ValueError."""
|
||||
with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
|
||||
determine_order_limits(
|
||||
MagicMock(),
|
||||
"EURUSD",
|
||||
"long",
|
||||
stop_loss_limit_ratio=stop_loss_ratio,
|
||||
take_profit_limit_ratio=take_profit_ratio,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "ratio"),
|
||||
[
|
||||
("stop_loss_limit_ratio", 1.0),
|
||||
("take_profit_limit_ratio", 1.0),
|
||||
],
|
||||
)
|
||||
def test_rejects_unit_boundary_protective_ratios(
|
||||
self,
|
||||
field: str,
|
||||
ratio: float,
|
||||
) -> None:
|
||||
"""Test protective ratios of exactly 1.0 are rejected."""
|
||||
kwargs = {
|
||||
"stop_loss_limit_ratio": 0.01,
|
||||
"take_profit_limit_ratio": 0.01,
|
||||
field: ratio,
|
||||
}
|
||||
with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
|
||||
determine_order_limits(
|
||||
MagicMock(),
|
||||
"EURUSD",
|
||||
"long",
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
class TestMt5TradingSession:
|
||||
"""Tests for the mt5_trading_session context manager."""
|
||||
|
||||
def test_yields_connected_client_and_shuts_down(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test mt5_trading_session connects, yields a client, and shuts down."""
|
||||
mock_client = MagicMock()
|
||||
trading_client = mocker.patch(
|
||||
"mt5cli.trading.Mt5TradingClient",
|
||||
return_value=mock_client,
|
||||
)
|
||||
|
||||
with mt5_trading_session(
|
||||
build_config(path="/opt/mt5/terminal64.exe"),
|
||||
retry_count=2,
|
||||
) as client:
|
||||
mock_client.initialize_and_login_mt5.assert_called_once()
|
||||
assert client is mock_client
|
||||
|
||||
trading_client.assert_called_once()
|
||||
assert trading_client.call_args.kwargs["retry_count"] == 2
|
||||
assert (
|
||||
trading_client.call_args.kwargs["config"].path == "/opt/mt5/terminal64.exe"
|
||||
)
|
||||
mock_client.shutdown.assert_called_once()
|
||||
|
||||
def test_shuts_down_when_initialize_raises(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test shutdown is called when initialization fails."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.initialize_and_login_mt5.side_effect = Mt5RuntimeError("boom")
|
||||
mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
|
||||
|
||||
with pytest.raises(Mt5RuntimeError, match="boom"), mt5_trading_session():
|
||||
pass
|
||||
|
||||
mock_client.shutdown.assert_called_once()
|
||||
|
||||
def test_shuts_down_when_body_raises(self, mocker: MockerFixture) -> None:
|
||||
"""Test shutdown is called when the context body raises."""
|
||||
mock_client = MagicMock()
|
||||
mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
|
||||
|
||||
body_error = "body error"
|
||||
with pytest.raises(RuntimeError, match=body_error), mt5_trading_session():
|
||||
raise RuntimeError(body_error)
|
||||
|
||||
mock_client.shutdown.assert_called_once()
|
||||
@@ -21,8 +21,10 @@ from mt5cli.utils import (
|
||||
TIMEFRAME_MAP,
|
||||
TIMEFRAME_TYPE,
|
||||
Dataset,
|
||||
IfExists,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
parse_datetime,
|
||||
parse_request,
|
||||
parse_tick_flags,
|
||||
@@ -130,6 +132,112 @@ class TestExportDataframe:
|
||||
export_dataframe(sample_df, tmp_path / "out.txt", "xml")
|
||||
|
||||
|
||||
class TestExportDataframeToSqlite:
|
||||
"""Tests for export_dataframe_to_sqlite."""
|
||||
|
||||
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
|
||||
"""Test append mode keeps prior rows in the SQLite table."""
|
||||
output = tmp_path / "append.db"
|
||||
first = pd.DataFrame({"id": [1], "value": ["a"]})
|
||||
second = pd.DataFrame({"id": [2], "value": ["b"]})
|
||||
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
|
||||
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
|
||||
with sqlite3.connect(output) as conn:
|
||||
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT id, value FROM items ORDER BY id",
|
||||
conn,
|
||||
)
|
||||
pd.testing.assert_frame_equal(
|
||||
result,
|
||||
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
|
||||
)
|
||||
|
||||
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
|
||||
"""Test deduplication keeps the latest ROWID for key columns."""
|
||||
output = tmp_path / "dedup.db"
|
||||
first = pd.DataFrame({
|
||||
"symbol": ["EURUSD", "EURUSD"],
|
||||
"time": ["2024-01-01", "2024-01-01"],
|
||||
"bid": [1.0, 1.1],
|
||||
})
|
||||
second = pd.DataFrame({
|
||||
"symbol": ["EURUSD"],
|
||||
"time": ["2024-01-01"],
|
||||
"bid": [1.2],
|
||||
})
|
||||
export_dataframe_to_sqlite(
|
||||
first,
|
||||
output,
|
||||
"ticks",
|
||||
if_exists=IfExists.REPLACE,
|
||||
deduplicate_on=("symbol", "time"),
|
||||
)
|
||||
export_dataframe_to_sqlite(
|
||||
second,
|
||||
output,
|
||||
"ticks",
|
||||
if_exists=IfExists.APPEND,
|
||||
deduplicate_on=("symbol", "time"),
|
||||
)
|
||||
with sqlite3.connect(output) as conn:
|
||||
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT symbol, time, bid FROM ticks",
|
||||
conn,
|
||||
)
|
||||
pd.testing.assert_frame_equal(
|
||||
result.reset_index(drop=True),
|
||||
pd.DataFrame({
|
||||
"symbol": ["EURUSD"],
|
||||
"time": ["2024-01-01"],
|
||||
"bid": [1.2],
|
||||
}),
|
||||
)
|
||||
|
||||
def test_default_if_exists_appends_without_dropping_rows(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test the default append mode keeps prior rows."""
|
||||
output = tmp_path / "default-append.db"
|
||||
first = pd.DataFrame({"id": [1], "value": ["a"]})
|
||||
second = pd.DataFrame({"id": [2], "value": ["b"]})
|
||||
export_dataframe_to_sqlite(first, output, "items")
|
||||
export_dataframe_to_sqlite(second, output, "items")
|
||||
with sqlite3.connect(output) as conn:
|
||||
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT id, value FROM items ORDER BY id",
|
||||
conn,
|
||||
)
|
||||
pd.testing.assert_frame_equal(
|
||||
result,
|
||||
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
|
||||
)
|
||||
|
||||
def test_writes_index_with_label(self, tmp_path: Path) -> None:
|
||||
"""Test optional index export with a custom label."""
|
||||
output = tmp_path / "index.db"
|
||||
frame = pd.DataFrame(
|
||||
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
|
||||
)
|
||||
export_dataframe_to_sqlite(
|
||||
frame,
|
||||
output,
|
||||
"margins",
|
||||
if_exists=IfExists.REPLACE,
|
||||
index=True,
|
||||
index_label="symbol",
|
||||
)
|
||||
with sqlite3.connect(output) as conn:
|
||||
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
|
||||
"SELECT symbol, value FROM margins",
|
||||
conn,
|
||||
)
|
||||
pd.testing.assert_frame_equal(
|
||||
result,
|
||||
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parse helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -487,7 +487,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "mt5cli"
|
||||
version = "0.4.2"
|
||||
version = "0.6.1"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
@@ -836,11 +836,11 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pygments"
|
||||
version = "2.19.2"
|
||||
version = "2.20.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user