Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Daichi Narushima
|
1ffac45d57 | ||
|
Daichi Narushima
|
d27da02f3f | ||
|
Daichi Narushima
|
4bc36d09d2 | ||
|
dceoy
|
3a126ced30 | ||
|
Daichi Narushima
|
80c3f3f65e | ||
|
Daichi Narushima
|
43f632bc40 | ||
|
dceoy
|
8028263b24 | ||
|
dceoy
|
63a8d67419 | ||
|
Daichi Narushima
|
93565681e1 | ||
|
Daichi Narushima
|
f435544f07 | ||
|
Daichi Narushima
|
668f38d8aa | ||
|
dceoy
|
8da5ee9242 | ||
|
Daichi Narushima
|
9dbb46fbb1 | ||
|
Daichi Narushima
|
dfe80ce500 | ||
|
Daichi Narushima
|
15bfd17db3 | ||
|
Daichi Narushima
|
37eef16e99 | ||
|
Daichi Narushima
|
96c75f7852 | ||
|
Daichi Narushima
|
292fac899a |
@@ -10,7 +10,7 @@ uv run pyright .
|
||||
uv run pytest
|
||||
|
||||
# Markdown
|
||||
npx -y prettier --write './**/*.md'
|
||||
npx -y prettier --write './**/*.{md,json}'
|
||||
|
||||
# GitHub Actions
|
||||
case "${OSTYPE}" in
|
||||
|
||||
@@ -29,25 +29,29 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
|
||||
pip install -U mt5cli MetaTrader5
|
||||
```
|
||||
|
||||
Parquet export is not included by default. To enable it, install the `parquet` extra:
|
||||
|
||||
```bash
|
||||
pip install -U "mt5cli[parquet]" MetaTrader5
|
||||
```
|
||||
|
||||
## Python API (downstream packages)
|
||||
|
||||
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives. `Mt5CliClient` remains available as a backward-compatible alias.
|
||||
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives.
|
||||
|
||||
```python
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import (
|
||||
DataKind,
|
||||
Dataset,
|
||||
MT5Client,
|
||||
build_config,
|
||||
collect_history,
|
||||
export_dataframe,
|
||||
mt5_session,
|
||||
normalize_dataframe,
|
||||
update_history_with_config,
|
||||
)
|
||||
from mt5cli.schemas import DataKind, normalize_dataframe
|
||||
from mt5cli.utils import Dataset, export_dataframe
|
||||
|
||||
# Persistent session for multiple calls
|
||||
with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
|
||||
@@ -83,7 +87,7 @@ update_history_with_config(
|
||||
)
|
||||
```
|
||||
|
||||
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
|
||||
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Export and storage helpers are in `mt5cli.utils` (`Dataset`, `export_dataframe`) and `mt5cli.history`.
|
||||
|
||||
`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly.
|
||||
|
||||
@@ -92,16 +96,21 @@ Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `norma
|
||||
Trading applications can depend on `mt5cli` imports only; terminal path,
|
||||
credentials, server, and timeout are forwarded to `pdmt5.Mt5Config`, numeric
|
||||
login strings are coerced to integers, and empty login strings are treated as
|
||||
unset.
|
||||
unset. Pass `allow_whole_dollar_env=True` to expand `${ENV_VAR}` and bare
|
||||
`$ENV_NAME` placeholders in connection string parameters before coercion.
|
||||
|
||||
```python
|
||||
from mt5cli import (
|
||||
build_config,
|
||||
calculate_spread_ratio,
|
||||
create_trading_client,
|
||||
get_account_snapshot,
|
||||
mt5_trading_session,
|
||||
)
|
||||
|
||||
# Login from environment — numeric string is coerced to int automatically
|
||||
config = build_config(login="$MT5_LOGIN", allow_whole_dollar_env=True)
|
||||
|
||||
with mt5_trading_session(
|
||||
path=r"C:\Program Files\MetaTrader 5\terminal64.exe",
|
||||
login="12345",
|
||||
@@ -148,39 +157,44 @@ python -m mt5cli -o account.csv account-info
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a recent trailing window |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a recent trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
|
||||
| Command | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `rates-from` | Export rates from a start date |
|
||||
| `rates-from-pos` | Export rates from a start position |
|
||||
| `latest-rates` | Export latest rates from a start position |
|
||||
| `rates-range` | Export rates for a date range |
|
||||
| `ticks-from` | Export ticks from a start date |
|
||||
| `ticks-range` | Export ticks for a date range |
|
||||
| `ticks-recent` | Export ticks from a recent trailing window |
|
||||
| `account-info` | Export account information |
|
||||
| `terminal-info` | Export terminal information |
|
||||
| `version` | Export MetaTrader 5 version information |
|
||||
| `last-error` | Export the last error information |
|
||||
| `symbols` | Export symbol list |
|
||||
| `symbol-info` | Export symbol details |
|
||||
| `symbol-info-tick` | Export the last tick for a symbol |
|
||||
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a recent trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a raw trade request to the trade server (`--yes` required; expert path) |
|
||||
| `close-positions` | Close open positions by `--symbol` or `--ticket` (`--yes` required for live; `--dry-run` available) |
|
||||
| `collect-history` | Collect rates, history-orders, and history-deals for one or more symbols into a single SQLite database (ticks opt-in via `--dataset ticks`) |
|
||||
| `grafana-schema` | Create or refresh Grafana-ready views and indexes in an existing SQLite database (idempotent, no MT5 connection) |
|
||||
| `snapshot` | Snapshot current account, position, order, and terminal state into SQLite for live Grafana dashboards |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
`close-positions` is the safer high-level alternative that builds correct close
|
||||
requests automatically. At least one `--symbol` or `--ticket` must be provided.
|
||||
|
||||
### `collect-history`
|
||||
|
||||
Collect several historical datasets per symbol into one SQLite database in a single MT5 session. Pick datasets with repeatable `--dataset` (default: all four), choose conflict behavior with `--if-exists append|replace|fail` (default: `fail`), and optionally derive `cash_events` / `positions_reconstructed` views from `history_deals` via `--with-views`.
|
||||
Collect several historical datasets per symbol into one SQLite database in a single MT5 session. Pick datasets with repeatable `--dataset` (default: `rates`, `history-orders`, `history-deals`; add `--dataset ticks` when tick-level history is required — tick data can grow the SQLite database quickly), choose conflict behavior with `--if-exists append|replace|fail` (default: `fail`), and optionally derive `cash_events` / `positions_reconstructed` views from `history_deals` via `--with-views`.
|
||||
|
||||
```bash
|
||||
mt5cli -o history.db collect-history \
|
||||
@@ -192,13 +206,112 @@ mt5cli -o history.db collect-history \
|
||||
|
||||
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
|
||||
|
||||
### Grafana-ready SQLite dashboards
|
||||
|
||||
mt5cli can prepare a SQLite database for use as a Grafana datasource (via the [SQLite plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/) or similar). Most `grafana_*` views expose an integer epoch-second `time` column for use in Grafana time-series panels. Two views (`grafana_realized_pnl`, `grafana_trade_stats`) are static symbol-level summaries with no `time` column — use them in table or stat panels.
|
||||
|
||||
#### Prepare the schema (idempotent, no MT5 connection needed)
|
||||
|
||||
```bash
|
||||
mt5cli -o history.db grafana-schema
|
||||
```
|
||||
|
||||
This creates snapshot tables (`account_snapshots`, `position_snapshots`, `order_snapshots`, `terminal_snapshots`, `snapshot_runs`) and all `grafana_*` views and indexes in the SQLite database. Safe to run repeatedly — all operations are idempotent.
|
||||
|
||||
#### Snapshot current account state
|
||||
|
||||
```bash
|
||||
mt5cli -o history.db snapshot \
|
||||
--symbol JP225 --symbol HK50 --symbol NL25 \
|
||||
--with-account --with-positions --with-orders --with-terminal \
|
||||
--with-grafana-schema
|
||||
```
|
||||
|
||||
Appends one timestamped row per data type. Never places orders or modifies trading state. Run periodically (e.g. from a cron job or a loop) to build a time-series account history.
|
||||
|
||||
#### SDK usage
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5DataClient, Mt5Config
|
||||
from mt5cli import update_observability, update_observability_with_config
|
||||
|
||||
# Reuse an already-connected client
|
||||
client = Mt5DataClient(config=Mt5Config(login=12345))
|
||||
client.initialize_and_login_mt5()
|
||||
try:
|
||||
update_observability(
|
||||
client=client,
|
||||
output="history.db",
|
||||
symbols=["EURUSD", "GBPUSD"], # optional position/order filter
|
||||
include_account=True,
|
||||
include_positions=True,
|
||||
include_orders=True,
|
||||
include_terminal=True,
|
||||
with_grafana_schema=True,
|
||||
)
|
||||
finally:
|
||||
client.shutdown()
|
||||
|
||||
# Standalone wrapper that opens/closes MT5 automatically
|
||||
update_observability_with_config(
|
||||
output="history.db",
|
||||
config=Mt5Config(login=12345),
|
||||
)
|
||||
```
|
||||
|
||||
#### Available Grafana views
|
||||
|
||||
**Time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
|
||||
|
||||
| View | Source | Description |
|
||||
| ---------------------------- | -------------------- | ---------------------------------------------------------- |
|
||||
| `grafana_rates` | `rates` | OHLCV bars with integer epoch `time` |
|
||||
| `grafana_ticks` | `ticks` | Tick data with integer epoch `time` |
|
||||
| `grafana_history_deals` | `history_deals` | All deals with epoch `time` |
|
||||
| `grafana_history_orders` | `history_orders` | All historical orders; adds epoch `time` from `time_setup` |
|
||||
| `grafana_trade_deals` | `history_deals` | Trade deals only (`type IN (0,1)`) |
|
||||
| `grafana_cash_events` | `history_deals` | Non-trade deals (deposits, dividends, etc.) |
|
||||
| `grafana_symbol_pnl` | `history_deals` | Per-close-deal profit/loss per symbol |
|
||||
| `grafana_account_snapshots` | `account_snapshots` | Account balance/equity/margin time series |
|
||||
| `grafana_position_snapshots` | `position_snapshots` | Open position snapshots over time |
|
||||
| `grafana_order_snapshots` | `order_snapshots` | Active order snapshots over time |
|
||||
| `grafana_terminal_snapshots` | `terminal_snapshots` | Terminal connectivity snapshots |
|
||||
|
||||
**Static summary views** (no `time` column; use in table or stat panels, not time-series):
|
||||
|
||||
| View | Source | Description |
|
||||
| ---------------------- | --------------- | ------------------------------------- |
|
||||
| `grafana_realized_pnl` | `history_deals` | Cumulative realized PnL per symbol |
|
||||
| `grafana_trade_stats` | `history_deals` | Win/loss counts and profit per symbol |
|
||||
|
||||
#### Example Grafana queries
|
||||
|
||||
```sql
|
||||
-- Equity curve over time
|
||||
SELECT time, equity FROM grafana_account_snapshots ORDER BY time;
|
||||
|
||||
-- Rolling balance by account login
|
||||
SELECT time, login, balance FROM grafana_account_snapshots
|
||||
WHERE login = $login ORDER BY time;
|
||||
|
||||
-- Open positions at latest successful snapshot
|
||||
SELECT symbol, volume, profit FROM grafana_position_snapshots
|
||||
WHERE run_id = (SELECT MAX(run_id) FROM snapshot_runs WHERE status = 'ok');
|
||||
|
||||
-- Realized PnL by symbol
|
||||
SELECT symbol, total_profit FROM grafana_trade_stats ORDER BY total_profit DESC;
|
||||
```
|
||||
|
||||
> **Note**: OpenTelemetry integration is intentionally not part of this release and is tracked separately.
|
||||
|
||||
### Incremental history SDK
|
||||
|
||||
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
|
||||
|
||||
```python
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
from mt5cli import Dataset, update_history, update_history_with_config
|
||||
from mt5cli import update_history, update_history_with_config
|
||||
from mt5cli.utils import Dataset
|
||||
|
||||
# Reuse an already-connected pdmt5 client (does not open/close MT5)
|
||||
client = Mt5DataClient(config=Mt5Config(login=12345))
|
||||
@@ -233,7 +346,7 @@ update_history_with_config(
|
||||
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
|
||||
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
|
||||
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
|
||||
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
|
||||
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only recoverable MT5 errors and re-raises once `retry_count` is exhausted.
|
||||
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
|
||||
|
||||
```python
|
||||
@@ -248,9 +361,9 @@ rates = collect_latest_closed_rates_by_granularity(
|
||||
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
|
||||
```
|
||||
|
||||
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
|
||||
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code. For config dicts or nested structures loaded from YAML/TOML, use `substitute_mapping_values(data, keys={"login", "password"})` to expand placeholders only for caller-specified keys — key names are never hard-coded in mt5cli.
|
||||
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`, `sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability errors for history API methods without advancing the throttle (other `AttributeError` / `TypeError` values always propagate). Pass `update_backend` to inject a custom history update callable (same keyword arguments as `update_history`) instead of monkey-patching `mt5cli.sdk.update_history`.
|
||||
- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable `pdmt5.Mt5TradingClient` that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. The read-only `mt5_session()` / `Mt5CliClient` SDK is unchanged.
|
||||
- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable client that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. Keep read-only collection on `mt5_session()` / `MT5Client`.
|
||||
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
|
||||
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `MT5Client` that shuts down on exit.
|
||||
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
|
||||
@@ -317,7 +430,7 @@ finally:
|
||||
client.shutdown()
|
||||
```
|
||||
|
||||
Read-only collectors can keep using `mt5_session()` and `MT5Client` (or the `Mt5CliClient` alias) without changes.
|
||||
Read-only collectors can keep using `mt5_session()` and `MT5Client`.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
+5
-3
@@ -182,12 +182,14 @@ targets without hard-coding view names:
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import (
|
||||
load_rate_data,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
resolve_rate_table_name,
|
||||
)
|
||||
from mt5cli.history import resolve_rate_view_name
|
||||
from mt5cli.history import (
|
||||
load_rate_data,
|
||||
resolve_rate_table_name,
|
||||
resolve_rate_view_name,
|
||||
)
|
||||
|
||||
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
|
||||
rates = load_rate_data(Path("history.db"), view, count=1000)
|
||||
|
||||
+3
-5
@@ -13,7 +13,6 @@ responsibilities.
|
||||
| [Public API Contract](public-contract.md) | Stable downstream SDK exports, CLI boundary, and out-of-scope items |
|
||||
| [Client](client.md) | `MT5Client` session abstraction for data access and order primitives |
|
||||
| [Schemas](schemas.md) | Canonical DataFrame contracts and normalization helpers |
|
||||
| [Storage](storage.md) | CSV/JSON/Parquet/SQLite export and history collection helpers |
|
||||
| [Converters](converters.md) | Symbol, timeframe, timezone, and date-range utilities |
|
||||
| [Exceptions](exceptions.md) | Stable mt5cli exception types and MT5 error normalization |
|
||||
| [SDK](sdk.md) | Module-level fetch helpers, multi-account collectors, incremental history |
|
||||
@@ -30,15 +29,14 @@ flowchart TD
|
||||
CLI["mt5cli CLI"] --> Client
|
||||
Client --> SDK["sdk / pdmt5"]
|
||||
Client --> Schemas["schemas"]
|
||||
Storage["storage"] --> History["history SQLite"]
|
||||
Storage --> Utils["utils export"]
|
||||
History["history SQLite"] --> Utils["utils export"]
|
||||
SDK --> PDMT5["pdmt5.Mt5DataClient"]
|
||||
```
|
||||
|
||||
Downstream packages should depend on the package root exports documented in the
|
||||
[Public API Contract](public-contract.md) (`MT5Client`,
|
||||
`DataKind`, `normalize_dataframe`, `collect_history`, `load_rate_data`,
|
||||
`resolve_rate_view_name`, etc.) rather than private modules.
|
||||
`collect_history`, `load_rate_series_from_sqlite`, etc.) rather than private
|
||||
modules. Lower-level helpers are accessible directly from their owning modules.
|
||||
|
||||
`MT5Client.order_send()` is a live execution primitive that can place real trades. mt5cli exposes minimal execution helpers only; strategy logic, signals, backtests, and optimization remain out of scope and must be implemented downstream with explicit execution gating.
|
||||
|
||||
|
||||
+164
-91
@@ -1,61 +1,53 @@
|
||||
# Public API Contract
|
||||
|
||||
mt5cli is the generic MT5 data and execution infrastructure layer for downstream
|
||||
Python applications. The intended dependency direction is:
|
||||
mt5cli is the canonical operational trading SDK and CLI/batch layer over pdmt5.
|
||||
The intended dependency direction is:
|
||||
|
||||
```text
|
||||
downstream app -> mt5cli -> pdmt5 -> MetaTrader 5
|
||||
```
|
||||
|
||||
## Responsibility boundary
|
||||
|
||||
| Layer | Owns |
|
||||
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **pdmt5** | MT5 core wrapper; DataFrame/dict conversion; canonical MT5 constants and parsers; direct low-level order primitives |
|
||||
| **mt5cli** | CLI/batch workflows; SQLite history collection; normalized datasets; closed-bar helpers; small downstream operational SDK; generic broker-facing margin/volume/order orchestration |
|
||||
| **downstream** | Strategy logic; signals; risk policy; backtesting; optimization; YAML/application semantics |
|
||||
|
||||
Downstream code should import raw pdmt5 types and constants (such as
|
||||
`Mt5Config`, `Mt5RuntimeError`, `TIMEFRAME_MAP`, `COPY_TICKS_MAP`) directly
|
||||
from `pdmt5` when needed. mt5cli does not serve as a pass-through compatibility
|
||||
namespace for pdmt5. mt5cli's trading helpers type their client parameter against
|
||||
an internal protocol backed by `pdmt5.Mt5DataClient`; `Mt5TradingClient` is no
|
||||
longer required. `Mt5TradingError` is conditionally imported where still present
|
||||
in pdmt5, but mt5cli raises `Mt5OperationError` for all trading-related failures.
|
||||
|
||||
Note: the former `mt5cli` re-export `TICK_FLAG_MAP` corresponds to `COPY_TICKS_MAP`
|
||||
in pdmt5 — the name changed, it was not simply moved.
|
||||
|
||||
Downstream packages should import from the package root (`from mt5cli import
|
||||
...`) and treat the symbols listed below as the stable SDK contract. CLI
|
||||
commands mirror the same behavior but are not importable Python APIs.
|
||||
...`). The contract set `STABLE_SDK_EXPORTS` in `mt5cli.contract` enumerates
|
||||
every package-root symbol. Lower-level helpers (schema utilities, export
|
||||
functions, parser helpers, low-level MT5 wrappers) are available directly from
|
||||
their owning modules (`mt5cli.schemas`, `mt5cli.utils`, `mt5cli.converters`,
|
||||
`mt5cli.sdk`, etc.) and are not part of the root SDK surface.
|
||||
|
||||
## Stable downstream SDK API
|
||||
|
||||
These names are exported from `mt5cli` and covered by the contract in
|
||||
`mt5cli.STABLE_SDK_EXPORTS` (defined in `mt5cli.contract`). Prefer `MT5Client` over the legacy `Mt5CliClient`
|
||||
alias for new code.
|
||||
These names are exported from `mt5cli` and enumerated in
|
||||
`mt5cli.STABLE_SDK_EXPORTS` (defined in `mt5cli.contract`).
|
||||
|
||||
### Session lifecycle and configuration
|
||||
|
||||
| Symbol | Role |
|
||||
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
||||
| `MT5Client`, `Mt5CliClient` | Read-only data client with optional `order_check` / `order_send` |
|
||||
| `build_config` | Build `pdmt5.Mt5Config` from connection fields |
|
||||
| `mt5_session` | Context manager: initialize, login, yield client, shutdown |
|
||||
| `create_trading_client`, `mt5_trading_session` | Trading-capable `pdmt5.Mt5TradingClient` lifecycle |
|
||||
| `AccountSpec` | Generic account group: symbols plus optional credentials |
|
||||
| `resolve_account_spec`, `resolve_account_specs` | Merge overrides and expand `${ENV_VAR}` placeholders; opt-in `allow_whole_dollar_env` for bare `$NAME` |
|
||||
| `substitute_env_placeholders` | Replace `${NAME}` substrings from the environment; opt-in `allow_whole_dollar_env` for whole-value `$NAME` |
|
||||
|
||||
Credential resolution is generic: any environment variable name may appear inside
|
||||
`${...}`. mt5cli does not hard-code application-specific keys such as
|
||||
`mt5_login` or `mt5_exe`.
|
||||
|
||||
Pass `allow_whole_dollar_env=True` to `substitute_env_placeholders()`,
|
||||
`resolve_account_spec()`, `resolve_account_specs()`, and `build_config()` to
|
||||
additionally expand strings whose entire value is a bare `$ENV_NAME` identifier.
|
||||
Partial strings such as `"plan$pass"`, `"abc$ENV"`, or `"$ENV-suffix"` are
|
||||
**never** expanded — only an exact `$IDENTIFIER` whole-string match qualifies.
|
||||
Default is `False` to preserve backward compatibility.
|
||||
|
||||
### Read-only MT5 data access
|
||||
|
||||
Module-level helpers open a transient connection per call. Prefer `mt5_session`
|
||||
or `MT5Client` when making many requests in one process.
|
||||
|
||||
| Area | Symbols |
|
||||
| -------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| Rates | `copy_rates_from`, `copy_rates_from_pos`, `copy_rates_range`, `latest_rates`, `collect_latest_rates` |
|
||||
| Ticks | `copy_ticks_from`, `copy_ticks_range`, `recent_ticks` |
|
||||
| Account / terminal | `account_info`, `terminal_info`, `mt5_version`, `last_error`, `mt5_summary`, `mt5_summary_as_df` |
|
||||
| Symbols / market | `symbols`, `symbol_info`, `symbol_info_tick`, `market_book`, `minimum_margins` |
|
||||
| Trading state (read) | `orders`, `positions`, `history_orders`, `history_deals`, `recent_history_deals` |
|
||||
|
||||
Use `mt5_version` for MetaTrader 5 terminal version data. The name `version` at
|
||||
the package root refers to `importlib.metadata.version` (package metadata), not
|
||||
the MT5 SDK helper.
|
||||
| Symbol | Role |
|
||||
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MT5Client` | Read-only data client with optional `order_check` / `order_send` |
|
||||
| `build_config` | Build `pdmt5.Mt5Config` from connection fields; `login` accepts `int \| str \| None` — numeric strings are coerced to `int`, blank strings are treated as unset, and `${ENV_VAR}` / `$ENV_NAME` placeholders in string parameters are expanded when `allow_whole_dollar_env=True` |
|
||||
| `mt5_session` | Context manager: initialize, login, yield client, shutdown |
|
||||
| `create_trading_client`, `mt5_trading_session` | Trading-capable MT5 client lifecycle; returns a client supporting order execution and account management |
|
||||
| `AccountSpec` | Generic account group: symbols plus optional credentials |
|
||||
| `resolve_account_spec`, `resolve_account_specs` | Merge overrides and expand `${ENV_VAR}` placeholders; opt-in `allow_whole_dollar_env` for bare `$NAME` |
|
||||
|
||||
### Closed-bar rate helpers
|
||||
|
||||
@@ -67,29 +59,21 @@ timestamp normalization in downstream apps.
|
||||
| ------------------------------------------------ | ------------------------------------------------------------------------------- |
|
||||
| `drop_forming_rate_bar` | Remove the last row from chronologically ordered rate data |
|
||||
| `fetch_latest_closed_rates` | Single connected client: fetch `count + 1`, drop forming bar |
|
||||
| `fetch_latest_closed_rates_for_trading_client` | Closed bars from an active `Mt5TradingClient` session; returns RangeIndex |
|
||||
| `fetch_latest_closed_rates_for_trading_client` | Closed bars from an active trading client session; returns RangeIndex |
|
||||
| `fetch_latest_closed_rates_indexed` | Same as above but returns a UTC `DatetimeIndex` named `"time"` (no time column) |
|
||||
| `collect_latest_closed_rates_for_accounts` | Multi-account closed bars with optional retry wrapper |
|
||||
| `collect_latest_closed_rates_by_granularity` | Same data keyed by `(symbol, granularity_name)` |
|
||||
| `collect_latest_rates_for_accounts` | Latest bars including the forming bar when `start_pos=0` |
|
||||
| `collect_latest_rates_for_accounts_with_retries` | Bounded exponential backoff for transient MT5 errors |
|
||||
|
||||
### SQLite history collection and rate loading
|
||||
|
||||
| Symbol | Role |
|
||||
| ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| `collect_history` | One-shot date-range export into SQLite |
|
||||
| `update_history`, `update_history_with_config` | Incremental append from `MAX(time)` cursors |
|
||||
| `ThrottledHistoryUpdater` | Minimum interval between successful incremental updates; optional `update_backend` injection |
|
||||
| `resolve_history_datasets`, `resolve_history_timeframes`, `resolve_history_tick_flags` | History pipeline configuration |
|
||||
| `build_rate_view_name`, `resolve_rate_table_name`, `resolve_rate_view_name`, `resolve_rate_view_names`, `resolve_rate_tables` | Map symbols/timeframes to mt5cli-managed table or view names |
|
||||
| `RateTarget`, `build_rate_targets` | Neutral `(symbol, timeframe)` series descriptors |
|
||||
| `load_rate_data`, `load_rate_data_from_connection` | Load one table/view into a time-indexed DataFrame |
|
||||
| `load_rate_series_from_sqlite`, `load_rate_series_by_granularity` | Load one or many series; fail clearly when managed views are missing |
|
||||
|
||||
Pass `require_existing=True` to rate view resolution helpers when downstream
|
||||
code must fail instead of receiving a best-guess view name. Multi-series loaders
|
||||
require existing managed `rate_*__*` views unless `explicit_tables` is supplied.
|
||||
| Symbol | Role |
|
||||
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| `collect_history` | One-shot date-range export into SQLite |
|
||||
| `update_history`, `update_history_with_config` | Incremental append from `MAX(time)` cursors |
|
||||
| `ThrottledHistoryUpdater` | Minimum interval between successful incremental updates; optional `update_backend` injection |
|
||||
| `RateTarget`, `build_rate_targets` | Neutral `(symbol, timeframe)` series descriptors |
|
||||
| `load_rate_series_from_sqlite`, `load_rate_series_by_granularity` | Load one or many series; fail clearly when managed views are missing |
|
||||
|
||||
See [History Collection (SQLite)](history.md) for schema, view naming, and ER
|
||||
diagrams.
|
||||
@@ -99,25 +83,38 @@ diagrams.
|
||||
These helpers implement broker-facing calculations only. They do not encode
|
||||
strategy entries, exits, Kelly sizing, or signal logic.
|
||||
|
||||
| Symbol | Role |
|
||||
| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
|
||||
| `get_account_snapshot`, `get_symbol_snapshot`, `get_tick_snapshot`, `get_positions_frame` | Normalized account/symbol/tick/position views |
|
||||
| `detect_position_side` | Net long / short / flat from open positions |
|
||||
| `calculate_spread_ratio` | Relative bid-ask spread |
|
||||
| `calculate_margin_and_volume`, `calculate_volume_by_margin`, `calculate_new_position_margin_ratio` | Margin budget and volume sizing |
|
||||
| `normalize_order_volume`, `estimate_order_margin`, `calculate_positions_margin` | Broker volume normalization and margin totals |
|
||||
| `calculate_positions_margin_by_symbol` | Per-symbol margin map (resilient, first-seen order) |
|
||||
| `calculate_positions_margin_safe` | Summed total margin across symbols (failed symbols skipped) |
|
||||
| `determine_order_limits` | SL/TP price levels from ratios |
|
||||
| `ensure_symbol_selected` | Select/verify Market Watch visibility |
|
||||
| `place_market_order`, `close_open_positions`, `update_sltp_for_open_positions` | Order execution helpers (`dry_run` supported) |
|
||||
| `MarginVolume`, `OrderLimits`, `OrderExecutionResult` | Typed return contracts for order helpers |
|
||||
| `OrderSide`, `OrderFillingMode`, `OrderTimeMode`, `PositionSide`, `ExecutionStatus` | Typed enums for order helpers |
|
||||
| Symbol | Role |
|
||||
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
|
||||
| `get_account_snapshot`, `get_symbol_snapshot`, `get_tick_snapshot`, `get_positions_frame` | Normalized account/symbol/tick/position views |
|
||||
| `extract_tick_price` | Positive finite bid/ask extraction from tick mappings |
|
||||
| `detect_position_side` | Net long / short / flat from open positions |
|
||||
| `calculate_spread_ratio` | Relative bid-ask spread |
|
||||
| `calculate_margin_and_volume`, `calculate_volume_by_margin`, `calculate_new_position_margin_ratio` | Margin budget and volume sizing |
|
||||
| `normalize_order_volume`, `estimate_order_margin`, `calculate_positions_margin` | Broker volume normalization and margin totals |
|
||||
| `calculate_positions_margin_by_symbol` | Per-symbol margin map (resilient, first-seen order) |
|
||||
| `calculate_positions_margin_safe` | Summed total margin across symbols (failed symbols skipped) |
|
||||
| `calculate_projected_margin_ratio` | Estimated symbol-scoped margin/equity after optional new exposure |
|
||||
| `calculate_account_projected_margin_ratio` | Account snapshot margin/equity after optional new exposure |
|
||||
| `calculate_symbol_group_margin_ratio` | Estimated symbol-group margin/equity with optional exposure |
|
||||
| `determine_order_limits` | SL/TP price levels from ratios |
|
||||
| `calculate_trailing_stop_updates` | Per-ticket generic trailing stop-loss update plan |
|
||||
| `ensure_symbol_selected` | Select/verify Market Watch visibility |
|
||||
| `place_market_order`, `close_open_positions`, `update_sltp_for_open_positions`, `update_trailing_stop_loss_for_open_positions` | Order execution helpers (`dry_run` supported) |
|
||||
| `MarginVolume`, `OrderLimits`, `OrderExecutionResult` | Typed return contracts for order helpers |
|
||||
| `OrderSide`, `OrderFillingMode`, `OrderTimeMode`, `PositionSide`, `ExecutionStatus` | Typed enums for order helpers |
|
||||
| `ProjectionMode` | Literal type for `calculate_symbol_group_margin_ratio` projection |
|
||||
|
||||
`calculate_symbol_group_margin_ratio` accepts an optional `projection_mode`
|
||||
parameter (`"add"` by default). Pass `projection_mode="replace_symbol"` to
|
||||
subtract current exposure for `new_symbol` before adding the candidate margin —
|
||||
useful for reversal-style projections. mt5cli only calculates broker-facing
|
||||
exposure; downstream applications own thresholds, risk guard actions, and
|
||||
strategy policy.
|
||||
|
||||
`MT5Client.order_send()` and CLI `order-send --yes` are live execution paths.
|
||||
|
||||
Order helpers validate broker stop-level distance in `determine_order_limits()` and
|
||||
raise `Mt5TradingError` when computed SL/TP prices are too close to the entry
|
||||
raise `Mt5OperationError` when computed SL/TP prices are too close to the entry
|
||||
quote. Validation uses `trade_stops_level * point` from the current quote and
|
||||
symbol metadata as a pre-check only; it does not guarantee live order acceptance
|
||||
after price movement and does not inspect `trade_freeze_level`. Live
|
||||
@@ -127,21 +124,83 @@ sending requests. Failed, malformed, or unknown broker retcodes are fail-closed
|
||||
and returned as `status="failed"` with normalized `request` / `response` details;
|
||||
`dry_run=True` never calls `ensure_symbol_selected()` or `order_send()`.
|
||||
|
||||
### Errors and MT5 type re-exports
|
||||
### Grafana observability (SQLite read model)
|
||||
|
||||
| Symbol | Role |
|
||||
| ------------------------------------------------------------------------------------ | ----------------------------------------------- |
|
||||
| `Mt5CliError`, `Mt5ConnectionError`, `Mt5OperationError`, `Mt5SchemaError` | Stable mt5cli exception types |
|
||||
| `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` | Error normalization and retry classification |
|
||||
| `Mt5Config`, `Mt5RuntimeError`, `Mt5TradingClient`, `Mt5TradingError` | Re-exported pdmt5 types for adapter convenience |
|
||||
These helpers prepare a SQLite database as a Grafana datasource. All DDL is
|
||||
idempotent (`CREATE TABLE IF NOT EXISTS`, `DROP VIEW IF EXISTS` + `CREATE
|
||||
VIEW`, `CREATE INDEX IF NOT EXISTS`). Missing source tables are skipped with a
|
||||
warning rather than raising an error.
|
||||
|
||||
### Additional public exports (secondary)
|
||||
| Symbol | Role |
|
||||
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `update_observability` | Append one timestamped snapshot row per data type; accepts an already-connected `Mt5DataClient` |
|
||||
| `update_observability_with_config` | Standalone wrapper: opens/closes MT5 connection automatically around `update_observability` |
|
||||
|
||||
The package root also exports schema, storage, and parsing helpers (for example
|
||||
`DataKind`, `Dataset`, `normalize_dataframe`, `export_dataframe`,
|
||||
`parse_timeframe`, `TIMEFRAME_MAP`). These are public but oriented toward export
|
||||
pipelines and advanced integration. Prefer the stable symbols above for core
|
||||
infrastructure.
|
||||
Both functions write to the SQLite path given by `output=`. The optional
|
||||
`symbols` parameter filters `positions_get` / `orders_get` by symbol.
|
||||
`with_grafana_schema=False` (default) skips Grafana view/index setup; run
|
||||
`grafana-schema` once to set up the schema, then call `snapshot` repeatedly
|
||||
without this flag.
|
||||
|
||||
**Snapshot tables** (created by `create_snapshot_tables` in `mt5cli.grafana`):
|
||||
|
||||
| Table | Content |
|
||||
| -------------------- | ----------------------------------------- |
|
||||
| `account_snapshots` | Balance, equity, margin, free-margin, P&L |
|
||||
| `position_snapshots` | Open positions: symbol, volume, profit, … |
|
||||
| `order_snapshots` | Active orders: symbol, type, price, … |
|
||||
| `terminal_snapshots` | Terminal connectivity and build info |
|
||||
| `snapshot_runs` | Per-run status (`ok` / `error`) timestamp |
|
||||
|
||||
**Grafana time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
|
||||
|
||||
| View | Source |
|
||||
| ---------------------------- | -------------------------------- |
|
||||
| `grafana_rates` | `rates` table |
|
||||
| `grafana_ticks` | `ticks` table |
|
||||
| `grafana_history_deals` | `history_deals` |
|
||||
| `grafana_history_orders` | `history_orders` |
|
||||
| `grafana_trade_deals` | `history_deals` trade types only |
|
||||
| `grafana_cash_events` | `history_deals` non-trade events |
|
||||
| `grafana_symbol_pnl` | Per-close-deal P&L per symbol |
|
||||
| `grafana_account_snapshots` | `account_snapshots` |
|
||||
| `grafana_position_snapshots` | `position_snapshots` |
|
||||
| `grafana_order_snapshots` | `order_snapshots` |
|
||||
| `grafana_terminal_snapshots` | `terminal_snapshots` |
|
||||
|
||||
**Grafana static summary views** (no `time` column; use for table/stat panels, not time-series):
|
||||
|
||||
| View | Source |
|
||||
| ---------------------- | ------------------------------------- |
|
||||
| `grafana_realized_pnl` | Cumulative realized PnL per symbol |
|
||||
| `grafana_trade_stats` | Win/loss counts and profit per symbol |
|
||||
|
||||
Lower-level helpers (`ensure_grafana_schema`, `create_grafana_views`,
|
||||
`create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`,
|
||||
`insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`,
|
||||
`insert_terminal_snapshot`, `record_snapshot_run`) are available directly from
|
||||
`mt5cli.grafana` and are not part of the package-root stable surface.
|
||||
|
||||
### Errors
|
||||
|
||||
| Symbol | Role |
|
||||
| -------------------------------------------------------------------------- | ----------------------------- |
|
||||
| `Mt5CliError`, `Mt5ConnectionError`, `Mt5OperationError`, `Mt5SchemaError` | Stable mt5cli exception types |
|
||||
|
||||
## Module-scoped helpers
|
||||
|
||||
Lower-level helpers are available from their owning modules and are not part
|
||||
of the package-root stable surface. Import them directly when needed:
|
||||
|
||||
| Module | Examples |
|
||||
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `mt5cli.grafana` | `ensure_grafana_schema`, `create_grafana_views`, `create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`, `insert_account_snapshot`, `record_snapshot_run` |
|
||||
| `mt5cli.history` | `resolve_rate_view_name`, `resolve_rate_tables`, `load_rate_data`, `build_rate_view_name` |
|
||||
| `mt5cli.sdk` | `copy_rates_from`, `copy_ticks_from`, `account_info`, `symbols`, `mt5_summary`, `latest_rates` |
|
||||
| `mt5cli.schemas` | `DataKind`, `normalize_dataframe`, `validate_schema`, `DEDUP_KEYS` |
|
||||
| `mt5cli.utils` | `Dataset`, `IfExists`, `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
|
||||
| `mt5cli.converters` | `normalize_symbol`, `ensure_utc`, `parse_date_range`, `granularity_name` |
|
||||
| `mt5cli.exceptions` | `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` |
|
||||
|
||||
## CLI commands
|
||||
|
||||
@@ -154,7 +213,21 @@ The Typer application in `mt5cli.cli` exposes file-export commands documented in
|
||||
- Delegate to the same Python APIs described here; they are not duplicated
|
||||
business logic.
|
||||
|
||||
`order-send` requires `--yes` before placing live trades.
|
||||
`grafana-schema` initializes Grafana views, indexes, and snapshot tables in the
|
||||
target SQLite database without connecting to MT5. It is idempotent and safe to
|
||||
run repeatedly.
|
||||
|
||||
`snapshot` appends one timestamped row per enabled data type
|
||||
(`--with-account`, `--with-positions`, `--with-orders`, `--with-terminal`) and
|
||||
never places orders or modifies trading state. Both commands require
|
||||
`-o/--output` to point at a `.db` / SQLite file.
|
||||
|
||||
`order-send` is the expert raw-request path; it requires `--yes` and a fully
|
||||
constructed request payload. `close-positions` is the safer high-level helper
|
||||
that closes open positions by `--symbol` or `--ticket` using
|
||||
`close_open_positions()`. Both `order-send --yes` and `close-positions --yes`
|
||||
are live execution paths. `close-positions --dry-run` previews close orders
|
||||
without placing them and does not require `--yes`.
|
||||
|
||||
## Internal helpers (not stable)
|
||||
|
||||
@@ -191,6 +264,6 @@ their own adapter layer.
|
||||
## Contract verification
|
||||
|
||||
`tests/test_contracts.py` asserts that every name in `STABLE_SDK_EXPORTS` is
|
||||
importable from `mt5cli`, documents key closed-bar, rate-view, SQLite loading,
|
||||
account-resolution, and trading-session behaviors, and keeps the contract set
|
||||
aligned with `__all__`.
|
||||
importable from `mt5cli`, that all package-root exports are covered by the
|
||||
stable set, and documents key closed-bar, SQLite loading, account-resolution,
|
||||
and trading-session behaviors.
|
||||
|
||||
+5
-4
@@ -31,7 +31,7 @@ rates = collect_latest_rates_for_accounts_with_retries(
|
||||
### Latest closed rate bars
|
||||
|
||||
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
|
||||
row. `fetch_latest_closed_rates()` handles one connected `Mt5CliClient`; use
|
||||
row. `fetch_latest_closed_rates()` handles one connected `MT5Client`; use
|
||||
`fetch_latest_closed_rates_for_trading_client()` from an active
|
||||
`Mt5TradingClient` session. Multi-account helpers fetch `count + 1` bars, drop
|
||||
that row with `drop_forming_rate_bar()`, and validate each series is non-empty. Returned frames are ordered
|
||||
@@ -117,7 +117,8 @@ call it every iteration without over-fetching.
|
||||
```python
|
||||
from pdmt5 import Mt5Config, Mt5DataClient
|
||||
|
||||
from mt5cli import Dataset, ThrottledHistoryUpdater
|
||||
from mt5cli import ThrottledHistoryUpdater
|
||||
from mt5cli.utils import Dataset
|
||||
|
||||
updater = ThrottledHistoryUpdater(
|
||||
output="history.db",
|
||||
@@ -170,5 +171,5 @@ resulting `ValueError` is suppressed along with other recoverable errors.
|
||||
## Trading-capable sessions
|
||||
|
||||
For order placement and trading calculations, use the dedicated
|
||||
[Trading module](trading.md). The read-only `Mt5CliClient` and `mt5_session()`
|
||||
helpers in this module are unchanged.
|
||||
[Trading module](trading.md). Use `mt5_session()` / `MT5Client` for read-only
|
||||
collection.
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
# Storage
|
||||
|
||||
::: mt5cli.storage
|
||||
+8
-7
@@ -6,8 +6,9 @@
|
||||
|
||||
`create_trading_client()` and `mt5_trading_session()` complement the read-only
|
||||
`mt5_session()` helper in `sdk.py`. They return or yield an initialized
|
||||
`pdmt5.Mt5TradingClient`, use `Mt5Config.path` to launch the terminal when
|
||||
configured, and `mt5_trading_session()` always calls `shutdown()` on exit.
|
||||
client supporting order execution and account management, use `Mt5Config.path`
|
||||
to launch the terminal when configured, and `mt5_trading_session()` always
|
||||
calls `shutdown()` on exit.
|
||||
|
||||
```python
|
||||
from mt5cli import create_trading_client, mt5_trading_session
|
||||
@@ -31,7 +32,7 @@ finally:
|
||||
`login` accepts `int`, numeric `str`, or an empty string; empty strings are
|
||||
treated as unset. `path`, `password`, `server`, and `timeout` are forwarded to
|
||||
`pdmt5.Mt5Config`, and omitted `timeout` values keep the lower-level default.
|
||||
The read-only `Mt5CliClient` / `mt5_session()` API is unchanged.
|
||||
Use `mt5_session()` / `MT5Client` for read-only data collection.
|
||||
|
||||
## State and order helpers
|
||||
|
||||
@@ -115,19 +116,19 @@ closed = close_open_positions(client, symbols="EURUSD", dry_run=True)
|
||||
`detect_position_side()` returns `long` for buy-only exposure, `short` for
|
||||
sell-only exposure, and `None` for no positions or mixed long/short exposure.
|
||||
`calculate_spread_ratio()` uses `(ask - bid) / ((ask + bid) / 2)` and raises
|
||||
`Mt5TradingError` when bid or ask is missing or non-positive.
|
||||
`Mt5OperationError` when bid or ask is missing or non-positive.
|
||||
`normalize_order_volume()` returns `0.0` for invalid constraints or
|
||||
sub-minimum requests; check the result before calling `estimate_order_margin()`,
|
||||
which requires a positive finite volume. `calculate_positions_margin()` silently
|
||||
skips rows with missing symbols, non-positive volumes, non-finite volumes, or
|
||||
unsupported position types, but propagates `Mt5TradingError` from `estimate_order_margin()` when a valid row
|
||||
unsupported position types, but propagates `Mt5OperationError` from `estimate_order_margin()` when a valid row
|
||||
encounters invalid tick data or margin results from the broker.
|
||||
|
||||
SL/TP ratios for `determine_order_limits()` must satisfy `0 <= ratio < 1`; `0`
|
||||
omits that level. SL/TP prices are rounded with symbol `digits` metadata when
|
||||
available. `determine_order_limits()` pre-validates computed SL/TP prices against
|
||||
available `trade_stops_level * point` metadata when present; violations raise
|
||||
`Mt5TradingError`. This is a planning helper only: it does not guarantee broker
|
||||
`Mt5OperationError`. This is a planning helper only: it does not guarantee broker
|
||||
acceptance because live validation can still depend on price movement, bid/ask
|
||||
side, freeze levels, and server-side rules, and it does not validate
|
||||
`trade_freeze_level`. When symbol metadata cannot be loaded, protective prices
|
||||
@@ -194,6 +195,6 @@ through the stable package root without embedding entry/exit policy.
|
||||
| Local SL/TP price derivation | `determine_order_limits()` |
|
||||
| Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)` |
|
||||
|
||||
Keep read-only data collection on `mt5_session()` / `Mt5CliClient`; use
|
||||
Keep read-only data collection on `mt5_session()` / `MT5Client`; use
|
||||
`mt5_trading_session()` only where order placement or trading calculations are
|
||||
required.
|
||||
|
||||
+50
-36
@@ -27,28 +27,30 @@ mt5cli provides a stable `MT5Client` Python API, standardized dataset schemas, s
|
||||
pip install mt5cli
|
||||
```
|
||||
|
||||
Parquet export is not included by default. To enable it, install the `parquet` extra:
|
||||
|
||||
```bash
|
||||
pip install "mt5cli[parquet]"
|
||||
```
|
||||
|
||||
## Python API for downstream packages
|
||||
|
||||
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives. `Mt5CliClient` remains available as a backward-compatible alias.
|
||||
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives.
|
||||
|
||||
```python
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli import (
|
||||
DataKind,
|
||||
Dataset,
|
||||
MT5Client,
|
||||
build_config,
|
||||
collect_history,
|
||||
export_dataframe,
|
||||
load_rate_data,
|
||||
minimum_margins,
|
||||
mt5_session,
|
||||
normalize_dataframe,
|
||||
recent_ticks,
|
||||
resolve_rate_view_name,
|
||||
)
|
||||
from mt5cli.history import load_rate_data, resolve_rate_view_name
|
||||
from mt5cli.schemas import DataKind, normalize_dataframe
|
||||
from mt5cli.sdk import minimum_margins, recent_ticks
|
||||
from mt5cli.utils import Dataset, export_dataframe
|
||||
|
||||
# Persistent session for multiple calls
|
||||
with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
|
||||
@@ -84,7 +86,7 @@ collect_history(
|
||||
)
|
||||
```
|
||||
|
||||
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
|
||||
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Export and storage helpers are in `mt5cli.utils` (`Dataset`, `export_dataframe`) and `mt5cli.history`.
|
||||
|
||||
`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly (the CLI requires `--yes` for `order-send`).
|
||||
|
||||
@@ -145,26 +147,38 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
|
||||
| `minimum-margins` | Export minimum-volume margin summary |
|
||||
| `market-book` | Export market depth (order book) |
|
||||
|
||||
### Trading
|
||||
### Trading State
|
||||
|
||||
| Command | Description |
|
||||
| ---------------------- | ----------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request |
|
||||
| `order-send` | Send a trade request to the trade server (`--yes` required) |
|
||||
| Command | Description |
|
||||
| ---------------------- | ------------------------------------------------------------------- |
|
||||
| `orders` | Export active orders |
|
||||
| `positions` | Export open positions |
|
||||
| `history-orders` | Export historical orders |
|
||||
| `history-deals` | Export historical deals |
|
||||
| `recent-history-deals` | Export historical deals from a trailing window |
|
||||
| `mt5-summary` | Export terminal/account status summary |
|
||||
| `order-check` | Check funds sufficiency for a trade request (read-only, no `--yes`) |
|
||||
|
||||
Use `order-check` to validate a request payload before running `order-send --yes`.
|
||||
### Execution (live / mutating)
|
||||
|
||||
These commands send requests to the live trade server and can place or close
|
||||
real trades. Both require `--yes` for live execution.
|
||||
|
||||
| Command | Description |
|
||||
| ----------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| `order-send` | Send a **raw** trade request directly to MT5 (`--yes` required; expert path — no extra validation) |
|
||||
| `close-positions` | Close open positions by `--symbol` or `--ticket` (`--yes` required for live; `--dry-run` to preview) |
|
||||
|
||||
Use `order-check` (Trading State) to validate funds before running `order-send --yes`.
|
||||
`close-positions` is the safer high-level alternative that builds correct close
|
||||
requests automatically. `order-send` is the expert raw path — downstream
|
||||
applications should prefer dedicated closing helpers or their own risk controls.
|
||||
|
||||
### Bulk Collection
|
||||
|
||||
| Command | Description |
|
||||
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `collect-history` | Collect rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database (optional cash-event/position views) |
|
||||
| Command | Description |
|
||||
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `collect-history` | Collect rates, history-orders, and history-deals (ticks opt-in via `--dataset ticks`) for one or more symbols into a single SQLite database (optional cash-event/position views) |
|
||||
|
||||
```bash
|
||||
mt5cli -o history.db collect-history \
|
||||
@@ -176,16 +190,16 @@ mt5cli -o history.db collect-history \
|
||||
|
||||
`collect-history` options:
|
||||
|
||||
| Option | Default | Description |
|
||||
| -------------- | ---------- | --------------------------------------------------------------------------------------------- |
|
||||
| `--symbol/-s` | _required_ | Symbol to collect (repeat for multiple). |
|
||||
| `--date-from` | _required_ | Start date in ISO 8601. |
|
||||
| `--date-to` | _required_ | End date in ISO 8601. |
|
||||
| `--dataset` | all four | Repeatable: `rates`, `ticks`, `history-orders`, `history-deals`. |
|
||||
| `--timeframe` | `M1` | Rates timeframe; recorded in a `timeframe` column on the `rates` table. |
|
||||
| `--flags` | `ALL` | Tick copy flags forwarded to `copy_ticks_range`. |
|
||||
| `--if-exists` | `fail` | `append`, `replace`, or `fail` when a target table already exists. |
|
||||
| `--with-views` | off | Add `cash_events` and `positions_reconstructed` views (requires the `history-deals` dataset). |
|
||||
| Option | Default | Description |
|
||||
| -------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--symbol/-s` | _required_ | Symbol to collect (repeat for multiple). |
|
||||
| `--date-from` | _required_ | Start date in ISO 8601. |
|
||||
| `--date-to` | _required_ | End date in ISO 8601. |
|
||||
| `--dataset` | rates, history-orders, history-deals | Repeatable: `rates`, `ticks`, `history-orders`, `history-deals`. Ticks are opt-in: pass `--dataset ticks` to include them. |
|
||||
| `--timeframe` | `M1` | Rates timeframe; recorded in a `timeframe` column on the `rates` table. |
|
||||
| `--flags` | `ALL` | Tick copy flags forwarded to `copy_ticks_range`. |
|
||||
| `--if-exists` | `fail` | `append`, `replace`, or `fail` when a target table already exists. |
|
||||
| `--with-views` | off | Add `cash_events` and `positions_reconstructed` views (requires the `history-deals` dataset). |
|
||||
|
||||
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
|
||||
|
||||
@@ -215,7 +229,7 @@ See the [History schema diagram](api/history.md#entity-relationship-diagram) for
|
||||
|
||||
Browse the API documentation for detailed module information:
|
||||
|
||||
- [CLI Module](api/cli.md) - CLI application with export commands
|
||||
- [CLI Module](api/cli.md) - CLI application with data export and execution commands
|
||||
- [SDK Module](api/sdk.md) - Programmatic read-only data collection API
|
||||
- [Utils Module](api/utils.md) - Constants, parameter types, parsers, and export utilities
|
||||
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# Grafana Integration for mt5cli
|
||||
|
||||
This directory contains example configuration and dashboard files for visualising
|
||||
mt5cli SQLite data in [Grafana](https://grafana.com/) using the
|
||||
[Grafana SQLite datasource plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- mt5cli installed and able to connect to MetaTrader 5
|
||||
- Grafana 10+ with the `frser-sqlite-datasource` plugin installed
|
||||
- (Optional) Docker and Docker Compose for the containerised setup
|
||||
|
||||
## Generating the SQLite database
|
||||
|
||||
Collect historical data and snapshot current account state:
|
||||
|
||||
```sh
|
||||
# Collect OHLCV history
|
||||
mt5cli -o history.db collect-history --symbol EURUSD --date-from 2024-01-01 --date-to 2024-12-31
|
||||
|
||||
# Create Grafana-ready views and indexes
|
||||
mt5cli -o history.db grafana-schema
|
||||
|
||||
# Snapshot current account, positions, and orders
|
||||
mt5cli -o history.db snapshot --with-grafana-schema
|
||||
```
|
||||
|
||||
## Publishing a Grafana-readable copy
|
||||
|
||||
Grafana reads the SQLite file directly. To avoid read/write conflicts, publish
|
||||
a consistent copy after each update:
|
||||
|
||||
```sh
|
||||
mt5cli -o history.db grafana-schema --publish-copy history.mt5cli.db
|
||||
mt5cli -o history.db snapshot --publish-copy history.mt5cli.db
|
||||
```
|
||||
|
||||
The `--publish-copy` option uses the SQLite online backup API, which is safe
|
||||
even when the source database uses WAL journal mode.
|
||||
|
||||
## Configuring the datasource path
|
||||
|
||||
Edit `provisioning/datasources/mt5cli-sqlite.yml` and set the `path` field
|
||||
to the absolute path of your published `.db` file:
|
||||
|
||||
```yaml
|
||||
jsonData:
|
||||
path: /absolute/path/to/history.mt5cli.db
|
||||
```
|
||||
|
||||
## Running Grafana on Windows (native)
|
||||
|
||||
1. Download and install Grafana from <https://grafana.com/grafana/download/>.
|
||||
2. Install the SQLite plugin: `grafana-cli plugins install frser-sqlite-datasource`.
|
||||
3. Copy `provisioning/datasources/mt5cli-sqlite.yml` into
|
||||
`%ProgramFiles%\GrafanaLabs\grafana\conf\provisioning\datasources\`.
|
||||
Do not copy `provisioning/dashboards/mt5cli.yml` — it contains a
|
||||
Docker-specific dashboard path that is not valid on Windows.
|
||||
4. Import the dashboards from `dashboards/` via the Grafana UI
|
||||
(Dashboards → Import → Upload JSON file).
|
||||
|
||||
## Running with Docker Compose
|
||||
|
||||
Set `MT5CLI_DB_PATH` to the absolute path of your published `.db` file, then
|
||||
start the stack:
|
||||
|
||||
```sh
|
||||
# From the examples/grafana directory
|
||||
MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db docker compose up -d
|
||||
```
|
||||
|
||||
Alternatively, create a `.env` file in `examples/grafana/` containing
|
||||
`MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db` and run
|
||||
`docker compose up -d`. Compose refuses to start if the variable is unset or
|
||||
empty.
|
||||
|
||||
Then open <http://localhost:3000> (default credentials: admin / admin).
|
||||
|
||||
## Dashboard overview
|
||||
|
||||
| Dashboard | Description |
|
||||
| ---------------------- | ------------------------------------------------------- |
|
||||
| `mt5cli-overview.json` | Account balance, equity, margin, and snapshot freshness |
|
||||
| `mt5cli-trades.json` | Trade P/L, win rate, symbol breakdown |
|
||||
| `mt5cli-market.json` | OHLCV rates, spreads, and tick volume |
|
||||
|
||||
All panel queries use the `grafana_*` views; they do not read internal storage
|
||||
tables directly.
|
||||
|
||||
## Importing dashboards
|
||||
|
||||
1. Open Grafana and navigate to **Dashboards → Import**.
|
||||
2. Click **Upload JSON file** and select one of the files in `dashboards/`.
|
||||
3. Select the `mt5cli-SQLite` datasource when prompted.
|
||||
4. Click **Import**.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Docker Compose for Grafana with mt5cli SQLite datasource.
|
||||
#
|
||||
# MT5CLI_DB_PATH must be set to the absolute host path of your published .db
|
||||
# file before running `docker compose up -d`. Compose will refuse to start if
|
||||
# the variable is missing or empty.
|
||||
#
|
||||
# Example:
|
||||
# MT5CLI_DB_PATH=/home/user/history.mt5cli.db docker compose up -d
|
||||
|
||||
services:
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
|
||||
GF_INSTALL_PLUGINS: frser-sqlite-datasource
|
||||
volumes:
|
||||
- ./provisioning:/etc/grafana/provisioning:ro
|
||||
- ./dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana-storage:/var/lib/grafana
|
||||
- ${MT5CLI_DB_PATH:?Set MT5CLI_DB_PATH to the path of your published mt5cli SQLite DB}:/data/mt5cli.db:ro
|
||||
user: "472"
|
||||
|
||||
volumes:
|
||||
grafana-storage:
|
||||
@@ -0,0 +1,98 @@
|
||||
{
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"label": "mt5cli-SQLite",
|
||||
"description": "",
|
||||
"type": "datasource",
|
||||
"pluginId": "frser-sqlite-datasource",
|
||||
"pluginName": "SQLite"
|
||||
}
|
||||
],
|
||||
"__requires": [
|
||||
{
|
||||
"type": "datasource",
|
||||
"id": "frser-sqlite-datasource",
|
||||
"name": "SQLite",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
],
|
||||
"annotations": { "list": [] },
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": { "defaults": {}, "overrides": [] },
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 0 },
|
||||
"id": 1,
|
||||
"title": "Close Price Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"close\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": { "defaults": {}, "overrides": [] },
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||
"id": 2,
|
||||
"title": "Spread Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"spread\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": { "defaults": {}, "overrides": [] },
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
|
||||
"id": 3,
|
||||
"title": "Tick Volume Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"tick_volume\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "1m",
|
||||
"schemaVersion": 36,
|
||||
"tags": ["mt5cli", "market"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Data Source",
|
||||
"multi": false,
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"options": [],
|
||||
"query": "frser-sqlite-datasource",
|
||||
"refresh": 1,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": { "from": "now-24h", "to": "now" },
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "MT5CLI - Market Data",
|
||||
"uid": "mt5cli-market",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
{
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"label": "mt5cli-SQLite",
|
||||
"description": "",
|
||||
"type": "datasource",
|
||||
"pluginId": "frser-sqlite-datasource",
|
||||
"pluginName": "SQLite"
|
||||
}
|
||||
],
|
||||
"__requires": [
|
||||
{
|
||||
"type": "datasource",
|
||||
"id": "frser-sqlite-datasource",
|
||||
"name": "SQLite",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "currencyUSD"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "Balance",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "currencyUSD"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "Equity",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "currencyUSD"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "Free Margin",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"margin_free\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "Margin Level",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"margin_level\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "dateTimeFromNow"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "Last Snapshot",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT MAX(\"time\") * 1000 AS \"Last Snapshot\" FROM grafana_account_snapshots",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 5,
|
||||
"title": "Account Balance Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"title": "Equity Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "1m",
|
||||
"schemaVersion": 36,
|
||||
"tags": ["mt5cli", "account"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Data Source",
|
||||
"multi": false,
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"options": [],
|
||||
"query": "frser-sqlite-datasource",
|
||||
"refresh": 1,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-7d",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "MT5CLI - Account Overview",
|
||||
"uid": "mt5cli-overview",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
{
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"label": "mt5cli-SQLite",
|
||||
"description": "",
|
||||
"type": "datasource",
|
||||
"pluginId": "frser-sqlite-datasource",
|
||||
"pluginName": "SQLite"
|
||||
}
|
||||
],
|
||||
"__requires": [
|
||||
{
|
||||
"type": "datasource",
|
||||
"id": "frser-sqlite-datasource",
|
||||
"name": "SQLite",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"title": "Realized P/L by Symbol",
|
||||
"type": "table",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"symbol\", \"cumulative_pnl\", \"deal_count\" FROM grafana_realized_pnl ORDER BY cumulative_pnl DESC",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "win_rate_pct"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "unit",
|
||||
"value": "percent"
|
||||
},
|
||||
{
|
||||
"id": "displayName",
|
||||
"value": "Win Rate (%)"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"title": "Trade Statistics by Symbol",
|
||||
"type": "table",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"symbol\", \"total_deals\", \"winning_deals\", \"losing_deals\", \"total_profit\", \"avg_profit\", 100.0 * \"winning_deals\" / NULLIF(\"total_deals\", 0) AS \"win_rate_pct\" FROM grafana_trade_stats ORDER BY total_profit DESC",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"title": "Open Position Profit Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"symbol\", SUM(\"profit\") AS profit FROM grafana_position_snapshots GROUP BY time, symbol ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": "${DS_MT5CLI_SQLITE}",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 4,
|
||||
"title": "Cash Events Over Time",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"rawSql": "SELECT \"time\" AS time, \"profit\" FROM grafana_cash_events ORDER BY time",
|
||||
"format": "time_series",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "5m",
|
||||
"schemaVersion": 36,
|
||||
"tags": ["mt5cli", "trades"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Data Source",
|
||||
"multi": false,
|
||||
"name": "DS_MT5CLI_SQLITE",
|
||||
"options": [],
|
||||
"query": "frser-sqlite-datasource",
|
||||
"refresh": 1,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-30d",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "MT5CLI - Trade Analytics",
|
||||
"uid": "mt5cli-trades",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# Grafana dashboard provisioning for mt5cli dashboards.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: mt5cli
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 30
|
||||
allowUiUpdates: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
foldersFromFilesStructure: false
|
||||
@@ -0,0 +1,16 @@
|
||||
# Grafana datasource provisioning for mt5cli SQLite.
|
||||
#
|
||||
# Requires the frser-sqlite-datasource plugin:
|
||||
# grafana-cli plugins install frser-sqlite-datasource
|
||||
#
|
||||
# Set `path` to the absolute path of your published history.mt5cli.db file.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: mt5cli-SQLite
|
||||
type: frser-sqlite-datasource
|
||||
access: proxy
|
||||
isDefault: true
|
||||
jsonData:
|
||||
path: /data/mt5cli.db
|
||||
@@ -59,7 +59,6 @@ nav:
|
||||
- Public API Contract: api/public-contract.md
|
||||
- Client: api/client.md
|
||||
- Schemas: api/schemas.md
|
||||
- Storage: api/storage.md
|
||||
- Converters: api/converters.md
|
||||
- Exceptions: api/exceptions.md
|
||||
- CLI: api/cli.md
|
||||
|
||||
+18
-148
@@ -8,106 +8,37 @@ strategy responsibilities.
|
||||
|
||||
from importlib.metadata import version
|
||||
|
||||
from pdmt5 import Mt5Config, Mt5RuntimeError, Mt5TradingClient, Mt5TradingError
|
||||
|
||||
from .client import MT5Client, build_config, mt5_session
|
||||
from .contract import STABLE_SDK_EXPORTS
|
||||
from .converters import (
|
||||
ensure_utc,
|
||||
granularity_name,
|
||||
normalize_symbol,
|
||||
normalize_symbols,
|
||||
parse_date_range,
|
||||
recent_window,
|
||||
)
|
||||
from .exceptions import (
|
||||
Mt5CliError,
|
||||
Mt5ConnectionError,
|
||||
Mt5OperationError,
|
||||
Mt5SchemaError,
|
||||
call_with_normalized_errors,
|
||||
is_recoverable_mt5_error,
|
||||
normalize_mt5_exception,
|
||||
)
|
||||
from .history import (
|
||||
RateTarget,
|
||||
build_rate_targets,
|
||||
build_rate_view_name,
|
||||
drop_forming_rate_bar,
|
||||
load_rate_data,
|
||||
load_rate_data_from_connection,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
resolve_history_datasets,
|
||||
resolve_history_tick_flags,
|
||||
resolve_history_timeframes,
|
||||
resolve_rate_table_name,
|
||||
resolve_rate_tables,
|
||||
resolve_rate_view_name,
|
||||
resolve_rate_view_names,
|
||||
)
|
||||
from .schemas import (
|
||||
DEDUP_KEYS,
|
||||
KNOWN_MT5_TIME_COLUMNS,
|
||||
REQUIRED_COLUMNS,
|
||||
TIME_COLUMNS,
|
||||
DataKind,
|
||||
normalize_dataframe,
|
||||
normalize_time_columns,
|
||||
schema_columns,
|
||||
validate_schema,
|
||||
)
|
||||
from .sdk import (
|
||||
AccountSpec,
|
||||
Mt5CliClient,
|
||||
ThrottledHistoryUpdater,
|
||||
account_info,
|
||||
collect_history,
|
||||
collect_latest_closed_rates_by_granularity,
|
||||
collect_latest_closed_rates_for_accounts,
|
||||
collect_latest_rates,
|
||||
collect_latest_rates_for_accounts,
|
||||
collect_latest_rates_for_accounts_with_retries,
|
||||
copy_rates_from,
|
||||
copy_rates_from_pos,
|
||||
copy_rates_range,
|
||||
copy_ticks_from,
|
||||
copy_ticks_range,
|
||||
fetch_latest_closed_rates,
|
||||
history_deals,
|
||||
history_orders,
|
||||
last_error,
|
||||
latest_rates,
|
||||
market_book,
|
||||
minimum_margins,
|
||||
mt5_summary,
|
||||
mt5_summary_as_df,
|
||||
orders,
|
||||
positions,
|
||||
recent_history_deals,
|
||||
recent_ticks,
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
substitute_env_placeholders,
|
||||
symbol_info,
|
||||
symbol_info_tick,
|
||||
symbols,
|
||||
terminal_info,
|
||||
update_history,
|
||||
update_history_with_config,
|
||||
)
|
||||
from .sdk import (
|
||||
version as mt5_version,
|
||||
)
|
||||
from .storage import (
|
||||
Dataset,
|
||||
IfExists,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
update_observability,
|
||||
update_observability_with_config,
|
||||
)
|
||||
from .trading import (
|
||||
POSITION_COLUMNS,
|
||||
ExecutionStatus,
|
||||
MarginVolume,
|
||||
OrderExecutionResult,
|
||||
@@ -116,12 +47,17 @@ from .trading import (
|
||||
OrderSide,
|
||||
OrderTimeMode,
|
||||
PositionSide,
|
||||
ProjectionMode,
|
||||
calculate_account_projected_margin_ratio,
|
||||
calculate_margin_and_volume,
|
||||
calculate_new_position_margin_ratio,
|
||||
calculate_positions_margin,
|
||||
calculate_positions_margin_by_symbol,
|
||||
calculate_positions_margin_safe,
|
||||
calculate_projected_margin_ratio,
|
||||
calculate_spread_ratio,
|
||||
calculate_symbol_group_margin_ratio,
|
||||
calculate_trailing_stop_updates,
|
||||
calculate_volume_by_margin,
|
||||
close_open_positions,
|
||||
create_trading_client,
|
||||
@@ -129,6 +65,7 @@ from .trading import (
|
||||
determine_order_limits,
|
||||
ensure_symbol_selected,
|
||||
estimate_order_margin,
|
||||
extract_tick_price,
|
||||
fetch_latest_closed_rates_for_trading_client,
|
||||
fetch_latest_closed_rates_indexed,
|
||||
get_account_snapshot,
|
||||
@@ -139,84 +76,55 @@ from .trading import (
|
||||
normalize_order_volume,
|
||||
place_market_order,
|
||||
update_sltp_for_open_positions,
|
||||
)
|
||||
from .utils import (
|
||||
TICK_FLAG_MAP,
|
||||
TIMEFRAME_MAP,
|
||||
parse_datetime,
|
||||
parse_tick_flags,
|
||||
parse_timeframe,
|
||||
update_trailing_stop_loss_for_open_positions,
|
||||
)
|
||||
|
||||
__version__ = version(__package__) if __package__ else None
|
||||
|
||||
__all__ = [
|
||||
"DEDUP_KEYS",
|
||||
"KNOWN_MT5_TIME_COLUMNS",
|
||||
"POSITION_COLUMNS",
|
||||
"REQUIRED_COLUMNS",
|
||||
"STABLE_SDK_EXPORTS",
|
||||
"TICK_FLAG_MAP",
|
||||
"TIMEFRAME_MAP",
|
||||
"TIME_COLUMNS",
|
||||
"AccountSpec",
|
||||
"DataKind",
|
||||
"Dataset",
|
||||
"ExecutionStatus",
|
||||
"IfExists",
|
||||
"MT5Client",
|
||||
"MarginVolume",
|
||||
"Mt5CliClient",
|
||||
"Mt5CliError",
|
||||
"Mt5Config",
|
||||
"Mt5ConnectionError",
|
||||
"Mt5OperationError",
|
||||
"Mt5RuntimeError",
|
||||
"Mt5SchemaError",
|
||||
"Mt5TradingClient",
|
||||
"Mt5TradingError",
|
||||
"OrderExecutionResult",
|
||||
"OrderFillingMode",
|
||||
"OrderLimits",
|
||||
"OrderSide",
|
||||
"OrderTimeMode",
|
||||
"PositionSide",
|
||||
"ProjectionMode",
|
||||
"RateTarget",
|
||||
"ThrottledHistoryUpdater",
|
||||
"account_info",
|
||||
"build_config",
|
||||
"build_rate_targets",
|
||||
"build_rate_view_name",
|
||||
"calculate_account_projected_margin_ratio",
|
||||
"calculate_margin_and_volume",
|
||||
"calculate_new_position_margin_ratio",
|
||||
"calculate_positions_margin",
|
||||
"calculate_positions_margin_by_symbol",
|
||||
"calculate_positions_margin_safe",
|
||||
"calculate_projected_margin_ratio",
|
||||
"calculate_spread_ratio",
|
||||
"calculate_symbol_group_margin_ratio",
|
||||
"calculate_trailing_stop_updates",
|
||||
"calculate_volume_by_margin",
|
||||
"call_with_normalized_errors",
|
||||
"close_open_positions",
|
||||
"collect_history",
|
||||
"collect_latest_closed_rates_by_granularity",
|
||||
"collect_latest_closed_rates_for_accounts",
|
||||
"collect_latest_rates",
|
||||
"collect_latest_rates_for_accounts",
|
||||
"collect_latest_rates_for_accounts_with_retries",
|
||||
"copy_rates_from",
|
||||
"copy_rates_from_pos",
|
||||
"copy_rates_range",
|
||||
"copy_ticks_from",
|
||||
"copy_ticks_range",
|
||||
"create_trading_client",
|
||||
"detect_format",
|
||||
"detect_position_side",
|
||||
"determine_order_limits",
|
||||
"drop_forming_rate_bar",
|
||||
"ensure_symbol_selected",
|
||||
"ensure_utc",
|
||||
"estimate_order_margin",
|
||||
"export_dataframe",
|
||||
"export_dataframe_to_sqlite",
|
||||
"extract_tick_price",
|
||||
"fetch_latest_closed_rates",
|
||||
"fetch_latest_closed_rates_for_trading_client",
|
||||
"fetch_latest_closed_rates_indexed",
|
||||
@@ -224,56 +132,18 @@ __all__ = [
|
||||
"get_positions_frame",
|
||||
"get_symbol_snapshot",
|
||||
"get_tick_snapshot",
|
||||
"granularity_name",
|
||||
"history_deals",
|
||||
"history_orders",
|
||||
"is_recoverable_mt5_error",
|
||||
"last_error",
|
||||
"latest_rates",
|
||||
"load_rate_data",
|
||||
"load_rate_data_from_connection",
|
||||
"load_rate_series_by_granularity",
|
||||
"load_rate_series_from_sqlite",
|
||||
"market_book",
|
||||
"minimum_margins",
|
||||
"mt5_session",
|
||||
"mt5_summary",
|
||||
"mt5_summary_as_df",
|
||||
"mt5_trading_session",
|
||||
"mt5_version",
|
||||
"normalize_dataframe",
|
||||
"normalize_mt5_exception",
|
||||
"normalize_order_volume",
|
||||
"normalize_symbol",
|
||||
"normalize_symbols",
|
||||
"normalize_time_columns",
|
||||
"orders",
|
||||
"parse_date_range",
|
||||
"parse_datetime",
|
||||
"parse_tick_flags",
|
||||
"parse_timeframe",
|
||||
"place_market_order",
|
||||
"positions",
|
||||
"recent_history_deals",
|
||||
"recent_ticks",
|
||||
"recent_window",
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"resolve_history_datasets",
|
||||
"resolve_history_tick_flags",
|
||||
"resolve_history_timeframes",
|
||||
"resolve_rate_table_name",
|
||||
"resolve_rate_tables",
|
||||
"resolve_rate_view_name",
|
||||
"resolve_rate_view_names",
|
||||
"schema_columns",
|
||||
"substitute_env_placeholders",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
"terminal_info",
|
||||
"update_history",
|
||||
"update_history_with_config",
|
||||
"update_observability",
|
||||
"update_observability_with_config",
|
||||
"update_sltp_for_open_positions",
|
||||
"validate_schema",
|
||||
"update_trailing_stop_loss_for_open_positions",
|
||||
]
|
||||
|
||||
+267
-37
@@ -1,18 +1,21 @@
|
||||
"""Command-line interface for MetaTrader 5 data export."""
|
||||
"""Command-line interface for MetaTrader 5 data and execution utilities."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime # noqa: TC003
|
||||
from pathlib import Path # noqa: TC003
|
||||
from typing import TYPE_CHECKING, Annotated, Any, cast
|
||||
|
||||
import pandas as pd
|
||||
import typer
|
||||
from pdmt5 import Mt5Config
|
||||
|
||||
from . import sdk
|
||||
from .client import MT5Client
|
||||
from .trading import OrderExecutionResult, close_open_positions, create_trading_client
|
||||
from .utils import (
|
||||
DATETIME_TYPE,
|
||||
REQUEST_TYPE,
|
||||
@@ -29,8 +32,6 @@ from .utils import (
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
import pandas as pd
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -54,7 +55,12 @@ class _ExportContext:
|
||||
|
||||
app = typer.Typer(
|
||||
name="mt5cli",
|
||||
help="Export MetaTrader5 data to CSV, JSON, Parquet, or SQLite3.",
|
||||
help=(
|
||||
"MT5 data and execution utilities — read market data, inspect account"
|
||||
" state, and send trade requests. Data commands write to CSV, JSON,"
|
||||
" Parquet, or SQLite3. Execution commands (order-send, close-positions)"
|
||||
" require --yes for live mutations."
|
||||
),
|
||||
)
|
||||
|
||||
_REQUEST_OPTION_HELP = (
|
||||
@@ -150,7 +156,7 @@ def _callback( # pyright: ignore[reportUnusedFunction]
|
||||
typer.Option("--log-level", help="Logging level."),
|
||||
] = LogLevel.WARNING,
|
||||
) -> None:
|
||||
"""Configure shared options for all export commands.
|
||||
"""Configure shared connection and output options.
|
||||
|
||||
Raises:
|
||||
typer.BadParameter: If the output format cannot be determined.
|
||||
@@ -182,7 +188,7 @@ def _callback( # pyright: ignore[reportUnusedFunction]
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def rates_from(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -209,7 +215,7 @@ def rates_from(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def rates_from_pos(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -235,7 +241,7 @@ def rates_from_pos(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def latest_rates(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -264,7 +270,7 @@ def latest_rates(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def rates_range(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -291,7 +297,7 @@ def rates_range(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def ticks_from(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -315,7 +321,7 @@ def ticks_from(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def ticks_range(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -339,7 +345,7 @@ def ticks_range(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def ticks_recent(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -376,19 +382,19 @@ def ticks_recent(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def account_info(ctx: typer.Context) -> None:
|
||||
"""Export account information."""
|
||||
_export_command(ctx, lambda client: client.account_info())
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def terminal_info(ctx: typer.Context) -> None:
|
||||
"""Export terminal information."""
|
||||
_export_command(ctx, lambda client: client.terminal_info())
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def symbols(
|
||||
ctx: typer.Context,
|
||||
group: Annotated[
|
||||
@@ -400,7 +406,7 @@ def symbols(
|
||||
_export_command(ctx, lambda client: client.symbols(group=group))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def symbol_info(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -409,7 +415,7 @@ def symbol_info(
|
||||
_export_command(ctx, lambda client: client.symbol_info(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def minimum_margins(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -418,7 +424,7 @@ def minimum_margins(
|
||||
_export_command(ctx, lambda client: client.minimum_margins(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def orders(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
|
||||
@@ -432,7 +438,7 @@ def orders(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def positions(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
|
||||
@@ -446,7 +452,7 @@ def positions(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def history_orders(
|
||||
ctx: typer.Context,
|
||||
date_from: Annotated[
|
||||
@@ -476,7 +482,7 @@ def history_orders(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def history_deals(
|
||||
ctx: typer.Context,
|
||||
date_from: Annotated[
|
||||
@@ -506,7 +512,7 @@ def history_deals(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def recent_history_deals(
|
||||
ctx: typer.Context,
|
||||
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
|
||||
@@ -529,25 +535,25 @@ def recent_history_deals(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def mt5_summary(ctx: typer.Context) -> None:
|
||||
"""Export a compact terminal/account status summary."""
|
||||
_export_command(ctx, lambda client: client.mt5_summary_as_df())
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def version(ctx: typer.Context) -> None:
|
||||
"""Export MetaTrader5 version information."""
|
||||
_export_command(ctx, lambda client: client.version())
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def last_error(ctx: typer.Context) -> None:
|
||||
"""Export the last error information."""
|
||||
_export_command(ctx, lambda client: client.last_error())
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def symbol_info_tick(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -556,7 +562,7 @@ def symbol_info_tick(
|
||||
_export_command(ctx, lambda client: client.symbol_info_tick(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def market_book(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[str, typer.Option(help="Symbol name.")],
|
||||
@@ -565,7 +571,7 @@ def market_book(
|
||||
_export_command(ctx, lambda client: client.market_book(symbol))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Data / Export")
|
||||
def order_check(
|
||||
ctx: typer.Context,
|
||||
request: Annotated[
|
||||
@@ -577,7 +583,7 @@ def order_check(
|
||||
_export_command(ctx, lambda client: client.order_check(request))
|
||||
|
||||
|
||||
@app.command()
|
||||
@app.command(rich_help_panel="Execution")
|
||||
def order_send(
|
||||
ctx: typer.Context,
|
||||
request: Annotated[
|
||||
@@ -589,7 +595,13 @@ def order_send(
|
||||
typer.Option("--yes", help="Confirm the live trade request."),
|
||||
] = False,
|
||||
) -> None:
|
||||
"""Send a trading operation request to the trade server.
|
||||
"""Send a raw trade request to the trade server (expert path, live execution).
|
||||
|
||||
Passes the request JSON directly to MT5 ``order_send``. This is the
|
||||
low-level expert path — it places real trades on the connected account
|
||||
with no additional validation beyond what MT5 itself performs. Use
|
||||
``order-check`` first to validate funds sufficiency. Prefer
|
||||
``close-positions`` for closing open positions. ``--yes`` is required.
|
||||
|
||||
Raises:
|
||||
typer.BadParameter: If --yes is not provided.
|
||||
@@ -600,7 +612,97 @@ def order_send(
|
||||
_export_command(ctx, lambda client: client.order_send(request))
|
||||
|
||||
|
||||
@app.command()
|
||||
_EXECUTION_RESULT_COLUMNS: list[str] = [
|
||||
"status",
|
||||
"symbol",
|
||||
"order_side",
|
||||
"volume",
|
||||
"retcode",
|
||||
"comment",
|
||||
"request",
|
||||
"response",
|
||||
"dry_run",
|
||||
]
|
||||
|
||||
|
||||
def _execution_results_to_df(results: list[OrderExecutionResult]) -> pd.DataFrame:
|
||||
if not results:
|
||||
return pd.DataFrame(columns=_EXECUTION_RESULT_COLUMNS)
|
||||
rows = [
|
||||
{
|
||||
**r,
|
||||
"request": json.dumps(r["request"]),
|
||||
"response": json.dumps(r["response"]),
|
||||
}
|
||||
for r in results
|
||||
]
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
@app.command(rich_help_panel="Execution")
|
||||
def close_positions(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[
|
||||
list[str] | None,
|
||||
typer.Option(
|
||||
"--symbol",
|
||||
"-s",
|
||||
help="Symbol to close (repeat for multiple symbols).",
|
||||
),
|
||||
] = None,
|
||||
ticket: Annotated[
|
||||
list[int] | None,
|
||||
typer.Option(
|
||||
"--ticket",
|
||||
"-t",
|
||||
help="Position ticket to close (repeat for multiple tickets).",
|
||||
),
|
||||
] = None,
|
||||
dry_run: Annotated[
|
||||
bool,
|
||||
typer.Option("--dry-run", help="Preview close orders without executing them."),
|
||||
] = False,
|
||||
yes: Annotated[
|
||||
bool,
|
||||
typer.Option("--yes", help="Confirm live position closing."),
|
||||
] = False,
|
||||
) -> None:
|
||||
"""Close open positions by symbol or ticket.
|
||||
|
||||
Delegates to :func:`mt5cli.trading.close_open_positions`. At least one
|
||||
``--symbol`` or ``--ticket`` must be provided to avoid accidentally closing
|
||||
all positions. Use ``--dry-run`` to preview without executing; ``--yes`` is
|
||||
required for live execution.
|
||||
|
||||
``order-send`` is the expert raw-request path. ``close-positions`` is the
|
||||
safer high-level helper that builds correct close requests automatically.
|
||||
|
||||
Raises:
|
||||
typer.BadParameter: If neither ``--symbol`` nor ``--ticket`` is given,
|
||||
or if ``--yes`` is missing for a live (non-dry-run) run.
|
||||
"""
|
||||
if not symbol and not ticket:
|
||||
msg = "Provide at least one --symbol or --ticket to close positions."
|
||||
raise typer.BadParameter(msg)
|
||||
if not dry_run and not yes:
|
||||
msg = "Pass --yes to close live positions."
|
||||
raise typer.BadParameter(msg, param_hint="--yes")
|
||||
export_ctx = _get_export_context(ctx)
|
||||
client = create_trading_client(config=export_ctx.config)
|
||||
try:
|
||||
results = close_open_positions(
|
||||
client,
|
||||
symbols=list(symbol) if symbol else None,
|
||||
tickets=list(ticket) if ticket else None,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
finally:
|
||||
client.shutdown()
|
||||
df = _execution_results_to_df(results)
|
||||
_execute_export(ctx, lambda: df)
|
||||
|
||||
|
||||
@app.command(rich_help_panel="Collection")
|
||||
def collect_history(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[
|
||||
@@ -625,7 +727,8 @@ def collect_history(
|
||||
"--dataset",
|
||||
help=(
|
||||
"Dataset to include (repeat for multiple)."
|
||||
" Defaults to all: rates, ticks, history-orders, history-deals."
|
||||
" Defaults to rates, history-orders, history-deals."
|
||||
" Ticks are opt-in: pass --dataset ticks to include them."
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
@@ -663,10 +766,12 @@ def collect_history(
|
||||
) -> None:
|
||||
"""Collect historical datasets into a single SQLite database.
|
||||
|
||||
Tables written depend on ``--dataset``: ``rates``, ``ticks``,
|
||||
``history_orders``, ``history_deals``. History datasets are fetched per
|
||||
symbol and concatenated. Rates rows carry the requested ``timeframe`` so
|
||||
appended runs at different timeframes remain distinguishable.
|
||||
Tables written depend on ``--dataset``: ``rates``, ``history_orders``,
|
||||
``history_deals`` by default. ``ticks`` are opt-in: pass
|
||||
``--dataset ticks`` to include them (tick data grows the database quickly).
|
||||
History datasets are fetched per symbol and concatenated. Rates rows carry
|
||||
the requested ``timeframe`` so appended runs at different timeframes remain
|
||||
distinguishable.
|
||||
|
||||
With ``--with-views`` (requires the ``history-deals`` dataset), optional
|
||||
views ``cash_events`` and ``positions_reconstructed`` are derived from
|
||||
@@ -682,7 +787,7 @@ def collect_history(
|
||||
" Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
|
||||
)
|
||||
raise typer.BadParameter(msg)
|
||||
datasets = set(dataset) if dataset else set(Dataset)
|
||||
datasets = set(dataset) if dataset is not None else None
|
||||
sdk.collect_history(
|
||||
output=export_ctx.output,
|
||||
symbols=symbol,
|
||||
@@ -697,6 +802,131 @@ def collect_history(
|
||||
)
|
||||
|
||||
|
||||
@app.command(rich_help_panel="Collection")
|
||||
def grafana_schema(
|
||||
ctx: typer.Context,
|
||||
publish_copy: Annotated[
|
||||
Path | None,
|
||||
typer.Option(
|
||||
"--publish-copy",
|
||||
help=(
|
||||
"Publish a Grafana-ready SQLite copy to this path"
|
||||
" after schema creation."
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""Create or refresh Grafana-ready views and indexes in a SQLite database.
|
||||
|
||||
Idempotent — safe to run repeatedly on the same database. Requires SQLite
|
||||
output. Does not connect to MetaTrader 5.
|
||||
|
||||
Raises:
|
||||
typer.BadParameter: If the output format is not SQLite3.
|
||||
"""
|
||||
import sqlite3 as _sqlite3 # noqa: PLC0415
|
||||
|
||||
from .grafana import ( # noqa: PLC0415
|
||||
create_snapshot_tables,
|
||||
ensure_grafana_schema,
|
||||
publish_grafana_copy,
|
||||
)
|
||||
|
||||
export_ctx = _get_export_context(ctx)
|
||||
if export_ctx.output_format != "sqlite3":
|
||||
msg = (
|
||||
"grafana-schema requires SQLite3 output."
|
||||
" Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
|
||||
)
|
||||
raise typer.BadParameter(msg)
|
||||
with _sqlite3.connect(export_ctx.output) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA synchronous=NORMAL")
|
||||
create_snapshot_tables(conn)
|
||||
ensure_grafana_schema(conn)
|
||||
logger.info("Grafana schema applied to %s", export_ctx.output)
|
||||
if publish_copy is not None:
|
||||
publish_grafana_copy(export_ctx.output, publish_copy)
|
||||
logger.info("Grafana copy published to %s", publish_copy)
|
||||
|
||||
|
||||
@app.command(rich_help_panel="Collection")
|
||||
def snapshot(
|
||||
ctx: typer.Context,
|
||||
symbol: Annotated[
|
||||
list[str] | None,
|
||||
typer.Option(
|
||||
"--symbol",
|
||||
"-s",
|
||||
help="Symbol filter for positions/orders (repeat for multiple).",
|
||||
),
|
||||
] = None,
|
||||
with_account: Annotated[
|
||||
bool,
|
||||
typer.Option("--with-account/--no-account", help="Snapshot account info."),
|
||||
] = True,
|
||||
with_positions: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
"--with-positions/--no-positions", help="Snapshot open positions."
|
||||
),
|
||||
] = True,
|
||||
with_orders: Annotated[
|
||||
bool,
|
||||
typer.Option("--with-orders/--no-orders", help="Snapshot active orders."),
|
||||
] = True,
|
||||
with_terminal: Annotated[
|
||||
bool,
|
||||
typer.Option("--with-terminal/--no-terminal", help="Snapshot terminal info."),
|
||||
] = True,
|
||||
with_grafana_schema: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
"--with-grafana-schema/--no-grafana-schema",
|
||||
help="Ensure Grafana views and indexes exist.",
|
||||
),
|
||||
] = False,
|
||||
publish_copy: Annotated[
|
||||
Path | None,
|
||||
typer.Option(
|
||||
"--publish-copy",
|
||||
help=("Publish a Grafana-ready SQLite copy to this path after snapshot."),
|
||||
),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""Snapshot current account, position, order, and terminal state into SQLite.
|
||||
|
||||
Appends a timestamped snapshot row for each data type. Never places
|
||||
orders or modifies trading state.
|
||||
|
||||
Raises:
|
||||
typer.BadParameter: If the output format is not SQLite3.
|
||||
"""
|
||||
export_ctx = _get_export_context(ctx)
|
||||
if export_ctx.output_format != "sqlite3":
|
||||
msg = (
|
||||
"snapshot requires SQLite3 output."
|
||||
" Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
|
||||
)
|
||||
raise typer.BadParameter(msg)
|
||||
sdk.update_observability_with_config(
|
||||
output=export_ctx.output,
|
||||
config=export_ctx.config,
|
||||
symbols=list(symbol) if symbol else None,
|
||||
include_account=with_account,
|
||||
include_positions=with_positions,
|
||||
include_orders=with_orders,
|
||||
include_terminal=with_terminal,
|
||||
with_grafana_schema=with_grafana_schema,
|
||||
)
|
||||
logger.info("Snapshot written to %s", export_ctx.output)
|
||||
if publish_copy is not None:
|
||||
from .grafana import publish_grafana_copy # noqa: PLC0415
|
||||
|
||||
publish_grafana_copy(export_ctx.output, publish_copy)
|
||||
logger.info("Grafana copy published to %s", publish_copy)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Run the mt5cli CLI."""
|
||||
app()
|
||||
|
||||
+1
-3
@@ -24,9 +24,7 @@ class MT5Client(Mt5CliClient):
|
||||
"""Public client for generic MT5 data access and order primitives.
|
||||
|
||||
Extends the read-only SDK client with optional order check/send helpers and
|
||||
exposes the same connection lifecycle as :class:`~mt5cli.sdk.Mt5CliClient`.
|
||||
Downstream applications such as private trading packages should prefer this
|
||||
type over the legacy ``Mt5CliClient`` name.
|
||||
exposes the same connection lifecycle as :func:`mt5_session`.
|
||||
|
||||
mt5cli intentionally exposes minimal execution primitives only. Trading
|
||||
decisions, signals, strategies, backtests, and optimization remain the
|
||||
|
||||
+10
-47
@@ -1,61 +1,50 @@
|
||||
"""Stable downstream SDK export names for mt5cli."""
|
||||
"""Downstream SDK export tier for mt5cli."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
|
||||
"AccountSpec",
|
||||
"MT5Client",
|
||||
"Mt5CliClient",
|
||||
"Mt5CliError",
|
||||
"Mt5Config",
|
||||
"Mt5ConnectionError",
|
||||
"Mt5OperationError",
|
||||
"Mt5RuntimeError",
|
||||
"Mt5SchemaError",
|
||||
"Mt5TradingClient",
|
||||
"Mt5TradingError",
|
||||
"OrderFillingMode",
|
||||
"OrderSide",
|
||||
"OrderTimeMode",
|
||||
"PositionSide",
|
||||
"ProjectionMode",
|
||||
"ExecutionStatus",
|
||||
"MarginVolume",
|
||||
"OrderExecutionResult",
|
||||
"OrderLimits",
|
||||
"RateTarget",
|
||||
"ThrottledHistoryUpdater",
|
||||
"account_info",
|
||||
"build_config",
|
||||
"build_rate_targets",
|
||||
"build_rate_view_name",
|
||||
"calculate_account_projected_margin_ratio",
|
||||
"calculate_margin_and_volume",
|
||||
"calculate_new_position_margin_ratio",
|
||||
"calculate_projected_margin_ratio",
|
||||
"calculate_positions_margin",
|
||||
"calculate_positions_margin_by_symbol",
|
||||
"calculate_positions_margin_safe",
|
||||
"calculate_spread_ratio",
|
||||
"calculate_symbol_group_margin_ratio",
|
||||
"calculate_trailing_stop_updates",
|
||||
"calculate_volume_by_margin",
|
||||
"call_with_normalized_errors",
|
||||
"close_open_positions",
|
||||
"collect_history",
|
||||
"collect_latest_closed_rates_by_granularity",
|
||||
"collect_latest_closed_rates_for_accounts",
|
||||
"collect_latest_rates",
|
||||
"collect_latest_rates_for_accounts",
|
||||
"collect_latest_rates_for_accounts_with_retries",
|
||||
"copy_rates_from",
|
||||
"copy_rates_from_pos",
|
||||
"copy_rates_range",
|
||||
"copy_ticks_from",
|
||||
"copy_ticks_range",
|
||||
"create_trading_client",
|
||||
"detect_position_side",
|
||||
"determine_order_limits",
|
||||
"drop_forming_rate_bar",
|
||||
"ensure_symbol_selected",
|
||||
"estimate_order_margin",
|
||||
"export_dataframe",
|
||||
"export_dataframe_to_sqlite",
|
||||
"extract_tick_price",
|
||||
"fetch_latest_closed_rates",
|
||||
"fetch_latest_closed_rates_for_trading_client",
|
||||
"fetch_latest_closed_rates_indexed",
|
||||
@@ -63,46 +52,20 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
|
||||
"get_positions_frame",
|
||||
"get_symbol_snapshot",
|
||||
"get_tick_snapshot",
|
||||
"history_deals",
|
||||
"history_orders",
|
||||
"is_recoverable_mt5_error",
|
||||
"last_error",
|
||||
"latest_rates",
|
||||
"load_rate_data",
|
||||
"load_rate_data_from_connection",
|
||||
"load_rate_series_by_granularity",
|
||||
"load_rate_series_from_sqlite",
|
||||
"market_book",
|
||||
"minimum_margins",
|
||||
"mt5_session",
|
||||
"mt5_summary",
|
||||
"mt5_summary_as_df",
|
||||
"mt5_trading_session",
|
||||
"mt5_version",
|
||||
"normalize_mt5_exception",
|
||||
"normalize_order_volume",
|
||||
"orders",
|
||||
"place_market_order",
|
||||
"positions",
|
||||
"recent_history_deals",
|
||||
"recent_ticks",
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"resolve_history_datasets",
|
||||
"resolve_history_tick_flags",
|
||||
"resolve_history_timeframes",
|
||||
"resolve_rate_table_name",
|
||||
"resolve_rate_tables",
|
||||
"resolve_rate_view_name",
|
||||
"resolve_rate_view_names",
|
||||
"substitute_env_placeholders",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
"terminal_info",
|
||||
"update_history",
|
||||
"update_history_with_config",
|
||||
"update_observability",
|
||||
"update_observability_with_config",
|
||||
"update_sltp_for_open_positions",
|
||||
"update_trailing_stop_loss_for_open_positions",
|
||||
})
|
||||
|
||||
__all__ = ["STABLE_SDK_EXPORTS"]
|
||||
|
||||
@@ -4,11 +4,16 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, TypeVar
|
||||
|
||||
from pdmt5 import Mt5RuntimeError, Mt5TradingError
|
||||
from pdmt5 import Mt5RuntimeError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
try:
|
||||
from pdmt5 import Mt5TradingError
|
||||
except ImportError: # pragma: no cover
|
||||
Mt5TradingError = None # type: ignore[assignment]
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
__all__ = [
|
||||
@@ -22,7 +27,7 @@ __all__ = [
|
||||
]
|
||||
|
||||
_RECOVERABLE_MT5_ERRORS: tuple[type[BaseException], ...] = (
|
||||
Mt5TradingError,
|
||||
*([Mt5TradingError] if Mt5TradingError is not None else []), # type: ignore[misc]
|
||||
Mt5RuntimeError,
|
||||
)
|
||||
|
||||
@@ -50,7 +55,7 @@ def is_recoverable_mt5_error(exc: BaseException) -> bool:
|
||||
exc: Exception raised by MT5 or pdmt5.
|
||||
|
||||
Returns:
|
||||
True for ``Mt5RuntimeError`` and ``Mt5TradingError``.
|
||||
True for ``Mt5RuntimeError`` and ``Mt5TradingError`` (if available).
|
||||
"""
|
||||
return isinstance(exc, _RECOVERABLE_MT5_ERRORS)
|
||||
|
||||
@@ -65,7 +70,7 @@ def normalize_mt5_exception(exc: BaseException) -> Mt5CliError:
|
||||
``Mt5ConnectionError`` for runtime failures, ``Mt5OperationError`` for
|
||||
trading failures, or the original exception when it is not recognized.
|
||||
"""
|
||||
if isinstance(exc, Mt5TradingError):
|
||||
if Mt5TradingError is not None and isinstance(exc, Mt5TradingError):
|
||||
return Mt5OperationError(str(exc))
|
||||
if isinstance(exc, Mt5RuntimeError):
|
||||
return Mt5ConnectionError(str(exc))
|
||||
|
||||
@@ -0,0 +1,682 @@
|
||||
"""Grafana-oriented SQLite views, indexes, and snapshot tables."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import datetime
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
from .history import get_table_columns
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TRADE_DEAL_TYPES_SQL = "(0, 1)"
|
||||
|
||||
_GRAFANA_VIEW_NAMES = (
|
||||
"grafana_rates",
|
||||
"grafana_ticks",
|
||||
"grafana_history_deals",
|
||||
"grafana_history_orders",
|
||||
"grafana_trade_deals",
|
||||
"grafana_cash_events",
|
||||
"grafana_realized_pnl",
|
||||
"grafana_symbol_pnl",
|
||||
"grafana_trade_stats",
|
||||
"grafana_account_snapshots",
|
||||
"grafana_position_snapshots",
|
||||
"grafana_order_snapshots",
|
||||
"grafana_terminal_snapshots",
|
||||
)
|
||||
|
||||
|
||||
def _to_epoch_int(value: object) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime.datetime):
|
||||
return int(value.timestamp())
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
return None
|
||||
|
||||
|
||||
def _time_col_expr(col: str) -> str:
|
||||
return (
|
||||
f"CASE WHEN typeof(\"{col}\") IN ('integer', 'real')"
|
||||
f' THEN CAST("{col}" AS INTEGER)'
|
||||
f" ELSE CAST(strftime('%s', \"{col}\") AS INTEGER) END"
|
||||
)
|
||||
|
||||
|
||||
def _create_view_safe(
|
||||
conn: sqlite3.Connection,
|
||||
name: str,
|
||||
select_sql: str,
|
||||
) -> None:
|
||||
try:
|
||||
conn.execute(f'DROP VIEW IF EXISTS "{name}"')
|
||||
conn.execute(f'CREATE VIEW "{name}" AS {select_sql}')
|
||||
except sqlite3.Error as exc:
|
||||
logger.warning("Skipping view %s: %s", name, exc)
|
||||
|
||||
|
||||
def _other_cols(all_cols: set[str], exclude: set[str]) -> list[str]:
|
||||
return sorted(all_cols - exclude)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Snapshot table DDL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SNAPSHOT_TABLE_DDLS: list[str] = [
|
||||
"""CREATE TABLE IF NOT EXISTS snapshot_runs (
|
||||
run_id INTEGER PRIMARY KEY,
|
||||
observed_at INTEGER NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
detail TEXT
|
||||
)""",
|
||||
"""CREATE TABLE IF NOT EXISTS account_snapshots (
|
||||
run_id INTEGER NOT NULL,
|
||||
login INTEGER,
|
||||
currency TEXT,
|
||||
balance REAL,
|
||||
equity REAL,
|
||||
margin REAL,
|
||||
margin_free REAL,
|
||||
margin_level REAL,
|
||||
profit REAL,
|
||||
leverage INTEGER
|
||||
)""",
|
||||
"""CREATE TABLE IF NOT EXISTS position_snapshots (
|
||||
run_id INTEGER NOT NULL,
|
||||
login INTEGER,
|
||||
ticket INTEGER,
|
||||
position_id INTEGER,
|
||||
symbol TEXT,
|
||||
type INTEGER,
|
||||
volume REAL,
|
||||
price_open REAL,
|
||||
price_current REAL,
|
||||
profit REAL,
|
||||
swap REAL,
|
||||
comment TEXT,
|
||||
magic INTEGER
|
||||
)""",
|
||||
"""CREATE TABLE IF NOT EXISTS order_snapshots (
|
||||
run_id INTEGER NOT NULL,
|
||||
login INTEGER,
|
||||
ticket INTEGER,
|
||||
symbol TEXT,
|
||||
type INTEGER,
|
||||
volume_current REAL,
|
||||
price_open REAL,
|
||||
price_current REAL,
|
||||
state INTEGER,
|
||||
comment TEXT,
|
||||
magic INTEGER,
|
||||
time_setup INTEGER
|
||||
)""",
|
||||
"""CREATE TABLE IF NOT EXISTS terminal_snapshots (
|
||||
run_id INTEGER NOT NULL,
|
||||
name TEXT,
|
||||
connected INTEGER,
|
||||
community_account INTEGER,
|
||||
trade_allowed INTEGER,
|
||||
trade_expert INTEGER,
|
||||
path TEXT,
|
||||
company TEXT,
|
||||
language TEXT
|
||||
)""",
|
||||
]
|
||||
|
||||
|
||||
def create_snapshot_tables(conn: sqlite3.Connection) -> None:
|
||||
"""Create snapshot tables idempotently."""
|
||||
for ddl in _SNAPSHOT_TABLE_DDLS:
|
||||
conn.execute(ddl)
|
||||
|
||||
|
||||
def start_snapshot_run(conn: sqlite3.Connection, observed_at: int) -> int:
|
||||
"""Insert a snapshot_runs row with status 'running' and return its run_id.
|
||||
|
||||
Returns:
|
||||
The auto-assigned run_id for the new row.
|
||||
"""
|
||||
cursor = conn.execute(
|
||||
"INSERT INTO snapshot_runs (observed_at, status) VALUES (?, 'running')",
|
||||
(observed_at,),
|
||||
)
|
||||
return cast("int", cursor.lastrowid)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# View builders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_grafana_rates(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "rates")
|
||||
required = {"time", "symbol", "timeframe"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_rates: rates table missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
others = _other_cols(cols, {"time"})
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_rates",
|
||||
f'SELECT {time_expr} AS "time", {other_sql} FROM "rates"', # noqa: S608
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_ticks(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "ticks")
|
||||
required = {"time", "symbol"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_ticks: ticks table missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
others = _other_cols(cols, {"time"})
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_ticks",
|
||||
f'SELECT {time_expr} AS "time", {other_sql} FROM "ticks"', # noqa: S608
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_history_deals(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
if "time" not in cols:
|
||||
logger.warning("Skipping grafana_history_deals: history_deals.time is missing")
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
others = _other_cols(cols, {"time"})
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_history_deals",
|
||||
f'SELECT {time_expr} AS "time", {other_sql} FROM "history_deals"', # noqa: S608
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_history_orders(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_orders")
|
||||
if "time_setup" not in cols:
|
||||
logger.warning(
|
||||
"Skipping grafana_history_orders: history_orders.time_setup is missing"
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time_setup")
|
||||
others = _other_cols(cols, set())
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_history_orders",
|
||||
f'SELECT {time_expr} AS "time", {other_sql} FROM "history_orders"', # noqa: S608
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_trade_deals(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
required = {"time", "type"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_trade_deals: history_deals missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
others = _other_cols(cols, {"time"})
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_trade_deals",
|
||||
f'SELECT {time_expr} AS "time", {other_sql}' # noqa: S608
|
||||
f' FROM "history_deals" WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}',
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_cash_events(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
required = {"time", "type"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_cash_events: history_deals missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
others = _other_cols(cols, {"time"})
|
||||
other_sql = ", ".join(f'"{c}"' for c in others)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_cash_events",
|
||||
f'SELECT {time_expr} AS "time", {other_sql}' # noqa: S608
|
||||
f' FROM "history_deals" WHERE "type" NOT IN {_TRADE_DEAL_TYPES_SQL}',
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_realized_pnl(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
required = {"symbol", "profit", "type", "entry"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_realized_pnl: history_deals missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_realized_pnl",
|
||||
'SELECT "symbol",' # noqa: S608
|
||||
' SUM("profit") AS cumulative_pnl, COUNT(*) AS deal_count'
|
||||
' FROM "history_deals"'
|
||||
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
|
||||
' AND "entry" IN (1, 2, 3)'
|
||||
' AND "symbol" IS NOT NULL AND "symbol" != \'\''
|
||||
' GROUP BY "symbol"',
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_symbol_pnl(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
required = {"time", "symbol", "profit", "type", "entry"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_symbol_pnl: history_deals missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
time_expr = _time_col_expr("time")
|
||||
select_parts = [f'{time_expr} AS "time"', '"symbol"', '"profit"']
|
||||
if "volume" in cols:
|
||||
select_parts.append('"volume"')
|
||||
if "price" in cols:
|
||||
select_parts.append('"price"')
|
||||
select_sql = ", ".join(select_parts)
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_symbol_pnl",
|
||||
f'SELECT {select_sql} FROM "history_deals"' # noqa: S608
|
||||
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
|
||||
' AND "entry" IN (1, 2, 3)'
|
||||
' AND "symbol" IS NOT NULL AND "symbol" != \'\'',
|
||||
)
|
||||
|
||||
|
||||
def _build_grafana_trade_stats(conn: sqlite3.Connection) -> None:
|
||||
cols = get_table_columns(conn, "history_deals")
|
||||
required = {"symbol", "profit", "type"}
|
||||
if not required.issubset(cols):
|
||||
logger.warning(
|
||||
"Skipping grafana_trade_stats: history_deals missing columns %s",
|
||||
sorted(required - cols),
|
||||
)
|
||||
return
|
||||
has_entry = "entry" in cols
|
||||
entry_filter = ' AND "entry" IN (1, 2, 3)' if has_entry else ""
|
||||
_create_view_safe(
|
||||
conn,
|
||||
"grafana_trade_stats",
|
||||
'SELECT "symbol",' # noqa: S608
|
||||
" COUNT(*) AS total_deals,"
|
||||
' SUM(CASE WHEN "profit" > 0 THEN 1 ELSE 0 END) AS winning_deals,'
|
||||
' SUM(CASE WHEN "profit" <= 0 THEN 1 ELSE 0 END) AS losing_deals,'
|
||||
' SUM("profit") AS total_profit,'
|
||||
' AVG("profit") AS avg_profit,'
|
||||
' MAX("profit") AS max_profit,'
|
||||
' MIN("profit") AS min_profit'
|
||||
' FROM "history_deals"'
|
||||
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
|
||||
f"{entry_filter}"
|
||||
' AND "symbol" IS NOT NULL AND "symbol" != \'\''
|
||||
' GROUP BY "symbol"',
|
||||
)
|
||||
|
||||
|
||||
def _build_snapshot_view(
|
||||
conn: sqlite3.Connection,
|
||||
view_name: str,
|
||||
table_name: str,
|
||||
) -> None:
|
||||
cols = get_table_columns(conn, table_name)
|
||||
if not cols:
|
||||
logger.warning("Skipping %s: %s table missing", view_name, table_name)
|
||||
return
|
||||
if "run_id" not in cols:
|
||||
logger.warning("Skipping %s: %s missing run_id column", view_name, table_name)
|
||||
return
|
||||
others = _other_cols(cols, {"run_id"})
|
||||
run_cols = get_table_columns(conn, "snapshot_runs")
|
||||
if {"run_id", "observed_at", "status"}.issubset(run_cols):
|
||||
other_sql = (", " + ", ".join(f's."{c}"' for c in others)) if others else ""
|
||||
select_cols = f'r."observed_at" AS "time", s."run_id"{other_sql}'
|
||||
_create_view_safe(
|
||||
conn,
|
||||
view_name,
|
||||
f'SELECT {select_cols} FROM "{table_name}" s' # noqa: S608
|
||||
f' JOIN "snapshot_runs" r ON s."run_id" = r."run_id"'
|
||||
f" WHERE r.\"status\" = 'ok'",
|
||||
)
|
||||
else:
|
||||
logger.warning("Skipping %s: snapshot_runs missing required columns", view_name)
|
||||
|
||||
|
||||
def _build_grafana_account_snapshots(conn: sqlite3.Connection) -> None:
|
||||
_build_snapshot_view(conn, "grafana_account_snapshots", "account_snapshots")
|
||||
|
||||
|
||||
def _build_grafana_position_snapshots(conn: sqlite3.Connection) -> None:
|
||||
_build_snapshot_view(conn, "grafana_position_snapshots", "position_snapshots")
|
||||
|
||||
|
||||
def _build_grafana_order_snapshots(conn: sqlite3.Connection) -> None:
|
||||
_build_snapshot_view(conn, "grafana_order_snapshots", "order_snapshots")
|
||||
|
||||
|
||||
def _build_grafana_terminal_snapshots(conn: sqlite3.Connection) -> None:
|
||||
_build_snapshot_view(conn, "grafana_terminal_snapshots", "terminal_snapshots")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def create_grafana_views(conn: sqlite3.Connection) -> None:
|
||||
"""Create all Grafana-facing views idempotently.
|
||||
|
||||
Missing source tables cause the affected view to be skipped with a warning;
|
||||
other views are unaffected. Stale views whose source table or required
|
||||
columns have disappeared are dropped before rebuild.
|
||||
"""
|
||||
for name in _GRAFANA_VIEW_NAMES:
|
||||
conn.execute(f'DROP VIEW IF EXISTS "{name}"')
|
||||
_build_grafana_rates(conn)
|
||||
_build_grafana_ticks(conn)
|
||||
_build_grafana_history_deals(conn)
|
||||
_build_grafana_history_orders(conn)
|
||||
_build_grafana_trade_deals(conn)
|
||||
_build_grafana_cash_events(conn)
|
||||
_build_grafana_realized_pnl(conn)
|
||||
_build_grafana_symbol_pnl(conn)
|
||||
_build_grafana_trade_stats(conn)
|
||||
_build_grafana_account_snapshots(conn)
|
||||
_build_grafana_position_snapshots(conn)
|
||||
_build_grafana_order_snapshots(conn)
|
||||
_build_grafana_terminal_snapshots(conn)
|
||||
|
||||
|
||||
def create_grafana_indexes(conn: sqlite3.Connection) -> None:
|
||||
"""Create Grafana query performance indexes idempotently."""
|
||||
rates_cols = get_table_columns(conn, "rates")
|
||||
if {"time", "symbol", "timeframe"}.issubset(rates_cols):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_rates_time_symbol_timeframe"
|
||||
' ON "rates"("time", "symbol", "timeframe")',
|
||||
)
|
||||
|
||||
ticks_cols = get_table_columns(conn, "ticks")
|
||||
if {"time", "symbol"}.issubset(ticks_cols):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_ticks_time_symbol"
|
||||
' ON "ticks"("time", "symbol")',
|
||||
)
|
||||
|
||||
deals_cols = get_table_columns(conn, "history_deals")
|
||||
if {"time", "symbol"}.issubset(deals_cols):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_history_deals_time_symbol"
|
||||
' ON "history_deals"("time", "symbol")',
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_history_deals_symbol_time"
|
||||
' ON "history_deals"("symbol", "time")',
|
||||
)
|
||||
|
||||
orders_cols = get_table_columns(conn, "history_orders")
|
||||
if {"time_setup", "symbol"}.issubset(orders_cols):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_history_orders_time_setup_symbol"
|
||||
' ON "history_orders"("time_setup", "symbol")',
|
||||
)
|
||||
|
||||
if {"run_id", "login"}.issubset(get_table_columns(conn, "account_snapshots")):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_account_snapshots_time_login"
|
||||
' ON "account_snapshots"("run_id", "login")',
|
||||
)
|
||||
if {"run_id", "symbol"}.issubset(get_table_columns(conn, "position_snapshots")):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_position_snapshots_time_symbol"
|
||||
' ON "position_snapshots"("run_id", "symbol")',
|
||||
)
|
||||
if {"run_id", "symbol"}.issubset(get_table_columns(conn, "order_snapshots")):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_order_snapshots_time_symbol"
|
||||
' ON "order_snapshots"("run_id", "symbol")',
|
||||
)
|
||||
if {"observed_at", "status"}.issubset(get_table_columns(conn, "snapshot_runs")):
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_snapshot_runs_time_status"
|
||||
' ON "snapshot_runs"("observed_at", "status")',
|
||||
)
|
||||
|
||||
|
||||
def ensure_grafana_schema(conn: sqlite3.Connection) -> None:
|
||||
"""Create snapshot tables, Grafana views, and indexes idempotently."""
|
||||
create_snapshot_tables(conn)
|
||||
create_grafana_views(conn)
|
||||
create_grafana_indexes(conn)
|
||||
|
||||
|
||||
def publish_grafana_copy(
|
||||
source: str | Path,
|
||||
target: str | Path,
|
||||
) -> Path:
|
||||
"""Publish a consistent SQLite copy for Grafana using the backup API.
|
||||
|
||||
Uses the SQLite online backup API for a WAL-safe, consistent snapshot of
|
||||
the source database. Writes to a temporary file beside the target, then
|
||||
atomically replaces it so that a previous published copy is preserved if
|
||||
publishing fails.
|
||||
|
||||
Args:
|
||||
source: Path to the source SQLite database.
|
||||
target: Destination path for the published copy.
|
||||
|
||||
Returns:
|
||||
The resolved absolute target path.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the source database does not exist.
|
||||
ValueError: If source and target resolve to the same path.
|
||||
"""
|
||||
source_path = Path(source)
|
||||
target_path = Path(target)
|
||||
if source_path.resolve() == target_path.resolve():
|
||||
msg = "--publish-copy target must differ from the source database: " + str(
|
||||
source_path
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if not source_path.exists():
|
||||
raise FileNotFoundError(source_path)
|
||||
|
||||
target_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tmp_fd, tmp_str = tempfile.mkstemp(
|
||||
dir=target_path.parent,
|
||||
suffix=".tmp",
|
||||
prefix=target_path.name + ".",
|
||||
)
|
||||
tmp_path = Path(tmp_str)
|
||||
try:
|
||||
os.close(tmp_fd)
|
||||
with (
|
||||
contextlib.closing(sqlite3.connect(source_path)) as src,
|
||||
contextlib.closing(sqlite3.connect(tmp_path)) as dst,
|
||||
):
|
||||
src.backup(dst)
|
||||
try:
|
||||
target_mode = target_path.stat().st_mode & 0o777
|
||||
except FileNotFoundError:
|
||||
target_mode = 0o644
|
||||
Path(tmp_path).chmod(target_mode)
|
||||
tmp_path.replace(target_path)
|
||||
except Exception:
|
||||
with contextlib.suppress(OSError):
|
||||
tmp_path.unlink()
|
||||
raise
|
||||
|
||||
logger.info("Published Grafana copy: %s -> %s", source_path, target_path)
|
||||
return target_path.resolve()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Snapshot insert helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def insert_account_snapshot(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
row: dict[str, object],
|
||||
) -> None:
|
||||
"""Append one account state row to account_snapshots."""
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots"
|
||||
" (run_id, login, currency, balance, equity,"
|
||||
" margin, margin_free, margin_level, profit, leverage)"
|
||||
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
run_id,
|
||||
row.get("login"),
|
||||
row.get("currency"),
|
||||
row.get("balance"),
|
||||
row.get("equity"),
|
||||
row.get("margin"),
|
||||
row.get("margin_free"),
|
||||
row.get("margin_level"),
|
||||
row.get("profit"),
|
||||
row.get("leverage"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def insert_position_snapshots(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
login: int | None,
|
||||
rows: list[dict[str, object]],
|
||||
) -> None:
|
||||
"""Append position rows to position_snapshots; no-op when rows is empty."""
|
||||
if not rows:
|
||||
return
|
||||
conn.executemany(
|
||||
"INSERT INTO position_snapshots"
|
||||
" (run_id, login, ticket, position_id, symbol, type, volume,"
|
||||
" price_open, price_current, profit, swap, comment, magic)"
|
||||
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
[
|
||||
(
|
||||
run_id,
|
||||
login,
|
||||
r.get("ticket"),
|
||||
r.get("position_id"),
|
||||
r.get("symbol"),
|
||||
r.get("type"),
|
||||
r.get("volume"),
|
||||
r.get("price_open"),
|
||||
r.get("price_current"),
|
||||
r.get("profit"),
|
||||
r.get("swap"),
|
||||
r.get("comment"),
|
||||
r.get("magic"),
|
||||
)
|
||||
for r in rows
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def insert_order_snapshots(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
login: int | None,
|
||||
rows: list[dict[str, object]],
|
||||
) -> None:
|
||||
"""Append order rows to order_snapshots; no-op when rows is empty."""
|
||||
if not rows:
|
||||
return
|
||||
conn.executemany(
|
||||
"INSERT INTO order_snapshots"
|
||||
" (run_id, login, ticket, symbol, type, volume_current,"
|
||||
" price_open, price_current, state, comment, magic, time_setup)"
|
||||
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
[
|
||||
(
|
||||
run_id,
|
||||
login,
|
||||
r.get("ticket"),
|
||||
r.get("symbol"),
|
||||
r.get("type"),
|
||||
r.get("volume_current"),
|
||||
r.get("price_open"),
|
||||
r.get("price_current"),
|
||||
r.get("state"),
|
||||
r.get("comment"),
|
||||
r.get("magic"),
|
||||
_to_epoch_int(r.get("time_setup")),
|
||||
)
|
||||
for r in rows
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def insert_terminal_snapshot(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
row: dict[str, object],
|
||||
) -> None:
|
||||
"""Append one terminal state row to terminal_snapshots."""
|
||||
conn.execute(
|
||||
"INSERT INTO terminal_snapshots"
|
||||
" (run_id, name, connected, community_account,"
|
||||
" trade_allowed, trade_expert, path, company, language)"
|
||||
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
run_id,
|
||||
row.get("name"),
|
||||
row.get("connected"),
|
||||
row.get("community_account"),
|
||||
row.get("trade_allowed"),
|
||||
row.get("trade_expert"),
|
||||
row.get("path"),
|
||||
row.get("company"),
|
||||
row.get("language"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def record_snapshot_run(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
status: str,
|
||||
detail: str | None = None,
|
||||
) -> None:
|
||||
"""Finalize a snapshot run by setting its status."""
|
||||
conn.execute(
|
||||
"UPDATE snapshot_runs SET status = ?, detail = ? WHERE run_id = ?",
|
||||
(status, detail, run_id),
|
||||
)
|
||||
+9
-3
@@ -30,6 +30,11 @@ if TYPE_CHECKING:
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_HISTORY_TIMEFRAMES: tuple[str, ...] = TIMEFRAME_NAMES
|
||||
DEFAULT_HISTORY_DATASETS: frozenset[Dataset] = frozenset({
|
||||
Dataset.rates,
|
||||
Dataset.history_orders,
|
||||
Dataset.history_deals,
|
||||
})
|
||||
|
||||
_HISTORY_DEDUP_KEYS: dict[Dataset, tuple[tuple[str, ...], ...]] = {
|
||||
Dataset.rates: DEDUP_KEYS[DataKind.rates],
|
||||
@@ -62,11 +67,12 @@ def resolve_history_datasets(datasets: set[Dataset] | None) -> set[Dataset]:
|
||||
"""Resolve configured history datasets.
|
||||
|
||||
Returns:
|
||||
All supported datasets when ``datasets`` is None, otherwise the
|
||||
configured selection (which may be empty).
|
||||
``DEFAULT_HISTORY_DATASETS`` (rates, history-orders, history-deals)
|
||||
when ``datasets`` is None, otherwise the configured selection (which
|
||||
may be empty or explicitly include ``Dataset.ticks``).
|
||||
"""
|
||||
if datasets is None:
|
||||
return set(Dataset)
|
||||
return set(DEFAULT_HISTORY_DATASETS)
|
||||
return set(datasets)
|
||||
|
||||
|
||||
|
||||
+352
-29
@@ -8,15 +8,30 @@ import os
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from contextlib import closing, contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Self, TypeVar, cast
|
||||
|
||||
import pandas as pd
|
||||
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError, Mt5TradingError
|
||||
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError
|
||||
|
||||
try:
|
||||
from pdmt5 import Mt5TradingError
|
||||
except ImportError: # pragma: no cover
|
||||
Mt5TradingError = None # type: ignore[assignment]
|
||||
|
||||
from .grafana import (
|
||||
create_snapshot_tables,
|
||||
ensure_grafana_schema,
|
||||
insert_account_snapshot,
|
||||
insert_order_snapshots,
|
||||
insert_position_snapshots,
|
||||
insert_terminal_snapshot,
|
||||
record_snapshot_run,
|
||||
start_snapshot_run,
|
||||
)
|
||||
from .history import (
|
||||
create_cash_events_view,
|
||||
create_history_indexes,
|
||||
@@ -30,6 +45,7 @@ from .history import (
|
||||
write_incremental_datasets,
|
||||
)
|
||||
from .retry import retry_with_backoff
|
||||
from .telemetry import get_metrics
|
||||
from .utils import (
|
||||
Dataset,
|
||||
IfExists,
|
||||
@@ -40,7 +56,7 @@ from .utils import (
|
||||
from .utils import coerce_login as _coerce_login
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Iterator, Sequence
|
||||
from collections.abc import Callable, Collection, Iterator, Sequence
|
||||
|
||||
UpdateHistoryBackend = Callable[..., None]
|
||||
|
||||
@@ -49,7 +65,7 @@ T = TypeVar("T")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_RECOVERABLE_HISTORY_UPDATE_ERRORS: tuple[type[BaseException], ...] = (
|
||||
Mt5TradingError,
|
||||
*([Mt5TradingError] if Mt5TradingError is not None else []), # type: ignore[assignment]
|
||||
Mt5RuntimeError,
|
||||
sqlite3.Error,
|
||||
ValueError,
|
||||
@@ -142,12 +158,15 @@ __all__ = [
|
||||
"resolve_account_spec",
|
||||
"resolve_account_specs",
|
||||
"substitute_env_placeholders",
|
||||
"substitute_mapping_values",
|
||||
"symbol_info",
|
||||
"symbol_info_tick",
|
||||
"symbols",
|
||||
"terminal_info",
|
||||
"update_history",
|
||||
"update_history_with_config",
|
||||
"update_observability",
|
||||
"update_observability_with_config",
|
||||
"version",
|
||||
]
|
||||
|
||||
@@ -305,7 +324,7 @@ def _fetch_minimum_margins(client: Mt5DataClient, symbol: str) -> pd.DataFrame:
|
||||
def build_config(
|
||||
*,
|
||||
path: str | None = None,
|
||||
login: int | None = None,
|
||||
login: int | str | None = None,
|
||||
password: str | None = None,
|
||||
server: str | None = None,
|
||||
timeout: int | None = None,
|
||||
@@ -315,14 +334,19 @@ def build_config(
|
||||
|
||||
Args:
|
||||
path: Optional terminal executable path.
|
||||
login: Optional trading account login.
|
||||
login: Optional trading account login. Integers are preserved. String
|
||||
values are coerced: empty or whitespace-only strings become
|
||||
``None``; numeric strings such as ``"12345"`` are converted to
|
||||
``int``; non-numeric strings raise ``ValueError``. When
|
||||
``allow_whole_dollar_env=True``, ``$ENV_NAME`` and
|
||||
``${ENV_NAME}`` placeholders are expanded before coercion.
|
||||
password: Optional trading account password.
|
||||
server: Optional trading server name.
|
||||
timeout: Optional connection timeout in milliseconds.
|
||||
allow_whole_dollar_env: When ``True``, string parameters that are
|
||||
exactly ``$ENV_NAME`` are expanded from the environment. Applies
|
||||
to ``path``, ``password``, and ``server``. Default ``False``
|
||||
preserves existing behavior.
|
||||
to ``path``, ``login``, ``password``, and ``server``. Default
|
||||
``False`` preserves existing behavior.
|
||||
|
||||
Returns:
|
||||
Configured ``Mt5Config`` instance.
|
||||
@@ -330,6 +354,8 @@ def build_config(
|
||||
if allow_whole_dollar_env:
|
||||
if path is not None:
|
||||
path = substitute_env_placeholders(path, allow_whole_dollar_env=True)
|
||||
if isinstance(login, str):
|
||||
login = substitute_env_placeholders(login, allow_whole_dollar_env=True)
|
||||
if password is not None:
|
||||
password = substitute_env_placeholders(
|
||||
password, allow_whole_dollar_env=True
|
||||
@@ -338,7 +364,7 @@ def build_config(
|
||||
server = substitute_env_placeholders(server, allow_whole_dollar_env=True)
|
||||
return Mt5Config(
|
||||
path=path,
|
||||
login=login,
|
||||
login=_coerce_login(login),
|
||||
password=password,
|
||||
server=server,
|
||||
timeout=timeout,
|
||||
@@ -968,7 +994,8 @@ def update_history( # noqa: PLR0913
|
||||
client: Connected MT5 data client.
|
||||
output: SQLite database path.
|
||||
symbols: Symbols to update.
|
||||
datasets: Datasets to include (defaults to all).
|
||||
datasets: Datasets to include (defaults to rates, history-orders,
|
||||
history-deals; pass ``{Dataset.ticks}`` to opt in to ticks).
|
||||
timeframes: Rate timeframes to update (defaults to all fixed MT5
|
||||
timeframes when None).
|
||||
flags: Tick copy flags as integer or name (e.g. ``ALL``).
|
||||
@@ -997,23 +1024,27 @@ def update_history( # noqa: PLR0913
|
||||
sorted(dataset.value for dataset in request.selected),
|
||||
request.output_path,
|
||||
)
|
||||
with sqlite3.connect(request.output_path) as conn:
|
||||
with closing(sqlite3.connect(request.output_path)) as conn, conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA synchronous=NORMAL")
|
||||
write_incremental_datasets(
|
||||
conn,
|
||||
client,
|
||||
symbols,
|
||||
request.selected,
|
||||
request.resolved_timeframes,
|
||||
request.resolved_tick_flags,
|
||||
request.fallback_start,
|
||||
request.end,
|
||||
deduplicate=deduplicate,
|
||||
create_rate_views=create_rate_views,
|
||||
with_views=with_views,
|
||||
include_account_events=include_account_events,
|
||||
)
|
||||
m = get_metrics()
|
||||
with m.record_history_update(dataset="history"):
|
||||
before = conn.total_changes
|
||||
write_incremental_datasets(
|
||||
conn,
|
||||
client,
|
||||
symbols,
|
||||
request.selected,
|
||||
request.resolved_timeframes,
|
||||
request.resolved_tick_flags,
|
||||
request.fallback_start,
|
||||
request.end,
|
||||
deduplicate=deduplicate,
|
||||
create_rate_views=create_rate_views,
|
||||
with_views=with_views,
|
||||
include_account_events=include_account_events,
|
||||
)
|
||||
m.add_history_rows(conn.total_changes - before, dataset="history")
|
||||
|
||||
|
||||
def update_history_with_config( # noqa: PLR0913
|
||||
@@ -1097,7 +1128,8 @@ class ThrottledHistoryUpdater:
|
||||
|
||||
Args:
|
||||
output: SQLite database path.
|
||||
datasets: Datasets to include (defaults to all).
|
||||
datasets: Datasets to include (defaults to rates, history-orders,
|
||||
history-deals; pass ``{Dataset.ticks}`` to opt in to ticks).
|
||||
timeframes: Rate timeframes to update (defaults to all fixed MT5
|
||||
timeframes).
|
||||
flags: Tick copy flags as integer or name (e.g. ``ALL``).
|
||||
@@ -1229,7 +1261,8 @@ def collect_history(
|
||||
symbols: Symbols to collect.
|
||||
date_from: Start date.
|
||||
date_to: End date.
|
||||
datasets: Datasets to include (defaults to all).
|
||||
datasets: Datasets to include (defaults to rates, history-orders,
|
||||
history-deals; pass ``{Dataset.ticks}`` to opt in to ticks).
|
||||
timeframe: Rates timeframe as integer or name (e.g. ``M1``).
|
||||
flags: Tick copy flags as integer or name (e.g. ``ALL``).
|
||||
if_exists: Behavior when a target table already exists.
|
||||
@@ -1238,11 +1271,15 @@ def collect_history(
|
||||
"""
|
||||
start = _require_datetime(date_from)
|
||||
end = _require_datetime(date_to)
|
||||
selected = datasets if datasets is not None else set(Dataset)
|
||||
selected = resolve_history_datasets(datasets)
|
||||
tf = _coerce_timeframe(timeframe)
|
||||
tick_flags = _coerce_tick_flags(flags)
|
||||
mt5_config = config or build_config()
|
||||
with connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
|
||||
with (
|
||||
connected_client(mt5_config) as client,
|
||||
closing(sqlite3.connect(output)) as conn,
|
||||
conn,
|
||||
):
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA synchronous=NORMAL")
|
||||
written_tables, written_columns = write_collected_datasets(
|
||||
@@ -1442,6 +1479,75 @@ def substitute_env_placeholders(
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def substitute_mapping_values(
|
||||
data: object,
|
||||
*,
|
||||
keys: Collection[str],
|
||||
allow_whole_dollar_env: bool = False,
|
||||
blank_string_keys_as_none: Collection[str] = (),
|
||||
) -> object:
|
||||
"""Recursively substitute environment placeholders for selected mapping keys.
|
||||
|
||||
Traverses nested dicts and lists, expanding ``${ENV_VAR}`` (and
|
||||
``$ENV_NAME`` when ``allow_whole_dollar_env=True``) in string values
|
||||
whose immediate parent dict key is in ``keys``. Fields whose key is
|
||||
not in ``keys`` are preserved exactly, including literal dollar signs.
|
||||
Strings that are direct elements of a list are never substituted;
|
||||
substitution only applies to strings that are immediate dict values.
|
||||
|
||||
This is a generic downstream config utility. Key names such as
|
||||
``mt5_login`` or ``mt5_password`` must be supplied by the caller;
|
||||
mt5cli does not hard-code any application-specific key names.
|
||||
Callers are responsible for ensuring ``data`` has bounded nesting depth;
|
||||
deeply nested or self-referential structures will hit Python's recursion
|
||||
limit.
|
||||
|
||||
Args:
|
||||
data: Arbitrarily nested dict/list/scalar value to process.
|
||||
keys: Mapping keys whose string values receive placeholder
|
||||
substitution.
|
||||
allow_whole_dollar_env: When ``True``, a string that is exactly
|
||||
``$ENV_NAME`` (whole value) is also expanded from the
|
||||
environment in addition to ``${ENV_NAME}`` placeholders.
|
||||
Default ``False`` expands ``${ENV_NAME}`` only.
|
||||
blank_string_keys_as_none: Mapping keys for which blank strings
|
||||
(after any substitution) are normalised to ``None``. A key
|
||||
may appear in ``blank_string_keys_as_none`` without also
|
||||
appearing in ``keys``.
|
||||
|
||||
Returns:
|
||||
The processed value. Dicts and lists are rebuilt into new
|
||||
containers with selected string values substituted and
|
||||
blank-normalised. Scalar inputs (non-dict, non-list) are
|
||||
returned as-is.
|
||||
"""
|
||||
keys_set: frozenset[str] = frozenset(keys)
|
||||
blank_keys_set: frozenset[str] = frozenset(blank_string_keys_as_none)
|
||||
|
||||
def _visit(node: object, current_key: str | None) -> object:
|
||||
if isinstance(node, dict):
|
||||
typed = cast("dict[object, object]", node)
|
||||
return {
|
||||
k: _visit(v, k if isinstance(k, str) else None)
|
||||
for k, v in typed.items()
|
||||
}
|
||||
if isinstance(node, list):
|
||||
typed_list = cast("list[object]", node)
|
||||
return [_visit(item, None) for item in typed_list]
|
||||
if not isinstance(node, str):
|
||||
return node
|
||||
text = node
|
||||
if current_key in keys_set:
|
||||
text = substitute_env_placeholders(
|
||||
node, allow_whole_dollar_env=allow_whole_dollar_env
|
||||
)
|
||||
if current_key in blank_keys_set and not text.strip():
|
||||
return None
|
||||
return text
|
||||
|
||||
return _visit(data, None)
|
||||
|
||||
|
||||
def _resolve_field(
|
||||
override: str | None,
|
||||
account_value: str | None,
|
||||
@@ -2063,3 +2169,220 @@ def mt5_summary(*, config: Mt5Config | None = None) -> dict[str, object]:
|
||||
def mt5_summary_as_df(*, config: Mt5Config | None = None) -> pd.DataFrame:
|
||||
"""Return an export-safe terminal/account status summary DataFrame."""
|
||||
return _make_client(config=config).mt5_summary_as_df()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Observability: account / position / order / terminal snapshots
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _emit_account_metrics(row: dict[str, object]) -> None:
|
||||
login = str(row.get("login", ""))
|
||||
server = str(row.get("server", ""))
|
||||
get_metrics().record_account_state(
|
||||
login=login,
|
||||
server=server,
|
||||
balance=float(row.get("balance") or 0.0), # type: ignore[arg-type]
|
||||
equity=float(row.get("equity") or 0.0), # type: ignore[arg-type]
|
||||
margin=float(row.get("margin") or 0.0), # type: ignore[arg-type]
|
||||
margin_free=float(row.get("margin_free") or 0.0), # type: ignore[arg-type]
|
||||
margin_level=float(row.get("margin_level") or 0.0), # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
def _emit_position_metrics(
|
||||
rows: list[dict[str, object]],
|
||||
login: int | None,
|
||||
) -> None:
|
||||
m = get_metrics()
|
||||
login_str = str(login) if login is not None else ""
|
||||
# Aggregate profit and volume by symbol so hedging accounts (multiple open
|
||||
# positions sharing the same symbol) emit a single gauge value per symbol
|
||||
# instead of overwriting with each row's value.
|
||||
totals: dict[str, tuple[float, float]] = {}
|
||||
for r in rows:
|
||||
symbol = str(r.get("symbol", ""))
|
||||
profit = float(r.get("profit") or 0.0) # type: ignore[arg-type]
|
||||
volume = float(r.get("volume") or 0.0) # type: ignore[arg-type]
|
||||
if symbol in totals:
|
||||
prev_p, prev_v = totals[symbol]
|
||||
totals[symbol] = (prev_p + profit, prev_v + volume)
|
||||
else:
|
||||
totals[symbol] = (profit, volume)
|
||||
for symbol, (profit, volume) in totals.items():
|
||||
m.record_position_state(
|
||||
login=login_str,
|
||||
server="",
|
||||
symbol=symbol,
|
||||
profit=profit,
|
||||
volume=volume,
|
||||
)
|
||||
|
||||
|
||||
def _snapshot_account(
|
||||
conn: sqlite3.Connection,
|
||||
client: Mt5DataClient,
|
||||
run_id: int,
|
||||
) -> int | None:
|
||||
df = client.account_info_as_df()
|
||||
if df.empty:
|
||||
logger.warning(
|
||||
"account_info_as_df returned empty frame; skipping account snapshot"
|
||||
)
|
||||
return None
|
||||
row = cast("dict[str, object]", df.iloc[0].to_dict())
|
||||
insert_account_snapshot(conn, run_id, row)
|
||||
_emit_account_metrics(row)
|
||||
login_val = row.get("login")
|
||||
return int(login_val) if login_val is not None else None # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _snapshot_positions(
|
||||
conn: sqlite3.Connection,
|
||||
client: Mt5DataClient,
|
||||
run_id: int,
|
||||
login: int | None,
|
||||
symbols: Sequence[str] | None,
|
||||
) -> None:
|
||||
df: pd.DataFrame = client.positions_get_as_df()
|
||||
if symbols is not None and not df.empty and "symbol" in df.columns:
|
||||
df = df[df["symbol"].isin(symbols)].reset_index(drop=True)
|
||||
raw = df.to_dict(orient="records") if not df.empty else []
|
||||
rows = cast("list[dict[str, object]]", raw)
|
||||
insert_position_snapshots(conn, run_id, login, rows)
|
||||
_emit_position_metrics(rows, login)
|
||||
|
||||
|
||||
def _snapshot_orders(
|
||||
conn: sqlite3.Connection,
|
||||
client: Mt5DataClient,
|
||||
run_id: int,
|
||||
login: int | None,
|
||||
symbols: Sequence[str] | None,
|
||||
) -> None:
|
||||
df: pd.DataFrame = client.orders_get_as_df()
|
||||
if symbols is not None and not df.empty and "symbol" in df.columns:
|
||||
df = df[df["symbol"].isin(symbols)].reset_index(drop=True)
|
||||
raw = df.to_dict(orient="records") if not df.empty else []
|
||||
rows = cast("list[dict[str, object]]", raw)
|
||||
insert_order_snapshots(conn, run_id, login, rows)
|
||||
|
||||
|
||||
def _emit_terminal_metrics(row: dict[str, object]) -> None:
|
||||
get_metrics().record_terminal_state(
|
||||
connected=float(row.get("connected") or 0.0), # type: ignore[arg-type]
|
||||
trade_allowed=float(row.get("trade_allowed") or 0.0), # type: ignore[arg-type]
|
||||
trade_expert=float(row.get("trade_expert") or 0.0), # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
def _snapshot_terminal(
|
||||
conn: sqlite3.Connection,
|
||||
client: Mt5DataClient,
|
||||
run_id: int,
|
||||
) -> None:
|
||||
df = client.terminal_info_as_df()
|
||||
if df.empty:
|
||||
logger.warning(
|
||||
"terminal_info_as_df returned empty frame; skipping terminal snapshot"
|
||||
)
|
||||
return
|
||||
row = cast("dict[str, object]", df.iloc[0].to_dict())
|
||||
insert_terminal_snapshot(conn, run_id, row)
|
||||
_emit_terminal_metrics(row)
|
||||
|
||||
|
||||
def update_observability(
|
||||
*,
|
||||
client: Mt5DataClient,
|
||||
output: Path | str,
|
||||
symbols: Sequence[str] | None = None,
|
||||
include_account: bool = True,
|
||||
include_positions: bool = True,
|
||||
include_orders: bool = True,
|
||||
include_terminal: bool = True,
|
||||
with_grafana_schema: bool = False,
|
||||
) -> None:
|
||||
"""Snapshot current account/position/order/terminal state into SQLite.
|
||||
|
||||
Reads the current MT5 state and appends timestamped snapshot rows. Never
|
||||
places orders or modifies trading state.
|
||||
|
||||
Args:
|
||||
client: Connected MT5 data client.
|
||||
output: SQLite database path.
|
||||
symbols: Optional symbol filter for positions and orders. When None,
|
||||
all positions and orders are snapshotted.
|
||||
include_account: Snapshot account info into ``account_snapshots``.
|
||||
include_positions: Snapshot open positions into ``position_snapshots``.
|
||||
include_orders: Snapshot active orders into ``order_snapshots``.
|
||||
include_terminal: Snapshot terminal info into ``terminal_snapshots``.
|
||||
with_grafana_schema: Ensure Grafana views and indexes exist. Defaults
|
||||
to ``False``; run ``grafana-schema`` once to set up the schema,
|
||||
then use ``snapshot`` repeatedly without this flag.
|
||||
"""
|
||||
observed_at = int(datetime.now(UTC).timestamp())
|
||||
with closing(sqlite3.connect(Path(output))) as conn, conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA synchronous=NORMAL")
|
||||
if with_grafana_schema:
|
||||
ensure_grafana_schema(conn)
|
||||
else:
|
||||
create_snapshot_tables(conn)
|
||||
with get_metrics().record_snapshot_update():
|
||||
run_id = start_snapshot_run(conn, observed_at)
|
||||
login: int | None = None
|
||||
try:
|
||||
if include_account:
|
||||
login = _snapshot_account(conn, client, run_id)
|
||||
if include_positions:
|
||||
_snapshot_positions(conn, client, run_id, login, symbols)
|
||||
if include_orders:
|
||||
_snapshot_orders(conn, client, run_id, login, symbols)
|
||||
if include_terminal:
|
||||
_snapshot_terminal(conn, client, run_id)
|
||||
record_snapshot_run(conn, run_id, "ok")
|
||||
except Exception:
|
||||
record_snapshot_run(conn, run_id, "error")
|
||||
conn.commit()
|
||||
raise
|
||||
|
||||
|
||||
def update_observability_with_config(
|
||||
*,
|
||||
output: Path | str,
|
||||
config: Mt5Config | None = None,
|
||||
symbols: Sequence[str] | None = None,
|
||||
include_account: bool = True,
|
||||
include_positions: bool = True,
|
||||
include_orders: bool = True,
|
||||
include_terminal: bool = True,
|
||||
with_grafana_schema: bool = False,
|
||||
) -> None:
|
||||
"""Snapshot current MT5 state, opening and closing the MT5 connection.
|
||||
|
||||
Convenience wrapper around :func:`update_observability` for standalone use.
|
||||
|
||||
Args:
|
||||
output: SQLite database path.
|
||||
config: MT5 connection configuration. Defaults to an empty config that
|
||||
attaches to a running terminal.
|
||||
symbols: Optional symbol filter for positions and orders.
|
||||
include_account: Snapshot account info.
|
||||
include_positions: Snapshot open positions.
|
||||
include_orders: Snapshot active orders.
|
||||
include_terminal: Snapshot terminal info.
|
||||
with_grafana_schema: Ensure Grafana views and indexes exist.
|
||||
"""
|
||||
mt5_config = config or build_config()
|
||||
with connected_client(mt5_config) as client:
|
||||
update_observability(
|
||||
client=client,
|
||||
output=output,
|
||||
symbols=symbols,
|
||||
include_account=include_account,
|
||||
include_positions=include_positions,
|
||||
include_orders=include_orders,
|
||||
include_terminal=include_terminal,
|
||||
with_grafana_schema=with_grafana_schema,
|
||||
)
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
"""Generic storage helpers for MT5 market and account history."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .history import (
|
||||
RateTarget,
|
||||
build_rate_targets,
|
||||
build_rate_view_name,
|
||||
drop_forming_rate_bar,
|
||||
load_rate_data,
|
||||
load_rate_data_from_connection,
|
||||
load_rate_series_by_granularity,
|
||||
load_rate_series_from_sqlite,
|
||||
resolve_rate_tables,
|
||||
resolve_rate_view_name,
|
||||
resolve_rate_view_names,
|
||||
)
|
||||
from .sdk import collect_history, update_history, update_history_with_config
|
||||
from .utils import (
|
||||
Dataset,
|
||||
IfExists,
|
||||
OutputFormat,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"Dataset",
|
||||
"IfExists",
|
||||
"OutputFormat",
|
||||
"RateTarget",
|
||||
"build_rate_targets",
|
||||
"build_rate_view_name",
|
||||
"collect_history",
|
||||
"detect_format",
|
||||
"drop_forming_rate_bar",
|
||||
"export_dataframe",
|
||||
"export_dataframe_to_sqlite",
|
||||
"load_rate_data",
|
||||
"load_rate_data_from_connection",
|
||||
"load_rate_series_by_granularity",
|
||||
"load_rate_series_from_sqlite",
|
||||
"resolve_rate_tables",
|
||||
"resolve_rate_view_name",
|
||||
"resolve_rate_view_names",
|
||||
"update_history",
|
||||
"update_history_with_config",
|
||||
]
|
||||
@@ -0,0 +1,354 @@
|
||||
"""Optional OpenTelemetry metrics for MT5 history and snapshot observability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_otel_available_flag = False
|
||||
|
||||
try:
|
||||
import opentelemetry.metrics as _otel_metrics_mod
|
||||
from opentelemetry.sdk.metrics import MeterProvider as _OtelMeterProvider
|
||||
from opentelemetry.sdk.metrics.export import (
|
||||
PeriodicExportingMetricReader as _OtelPeriodicReader,
|
||||
)
|
||||
from opentelemetry.sdk.resources import Resource as _OtelResource
|
||||
|
||||
_otel_available_flag = True
|
||||
except ImportError: # pragma: no cover
|
||||
_otel_metrics_mod = None # type: ignore[assignment]
|
||||
_OtelMeterProvider = None # type: ignore[assignment]
|
||||
_OtelPeriodicReader = None # type: ignore[assignment]
|
||||
_OtelResource = None # type: ignore[assignment]
|
||||
|
||||
_OTEL_AVAILABLE: bool = _otel_available_flag
|
||||
|
||||
try:
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import ( # type: ignore[import]
|
||||
OTLPMetricExporter as _OtelOTLPExporter, # type: ignore[reportUnknownVariableType]
|
||||
)
|
||||
except ImportError: # pragma: no cover
|
||||
_OtelOTLPExporter = None # type: ignore[assignment, misc]
|
||||
|
||||
|
||||
class _NoOp:
|
||||
"""No-op instrument that silently ignores all calls."""
|
||||
|
||||
def add(
|
||||
self,
|
||||
amount: float,
|
||||
attributes: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
"""No-op add."""
|
||||
|
||||
def set(
|
||||
self,
|
||||
amount: float,
|
||||
attributes: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
"""No-op set."""
|
||||
|
||||
def record(
|
||||
self,
|
||||
amount: float,
|
||||
attributes: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
"""No-op record."""
|
||||
|
||||
|
||||
_NOOP: _NoOp = _NoOp()
|
||||
|
||||
|
||||
class _Mt5Metrics:
|
||||
"""MT5 metric instrument registry.
|
||||
|
||||
Holds references to OTel instruments. All instruments are no-op until
|
||||
:meth:`configure` is called with a compatible meter object.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._history_duration: Any = _NOOP
|
||||
self._history_rows: Any = _NOOP
|
||||
self._history_failures: Any = _NOOP
|
||||
self._snapshot_duration: Any = _NOOP
|
||||
self._snapshot_failures: Any = _NOOP
|
||||
self._account_balance: Any = _NOOP
|
||||
self._account_equity: Any = _NOOP
|
||||
self._account_margin: Any = _NOOP
|
||||
self._account_margin_free: Any = _NOOP
|
||||
self._account_margin_level: Any = _NOOP
|
||||
self._position_profit: Any = _NOOP
|
||||
self._position_volume: Any = _NOOP
|
||||
self._terminal_connected: Any = _NOOP
|
||||
self._terminal_trade_allowed: Any = _NOOP
|
||||
self._terminal_trade_expert: Any = _NOOP
|
||||
self._last_successful_update: Any = _NOOP
|
||||
|
||||
def configure(self, meter: Any) -> None: # noqa: ANN401
|
||||
"""Set up metric instruments from a meter object.
|
||||
|
||||
Args:
|
||||
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object
|
||||
that supports ``create_counter``, ``create_histogram``, and
|
||||
``create_gauge``.
|
||||
"""
|
||||
self._history_duration = meter.create_histogram(
|
||||
"mt5_history_update_duration_seconds",
|
||||
unit="s",
|
||||
description="Duration of incremental history update operations.",
|
||||
)
|
||||
self._history_rows = meter.create_counter(
|
||||
"mt5_history_update_rows_total",
|
||||
description="Rows written during incremental history updates.",
|
||||
)
|
||||
self._history_failures = meter.create_counter(
|
||||
"mt5_history_update_failures_total",
|
||||
description="Number of incremental history update failures.",
|
||||
)
|
||||
self._snapshot_duration = meter.create_histogram(
|
||||
"mt5_snapshot_update_duration_seconds",
|
||||
unit="s",
|
||||
description="Duration of snapshot update operations.",
|
||||
)
|
||||
self._snapshot_failures = meter.create_counter(
|
||||
"mt5_snapshot_update_failures_total",
|
||||
description="Number of snapshot update failures.",
|
||||
)
|
||||
self._account_balance = meter.create_gauge(
|
||||
"mt5_account_balance",
|
||||
description="Account balance.",
|
||||
)
|
||||
self._account_equity = meter.create_gauge(
|
||||
"mt5_account_equity",
|
||||
description="Account equity.",
|
||||
)
|
||||
self._account_margin = meter.create_gauge(
|
||||
"mt5_account_margin",
|
||||
description="Account margin used.",
|
||||
)
|
||||
self._account_margin_free = meter.create_gauge(
|
||||
"mt5_account_margin_free",
|
||||
description="Account free margin.",
|
||||
)
|
||||
self._account_margin_level = meter.create_gauge(
|
||||
"mt5_account_margin_level",
|
||||
description="Account margin level as a percentage.",
|
||||
)
|
||||
self._position_profit = meter.create_gauge(
|
||||
"mt5_position_profit",
|
||||
description="Floating profit for an open position.",
|
||||
)
|
||||
self._position_volume = meter.create_gauge(
|
||||
"mt5_position_volume",
|
||||
description="Volume of an open position.",
|
||||
)
|
||||
self._terminal_connected = meter.create_gauge(
|
||||
"mt5_terminal_connected",
|
||||
description="1 if the terminal is connected to the broker, 0 otherwise.",
|
||||
)
|
||||
self._terminal_trade_allowed = meter.create_gauge(
|
||||
"mt5_terminal_trade_allowed",
|
||||
description="1 if trading is allowed by the broker server, 0 otherwise.",
|
||||
)
|
||||
self._terminal_trade_expert = meter.create_gauge(
|
||||
"mt5_terminal_trade_expert",
|
||||
description="1 if Expert Advisor trading is enabled, 0 otherwise.",
|
||||
)
|
||||
self._last_successful_update = meter.create_gauge(
|
||||
"mt5_last_successful_update_timestamp",
|
||||
description="Unix timestamp of the last successful history update.",
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def record_history_update(
|
||||
self,
|
||||
*,
|
||||
dataset: str,
|
||||
) -> Iterator[None]:
|
||||
"""Context manager recording history update duration and failures.
|
||||
|
||||
Args:
|
||||
dataset: Dataset label (e.g. ``"rates"``).
|
||||
|
||||
Yields:
|
||||
None inside the update operation.
|
||||
"""
|
||||
attrs = {"dataset": dataset}
|
||||
start = time.monotonic()
|
||||
try:
|
||||
yield
|
||||
self._history_duration.record(time.monotonic() - start, attrs)
|
||||
self._last_successful_update.set(time.time(), attrs)
|
||||
except Exception:
|
||||
self._history_failures.add(1, attrs)
|
||||
raise
|
||||
|
||||
def add_history_rows(self, count: int, *, dataset: str) -> None:
|
||||
"""Increment the history rows-written counter.
|
||||
|
||||
Args:
|
||||
count: Number of rows written during this update.
|
||||
dataset: Dataset label (e.g. ``"rates"``).
|
||||
"""
|
||||
self._history_rows.add(count, {"dataset": dataset})
|
||||
|
||||
@contextmanager
|
||||
def record_snapshot_update(self) -> Iterator[None]:
|
||||
"""Context manager recording snapshot update duration and failures.
|
||||
|
||||
Yields:
|
||||
None inside the snapshot operation.
|
||||
"""
|
||||
start = time.monotonic()
|
||||
try:
|
||||
yield
|
||||
self._snapshot_duration.record(time.monotonic() - start, {})
|
||||
except Exception:
|
||||
self._snapshot_failures.add(1, {})
|
||||
raise
|
||||
|
||||
def record_account_state(
|
||||
self,
|
||||
*,
|
||||
login: str,
|
||||
server: str,
|
||||
balance: float,
|
||||
equity: float,
|
||||
margin: float,
|
||||
margin_free: float,
|
||||
margin_level: float,
|
||||
) -> None:
|
||||
"""Emit account metric gauges.
|
||||
|
||||
Args:
|
||||
login: Account login number (as string; not a password or secret).
|
||||
server: Broker server name.
|
||||
balance: Account balance.
|
||||
equity: Account equity.
|
||||
margin: Margin used.
|
||||
margin_free: Free margin.
|
||||
margin_level: Margin level percentage.
|
||||
"""
|
||||
attrs: dict[str, str] = {"login": login, "server": server}
|
||||
self._account_balance.set(balance, attrs)
|
||||
self._account_equity.set(equity, attrs)
|
||||
self._account_margin.set(margin, attrs)
|
||||
self._account_margin_free.set(margin_free, attrs)
|
||||
self._account_margin_level.set(margin_level, attrs)
|
||||
|
||||
def record_position_state(
|
||||
self,
|
||||
*,
|
||||
login: str,
|
||||
server: str,
|
||||
symbol: str,
|
||||
profit: float,
|
||||
volume: float,
|
||||
) -> None:
|
||||
"""Emit position metric gauges.
|
||||
|
||||
Args:
|
||||
login: Account login number (as string).
|
||||
server: Broker server name.
|
||||
symbol: Position symbol.
|
||||
profit: Floating profit/loss.
|
||||
volume: Position volume.
|
||||
"""
|
||||
attrs: dict[str, str] = {"login": login, "server": server, "symbol": symbol}
|
||||
self._position_profit.set(profit, attrs)
|
||||
self._position_volume.set(volume, attrs)
|
||||
|
||||
def record_terminal_state(
|
||||
self,
|
||||
*,
|
||||
connected: float,
|
||||
trade_allowed: float,
|
||||
trade_expert: float,
|
||||
) -> None:
|
||||
"""Emit terminal connection and trading status gauges.
|
||||
|
||||
Args:
|
||||
connected: 1.0 if connected to the broker, 0.0 otherwise.
|
||||
trade_allowed: 1.0 if broker server allows trading, 0.0 otherwise.
|
||||
trade_expert: 1.0 if Expert Advisor trading is enabled, 0.0 otherwise.
|
||||
"""
|
||||
self._terminal_connected.set(connected, {})
|
||||
self._terminal_trade_allowed.set(trade_allowed, {})
|
||||
self._terminal_trade_expert.set(trade_expert, {})
|
||||
|
||||
|
||||
_metrics = _Mt5Metrics()
|
||||
|
||||
|
||||
def configure_metrics(meter: Any) -> None: # noqa: ANN401
|
||||
"""Configure MT5 metrics using the provided meter.
|
||||
|
||||
Args:
|
||||
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object.
|
||||
"""
|
||||
_metrics.configure(meter)
|
||||
|
||||
|
||||
def enable_otel_metrics(
|
||||
service_name: str = "mt5cli",
|
||||
readers: list[Any] | None = None,
|
||||
) -> None:
|
||||
"""Enable OTel metrics by wiring up an SDK ``MeterProvider`` pipeline.
|
||||
|
||||
Requires the ``otel`` optional dependency group:
|
||||
``pip install "mt5cli[otel]"``.
|
||||
|
||||
Args:
|
||||
service_name: OTel meter/service name used for the ``Resource`` and
|
||||
the meter itself.
|
||||
readers: Optional list of metric readers. When *None* (the default),
|
||||
a :class:`~opentelemetry.sdk.metrics.export.PeriodicExportingMetricReader`
|
||||
backed by an OTLP HTTP exporter is created automatically
|
||||
(reads the endpoint from ``OTEL_EXPORTER_OTLP_ENDPOINT``).
|
||||
Pass a custom list (e.g. ``InMemoryMetricReader`` for tests)
|
||||
to override.
|
||||
|
||||
Raises:
|
||||
ImportError: If ``opentelemetry-api`` is not installed, or if
|
||||
``readers`` is *None* and
|
||||
``opentelemetry-exporter-otlp-proto-http`` is not installed.
|
||||
"""
|
||||
if not _OTEL_AVAILABLE:
|
||||
msg = (
|
||||
"opentelemetry-api is not installed. "
|
||||
'Install it with: pip install "mt5cli[otel]"'
|
||||
)
|
||||
raise ImportError(msg)
|
||||
if readers is None:
|
||||
if _OtelOTLPExporter is None:
|
||||
msg = (
|
||||
"opentelemetry-exporter-otlp-proto-http is required for the "
|
||||
"default OTLP export pipeline. "
|
||||
'Install it with: pip install "mt5cli[otel]" or pass a '
|
||||
"custom readers list."
|
||||
)
|
||||
raise ImportError(msg)
|
||||
readers = [_OtelPeriodicReader(_OtelOTLPExporter())] # type: ignore[misc]
|
||||
resource = _OtelResource.create({"service.name": service_name}) # type: ignore[union-attr]
|
||||
provider = _OtelMeterProvider(resource=resource, metric_readers=readers) # type: ignore[misc]
|
||||
_otel_metrics_mod.set_meter_provider(provider) # type: ignore[union-attr]
|
||||
meter = provider.get_meter(service_name)
|
||||
configure_metrics(meter)
|
||||
|
||||
|
||||
def get_metrics() -> _Mt5Metrics:
|
||||
"""Return the global :class:`_Mt5Metrics` instance.
|
||||
|
||||
Returns:
|
||||
The global metric registry (no-op until :func:`configure_metrics` is
|
||||
called).
|
||||
"""
|
||||
return _metrics
|
||||
+446
-96
@@ -6,11 +6,12 @@ import logging
|
||||
from contextlib import contextmanager
|
||||
from math import floor, isfinite
|
||||
from numbers import Integral, Real
|
||||
from typing import TYPE_CHECKING, Literal, TypedDict, cast
|
||||
from typing import TYPE_CHECKING, Literal, Protocol, TypedDict, cast
|
||||
|
||||
import pandas as pd
|
||||
from pdmt5 import Mt5Config, Mt5RuntimeError, Mt5TradingClient, Mt5TradingError
|
||||
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError
|
||||
|
||||
from .exceptions import Mt5OperationError
|
||||
from .history import drop_forming_rate_bar
|
||||
from .sdk import build_config
|
||||
from .utils import coerce_login as _coerce_login
|
||||
@@ -18,14 +19,73 @@ from .utils import parse_timeframe
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _Mt5ClientProtocol(Protocol):
|
||||
"""Minimal protocol for MT5 clients with methods required by mt5cli.
|
||||
|
||||
This protocol describes the interface required by mt5cli trading helpers.
|
||||
It uses positional-only parameters to avoid structural subtyping issues with
|
||||
different client implementations that may use different parameter names.
|
||||
"""
|
||||
|
||||
@property
|
||||
def mt5(self) -> Any: # noqa: ANN401
|
||||
"""MT5 module with trading constants (POSITION_TYPE_*, ORDER_TYPE_*, etc.)."""
|
||||
...
|
||||
|
||||
def account_info_as_dict(self) -> dict[str, Any]:
|
||||
"""Return account information as a dictionary."""
|
||||
...
|
||||
|
||||
def symbol_info(self, symbol: str, /) -> object:
|
||||
"""Return symbol information."""
|
||||
...
|
||||
|
||||
def symbol_info_tick(self, symbol: str, /) -> object:
|
||||
"""Return latest symbol tick information."""
|
||||
...
|
||||
|
||||
def positions_get_as_df(self, symbol: str | None = None) -> pd.DataFrame:
|
||||
"""Return open positions as a DataFrame."""
|
||||
...
|
||||
|
||||
def order_calc_margin(
|
||||
self, /, action: int, symbol: str, volume: float, price: float
|
||||
) -> Any: # noqa: ANN401
|
||||
"""Calculate required margin for an order."""
|
||||
...
|
||||
|
||||
def order_send(self, request: dict[str, Any], /) -> Any: # noqa: ANN401
|
||||
"""Send an order request and return the response."""
|
||||
...
|
||||
|
||||
def symbol_select(self, symbol: str, enable: bool = True) -> bool:
|
||||
"""Select/deselect a symbol in Market Watch."""
|
||||
...
|
||||
|
||||
def last_error(self) -> object:
|
||||
"""Return the last error message or info."""
|
||||
...
|
||||
|
||||
def shutdown(self) -> None:
|
||||
"""Shut down the MT5 client."""
|
||||
...
|
||||
|
||||
def initialize_and_login_mt5(self) -> None:
|
||||
"""Initialize and login to MT5."""
|
||||
...
|
||||
|
||||
|
||||
PositionSide = Literal["long", "short"]
|
||||
OrderSide = Literal["BUY", "SELL"]
|
||||
OrderFillingMode = Literal["IOC", "FOK", "RETURN"]
|
||||
OrderTimeMode = Literal["GTC", "DAY", "SPECIFIED", "SPECIFIED_DAY"]
|
||||
ExecutionStatus = Literal["executed", "dry_run", "skipped", "failed"]
|
||||
ProjectionMode = Literal["add", "replace_symbol"]
|
||||
|
||||
|
||||
class MarginVolume(TypedDict):
|
||||
@@ -128,12 +188,17 @@ __all__ = [
|
||||
"OrderSide",
|
||||
"OrderTimeMode",
|
||||
"PositionSide",
|
||||
"ProjectionMode",
|
||||
"calculate_account_projected_margin_ratio",
|
||||
"calculate_margin_and_volume",
|
||||
"calculate_new_position_margin_ratio",
|
||||
"calculate_positions_margin",
|
||||
"calculate_positions_margin_by_symbol",
|
||||
"calculate_positions_margin_safe",
|
||||
"calculate_projected_margin_ratio",
|
||||
"calculate_spread_ratio",
|
||||
"calculate_symbol_group_margin_ratio",
|
||||
"calculate_trailing_stop_updates",
|
||||
"calculate_volume_by_margin",
|
||||
"close_open_positions",
|
||||
"create_trading_client",
|
||||
@@ -141,6 +206,7 @@ __all__ = [
|
||||
"determine_order_limits",
|
||||
"ensure_symbol_selected",
|
||||
"estimate_order_margin",
|
||||
"extract_tick_price",
|
||||
"fetch_latest_closed_rates_for_trading_client",
|
||||
"fetch_latest_closed_rates_indexed",
|
||||
"get_account_snapshot",
|
||||
@@ -151,6 +217,7 @@ __all__ = [
|
||||
"normalize_order_volume",
|
||||
"place_market_order",
|
||||
"update_sltp_for_open_positions",
|
||||
"update_trailing_stop_loss_for_open_positions",
|
||||
]
|
||||
|
||||
|
||||
@@ -181,7 +248,7 @@ def _validate_protective_prices(
|
||||
"""Validate SL/TP distances against broker stop-level constraints.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: When a protective price is closer than ``min_distance``.
|
||||
Mt5OperationError: When a protective price is closer than ``min_distance``.
|
||||
"""
|
||||
if min_distance <= 0:
|
||||
return
|
||||
@@ -191,37 +258,37 @@ def _validate_protective_prices(
|
||||
f"Stop loss for {symbol!r} violates broker stop level "
|
||||
f"(minimum distance {min_distance})."
|
||||
)
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
if take_profit is not None and (take_profit - entry) < min_distance:
|
||||
msg = (
|
||||
f"Take profit for {symbol!r} violates broker stop level "
|
||||
f"(minimum distance {min_distance})."
|
||||
)
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
return
|
||||
if stop_loss is not None and (stop_loss - entry) < min_distance:
|
||||
msg = (
|
||||
f"Stop loss for {symbol!r} violates broker stop level "
|
||||
f"(minimum distance {min_distance})."
|
||||
)
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
if take_profit is not None and (entry - take_profit) < min_distance:
|
||||
msg = (
|
||||
f"Take profit for {symbol!r} violates broker stop level "
|
||||
f"(minimum distance {min_distance})."
|
||||
)
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
|
||||
|
||||
def ensure_symbol_selected(client: Mt5TradingClient, symbol: str) -> None:
|
||||
def ensure_symbol_selected(client: _Mt5ClientProtocol, symbol: str) -> None:
|
||||
"""Ensure a symbol is visible in Market Watch before sending orders.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbol: Symbol to select.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If the symbol cannot be selected in Market Watch or
|
||||
Mt5OperationError: If the symbol cannot be selected in Market Watch or
|
||||
``symbol_select`` is unavailable on the client.
|
||||
"""
|
||||
snapshot = get_symbol_snapshot(client, symbol)
|
||||
@@ -230,13 +297,13 @@ def ensure_symbol_selected(client: Mt5TradingClient, symbol: str) -> None:
|
||||
select = getattr(client, "symbol_select", None)
|
||||
if not callable(select):
|
||||
msg = "MT5 client is missing required method: symbol_select"
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
if select(symbol, enable=True):
|
||||
return
|
||||
last_error = getattr(client, "last_error", None)
|
||||
detail = f" ({last_error()})" if callable(last_error) else ""
|
||||
msg = f"Failed to select symbol {symbol!r} in Market Watch{detail}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
|
||||
|
||||
def _require_unit_ratio(value: float, name: str) -> None:
|
||||
@@ -363,7 +430,7 @@ def _snapshot_from_value(value: object, fields: tuple[str, ...]) -> dict[str, ob
|
||||
return {field: row.get(field) for field in fields}
|
||||
|
||||
|
||||
def _call_snapshot_method(client: Mt5TradingClient, *names: str) -> object:
|
||||
def _call_snapshot_method(client: _Mt5ClientProtocol, *names: str) -> object:
|
||||
for name in names:
|
||||
method = getattr(client, name, None)
|
||||
if callable(method):
|
||||
@@ -387,7 +454,7 @@ def _resolve_mt5_constant(
|
||||
return cast("int", getattr(mt5, name))
|
||||
except AttributeError as exc:
|
||||
msg = f"MT5 module is missing required constant: {name}"
|
||||
raise Mt5TradingError(msg) from exc
|
||||
raise Mt5OperationError(msg) from exc
|
||||
|
||||
|
||||
def _parse_digit_string(value: str) -> int | None:
|
||||
@@ -428,7 +495,7 @@ def _optional_price(value: object) -> float | None:
|
||||
return price
|
||||
|
||||
|
||||
def _valid_tick_price(tick: Mapping[str, object], key: str) -> float | None:
|
||||
def extract_tick_price(tick: Mapping[str, object], key: str) -> float | None:
|
||||
"""Return a positive finite float from tick[key], or None if invalid.
|
||||
|
||||
Accepts int, float, or numeric string values. Returns None when the key is
|
||||
@@ -471,7 +538,7 @@ def _order_status_from_retcode(mt5: object, retcode: object) -> ExecutionStatus:
|
||||
|
||||
|
||||
def _calculate_min_volume_if_affordable(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
available_margin: float,
|
||||
order_side: OrderSide,
|
||||
@@ -488,14 +555,14 @@ def _calculate_min_volume_if_affordable(
|
||||
or (volume_max > 0 and volume_min > volume_max)
|
||||
):
|
||||
msg = f"Invalid volume constraints for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
side = _normalize_order_side(order_side)
|
||||
price = _valid_tick_price(
|
||||
price = extract_tick_price(
|
||||
get_tick_snapshot(client, symbol), "ask" if side == "BUY" else "bid"
|
||||
)
|
||||
if price is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
order_type = (
|
||||
client.mt5.ORDER_TYPE_BUY if side == "BUY" else client.mt5.ORDER_TYPE_SELL
|
||||
)
|
||||
@@ -512,8 +579,12 @@ def create_trading_client(
|
||||
path: str | None = None,
|
||||
timeout: int | None = None,
|
||||
retry_count: int = 0,
|
||||
) -> Mt5TradingClient:
|
||||
"""Return an initialized and logged-in trading client."""
|
||||
) -> _Mt5ClientProtocol:
|
||||
"""Return an initialized and logged-in trading client.
|
||||
|
||||
Returns:
|
||||
A client instance supporting the required MT5 trading methods.
|
||||
"""
|
||||
mt5_config = _resolve_config(
|
||||
config=config,
|
||||
login=login,
|
||||
@@ -522,7 +593,7 @@ def create_trading_client(
|
||||
path=path,
|
||||
timeout=timeout,
|
||||
)
|
||||
client = Mt5TradingClient(config=mt5_config, retry_count=retry_count)
|
||||
client = Mt5DataClient(config=mt5_config, retry_count=retry_count)
|
||||
try:
|
||||
client.initialize_and_login_mt5()
|
||||
except Exception:
|
||||
@@ -532,13 +603,13 @@ def create_trading_client(
|
||||
|
||||
|
||||
def detect_position_side(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
) -> PositionSide | None:
|
||||
"""Detect the net open position side for a symbol.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbol: Symbol to inspect.
|
||||
|
||||
Returns:
|
||||
@@ -562,7 +633,7 @@ def detect_position_side(
|
||||
|
||||
|
||||
def get_account_snapshot(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
) -> dict[str, float | int | str | None]:
|
||||
"""Return normalized account state with stable keys."""
|
||||
value = _call_snapshot_method(client, "account_info_as_dict", "account_info")
|
||||
@@ -573,7 +644,7 @@ def get_account_snapshot(
|
||||
|
||||
|
||||
def get_symbol_snapshot(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
) -> dict[str, float | int | str | bool | None]:
|
||||
"""Return normalized symbol metadata required for trading decisions."""
|
||||
@@ -585,7 +656,7 @@ def get_symbol_snapshot(
|
||||
|
||||
|
||||
def get_tick_snapshot(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
) -> dict[str, float | int | None]:
|
||||
"""Return normalized latest tick data, including bid, ask, and timestamp."""
|
||||
@@ -599,7 +670,7 @@ def get_tick_snapshot(
|
||||
|
||||
|
||||
def get_positions_frame(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Return open positions as a DataFrame with stable baseline columns."""
|
||||
@@ -611,7 +682,7 @@ def get_positions_frame(
|
||||
|
||||
|
||||
def _order_side_from_position_type(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
position_type: object,
|
||||
) -> OrderSide | None:
|
||||
if position_type == client.mt5.POSITION_TYPE_BUY:
|
||||
@@ -633,7 +704,7 @@ def _ensure_rate_time_column(frame: pd.DataFrame) -> pd.DataFrame:
|
||||
|
||||
|
||||
def estimate_order_margin(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
order_side: OrderSide | str,
|
||||
volume: float,
|
||||
@@ -644,17 +715,17 @@ def estimate_order_margin(
|
||||
Positive finite margin required for the order at the current quote.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If volume, tick data, or margin estimation is invalid.
|
||||
Mt5OperationError: If volume, tick data, or margin estimation is invalid.
|
||||
"""
|
||||
if not _is_positive_finite_number(volume):
|
||||
msg = "Volume must be a positive finite number to estimate order margin."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
side = _normalize_order_side(order_side)
|
||||
tick = get_tick_snapshot(client, symbol)
|
||||
price = _valid_tick_price(tick, "ask" if side == "BUY" else "bid")
|
||||
price = extract_tick_price(tick, "ask" if side == "BUY" else "bid")
|
||||
if price is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
order_type = (
|
||||
client.mt5.ORDER_TYPE_BUY if side == "BUY" else client.mt5.ORDER_TYPE_SELL
|
||||
)
|
||||
@@ -663,22 +734,22 @@ def estimate_order_margin(
|
||||
margin = float(raw_margin)
|
||||
except (TypeError, ValueError) as exc:
|
||||
msg = f"Margin estimate is invalid for {symbol!r}."
|
||||
raise Mt5TradingError(msg) from exc
|
||||
raise Mt5OperationError(msg) from exc
|
||||
if margin <= 0 or not isfinite(margin):
|
||||
msg = f"Margin estimate is invalid for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
return margin
|
||||
|
||||
|
||||
def calculate_positions_margin(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbols: Sequence[str] | None = None,
|
||||
) -> float:
|
||||
"""Return the sum of estimated current margin for open positions.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbols: Optional symbol filter. When omitted, all open positions are
|
||||
included.
|
||||
|
||||
@@ -713,7 +784,7 @@ def calculate_positions_margin(
|
||||
|
||||
|
||||
def calculate_positions_margin_by_symbol(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbols: Sequence[str],
|
||||
suppress_errors: bool = True,
|
||||
@@ -725,10 +796,10 @@ def calculate_positions_margin_by_symbol(
|
||||
first-seen order.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbols: Symbols to compute margin for.
|
||||
suppress_errors: When ``True``, log and skip symbols that raise
|
||||
``Mt5TradingError``, ``Mt5RuntimeError``, or ``AttributeError``.
|
||||
``Mt5OperationError``, ``Mt5RuntimeError``, or ``AttributeError``.
|
||||
When ``False``, re-raise the first failure.
|
||||
|
||||
Returns:
|
||||
@@ -737,7 +808,7 @@ def calculate_positions_margin_by_symbol(
|
||||
with ``suppress_errors=True``.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: When a symbol raises ``Mt5TradingError`` and
|
||||
Mt5OperationError: When a symbol raises ``Mt5OperationError`` and
|
||||
``suppress_errors=False``.
|
||||
Mt5RuntimeError: When a symbol raises ``Mt5RuntimeError`` and
|
||||
``suppress_errors=False``.
|
||||
@@ -748,7 +819,7 @@ def calculate_positions_margin_by_symbol(
|
||||
for symbol in dict.fromkeys(symbols):
|
||||
try:
|
||||
result[symbol] = calculate_positions_margin(client, symbols=[symbol])
|
||||
except (Mt5TradingError, Mt5RuntimeError, AttributeError) as exc:
|
||||
except (Mt5OperationError, Mt5RuntimeError, AttributeError) as exc:
|
||||
if not suppress_errors:
|
||||
raise
|
||||
_logger.warning("Skipping margin for %r: %s", symbol, exc)
|
||||
@@ -756,7 +827,7 @@ def calculate_positions_margin_by_symbol(
|
||||
|
||||
|
||||
def calculate_positions_margin_safe(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbols: Sequence[str],
|
||||
) -> float:
|
||||
@@ -766,7 +837,7 @@ def calculate_positions_margin_safe(
|
||||
``suppress_errors=True``. Failed symbols are silently skipped.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbols: Symbols to include.
|
||||
|
||||
Returns:
|
||||
@@ -778,23 +849,23 @@ def calculate_positions_margin_safe(
|
||||
)
|
||||
|
||||
|
||||
def calculate_spread_ratio(client: Mt5TradingClient, symbol: str) -> float:
|
||||
def calculate_spread_ratio(client: _Mt5ClientProtocol, symbol: str) -> float:
|
||||
"""Return ``(ask - bid) / ((ask + bid) / 2)`` for the latest tick.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If bid or ask is unavailable.
|
||||
Mt5OperationError: If bid or ask is unavailable.
|
||||
"""
|
||||
tick = get_tick_snapshot(client, symbol)
|
||||
bid = _valid_tick_price(tick, "bid")
|
||||
ask = _valid_tick_price(tick, "ask")
|
||||
bid = extract_tick_price(tick, "bid")
|
||||
ask = extract_tick_price(tick, "ask")
|
||||
if bid is None or ask is None:
|
||||
msg = f"Tick bid/ask is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
return (ask - bid) / ((ask + bid) / 2.0)
|
||||
|
||||
|
||||
def calculate_new_position_margin_ratio(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
new_position_side: OrderSide | None = None,
|
||||
@@ -803,22 +874,22 @@ def calculate_new_position_margin_ratio(
|
||||
"""Return total margin/equity ratio after an optional hypothetical position.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If equity or required tick data is invalid.
|
||||
Mt5OperationError: If equity or required tick data is invalid.
|
||||
"""
|
||||
account = get_account_snapshot(client)
|
||||
equity = float(account.get("equity") or 0.0)
|
||||
if equity <= 0:
|
||||
msg = "Account equity must be positive to calculate margin ratio."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
margin = float(account.get("margin") or 0.0)
|
||||
if new_position_side is not None and new_position_volume > 0:
|
||||
side = _normalize_order_side(new_position_side)
|
||||
price = _valid_tick_price(
|
||||
price = extract_tick_price(
|
||||
get_tick_snapshot(client, symbol), "ask" if side == "BUY" else "bid"
|
||||
)
|
||||
if price is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
order_type = (
|
||||
client.mt5.ORDER_TYPE_BUY if side == "BUY" else client.mt5.ORDER_TYPE_SELL
|
||||
)
|
||||
@@ -828,8 +899,171 @@ def calculate_new_position_margin_ratio(
|
||||
return margin / equity
|
||||
|
||||
|
||||
def _account_equity(client: _Mt5ClientProtocol) -> float:
|
||||
account = get_account_snapshot(client)
|
||||
return _required_account_number(account, "equity", allow_zero=False)
|
||||
|
||||
|
||||
def _required_account_number(
|
||||
account: Mapping[str, object],
|
||||
field: str,
|
||||
*,
|
||||
allow_zero: bool,
|
||||
) -> float:
|
||||
raw_value = account.get(field)
|
||||
if isinstance(raw_value, bool) or not isinstance(raw_value, Real):
|
||||
msg = f"Account {field} must be a finite number to calculate margin ratio."
|
||||
raise Mt5OperationError(msg)
|
||||
value = float(raw_value)
|
||||
if (
|
||||
not isfinite(value)
|
||||
or (not allow_zero and value <= 0)
|
||||
or (allow_zero and value < 0)
|
||||
):
|
||||
msg = (
|
||||
f"Account {field} must be a non-negative finite number."
|
||||
if allow_zero
|
||||
else f"Account {field} must be a positive finite number."
|
||||
)
|
||||
raise Mt5OperationError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def calculate_account_projected_margin_ratio(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str | None = None,
|
||||
new_position_side: OrderSide | None = None,
|
||||
new_position_volume: float = 0.0,
|
||||
) -> float:
|
||||
"""Return account-wide current plus optional new-position margin over equity.
|
||||
|
||||
Current exposure comes from the broker account snapshot ``margin`` field so
|
||||
unrelated open positions remain in the baseline. Optional projected
|
||||
exposure is added via :func:`estimate_order_margin` only when a symbol, side,
|
||||
and positive volume are all supplied.
|
||||
|
||||
"""
|
||||
account = get_account_snapshot(client)
|
||||
equity = _required_account_number(account, "equity", allow_zero=False)
|
||||
margin = _required_account_number(account, "margin", allow_zero=True)
|
||||
if symbol is not None and new_position_side is not None and new_position_volume > 0:
|
||||
margin += estimate_order_margin(
|
||||
client,
|
||||
symbol,
|
||||
new_position_side,
|
||||
new_position_volume,
|
||||
)
|
||||
return margin / equity
|
||||
|
||||
|
||||
def calculate_projected_margin_ratio(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
new_position_side: OrderSide | None = None,
|
||||
new_position_volume: float = 0.0,
|
||||
) -> float:
|
||||
"""Return estimated current plus optional new-position margin over equity.
|
||||
|
||||
Current exposure is estimated from open positions with
|
||||
:func:`calculate_positions_margin`. Optional projected exposure is added via
|
||||
:func:`estimate_order_margin`. Thresholds and guard actions are intentionally
|
||||
left to downstream applications.
|
||||
|
||||
Account equity, position margin, and optional projected margin errors from
|
||||
the composed MT5 helpers propagate to the caller.
|
||||
"""
|
||||
equity = _account_equity(client)
|
||||
margin = calculate_positions_margin(client, symbols=[symbol])
|
||||
if new_position_side is not None and new_position_volume > 0:
|
||||
margin += estimate_order_margin(
|
||||
client,
|
||||
symbol,
|
||||
new_position_side,
|
||||
new_position_volume,
|
||||
)
|
||||
return margin / equity
|
||||
|
||||
|
||||
def _validate_projection_mode(projection_mode: str) -> ProjectionMode:
|
||||
if projection_mode not in {"add", "replace_symbol"}:
|
||||
msg = (
|
||||
f"Unsupported projection mode: {projection_mode!r}. "
|
||||
"Expected 'add' or 'replace_symbol'."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
return cast("ProjectionMode", projection_mode)
|
||||
|
||||
|
||||
def calculate_symbol_group_margin_ratio(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbols: Sequence[str],
|
||||
new_symbol: str | None = None,
|
||||
new_position_side: OrderSide | None = None,
|
||||
new_position_volume: float = 0.0,
|
||||
suppress_errors: bool = True,
|
||||
projection_mode: ProjectionMode = "add",
|
||||
) -> float:
|
||||
"""Return estimated symbol-group margin over account equity.
|
||||
|
||||
Per-symbol current exposure is summed with
|
||||
:func:`calculate_positions_margin_by_symbol`. When ``new_symbol`` is inside
|
||||
the input symbol group and candidate side/volume are provided, projected order
|
||||
margin is applied according to ``projection_mode``:
|
||||
|
||||
- ``"add"`` (default): adds candidate margin to the group total.
|
||||
- ``"replace_symbol"``: subtracts current margin for ``new_symbol``, then
|
||||
adds candidate margin. Useful for reversal-style projections where the new
|
||||
order is intended to replace existing exposure for that symbol.
|
||||
|
||||
If the candidate margin estimation fails, the subtraction is also skipped so
|
||||
the operation is atomic. Invalid equity always raises to fail closed.
|
||||
|
||||
Raises:
|
||||
AttributeError: When symbol margin lookup or projected margin lookup
|
||||
fails and ``suppress_errors`` is ``False``.
|
||||
Mt5RuntimeError: When symbol margin lookup or projected margin lookup
|
||||
fails and ``suppress_errors`` is ``False``.
|
||||
Mt5OperationError: When account equity is invalid, or when symbol margin
|
||||
lookup or projected margin lookup fails and ``suppress_errors`` is
|
||||
``False``.
|
||||
"""
|
||||
projection_mode = _validate_projection_mode(projection_mode)
|
||||
equity = _account_equity(client)
|
||||
unique_symbols = list(dict.fromkeys(symbols))
|
||||
per_symbol = calculate_positions_margin_by_symbol(
|
||||
client,
|
||||
symbols=unique_symbols,
|
||||
suppress_errors=suppress_errors,
|
||||
)
|
||||
margin = sum(per_symbol.values(), 0.0)
|
||||
if (
|
||||
new_symbol in unique_symbols
|
||||
and new_position_side is not None
|
||||
and new_position_volume > 0
|
||||
):
|
||||
try:
|
||||
candidate_margin = estimate_order_margin(
|
||||
client,
|
||||
new_symbol,
|
||||
new_position_side,
|
||||
new_position_volume,
|
||||
)
|
||||
except (Mt5OperationError, Mt5RuntimeError, AttributeError):
|
||||
if not suppress_errors:
|
||||
raise
|
||||
_logger.warning("Skipping projected margin for %r.", new_symbol)
|
||||
else:
|
||||
if projection_mode == "replace_symbol":
|
||||
margin = max(0.0, margin - per_symbol.get(new_symbol, 0.0))
|
||||
margin += candidate_margin
|
||||
return margin / equity
|
||||
|
||||
|
||||
def calculate_margin_and_volume(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
unit_margin_ratio: float,
|
||||
preserved_margin_ratio: float,
|
||||
@@ -843,7 +1077,7 @@ def calculate_margin_and_volume(
|
||||
side when the post-reserve margin can afford it.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbol: Symbol used for minimum-lot margin and volume calculations.
|
||||
unit_margin_ratio: Fraction of post-reserve margin to allocate per unit.
|
||||
preserved_margin_ratio: Fraction of ``margin_free`` to preserve.
|
||||
@@ -896,7 +1130,7 @@ def calculate_margin_and_volume(
|
||||
|
||||
|
||||
def calculate_volume_by_margin(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
available_margin: float,
|
||||
order_side: OrderSide,
|
||||
@@ -909,7 +1143,7 @@ def calculate_volume_by_margin(
|
||||
constraints; ``0.0`` when no affordable step exists.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If symbol volume constraints or tick data are invalid.
|
||||
Mt5OperationError: If symbol volume constraints or tick data are invalid.
|
||||
"""
|
||||
if available_margin <= 0:
|
||||
return 0.0
|
||||
@@ -919,14 +1153,14 @@ def calculate_volume_by_margin(
|
||||
volume_step = float(symbol_info.get("volume_step") or volume_min or 0.0)
|
||||
if volume_min <= 0 or volume_step <= 0:
|
||||
msg = f"Invalid volume constraints for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
side = _normalize_order_side(order_side)
|
||||
price = _valid_tick_price(
|
||||
price = extract_tick_price(
|
||||
get_tick_snapshot(client, symbol), "ask" if side == "BUY" else "bid"
|
||||
)
|
||||
if price is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
order_type = (
|
||||
client.mt5.ORDER_TYPE_BUY if side == "BUY" else client.mt5.ORDER_TYPE_SELL
|
||||
)
|
||||
@@ -968,7 +1202,7 @@ def calculate_volume_by_margin(
|
||||
|
||||
|
||||
def determine_order_limits(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
symbol: str,
|
||||
side: PositionSide | str,
|
||||
stop_loss_limit_ratio: float | None = None,
|
||||
@@ -977,7 +1211,7 @@ def determine_order_limits(
|
||||
"""Derive entry and protective order prices from current market quotes.
|
||||
|
||||
Args:
|
||||
client: Connected ``Mt5TradingClient`` instance.
|
||||
client: Connected MT5 client instance.
|
||||
symbol: Symbol used for the quote lookup.
|
||||
side: Position side as ``"long"``/``"short"`` (``"buy"``/``"sell"``
|
||||
aliases are accepted).
|
||||
@@ -991,7 +1225,7 @@ def determine_order_limits(
|
||||
Omitted protective levels are returned as ``None``.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If required tick data is invalid or computed SL/TP
|
||||
Mt5OperationError: If required tick data is invalid or computed SL/TP
|
||||
prices violate available ``trade_stops_level`` pre-validation.
|
||||
"""
|
||||
stop_loss_ratio = stop_loss_limit_ratio or 0.0
|
||||
@@ -1001,10 +1235,10 @@ def determine_order_limits(
|
||||
normalized_side = _position_side_from_order_side(side)
|
||||
tick = get_tick_snapshot(client, symbol)
|
||||
entry_key = "ask" if normalized_side == "long" else "bid"
|
||||
entry = _valid_tick_price(tick, entry_key)
|
||||
entry = extract_tick_price(tick, entry_key)
|
||||
if entry is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
try:
|
||||
symbol_info = get_symbol_snapshot(client, symbol)
|
||||
except (AttributeError, KeyError, TypeError, ValueError):
|
||||
@@ -1048,7 +1282,7 @@ def determine_order_limits(
|
||||
|
||||
|
||||
def place_market_order(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
volume: float,
|
||||
@@ -1062,28 +1296,28 @@ def place_market_order(
|
||||
) -> OrderExecutionResult:
|
||||
"""Place one normalized market order or return a dry-run result.
|
||||
|
||||
``pdmt5.Mt5TradingClient.order_send()`` raises only when MT5 returns no
|
||||
response. When MT5 returns a response with a known non-success retcode, this
|
||||
helper returns ``status="failed"`` and keeps the normalized response
|
||||
details for callers to inspect.
|
||||
``order_send()`` raises only when MT5 returns no response. When MT5 returns
|
||||
a response with a known non-success retcode, this helper returns
|
||||
``status="failed"`` and keeps the normalized response details for callers
|
||||
to inspect.
|
||||
|
||||
Returns:
|
||||
Normalized execution result containing request and response details.
|
||||
|
||||
Raises:
|
||||
Mt5TradingError: If volume or required tick data is invalid.
|
||||
Mt5OperationError: If volume or required tick data is invalid.
|
||||
"""
|
||||
if volume <= 0:
|
||||
msg = "volume must be positive."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
side = _normalize_order_side(order_side)
|
||||
if not dry_run:
|
||||
ensure_symbol_selected(client, symbol)
|
||||
tick = get_tick_snapshot(client, symbol)
|
||||
price = _valid_tick_price(tick, "ask" if side == "BUY" else "bid")
|
||||
price = extract_tick_price(tick, "ask" if side == "BUY" else "bid")
|
||||
if price is None:
|
||||
msg = f"Tick price is unavailable for {symbol!r}."
|
||||
raise Mt5TradingError(msg)
|
||||
raise Mt5OperationError(msg)
|
||||
request = {
|
||||
"action": client.mt5.TRADE_ACTION_DEAL,
|
||||
"symbol": symbol,
|
||||
@@ -1156,7 +1390,7 @@ def _filter_positions(
|
||||
|
||||
|
||||
def close_open_positions(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbols: str | list[str] | None = None,
|
||||
tickets: list[int] | None = None,
|
||||
@@ -1188,8 +1422,127 @@ def close_open_positions(
|
||||
return results
|
||||
|
||||
|
||||
def _symbol_digits(client: _Mt5ClientProtocol, symbol: str) -> int | None:
|
||||
try:
|
||||
raw_digits = get_symbol_snapshot(client, symbol).get("digits")
|
||||
if raw_digits is None:
|
||||
return None
|
||||
digits = int(raw_digits)
|
||||
except (AttributeError, TypeError, ValueError):
|
||||
return None
|
||||
return digits if digits >= 0 else None
|
||||
|
||||
|
||||
def _position_ticket(value: object) -> int | None:
|
||||
ticket = _optional_int(value)
|
||||
return ticket if ticket is not None and ticket > 0 else None
|
||||
|
||||
|
||||
def _current_stop_loss(value: object) -> float | None:
|
||||
return _optional_price(value)
|
||||
|
||||
|
||||
def _trailing_stop_loss(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
position_type: object,
|
||||
current_sl: float | None,
|
||||
bid: float | None,
|
||||
ask: float | None,
|
||||
digits: int,
|
||||
trailing_stop_ratio: float,
|
||||
) -> float | None:
|
||||
next_sl: float | None = None
|
||||
if position_type == client.mt5.POSITION_TYPE_BUY:
|
||||
if bid is not None:
|
||||
next_sl = round(bid * (1.0 - trailing_stop_ratio), digits)
|
||||
if current_sl is not None and current_sl >= next_sl:
|
||||
next_sl = None
|
||||
elif position_type == client.mt5.POSITION_TYPE_SELL and ask is not None:
|
||||
next_sl = round(ask * (1.0 + trailing_stop_ratio), digits)
|
||||
if current_sl is not None and current_sl <= next_sl:
|
||||
next_sl = None
|
||||
return next_sl
|
||||
|
||||
|
||||
def calculate_trailing_stop_updates(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
trailing_stop_ratio: float,
|
||||
) -> dict[int, float]:
|
||||
"""Return per-ticket trailing stop-loss updates for open symbol positions.
|
||||
|
||||
Buy positions trail from bid using ``bid * (1 - trailing_stop_ratio)``.
|
||||
Sell positions trail from ask using ``ask * (1 + trailing_stop_ratio)``.
|
||||
Existing stop losses are preserved when they are already more favorable.
|
||||
Missing symbol metadata returns an empty update map. Positions with a
|
||||
missing side-specific tick price are skipped.
|
||||
"""
|
||||
_require_protective_ratio(trailing_stop_ratio, "trailing_stop_ratio")
|
||||
positions = get_positions_frame(client, symbol=symbol)
|
||||
if positions.empty:
|
||||
return {}
|
||||
tick = get_tick_snapshot(client, symbol)
|
||||
bid = extract_tick_price(tick, "bid")
|
||||
ask = extract_tick_price(tick, "ask")
|
||||
digits = _symbol_digits(client, symbol)
|
||||
if digits is None:
|
||||
return {}
|
||||
|
||||
updates: dict[int, float] = {}
|
||||
for row in positions.to_dict("records"):
|
||||
ticket = _position_ticket(row.get("ticket"))
|
||||
if ticket is None:
|
||||
continue
|
||||
next_sl = _trailing_stop_loss(
|
||||
client,
|
||||
position_type=row.get("type"),
|
||||
current_sl=_current_stop_loss(row.get("sl")),
|
||||
bid=bid,
|
||||
ask=ask,
|
||||
digits=digits,
|
||||
trailing_stop_ratio=trailing_stop_ratio,
|
||||
)
|
||||
if next_sl is None:
|
||||
continue
|
||||
updates[ticket] = next_sl
|
||||
return updates
|
||||
|
||||
|
||||
def update_trailing_stop_loss_for_open_positions(
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
trailing_stop_ratio: float,
|
||||
dry_run: bool = False,
|
||||
) -> list[OrderExecutionResult]:
|
||||
"""Update open positions whose trailing stop loss should move favorably.
|
||||
|
||||
Returns:
|
||||
Normalized execution results for positions that need an SL update.
|
||||
"""
|
||||
updates = calculate_trailing_stop_updates(
|
||||
client,
|
||||
symbol=symbol,
|
||||
trailing_stop_ratio=trailing_stop_ratio,
|
||||
)
|
||||
results: list[OrderExecutionResult] = []
|
||||
for ticket, stop_loss in updates.items():
|
||||
results.extend(
|
||||
update_sltp_for_open_positions(
|
||||
client,
|
||||
symbol=symbol,
|
||||
tickets=[ticket],
|
||||
stop_loss=stop_loss,
|
||||
dry_run=dry_run,
|
||||
),
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
def update_sltp_for_open_positions(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str | None = None,
|
||||
tickets: list[int] | None = None,
|
||||
@@ -1253,7 +1606,7 @@ def update_sltp_for_open_positions(
|
||||
|
||||
|
||||
def fetch_latest_closed_rates_for_trading_client(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
granularity: str,
|
||||
@@ -1267,25 +1620,23 @@ def fetch_latest_closed_rates_for_trading_client(
|
||||
Raises:
|
||||
ValueError: If ``count`` is not positive, rate data is empty or
|
||||
malformed, or the ``time`` column is missing.
|
||||
Mt5TradingError: If the trading client cannot fetch rate data.
|
||||
Mt5OperationError: If the trading client cannot fetch rate data.
|
||||
"""
|
||||
if count <= 0:
|
||||
msg = "count must be positive."
|
||||
raise ValueError(msg)
|
||||
timeframe = parse_timeframe(granularity)
|
||||
fetch_method = getattr(client, "fetch_latest_rates_as_df", None)
|
||||
copy_method = getattr(client, "copy_rates_from_pos_as_df", None)
|
||||
if callable(fetch_method):
|
||||
fetched = fetch_method(symbol, granularity, count + 1)
|
||||
else:
|
||||
copy_method = getattr(client, "copy_rates_from_pos_as_df", None)
|
||||
if not callable(copy_method):
|
||||
msg = "MT5 trading client cannot fetch rate data."
|
||||
raise Mt5TradingError(msg)
|
||||
elif callable(copy_method):
|
||||
fetched = copy_method(
|
||||
symbol=symbol,
|
||||
timeframe=parse_timeframe(granularity),
|
||||
start_pos=0,
|
||||
count=count + 1,
|
||||
symbol=symbol, timeframe=timeframe, start_pos=0, count=count + 1
|
||||
)
|
||||
else:
|
||||
msg = "MT5 trading client cannot fetch rate data."
|
||||
raise Mt5OperationError(msg)
|
||||
if not isinstance(fetched, pd.DataFrame):
|
||||
msg = (
|
||||
f"Malformed rate data for {symbol!r} at granularity {granularity!r}: "
|
||||
@@ -1347,7 +1698,7 @@ def _rate_time_to_utc(series: pd.Series, symbol: str) -> pd.DatetimeIndex:
|
||||
|
||||
|
||||
def fetch_latest_closed_rates_indexed(
|
||||
client: Mt5TradingClient,
|
||||
client: _Mt5ClientProtocol,
|
||||
*,
|
||||
symbol: str,
|
||||
granularity: str,
|
||||
@@ -1403,13 +1754,13 @@ def mt5_trading_session(
|
||||
path: str | None = None,
|
||||
timeout: int | None = None,
|
||||
retry_count: int = 0,
|
||||
) -> Iterator[Mt5TradingClient]:
|
||||
) -> Iterator[_Mt5ClientProtocol]:
|
||||
"""Open a trading-capable MT5 session and always shut down safely.
|
||||
|
||||
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` when set,
|
||||
initializes and logs in via ``initialize_and_login_mt5()``, yields a
|
||||
connected :class:`~pdmt5.Mt5TradingClient`, and calls ``shutdown()`` on
|
||||
exit even when an error is raised inside the context.
|
||||
connected client supporting required MT5 methods, and calls ``shutdown()``
|
||||
on exit even when an error is raised inside the context.
|
||||
|
||||
Args:
|
||||
config: MT5 connection configuration. Defaults to an empty config that
|
||||
@@ -1419,11 +1770,10 @@ def mt5_trading_session(
|
||||
server: Optional trading server name.
|
||||
path: Optional terminal executable path.
|
||||
timeout: Optional connection timeout in milliseconds.
|
||||
retry_count: Number of initialization retries passed to
|
||||
``Mt5TradingClient``.
|
||||
retry_count: Number of initialization retries.
|
||||
|
||||
Yields:
|
||||
Connected ``Mt5TradingClient`` bound to the session.
|
||||
Connected client supporting required MT5 trading methods.
|
||||
"""
|
||||
client = create_trading_client(
|
||||
config=config,
|
||||
|
||||
+15
-7
@@ -4,13 +4,15 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, TypeGuard
|
||||
|
||||
import click
|
||||
from pdmt5 import COPY_TICKS_MAP, TIMEFRAME_MAP
|
||||
from pdmt5 import COPY_TICKS_MAP as _COPY_TICKS_MAP
|
||||
from pdmt5 import TIMEFRAME_MAP as _TIMEFRAME_MAP
|
||||
from pdmt5 import parse_copy_ticks as _parse_copy_ticks
|
||||
from pdmt5 import parse_timeframe as _parse_timeframe
|
||||
|
||||
@@ -23,14 +25,11 @@ if TYPE_CHECKING:
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Backward-compatible snapshot; prefer ``COPY_TICKS_MAP`` from pdmt5 directly.
|
||||
TICK_FLAG_MAP: dict[str, int] = dict(COPY_TICKS_MAP)
|
||||
|
||||
TIMEFRAME_NAMES: tuple[str, ...] = tuple(
|
||||
name for name in TIMEFRAME_MAP if not name.startswith("TIMEFRAME_")
|
||||
name for name in _TIMEFRAME_MAP if not name.startswith("TIMEFRAME_")
|
||||
)
|
||||
_TICK_FLAG_NAMES: tuple[str, ...] = tuple(
|
||||
name for name in COPY_TICKS_MAP if not name.startswith("COPY_TICKS_")
|
||||
name for name in _COPY_TICKS_MAP if not name.startswith("COPY_TICKS_")
|
||||
)
|
||||
|
||||
_FORMAT_EXTENSIONS: dict[str, str] = {
|
||||
@@ -279,7 +278,7 @@ def export_dataframe_to_sqlite(
|
||||
full table, so repeated appends cost O(table size); index the key
|
||||
columns when appending frequently.
|
||||
"""
|
||||
with sqlite3.connect(output_path) as conn:
|
||||
with closing(sqlite3.connect(output_path)) as conn, conn:
|
||||
df.to_sql( # type: ignore[reportUnknownMemberType]
|
||||
table_name,
|
||||
conn,
|
||||
@@ -314,6 +313,7 @@ def export_dataframe(
|
||||
table_name: Table name for SQLite3 output.
|
||||
|
||||
Raises:
|
||||
ImportError: If the parquet format is requested but pyarrow is not installed.
|
||||
ValueError: If the output format is not supported.
|
||||
"""
|
||||
if output_format == "csv":
|
||||
@@ -326,6 +326,14 @@ def export_dataframe(
|
||||
indent=2,
|
||||
)
|
||||
elif output_format == "parquet":
|
||||
try:
|
||||
__import__("pyarrow")
|
||||
except ImportError as exc:
|
||||
msg = (
|
||||
"Parquet export requires the optional dependency pyarrow. "
|
||||
'Install it with: pip install "mt5cli[parquet]"'
|
||||
)
|
||||
raise ImportError(msg) from exc
|
||||
df.to_parquet(output_path, index=False)
|
||||
elif output_format == "sqlite3":
|
||||
export_dataframe_to_sqlite(
|
||||
|
||||
+17
-4
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "mt5cli"
|
||||
version = "0.9.2"
|
||||
version = "1.1.0"
|
||||
description = "Generic MT5 data and execution infrastructure for Python applications"
|
||||
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
|
||||
@@ -9,9 +9,8 @@ license-files = ["LICENSE"]
|
||||
readme = "README.md"
|
||||
requires-python = ">= 3.11, < 3.14"
|
||||
dependencies = [
|
||||
"pdmt5>=0.3.0",
|
||||
"pdmt5>=1.0.0",
|
||||
"click >= 8.1.0",
|
||||
"pyarrow >= 19.0.0",
|
||||
"typer >= 0.15.0",
|
||||
]
|
||||
classifiers = [
|
||||
@@ -25,6 +24,14 @@ classifiers = [
|
||||
"Topic :: Office/Business :: Financial :: Investment",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
parquet = ["pyarrow >= 19.0.0"]
|
||||
otel = [
|
||||
"opentelemetry-api",
|
||||
"opentelemetry-sdk",
|
||||
"opentelemetry-exporter-otlp-proto-http",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
mt5cli = "mt5cli.cli:main"
|
||||
|
||||
@@ -42,6 +49,9 @@ dev = [
|
||||
"pytest-mock >= 3.12.0",
|
||||
"pytest-cov >= 5.0.0",
|
||||
"pandas-stubs >= 2.2.3.250527",
|
||||
"pyarrow >= 19.0.0",
|
||||
"opentelemetry-api",
|
||||
"opentelemetry-sdk",
|
||||
"mkdocs >= 1.6.1",
|
||||
"mkdocs-material >= 9.7.6",
|
||||
"mkdocstrings[python] >= 1.0.4",
|
||||
@@ -175,7 +185,10 @@ omit = [
|
||||
[tool.coverage.report]
|
||||
show_missing = true
|
||||
fail_under = 100
|
||||
exclude_lines = ["if TYPE_CHECKING:"]
|
||||
exclude_also = [
|
||||
"if TYPE_CHECKING:",
|
||||
"^\\s+\\.\\.\\.$",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -2,12 +2,17 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from pytest_mock import MockerFixture # noqa: TC002
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from types import TracebackType
|
||||
|
||||
_DATAFRAME_METHODS = (
|
||||
"copy_rates_from_as_df",
|
||||
"copy_rates_from_pos_as_df",
|
||||
@@ -30,6 +35,25 @@ _DATAFRAME_METHODS = (
|
||||
"order_send_as_df",
|
||||
)
|
||||
|
||||
_ORIGINAL_SQLITE_CONNECT = sqlite3.connect
|
||||
|
||||
|
||||
class ClosingSqliteConnection(sqlite3.Connection):
|
||||
"""SQLite connection that closes after context-manager exit in tests."""
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_value: BaseException | None,
|
||||
traceback: TracebackType | None,
|
||||
) -> Literal[False]:
|
||||
"""Commit or roll back the transaction, then close the connection."""
|
||||
try:
|
||||
super().__exit__(exc_type, exc_value, traceback)
|
||||
finally:
|
||||
self.close()
|
||||
return False
|
||||
|
||||
|
||||
def build_mock_mt5_data_client() -> MagicMock:
|
||||
"""Return a MagicMock Mt5DataClient with common DataFrame stubs."""
|
||||
@@ -50,3 +74,17 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
|
||||
client = build_mock_mt5_data_client()
|
||||
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
|
||||
return client
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def close_sqlite_context_connections(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Make test SQLite context managers close their connection handles."""
|
||||
|
||||
def connect(
|
||||
*args: Any, # noqa: ANN401
|
||||
**kwargs: Any, # noqa: ANN401
|
||||
) -> sqlite3.Connection:
|
||||
kwargs.setdefault("factory", ClosingSqliteConnection)
|
||||
return _ORIGINAL_SQLITE_CONNECT(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(sqlite3, "connect", connect)
|
||||
|
||||
+642
-6
@@ -740,6 +740,366 @@ class TestCommands:
|
||||
assert "must be a JSON object" in normalize_cli_output(result.output)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Help text / scope tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestHelpText:
|
||||
"""Tests verifying CLI help text matches the documented scope."""
|
||||
|
||||
def test_top_level_help_mentions_execution(self) -> None:
|
||||
"""Top-level help must describe execution utilities, not export only."""
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
output = normalize_cli_output(result.output)
|
||||
assert "execution" in output.lower()
|
||||
|
||||
def test_top_level_help_has_execution_panel(self) -> None:
|
||||
"""Top-level help must show an Execution command group."""
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Execution" in result.output
|
||||
|
||||
def test_top_level_help_has_data_export_panel(self) -> None:
|
||||
"""Top-level help must show a Data / Export command group."""
|
||||
result = runner.invoke(app, ["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Data / Export" in result.output
|
||||
|
||||
def test_order_send_help_mentions_expert_and_raw(self) -> None:
|
||||
"""order-send help must communicate it is the expert raw-request path."""
|
||||
result2 = runner.invoke(
|
||||
app,
|
||||
["-o", "out.csv", "order-send", "--help"],
|
||||
)
|
||||
assert result2.exit_code == 0
|
||||
output = normalize_cli_output(result2.output)
|
||||
assert "raw" in output.lower()
|
||||
assert "expert" in output.lower()
|
||||
|
||||
def test_order_send_help_mentions_live_execution(self) -> None:
|
||||
"""order-send help must warn about live execution."""
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", "out.csv", "order-send", "--help"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
output = normalize_cli_output(result.output)
|
||||
assert "live" in output.lower()
|
||||
|
||||
def test_close_positions_help_mentions_dry_run_and_yes(self) -> None:
|
||||
"""close-positions help must document both safety gates."""
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", "out.csv", "close-positions", "--help"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
output = normalize_cli_output(result.output)
|
||||
assert "--dry-run" in output
|
||||
assert "--yes" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# close-positions command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_mock_trading_client() -> MagicMock:
|
||||
"""Return a MagicMock Mt5TradingClient with trading constants set."""
|
||||
client = MagicMock()
|
||||
client.mt5.POSITION_TYPE_BUY = 0
|
||||
client.mt5.POSITION_TYPE_SELL = 1
|
||||
client.mt5.ORDER_TYPE_BUY = 10
|
||||
client.mt5.ORDER_TYPE_SELL = 11
|
||||
client.mt5.TRADE_ACTION_DEAL = 20
|
||||
client.mt5.ORDER_FILLING_IOC = 30
|
||||
client.mt5.ORDER_TIME_GTC = 40
|
||||
client.mt5.TRADE_RETCODE_DONE = 10009
|
||||
client.mt5.TRADE_RETCODE_PLACED = 10008
|
||||
client.mt5.TRADE_RETCODE_DONE_PARTIAL = 10010
|
||||
return client
|
||||
|
||||
|
||||
class TestClosePositions:
|
||||
"""Tests for the close-positions command."""
|
||||
|
||||
@pytest.fixture
|
||||
def trading_client(self, mocker: MockerFixture) -> MagicMock:
|
||||
"""Patch create_trading_client and return a mock trading client."""
|
||||
client = _build_mock_trading_client()
|
||||
client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 1, "symbol": "JP225", "type": 0, "volume": 1.0},
|
||||
{"ticket": 2, "symbol": "EURUSD", "type": 1, "volume": 0.5},
|
||||
])
|
||||
client.symbol_info_tick_as_dict.return_value = {"ask": 1.2, "bid": 1.1}
|
||||
mocker.patch("mt5cli.cli.create_trading_client", return_value=client)
|
||||
return client
|
||||
|
||||
def test_dry_run_does_not_require_yes(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test --dry-run mode succeeds without --yes."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--symbol", "JP225", "--dry-run"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert output.exists()
|
||||
trading_client.order_send.assert_not_called()
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_live_requires_yes(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test live close-positions fails without --yes."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--symbol", "JP225"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "Pass --yes" in normalize_cli_output(result.output)
|
||||
trading_client.order_send.assert_not_called()
|
||||
|
||||
def test_live_with_yes_calls_order_send(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test --yes triggers live execution for matching positions."""
|
||||
trading_client.order_send.return_value = {"retcode": 10009, "comment": "ok"}
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--symbol", "JP225", "--yes"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
trading_client.order_send.assert_called_once()
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_symbol_filter_passed_through(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test --symbol values are used to filter positions."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--symbol",
|
||||
"JP225",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
data = json.loads(output.read_text())
|
||||
assert len(data) == 1
|
||||
assert data[0]["symbol"] == "JP225"
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_multiple_symbols_filter(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test multiple --symbol options are combined."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--symbol",
|
||||
"JP225",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
data = json.loads(output.read_text())
|
||||
assert len(data) == 2
|
||||
symbols = {row["symbol"] for row in data}
|
||||
assert symbols == {"JP225", "EURUSD"}
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_ticket_filter_passed_through(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test --ticket values are used to filter positions."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--ticket",
|
||||
"2",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
data = json.loads(output.read_text())
|
||||
assert len(data) == 1
|
||||
assert data[0]["symbol"] == "EURUSD"
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_symbol_and_ticket_combined(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test --symbol and --ticket apply AND semantics when combined."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--symbol",
|
||||
"JP225",
|
||||
"--ticket",
|
||||
"1",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
data = json.loads(output.read_text())
|
||||
# symbol=JP225 AND ticket=1 → exactly one match
|
||||
assert len(data) == 1
|
||||
assert data[0]["symbol"] == "JP225"
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_missing_symbol_and_ticket_fails(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test that omitting both --symbol and --ticket fails closed."""
|
||||
mocker.patch("mt5cli.cli.create_trading_client")
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--dry-run"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "symbol" in normalize_cli_output(result.output).lower()
|
||||
|
||||
def test_output_export_dry_run(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test dry-run results export with status=dry_run."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--symbol", "JP225", "--dry-run"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
trading_client.shutdown.assert_called_once()
|
||||
data = json.loads(output.read_text())
|
||||
assert data[0]["status"] == "dry_run"
|
||||
assert data[0]["dry_run"] is True
|
||||
assert data[0]["order_side"] == "SELL"
|
||||
|
||||
def test_order_send_unchanged(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mock_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that order-send behavior is unchanged by close-positions addition."""
|
||||
output = tmp_path / "out.csv"
|
||||
request = json.dumps({"action": 1, "symbol": "EURUSD", "volume": 0.1})
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "order-send", "--request", request, "--yes"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_client.order_send_as_df.assert_called_once()
|
||||
|
||||
def test_shutdown_called_on_close_error(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Test that shutdown is called even when close_open_positions raises."""
|
||||
client = _build_mock_trading_client()
|
||||
client.positions_get_as_df.side_effect = RuntimeError("connection lost")
|
||||
mocker.patch("mt5cli.cli.create_trading_client", return_value=client)
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(output), "close-positions", "--symbol", "JP225", "--dry-run"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
client.shutdown.assert_called_once()
|
||||
|
||||
def test_dry_run_wins_over_yes(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that --dry-run takes precedence when combined with --yes."""
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--symbol",
|
||||
"JP225",
|
||||
"--dry-run",
|
||||
"--yes",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
trading_client.order_send.assert_not_called()
|
||||
trading_client.shutdown.assert_called_once()
|
||||
|
||||
def test_no_matching_positions_exports_empty_result(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
trading_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that zero filter matches produces an empty JSON array."""
|
||||
trading_client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 1, "symbol": "JP225", "type": 0, "volume": 1.0},
|
||||
])
|
||||
output = tmp_path / "close.json"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"close-positions",
|
||||
"--symbol",
|
||||
"NONEXISTENT",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
trading_client.shutdown.assert_called_once()
|
||||
assert output.exists()
|
||||
assert json.loads(output.read_text()) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Callback / shared options
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -938,12 +1298,12 @@ class TestCollectHistory:
|
||||
"""Create a mocked Mt5DataClient with history-style DataFrames."""
|
||||
return _build_history_client(mocker)
|
||||
|
||||
def test_collect_history_writes_all_tables(
|
||||
def test_collect_history_writes_default_tables(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
history_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that collect-history writes rates, ticks, and history tables."""
|
||||
"""Test that collect-history default excludes ticks."""
|
||||
output = tmp_path / "history.db"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
@@ -963,8 +1323,42 @@ class TestCollectHistory:
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert history_client.copy_rates_range_as_df.call_count == 2
|
||||
assert history_client.copy_ticks_range_as_df.call_count == 2
|
||||
history_client.copy_ticks_range_as_df.assert_any_call(
|
||||
assert history_client.copy_ticks_range_as_df.call_count == 0
|
||||
with sqlite3.connect(output) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'",
|
||||
).fetchall()
|
||||
}
|
||||
assert {"rates", "history_orders", "history_deals"} <= tables
|
||||
assert "ticks" not in tables
|
||||
|
||||
def test_collect_history_explicit_ticks_dataset(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
history_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that --dataset ticks writes the ticks table with the correct flags."""
|
||||
output = tmp_path / "history.db"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"collect-history",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
"--date-from",
|
||||
"2024-01-01",
|
||||
"--date-to",
|
||||
"2024-02-01",
|
||||
"--dataset",
|
||||
"ticks",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
history_client.copy_ticks_range_as_df.assert_called_once_with(
|
||||
symbol="EURUSD",
|
||||
date_from=datetime(2024, 1, 1, tzinfo=UTC),
|
||||
date_to=datetime(2024, 2, 1, tzinfo=UTC),
|
||||
@@ -977,7 +1371,8 @@ class TestCollectHistory:
|
||||
"SELECT name FROM sqlite_master WHERE type='table'",
|
||||
).fetchall()
|
||||
}
|
||||
assert {"rates", "ticks", "history_orders", "history_deals"} <= tables
|
||||
assert "ticks" in tables
|
||||
assert "rates" not in tables
|
||||
|
||||
def test_collect_history_history_fetched_per_symbol(
|
||||
self,
|
||||
@@ -1160,7 +1555,7 @@ class TestCollectHistory:
|
||||
tmp_path: Path,
|
||||
history_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that --flags defaults to ALL for ticks."""
|
||||
"""Test that --flags defaults to ALL when --dataset ticks is explicit."""
|
||||
output = tmp_path / "history.db"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
@@ -1174,6 +1569,8 @@ class TestCollectHistory:
|
||||
"2024-01-01",
|
||||
"--date-to",
|
||||
"2024-02-01",
|
||||
"--dataset",
|
||||
"ticks",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
@@ -1458,6 +1855,245 @@ class TestCollectHistory:
|
||||
)
|
||||
|
||||
|
||||
class TestGrafanaSchemaCommand:
|
||||
"""Tests for the grafana-schema CLI command."""
|
||||
|
||||
def test_grafana_schema_creates_snapshot_tables_in_sqlite(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""grafana-schema applies Grafana schema to a SQLite database."""
|
||||
output = tmp_path / "out.db"
|
||||
result = runner.invoke(app, ["-o", str(output), "grafana-schema"])
|
||||
assert result.exit_code == 0, result.output
|
||||
with sqlite3.connect(output) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
).fetchall()
|
||||
}
|
||||
assert "snapshot_runs" in tables
|
||||
assert "account_snapshots" in tables
|
||||
|
||||
def test_grafana_schema_is_idempotent(self, tmp_path: Path) -> None:
|
||||
"""grafana-schema can be invoked multiple times without error."""
|
||||
output = tmp_path / "out.db"
|
||||
result1 = runner.invoke(app, ["-o", str(output), "grafana-schema"])
|
||||
result2 = runner.invoke(app, ["-o", str(output), "grafana-schema"])
|
||||
assert result1.exit_code == 0, result1.output
|
||||
assert result2.exit_code == 0, result2.output
|
||||
|
||||
def test_grafana_schema_rejects_non_sqlite_output(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""grafana-schema fails when output is not a SQLite3 format."""
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.csv"), "grafana-schema"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "grafana-schema requires SQLite3 output" in result.output
|
||||
|
||||
|
||||
class TestSnapshotCommand:
|
||||
"""Tests for the snapshot CLI command."""
|
||||
|
||||
def test_snapshot_rejects_non_sqlite_output(self, tmp_path: Path) -> None:
|
||||
"""Snapshot fails when output is not a SQLite3 format."""
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.csv"), "snapshot"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "snapshot requires SQLite3 output" in result.output
|
||||
|
||||
def test_snapshot_delegates_to_update_observability_with_config(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Snapshot calls sdk.update_observability_with_config."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
output = tmp_path / "out.db"
|
||||
result = runner.invoke(app, ["-o", str(output), "snapshot"])
|
||||
assert result.exit_code == 0, result.output
|
||||
updater.assert_called_once()
|
||||
kwargs = updater.call_args.kwargs
|
||||
assert kwargs["output"] == output
|
||||
assert kwargs["symbols"] is None
|
||||
assert kwargs["include_account"] is True
|
||||
assert kwargs["include_positions"] is True
|
||||
assert kwargs["include_orders"] is True
|
||||
assert kwargs["include_terminal"] is True
|
||||
assert kwargs["with_grafana_schema"] is False
|
||||
|
||||
def test_snapshot_with_symbol_filter(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Snapshot passes symbol list to update_observability_with_config."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(tmp_path / "out.db"),
|
||||
"snapshot",
|
||||
"--symbol",
|
||||
"EURUSD",
|
||||
"--symbol",
|
||||
"GBPUSD",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
kwargs = updater.call_args.kwargs
|
||||
assert kwargs["symbols"] == ["EURUSD", "GBPUSD"]
|
||||
|
||||
def test_snapshot_with_no_account_flag(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--no-account disables account snapshotting."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.db"), "snapshot", "--no-account"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert updater.call_args.kwargs["include_account"] is False
|
||||
|
||||
def test_snapshot_with_no_positions_flag(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--no-positions disables position snapshotting."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.db"), "snapshot", "--no-positions"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert updater.call_args.kwargs["include_positions"] is False
|
||||
|
||||
def test_snapshot_with_no_orders_flag(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--no-orders disables order snapshotting."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.db"), "snapshot", "--no-orders"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert updater.call_args.kwargs["include_orders"] is False
|
||||
|
||||
def test_snapshot_with_no_terminal_flag(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--no-terminal disables terminal snapshotting."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.db"), "snapshot", "--no-terminal"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert updater.call_args.kwargs["include_terminal"] is False
|
||||
|
||||
def test_snapshot_with_no_grafana_schema_flag(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--no-grafana-schema disables Grafana schema creation."""
|
||||
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
result = runner.invoke(
|
||||
app,
|
||||
["-o", str(tmp_path / "out.db"), "snapshot", "--no-grafana-schema"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert updater.call_args.kwargs["with_grafana_schema"] is False
|
||||
|
||||
def test_snapshot_with_publish_copy(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""--publish-copy calls publish_grafana_copy after update_observability."""
|
||||
mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
|
||||
copy_path = tmp_path / "grafana.db"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(tmp_path / "out.db"),
|
||||
"snapshot",
|
||||
"--publish-copy",
|
||||
str(copy_path),
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_publish.assert_called_once()
|
||||
|
||||
def test_snapshot_no_publish_copy_by_default(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""Snapshot does not call publish_grafana_copy without --publish-copy."""
|
||||
mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
|
||||
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
|
||||
result = runner.invoke(app, ["-o", str(tmp_path / "out.db"), "snapshot"])
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_publish.assert_not_called()
|
||||
|
||||
|
||||
class TestGrafanaSchemaPublishCopy:
|
||||
"""Tests for grafana-schema --publish-copy option."""
|
||||
|
||||
def test_grafana_schema_with_publish_copy(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""grafana-schema --publish-copy calls publish_grafana_copy."""
|
||||
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
|
||||
output = tmp_path / "out.db"
|
||||
copy_path = tmp_path / "grafana.db"
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"-o",
|
||||
str(output),
|
||||
"grafana-schema",
|
||||
"--publish-copy",
|
||||
str(copy_path),
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_publish.assert_called_once()
|
||||
|
||||
def test_grafana_schema_no_publish_copy_by_default(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""grafana-schema does not call publish_grafana_copy by default."""
|
||||
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
|
||||
result = runner.invoke(app, ["-o", str(tmp_path / "out.db"), "grafana-schema"])
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_publish.assert_not_called()
|
||||
|
||||
|
||||
class TestMain:
|
||||
"""Tests for the main entry point."""
|
||||
|
||||
|
||||
+188
-93
@@ -2,11 +2,16 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
from importlib.metadata import requires
|
||||
from typing import TYPE_CHECKING, get_type_hints
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from pdmt5 import Mt5RuntimeError, Mt5TradingError
|
||||
@@ -14,13 +19,8 @@ from pytest_mock import MockerFixture # noqa: TC002
|
||||
|
||||
import mt5cli
|
||||
from mt5cli import (
|
||||
DEDUP_KEYS,
|
||||
REQUIRED_COLUMNS,
|
||||
STABLE_SDK_EXPORTS,
|
||||
TIME_COLUMNS,
|
||||
AccountSpec,
|
||||
DataKind,
|
||||
Dataset,
|
||||
ExecutionStatus,
|
||||
MarginVolume,
|
||||
MT5Client,
|
||||
@@ -33,44 +33,62 @@ from mt5cli import (
|
||||
RateTarget,
|
||||
build_config,
|
||||
build_rate_targets,
|
||||
calculate_account_projected_margin_ratio,
|
||||
calculate_margin_and_volume,
|
||||
calculate_positions_margin,
|
||||
call_with_normalized_errors,
|
||||
detect_format,
|
||||
calculate_projected_margin_ratio,
|
||||
calculate_symbol_group_margin_ratio,
|
||||
calculate_trailing_stop_updates,
|
||||
drop_forming_rate_bar,
|
||||
ensure_symbol_selected,
|
||||
ensure_utc,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
extract_tick_price,
|
||||
fetch_latest_closed_rates,
|
||||
fetch_latest_closed_rates_for_trading_client,
|
||||
fetch_latest_closed_rates_indexed,
|
||||
granularity_name,
|
||||
is_recoverable_mt5_error,
|
||||
load_rate_data,
|
||||
load_rate_series_from_sqlite,
|
||||
mt5_session,
|
||||
mt5_trading_session,
|
||||
normalize_dataframe,
|
||||
normalize_mt5_exception,
|
||||
normalize_order_volume,
|
||||
place_market_order,
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
)
|
||||
from mt5cli.converters import (
|
||||
ensure_utc,
|
||||
granularity_name,
|
||||
normalize_symbol,
|
||||
normalize_symbols,
|
||||
parse_date_range,
|
||||
place_market_order,
|
||||
recent_window,
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
)
|
||||
from mt5cli.exceptions import (
|
||||
call_with_normalized_errors,
|
||||
is_recoverable_mt5_error,
|
||||
normalize_mt5_exception,
|
||||
)
|
||||
from mt5cli.history import (
|
||||
create_rate_compatibility_views,
|
||||
load_rate_data,
|
||||
resolve_rate_view_name,
|
||||
)
|
||||
from mt5cli.retry import retry_with_backoff
|
||||
from mt5cli.schemas import (
|
||||
DEDUP_KEYS,
|
||||
REQUIRED_COLUMNS,
|
||||
TIME_COLUMNS,
|
||||
DataKind,
|
||||
ensure_utc_columns,
|
||||
normalize_dataframe,
|
||||
normalize_time_columns,
|
||||
schema_columns,
|
||||
validate_schema,
|
||||
)
|
||||
from mt5cli.history import create_rate_compatibility_views
|
||||
from mt5cli.retry import retry_with_backoff
|
||||
from mt5cli.schemas import ensure_utc_columns, normalize_time_columns
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
from mt5cli.utils import (
|
||||
Dataset,
|
||||
detect_format,
|
||||
export_dataframe,
|
||||
export_dataframe_to_sqlite,
|
||||
)
|
||||
|
||||
|
||||
def _sample_frame(kind: DataKind) -> pd.DataFrame:
|
||||
@@ -228,16 +246,19 @@ def test_is_recoverable_mt5_error(exc: Exception) -> None:
|
||||
assert is_recoverable_mt5_error(exc)
|
||||
|
||||
|
||||
def test_normalize_mt5_exception_maps_types() -> None:
|
||||
@pytest.mark.parametrize(
|
||||
("exc", "expected_type"),
|
||||
[
|
||||
(Mt5RuntimeError("x"), Mt5ConnectionError),
|
||||
(Mt5TradingError("x"), Mt5OperationError),
|
||||
],
|
||||
)
|
||||
def test_normalize_mt5_exception_maps_types(
|
||||
exc: Exception,
|
||||
expected_type: type[Mt5ConnectionError | Mt5OperationError],
|
||||
) -> None:
|
||||
"""MT5 exceptions map to stable mt5cli types."""
|
||||
assert isinstance(
|
||||
normalize_mt5_exception(Mt5RuntimeError("x")),
|
||||
Mt5ConnectionError,
|
||||
)
|
||||
assert isinstance(
|
||||
normalize_mt5_exception(Mt5TradingError("x")),
|
||||
Mt5OperationError,
|
||||
)
|
||||
assert isinstance(normalize_mt5_exception(exc), expected_type)
|
||||
|
||||
|
||||
def test_call_with_normalized_errors_reraises_mapped_type() -> None:
|
||||
@@ -414,26 +435,24 @@ def test_normalize_time_columns_skips_absent_time_fields() -> None:
|
||||
assert list(result.columns) == ["open"]
|
||||
|
||||
|
||||
def test_normalize_time_columns_converts_unix_seconds() -> None:
|
||||
"""Numeric MT5 ``time`` values are interpreted as Unix seconds."""
|
||||
frame = pd.DataFrame({"time": [1704067200]})
|
||||
result = normalize_time_columns(frame, DataKind.rates)
|
||||
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
|
||||
|
||||
|
||||
def test_normalize_time_columns_converts_unix_milliseconds() -> None:
|
||||
"""Numeric MT5 ``time_msc`` values are interpreted as Unix milliseconds."""
|
||||
frame = pd.DataFrame({"time_msc": [1704067200000]})
|
||||
result = normalize_time_columns(frame, DataKind.ticks)
|
||||
assert result.loc[0, "time_msc"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
|
||||
|
||||
|
||||
def test_normalize_time_columns_preserves_utc_datetimes() -> None:
|
||||
"""Already-converted datetime values remain UTC-normalized."""
|
||||
aware = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
frame = pd.DataFrame({"time": [aware]})
|
||||
result = normalize_time_columns(frame, DataKind.rates)
|
||||
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
|
||||
@pytest.mark.parametrize(
|
||||
("col", "value", "kind"),
|
||||
[
|
||||
("time", 1704067200, DataKind.rates),
|
||||
("time_msc", 1704067200000, DataKind.ticks),
|
||||
("time", datetime(2024, 1, 1, tzinfo=UTC), DataKind.rates),
|
||||
("time", "2024-01-01T00:00:00+00:00", DataKind.rates),
|
||||
],
|
||||
)
|
||||
def test_normalize_time_columns_coerces_value(
|
||||
col: str,
|
||||
value: object,
|
||||
kind: DataKind,
|
||||
) -> None:
|
||||
"""Time column values are coerced to UTC timestamps regardless of input type."""
|
||||
frame = pd.DataFrame({col: [value]})
|
||||
result = normalize_time_columns(frame, kind)
|
||||
assert result.loc[0, col] == pd.Timestamp("2024-01-01T00:00:00+00:00")
|
||||
|
||||
|
||||
def test_normalize_time_columns_handles_optional_order_times() -> None:
|
||||
@@ -483,13 +502,6 @@ def test_ensure_utc_columns_skips_missing_columns() -> None:
|
||||
assert "time" in result.columns
|
||||
|
||||
|
||||
def test_normalize_time_columns_coerces_string_timestamps() -> None:
|
||||
"""String timestamps are parsed with timezone-aware datetime coercion."""
|
||||
frame = pd.DataFrame({"time": ["2024-01-01T00:00:00+00:00"]})
|
||||
result = normalize_time_columns(frame, DataKind.rates)
|
||||
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
|
||||
|
||||
|
||||
def test_ensure_utc_columns_coerces_non_mt5_columns() -> None:
|
||||
"""Non-MT5 columns still coerce to UTC datetimes."""
|
||||
frame = pd.DataFrame({"created_at": ["2024-01-01T00:00:00+00:00"]})
|
||||
@@ -539,6 +551,12 @@ def test_storage_export_round_trip_sqlite(tmp_path: Path) -> None:
|
||||
assert count == 1
|
||||
|
||||
|
||||
def test_storage_module_does_not_exist() -> None:
|
||||
"""mt5cli.storage re-export module has been removed."""
|
||||
with pytest.raises(ModuleNotFoundError):
|
||||
importlib.import_module("mt5cli.storage")
|
||||
|
||||
|
||||
class TestStableSdkContract:
|
||||
"""Tests for the documented stable downstream SDK contract."""
|
||||
|
||||
@@ -547,6 +565,21 @@ class TestStableSdkContract:
|
||||
missing = sorted(STABLE_SDK_EXPORTS - set(mt5cli.__all__))
|
||||
assert not missing, f"STABLE_SDK_EXPORTS missing from __all__: {missing}"
|
||||
|
||||
def test_stable_exports_cover_root_api(self) -> None:
|
||||
"""STABLE_SDK_EXPORTS classifies every package-root symbol."""
|
||||
tier_metadata = {"STABLE_SDK_EXPORTS"}
|
||||
root_exports = set(mt5cli.__all__)
|
||||
|
||||
missing_from_root = sorted(STABLE_SDK_EXPORTS - root_exports)
|
||||
assert not missing_from_root, (
|
||||
f"STABLE_SDK_EXPORTS missing from __all__: {missing_from_root}"
|
||||
)
|
||||
|
||||
unclassified = sorted(root_exports - STABLE_SDK_EXPORTS - tier_metadata)
|
||||
assert not unclassified, (
|
||||
f"Root exports not in STABLE_SDK_EXPORTS: {unclassified}"
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("name", sorted(STABLE_SDK_EXPORTS))
|
||||
def test_stable_exports_are_importable_from_package_root(self, name: str) -> None:
|
||||
"""Stable SDK names resolve through ``from mt5cli import ...``."""
|
||||
@@ -615,37 +648,15 @@ class TestStableSdkContract:
|
||||
|
||||
assert calculate_positions_margin(client) == 0
|
||||
|
||||
def test_resolve_rate_view_name_from_package_root(self, tmp_path: Path) -> None:
|
||||
"""Rate view resolution is importable and honors require_existing."""
|
||||
db_path = tmp_path / "rates.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
|
||||
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
|
||||
missing = tmp_path / "missing.db"
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_view_name(missing, "EURUSD", "M1", require_existing=True)
|
||||
|
||||
def test_load_rate_data_from_package_root(self, tmp_path: Path) -> None:
|
||||
"""SQLite rate loading normalizes timestamps through the stable API."""
|
||||
db_path = tmp_path / "view.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
'CREATE VIEW "rate_EURUSD__1" AS'
|
||||
" SELECT '2024-01-01T00:00:00+00:00' AS time, 1.1 AS close",
|
||||
)
|
||||
|
||||
frame = load_rate_data(db_path, "rate_EURUSD__1")
|
||||
assert frame.index.name == "time"
|
||||
assert abs(float(frame.iloc[0]["close"]) - 1.1) < 1e-9
|
||||
def test_generic_trading_helpers_from_package_root(self) -> None:
|
||||
"""New generic trading helpers resolve through the stable surface."""
|
||||
price = extract_tick_price({"bid": "1.2"}, "bid")
|
||||
assert price is not None
|
||||
assert abs(price - 1.2) < 1e-9
|
||||
assert callable(calculate_trailing_stop_updates)
|
||||
assert callable(calculate_account_projected_margin_ratio)
|
||||
assert callable(calculate_projected_margin_ratio)
|
||||
assert callable(calculate_symbol_group_margin_ratio)
|
||||
|
||||
def test_load_rate_series_from_sqlite_requires_managed_views(
|
||||
self,
|
||||
@@ -694,7 +705,7 @@ class TestStableSdkContract:
|
||||
"""Trading session helper initializes and always shuts down."""
|
||||
mock_client = MagicMock()
|
||||
mocker.patch(
|
||||
"mt5cli.trading.Mt5TradingClient",
|
||||
"mt5cli.trading.Mt5DataClient",
|
||||
return_value=mock_client,
|
||||
)
|
||||
|
||||
@@ -723,7 +734,7 @@ class TestStableSdkContract:
|
||||
"""Trading session helper shuts down even when the body raises."""
|
||||
mock_client = MagicMock()
|
||||
mocker.patch(
|
||||
"mt5cli.trading.Mt5TradingClient",
|
||||
"mt5cli.trading.Mt5DataClient",
|
||||
return_value=mock_client,
|
||||
)
|
||||
|
||||
@@ -764,3 +775,87 @@ class TestStableSdkContract:
|
||||
assert result.index.tz is not None
|
||||
assert "time" not in result.columns
|
||||
assert "close" in result.columns
|
||||
|
||||
def test_rate_view_helpers_in_history_module(self, tmp_path: Path) -> None:
|
||||
"""Rate view helpers are available from mt5cli.history."""
|
||||
db_path = tmp_path / "rates.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates("
|
||||
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
|
||||
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
|
||||
)
|
||||
create_rate_compatibility_views(conn)
|
||||
|
||||
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
|
||||
missing = tmp_path / "missing.db"
|
||||
with pytest.raises(ValueError, match="SQLite database not found"):
|
||||
resolve_rate_view_name(missing, "EURUSD", "M1", require_existing=True)
|
||||
|
||||
def test_load_rate_data_in_history_module(self, tmp_path: Path) -> None:
|
||||
"""SQLite rate loading normalizes timestamps through mt5cli.history."""
|
||||
db_path = tmp_path / "view.db"
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
'CREATE VIEW "rate_EURUSD__1" AS'
|
||||
" SELECT '2024-01-01T00:00:00+00:00' AS time, 1.1 AS close",
|
||||
)
|
||||
|
||||
frame = load_rate_data(db_path, "rate_EURUSD__1")
|
||||
assert frame.index.name == "time"
|
||||
assert abs(float(frame.iloc[0]["close"]) - 1.1) < 1e-9
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"name",
|
||||
[
|
||||
"Mt5Config",
|
||||
"Mt5RuntimeError",
|
||||
"Mt5TradingClient",
|
||||
"Mt5TradingError",
|
||||
"TICK_FLAG_MAP",
|
||||
"TIMEFRAME_MAP",
|
||||
],
|
||||
)
|
||||
def test_pdmt5_pass_through_names_removed_from_public_contract(name: str) -> None:
|
||||
"""Removed pdmt5 pass-through names are not part of the public contract."""
|
||||
assert name not in STABLE_SDK_EXPORTS, (
|
||||
f"{name!r} should not be in STABLE_SDK_EXPORTS"
|
||||
)
|
||||
assert name not in mt5cli.__all__, f"{name!r} should not be in mt5cli.__all__"
|
||||
|
||||
|
||||
def test_mt5cli_does_not_import_high_level_trading_symbols() -> None:
|
||||
"""mt5cli doesn't import Mt5TradingClient or Mt5TradingError at module level."""
|
||||
trading_module = importlib.import_module("mt5cli.trading")
|
||||
module_dict = vars(trading_module)
|
||||
assert "Mt5TradingClient" not in module_dict, (
|
||||
"mt5cli.trading should not import Mt5TradingClient at module level"
|
||||
)
|
||||
assert "Mt5TradingError" not in module_dict, (
|
||||
"mt5cli.trading should not import Mt5TradingError at module level"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Packaging metadata
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parquet_extra_declares_pyarrow() -> None:
|
||||
"""Package metadata lists pyarrow under the parquet optional extra."""
|
||||
reqs = requires("mt5cli") or []
|
||||
parquet_reqs = [r for r in reqs if "pyarrow" in r and "parquet" in r]
|
||||
assert parquet_reqs, "pyarrow not found in parquet optional extra"
|
||||
|
||||
|
||||
def test_pyarrow_not_in_core_dependencies() -> None:
|
||||
"""Pyarrow is not a core dependency; it belongs only in the parquet extra."""
|
||||
reqs = requires("mt5cli") or []
|
||||
core_reqs = [r for r in reqs if "extra ==" not in r]
|
||||
assert not any("pyarrow" in r for r in core_reqs), (
|
||||
"pyarrow should not appear in core dependencies"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Tests for example files in examples/grafana/."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
_EXAMPLES_DIR = Path(__file__).parent.parent / "examples" / "grafana"
|
||||
_DASHBOARDS_DIR = _EXAMPLES_DIR / "dashboards"
|
||||
|
||||
|
||||
class TestGrafanaExamples:
|
||||
"""Validate structure and content of bundled Grafana example files."""
|
||||
|
||||
def test_dashboard_json_files_are_valid_json(self) -> None:
|
||||
"""All dashboard JSON files parse without error."""
|
||||
paths = list(_DASHBOARDS_DIR.glob("*.json"))
|
||||
assert paths, "No dashboard JSON files found"
|
||||
for path in paths:
|
||||
content = path.read_text(encoding="utf-8")
|
||||
obj = json.loads(content)
|
||||
assert isinstance(obj, dict), f"{path.name} root must be a JSON object"
|
||||
|
||||
def test_dashboard_json_has_no_private_placeholders(self) -> None:
|
||||
"""Dashboard JSON files contain no obvious credential placeholders."""
|
||||
private_patterns = ["password", "api_key", "apikey"]
|
||||
for path in _DASHBOARDS_DIR.glob("*.json"):
|
||||
content = path.read_text(encoding="utf-8").lower()
|
||||
for pat in private_patterns:
|
||||
assert pat not in content, f"{path.name} contains {pat!r}"
|
||||
|
||||
def test_dashboard_json_uses_grafana_views(self) -> None:
|
||||
"""All dashboard JSON files query grafana_* views."""
|
||||
for path in _DASHBOARDS_DIR.glob("*.json"):
|
||||
content = path.read_text(encoding="utf-8")
|
||||
assert "grafana_" in content, (
|
||||
f"{path.name} must contain queries against grafana_* views"
|
||||
)
|
||||
|
||||
def test_dashboard_json_has_uid(self) -> None:
|
||||
"""All dashboard JSON files have a non-empty uid field."""
|
||||
for path in _DASHBOARDS_DIR.glob("*.json"):
|
||||
obj = json.loads(path.read_text(encoding="utf-8"))
|
||||
assert obj.get("uid"), f"{path.name} must have a uid"
|
||||
|
||||
def test_dashboard_json_has_title(self) -> None:
|
||||
"""All dashboard JSON files have a non-empty title field."""
|
||||
for path in _DASHBOARDS_DIR.glob("*.json"):
|
||||
obj = json.loads(path.read_text(encoding="utf-8"))
|
||||
assert obj.get("title"), f"{path.name} must have a title"
|
||||
|
||||
def test_expected_dashboards_present(self) -> None:
|
||||
"""The three expected dashboard files are present."""
|
||||
names = {p.name for p in _DASHBOARDS_DIR.glob("*.json")}
|
||||
assert "mt5cli-overview.json" in names
|
||||
assert "mt5cli-trades.json" in names
|
||||
assert "mt5cli-market.json" in names
|
||||
|
||||
def test_readme_exists(self) -> None:
|
||||
"""examples/grafana/README.md is present."""
|
||||
assert (_EXAMPLES_DIR / "README.md").is_file()
|
||||
|
||||
def test_compose_file_exists(self) -> None:
|
||||
"""examples/grafana/compose.yml is present."""
|
||||
assert (_EXAMPLES_DIR / "compose.yml").is_file()
|
||||
|
||||
def test_datasource_provisioning_exists(self) -> None:
|
||||
"""Datasource provisioning YAML is present."""
|
||||
assert (
|
||||
_EXAMPLES_DIR / "provisioning" / "datasources" / "mt5cli-sqlite.yml"
|
||||
).is_file()
|
||||
|
||||
def test_dashboard_provisioning_exists(self) -> None:
|
||||
"""Dashboard provisioning YAML is present."""
|
||||
assert (_EXAMPLES_DIR / "provisioning" / "dashboards" / "mt5cli.yml").is_file()
|
||||
@@ -0,0 +1,991 @@
|
||||
"""Tests for mt5cli.grafana module."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
from mt5cli.grafana import (
|
||||
_build_snapshot_view, # type: ignore[reportPrivateUsage]
|
||||
_create_view_safe, # type: ignore[reportPrivateUsage]
|
||||
create_grafana_indexes,
|
||||
create_grafana_views,
|
||||
create_snapshot_tables,
|
||||
ensure_grafana_schema,
|
||||
insert_account_snapshot,
|
||||
insert_order_snapshots,
|
||||
insert_position_snapshots,
|
||||
insert_terminal_snapshot,
|
||||
publish_grafana_copy,
|
||||
record_snapshot_run,
|
||||
start_snapshot_run,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conn() -> Iterator[sqlite3.Connection]:
|
||||
"""Yield an in-memory SQLite connection for each test."""
|
||||
with sqlite3.connect(":memory:") as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _get_names(conn: sqlite3.Connection, type_: str) -> set[str]:
|
||||
return {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type=?",
|
||||
(type_,),
|
||||
).fetchall()
|
||||
}
|
||||
|
||||
|
||||
def _make_rates_table(conn: sqlite3.Connection) -> None:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates"
|
||||
" (time TEXT, symbol TEXT, timeframe INTEGER,"
|
||||
" open REAL, high REAL, low REAL, close REAL)"
|
||||
)
|
||||
|
||||
|
||||
def _make_ticks_table(conn: sqlite3.Connection) -> None:
|
||||
conn.execute("CREATE TABLE ticks (time TEXT, symbol TEXT, bid REAL, ask REAL)")
|
||||
|
||||
|
||||
def _make_history_deals_full(conn: sqlite3.Connection) -> None:
|
||||
conn.execute(
|
||||
"CREATE TABLE history_deals"
|
||||
" (time TEXT, symbol TEXT, profit REAL, type INTEGER,"
|
||||
" entry INTEGER, volume REAL, price REAL, ticket INTEGER, position_id INTEGER)"
|
||||
)
|
||||
|
||||
|
||||
def _make_history_deals_minimal(conn: sqlite3.Connection) -> None:
|
||||
"""history_deals with only time, type, symbol, profit — no entry/volume/price."""
|
||||
conn.execute(
|
||||
"CREATE TABLE history_deals (time TEXT, symbol TEXT, profit REAL, type INTEGER)"
|
||||
)
|
||||
|
||||
|
||||
def _make_history_orders_table(conn: sqlite3.Connection) -> None:
|
||||
conn.execute(
|
||||
"CREATE TABLE history_orders"
|
||||
" (time_setup TEXT, symbol TEXT, ticket INTEGER, type INTEGER)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSnapshotTables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSnapshotTables:
|
||||
"""Tests for create_snapshot_tables."""
|
||||
|
||||
def test_creates_all_five_tables(self, conn: sqlite3.Connection) -> None:
|
||||
"""All five snapshot tables are created."""
|
||||
create_snapshot_tables(conn)
|
||||
tables = _get_names(conn, "table")
|
||||
assert "snapshot_runs" in tables
|
||||
assert "account_snapshots" in tables
|
||||
assert "position_snapshots" in tables
|
||||
assert "order_snapshots" in tables
|
||||
assert "terminal_snapshots" in tables
|
||||
|
||||
def test_is_idempotent(self, conn: sqlite3.Connection) -> None:
|
||||
"""Calling create_snapshot_tables twice does not raise."""
|
||||
create_snapshot_tables(conn)
|
||||
create_snapshot_tables(conn)
|
||||
tables = _get_names(conn, "table")
|
||||
assert "snapshot_runs" in tables
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestCreateViewSafe
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCreateViewSafe:
|
||||
"""Tests for _create_view_safe."""
|
||||
|
||||
def test_creates_view_successfully(self, conn: sqlite3.Connection) -> None:
|
||||
"""A valid select SQL creates the named view."""
|
||||
_create_view_safe(conn, "test_view", "SELECT 1 AS val")
|
||||
views = _get_names(conn, "view")
|
||||
assert "test_view" in views
|
||||
|
||||
def test_replaces_existing_view(self, conn: sqlite3.Connection) -> None:
|
||||
"""Calling again with a new SQL replaces the existing view."""
|
||||
_create_view_safe(conn, "test_view", "SELECT 1 AS val")
|
||||
_create_view_safe(conn, "test_view", "SELECT 2 AS val")
|
||||
result = conn.execute("SELECT val FROM test_view").fetchone()
|
||||
assert result == (2,)
|
||||
|
||||
def test_logs_warning_on_sqlite_error(
|
||||
self,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""sqlite3.Error during CREATE VIEW logs a warning instead of raising."""
|
||||
mock_conn = MagicMock()
|
||||
mock_conn.execute.side_effect = [
|
||||
None,
|
||||
sqlite3.OperationalError("parse error"),
|
||||
]
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
_create_view_safe(mock_conn, "bad_view", "SELECT 1")
|
||||
assert "Skipping view bad_view" in caplog.text
|
||||
assert "parse error" in caplog.text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestGrafanaViews
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestGrafanaViews:
|
||||
"""Tests for create_grafana_views and individual view builders."""
|
||||
|
||||
def test_all_views_created_with_full_schema(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""All 13 Grafana views are created when all source tables are present."""
|
||||
_make_rates_table(conn)
|
||||
_make_ticks_table(conn)
|
||||
_make_history_deals_full(conn)
|
||||
_make_history_orders_table(conn)
|
||||
create_snapshot_tables(conn)
|
||||
create_grafana_views(conn)
|
||||
views = _get_names(conn, "view")
|
||||
expected = {
|
||||
"grafana_rates",
|
||||
"grafana_ticks",
|
||||
"grafana_history_deals",
|
||||
"grafana_history_orders",
|
||||
"grafana_trade_deals",
|
||||
"grafana_cash_events",
|
||||
"grafana_realized_pnl",
|
||||
"grafana_symbol_pnl",
|
||||
"grafana_trade_stats",
|
||||
"grafana_account_snapshots",
|
||||
"grafana_position_snapshots",
|
||||
"grafana_order_snapshots",
|
||||
"grafana_terminal_snapshots",
|
||||
}
|
||||
assert expected.issubset(views)
|
||||
|
||||
def test_stale_view_dropped_when_source_table_disappears(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""create_grafana_views drops a previously created view whose source is gone."""
|
||||
_make_ticks_table(conn)
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_ticks" in _get_names(conn, "view")
|
||||
conn.execute("DROP TABLE ticks")
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_ticks" not in _get_names(conn, "view")
|
||||
|
||||
def test_grafana_rates_skipped_when_table_absent(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_rates is skipped when rates table is missing."""
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_rates" not in _get_names(conn, "view")
|
||||
|
||||
def test_grafana_rates_skipped_when_required_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_rates is skipped when rates table lacks required columns."""
|
||||
conn.execute("CREATE TABLE rates (open REAL)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_rates" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_rates" in caplog.text
|
||||
|
||||
def test_grafana_ticks_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_ticks is skipped when ticks table lacks required columns."""
|
||||
conn.execute("CREATE TABLE ticks (bid REAL)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_ticks" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_ticks" in caplog.text
|
||||
|
||||
def test_grafana_history_deals_skipped_when_time_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_history_deals is skipped when history_deals.time is missing."""
|
||||
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_history_deals" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_history_deals" in caplog.text
|
||||
|
||||
def test_grafana_history_orders_skipped_when_time_setup_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_history_orders is skipped when time_setup is absent."""
|
||||
conn.execute("CREATE TABLE history_orders (symbol TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_history_orders" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_history_orders" in caplog.text
|
||||
|
||||
def test_grafana_trade_deals_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_trade_deals is skipped when history_deals missing time/type."""
|
||||
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_trade_deals" not in _get_names(conn, "view")
|
||||
|
||||
def test_grafana_cash_events_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_cash_events is skipped when history_deals missing time/type."""
|
||||
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_cash_events" not in _get_names(conn, "view")
|
||||
|
||||
def test_grafana_realized_pnl_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_realized_pnl is skipped when history_deals missing required cols."""
|
||||
conn.execute("CREATE TABLE history_deals (time TEXT, type INTEGER)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_realized_pnl" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_realized_pnl" in caplog.text
|
||||
|
||||
def test_grafana_realized_pnl_skipped_when_entry_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_realized_pnl is skipped when entry column is absent."""
|
||||
_make_history_deals_minimal(conn)
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_realized_pnl" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_realized_pnl" in caplog.text
|
||||
|
||||
def test_grafana_symbol_pnl_skipped_when_required_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_symbol_pnl is skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE history_deals (time TEXT, type INTEGER)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_symbol_pnl" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_symbol_pnl" in caplog.text
|
||||
|
||||
def test_grafana_symbol_pnl_without_volume_and_price(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""grafana_symbol_pnl is created with only required columns."""
|
||||
conn.execute(
|
||||
"CREATE TABLE history_deals"
|
||||
" (time TEXT, symbol TEXT, profit REAL, type INTEGER, entry INTEGER)"
|
||||
)
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_symbol_pnl" in _get_names(conn, "view")
|
||||
|
||||
def test_grafana_symbol_pnl_with_volume_and_price(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""grafana_symbol_pnl includes volume and price columns when present."""
|
||||
_make_history_deals_full(conn)
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_symbol_pnl" in _get_names(conn, "view")
|
||||
# View columns include volume and price
|
||||
cols = {row[1] for row in conn.execute("PRAGMA table_info(grafana_symbol_pnl)")}
|
||||
assert "volume" in cols
|
||||
assert "price" in cols
|
||||
|
||||
def test_grafana_trade_stats_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""grafana_trade_stats is skipped when history_deals missing required cols."""
|
||||
conn.execute("CREATE TABLE history_deals (time TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_trade_stats" not in _get_names(conn, "view")
|
||||
assert "Skipping grafana_trade_stats" in caplog.text
|
||||
|
||||
def test_grafana_trade_stats_without_entry_col(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""grafana_trade_stats is a static summary view with no time column."""
|
||||
_make_history_deals_minimal(conn)
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_trade_stats" in _get_names(conn, "view")
|
||||
cols = {
|
||||
row[1] for row in conn.execute("PRAGMA table_info(grafana_trade_stats)")
|
||||
}
|
||||
assert "time" not in cols
|
||||
assert "symbol" in cols
|
||||
|
||||
def test_grafana_trade_stats_with_entry_col(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""grafana_trade_stats is a static summary view with no time column."""
|
||||
_make_history_deals_full(conn)
|
||||
create_grafana_views(conn)
|
||||
assert "grafana_trade_stats" in _get_names(conn, "view")
|
||||
cols = {
|
||||
row[1] for row in conn.execute("PRAGMA table_info(grafana_trade_stats)")
|
||||
}
|
||||
assert "time" not in cols
|
||||
assert "symbol" in cols
|
||||
|
||||
def test_snapshot_views_skipped_when_snapshot_tables_absent(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Snapshot views are skipped when snapshot tables are not created."""
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
create_grafana_views(conn)
|
||||
views = _get_names(conn, "view")
|
||||
assert "grafana_account_snapshots" not in views
|
||||
assert "grafana_position_snapshots" not in views
|
||||
assert "grafana_order_snapshots" not in views
|
||||
assert "grafana_terminal_snapshots" not in views
|
||||
|
||||
def test_build_snapshot_view_with_only_run_id_col(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""_build_snapshot_view exposes time and run_id when table has only run_id."""
|
||||
create_snapshot_tables(conn)
|
||||
conn.execute("CREATE TABLE only_run (run_id INTEGER NOT NULL)")
|
||||
run_id = start_snapshot_run(conn, 1000)
|
||||
record_snapshot_run(conn, run_id, "ok")
|
||||
conn.execute("INSERT INTO only_run (run_id) VALUES (?)", (run_id,))
|
||||
_build_snapshot_view(conn, "test_view", "only_run")
|
||||
assert "test_view" in _get_names(conn, "view")
|
||||
cols = {row[1] for row in conn.execute("PRAGMA table_info(test_view)")}
|
||||
assert "time" in cols
|
||||
assert "run_id" in cols
|
||||
|
||||
def test_build_snapshot_view_skips_when_snapshot_runs_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""_build_snapshot_view skips view when snapshot_runs has wrong columns."""
|
||||
conn.execute("CREATE TABLE only_run (run_id INTEGER NOT NULL)")
|
||||
conn.execute("CREATE TABLE snapshot_runs (foo TEXT)")
|
||||
_build_snapshot_view(conn, "test_view", "only_run")
|
||||
views = _get_names(conn, "view")
|
||||
assert "test_view" not in views
|
||||
|
||||
def test_build_snapshot_view_skips_when_run_id_col_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""_build_snapshot_view skips view when the table lacks run_id."""
|
||||
create_snapshot_tables(conn)
|
||||
conn.execute("CREATE TABLE no_run_id (symbol TEXT)")
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
|
||||
_build_snapshot_view(conn, "test_view", "no_run_id")
|
||||
assert "test_view" not in _get_names(conn, "view")
|
||||
assert "missing run_id column" in caplog.text
|
||||
|
||||
def test_snapshot_view_excludes_failed_run_rows(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Snapshot views hide rows from failed runs."""
|
||||
create_snapshot_tables(conn)
|
||||
run_id = start_snapshot_run(conn, 1000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots"
|
||||
" (run_id, login, balance, equity, margin, margin_free, profit)"
|
||||
" VALUES (?, 12345, 10000.0, 9800.0, 200.0, 9600.0, -200.0)",
|
||||
(run_id,),
|
||||
)
|
||||
record_snapshot_run(conn, run_id, "error", "terminal offline")
|
||||
create_grafana_views(conn)
|
||||
rows = conn.execute("SELECT * FROM grafana_account_snapshots").fetchall()
|
||||
assert rows == []
|
||||
|
||||
def test_snapshot_view_includes_ok_run_rows(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Snapshot views show rows from successful runs and expose run_id."""
|
||||
create_snapshot_tables(conn)
|
||||
run_id = start_snapshot_run(conn, 2000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots"
|
||||
" (run_id, login, balance, equity, margin, margin_free, profit)"
|
||||
" VALUES (?, 12345, 10000.0, 9800.0, 200.0, 9600.0, -200.0)",
|
||||
(run_id,),
|
||||
)
|
||||
record_snapshot_run(conn, run_id, "ok")
|
||||
create_grafana_views(conn)
|
||||
rows = conn.execute(
|
||||
"SELECT time, run_id, login FROM grafana_account_snapshots"
|
||||
).fetchall()
|
||||
assert rows == [(2000, run_id, 12345)]
|
||||
cols = {
|
||||
row[1]
|
||||
for row in conn.execute("PRAGMA table_info(grafana_account_snapshots)")
|
||||
}
|
||||
assert "run_id" in cols
|
||||
|
||||
def test_snapshot_view_same_second_ok_and_error_no_cross_contamination(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""An ok and error run sharing observed_at expose only the ok run's rows."""
|
||||
create_snapshot_tables(conn)
|
||||
run_err = start_snapshot_run(conn, 3000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 99)",
|
||||
(run_err,),
|
||||
)
|
||||
record_snapshot_run(conn, run_err, "error")
|
||||
run_ok = start_snapshot_run(conn, 3000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 12345)",
|
||||
(run_ok,),
|
||||
)
|
||||
record_snapshot_run(conn, run_ok, "ok")
|
||||
create_grafana_views(conn)
|
||||
rows = conn.execute("SELECT login FROM grafana_account_snapshots").fetchall()
|
||||
assert rows == [(12345,)]
|
||||
|
||||
def test_snapshot_view_two_ok_runs_same_second_no_duplication(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Two ok runs sharing observed_at each produce exactly one row in the view."""
|
||||
create_snapshot_tables(conn)
|
||||
run1 = start_snapshot_run(conn, 4000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 1)",
|
||||
(run1,),
|
||||
)
|
||||
record_snapshot_run(conn, run1, "ok")
|
||||
run2 = start_snapshot_run(conn, 4000)
|
||||
conn.execute(
|
||||
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 2)",
|
||||
(run2,),
|
||||
)
|
||||
record_snapshot_run(conn, run2, "ok")
|
||||
create_grafana_views(conn)
|
||||
rows = conn.execute("SELECT login FROM grafana_account_snapshots").fetchall()
|
||||
assert len(rows) == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestGrafanaIndexes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestGrafanaIndexes:
|
||||
"""Tests for create_grafana_indexes."""
|
||||
|
||||
def test_all_indexes_created_with_full_schema(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""All 9 indexes are created when all source tables are present."""
|
||||
_make_rates_table(conn)
|
||||
_make_ticks_table(conn)
|
||||
_make_history_deals_full(conn)
|
||||
_make_history_orders_table(conn)
|
||||
create_snapshot_tables(conn)
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_rates_time_symbol_timeframe" in indexes
|
||||
assert "idx_ticks_time_symbol" in indexes
|
||||
assert "idx_history_deals_time_symbol" in indexes
|
||||
assert "idx_history_deals_symbol_time" in indexes
|
||||
assert "idx_history_orders_time_setup_symbol" in indexes
|
||||
assert "idx_account_snapshots_time_login" in indexes
|
||||
assert "idx_position_snapshots_time_symbol" in indexes
|
||||
assert "idx_order_snapshots_time_symbol" in indexes
|
||||
assert "idx_snapshot_runs_time_status" in indexes
|
||||
|
||||
def test_no_indexes_created_when_tables_absent(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""No indexes are created when tables are absent."""
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert not any(name.startswith("idx_") for name in indexes)
|
||||
|
||||
def test_indexes_for_snapshot_tables_skipped_when_absent(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Snapshot table indexes are skipped when snapshot tables don't exist."""
|
||||
_make_history_deals_full(conn)
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_account_snapshots_time_login" not in indexes
|
||||
assert "idx_position_snapshots_time_symbol" not in indexes
|
||||
assert "idx_order_snapshots_time_symbol" not in indexes
|
||||
assert "idx_snapshot_runs_time_status" not in indexes
|
||||
|
||||
def test_rates_index_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Rates index is skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE rates (open REAL)")
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_rates_time_symbol_timeframe" not in indexes
|
||||
|
||||
def test_ticks_index_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Ticks index is skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE ticks (bid REAL)")
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_ticks_time_symbol" not in indexes
|
||||
|
||||
def test_deals_indexes_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""history_deals indexes are skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE history_deals (ticket INTEGER)")
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_history_deals_time_symbol" not in indexes
|
||||
|
||||
def test_orders_index_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""history_orders index is skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE history_orders (ticket INTEGER)")
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_history_orders_time_setup_symbol" not in indexes
|
||||
|
||||
def test_snapshot_indexes_skipped_when_cols_missing(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""Snapshot table indexes are skipped when required columns are absent."""
|
||||
conn.execute("CREATE TABLE account_snapshots (foo TEXT)")
|
||||
conn.execute("CREATE TABLE position_snapshots (foo TEXT)")
|
||||
conn.execute("CREATE TABLE order_snapshots (foo TEXT)")
|
||||
conn.execute("CREATE TABLE snapshot_runs (foo TEXT)")
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_account_snapshots_time_login" not in indexes
|
||||
assert "idx_position_snapshots_time_symbol" not in indexes
|
||||
assert "idx_order_snapshots_time_symbol" not in indexes
|
||||
assert "idx_snapshot_runs_time_status" not in indexes
|
||||
|
||||
def test_indexes_are_idempotent(self, conn: sqlite3.Connection) -> None:
|
||||
"""Creating indexes twice does not raise (IF NOT EXISTS)."""
|
||||
_make_rates_table(conn)
|
||||
create_grafana_indexes(conn)
|
||||
create_grafana_indexes(conn)
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_rates_time_symbol_timeframe" in indexes
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestEnsureGrafanaSchema
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEnsureGrafanaSchema:
|
||||
"""Tests for ensure_grafana_schema."""
|
||||
|
||||
def test_creates_all_tables_views_and_indexes(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""ensure_grafana_schema creates snapshot tables, views, and indexes."""
|
||||
_make_rates_table(conn)
|
||||
_make_history_deals_full(conn)
|
||||
ensure_grafana_schema(conn)
|
||||
tables = _get_names(conn, "table")
|
||||
assert "snapshot_runs" in tables
|
||||
assert "account_snapshots" in tables
|
||||
views = _get_names(conn, "view")
|
||||
assert "grafana_rates" in views
|
||||
assert "grafana_account_snapshots" in views
|
||||
indexes = _get_names(conn, "index")
|
||||
assert "idx_rates_time_symbol_timeframe" in indexes
|
||||
|
||||
def test_is_idempotent(self, conn: sqlite3.Connection) -> None:
|
||||
"""Calling ensure_grafana_schema twice does not raise."""
|
||||
ensure_grafana_schema(conn)
|
||||
ensure_grafana_schema(conn)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSnapshotInserts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSnapshotInserts:
|
||||
"""Tests for snapshot insert helpers."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_tables(self, conn: sqlite3.Connection) -> None:
|
||||
"""Create snapshot tables before each insert test."""
|
||||
create_snapshot_tables(conn)
|
||||
|
||||
def test_insert_account_snapshot(self, conn: sqlite3.Connection) -> None:
|
||||
"""insert_account_snapshot appends a row with correct values."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
row: dict[str, object] = {
|
||||
"login": 12345,
|
||||
"currency": "USD",
|
||||
"balance": 10000.0,
|
||||
"equity": 9800.0,
|
||||
"margin": 200.0,
|
||||
"margin_free": 9800.0,
|
||||
"margin_level": 4900.0,
|
||||
"profit": -200.0,
|
||||
"leverage": 100,
|
||||
}
|
||||
insert_account_snapshot(conn, run_id, row)
|
||||
result = conn.execute(
|
||||
"SELECT login, currency, balance FROM account_snapshots"
|
||||
).fetchone()
|
||||
assert result == (12345, "USD", 10000.0)
|
||||
|
||||
def test_insert_account_snapshot_partial_row(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_account_snapshot works when some fields are missing (uses None)."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
insert_account_snapshot(conn, run_id, {"login": 1})
|
||||
result = conn.execute(
|
||||
"SELECT login, currency FROM account_snapshots"
|
||||
).fetchone()
|
||||
assert result == (1, None)
|
||||
|
||||
def test_insert_position_snapshots_with_rows(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_position_snapshots appends each position row."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
rows: list[dict[str, object]] = [
|
||||
{"ticket": 1, "symbol": "EURUSD", "volume": 0.1, "profit": 10.0},
|
||||
{"ticket": 2, "symbol": "GBPUSD", "volume": 0.2, "profit": -5.0},
|
||||
]
|
||||
insert_position_snapshots(conn, run_id, 12345, rows)
|
||||
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[0]
|
||||
assert count == 2
|
||||
|
||||
def test_insert_position_snapshots_noop_when_empty(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_position_snapshots is a no-op when rows is empty."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
insert_position_snapshots(conn, run_id, 12345, [])
|
||||
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[0]
|
||||
assert count == 0
|
||||
|
||||
def test_insert_order_snapshots_with_rows(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_order_snapshots appends each order row."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
rows: list[dict[str, object]] = [
|
||||
{"ticket": 10, "symbol": "EURUSD", "type": 2, "volume_current": 0.1},
|
||||
]
|
||||
insert_order_snapshots(conn, run_id, 12345, rows)
|
||||
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
|
||||
assert count == 1
|
||||
|
||||
def test_insert_order_snapshots_noop_when_empty(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_order_snapshots is a no-op when rows is empty."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
insert_order_snapshots(conn, run_id, 12345, [])
|
||||
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
|
||||
assert count == 0
|
||||
|
||||
def test_insert_order_snapshots_normalizes_timestamp_time_setup(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_order_snapshots converts pd.Timestamp time_setup to epoch int."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
ts = pd.Timestamp("2024-01-15 10:30:00", tz="UTC")
|
||||
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": ts}]
|
||||
insert_order_snapshots(conn, run_id, 12345, rows)
|
||||
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
|
||||
assert stored == int(ts.timestamp())
|
||||
|
||||
def test_insert_order_snapshots_stores_int_time_setup(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_order_snapshots stores an integer time_setup as-is."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": 1705314600}]
|
||||
insert_order_snapshots(conn, run_id, 12345, rows)
|
||||
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
|
||||
assert stored == 1705314600
|
||||
|
||||
def test_insert_order_snapshots_stores_null_for_unknown_time_setup_type(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""insert_order_snapshots stores NULL for an unrecognized time_setup type."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": "not_a_time"}]
|
||||
insert_order_snapshots(conn, run_id, 12345, rows)
|
||||
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
|
||||
assert stored is None
|
||||
|
||||
def test_insert_terminal_snapshot(self, conn: sqlite3.Connection) -> None:
|
||||
"""insert_terminal_snapshot appends a terminal info row."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
row: dict[str, object] = {
|
||||
"name": "MetaTrader 5",
|
||||
"connected": 1,
|
||||
"community_account": 0,
|
||||
"trade_allowed": 1,
|
||||
"trade_expert": 1,
|
||||
"path": "/mt5",
|
||||
"company": "Broker",
|
||||
"language": "en",
|
||||
}
|
||||
insert_terminal_snapshot(conn, run_id, row)
|
||||
result = conn.execute(
|
||||
"SELECT name, connected FROM terminal_snapshots"
|
||||
).fetchone()
|
||||
assert result == ("MetaTrader 5", 1)
|
||||
|
||||
def test_start_snapshot_run_returns_incrementing_ids(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""start_snapshot_run returns a unique run_id for each call."""
|
||||
run1 = start_snapshot_run(conn, 1700000000)
|
||||
run2 = start_snapshot_run(conn, 1700000000)
|
||||
assert run1 != run2
|
||||
|
||||
def test_record_snapshot_run_with_detail(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""record_snapshot_run stores status and detail text."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
record_snapshot_run(conn, run_id, "error", "RuntimeError: boom")
|
||||
row = conn.execute("SELECT status, detail FROM snapshot_runs").fetchone()
|
||||
assert row == ("error", "RuntimeError: boom")
|
||||
|
||||
def test_record_snapshot_run_without_detail(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
) -> None:
|
||||
"""record_snapshot_run stores None for detail when omitted."""
|
||||
run_id = start_snapshot_run(conn, 1700000000)
|
||||
record_snapshot_run(conn, run_id, "ok")
|
||||
row = conn.execute("SELECT status, detail FROM snapshot_runs").fetchone()
|
||||
assert row == ("ok", None)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestPublishGrafanaCopy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_source_db(path: Path) -> None:
|
||||
"""Create a minimal source SQLite database with snapshot tables."""
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
create_snapshot_tables(conn)
|
||||
conn.execute(
|
||||
"INSERT INTO snapshot_runs (observed_at, status) VALUES (?, 'ok')",
|
||||
(1700000000,),
|
||||
)
|
||||
|
||||
|
||||
class TestPublishGrafanaCopy:
|
||||
"""Tests for publish_grafana_copy."""
|
||||
|
||||
def test_publish_to_fresh_target(self, tmp_path: Path) -> None:
|
||||
"""publish_grafana_copy creates the target file."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "out" / "grafana.db"
|
||||
_make_source_db(source)
|
||||
result = publish_grafana_copy(source, target)
|
||||
assert target.exists()
|
||||
assert result == target.resolve()
|
||||
|
||||
def test_overwrite_existing_target(self, tmp_path: Path) -> None:
|
||||
"""publish_grafana_copy replaces an existing target without error."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
target.write_bytes(b"stale")
|
||||
publish_grafana_copy(source, target)
|
||||
# Target must now be a valid SQLite file from source
|
||||
with sqlite3.connect(target) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
).fetchall()
|
||||
}
|
||||
assert "snapshot_runs" in tables
|
||||
|
||||
def test_target_contains_source_tables(self, tmp_path: Path) -> None:
|
||||
"""Published target contains the same tables as the source."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
publish_grafana_copy(source, target)
|
||||
with sqlite3.connect(target) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
).fetchall()
|
||||
}
|
||||
assert {"snapshot_runs", "account_snapshots"}.issubset(tables)
|
||||
|
||||
def test_target_can_be_opened_readonly(self, tmp_path: Path) -> None:
|
||||
"""Published target can be opened with uri=True in read-only mode."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
publish_grafana_copy(source, target)
|
||||
uri = f"file:{target}?mode=ro"
|
||||
with sqlite3.connect(uri, uri=True) as conn:
|
||||
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
|
||||
assert row == ("ok",)
|
||||
|
||||
def test_same_path_raises(self, tmp_path: Path) -> None:
|
||||
"""publish_grafana_copy raises ValueError when source equals target."""
|
||||
db = tmp_path / "history.db"
|
||||
_make_source_db(db)
|
||||
with pytest.raises(ValueError, match="must differ from the source"):
|
||||
publish_grafana_copy(db, db)
|
||||
|
||||
def test_source_not_found_raises(self, tmp_path: Path) -> None:
|
||||
"""publish_grafana_copy raises FileNotFoundError when source is absent."""
|
||||
with pytest.raises(FileNotFoundError):
|
||||
publish_grafana_copy(tmp_path / "missing.db", tmp_path / "out.db")
|
||||
|
||||
def test_preserve_old_target_on_backup_failure(self, tmp_path: Path) -> None:
|
||||
"""Old target is preserved when the backup fails."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
original_content = b"original_data"
|
||||
target.write_bytes(original_content)
|
||||
with patch("sqlite3.connect") as mock_connect:
|
||||
mock_src = MagicMock()
|
||||
mock_src.__enter__ = MagicMock(return_value=mock_src)
|
||||
mock_src.__exit__ = MagicMock(return_value=False)
|
||||
mock_src.backup.side_effect = sqlite3.OperationalError("backup failed")
|
||||
mock_connect.return_value = mock_src
|
||||
with pytest.raises(sqlite3.OperationalError, match="backup failed"):
|
||||
publish_grafana_copy(source, target)
|
||||
assert target.read_bytes() == original_content
|
||||
|
||||
def test_temp_file_cleaned_up_on_failure(self, tmp_path: Path) -> None:
|
||||
"""Temporary file is removed when backup raises an exception."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
with patch("sqlite3.connect") as mock_connect:
|
||||
mock_src = MagicMock()
|
||||
mock_src.__enter__ = MagicMock(return_value=mock_src)
|
||||
mock_src.__exit__ = MagicMock(return_value=False)
|
||||
mock_src.backup.side_effect = sqlite3.OperationalError("fail")
|
||||
mock_connect.return_value = mock_src
|
||||
with pytest.raises(sqlite3.OperationalError):
|
||||
publish_grafana_copy(source, target)
|
||||
tmp_files = list(tmp_path.glob("grafana.db.*.tmp"))
|
||||
assert not tmp_files, "Temp file should be cleaned up on failure"
|
||||
|
||||
def test_returns_path_object(self, tmp_path: Path) -> None:
|
||||
"""publish_grafana_copy returns a Path instance."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
result = publish_grafana_copy(source, target)
|
||||
assert isinstance(result, Path)
|
||||
|
||||
def test_fresh_target_has_readable_permissions(self, tmp_path: Path) -> None:
|
||||
"""Published copy is readable by the owner."""
|
||||
import stat as _stat # noqa: PLC0415
|
||||
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
publish_grafana_copy(source, target)
|
||||
mode = target.stat().st_mode & 0o777
|
||||
assert bool(mode & _stat.S_IRUSR), "owner must be able to read"
|
||||
|
||||
@pytest.mark.skipif(
|
||||
__import__("sys").platform == "win32",
|
||||
reason="Windows does not support Unix-style group/other permission bits",
|
||||
)
|
||||
def test_overwrite_preserves_existing_target_mode(self, tmp_path: Path) -> None:
|
||||
"""Overwriting an existing target preserves that target's file mode."""
|
||||
source = tmp_path / "src.db"
|
||||
target = tmp_path / "grafana.db"
|
||||
_make_source_db(source)
|
||||
target.write_bytes(b"old")
|
||||
target.chmod(0o640)
|
||||
publish_grafana_copy(source, target)
|
||||
mode = target.stat().st_mode & 0o777
|
||||
assert mode == 0o640
|
||||
+38
-57
@@ -15,8 +15,11 @@ from pytest_mock import MockerFixture # noqa: TC002
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from pdmt5 import TIMEFRAME_MAP
|
||||
|
||||
from mt5cli import history
|
||||
from mt5cli.history import (
|
||||
DEFAULT_HISTORY_DATASETS,
|
||||
DEFAULT_HISTORY_TIMEFRAMES,
|
||||
DedupScope,
|
||||
RateTarget,
|
||||
@@ -58,7 +61,7 @@ from mt5cli.history import (
|
||||
write_rates_dataset,
|
||||
write_streamed_frame,
|
||||
)
|
||||
from mt5cli.utils import TIMEFRAME_MAP, Dataset, IfExists
|
||||
from mt5cli.utils import Dataset, IfExists
|
||||
|
||||
|
||||
class TestResolveRateViewName:
|
||||
@@ -545,10 +548,23 @@ class TestResolveHistorySettings:
|
||||
"""Tests for history dataset and timeframe resolution."""
|
||||
|
||||
def test_resolve_history_datasets_defaults_and_empty(self) -> None:
|
||||
"""Test dataset resolution distinguishes None from empty selection."""
|
||||
assert resolve_history_datasets(None) == set(Dataset)
|
||||
"""Test dataset resolution excludes ticks by default."""
|
||||
resolved = resolve_history_datasets(None)
|
||||
assert resolved == set(DEFAULT_HISTORY_DATASETS)
|
||||
assert Dataset.ticks not in resolved
|
||||
assert {
|
||||
Dataset.rates,
|
||||
Dataset.history_orders,
|
||||
Dataset.history_deals,
|
||||
} == resolved
|
||||
assert resolve_history_datasets(set()) == set()
|
||||
|
||||
def test_resolve_history_datasets_explicit_ticks(self) -> None:
|
||||
"""Test that explicit ticks selection is honored."""
|
||||
assert resolve_history_datasets({Dataset.ticks}) == {Dataset.ticks}
|
||||
all_ds = resolve_history_datasets(set(Dataset))
|
||||
assert Dataset.ticks in all_ds
|
||||
|
||||
def test_resolve_history_timeframes_defaults(self) -> None:
|
||||
"""Test default timeframes include all fixed MT5 values."""
|
||||
resolved = resolve_history_timeframes(None)
|
||||
@@ -705,19 +721,25 @@ class TestIncrementalStart:
|
||||
assert starts["EURUSD", 1] == datetime(2024, 1, 2, tzinfo=UTC)
|
||||
assert starts["GBPUSD", 1] == datetime(2024, 1, 3, tzinfo=UTC)
|
||||
|
||||
def test_load_incremental_start_datetimes_requires_timeframe_column(
|
||||
@pytest.mark.parametrize(
|
||||
("ddl", "missing_col"),
|
||||
[
|
||||
("CREATE TABLE rates(symbol TEXT, time TEXT, open REAL)", "timeframe"),
|
||||
("CREATE TABLE rates(timeframe INTEGER, time TEXT, open REAL)", "symbol"),
|
||||
("CREATE TABLE rates(symbol TEXT, timeframe INTEGER, open REAL)", "time"),
|
||||
],
|
||||
)
|
||||
def test_load_incremental_start_datetimes_requires_column(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
ddl: str,
|
||||
missing_col: str,
|
||||
) -> None:
|
||||
"""Test rates tables without timeframe fail fast during incremental resume."""
|
||||
"""Test rates tables missing a required column fail fast."""
|
||||
fallback = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "rates-without-timeframe.db") as conn:
|
||||
conn.execute("CREATE TABLE rates(symbol TEXT, time TEXT, open REAL)")
|
||||
conn.execute(
|
||||
"INSERT INTO rates(symbol, time, open) VALUES (?, ?, ?)",
|
||||
("EURUSD", "2024-01-02T00:00:00+00:00", 1.0),
|
||||
)
|
||||
with pytest.raises(ValueError, match="missing: timeframe") as exc_info:
|
||||
with sqlite3.connect(tmp_path / f"rates-no-{missing_col}.db") as conn:
|
||||
conn.execute(ddl)
|
||||
with pytest.raises(ValueError, match=f"missing: {missing_col}") as exc_info:
|
||||
load_incremental_start_datetimes(
|
||||
conn,
|
||||
Dataset.rates,
|
||||
@@ -725,47 +747,7 @@ class TestIncrementalStart:
|
||||
timeframes=[1],
|
||||
fallback_start=fallback,
|
||||
)
|
||||
assert "timeframe" in str(exc_info.value)
|
||||
|
||||
def test_load_incremental_start_datetimes_requires_symbol_column(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test rates tables without symbol fail fast during incremental resume."""
|
||||
fallback = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "rates-no-symbol.db") as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates(timeframe INTEGER, time TEXT, open REAL)",
|
||||
)
|
||||
with pytest.raises(ValueError, match="missing: symbol") as exc_info:
|
||||
load_incremental_start_datetimes(
|
||||
conn,
|
||||
Dataset.rates,
|
||||
symbols=["EURUSD"],
|
||||
timeframes=[1],
|
||||
fallback_start=fallback,
|
||||
)
|
||||
assert "symbol" in str(exc_info.value)
|
||||
|
||||
def test_load_incremental_start_datetimes_requires_time_column(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test rates tables without time fail fast during incremental resume."""
|
||||
fallback = datetime(2024, 1, 1, tzinfo=UTC)
|
||||
with sqlite3.connect(tmp_path / "rates-no-time.db") as conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE rates(symbol TEXT, timeframe INTEGER, open REAL)",
|
||||
)
|
||||
with pytest.raises(ValueError, match="missing: time") as exc_info:
|
||||
load_incremental_start_datetimes(
|
||||
conn,
|
||||
Dataset.rates,
|
||||
symbols=["EURUSD"],
|
||||
timeframes=[1],
|
||||
fallback_start=fallback,
|
||||
)
|
||||
assert "time" in str(exc_info.value)
|
||||
assert missing_col in str(exc_info.value)
|
||||
|
||||
def test_load_incremental_start_datetimes_rejects_unrelated_rates_columns(
|
||||
self,
|
||||
@@ -1800,12 +1782,11 @@ class TestIncrementalIntegration:
|
||||
)
|
||||
assert written_tables == set()
|
||||
|
||||
def test_resolve_history_tick_flags_invalid(self) -> None:
|
||||
@pytest.mark.parametrize("flags", ["BAD", 7])
|
||||
def test_resolve_history_tick_flags_invalid(self, flags: str | int) -> None:
|
||||
"""Test invalid tick flags raise ValueError."""
|
||||
with pytest.raises(ValueError, match="Invalid tick flags"):
|
||||
resolve_history_tick_flags("BAD")
|
||||
with pytest.raises(ValueError, match="Invalid tick flags"):
|
||||
resolve_history_tick_flags(7)
|
||||
resolve_history_tick_flags(flags)
|
||||
|
||||
def test_resolve_history_timeframes_invalid(self) -> None:
|
||||
"""Test invalid timeframes raise ValueError."""
|
||||
|
||||
+872
-47
@@ -54,12 +54,15 @@ from mt5cli.sdk import (
|
||||
resolve_account_spec,
|
||||
resolve_account_specs,
|
||||
substitute_env_placeholders,
|
||||
substitute_mapping_values,
|
||||
symbol_info,
|
||||
symbol_info_tick,
|
||||
symbols,
|
||||
terminal_info,
|
||||
update_history,
|
||||
update_history_with_config,
|
||||
update_observability,
|
||||
update_observability_with_config,
|
||||
version,
|
||||
)
|
||||
from mt5cli.utils import Dataset, IfExists, coerce_login
|
||||
@@ -617,12 +620,12 @@ class TestCollectHistory:
|
||||
"""Create a mocked Mt5DataClient with history-style DataFrames."""
|
||||
return _build_history_client(mocker)
|
||||
|
||||
def test_collect_history_writes_all_tables(
|
||||
def test_collect_history_writes_default_tables(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
history_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that collect_history writes rates, ticks, and history tables."""
|
||||
"""Test that collect_history default excludes ticks."""
|
||||
output = tmp_path / "history.db"
|
||||
collect_history(
|
||||
output,
|
||||
@@ -631,7 +634,7 @@ class TestCollectHistory:
|
||||
"2024-02-01",
|
||||
)
|
||||
assert history_client.copy_rates_range_as_df.call_count == 2
|
||||
assert history_client.copy_ticks_range_as_df.call_count == 2
|
||||
assert history_client.copy_ticks_range_as_df.call_count == 0
|
||||
with sqlite3.connect(output) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
@@ -639,7 +642,34 @@ class TestCollectHistory:
|
||||
"SELECT name FROM sqlite_master WHERE type='table'",
|
||||
).fetchall()
|
||||
}
|
||||
assert {"rates", "ticks", "history_orders", "history_deals"} <= tables
|
||||
assert {"rates", "history_orders", "history_deals"} <= tables
|
||||
assert "ticks" not in tables
|
||||
|
||||
def test_collect_history_explicit_ticks_dataset(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
history_client: MagicMock,
|
||||
) -> None:
|
||||
"""Test that explicit datasets={Dataset.ticks} writes the ticks table."""
|
||||
output = tmp_path / "history.db"
|
||||
collect_history(
|
||||
output,
|
||||
["EURUSD", "GBPUSD"],
|
||||
"2024-01-01",
|
||||
"2024-02-01",
|
||||
datasets={Dataset.ticks},
|
||||
)
|
||||
assert history_client.copy_ticks_range_as_df.call_count == 2
|
||||
assert history_client.copy_rates_range_as_df.call_count == 0
|
||||
with sqlite3.connect(output) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'",
|
||||
).fetchall()
|
||||
}
|
||||
assert "ticks" in tables
|
||||
assert "rates" not in tables
|
||||
|
||||
def test_collect_history_with_views(
|
||||
self,
|
||||
@@ -1065,6 +1095,40 @@ class TestUpdateHistory:
|
||||
after = datetime.now(UTC)
|
||||
assert before <= captured["end"] <= after
|
||||
|
||||
def test_update_history_default_datasets_exclude_ticks(
|
||||
self,
|
||||
connected_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Test update_history with datasets=None does not collect ticks."""
|
||||
datasets_written: list[set[Dataset]] = []
|
||||
|
||||
def capture(
|
||||
*args: object,
|
||||
**_kwargs: object,
|
||||
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
|
||||
datasets_written.append(args[3]) # type: ignore[arg-type]
|
||||
return set(), {}
|
||||
|
||||
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
|
||||
update_history(
|
||||
client=connected_client,
|
||||
output=tmp_path / "default-datasets.db",
|
||||
symbols=["EURUSD"],
|
||||
datasets=None,
|
||||
timeframes=["M1"],
|
||||
lookback_hours=1,
|
||||
date_to=datetime(2024, 1, 1, tzinfo=UTC),
|
||||
)
|
||||
assert len(datasets_written) == 1
|
||||
assert Dataset.ticks not in datasets_written[0]
|
||||
assert {
|
||||
Dataset.rates,
|
||||
Dataset.history_orders,
|
||||
Dataset.history_deals,
|
||||
} == datasets_written[0]
|
||||
|
||||
|
||||
class TestRecentTicks:
|
||||
"""Tests for recent_ticks helper."""
|
||||
@@ -1937,29 +2001,28 @@ class TestResolveAccountSpec:
|
||||
assert [a.server for a in resolved] == ["Shared", "Fixed"]
|
||||
assert all(a.timeout == 1000 for a in resolved)
|
||||
|
||||
def test_resolve_account_spec_with_whole_dollar_env(
|
||||
@pytest.mark.parametrize(
|
||||
("allow_whole_dollar_env", "expected"),
|
||||
[
|
||||
(True, "secret"),
|
||||
(False, "$MT5_PASSWORD"),
|
||||
],
|
||||
)
|
||||
def test_resolve_account_spec_whole_dollar_password(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
allow_whole_dollar_env: bool,
|
||||
expected: str,
|
||||
) -> None:
|
||||
"""Account spec expands $ENV_NAME when allow_whole_dollar_env=True."""
|
||||
"""Test resolve_account_spec expands $ENV_NAME password only with opt-in."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
account = AccountSpec(symbols=["EURUSD"], password="$MT5_PASSWORD")
|
||||
|
||||
resolved = resolve_account_spec(account, allow_whole_dollar_env=True)
|
||||
resolved = resolve_account_spec(
|
||||
account, allow_whole_dollar_env=allow_whole_dollar_env
|
||||
)
|
||||
|
||||
assert resolved.password == "secret" # noqa: S105
|
||||
|
||||
def test_resolve_account_spec_whole_dollar_not_expanded_by_default(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test resolve_account_spec leaves $ENV_NAME literal by default."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
account = AccountSpec(symbols=["EURUSD"], password="$MT5_PASSWORD")
|
||||
|
||||
resolved = resolve_account_spec(account)
|
||||
|
||||
assert resolved.password == "$MT5_PASSWORD" # noqa: S105
|
||||
assert resolved.password == expected
|
||||
|
||||
def test_resolve_account_specs_with_whole_dollar_env(
|
||||
self,
|
||||
@@ -1993,38 +2056,27 @@ class TestResolveAccountSpec:
|
||||
class TestBuildConfigWholeDollarEnv:
|
||||
"""Tests for build_config with allow_whole_dollar_env."""
|
||||
|
||||
def test_build_config_substitutes_server_with_opt_in(
|
||||
@pytest.mark.parametrize(
|
||||
("env_var", "field", "env_value"),
|
||||
[
|
||||
("MT5_SERVER", "server", "Broker-Demo"),
|
||||
("MT5_PASSWORD", "password", "secret"),
|
||||
("MT5_PATH", "path", "/opt/mt5/terminal64.exe"),
|
||||
],
|
||||
)
|
||||
def test_build_config_substitutes_field_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
env_var: str,
|
||||
field: str,
|
||||
env_value: str,
|
||||
) -> None:
|
||||
"""build_config expands $ENV_NAME server when allow_whole_dollar_env=True."""
|
||||
monkeypatch.setenv("MT5_SERVER", "Broker-Demo")
|
||||
"""Test build_config expands $ENV_NAME fields when opt-in is enabled."""
|
||||
monkeypatch.setenv(env_var, env_value)
|
||||
|
||||
config = build_config(server="$MT5_SERVER", allow_whole_dollar_env=True)
|
||||
config = build_config(**{field: f"${env_var}"}, allow_whole_dollar_env=True) # type: ignore[arg-type]
|
||||
|
||||
assert config.server == "Broker-Demo"
|
||||
|
||||
def test_build_config_substitutes_password_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""build_config expands $ENV_NAME password when allow_whole_dollar_env=True."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
|
||||
config = build_config(password="$MT5_PASSWORD", allow_whole_dollar_env=True)
|
||||
|
||||
assert config.password == "secret" # noqa: S105
|
||||
|
||||
def test_build_config_substitutes_path_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test build_config expands $ENV_NAME path when allow_whole_dollar_env=True."""
|
||||
monkeypatch.setenv("MT5_PATH", "/opt/mt5/terminal64.exe")
|
||||
|
||||
config = build_config(path="$MT5_PATH", allow_whole_dollar_env=True)
|
||||
|
||||
assert config.path == "/opt/mt5/terminal64.exe"
|
||||
assert getattr(config, field) == env_value
|
||||
|
||||
def test_build_config_leaves_dollar_literal_by_default(
|
||||
self,
|
||||
@@ -2436,3 +2488,776 @@ class TestThrottledHistoryUpdater:
|
||||
updater.update(MagicMock(), ["EURUSD"])
|
||||
|
||||
assert updater.last_update_monotonic is None
|
||||
|
||||
|
||||
class TestBuildConfigStringLogin:
|
||||
"""Tests for build_config() string login coercion (issue #61)."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("login", "expected"),
|
||||
[
|
||||
(None, None),
|
||||
(12345, 12345),
|
||||
("12345", 12345),
|
||||
(" 12345 ", 12345),
|
||||
("", None),
|
||||
(" ", None),
|
||||
],
|
||||
)
|
||||
def test_coerces_login_from_string(
|
||||
self,
|
||||
login: int | str | None,
|
||||
expected: int | None,
|
||||
) -> None:
|
||||
"""Test build_config coerces string login to int or None."""
|
||||
config = build_config(login=login)
|
||||
assert config.login == expected
|
||||
|
||||
def test_rejects_non_numeric_string_login(self) -> None:
|
||||
"""Test build_config raises ValueError for non-numeric string login."""
|
||||
with pytest.raises(ValueError, match="invalid literal"):
|
||||
build_config(login="abc")
|
||||
|
||||
def test_expands_dollar_brace_login_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test build_config expands ${MT5_LOGIN} and coerces with opt-in."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "12345")
|
||||
config = build_config(login="${MT5_LOGIN}", allow_whole_dollar_env=True)
|
||||
assert config.login == 12345
|
||||
|
||||
def test_expands_whole_dollar_login_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test build_config expands $MT5_LOGIN and coerces with opt-in."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "99999")
|
||||
config = build_config(login="$MT5_LOGIN", allow_whole_dollar_env=True)
|
||||
assert config.login == 99999
|
||||
|
||||
def test_missing_env_variable_raises(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test build_config raises ValueError when referenced env var is not set."""
|
||||
monkeypatch.delenv("MT5_LOGIN", raising=False)
|
||||
with pytest.raises(ValueError, match="'MT5_LOGIN' is not set"):
|
||||
build_config(login="${MT5_LOGIN}", allow_whole_dollar_env=True)
|
||||
|
||||
def test_env_expands_to_blank_becomes_none(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test build_config coerces blank env-expanded login to None."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "")
|
||||
config = build_config(login="${MT5_LOGIN}", allow_whole_dollar_env=True)
|
||||
assert config.login is None
|
||||
|
||||
def test_dollar_brace_login_not_expanded_without_opt_in(self) -> None:
|
||||
"""Test ${MT5_LOGIN} is not expanded when allow_whole_dollar_env=False."""
|
||||
with pytest.raises(ValueError, match="invalid literal"):
|
||||
build_config(login="${MT5_LOGIN}")
|
||||
|
||||
def test_integer_login_preserved_backward_compat(self) -> None:
|
||||
"""Test existing int login callers remain backward-compatible."""
|
||||
config = build_config(login=54321)
|
||||
assert config.login == 54321
|
||||
|
||||
def test_none_login_preserved_backward_compat(self) -> None:
|
||||
"""Test existing None login callers remain backward-compatible."""
|
||||
config = build_config(login=None)
|
||||
assert config.login is None
|
||||
|
||||
|
||||
class TestSubstituteMappingValues:
|
||||
"""Tests for substitute_mapping_values() (issue #62)."""
|
||||
|
||||
def test_substitutes_selected_keys_in_flat_dict(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test selected keys are substituted in a flat mapping."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "12345")
|
||||
data: dict[str, object] = {
|
||||
"mt5_login": "${MT5_LOGIN}",
|
||||
"strategy_name": "${MT5_LOGIN}",
|
||||
}
|
||||
result = substitute_mapping_values(data, keys={"mt5_login"})
|
||||
assert result == {"mt5_login": "12345", "strategy_name": "${MT5_LOGIN}"}
|
||||
|
||||
def test_preserves_non_selected_literal_dollar_signs(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test literal dollar signs in non-selected fields are preserved exactly."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
data: dict[str, object] = {
|
||||
"mt5_password": "${MT5_PASSWORD}",
|
||||
"notes": "$NOT_EXPANDED",
|
||||
}
|
||||
result = substitute_mapping_values(data, keys={"mt5_password"})
|
||||
assert result == {"mt5_password": "secret", "notes": "$NOT_EXPANDED"}
|
||||
|
||||
def test_nested_dict_traversal_substitutes_selected_keys(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test selected keys inside nested dicts are substituted."""
|
||||
monkeypatch.setenv("MT5_SERVER", "Broker-Demo")
|
||||
data: dict[str, object] = {
|
||||
"outer": {
|
||||
"mt5_server": "${MT5_SERVER}",
|
||||
"other": "${MT5_SERVER}",
|
||||
}
|
||||
}
|
||||
result = substitute_mapping_values(data, keys={"mt5_server"})
|
||||
assert result == {
|
||||
"outer": {"mt5_server": "Broker-Demo", "other": "${MT5_SERVER}"}
|
||||
}
|
||||
|
||||
def test_nested_list_traversal_substitutes_selected_keys(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test selected keys inside list elements are substituted."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "42")
|
||||
data: dict[str, object] = {
|
||||
"accounts": [
|
||||
{"mt5_login": "${MT5_LOGIN}", "name": "${MT5_LOGIN}"},
|
||||
{"mt5_login": "${MT5_LOGIN}", "name": "fixed"},
|
||||
]
|
||||
}
|
||||
result = substitute_mapping_values(data, keys={"mt5_login"})
|
||||
assert result == {
|
||||
"accounts": [
|
||||
{"mt5_login": "42", "name": "${MT5_LOGIN}"},
|
||||
{"mt5_login": "42", "name": "fixed"},
|
||||
]
|
||||
}
|
||||
|
||||
def test_whole_dollar_expanded_with_opt_in(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test $ENV_NAME is expanded when allow_whole_dollar_env=True."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
data: dict[str, object] = {"mt5_password": "$MT5_PASSWORD"}
|
||||
result = substitute_mapping_values(
|
||||
data,
|
||||
keys={"mt5_password"},
|
||||
allow_whole_dollar_env=True,
|
||||
)
|
||||
assert result == {"mt5_password": "secret"}
|
||||
|
||||
def test_whole_dollar_not_expanded_by_default(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test $ENV_NAME in a selected key is preserved when opt-in is False."""
|
||||
monkeypatch.setenv("MT5_PASSWORD", "secret")
|
||||
data: dict[str, object] = {"mt5_password": "$MT5_PASSWORD"}
|
||||
result = substitute_mapping_values(data, keys={"mt5_password"})
|
||||
assert result == {"mt5_password": "$MT5_PASSWORD"}
|
||||
|
||||
def test_blank_string_becomes_none_for_blank_keys(self) -> None:
|
||||
"""Test blank strings are normalised to None for blank_string_keys_as_none."""
|
||||
data: dict[str, object] = {
|
||||
"mt5_login": "",
|
||||
"mt5_password": " ",
|
||||
"other": "",
|
||||
}
|
||||
result = substitute_mapping_values(
|
||||
data,
|
||||
keys=set(),
|
||||
blank_string_keys_as_none={"mt5_login", "mt5_password"},
|
||||
)
|
||||
assert result == {"mt5_login": None, "mt5_password": None, "other": ""}
|
||||
|
||||
def test_env_expanded_blank_becomes_none(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test env-expanded blank string is normalised to None."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "")
|
||||
data: dict[str, object] = {"mt5_login": "${MT5_LOGIN}"}
|
||||
result = substitute_mapping_values(
|
||||
data,
|
||||
keys={"mt5_login"},
|
||||
blank_string_keys_as_none={"mt5_login"},
|
||||
)
|
||||
assert result == {"mt5_login": None}
|
||||
|
||||
def test_missing_env_variable_raises_for_selected_key(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test missing env var for a selected key raises ValueError."""
|
||||
monkeypatch.delenv("MT5_MISSING", raising=False)
|
||||
data: dict[str, object] = {"mt5_login": "${MT5_MISSING}"}
|
||||
with pytest.raises(ValueError, match="'MT5_MISSING' is not set"):
|
||||
substitute_mapping_values(data, keys={"mt5_login"})
|
||||
|
||||
def test_non_string_values_preserved(self) -> None:
|
||||
"""Test non-string values under selected or non-selected keys are preserved."""
|
||||
data: dict[str, object] = {
|
||||
"mt5_login": 12345,
|
||||
"timeout": 5000,
|
||||
"enabled": True,
|
||||
"ratio": 1.5,
|
||||
"nothing": None,
|
||||
}
|
||||
result = substitute_mapping_values(
|
||||
data, keys={"mt5_login", "timeout", "enabled", "ratio", "nothing"}
|
||||
)
|
||||
assert result == data
|
||||
|
||||
def test_caller_supplied_key_set_substitutes_correctly(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test helper works with any caller-supplied key set."""
|
||||
monkeypatch.setenv("APP_LOGIN", "77777")
|
||||
monkeypatch.setenv("APP_PASSWORD", "p4ss")
|
||||
data: dict[str, object] = {
|
||||
"app_login": "${APP_LOGIN}",
|
||||
"app_password": "${APP_PASSWORD}",
|
||||
"unrelated": "${APP_LOGIN}",
|
||||
}
|
||||
credential_keys = {"app_login", "app_password"}
|
||||
result = substitute_mapping_values(data, keys=credential_keys)
|
||||
assert result == {
|
||||
"app_login": "77777",
|
||||
"app_password": "p4ss",
|
||||
"unrelated": "${APP_LOGIN}",
|
||||
}
|
||||
|
||||
def test_scalar_data_returned_unchanged(self) -> None:
|
||||
"""Test a scalar (non-dict, non-list) value is returned as-is."""
|
||||
assert substitute_mapping_values("hello", keys={"x"}) == "hello"
|
||||
assert substitute_mapping_values(42, keys={"x"}) == 42
|
||||
assert substitute_mapping_values(None, keys={"x"}) is None
|
||||
|
||||
def test_tuple_container_not_traversed(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test tuple containers are returned as-is without traversal."""
|
||||
monkeypatch.setenv("MT5_LOGIN", "42")
|
||||
data: dict[str, object] = {"accounts": ({"mt5_login": "${MT5_LOGIN}"},)}
|
||||
result = substitute_mapping_values(data, keys={"mt5_login"})
|
||||
# tuple is returned as-is; inner dict is NOT visited
|
||||
assert result == {"accounts": ({"mt5_login": "${MT5_LOGIN}"},)}
|
||||
|
||||
|
||||
class TestUpdateObservability:
|
||||
"""Tests for update_observability and update_observability_with_config."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self) -> MagicMock:
|
||||
"""Mock client returning minimal valid frames."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_df.return_value = pd.DataFrame([
|
||||
{
|
||||
"login": 12345,
|
||||
"currency": "USD",
|
||||
"balance": 10000.0,
|
||||
"equity": 10000.0,
|
||||
"margin": 0.0,
|
||||
"margin_free": 10000.0,
|
||||
"margin_level": 0.0,
|
||||
"profit": 0.0,
|
||||
"leverage": 100,
|
||||
}
|
||||
])
|
||||
client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
client.terminal_info_as_df.return_value = pd.DataFrame([
|
||||
{
|
||||
"name": "MetaTrader 5",
|
||||
"connected": 1,
|
||||
"community_account": 0,
|
||||
"trade_allowed": 1,
|
||||
"trade_expert": 1,
|
||||
"path": "/mt5",
|
||||
"company": "Broker",
|
||||
"language": "en",
|
||||
}
|
||||
])
|
||||
return client
|
||||
|
||||
def test_update_observability_creates_snapshot_tables(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Snapshot tables are created in the output database."""
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=mock_client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
).fetchall()
|
||||
}
|
||||
assert "snapshot_runs" in tables
|
||||
assert "account_snapshots" in tables
|
||||
assert "position_snapshots" in tables
|
||||
|
||||
def test_update_observability_records_ok_on_success(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""snapshot_runs records 'ok' status on a successful run."""
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=mock_client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
|
||||
assert row == ("ok",)
|
||||
|
||||
def test_update_observability_records_error_on_failure(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""snapshot_runs records 'error' and re-raises when a snapshot fails."""
|
||||
mock_client.account_info_as_df.side_effect = RuntimeError("boom")
|
||||
output = tmp_path / "obs.db"
|
||||
with pytest.raises(RuntimeError, match="boom"):
|
||||
update_observability(client=mock_client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
|
||||
assert row == ("error",)
|
||||
|
||||
def test_update_observability_skips_ensure_grafana_schema_when_disabled(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""with_grafana_schema=False does not call ensure_grafana_schema."""
|
||||
spy = mocker.spy(sdk, "ensure_grafana_schema")
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
with_grafana_schema=False,
|
||||
)
|
||||
spy.assert_not_called()
|
||||
|
||||
def test_update_observability_calls_ensure_grafana_schema_by_default(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""with_grafana_schema=True calls ensure_grafana_schema."""
|
||||
spy = mocker.spy(sdk, "ensure_grafana_schema")
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
with_grafana_schema=True,
|
||||
)
|
||||
spy.assert_called_once()
|
||||
|
||||
def test_update_observability_skips_account_when_disabled(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""include_account=False does not call account_info_as_df."""
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
include_account=False,
|
||||
)
|
||||
mock_client.account_info_as_df.assert_not_called()
|
||||
|
||||
def test_update_observability_skips_positions_when_disabled(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""include_positions=False does not call positions_get_as_df."""
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
include_positions=False,
|
||||
)
|
||||
mock_client.positions_get_as_df.assert_not_called()
|
||||
|
||||
def test_update_observability_skips_orders_when_disabled(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""include_orders=False does not call orders_get_as_df."""
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
include_orders=False,
|
||||
)
|
||||
mock_client.orders_get_as_df.assert_not_called()
|
||||
|
||||
def test_update_observability_skips_terminal_when_disabled(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""include_terminal=False does not call terminal_info_as_df."""
|
||||
update_observability(
|
||||
client=mock_client,
|
||||
output=tmp_path / "obs.db",
|
||||
include_terminal=False,
|
||||
)
|
||||
mock_client.terminal_info_as_df.assert_not_called()
|
||||
|
||||
def test_update_observability_with_positions_rows(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Non-empty positions are written to position_snapshots."""
|
||||
mock_client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{
|
||||
"ticket": 1,
|
||||
"position_id": 1,
|
||||
"symbol": "EURUSD",
|
||||
"type": 0,
|
||||
"volume": 0.1,
|
||||
"price_open": 1.1,
|
||||
"price_current": 1.1,
|
||||
"profit": 0.0,
|
||||
"swap": 0.0,
|
||||
"comment": "",
|
||||
"magic": 0,
|
||||
}
|
||||
])
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=mock_client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
|
||||
0
|
||||
]
|
||||
assert count == 1
|
||||
|
||||
def test_update_observability_with_order_rows(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Non-empty orders are written to order_snapshots."""
|
||||
mock_client.orders_get_as_df.return_value = pd.DataFrame([
|
||||
{
|
||||
"ticket": 10,
|
||||
"symbol": "EURUSD",
|
||||
"type": 2,
|
||||
"volume_current": 0.1,
|
||||
"price_open": 1.2,
|
||||
"price_current": 1.1,
|
||||
"state": 1,
|
||||
"comment": "",
|
||||
"magic": 0,
|
||||
"time_setup": 1700000000,
|
||||
}
|
||||
])
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=mock_client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
|
||||
assert count == 1
|
||||
|
||||
def test_update_observability_symbol_filter_positions(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Symbol filter fetches all positions in one call and filters client-side."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
|
||||
# All positions; only EURUSD matches the filter
|
||||
client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 1, "symbol": "EURUSD", "volume": 0.1, "profit": 0.0},
|
||||
{"ticket": 2, "symbol": "USDJPY", "volume": 0.2, "profit": 0.0},
|
||||
])
|
||||
client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
|
||||
assert client.positions_get_as_df.call_count == 1
|
||||
with sqlite3.connect(output) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
|
||||
0
|
||||
]
|
||||
assert count == 1
|
||||
|
||||
def test_update_observability_symbol_filter_orders(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Symbol filter fetches all orders in one call and filters client-side."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
|
||||
client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
# All orders; only EURUSD matches the filter
|
||||
client.orders_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 10, "symbol": "EURUSD", "volume_current": 0.1},
|
||||
{"ticket": 11, "symbol": "USDJPY", "volume_current": 0.5},
|
||||
])
|
||||
client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
|
||||
assert client.orders_get_as_df.call_count == 1
|
||||
with sqlite3.connect(output) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
|
||||
assert count == 1
|
||||
|
||||
def test_update_observability_symbol_filter_no_symbol_col(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Symbol filter is skipped when positions df has no symbol column."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
|
||||
# No symbol column in positions — all rows pass through unfiltered
|
||||
client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 1, "volume": 0.1},
|
||||
])
|
||||
client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=client, output=output, symbols=["EURUSD"])
|
||||
with sqlite3.connect(output) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
|
||||
0
|
||||
]
|
||||
assert count == 1
|
||||
|
||||
def test_update_observability_account_none_login(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Account row with no login key returns None login for downstream helpers."""
|
||||
client = MagicMock()
|
||||
client.account_info_as_df.return_value = pd.DataFrame([{"balance": 10000.0}])
|
||||
client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
output = tmp_path / "obs.db"
|
||||
update_observability(client=client, output=output)
|
||||
with sqlite3.connect(output) as conn:
|
||||
row = conn.execute("SELECT login FROM account_snapshots").fetchone()
|
||||
assert row is not None
|
||||
assert row[0] is None
|
||||
|
||||
def test_update_observability_empty_account_logs_warning(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Empty account_info_as_df logs a warning and does not write account row."""
|
||||
mock_client.account_info_as_df.return_value = pd.DataFrame()
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
assert "account_info_as_df returned empty frame" in caplog.text
|
||||
with sqlite3.connect(tmp_path / "obs.db") as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM account_snapshots").fetchone()[0]
|
||||
assert count == 0
|
||||
|
||||
def test_update_observability_empty_terminal_logs_warning(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Empty terminal_info_as_df logs a warning and does not write terminal row."""
|
||||
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
assert "terminal_info_as_df returned empty frame" in caplog.text
|
||||
with sqlite3.connect(tmp_path / "obs.db") as conn:
|
||||
count = conn.execute("SELECT COUNT(*) FROM terminal_snapshots").fetchone()[
|
||||
0
|
||||
]
|
||||
assert count == 0
|
||||
|
||||
def test_update_observability_with_config_opens_and_closes_connection(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""update_observability_with_config manages the MT5 connection lifecycle."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.account_info_as_df.return_value = pd.DataFrame()
|
||||
mock_client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
|
||||
update_observability_with_config(output=tmp_path / "obs.db")
|
||||
mock_client.initialize_and_login_mt5.assert_called_once()
|
||||
mock_client.shutdown.assert_called_once()
|
||||
|
||||
def test_update_observability_with_config_passes_symbols(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""update_observability_with_config forwards symbols to update_observability."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.account_info_as_df.return_value = pd.DataFrame()
|
||||
mock_client.positions_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
|
||||
spy = mocker.patch("mt5cli.sdk.update_observability")
|
||||
update_observability_with_config(
|
||||
output=tmp_path / "obs.db",
|
||||
symbols=["EURUSD"],
|
||||
include_account=False,
|
||||
)
|
||||
spy.assert_called_once()
|
||||
call_kwargs = spy.call_args.kwargs
|
||||
assert call_kwargs["symbols"] == ["EURUSD"]
|
||||
assert call_kwargs["include_account"] is False
|
||||
|
||||
def test_update_observability_invokes_snapshot_telemetry(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""update_observability calls record_snapshot_update on the global metrics."""
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_snapshot_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
mock_metrics.record_snapshot_update.assert_called_once()
|
||||
|
||||
def test_update_observability_emits_account_metrics(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""_snapshot_account emits account gauges via get_metrics."""
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_snapshot_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
mock_metrics.record_account_state.assert_called_once()
|
||||
|
||||
def test_update_observability_emits_terminal_metrics(
|
||||
self,
|
||||
mock_client: MagicMock,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""_snapshot_terminal emits connected/trade gauges via get_metrics."""
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_snapshot_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
mock_metrics.record_terminal_state.assert_called_once_with(
|
||||
connected=1.0, trade_allowed=1.0, trade_expert=1.0
|
||||
)
|
||||
|
||||
def test_update_observability_aggregates_same_symbol_positions(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Same-symbol positions are summed before emitting gauges (hedging)."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.account_info_as_df.return_value = pd.DataFrame([
|
||||
{
|
||||
"login": 1,
|
||||
"server": "demo",
|
||||
"balance": 1000.0,
|
||||
"equity": 1000.0,
|
||||
"margin": 0.0,
|
||||
"margin_free": 1000.0,
|
||||
"margin_level": 0.0,
|
||||
}
|
||||
])
|
||||
mock_client.positions_get_as_df.return_value = pd.DataFrame([
|
||||
{"ticket": 1, "symbol": "EURUSD", "profit": 10.0, "volume": 0.1},
|
||||
{"ticket": 2, "symbol": "EURUSD", "profit": -5.0, "volume": 0.2},
|
||||
{"ticket": 3, "symbol": "GBPUSD", "profit": 3.0, "volume": 0.05},
|
||||
])
|
||||
mock_client.orders_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_snapshot_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_observability(client=mock_client, output=tmp_path / "obs.db")
|
||||
calls = mock_metrics.record_position_state.call_args_list
|
||||
# Two EURUSD positions should be collapsed to one call; GBPUSD is one call.
|
||||
assert len(calls) == 2
|
||||
by_symbol = {c.kwargs["symbol"]: c.kwargs for c in calls}
|
||||
assert abs(float(by_symbol["EURUSD"]["profit"]) - 5.0) < 1e-9
|
||||
assert abs(float(by_symbol["EURUSD"]["volume"]) - 0.3) < 1e-9
|
||||
assert abs(float(by_symbol["GBPUSD"]["profit"]) - 3.0) < 1e-9
|
||||
assert abs(float(by_symbol["GBPUSD"]["volume"]) - 0.05) < 1e-9
|
||||
|
||||
|
||||
class TestUpdateHistoryTelemetry:
|
||||
"""Tests for telemetry hooks in update_history."""
|
||||
|
||||
def test_update_history_invokes_history_telemetry(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""update_history wraps write_incremental_datasets with telemetry."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.copy_rates_range_as_df.return_value = pd.DataFrame()
|
||||
mock_client.history_orders_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.history_deals_get_as_df.return_value = pd.DataFrame()
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_history_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_history(
|
||||
client=mock_client,
|
||||
output=tmp_path / "hist.db",
|
||||
symbols=["EURUSD"],
|
||||
)
|
||||
mock_metrics.record_history_update.assert_called_once_with(dataset="history")
|
||||
|
||||
def test_update_history_emits_history_rows(
|
||||
self,
|
||||
mocker: MockerFixture,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""update_history calls add_history_rows with the SQLite change delta."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.copy_rates_range_as_df.return_value = pd.DataFrame()
|
||||
mock_client.history_orders_get_as_df.return_value = pd.DataFrame()
|
||||
mock_client.history_deals_get_as_df.return_value = pd.DataFrame()
|
||||
mock_metrics = MagicMock()
|
||||
mock_cm = MagicMock()
|
||||
mock_cm.__enter__ = MagicMock(return_value=None)
|
||||
mock_cm.__exit__ = MagicMock(return_value=False)
|
||||
mock_metrics.record_history_update.return_value = mock_cm
|
||||
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
|
||||
update_history(
|
||||
client=mock_client,
|
||||
output=tmp_path / "hist.db",
|
||||
symbols=["EURUSD"],
|
||||
)
|
||||
mock_metrics.add_history_rows.assert_called_once_with(0, dataset="history")
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
"""Tests for mt5cli.telemetry module."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
|
||||
|
||||
from mt5cli.telemetry import (
|
||||
_OTEL_AVAILABLE, # type: ignore[reportPrivateUsage]
|
||||
_Mt5Metrics, # type: ignore[reportPrivateUsage]
|
||||
_NoOp, # type: ignore[reportPrivateUsage]
|
||||
configure_metrics,
|
||||
enable_otel_metrics,
|
||||
get_metrics,
|
||||
)
|
||||
|
||||
|
||||
class TestNoOp:
|
||||
"""Tests for _NoOp no-op instrument."""
|
||||
|
||||
def test_add_is_noop(self) -> None:
|
||||
"""_NoOp.add accepts amount and optional attributes without error."""
|
||||
noop = _NoOp()
|
||||
noop.add(1.0)
|
||||
noop.add(1.0, {"key": "val"})
|
||||
|
||||
def test_set_is_noop(self) -> None:
|
||||
"""_NoOp.set accepts amount and optional attributes without error."""
|
||||
noop = _NoOp()
|
||||
noop.set(2.0)
|
||||
noop.set(2.0, {"key": "val"})
|
||||
|
||||
def test_record_is_noop(self) -> None:
|
||||
"""_NoOp.record accepts amount and optional attributes without error."""
|
||||
noop = _NoOp()
|
||||
noop.record(3.0)
|
||||
noop.record(3.0, {"key": "val"})
|
||||
|
||||
|
||||
class TestMt5Metrics:
|
||||
"""Tests for _Mt5Metrics."""
|
||||
|
||||
def test_default_instruments_are_noop(self) -> None:
|
||||
"""Default _Mt5Metrics methods do not raise before configure is called."""
|
||||
m = _Mt5Metrics()
|
||||
m.record_account_state(
|
||||
login="123",
|
||||
server="demo",
|
||||
balance=1000.0,
|
||||
equity=1050.0,
|
||||
margin=100.0,
|
||||
margin_free=950.0,
|
||||
margin_level=1050.0,
|
||||
)
|
||||
|
||||
def test_configure_calls_meter(self) -> None:
|
||||
"""configure() calls create_histogram, create_counter, create_gauge on meter."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
assert meter.create_histogram.called
|
||||
assert meter.create_counter.called
|
||||
assert meter.create_gauge.called
|
||||
|
||||
def test_record_history_update_success(self) -> None:
|
||||
"""record_history_update records duration and timestamp on success."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
with m.record_history_update(dataset="rates"):
|
||||
pass
|
||||
m._history_duration.record.assert_called_once() # type: ignore[reportPrivateUsage]
|
||||
m._last_successful_update.set.assert_called_once() # type: ignore[reportPrivateUsage]
|
||||
m._history_failures.add.assert_not_called() # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_history_update_failure(self) -> None:
|
||||
"""record_history_update increments failure counter and re-raises on error."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
exc = ValueError("boom")
|
||||
with (
|
||||
pytest.raises(ValueError, match="boom"),
|
||||
m.record_history_update(dataset="rates"),
|
||||
):
|
||||
raise exc
|
||||
m._history_failures.add.assert_called_once_with( # type: ignore[reportPrivateUsage]
|
||||
1, {"dataset": "rates"}
|
||||
)
|
||||
m._history_duration.record.assert_not_called() # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_add_history_rows(self) -> None:
|
||||
"""add_history_rows increments the rows-written counter."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
m.add_history_rows(42, dataset="rates")
|
||||
m._history_rows.add.assert_called_once_with( # type: ignore[reportPrivateUsage]
|
||||
42, {"dataset": "rates"}
|
||||
)
|
||||
|
||||
def test_record_snapshot_update_success(self) -> None:
|
||||
"""record_snapshot_update records duration on success."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
with m.record_snapshot_update():
|
||||
pass
|
||||
m._snapshot_duration.record.assert_called_once() # type: ignore[reportPrivateUsage]
|
||||
m._snapshot_failures.add.assert_not_called() # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_snapshot_update_failure(self) -> None:
|
||||
"""record_snapshot_update increments failure counter and re-raises on error."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
exc = RuntimeError("snap fail")
|
||||
with (
|
||||
pytest.raises(RuntimeError, match="snap fail"),
|
||||
m.record_snapshot_update(),
|
||||
):
|
||||
raise exc
|
||||
m._snapshot_failures.add.assert_called_once_with(1, {}) # type: ignore[reportPrivateUsage]
|
||||
m._snapshot_duration.record.assert_not_called() # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_position_state(self) -> None:
|
||||
"""record_position_state emits profit and volume gauges."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
m.record_position_state(
|
||||
login="42",
|
||||
server="demo",
|
||||
symbol="EURUSD",
|
||||
profit=12.5,
|
||||
volume=0.01,
|
||||
)
|
||||
# Both profit and volume share the same gauge mock via create_gauge.
|
||||
# Verify that set was called exactly twice (once each).
|
||||
assert m._position_profit.set.call_count == 2 # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_terminal_state(self) -> None:
|
||||
"""record_terminal_state emits connected, trade_allowed, trade_expert gauges."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
m.record_terminal_state(connected=1.0, trade_allowed=1.0, trade_expert=0.0)
|
||||
# All three terminal gauges share the same mock; set is called 3 times.
|
||||
assert m._terminal_connected.set.call_count == 3 # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_account_state_after_configure(self) -> None:
|
||||
"""record_account_state emits all five account gauges."""
|
||||
meter = MagicMock()
|
||||
m = _Mt5Metrics()
|
||||
m.configure(meter)
|
||||
m.record_account_state(
|
||||
login="99",
|
||||
server="live",
|
||||
balance=5000.0,
|
||||
equity=5100.0,
|
||||
margin=200.0,
|
||||
margin_free=4800.0,
|
||||
margin_level=2550.0,
|
||||
)
|
||||
# All five account gauges share the same gauge mock; set is called 5 times.
|
||||
assert m._account_balance.set.call_count == 5 # type: ignore[reportPrivateUsage]
|
||||
|
||||
def test_record_history_update_noop_before_configure(self) -> None:
|
||||
"""record_history_update works without configure (no-op instruments)."""
|
||||
m = _Mt5Metrics()
|
||||
with m.record_history_update(dataset="ticks"):
|
||||
pass
|
||||
|
||||
def test_record_snapshot_update_noop_before_configure(self) -> None:
|
||||
"""record_snapshot_update works without configure (no-op instruments)."""
|
||||
m = _Mt5Metrics()
|
||||
with m.record_snapshot_update():
|
||||
pass
|
||||
|
||||
|
||||
class TestConfigureMetrics:
|
||||
"""Tests for configure_metrics and get_metrics."""
|
||||
|
||||
def test_configure_metrics_updates_global(self) -> None:
|
||||
"""configure_metrics wires up the global singleton."""
|
||||
meter = MagicMock()
|
||||
configure_metrics(meter)
|
||||
assert get_metrics() is get_metrics()
|
||||
|
||||
def test_get_metrics_returns_mt5metrics(self) -> None:
|
||||
"""get_metrics returns the global _Mt5Metrics instance."""
|
||||
assert isinstance(get_metrics(), _Mt5Metrics)
|
||||
|
||||
|
||||
class TestEnableOtelMetrics:
|
||||
"""Tests for enable_otel_metrics."""
|
||||
|
||||
def test_enable_raises_when_unavailable(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""enable_otel_metrics raises ImportError when OTel is not installed."""
|
||||
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", False)
|
||||
with pytest.raises(ImportError, match="opentelemetry-api"):
|
||||
enable_otel_metrics()
|
||||
|
||||
def test_enable_configures_sdk_pipeline_with_readers(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""enable_otel_metrics wires up an SDK MeterProvider with supplied readers."""
|
||||
mock_mod = MagicMock()
|
||||
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
|
||||
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
|
||||
reader = InMemoryMetricReader()
|
||||
enable_otel_metrics("my-service", readers=[reader])
|
||||
mock_mod.set_meter_provider.assert_called_once()
|
||||
provider = mock_mod.set_meter_provider.call_args[0][0]
|
||||
assert provider.get_meter("my-service") is not None
|
||||
|
||||
def test_enable_default_readers_uses_otlp(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""enable_otel_metrics with no readers creates an OTLP pipeline by default."""
|
||||
mock_mod = MagicMock()
|
||||
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
|
||||
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
|
||||
monkeypatch.setattr("mt5cli.telemetry._OtelOTLPExporter", MagicMock())
|
||||
enable_otel_metrics("my-service")
|
||||
mock_mod.set_meter_provider.assert_called_once()
|
||||
|
||||
def test_enable_default_readers_raises_when_otlp_missing(
|
||||
self,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""enable_otel_metrics raises ImportError when the OTLP exporter is missing."""
|
||||
mock_mod = MagicMock()
|
||||
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
|
||||
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
|
||||
monkeypatch.setattr("mt5cli.telemetry._OtelOTLPExporter", None)
|
||||
with pytest.raises(ImportError, match="opentelemetry-exporter-otlp-proto-http"):
|
||||
enable_otel_metrics()
|
||||
|
||||
def test_otel_available_flag_is_bool(self) -> None:
|
||||
"""_OTEL_AVAILABLE is a boolean."""
|
||||
assert isinstance(_OTEL_AVAILABLE, bool)
|
||||
+1142
-604
File diff suppressed because it is too large
Load Diff
+20
-12
@@ -4,21 +4,22 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
|
||||
import mt5cli.utils
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from mt5cli.utils import (
|
||||
DATETIME_TYPE,
|
||||
REQUEST_TYPE,
|
||||
TICK_FLAG_MAP,
|
||||
TICK_FLAGS_TYPE,
|
||||
TIMEFRAME_MAP,
|
||||
TIMEFRAME_TYPE,
|
||||
Dataset,
|
||||
IfExists,
|
||||
@@ -111,6 +112,17 @@ class TestExportDataframe:
|
||||
result = pd.read_parquet(output)
|
||||
pd.testing.assert_frame_equal(result, sample_df)
|
||||
|
||||
def test_export_parquet_without_pyarrow(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
sample_df: pd.DataFrame,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Test that a clear error is raised when pyarrow is not installed."""
|
||||
monkeypatch.setitem(sys.modules, "pyarrow", None)
|
||||
with pytest.raises(ImportError, match="mt5cli\\[parquet\\]"):
|
||||
export_dataframe(sample_df, tmp_path / "out.parquet", "parquet")
|
||||
|
||||
def test_export_sqlite3(self, tmp_path: Path, sample_df: pd.DataFrame) -> None:
|
||||
"""Test SQLite3 export."""
|
||||
output = tmp_path / "out.db"
|
||||
@@ -361,17 +373,13 @@ class TestParseRequest:
|
||||
class TestConstants:
|
||||
"""Tests for module constants."""
|
||||
|
||||
def test_timeframe_map_has_expected_keys(self) -> None:
|
||||
"""Test that TIMEFRAME_MAP contains standard timeframes."""
|
||||
for key in ("M1", "M5", "M15", "M30", "H1", "H4", "D1", "W1", "MN1"):
|
||||
assert key in TIMEFRAME_MAP
|
||||
def test_timeframe_map_is_private_in_utils(self) -> None:
|
||||
"""TIMEFRAME_MAP is a private implementation detail; not a public attribute."""
|
||||
assert not hasattr(mt5cli.utils, "TIMEFRAME_MAP")
|
||||
|
||||
def test_tick_flag_map_has_expected_keys(self) -> None:
|
||||
"""Test that TICK_FLAG_MAP contains standard flags with MT5 values."""
|
||||
assert {"ALL", "INFO", "TRADE"} <= set(TICK_FLAG_MAP)
|
||||
assert TICK_FLAG_MAP["ALL"] == -1
|
||||
assert TICK_FLAG_MAP["INFO"] == 1
|
||||
assert TICK_FLAG_MAP["TRADE"] == 2
|
||||
def test_tick_flag_map_absent_from_utils(self) -> None:
|
||||
"""TICK_FLAG_MAP is not exposed by mt5cli.utils."""
|
||||
assert not hasattr(mt5cli.utils, "TICK_FLAG_MAP")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("dataset", "expected"),
|
||||
|
||||
@@ -223,6 +223,18 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "googleapis-common-protos"
|
||||
version = "1.75.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b5/c8/f439cffde755cffa462bfbb156278fa6f9d09119719af9814b858fd4f81f/googleapis_common_protos-1.75.0.tar.gz", hash = "sha256:53a062ff3c32552fbd62c11fe23768b78e4ddf0494d5e5fd97d3f4689c75fbbd", size = 151035, upload-time = "2026-05-07T08:04:49.423Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/c8/e2645aa8ed02fd4c7a2f59d68783b65b1f3cbdfe39a6308e156509d1fee8/googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed", size = 300631, upload-time = "2026-05-07T08:03:30.345Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "griffelib"
|
||||
version = "2.0.2"
|
||||
@@ -487,21 +499,33 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "mt5cli"
|
||||
version = "0.9.2"
|
||||
version = "1.1.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
{ name = "pdmt5" },
|
||||
{ name = "pyarrow" },
|
||||
{ name = "typer" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
otel = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
]
|
||||
parquet = [
|
||||
{ name = "pyarrow" },
|
||||
]
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
{ name = "mkdocs" },
|
||||
{ name = "mkdocs-material" },
|
||||
{ name = "mkdocstrings", extra = ["python"] },
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "pandas-stubs" },
|
||||
{ name = "pyarrow" },
|
||||
{ name = "pymdown-extensions" },
|
||||
{ name = "pyright" },
|
||||
{ name = "pytest" },
|
||||
@@ -513,17 +537,24 @@ dev = [
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "click", specifier = ">=8.1.0" },
|
||||
{ name = "pdmt5", specifier = ">=0.3.0" },
|
||||
{ name = "pyarrow", specifier = ">=19.0.0" },
|
||||
{ name = "opentelemetry-api", marker = "extra == 'otel'" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http", marker = "extra == 'otel'" },
|
||||
{ name = "opentelemetry-sdk", marker = "extra == 'otel'" },
|
||||
{ name = "pdmt5", specifier = ">=1.0.0" },
|
||||
{ name = "pyarrow", marker = "extra == 'parquet'", specifier = ">=19.0.0" },
|
||||
{ name = "typer", specifier = ">=0.15.0" },
|
||||
]
|
||||
provides-extras = ["parquet", "otel"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [
|
||||
{ name = "mkdocs", specifier = ">=1.6.1" },
|
||||
{ name = "mkdocs-material", specifier = ">=9.7.6" },
|
||||
{ name = "mkdocstrings", extras = ["python"], specifier = ">=1.0.4" },
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "pandas-stubs", specifier = ">=2.2.3.250527" },
|
||||
{ name = "pyarrow", specifier = ">=19.0.0" },
|
||||
{ name = "pymdown-extensions", specifier = ">=10.21.2" },
|
||||
{ name = "pyright", specifier = ">=1.1.407" },
|
||||
{ name = "pytest", specifier = ">=9.0.3" },
|
||||
@@ -599,6 +630,87 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/57/a7/b35835e278c18b85206834b3aa3abe68e77a98769c59233d1f6300284781/numpy-2.4.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:4b42639cdde6d24e732ff823a3fa5b701d8acad89c4142bc1d0bd6dc85200ba5", size = 12504685, upload-time = "2026-03-09T07:58:50.525Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-api"
|
||||
version = "1.43.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ae/cc/e4c9584181f86494df0f6bdec1a4f3280c50db44704dc2a407e994fc87bb/opentelemetry_api-1.43.0.tar.gz", hash = "sha256:107d0d03857ea8fc7c5fcbbbd83f800c281f0d560553d61c1d675fccfd1761c1", size = 73476, upload-time = "2026-06-24T15:19:55.323Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/17/83/6dba32b85f31868400440dc7ad2ca1eab94cbbf3a7b0459ed39f8311a9e2/opentelemetry_api-1.43.0-py3-none-any.whl", hash = "sha256:20acf45e9b21851926835292e4045d290acade1edd2ff3de86d2f069687ba1fd", size = 61912, upload-time = "2026-06-24T15:19:35.434Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-exporter-otlp-proto-common"
|
||||
version = "1.43.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-proto" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/55/c1/e8098490ab15abf116dcaf9fa89ededcb35547c7d08d4b5a62f573dc1e63/opentelemetry_exporter_otlp_proto_common-1.43.0.tar.gz", hash = "sha256:c4e32ba6d6b13bdb2b8f6764c4fd28d00192826561aa04f6d14eedfce7ac076f", size = 20197, upload-time = "2026-06-24T15:20:00.247Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/b2/41ebc74ae1d5859901f1b69305de58724bf043381103d6ef413521cbc35a/opentelemetry_exporter_otlp_proto_common-1.43.0-py3-none-any.whl", hash = "sha256:123c3f9cc87218562490c63b36f497bf3a722faf174a515d1443f31ababa6264", size = 17048, upload-time = "2026-06-24T15:19:41.264Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-exporter-otlp-proto-http"
|
||||
version = "1.43.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "googleapis-common-protos" },
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-common" },
|
||||
{ name = "opentelemetry-proto" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "requests" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/fc/92/0b9f56412483a8891d4843890294796c9df8ab42417bd9bad8035d840cb3/opentelemetry_exporter_otlp_proto_http-1.43.0.tar.gz", hash = "sha256:fa8a42bb7d00ee5391f4c0b04d8e6a46c03caa437903296ab73a81dc11ba118f", size = 25406, upload-time = "2026-06-24T15:20:01.515Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/20/b685ed7af2e17c29ffc8af56f1fa8bc2033258fc30fb0d2b722f49d13ba0/opentelemetry_exporter_otlp_proto_http-1.43.0-py3-none-any.whl", hash = "sha256:647f603aa8efdbdb4dbff842e0729d0406a6fff26b295a72d3d60e7d963b2610", size = 21795, upload-time = "2026-06-24T15:19:43.164Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-proto"
|
||||
version = "1.43.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e0/b9/d357faefb40bda1d4799913e6af611171ff22a2dedcb93576bc92242d056/opentelemetry_proto-1.43.0.tar.gz", hash = "sha256:224778df17e1f3fafeaaa21d874236ca5f6ffc2f86e0899298ec7351aac27924", size = 46481, upload-time = "2026-06-24T15:20:07.625Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/a7/3e5308cf548b8f72529c7db1afdb3a404211982376a12927fd7759f77bf3/opentelemetry_proto-1.43.0-py3-none-any.whl", hash = "sha256:c58f1f7ef84bc7dc2834016c0c37fe0081dde7ca9f6339be1970fbf9cdaaa90d", size = 72489, upload-time = "2026-06-24T15:19:51.164Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-sdk"
|
||||
version = "1.43.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-semantic-conventions" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3e/eb/5041074274ac0956b03637cc039d434569112468e875eddfcc9a0674ce06/opentelemetry_sdk-1.43.0.tar.gz", hash = "sha256:d8187c81c162df9913e4003dd6485f7390d9a24fc17026ec7387b8b8218b08e9", size = 254744, upload-time = "2026-06-24T15:20:08.467Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/49/e3/b17be23af124201c9f52eececd4cc8ddfed1597d37b4ee771895d325805c/opentelemetry_sdk-1.43.0-py3-none-any.whl", hash = "sha256:d1323a547c1ce69d6a069a17a44b7da82bb8b332051ecb074041f87642c86823", size = 178852, upload-time = "2026-06-24T15:19:52.169Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-semantic-conventions"
|
||||
version = "0.64b0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5a/30/5f26df29509eccd86b99b481ac9ffa39da49ba9577cc69071c552ae30447/opentelemetry_semantic_conventions-0.64b0.tar.gz", hash = "sha256:72f76fb2d1582d9d033dd1fcd84532e961e6ff3d90d24ba6fabc72975a83864c", size = 148340, upload-time = "2026-06-24T15:20:09.267Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f2/ca/23ba87a221b574a7c5a99d48849d80bfe8b047624681357e2b002e566187/opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl", hash = "sha256:ea77e85e354b8f604ddbe5f3d9135216f982fa4d77e5859ac30f6d8a50505aa6", size = 203713, upload-time = "2026-06-24T15:19:53.339Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "packaging"
|
||||
version = "26.0"
|
||||
@@ -684,16 +796,16 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pdmt5"
|
||||
version = "0.3.0"
|
||||
version = "1.0.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "metatrader5", marker = "sys_platform == 'win32'" },
|
||||
{ name = "pandas" },
|
||||
{ name = "pydantic" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/bf/cc/c8fa3a01e0e34178fec8527992f7bb8eda5881477ce23aaacaa9b2ef7bec/pdmt5-0.3.0.tar.gz", hash = "sha256:bb612d5c2695eafac9b2a7b74756e13bd383d7e5517bd90c9a2efa92492c484c", size = 215100, upload-time = "2026-06-11T13:26:46.976Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/21/6d/b51d2d0ec4636e914210be03a7da20e5078bc8cd7a351edd13bc30b7d2b1/pdmt5-1.0.0.tar.gz", hash = "sha256:ba53a1a5db41fdf4c022ef55353f5a4f6884f625c9310de2b0ddb20457956716", size = 123155, upload-time = "2026-06-25T23:41:50.503Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f2/03/b12cc4c9db983d971c9172b3765161b6d91136d0624e6718a04dd815e7a1/pdmt5-0.3.0-py3-none-any.whl", hash = "sha256:5388b406cc583202600cfe22c9d781679b1d931b1ed5a2b5dcf37c566149b49f", size = 26250, upload-time = "2026-06-11T13:26:45.689Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5e/38/27b712c572d8146efddf571a0e4e7b6d2314a335eb0f47dec1f499ab2189/pdmt5-1.0.0-py3-none-any.whl", hash = "sha256:f969c17902f9ffcbf56d7ccf9285aab39ececd676fcca50c094abcf9d728d517", size = 23992, upload-time = "2026-06-25T23:41:49.067Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -714,6 +826,21 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "protobuf"
|
||||
version = "7.35.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/da/01/9ef0afd7999eb9badb3a768b4aedd78c86d4c65cfaf1958ab276199e76b4/protobuf-7.35.1.tar.gz", hash = "sha256:ce115a26fe0c39a2c29973d914d327e516a6455464489fe3cd1e51a1b354f81a", size = 458717, upload-time = "2026-06-11T21:55:40.257Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/10/03/8aeeb7458d22546bf64b5250ca1daeb5ff757d900e8e4a7476c6f0db843e/protobuf-7.35.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:24f857477359a85c0c235261b8ba905fd51b2562f4a64ca1df5473f29850cbf6", size = 433226, upload-time = "2026-06-11T21:55:31.719Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/37/4b/dfb89eb0e652a1ff073c39a59fb5e3a83cfe9b57a2c83fa6d78270101767/protobuf-7.35.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:11d6b0ec246892d85215b0a13ca6e0233cf5284b68f0ac02646427f4ff88a799", size = 328847, upload-time = "2026-06-11T21:55:34.035Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0f/58/dc12f2cd484951524af6e3382c785869b9b3fb5e52ee95ae23add53ee8f9/protobuf-7.35.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:b73f9489a4b8b1c9cb1f8ed951c736392592edb24b9d6819f36d2e10b171d5b4", size = 344030, upload-time = "2026-06-11T21:55:34.941Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e4/be/5b3cfe508bfab6761414ff944e3366eb13be4fd71efcd69450f89ba39f43/protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:74758715c53d7158fb76caf4f0cfdacc5329a4b1bb994f865d6cf302d413a1c4", size = 327130, upload-time = "2026-06-11T21:55:35.921Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/bc/6d6c7ba8709c85f8f2c390b2b118d6fb08a783676a572271851bf45a7d22/protobuf-7.35.1-cp310-abi3-win32.whl", hash = "sha256:353652e4efd0bca5b5fc2656abf8307ef351f0cf938c9eba09f0e09c20a25c30", size = 428945, upload-time = "2026-06-11T21:55:37.034Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0a/19/8d0cb6f20a1ef7b18f1c8986ad5783f22f84cce39c6ce9a6e645ea55192e/protobuf-7.35.1-cp310-abi3-win_amd64.whl", hash = "sha256:230a75ddfc2de4806e56696ce9640c1cdfdb6543b7cfce98d42a4c0a0e7bdb87", size = 439996, upload-time = "2026-06-11T21:55:38.123Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyarrow"
|
||||
version = "23.0.1"
|
||||
|
||||
Reference in New Issue
Block a user