Compare commits

...

2 Commits

Author SHA1 Message Date
Daichi Narushima 1ffac45d57 feat: add publish_grafana_copy, Grafana examples, and optional OTel metrics (#89)
* feat: Grafana copy publishing, dashboard examples, and optional OTel metrics

Implements three observability improvements:

#82 — publish_grafana_copy(): Uses SQLite online backup API (WAL-safe) to
atomically publish a consistent read-only copy beside the target. Adds
--publish-copy option to grafana-schema and snapshot CLI commands.

#83 — examples/grafana/: Minimal working Grafana setup with docker-compose,
provisioning datasource/dashboard YAML, and three dashboard JSON files
(mt5cli-overview, mt5cli-trades, mt5cli-market). All queries use grafana_*
views; no credentials or private paths included.

#84 — mt5cli/telemetry.py: Optional OTel metrics behind mt5cli[otel] extra.
Base install is unaffected. Adds _Mt5Metrics singleton (no-op until
configure_metrics() is called), wraps update_history() and
update_observability() with record_history_update / record_snapshot_update
context managers, and emits account/position gauges from snapshots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: replace ambiguous multiplication sign in comment

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: normalize markdown formatting in grafana README

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: preserve file mode on Grafana copy and fix unsupported time macro

- publish_grafana_copy: chmod temp file to match the existing target's
  permissions (or 0o644 when no prior target exists) before atomic
  replace, so Grafana running as a different OS user (e.g. UID 472 in
  Docker) can read the published database
- mt5cli-market.json: replace unsupported \$__timeFilter(time) with the
  epoch-based filter supported by frser-sqlite-datasource:
  "time" >= \$__from / 1000 AND "time" < \$__to / 1000

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: skip Windows-incompatible mode test, rename compose file to compose.yaml

- Skip test_overwrite_preserves_existing_target_mode on win32 since
  Windows chmod does not preserve Unix group/other permission bits
- Simplify test_fresh_target_has_readable_permissions to check owner
  read bit only (portable across platforms)
- Rename docker-compose.yml -> compose.yaml (modern Compose convention)
- Update README and test reference to match new filename

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: rename *.yaml to *.yml in examples/grafana

Renames compose.yaml, mt5cli-sqlite.yaml, and mt5cli.yaml to .yml;
updates README and test references accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: format Grafana dashboards and expand qa script to include JSON

- Update qa.sh prettier pattern to format JSON files alongside markdown
- Reformat Grafana dashboard JSONs with consistent spacing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address owner review comments before merge

- qa.sh: fix Prettier glob from `{,d,json}` to `{md,json}` so Markdown
  files are actually formatted by local QA (P2)
- compose.yml: add GF_INSTALL_PLUGINS env var so the frser-sqlite-datasource
  plugin is installed at container start (P1)
- telemetry.py: replace no-op get_meter() call with a real SDK MeterProvider
  pipeline; add optional `readers` kwarg so callers can inject custom readers
  (e.g. InMemoryMetricReader in tests) without needing the OTLP package (P1)
- sdk.py: aggregate profit and volume by symbol before emitting gauge values
  so hedging accounts with multiple same-symbol positions emit one point per
  symbol instead of overwriting with each row (P2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: emit mt5_history_update_rows_total via conn.total_changes delta

The counter was registered but never incremented, making the advertised
history-update throughput metric permanently zero. Add add_history_rows()
to _Mt5Metrics and call it in update_history() using the SQLite
total_changes delta measured around write_incremental_datasets().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address three owner review comments

- compose.yml: replace soft fallback with :? error expansion so Compose
  refuses to start when MT5CLI_DB_PATH is unset or empty (P1)
- README.md: tell native Windows users to copy only the datasource
  provisioning file; the dashboards yml contains a Docker-specific path
  that is invalid on Windows (P2)
- telemetry.py / sdk.py: emit mt5_terminal_connected,
  mt5_terminal_trade_allowed, and mt5_terminal_trade_expert gauges via a
  new record_terminal_state() method called from _snapshot_terminal(),
  completing the connection-status metric surface from issue #84 (P2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: add snapshot freshness panel and win-rate column to dashboards

- mt5cli-overview.json: add a full-width "Last Snapshot" stat panel
  (dateTimeFromNow unit) below the account stats, querying
  MAX(time)*1000 from grafana_account_snapshots so users can tell
  whether Grafana is reading a current published copy (#83)
- mt5cli-trades.json: add win_rate_pct computed column to the Trade
  Statistics by Symbol table via 100.0 * winning_deals / NULLIF(
  total_deals, 0), with a percent unit override and "Win Rate (%)"
  display label (#83)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: reject same source and target path in publish_grafana_copy

Adds an early same-path guard to publish_grafana_copy: resolves both
paths before any I/O and raises ValueError if they are identical,
preventing the function from overwriting the live source database with
its own backup copy. Also adds a unit test for the rejected case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address ruff EM102/TRY003/E501 in same-path guard

Assigns the ValueError message to a variable before raising and
shortens the test docstring to stay within the 88-char line limit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: apply ruff format to publish_grafana_copy error message

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: remove grafana_ticks panel from default market dashboard

The Tick Bid/Ask panel queried grafana_ticks which only exists when users
collect tick data (opt-in). Users following the default OHLCV-only setup
path hit "no such table: grafana_ticks" on dashboard load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: close SQLite connections before atomic replace in publish_grafana_copy

Wrap both src and dst connections with contextlib.closing() so they are
explicitly closed before tmp_path.replace(target_path) runs. Without
this, sqlite3.Connection's context manager only commits/rolls back but
leaves the file handle open, which can cause PermissionError on Windows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: rename history.grafana.db to history.mt5cli.db in Grafana examples

frser-sqlite-datasource blocks paths containing "grafana.db" via its
internal blocklist. Rename the recommended published filename in the
README, compose comment, and datasource provisioning comment to avoid
a blocked/denied datasource for native Windows users.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update Docker Compose quick-start to pass MT5CLI_DB_PATH

The compose.yml already required MT5CLI_DB_PATH via ${MT5CLI_DB_PATH:?...},
but the README still showed bare `docker compose up -d`. Update the section
to show the env-var-prefixed invocation and document the .env file alternative.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 17:14:17 +09:00
Daichi Narushima d27da02f3f feat: add Grafana-ready SQLite observability (#86)
* feat: add Grafana-ready SQLite observability (#79, #80, #81)

New `mt5cli/grafana.py` module with idempotent DDL helpers:
- `create_snapshot_tables` — five SQLite tables for time-series account,
  position, order, terminal, and run-status snapshots
- `create_grafana_views` — 13 `grafana_*` views with integer epoch-second
  `time` columns; missing source tables emit warnings and are skipped
- `create_grafana_indexes` — 9 performance indexes guarded by column checks
- `ensure_grafana_schema` — convenience wrapper calling all three above
- Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`,
  `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run`

New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`:
- `update_observability` — appends a timestamped snapshot to a SQLite db
  using an already-connected `Mt5DataClient`; never places orders
- `update_observability_with_config` — standalone wrapper that opens and
  closes the MT5 connection automatically

New CLI commands (Collection panel):
- `grafana-schema` — idempotent schema setup, no MT5 connection required
- `snapshot` — append account/position/order/terminal rows; supports
  `--symbol`, `--with-account/--no-account`, and equivalent flags

All public modules maintain 100 % branch coverage; 968 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply mdformat to docs after grafana observability additions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: bump version to 1.1.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address PR #86 review feedback

- Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX)
- Replace O(N) per-symbol API calls with single call + client-side filter
- Eliminate double create_snapshot_tables when with_grafana_schema=True
- Move grafana imports to module level in sdk.py (remove PLC0415 noqa)
- Default with_grafana_schema to False (run grafana-schema once for setup)
- Fix README position example to use snapshot_runs for latest snapshot
- Update tests to reflect new behavior and correct patch targets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply ruff formatting and sync lock file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address owner review feedback on PR #86

- Filter grafana_*_snapshots views to only expose rows from successful
  runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot
  visibility gap raised in PRRT_kwDORzI_286MvDJ6
- Add issubset column guards for snapshot table index creation, consistent
  with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1)
- Fix README example queries: views expose 'time' not 'observed_at'
  (PRRT_kwDORzI_286MvUxw)
- Correct public-contract.md default for with_grafana_schema (False, not
  True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: add run_id to snapshot schema and fix stale-positions README query

Replace second-level observed_at as the join key between snapshot_runs
and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs
in the same second now get distinct run_ids, preventing view duplication
and cross-contamination from a failed run. Update README example to use
snapshot_runs for latest-snapshot lookup so zero-position runs return an
empty result instead of stale rows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: format markdown tables

Align table column widths in README and public-contract documentation.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: close sqlite connections deterministically

* fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats

grafana_realized_pnl now requires the entry column and filters to
close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl
and grafana_trade_stats. grafana_trade_stats now requires the time column
and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana
view contract (integer epoch-second time column throughout).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots

orders_get_as_df() returns datetime-converted columns by default, so
time_setup is a pd.Timestamp in normal use. Passing it directly to
sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper
that converts datetime.datetime subclasses (including pd.Timestamp) and
raw int/float values to integer epoch seconds, returning None for other
types. Regression tests cover all four input paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: pre-drop all grafana_* views at start of create_grafana_views

Previously, a builder that skipped due to a missing source table or column
would not drop the view it owned, leaving stale views referencing gone
tables. Now create_grafana_views drops all 13 known grafana_* views before
calling any builder, so a schema refresh always removes views whose source
has disappeared. Regression test covers the create → drop-source → refresh
cycle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: expose run_id in snapshot views and drop time from summary views

Grafana snapshot views now expose run_id so latest-state queries can use
MAX(run_id) instead of the ambiguous second-level MAX(observed_at).
grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column
and are reclassified as static summary views; their all-time aggregates
are not filterable by Grafana time-range selectors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: guard snapshot views against missing run_id and fix README view docs

_build_snapshot_view now skips with a warning when the underlying
snapshot table exists but lacks a run_id column, preventing a broken
view that fails at query time. Adds a regression test for that path.

README Grafana section now qualifies that grafana_realized_pnl and
grafana_trade_stats are static summary views (no time column) and splits
the view table to match docs/api/public-contract.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 07:55:26 +09:00
25 changed files with 4507 additions and 31 deletions
+1 -1
View File
@@ -10,7 +10,7 @@ uv run pyright .
uv run pytest
# Markdown
npx -y prettier --write './**/*.md'
npx -y prettier --write './**/*.{md,json}'
# GitHub Actions
case "${OSTYPE}" in
+100
View File
@@ -185,6 +185,8 @@ python -m mt5cli -o account.csv account-info
| `order-send` | Send a raw trade request to the trade server (`--yes` required; expert path) |
| `close-positions` | Close open positions by `--symbol` or `--ticket` (`--yes` required for live; `--dry-run` available) |
| `collect-history` | Collect rates, history-orders, and history-deals for one or more symbols into a single SQLite database (ticks opt-in via `--dataset ticks`) |
| `grafana-schema` | Create or refresh Grafana-ready views and indexes in an existing SQLite database (idempotent, no MT5 connection) |
| `snapshot` | Snapshot current account, position, order, and terminal state into SQLite for live Grafana dashboards |
Use `order-check` to validate a request payload before running `order-send --yes`.
`close-positions` is the safer high-level alternative that builds correct close
@@ -204,6 +206,104 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
### Grafana-ready SQLite dashboards
mt5cli can prepare a SQLite database for use as a Grafana datasource (via the [SQLite plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/) or similar). Most `grafana_*` views expose an integer epoch-second `time` column for use in Grafana time-series panels. Two views (`grafana_realized_pnl`, `grafana_trade_stats`) are static symbol-level summaries with no `time` column — use them in table or stat panels.
#### Prepare the schema (idempotent, no MT5 connection needed)
```bash
mt5cli -o history.db grafana-schema
```
This creates snapshot tables (`account_snapshots`, `position_snapshots`, `order_snapshots`, `terminal_snapshots`, `snapshot_runs`) and all `grafana_*` views and indexes in the SQLite database. Safe to run repeatedly — all operations are idempotent.
#### Snapshot current account state
```bash
mt5cli -o history.db snapshot \
--symbol JP225 --symbol HK50 --symbol NL25 \
--with-account --with-positions --with-orders --with-terminal \
--with-grafana-schema
```
Appends one timestamped row per data type. Never places orders or modifies trading state. Run periodically (e.g. from a cron job or a loop) to build a time-series account history.
#### SDK usage
```python
from pdmt5 import Mt5DataClient, Mt5Config
from mt5cli import update_observability, update_observability_with_config
# Reuse an already-connected client
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
update_observability(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"], # optional position/order filter
include_account=True,
include_positions=True,
include_orders=True,
include_terminal=True,
with_grafana_schema=True,
)
finally:
client.shutdown()
# Standalone wrapper that opens/closes MT5 automatically
update_observability_with_config(
output="history.db",
config=Mt5Config(login=12345),
)
```
#### Available Grafana views
**Time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
| View | Source | Description |
| ---------------------------- | -------------------- | ---------------------------------------------------------- |
| `grafana_rates` | `rates` | OHLCV bars with integer epoch `time` |
| `grafana_ticks` | `ticks` | Tick data with integer epoch `time` |
| `grafana_history_deals` | `history_deals` | All deals with epoch `time` |
| `grafana_history_orders` | `history_orders` | All historical orders; adds epoch `time` from `time_setup` |
| `grafana_trade_deals` | `history_deals` | Trade deals only (`type IN (0,1)`) |
| `grafana_cash_events` | `history_deals` | Non-trade deals (deposits, dividends, etc.) |
| `grafana_symbol_pnl` | `history_deals` | Per-close-deal profit/loss per symbol |
| `grafana_account_snapshots` | `account_snapshots` | Account balance/equity/margin time series |
| `grafana_position_snapshots` | `position_snapshots` | Open position snapshots over time |
| `grafana_order_snapshots` | `order_snapshots` | Active order snapshots over time |
| `grafana_terminal_snapshots` | `terminal_snapshots` | Terminal connectivity snapshots |
**Static summary views** (no `time` column; use in table or stat panels, not time-series):
| View | Source | Description |
| ---------------------- | --------------- | ------------------------------------- |
| `grafana_realized_pnl` | `history_deals` | Cumulative realized PnL per symbol |
| `grafana_trade_stats` | `history_deals` | Win/loss counts and profit per symbol |
#### Example Grafana queries
```sql
-- Equity curve over time
SELECT time, equity FROM grafana_account_snapshots ORDER BY time;
-- Rolling balance by account login
SELECT time, login, balance FROM grafana_account_snapshots
WHERE login = $login ORDER BY time;
-- Open positions at latest successful snapshot
SELECT symbol, volume, profit FROM grafana_position_snapshots
WHERE run_id = (SELECT MAX(run_id) FROM snapshot_runs WHERE status = 'ok');
-- Realized PnL by symbol
SELECT symbol, total_profit FROM grafana_trade_stats ORDER BY total_profit DESC;
```
> **Note**: OpenTelemetry integration is intentionally not part of this release and is tracked separately.
### Incremental history SDK
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
+75 -8
View File
@@ -124,6 +124,63 @@ sending requests. Failed, malformed, or unknown broker retcodes are fail-closed
and returned as `status="failed"` with normalized `request` / `response` details;
`dry_run=True` never calls `ensure_symbol_selected()` or `order_send()`.
### Grafana observability (SQLite read model)
These helpers prepare a SQLite database as a Grafana datasource. All DDL is
idempotent (`CREATE TABLE IF NOT EXISTS`, `DROP VIEW IF EXISTS` + `CREATE
VIEW`, `CREATE INDEX IF NOT EXISTS`). Missing source tables are skipped with a
warning rather than raising an error.
| Symbol | Role |
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
| `update_observability` | Append one timestamped snapshot row per data type; accepts an already-connected `Mt5DataClient` |
| `update_observability_with_config` | Standalone wrapper: opens/closes MT5 connection automatically around `update_observability` |
Both functions write to the SQLite path given by `output=`. The optional
`symbols` parameter filters `positions_get` / `orders_get` by symbol.
`with_grafana_schema=False` (default) skips Grafana view/index setup; run
`grafana-schema` once to set up the schema, then call `snapshot` repeatedly
without this flag.
**Snapshot tables** (created by `create_snapshot_tables` in `mt5cli.grafana`):
| Table | Content |
| -------------------- | ----------------------------------------- |
| `account_snapshots` | Balance, equity, margin, free-margin, P&L |
| `position_snapshots` | Open positions: symbol, volume, profit, … |
| `order_snapshots` | Active orders: symbol, type, price, … |
| `terminal_snapshots` | Terminal connectivity and build info |
| `snapshot_runs` | Per-run status (`ok` / `error`) timestamp |
**Grafana time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
| View | Source |
| ---------------------------- | -------------------------------- |
| `grafana_rates` | `rates` table |
| `grafana_ticks` | `ticks` table |
| `grafana_history_deals` | `history_deals` |
| `grafana_history_orders` | `history_orders` |
| `grafana_trade_deals` | `history_deals` trade types only |
| `grafana_cash_events` | `history_deals` non-trade events |
| `grafana_symbol_pnl` | Per-close-deal P&L per symbol |
| `grafana_account_snapshots` | `account_snapshots` |
| `grafana_position_snapshots` | `position_snapshots` |
| `grafana_order_snapshots` | `order_snapshots` |
| `grafana_terminal_snapshots` | `terminal_snapshots` |
**Grafana static summary views** (no `time` column; use for table/stat panels, not time-series):
| View | Source |
| ---------------------- | ------------------------------------- |
| `grafana_realized_pnl` | Cumulative realized PnL per symbol |
| `grafana_trade_stats` | Win/loss counts and profit per symbol |
Lower-level helpers (`ensure_grafana_schema`, `create_grafana_views`,
`create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`,
`insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`,
`insert_terminal_snapshot`, `record_snapshot_run`) are available directly from
`mt5cli.grafana` and are not part of the package-root stable surface.
### Errors
| Symbol | Role |
@@ -135,14 +192,15 @@ and returned as `status="failed"` with normalized `request` / `response` details
Lower-level helpers are available from their owning modules and are not part
of the package-root stable surface. Import them directly when needed:
| Module | Examples |
| ------------------- | ---------------------------------------------------------------------------------------------- |
| `mt5cli.history` | `resolve_rate_view_name`, `resolve_rate_tables`, `load_rate_data`, `build_rate_view_name` |
| `mt5cli.sdk` | `copy_rates_from`, `copy_ticks_from`, `account_info`, `symbols`, `mt5_summary`, `latest_rates` |
| `mt5cli.schemas` | `DataKind`, `normalize_dataframe`, `validate_schema`, `DEDUP_KEYS` |
| `mt5cli.utils` | `Dataset`, `IfExists`, `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
| `mt5cli.converters` | `normalize_symbol`, `ensure_utc`, `parse_date_range`, `granularity_name` |
| `mt5cli.exceptions` | `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` |
| Module | Examples |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mt5cli.grafana` | `ensure_grafana_schema`, `create_grafana_views`, `create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`, `insert_account_snapshot`, `record_snapshot_run` |
| `mt5cli.history` | `resolve_rate_view_name`, `resolve_rate_tables`, `load_rate_data`, `build_rate_view_name` |
| `mt5cli.sdk` | `copy_rates_from`, `copy_ticks_from`, `account_info`, `symbols`, `mt5_summary`, `latest_rates` |
| `mt5cli.schemas` | `DataKind`, `normalize_dataframe`, `validate_schema`, `DEDUP_KEYS` |
| `mt5cli.utils` | `Dataset`, `IfExists`, `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
| `mt5cli.converters` | `normalize_symbol`, `ensure_utc`, `parse_date_range`, `granularity_name` |
| `mt5cli.exceptions` | `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` |
## CLI commands
@@ -155,6 +213,15 @@ The Typer application in `mt5cli.cli` exposes file-export commands documented in
- Delegate to the same Python APIs described here; they are not duplicated
business logic.
`grafana-schema` initializes Grafana views, indexes, and snapshot tables in the
target SQLite database without connecting to MT5. It is idempotent and safe to
run repeatedly.
`snapshot` appends one timestamped row per enabled data type
(`--with-account`, `--with-positions`, `--with-orders`, `--with-terminal`) and
never places orders or modifies trading state. Both commands require
`-o/--output` to point at a `.db` / SQLite file.
`order-send` is the expert raw-request path; it requires `--yes` and a fully
constructed request payload. `close-positions` is the safer high-level helper
that closes open positions by `--symbol` or `--ticket` using
+95
View File
@@ -0,0 +1,95 @@
# Grafana Integration for mt5cli
This directory contains example configuration and dashboard files for visualising
mt5cli SQLite data in [Grafana](https://grafana.com/) using the
[Grafana SQLite datasource plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/).
## Prerequisites
- mt5cli installed and able to connect to MetaTrader 5
- Grafana 10+ with the `frser-sqlite-datasource` plugin installed
- (Optional) Docker and Docker Compose for the containerised setup
## Generating the SQLite database
Collect historical data and snapshot current account state:
```sh
# Collect OHLCV history
mt5cli -o history.db collect-history --symbol EURUSD --date-from 2024-01-01 --date-to 2024-12-31
# Create Grafana-ready views and indexes
mt5cli -o history.db grafana-schema
# Snapshot current account, positions, and orders
mt5cli -o history.db snapshot --with-grafana-schema
```
## Publishing a Grafana-readable copy
Grafana reads the SQLite file directly. To avoid read/write conflicts, publish
a consistent copy after each update:
```sh
mt5cli -o history.db grafana-schema --publish-copy history.mt5cli.db
mt5cli -o history.db snapshot --publish-copy history.mt5cli.db
```
The `--publish-copy` option uses the SQLite online backup API, which is safe
even when the source database uses WAL journal mode.
## Configuring the datasource path
Edit `provisioning/datasources/mt5cli-sqlite.yml` and set the `path` field
to the absolute path of your published `.db` file:
```yaml
jsonData:
path: /absolute/path/to/history.mt5cli.db
```
## Running Grafana on Windows (native)
1. Download and install Grafana from <https://grafana.com/grafana/download/>.
2. Install the SQLite plugin: `grafana-cli plugins install frser-sqlite-datasource`.
3. Copy `provisioning/datasources/mt5cli-sqlite.yml` into
`%ProgramFiles%\GrafanaLabs\grafana\conf\provisioning\datasources\`.
Do not copy `provisioning/dashboards/mt5cli.yml` — it contains a
Docker-specific dashboard path that is not valid on Windows.
4. Import the dashboards from `dashboards/` via the Grafana UI
(Dashboards → Import → Upload JSON file).
## Running with Docker Compose
Set `MT5CLI_DB_PATH` to the absolute path of your published `.db` file, then
start the stack:
```sh
# From the examples/grafana directory
MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db docker compose up -d
```
Alternatively, create a `.env` file in `examples/grafana/` containing
`MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db` and run
`docker compose up -d`. Compose refuses to start if the variable is unset or
empty.
Then open <http://localhost:3000> (default credentials: admin / admin).
## Dashboard overview
| Dashboard | Description |
| ---------------------- | ------------------------------------------------------- |
| `mt5cli-overview.json` | Account balance, equity, margin, and snapshot freshness |
| `mt5cli-trades.json` | Trade P/L, win rate, symbol breakdown |
| `mt5cli-market.json` | OHLCV rates, spreads, and tick volume |
All panel queries use the `grafana_*` views; they do not read internal storage
tables directly.
## Importing dashboards
1. Open Grafana and navigate to **Dashboards → Import**.
2. Click **Upload JSON file** and select one of the files in `dashboards/`.
3. Select the `mt5cli-SQLite` datasource when prompted.
4. Click **Import**.
+26
View File
@@ -0,0 +1,26 @@
# Docker Compose for Grafana with mt5cli SQLite datasource.
#
# MT5CLI_DB_PATH must be set to the absolute host path of your published .db
# file before running `docker compose up -d`. Compose will refuse to start if
# the variable is missing or empty.
#
# Example:
# MT5CLI_DB_PATH=/home/user/history.mt5cli.db docker compose up -d
services:
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
GF_INSTALL_PLUGINS: frser-sqlite-datasource
volumes:
- ./provisioning:/etc/grafana/provisioning:ro
- ./dashboards:/var/lib/grafana/dashboards:ro
- grafana-storage:/var/lib/grafana
- ${MT5CLI_DB_PATH:?Set MT5CLI_DB_PATH to the path of your published mt5cli SQLite DB}:/data/mt5cli.db:ro
user: "472"
volumes:
grafana-storage:
@@ -0,0 +1,98 @@
{
"__inputs": [
{
"name": "DS_MT5CLI_SQLITE",
"label": "mt5cli-SQLite",
"description": "",
"type": "datasource",
"pluginId": "frser-sqlite-datasource",
"pluginName": "SQLite"
}
],
"__requires": [
{
"type": "datasource",
"id": "frser-sqlite-datasource",
"name": "SQLite",
"version": "1.0.0"
}
],
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 0 },
"id": 1,
"title": "Close Price Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"close\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
"id": 2,
"title": "Spread Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"spread\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
"id": 3,
"title": "Tick Volume Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"symbol\", \"tick_volume\" FROM grafana_rates WHERE \"time\" >= $__from / 1000 AND \"time\" < $__to / 1000 ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
}
],
"refresh": "1m",
"schemaVersion": 36,
"tags": ["mt5cli", "market"],
"templating": {
"list": [
{
"current": {},
"hide": 0,
"includeAll": false,
"label": "Data Source",
"multi": false,
"name": "DS_MT5CLI_SQLITE",
"options": [],
"query": "frser-sqlite-datasource",
"refresh": 1,
"type": "datasource"
}
]
},
"time": { "from": "now-24h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "MT5CLI - Market Data",
"uid": "mt5cli-market",
"version": 1
}
@@ -0,0 +1,254 @@
{
"__inputs": [
{
"name": "DS_MT5CLI_SQLITE",
"label": "mt5cli-SQLite",
"description": "",
"type": "datasource",
"pluginId": "frser-sqlite-datasource",
"pluginName": "SQLite"
}
],
"__requires": [
{
"type": "datasource",
"id": "frser-sqlite-datasource",
"name": "SQLite",
"version": "1.0.0"
}
],
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {
"unit": "currencyUSD"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"]
}
},
"title": "Balance",
"type": "stat",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {
"unit": "currencyUSD"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 0
},
"id": 2,
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"]
}
},
"title": "Equity",
"type": "stat",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {
"unit": "currencyUSD"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 0
},
"id": 3,
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"]
}
},
"title": "Free Margin",
"type": "stat",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"margin_free\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 0
},
"id": 4,
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"]
}
},
"title": "Margin Level",
"type": "stat",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"margin_level\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {
"unit": "dateTimeFromNow"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 24,
"x": 0,
"y": 4
},
"id": 7,
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"]
}
},
"title": "Last Snapshot",
"type": "stat",
"targets": [
{
"rawSql": "SELECT MAX(\"time\") * 1000 AS \"Last Snapshot\" FROM grafana_account_snapshots",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 8
},
"id": 5,
"title": "Account Balance Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 16
},
"id": 6,
"title": "Equity Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
}
],
"refresh": "1m",
"schemaVersion": 36,
"tags": ["mt5cli", "account"],
"templating": {
"list": [
{
"current": {},
"hide": 0,
"includeAll": false,
"label": "Data Source",
"multi": false,
"name": "DS_MT5CLI_SQLITE",
"options": [],
"query": "frser-sqlite-datasource",
"refresh": 1,
"type": "datasource"
}
]
},
"time": {
"from": "now-7d",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "MT5CLI - Account Overview",
"uid": "mt5cli-overview",
"version": 1
}
@@ -0,0 +1,167 @@
{
"__inputs": [
{
"name": "DS_MT5CLI_SQLITE",
"label": "mt5cli-SQLite",
"description": "",
"type": "datasource",
"pluginId": "frser-sqlite-datasource",
"pluginName": "SQLite"
}
],
"__requires": [
{
"type": "datasource",
"id": "frser-sqlite-datasource",
"name": "SQLite",
"version": "1.0.0"
}
],
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"title": "Realized P/L by Symbol",
"type": "table",
"targets": [
{
"rawSql": "SELECT \"symbol\", \"cumulative_pnl\", \"deal_count\" FROM grafana_realized_pnl ORDER BY cumulative_pnl DESC",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "win_rate_pct"
},
"properties": [
{
"id": "unit",
"value": "percent"
},
{
"id": "displayName",
"value": "Win Rate (%)"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"title": "Trade Statistics by Symbol",
"type": "table",
"targets": [
{
"rawSql": "SELECT \"symbol\", \"total_deals\", \"winning_deals\", \"losing_deals\", \"total_profit\", \"avg_profit\", 100.0 * \"winning_deals\" / NULLIF(\"total_deals\", 0) AS \"win_rate_pct\" FROM grafana_trade_stats ORDER BY total_profit DESC",
"format": "table",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 8
},
"id": 3,
"title": "Open Position Profit Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"symbol\", SUM(\"profit\") AS profit FROM grafana_position_snapshots GROUP BY time, symbol ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
},
{
"datasource": "${DS_MT5CLI_SQLITE}",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 16
},
"id": 4,
"title": "Cash Events Over Time",
"type": "timeseries",
"targets": [
{
"rawSql": "SELECT \"time\" AS time, \"profit\" FROM grafana_cash_events ORDER BY time",
"format": "time_series",
"refId": "A"
}
]
}
],
"refresh": "5m",
"schemaVersion": 36,
"tags": ["mt5cli", "trades"],
"templating": {
"list": [
{
"current": {},
"hide": 0,
"includeAll": false,
"label": "Data Source",
"multi": false,
"name": "DS_MT5CLI_SQLITE",
"options": [],
"query": "frser-sqlite-datasource",
"refresh": 1,
"type": "datasource"
}
]
},
"time": {
"from": "now-30d",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "MT5CLI - Trade Analytics",
"uid": "mt5cli-trades",
"version": 1
}
@@ -0,0 +1,13 @@
# Grafana dashboard provisioning for mt5cli dashboards.
apiVersion: 1
providers:
- name: mt5cli
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: false
@@ -0,0 +1,16 @@
# Grafana datasource provisioning for mt5cli SQLite.
#
# Requires the frser-sqlite-datasource plugin:
# grafana-cli plugins install frser-sqlite-datasource
#
# Set `path` to the absolute path of your published history.mt5cli.db file.
apiVersion: 1
datasources:
- name: mt5cli-SQLite
type: frser-sqlite-datasource
access: proxy
isDefault: true
jsonData:
path: /data/mt5cli.db
+4
View File
@@ -35,6 +35,8 @@ from .sdk import (
resolve_account_specs,
update_history,
update_history_with_config,
update_observability,
update_observability_with_config,
)
from .trading import (
ExecutionStatus,
@@ -140,6 +142,8 @@ __all__ = [
"resolve_account_specs",
"update_history",
"update_history_with_config",
"update_observability",
"update_observability_with_config",
"update_sltp_for_open_positions",
"update_trailing_stop_loss_for_open_positions",
]
+125
View File
@@ -802,6 +802,131 @@ def collect_history(
)
@app.command(rich_help_panel="Collection")
def grafana_schema(
ctx: typer.Context,
publish_copy: Annotated[
Path | None,
typer.Option(
"--publish-copy",
help=(
"Publish a Grafana-ready SQLite copy to this path"
" after schema creation."
),
),
] = None,
) -> None:
"""Create or refresh Grafana-ready views and indexes in a SQLite database.
Idempotent safe to run repeatedly on the same database. Requires SQLite
output. Does not connect to MetaTrader 5.
Raises:
typer.BadParameter: If the output format is not SQLite3.
"""
import sqlite3 as _sqlite3 # noqa: PLC0415
from .grafana import ( # noqa: PLC0415
create_snapshot_tables,
ensure_grafana_schema,
publish_grafana_copy,
)
export_ctx = _get_export_context(ctx)
if export_ctx.output_format != "sqlite3":
msg = (
"grafana-schema requires SQLite3 output."
" Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
)
raise typer.BadParameter(msg)
with _sqlite3.connect(export_ctx.output) as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
create_snapshot_tables(conn)
ensure_grafana_schema(conn)
logger.info("Grafana schema applied to %s", export_ctx.output)
if publish_copy is not None:
publish_grafana_copy(export_ctx.output, publish_copy)
logger.info("Grafana copy published to %s", publish_copy)
@app.command(rich_help_panel="Collection")
def snapshot(
ctx: typer.Context,
symbol: Annotated[
list[str] | None,
typer.Option(
"--symbol",
"-s",
help="Symbol filter for positions/orders (repeat for multiple).",
),
] = None,
with_account: Annotated[
bool,
typer.Option("--with-account/--no-account", help="Snapshot account info."),
] = True,
with_positions: Annotated[
bool,
typer.Option(
"--with-positions/--no-positions", help="Snapshot open positions."
),
] = True,
with_orders: Annotated[
bool,
typer.Option("--with-orders/--no-orders", help="Snapshot active orders."),
] = True,
with_terminal: Annotated[
bool,
typer.Option("--with-terminal/--no-terminal", help="Snapshot terminal info."),
] = True,
with_grafana_schema: Annotated[
bool,
typer.Option(
"--with-grafana-schema/--no-grafana-schema",
help="Ensure Grafana views and indexes exist.",
),
] = False,
publish_copy: Annotated[
Path | None,
typer.Option(
"--publish-copy",
help=("Publish a Grafana-ready SQLite copy to this path after snapshot."),
),
] = None,
) -> None:
"""Snapshot current account, position, order, and terminal state into SQLite.
Appends a timestamped snapshot row for each data type. Never places
orders or modifies trading state.
Raises:
typer.BadParameter: If the output format is not SQLite3.
"""
export_ctx = _get_export_context(ctx)
if export_ctx.output_format != "sqlite3":
msg = (
"snapshot requires SQLite3 output."
" Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
)
raise typer.BadParameter(msg)
sdk.update_observability_with_config(
output=export_ctx.output,
config=export_ctx.config,
symbols=list(symbol) if symbol else None,
include_account=with_account,
include_positions=with_positions,
include_orders=with_orders,
include_terminal=with_terminal,
with_grafana_schema=with_grafana_schema,
)
logger.info("Snapshot written to %s", export_ctx.output)
if publish_copy is not None:
from .grafana import publish_grafana_copy # noqa: PLC0415
publish_grafana_copy(export_ctx.output, publish_copy)
logger.info("Grafana copy published to %s", publish_copy)
def main() -> None:
"""Run the mt5cli CLI."""
app()
+2
View File
@@ -62,6 +62,8 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
"resolve_account_specs",
"update_history",
"update_history_with_config",
"update_observability",
"update_observability_with_config",
"update_sltp_for_open_positions",
"update_trailing_stop_loss_for_open_positions",
})
+682
View File
@@ -0,0 +1,682 @@
"""Grafana-oriented SQLite views, indexes, and snapshot tables."""
from __future__ import annotations
import contextlib
import datetime
import logging
import os
import sqlite3
import tempfile
from pathlib import Path
from typing import cast
from .history import get_table_columns
logger = logging.getLogger(__name__)
_TRADE_DEAL_TYPES_SQL = "(0, 1)"
_GRAFANA_VIEW_NAMES = (
"grafana_rates",
"grafana_ticks",
"grafana_history_deals",
"grafana_history_orders",
"grafana_trade_deals",
"grafana_cash_events",
"grafana_realized_pnl",
"grafana_symbol_pnl",
"grafana_trade_stats",
"grafana_account_snapshots",
"grafana_position_snapshots",
"grafana_order_snapshots",
"grafana_terminal_snapshots",
)
def _to_epoch_int(value: object) -> int | None:
if value is None:
return None
if isinstance(value, datetime.datetime):
return int(value.timestamp())
if isinstance(value, (int, float)):
return int(value)
return None
def _time_col_expr(col: str) -> str:
return (
f"CASE WHEN typeof(\"{col}\") IN ('integer', 'real')"
f' THEN CAST("{col}" AS INTEGER)'
f" ELSE CAST(strftime('%s', \"{col}\") AS INTEGER) END"
)
def _create_view_safe(
conn: sqlite3.Connection,
name: str,
select_sql: str,
) -> None:
try:
conn.execute(f'DROP VIEW IF EXISTS "{name}"')
conn.execute(f'CREATE VIEW "{name}" AS {select_sql}')
except sqlite3.Error as exc:
logger.warning("Skipping view %s: %s", name, exc)
def _other_cols(all_cols: set[str], exclude: set[str]) -> list[str]:
return sorted(all_cols - exclude)
# ---------------------------------------------------------------------------
# Snapshot table DDL
# ---------------------------------------------------------------------------
_SNAPSHOT_TABLE_DDLS: list[str] = [
"""CREATE TABLE IF NOT EXISTS snapshot_runs (
run_id INTEGER PRIMARY KEY,
observed_at INTEGER NOT NULL,
status TEXT NOT NULL,
detail TEXT
)""",
"""CREATE TABLE IF NOT EXISTS account_snapshots (
run_id INTEGER NOT NULL,
login INTEGER,
currency TEXT,
balance REAL,
equity REAL,
margin REAL,
margin_free REAL,
margin_level REAL,
profit REAL,
leverage INTEGER
)""",
"""CREATE TABLE IF NOT EXISTS position_snapshots (
run_id INTEGER NOT NULL,
login INTEGER,
ticket INTEGER,
position_id INTEGER,
symbol TEXT,
type INTEGER,
volume REAL,
price_open REAL,
price_current REAL,
profit REAL,
swap REAL,
comment TEXT,
magic INTEGER
)""",
"""CREATE TABLE IF NOT EXISTS order_snapshots (
run_id INTEGER NOT NULL,
login INTEGER,
ticket INTEGER,
symbol TEXT,
type INTEGER,
volume_current REAL,
price_open REAL,
price_current REAL,
state INTEGER,
comment TEXT,
magic INTEGER,
time_setup INTEGER
)""",
"""CREATE TABLE IF NOT EXISTS terminal_snapshots (
run_id INTEGER NOT NULL,
name TEXT,
connected INTEGER,
community_account INTEGER,
trade_allowed INTEGER,
trade_expert INTEGER,
path TEXT,
company TEXT,
language TEXT
)""",
]
def create_snapshot_tables(conn: sqlite3.Connection) -> None:
"""Create snapshot tables idempotently."""
for ddl in _SNAPSHOT_TABLE_DDLS:
conn.execute(ddl)
def start_snapshot_run(conn: sqlite3.Connection, observed_at: int) -> int:
"""Insert a snapshot_runs row with status 'running' and return its run_id.
Returns:
The auto-assigned run_id for the new row.
"""
cursor = conn.execute(
"INSERT INTO snapshot_runs (observed_at, status) VALUES (?, 'running')",
(observed_at,),
)
return cast("int", cursor.lastrowid)
# ---------------------------------------------------------------------------
# View builders
# ---------------------------------------------------------------------------
def _build_grafana_rates(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "rates")
required = {"time", "symbol", "timeframe"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_rates: rates table missing columns %s",
sorted(required - cols),
)
return
time_expr = _time_col_expr("time")
others = _other_cols(cols, {"time"})
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_rates",
f'SELECT {time_expr} AS "time", {other_sql} FROM "rates"', # noqa: S608
)
def _build_grafana_ticks(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "ticks")
required = {"time", "symbol"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_ticks: ticks table missing columns %s",
sorted(required - cols),
)
return
time_expr = _time_col_expr("time")
others = _other_cols(cols, {"time"})
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_ticks",
f'SELECT {time_expr} AS "time", {other_sql} FROM "ticks"', # noqa: S608
)
def _build_grafana_history_deals(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
if "time" not in cols:
logger.warning("Skipping grafana_history_deals: history_deals.time is missing")
return
time_expr = _time_col_expr("time")
others = _other_cols(cols, {"time"})
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_history_deals",
f'SELECT {time_expr} AS "time", {other_sql} FROM "history_deals"', # noqa: S608
)
def _build_grafana_history_orders(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_orders")
if "time_setup" not in cols:
logger.warning(
"Skipping grafana_history_orders: history_orders.time_setup is missing"
)
return
time_expr = _time_col_expr("time_setup")
others = _other_cols(cols, set())
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_history_orders",
f'SELECT {time_expr} AS "time", {other_sql} FROM "history_orders"', # noqa: S608
)
def _build_grafana_trade_deals(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
required = {"time", "type"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_trade_deals: history_deals missing columns %s",
sorted(required - cols),
)
return
time_expr = _time_col_expr("time")
others = _other_cols(cols, {"time"})
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_trade_deals",
f'SELECT {time_expr} AS "time", {other_sql}' # noqa: S608
f' FROM "history_deals" WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}',
)
def _build_grafana_cash_events(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
required = {"time", "type"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_cash_events: history_deals missing columns %s",
sorted(required - cols),
)
return
time_expr = _time_col_expr("time")
others = _other_cols(cols, {"time"})
other_sql = ", ".join(f'"{c}"' for c in others)
_create_view_safe(
conn,
"grafana_cash_events",
f'SELECT {time_expr} AS "time", {other_sql}' # noqa: S608
f' FROM "history_deals" WHERE "type" NOT IN {_TRADE_DEAL_TYPES_SQL}',
)
def _build_grafana_realized_pnl(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
required = {"symbol", "profit", "type", "entry"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_realized_pnl: history_deals missing columns %s",
sorted(required - cols),
)
return
_create_view_safe(
conn,
"grafana_realized_pnl",
'SELECT "symbol",' # noqa: S608
' SUM("profit") AS cumulative_pnl, COUNT(*) AS deal_count'
' FROM "history_deals"'
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
' AND "entry" IN (1, 2, 3)'
' AND "symbol" IS NOT NULL AND "symbol" != \'\''
' GROUP BY "symbol"',
)
def _build_grafana_symbol_pnl(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
required = {"time", "symbol", "profit", "type", "entry"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_symbol_pnl: history_deals missing columns %s",
sorted(required - cols),
)
return
time_expr = _time_col_expr("time")
select_parts = [f'{time_expr} AS "time"', '"symbol"', '"profit"']
if "volume" in cols:
select_parts.append('"volume"')
if "price" in cols:
select_parts.append('"price"')
select_sql = ", ".join(select_parts)
_create_view_safe(
conn,
"grafana_symbol_pnl",
f'SELECT {select_sql} FROM "history_deals"' # noqa: S608
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
' AND "entry" IN (1, 2, 3)'
' AND "symbol" IS NOT NULL AND "symbol" != \'\'',
)
def _build_grafana_trade_stats(conn: sqlite3.Connection) -> None:
cols = get_table_columns(conn, "history_deals")
required = {"symbol", "profit", "type"}
if not required.issubset(cols):
logger.warning(
"Skipping grafana_trade_stats: history_deals missing columns %s",
sorted(required - cols),
)
return
has_entry = "entry" in cols
entry_filter = ' AND "entry" IN (1, 2, 3)' if has_entry else ""
_create_view_safe(
conn,
"grafana_trade_stats",
'SELECT "symbol",' # noqa: S608
" COUNT(*) AS total_deals,"
' SUM(CASE WHEN "profit" > 0 THEN 1 ELSE 0 END) AS winning_deals,'
' SUM(CASE WHEN "profit" <= 0 THEN 1 ELSE 0 END) AS losing_deals,'
' SUM("profit") AS total_profit,'
' AVG("profit") AS avg_profit,'
' MAX("profit") AS max_profit,'
' MIN("profit") AS min_profit'
' FROM "history_deals"'
f' WHERE "type" IN {_TRADE_DEAL_TYPES_SQL}'
f"{entry_filter}"
' AND "symbol" IS NOT NULL AND "symbol" != \'\''
' GROUP BY "symbol"',
)
def _build_snapshot_view(
conn: sqlite3.Connection,
view_name: str,
table_name: str,
) -> None:
cols = get_table_columns(conn, table_name)
if not cols:
logger.warning("Skipping %s: %s table missing", view_name, table_name)
return
if "run_id" not in cols:
logger.warning("Skipping %s: %s missing run_id column", view_name, table_name)
return
others = _other_cols(cols, {"run_id"})
run_cols = get_table_columns(conn, "snapshot_runs")
if {"run_id", "observed_at", "status"}.issubset(run_cols):
other_sql = (", " + ", ".join(f's."{c}"' for c in others)) if others else ""
select_cols = f'r."observed_at" AS "time", s."run_id"{other_sql}'
_create_view_safe(
conn,
view_name,
f'SELECT {select_cols} FROM "{table_name}" s' # noqa: S608
f' JOIN "snapshot_runs" r ON s."run_id" = r."run_id"'
f" WHERE r.\"status\" = 'ok'",
)
else:
logger.warning("Skipping %s: snapshot_runs missing required columns", view_name)
def _build_grafana_account_snapshots(conn: sqlite3.Connection) -> None:
_build_snapshot_view(conn, "grafana_account_snapshots", "account_snapshots")
def _build_grafana_position_snapshots(conn: sqlite3.Connection) -> None:
_build_snapshot_view(conn, "grafana_position_snapshots", "position_snapshots")
def _build_grafana_order_snapshots(conn: sqlite3.Connection) -> None:
_build_snapshot_view(conn, "grafana_order_snapshots", "order_snapshots")
def _build_grafana_terminal_snapshots(conn: sqlite3.Connection) -> None:
_build_snapshot_view(conn, "grafana_terminal_snapshots", "terminal_snapshots")
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def create_grafana_views(conn: sqlite3.Connection) -> None:
"""Create all Grafana-facing views idempotently.
Missing source tables cause the affected view to be skipped with a warning;
other views are unaffected. Stale views whose source table or required
columns have disappeared are dropped before rebuild.
"""
for name in _GRAFANA_VIEW_NAMES:
conn.execute(f'DROP VIEW IF EXISTS "{name}"')
_build_grafana_rates(conn)
_build_grafana_ticks(conn)
_build_grafana_history_deals(conn)
_build_grafana_history_orders(conn)
_build_grafana_trade_deals(conn)
_build_grafana_cash_events(conn)
_build_grafana_realized_pnl(conn)
_build_grafana_symbol_pnl(conn)
_build_grafana_trade_stats(conn)
_build_grafana_account_snapshots(conn)
_build_grafana_position_snapshots(conn)
_build_grafana_order_snapshots(conn)
_build_grafana_terminal_snapshots(conn)
def create_grafana_indexes(conn: sqlite3.Connection) -> None:
"""Create Grafana query performance indexes idempotently."""
rates_cols = get_table_columns(conn, "rates")
if {"time", "symbol", "timeframe"}.issubset(rates_cols):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_rates_time_symbol_timeframe"
' ON "rates"("time", "symbol", "timeframe")',
)
ticks_cols = get_table_columns(conn, "ticks")
if {"time", "symbol"}.issubset(ticks_cols):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ticks_time_symbol"
' ON "ticks"("time", "symbol")',
)
deals_cols = get_table_columns(conn, "history_deals")
if {"time", "symbol"}.issubset(deals_cols):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_history_deals_time_symbol"
' ON "history_deals"("time", "symbol")',
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_history_deals_symbol_time"
' ON "history_deals"("symbol", "time")',
)
orders_cols = get_table_columns(conn, "history_orders")
if {"time_setup", "symbol"}.issubset(orders_cols):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_history_orders_time_setup_symbol"
' ON "history_orders"("time_setup", "symbol")',
)
if {"run_id", "login"}.issubset(get_table_columns(conn, "account_snapshots")):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_account_snapshots_time_login"
' ON "account_snapshots"("run_id", "login")',
)
if {"run_id", "symbol"}.issubset(get_table_columns(conn, "position_snapshots")):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_position_snapshots_time_symbol"
' ON "position_snapshots"("run_id", "symbol")',
)
if {"run_id", "symbol"}.issubset(get_table_columns(conn, "order_snapshots")):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_order_snapshots_time_symbol"
' ON "order_snapshots"("run_id", "symbol")',
)
if {"observed_at", "status"}.issubset(get_table_columns(conn, "snapshot_runs")):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_snapshot_runs_time_status"
' ON "snapshot_runs"("observed_at", "status")',
)
def ensure_grafana_schema(conn: sqlite3.Connection) -> None:
"""Create snapshot tables, Grafana views, and indexes idempotently."""
create_snapshot_tables(conn)
create_grafana_views(conn)
create_grafana_indexes(conn)
def publish_grafana_copy(
source: str | Path,
target: str | Path,
) -> Path:
"""Publish a consistent SQLite copy for Grafana using the backup API.
Uses the SQLite online backup API for a WAL-safe, consistent snapshot of
the source database. Writes to a temporary file beside the target, then
atomically replaces it so that a previous published copy is preserved if
publishing fails.
Args:
source: Path to the source SQLite database.
target: Destination path for the published copy.
Returns:
The resolved absolute target path.
Raises:
FileNotFoundError: If the source database does not exist.
ValueError: If source and target resolve to the same path.
"""
source_path = Path(source)
target_path = Path(target)
if source_path.resolve() == target_path.resolve():
msg = "--publish-copy target must differ from the source database: " + str(
source_path
)
raise ValueError(msg)
if not source_path.exists():
raise FileNotFoundError(source_path)
target_path.parent.mkdir(parents=True, exist_ok=True)
tmp_fd, tmp_str = tempfile.mkstemp(
dir=target_path.parent,
suffix=".tmp",
prefix=target_path.name + ".",
)
tmp_path = Path(tmp_str)
try:
os.close(tmp_fd)
with (
contextlib.closing(sqlite3.connect(source_path)) as src,
contextlib.closing(sqlite3.connect(tmp_path)) as dst,
):
src.backup(dst)
try:
target_mode = target_path.stat().st_mode & 0o777
except FileNotFoundError:
target_mode = 0o644
Path(tmp_path).chmod(target_mode)
tmp_path.replace(target_path)
except Exception:
with contextlib.suppress(OSError):
tmp_path.unlink()
raise
logger.info("Published Grafana copy: %s -> %s", source_path, target_path)
return target_path.resolve()
# ---------------------------------------------------------------------------
# Snapshot insert helpers
# ---------------------------------------------------------------------------
def insert_account_snapshot(
conn: sqlite3.Connection,
run_id: int,
row: dict[str, object],
) -> None:
"""Append one account state row to account_snapshots."""
conn.execute(
"INSERT INTO account_snapshots"
" (run_id, login, currency, balance, equity,"
" margin, margin_free, margin_level, profit, leverage)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
run_id,
row.get("login"),
row.get("currency"),
row.get("balance"),
row.get("equity"),
row.get("margin"),
row.get("margin_free"),
row.get("margin_level"),
row.get("profit"),
row.get("leverage"),
),
)
def insert_position_snapshots(
conn: sqlite3.Connection,
run_id: int,
login: int | None,
rows: list[dict[str, object]],
) -> None:
"""Append position rows to position_snapshots; no-op when rows is empty."""
if not rows:
return
conn.executemany(
"INSERT INTO position_snapshots"
" (run_id, login, ticket, position_id, symbol, type, volume,"
" price_open, price_current, profit, swap, comment, magic)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
[
(
run_id,
login,
r.get("ticket"),
r.get("position_id"),
r.get("symbol"),
r.get("type"),
r.get("volume"),
r.get("price_open"),
r.get("price_current"),
r.get("profit"),
r.get("swap"),
r.get("comment"),
r.get("magic"),
)
for r in rows
],
)
def insert_order_snapshots(
conn: sqlite3.Connection,
run_id: int,
login: int | None,
rows: list[dict[str, object]],
) -> None:
"""Append order rows to order_snapshots; no-op when rows is empty."""
if not rows:
return
conn.executemany(
"INSERT INTO order_snapshots"
" (run_id, login, ticket, symbol, type, volume_current,"
" price_open, price_current, state, comment, magic, time_setup)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
[
(
run_id,
login,
r.get("ticket"),
r.get("symbol"),
r.get("type"),
r.get("volume_current"),
r.get("price_open"),
r.get("price_current"),
r.get("state"),
r.get("comment"),
r.get("magic"),
_to_epoch_int(r.get("time_setup")),
)
for r in rows
],
)
def insert_terminal_snapshot(
conn: sqlite3.Connection,
run_id: int,
row: dict[str, object],
) -> None:
"""Append one terminal state row to terminal_snapshots."""
conn.execute(
"INSERT INTO terminal_snapshots"
" (run_id, name, connected, community_account,"
" trade_allowed, trade_expert, path, company, language)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
run_id,
row.get("name"),
row.get("connected"),
row.get("community_account"),
row.get("trade_allowed"),
row.get("trade_expert"),
row.get("path"),
row.get("company"),
row.get("language"),
),
)
def record_snapshot_run(
conn: sqlite3.Connection,
run_id: int,
status: str,
detail: str | None = None,
) -> None:
"""Finalize a snapshot run by setting its status."""
conn.execute(
"UPDATE snapshot_runs SET status = ?, detail = ? WHERE run_id = ?",
(status, detail, run_id),
)
+255 -17
View File
@@ -8,7 +8,7 @@ import os
import re
import sqlite3
import time
from contextlib import contextmanager
from contextlib import closing, contextmanager
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -22,6 +22,16 @@ try:
except ImportError: # pragma: no cover
Mt5TradingError = None # type: ignore[assignment]
from .grafana import (
create_snapshot_tables,
ensure_grafana_schema,
insert_account_snapshot,
insert_order_snapshots,
insert_position_snapshots,
insert_terminal_snapshot,
record_snapshot_run,
start_snapshot_run,
)
from .history import (
create_cash_events_view,
create_history_indexes,
@@ -35,6 +45,7 @@ from .history import (
write_incremental_datasets,
)
from .retry import retry_with_backoff
from .telemetry import get_metrics
from .utils import (
Dataset,
IfExists,
@@ -154,6 +165,8 @@ __all__ = [
"terminal_info",
"update_history",
"update_history_with_config",
"update_observability",
"update_observability_with_config",
"version",
]
@@ -1011,23 +1024,27 @@ def update_history( # noqa: PLR0913
sorted(dataset.value for dataset in request.selected),
request.output_path,
)
with sqlite3.connect(request.output_path) as conn:
with closing(sqlite3.connect(request.output_path)) as conn, conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
write_incremental_datasets(
conn,
client,
symbols,
request.selected,
request.resolved_timeframes,
request.resolved_tick_flags,
request.fallback_start,
request.end,
deduplicate=deduplicate,
create_rate_views=create_rate_views,
with_views=with_views,
include_account_events=include_account_events,
)
m = get_metrics()
with m.record_history_update(dataset="history"):
before = conn.total_changes
write_incremental_datasets(
conn,
client,
symbols,
request.selected,
request.resolved_timeframes,
request.resolved_tick_flags,
request.fallback_start,
request.end,
deduplicate=deduplicate,
create_rate_views=create_rate_views,
with_views=with_views,
include_account_events=include_account_events,
)
m.add_history_rows(conn.total_changes - before, dataset="history")
def update_history_with_config( # noqa: PLR0913
@@ -1258,7 +1275,11 @@ def collect_history(
tf = _coerce_timeframe(timeframe)
tick_flags = _coerce_tick_flags(flags)
mt5_config = config or build_config()
with connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
with (
connected_client(mt5_config) as client,
closing(sqlite3.connect(output)) as conn,
conn,
):
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
written_tables, written_columns = write_collected_datasets(
@@ -2148,3 +2169,220 @@ def mt5_summary(*, config: Mt5Config | None = None) -> dict[str, object]:
def mt5_summary_as_df(*, config: Mt5Config | None = None) -> pd.DataFrame:
"""Return an export-safe terminal/account status summary DataFrame."""
return _make_client(config=config).mt5_summary_as_df()
# ---------------------------------------------------------------------------
# Observability: account / position / order / terminal snapshots
# ---------------------------------------------------------------------------
def _emit_account_metrics(row: dict[str, object]) -> None:
login = str(row.get("login", ""))
server = str(row.get("server", ""))
get_metrics().record_account_state(
login=login,
server=server,
balance=float(row.get("balance") or 0.0), # type: ignore[arg-type]
equity=float(row.get("equity") or 0.0), # type: ignore[arg-type]
margin=float(row.get("margin") or 0.0), # type: ignore[arg-type]
margin_free=float(row.get("margin_free") or 0.0), # type: ignore[arg-type]
margin_level=float(row.get("margin_level") or 0.0), # type: ignore[arg-type]
)
def _emit_position_metrics(
rows: list[dict[str, object]],
login: int | None,
) -> None:
m = get_metrics()
login_str = str(login) if login is not None else ""
# Aggregate profit and volume by symbol so hedging accounts (multiple open
# positions sharing the same symbol) emit a single gauge value per symbol
# instead of overwriting with each row's value.
totals: dict[str, tuple[float, float]] = {}
for r in rows:
symbol = str(r.get("symbol", ""))
profit = float(r.get("profit") or 0.0) # type: ignore[arg-type]
volume = float(r.get("volume") or 0.0) # type: ignore[arg-type]
if symbol in totals:
prev_p, prev_v = totals[symbol]
totals[symbol] = (prev_p + profit, prev_v + volume)
else:
totals[symbol] = (profit, volume)
for symbol, (profit, volume) in totals.items():
m.record_position_state(
login=login_str,
server="",
symbol=symbol,
profit=profit,
volume=volume,
)
def _snapshot_account(
conn: sqlite3.Connection,
client: Mt5DataClient,
run_id: int,
) -> int | None:
df = client.account_info_as_df()
if df.empty:
logger.warning(
"account_info_as_df returned empty frame; skipping account snapshot"
)
return None
row = cast("dict[str, object]", df.iloc[0].to_dict())
insert_account_snapshot(conn, run_id, row)
_emit_account_metrics(row)
login_val = row.get("login")
return int(login_val) if login_val is not None else None # type: ignore[arg-type]
def _snapshot_positions(
conn: sqlite3.Connection,
client: Mt5DataClient,
run_id: int,
login: int | None,
symbols: Sequence[str] | None,
) -> None:
df: pd.DataFrame = client.positions_get_as_df()
if symbols is not None and not df.empty and "symbol" in df.columns:
df = df[df["symbol"].isin(symbols)].reset_index(drop=True)
raw = df.to_dict(orient="records") if not df.empty else []
rows = cast("list[dict[str, object]]", raw)
insert_position_snapshots(conn, run_id, login, rows)
_emit_position_metrics(rows, login)
def _snapshot_orders(
conn: sqlite3.Connection,
client: Mt5DataClient,
run_id: int,
login: int | None,
symbols: Sequence[str] | None,
) -> None:
df: pd.DataFrame = client.orders_get_as_df()
if symbols is not None and not df.empty and "symbol" in df.columns:
df = df[df["symbol"].isin(symbols)].reset_index(drop=True)
raw = df.to_dict(orient="records") if not df.empty else []
rows = cast("list[dict[str, object]]", raw)
insert_order_snapshots(conn, run_id, login, rows)
def _emit_terminal_metrics(row: dict[str, object]) -> None:
get_metrics().record_terminal_state(
connected=float(row.get("connected") or 0.0), # type: ignore[arg-type]
trade_allowed=float(row.get("trade_allowed") or 0.0), # type: ignore[arg-type]
trade_expert=float(row.get("trade_expert") or 0.0), # type: ignore[arg-type]
)
def _snapshot_terminal(
conn: sqlite3.Connection,
client: Mt5DataClient,
run_id: int,
) -> None:
df = client.terminal_info_as_df()
if df.empty:
logger.warning(
"terminal_info_as_df returned empty frame; skipping terminal snapshot"
)
return
row = cast("dict[str, object]", df.iloc[0].to_dict())
insert_terminal_snapshot(conn, run_id, row)
_emit_terminal_metrics(row)
def update_observability(
*,
client: Mt5DataClient,
output: Path | str,
symbols: Sequence[str] | None = None,
include_account: bool = True,
include_positions: bool = True,
include_orders: bool = True,
include_terminal: bool = True,
with_grafana_schema: bool = False,
) -> None:
"""Snapshot current account/position/order/terminal state into SQLite.
Reads the current MT5 state and appends timestamped snapshot rows. Never
places orders or modifies trading state.
Args:
client: Connected MT5 data client.
output: SQLite database path.
symbols: Optional symbol filter for positions and orders. When None,
all positions and orders are snapshotted.
include_account: Snapshot account info into ``account_snapshots``.
include_positions: Snapshot open positions into ``position_snapshots``.
include_orders: Snapshot active orders into ``order_snapshots``.
include_terminal: Snapshot terminal info into ``terminal_snapshots``.
with_grafana_schema: Ensure Grafana views and indexes exist. Defaults
to ``False``; run ``grafana-schema`` once to set up the schema,
then use ``snapshot`` repeatedly without this flag.
"""
observed_at = int(datetime.now(UTC).timestamp())
with closing(sqlite3.connect(Path(output))) as conn, conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
if with_grafana_schema:
ensure_grafana_schema(conn)
else:
create_snapshot_tables(conn)
with get_metrics().record_snapshot_update():
run_id = start_snapshot_run(conn, observed_at)
login: int | None = None
try:
if include_account:
login = _snapshot_account(conn, client, run_id)
if include_positions:
_snapshot_positions(conn, client, run_id, login, symbols)
if include_orders:
_snapshot_orders(conn, client, run_id, login, symbols)
if include_terminal:
_snapshot_terminal(conn, client, run_id)
record_snapshot_run(conn, run_id, "ok")
except Exception:
record_snapshot_run(conn, run_id, "error")
conn.commit()
raise
def update_observability_with_config(
*,
output: Path | str,
config: Mt5Config | None = None,
symbols: Sequence[str] | None = None,
include_account: bool = True,
include_positions: bool = True,
include_orders: bool = True,
include_terminal: bool = True,
with_grafana_schema: bool = False,
) -> None:
"""Snapshot current MT5 state, opening and closing the MT5 connection.
Convenience wrapper around :func:`update_observability` for standalone use.
Args:
output: SQLite database path.
config: MT5 connection configuration. Defaults to an empty config that
attaches to a running terminal.
symbols: Optional symbol filter for positions and orders.
include_account: Snapshot account info.
include_positions: Snapshot open positions.
include_orders: Snapshot active orders.
include_terminal: Snapshot terminal info.
with_grafana_schema: Ensure Grafana views and indexes exist.
"""
mt5_config = config or build_config()
with connected_client(mt5_config) as client:
update_observability(
client=client,
output=output,
symbols=symbols,
include_account=include_account,
include_positions=include_positions,
include_orders=include_orders,
include_terminal=include_terminal,
with_grafana_schema=with_grafana_schema,
)
+354
View File
@@ -0,0 +1,354 @@
"""Optional OpenTelemetry metrics for MT5 history and snapshot observability."""
from __future__ import annotations
import logging
import time
from contextlib import contextmanager
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Iterator
logger = logging.getLogger(__name__)
_otel_available_flag = False
try:
import opentelemetry.metrics as _otel_metrics_mod
from opentelemetry.sdk.metrics import MeterProvider as _OtelMeterProvider
from opentelemetry.sdk.metrics.export import (
PeriodicExportingMetricReader as _OtelPeriodicReader,
)
from opentelemetry.sdk.resources import Resource as _OtelResource
_otel_available_flag = True
except ImportError: # pragma: no cover
_otel_metrics_mod = None # type: ignore[assignment]
_OtelMeterProvider = None # type: ignore[assignment]
_OtelPeriodicReader = None # type: ignore[assignment]
_OtelResource = None # type: ignore[assignment]
_OTEL_AVAILABLE: bool = _otel_available_flag
try:
from opentelemetry.exporter.otlp.proto.http.metric_exporter import ( # type: ignore[import]
OTLPMetricExporter as _OtelOTLPExporter, # type: ignore[reportUnknownVariableType]
)
except ImportError: # pragma: no cover
_OtelOTLPExporter = None # type: ignore[assignment, misc]
class _NoOp:
"""No-op instrument that silently ignores all calls."""
def add(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op add."""
def set(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op set."""
def record(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op record."""
_NOOP: _NoOp = _NoOp()
class _Mt5Metrics:
"""MT5 metric instrument registry.
Holds references to OTel instruments. All instruments are no-op until
:meth:`configure` is called with a compatible meter object.
"""
def __init__(self) -> None:
self._history_duration: Any = _NOOP
self._history_rows: Any = _NOOP
self._history_failures: Any = _NOOP
self._snapshot_duration: Any = _NOOP
self._snapshot_failures: Any = _NOOP
self._account_balance: Any = _NOOP
self._account_equity: Any = _NOOP
self._account_margin: Any = _NOOP
self._account_margin_free: Any = _NOOP
self._account_margin_level: Any = _NOOP
self._position_profit: Any = _NOOP
self._position_volume: Any = _NOOP
self._terminal_connected: Any = _NOOP
self._terminal_trade_allowed: Any = _NOOP
self._terminal_trade_expert: Any = _NOOP
self._last_successful_update: Any = _NOOP
def configure(self, meter: Any) -> None: # noqa: ANN401
"""Set up metric instruments from a meter object.
Args:
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object
that supports ``create_counter``, ``create_histogram``, and
``create_gauge``.
"""
self._history_duration = meter.create_histogram(
"mt5_history_update_duration_seconds",
unit="s",
description="Duration of incremental history update operations.",
)
self._history_rows = meter.create_counter(
"mt5_history_update_rows_total",
description="Rows written during incremental history updates.",
)
self._history_failures = meter.create_counter(
"mt5_history_update_failures_total",
description="Number of incremental history update failures.",
)
self._snapshot_duration = meter.create_histogram(
"mt5_snapshot_update_duration_seconds",
unit="s",
description="Duration of snapshot update operations.",
)
self._snapshot_failures = meter.create_counter(
"mt5_snapshot_update_failures_total",
description="Number of snapshot update failures.",
)
self._account_balance = meter.create_gauge(
"mt5_account_balance",
description="Account balance.",
)
self._account_equity = meter.create_gauge(
"mt5_account_equity",
description="Account equity.",
)
self._account_margin = meter.create_gauge(
"mt5_account_margin",
description="Account margin used.",
)
self._account_margin_free = meter.create_gauge(
"mt5_account_margin_free",
description="Account free margin.",
)
self._account_margin_level = meter.create_gauge(
"mt5_account_margin_level",
description="Account margin level as a percentage.",
)
self._position_profit = meter.create_gauge(
"mt5_position_profit",
description="Floating profit for an open position.",
)
self._position_volume = meter.create_gauge(
"mt5_position_volume",
description="Volume of an open position.",
)
self._terminal_connected = meter.create_gauge(
"mt5_terminal_connected",
description="1 if the terminal is connected to the broker, 0 otherwise.",
)
self._terminal_trade_allowed = meter.create_gauge(
"mt5_terminal_trade_allowed",
description="1 if trading is allowed by the broker server, 0 otherwise.",
)
self._terminal_trade_expert = meter.create_gauge(
"mt5_terminal_trade_expert",
description="1 if Expert Advisor trading is enabled, 0 otherwise.",
)
self._last_successful_update = meter.create_gauge(
"mt5_last_successful_update_timestamp",
description="Unix timestamp of the last successful history update.",
)
@contextmanager
def record_history_update(
self,
*,
dataset: str,
) -> Iterator[None]:
"""Context manager recording history update duration and failures.
Args:
dataset: Dataset label (e.g. ``"rates"``).
Yields:
None inside the update operation.
"""
attrs = {"dataset": dataset}
start = time.monotonic()
try:
yield
self._history_duration.record(time.monotonic() - start, attrs)
self._last_successful_update.set(time.time(), attrs)
except Exception:
self._history_failures.add(1, attrs)
raise
def add_history_rows(self, count: int, *, dataset: str) -> None:
"""Increment the history rows-written counter.
Args:
count: Number of rows written during this update.
dataset: Dataset label (e.g. ``"rates"``).
"""
self._history_rows.add(count, {"dataset": dataset})
@contextmanager
def record_snapshot_update(self) -> Iterator[None]:
"""Context manager recording snapshot update duration and failures.
Yields:
None inside the snapshot operation.
"""
start = time.monotonic()
try:
yield
self._snapshot_duration.record(time.monotonic() - start, {})
except Exception:
self._snapshot_failures.add(1, {})
raise
def record_account_state(
self,
*,
login: str,
server: str,
balance: float,
equity: float,
margin: float,
margin_free: float,
margin_level: float,
) -> None:
"""Emit account metric gauges.
Args:
login: Account login number (as string; not a password or secret).
server: Broker server name.
balance: Account balance.
equity: Account equity.
margin: Margin used.
margin_free: Free margin.
margin_level: Margin level percentage.
"""
attrs: dict[str, str] = {"login": login, "server": server}
self._account_balance.set(balance, attrs)
self._account_equity.set(equity, attrs)
self._account_margin.set(margin, attrs)
self._account_margin_free.set(margin_free, attrs)
self._account_margin_level.set(margin_level, attrs)
def record_position_state(
self,
*,
login: str,
server: str,
symbol: str,
profit: float,
volume: float,
) -> None:
"""Emit position metric gauges.
Args:
login: Account login number (as string).
server: Broker server name.
symbol: Position symbol.
profit: Floating profit/loss.
volume: Position volume.
"""
attrs: dict[str, str] = {"login": login, "server": server, "symbol": symbol}
self._position_profit.set(profit, attrs)
self._position_volume.set(volume, attrs)
def record_terminal_state(
self,
*,
connected: float,
trade_allowed: float,
trade_expert: float,
) -> None:
"""Emit terminal connection and trading status gauges.
Args:
connected: 1.0 if connected to the broker, 0.0 otherwise.
trade_allowed: 1.0 if broker server allows trading, 0.0 otherwise.
trade_expert: 1.0 if Expert Advisor trading is enabled, 0.0 otherwise.
"""
self._terminal_connected.set(connected, {})
self._terminal_trade_allowed.set(trade_allowed, {})
self._terminal_trade_expert.set(trade_expert, {})
_metrics = _Mt5Metrics()
def configure_metrics(meter: Any) -> None: # noqa: ANN401
"""Configure MT5 metrics using the provided meter.
Args:
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object.
"""
_metrics.configure(meter)
def enable_otel_metrics(
service_name: str = "mt5cli",
readers: list[Any] | None = None,
) -> None:
"""Enable OTel metrics by wiring up an SDK ``MeterProvider`` pipeline.
Requires the ``otel`` optional dependency group:
``pip install "mt5cli[otel]"``.
Args:
service_name: OTel meter/service name used for the ``Resource`` and
the meter itself.
readers: Optional list of metric readers. When *None* (the default),
a :class:`~opentelemetry.sdk.metrics.export.PeriodicExportingMetricReader`
backed by an OTLP HTTP exporter is created automatically
(reads the endpoint from ``OTEL_EXPORTER_OTLP_ENDPOINT``).
Pass a custom list (e.g. ``InMemoryMetricReader`` for tests)
to override.
Raises:
ImportError: If ``opentelemetry-api`` is not installed, or if
``readers`` is *None* and
``opentelemetry-exporter-otlp-proto-http`` is not installed.
"""
if not _OTEL_AVAILABLE:
msg = (
"opentelemetry-api is not installed. "
'Install it with: pip install "mt5cli[otel]"'
)
raise ImportError(msg)
if readers is None:
if _OtelOTLPExporter is None:
msg = (
"opentelemetry-exporter-otlp-proto-http is required for the "
"default OTLP export pipeline. "
'Install it with: pip install "mt5cli[otel]" or pass a '
"custom readers list."
)
raise ImportError(msg)
readers = [_OtelPeriodicReader(_OtelOTLPExporter())] # type: ignore[misc]
resource = _OtelResource.create({"service.name": service_name}) # type: ignore[union-attr]
provider = _OtelMeterProvider(resource=resource, metric_readers=readers) # type: ignore[misc]
_otel_metrics_mod.set_meter_provider(provider) # type: ignore[union-attr]
meter = provider.get_meter(service_name)
configure_metrics(meter)
def get_metrics() -> _Mt5Metrics:
"""Return the global :class:`_Mt5Metrics` instance.
Returns:
The global metric registry (no-op until :func:`configure_metrics` is
called).
"""
return _metrics
+2 -1
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import json
import sqlite3
from contextlib import closing
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
@@ -277,7 +278,7 @@ def export_dataframe_to_sqlite(
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
with closing(sqlite3.connect(output_path)) as conn, conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
+8 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "1.0.3"
version = "1.1.0"
description = "Generic MT5 data and execution infrastructure for Python applications"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -26,6 +26,11 @@ classifiers = [
[project.optional-dependencies]
parquet = ["pyarrow >= 19.0.0"]
otel = [
"opentelemetry-api",
"opentelemetry-sdk",
"opentelemetry-exporter-otlp-proto-http",
]
[project.scripts]
mt5cli = "mt5cli.cli:main"
@@ -45,6 +50,8 @@ dev = [
"pytest-cov >= 5.0.0",
"pandas-stubs >= 2.2.3.250527",
"pyarrow >= 19.0.0",
"opentelemetry-api",
"opentelemetry-sdk",
"mkdocs >= 1.6.1",
"mkdocs-material >= 9.7.6",
"mkdocstrings[python] >= 1.0.4",
+38
View File
@@ -2,12 +2,17 @@
from __future__ import annotations
import sqlite3
from typing import TYPE_CHECKING, Any, Literal
from unittest.mock import MagicMock
import pandas as pd
import pytest
from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from types import TracebackType
_DATAFRAME_METHODS = (
"copy_rates_from_as_df",
"copy_rates_from_pos_as_df",
@@ -30,6 +35,25 @@ _DATAFRAME_METHODS = (
"order_send_as_df",
)
_ORIGINAL_SQLITE_CONNECT = sqlite3.connect
class ClosingSqliteConnection(sqlite3.Connection):
"""SQLite connection that closes after context-manager exit in tests."""
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_value: BaseException | None,
traceback: TracebackType | None,
) -> Literal[False]:
"""Commit or roll back the transaction, then close the connection."""
try:
super().__exit__(exc_type, exc_value, traceback)
finally:
self.close()
return False
def build_mock_mt5_data_client() -> MagicMock:
"""Return a MagicMock Mt5DataClient with common DataFrame stubs."""
@@ -50,3 +74,17 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client = build_mock_mt5_data_client()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@pytest.fixture(autouse=True)
def close_sqlite_context_connections(monkeypatch: pytest.MonkeyPatch) -> None:
"""Make test SQLite context managers close their connection handles."""
def connect(
*args: Any, # noqa: ANN401
**kwargs: Any, # noqa: ANN401
) -> sqlite3.Connection:
kwargs.setdefault("factory", ClosingSqliteConnection)
return _ORIGINAL_SQLITE_CONNECT(*args, **kwargs)
monkeypatch.setattr(sqlite3, "connect", connect)
+239
View File
@@ -1855,6 +1855,245 @@ class TestCollectHistory:
)
class TestGrafanaSchemaCommand:
"""Tests for the grafana-schema CLI command."""
def test_grafana_schema_creates_snapshot_tables_in_sqlite(
self,
tmp_path: Path,
) -> None:
"""grafana-schema applies Grafana schema to a SQLite database."""
output = tmp_path / "out.db"
result = runner.invoke(app, ["-o", str(output), "grafana-schema"])
assert result.exit_code == 0, result.output
with sqlite3.connect(output) as conn:
tables = {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
}
assert "snapshot_runs" in tables
assert "account_snapshots" in tables
def test_grafana_schema_is_idempotent(self, tmp_path: Path) -> None:
"""grafana-schema can be invoked multiple times without error."""
output = tmp_path / "out.db"
result1 = runner.invoke(app, ["-o", str(output), "grafana-schema"])
result2 = runner.invoke(app, ["-o", str(output), "grafana-schema"])
assert result1.exit_code == 0, result1.output
assert result2.exit_code == 0, result2.output
def test_grafana_schema_rejects_non_sqlite_output(
self,
tmp_path: Path,
) -> None:
"""grafana-schema fails when output is not a SQLite3 format."""
result = runner.invoke(
app,
["-o", str(tmp_path / "out.csv"), "grafana-schema"],
)
assert result.exit_code != 0
assert "grafana-schema requires SQLite3 output" in result.output
class TestSnapshotCommand:
"""Tests for the snapshot CLI command."""
def test_snapshot_rejects_non_sqlite_output(self, tmp_path: Path) -> None:
"""Snapshot fails when output is not a SQLite3 format."""
result = runner.invoke(
app,
["-o", str(tmp_path / "out.csv"), "snapshot"],
)
assert result.exit_code != 0
assert "snapshot requires SQLite3 output" in result.output
def test_snapshot_delegates_to_update_observability_with_config(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Snapshot calls sdk.update_observability_with_config."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
output = tmp_path / "out.db"
result = runner.invoke(app, ["-o", str(output), "snapshot"])
assert result.exit_code == 0, result.output
updater.assert_called_once()
kwargs = updater.call_args.kwargs
assert kwargs["output"] == output
assert kwargs["symbols"] is None
assert kwargs["include_account"] is True
assert kwargs["include_positions"] is True
assert kwargs["include_orders"] is True
assert kwargs["include_terminal"] is True
assert kwargs["with_grafana_schema"] is False
def test_snapshot_with_symbol_filter(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Snapshot passes symbol list to update_observability_with_config."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
[
"-o",
str(tmp_path / "out.db"),
"snapshot",
"--symbol",
"EURUSD",
"--symbol",
"GBPUSD",
],
)
assert result.exit_code == 0, result.output
kwargs = updater.call_args.kwargs
assert kwargs["symbols"] == ["EURUSD", "GBPUSD"]
def test_snapshot_with_no_account_flag(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--no-account disables account snapshotting."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
["-o", str(tmp_path / "out.db"), "snapshot", "--no-account"],
)
assert result.exit_code == 0, result.output
assert updater.call_args.kwargs["include_account"] is False
def test_snapshot_with_no_positions_flag(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--no-positions disables position snapshotting."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
["-o", str(tmp_path / "out.db"), "snapshot", "--no-positions"],
)
assert result.exit_code == 0, result.output
assert updater.call_args.kwargs["include_positions"] is False
def test_snapshot_with_no_orders_flag(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--no-orders disables order snapshotting."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
["-o", str(tmp_path / "out.db"), "snapshot", "--no-orders"],
)
assert result.exit_code == 0, result.output
assert updater.call_args.kwargs["include_orders"] is False
def test_snapshot_with_no_terminal_flag(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--no-terminal disables terminal snapshotting."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
["-o", str(tmp_path / "out.db"), "snapshot", "--no-terminal"],
)
assert result.exit_code == 0, result.output
assert updater.call_args.kwargs["include_terminal"] is False
def test_snapshot_with_no_grafana_schema_flag(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--no-grafana-schema disables Grafana schema creation."""
updater = mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
result = runner.invoke(
app,
["-o", str(tmp_path / "out.db"), "snapshot", "--no-grafana-schema"],
)
assert result.exit_code == 0, result.output
assert updater.call_args.kwargs["with_grafana_schema"] is False
def test_snapshot_with_publish_copy(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""--publish-copy calls publish_grafana_copy after update_observability."""
mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
copy_path = tmp_path / "grafana.db"
result = runner.invoke(
app,
[
"-o",
str(tmp_path / "out.db"),
"snapshot",
"--publish-copy",
str(copy_path),
],
)
assert result.exit_code == 0, result.output
mock_publish.assert_called_once()
def test_snapshot_no_publish_copy_by_default(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Snapshot does not call publish_grafana_copy without --publish-copy."""
mocker.patch("mt5cli.cli.sdk.update_observability_with_config")
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
result = runner.invoke(app, ["-o", str(tmp_path / "out.db"), "snapshot"])
assert result.exit_code == 0, result.output
mock_publish.assert_not_called()
class TestGrafanaSchemaPublishCopy:
"""Tests for grafana-schema --publish-copy option."""
def test_grafana_schema_with_publish_copy(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""grafana-schema --publish-copy calls publish_grafana_copy."""
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
output = tmp_path / "out.db"
copy_path = tmp_path / "grafana.db"
result = runner.invoke(
app,
[
"-o",
str(output),
"grafana-schema",
"--publish-copy",
str(copy_path),
],
)
assert result.exit_code == 0, result.output
mock_publish.assert_called_once()
def test_grafana_schema_no_publish_copy_by_default(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""grafana-schema does not call publish_grafana_copy by default."""
mock_publish = mocker.patch("mt5cli.grafana.publish_grafana_copy")
result = runner.invoke(app, ["-o", str(tmp_path / "out.db"), "grafana-schema"])
assert result.exit_code == 0, result.output
mock_publish.assert_not_called()
class TestMain:
"""Tests for the main entry point."""
+75
View File
@@ -0,0 +1,75 @@
"""Tests for example files in examples/grafana/."""
from __future__ import annotations
import json
from pathlib import Path
_EXAMPLES_DIR = Path(__file__).parent.parent / "examples" / "grafana"
_DASHBOARDS_DIR = _EXAMPLES_DIR / "dashboards"
class TestGrafanaExamples:
"""Validate structure and content of bundled Grafana example files."""
def test_dashboard_json_files_are_valid_json(self) -> None:
"""All dashboard JSON files parse without error."""
paths = list(_DASHBOARDS_DIR.glob("*.json"))
assert paths, "No dashboard JSON files found"
for path in paths:
content = path.read_text(encoding="utf-8")
obj = json.loads(content)
assert isinstance(obj, dict), f"{path.name} root must be a JSON object"
def test_dashboard_json_has_no_private_placeholders(self) -> None:
"""Dashboard JSON files contain no obvious credential placeholders."""
private_patterns = ["password", "api_key", "apikey"]
for path in _DASHBOARDS_DIR.glob("*.json"):
content = path.read_text(encoding="utf-8").lower()
for pat in private_patterns:
assert pat not in content, f"{path.name} contains {pat!r}"
def test_dashboard_json_uses_grafana_views(self) -> None:
"""All dashboard JSON files query grafana_* views."""
for path in _DASHBOARDS_DIR.glob("*.json"):
content = path.read_text(encoding="utf-8")
assert "grafana_" in content, (
f"{path.name} must contain queries against grafana_* views"
)
def test_dashboard_json_has_uid(self) -> None:
"""All dashboard JSON files have a non-empty uid field."""
for path in _DASHBOARDS_DIR.glob("*.json"):
obj = json.loads(path.read_text(encoding="utf-8"))
assert obj.get("uid"), f"{path.name} must have a uid"
def test_dashboard_json_has_title(self) -> None:
"""All dashboard JSON files have a non-empty title field."""
for path in _DASHBOARDS_DIR.glob("*.json"):
obj = json.loads(path.read_text(encoding="utf-8"))
assert obj.get("title"), f"{path.name} must have a title"
def test_expected_dashboards_present(self) -> None:
"""The three expected dashboard files are present."""
names = {p.name for p in _DASHBOARDS_DIR.glob("*.json")}
assert "mt5cli-overview.json" in names
assert "mt5cli-trades.json" in names
assert "mt5cli-market.json" in names
def test_readme_exists(self) -> None:
"""examples/grafana/README.md is present."""
assert (_EXAMPLES_DIR / "README.md").is_file()
def test_compose_file_exists(self) -> None:
"""examples/grafana/compose.yml is present."""
assert (_EXAMPLES_DIR / "compose.yml").is_file()
def test_datasource_provisioning_exists(self) -> None:
"""Datasource provisioning YAML is present."""
assert (
_EXAMPLES_DIR / "provisioning" / "datasources" / "mt5cli-sqlite.yml"
).is_file()
def test_dashboard_provisioning_exists(self) -> None:
"""Dashboard provisioning YAML is present."""
assert (_EXAMPLES_DIR / "provisioning" / "dashboards" / "mt5cli.yml").is_file()
+991
View File
@@ -0,0 +1,991 @@
"""Tests for mt5cli.grafana module."""
from __future__ import annotations
import logging
import sqlite3
from pathlib import Path
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pandas as pd
import pytest
if TYPE_CHECKING:
from collections.abc import Iterator
from mt5cli.grafana import (
_build_snapshot_view, # type: ignore[reportPrivateUsage]
_create_view_safe, # type: ignore[reportPrivateUsage]
create_grafana_indexes,
create_grafana_views,
create_snapshot_tables,
ensure_grafana_schema,
insert_account_snapshot,
insert_order_snapshots,
insert_position_snapshots,
insert_terminal_snapshot,
publish_grafana_copy,
record_snapshot_run,
start_snapshot_run,
)
@pytest.fixture
def conn() -> Iterator[sqlite3.Connection]:
"""Yield an in-memory SQLite connection for each test."""
with sqlite3.connect(":memory:") as c:
yield c
def _get_names(conn: sqlite3.Connection, type_: str) -> set[str]:
return {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type=?",
(type_,),
).fetchall()
}
def _make_rates_table(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE rates"
" (time TEXT, symbol TEXT, timeframe INTEGER,"
" open REAL, high REAL, low REAL, close REAL)"
)
def _make_ticks_table(conn: sqlite3.Connection) -> None:
conn.execute("CREATE TABLE ticks (time TEXT, symbol TEXT, bid REAL, ask REAL)")
def _make_history_deals_full(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE history_deals"
" (time TEXT, symbol TEXT, profit REAL, type INTEGER,"
" entry INTEGER, volume REAL, price REAL, ticket INTEGER, position_id INTEGER)"
)
def _make_history_deals_minimal(conn: sqlite3.Connection) -> None:
"""history_deals with only time, type, symbol, profit — no entry/volume/price."""
conn.execute(
"CREATE TABLE history_deals (time TEXT, symbol TEXT, profit REAL, type INTEGER)"
)
def _make_history_orders_table(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE history_orders"
" (time_setup TEXT, symbol TEXT, ticket INTEGER, type INTEGER)"
)
# ---------------------------------------------------------------------------
# TestSnapshotTables
# ---------------------------------------------------------------------------
class TestSnapshotTables:
"""Tests for create_snapshot_tables."""
def test_creates_all_five_tables(self, conn: sqlite3.Connection) -> None:
"""All five snapshot tables are created."""
create_snapshot_tables(conn)
tables = _get_names(conn, "table")
assert "snapshot_runs" in tables
assert "account_snapshots" in tables
assert "position_snapshots" in tables
assert "order_snapshots" in tables
assert "terminal_snapshots" in tables
def test_is_idempotent(self, conn: sqlite3.Connection) -> None:
"""Calling create_snapshot_tables twice does not raise."""
create_snapshot_tables(conn)
create_snapshot_tables(conn)
tables = _get_names(conn, "table")
assert "snapshot_runs" in tables
# ---------------------------------------------------------------------------
# TestCreateViewSafe
# ---------------------------------------------------------------------------
class TestCreateViewSafe:
"""Tests for _create_view_safe."""
def test_creates_view_successfully(self, conn: sqlite3.Connection) -> None:
"""A valid select SQL creates the named view."""
_create_view_safe(conn, "test_view", "SELECT 1 AS val")
views = _get_names(conn, "view")
assert "test_view" in views
def test_replaces_existing_view(self, conn: sqlite3.Connection) -> None:
"""Calling again with a new SQL replaces the existing view."""
_create_view_safe(conn, "test_view", "SELECT 1 AS val")
_create_view_safe(conn, "test_view", "SELECT 2 AS val")
result = conn.execute("SELECT val FROM test_view").fetchone()
assert result == (2,)
def test_logs_warning_on_sqlite_error(
self,
caplog: pytest.LogCaptureFixture,
) -> None:
"""sqlite3.Error during CREATE VIEW logs a warning instead of raising."""
mock_conn = MagicMock()
mock_conn.execute.side_effect = [
None,
sqlite3.OperationalError("parse error"),
]
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
_create_view_safe(mock_conn, "bad_view", "SELECT 1")
assert "Skipping view bad_view" in caplog.text
assert "parse error" in caplog.text
# ---------------------------------------------------------------------------
# TestGrafanaViews
# ---------------------------------------------------------------------------
class TestGrafanaViews:
"""Tests for create_grafana_views and individual view builders."""
def test_all_views_created_with_full_schema(
self,
conn: sqlite3.Connection,
) -> None:
"""All 13 Grafana views are created when all source tables are present."""
_make_rates_table(conn)
_make_ticks_table(conn)
_make_history_deals_full(conn)
_make_history_orders_table(conn)
create_snapshot_tables(conn)
create_grafana_views(conn)
views = _get_names(conn, "view")
expected = {
"grafana_rates",
"grafana_ticks",
"grafana_history_deals",
"grafana_history_orders",
"grafana_trade_deals",
"grafana_cash_events",
"grafana_realized_pnl",
"grafana_symbol_pnl",
"grafana_trade_stats",
"grafana_account_snapshots",
"grafana_position_snapshots",
"grafana_order_snapshots",
"grafana_terminal_snapshots",
}
assert expected.issubset(views)
def test_stale_view_dropped_when_source_table_disappears(
self,
conn: sqlite3.Connection,
) -> None:
"""create_grafana_views drops a previously created view whose source is gone."""
_make_ticks_table(conn)
create_grafana_views(conn)
assert "grafana_ticks" in _get_names(conn, "view")
conn.execute("DROP TABLE ticks")
create_grafana_views(conn)
assert "grafana_ticks" not in _get_names(conn, "view")
def test_grafana_rates_skipped_when_table_absent(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_rates is skipped when rates table is missing."""
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_rates" not in _get_names(conn, "view")
def test_grafana_rates_skipped_when_required_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_rates is skipped when rates table lacks required columns."""
conn.execute("CREATE TABLE rates (open REAL)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_rates" not in _get_names(conn, "view")
assert "Skipping grafana_rates" in caplog.text
def test_grafana_ticks_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_ticks is skipped when ticks table lacks required columns."""
conn.execute("CREATE TABLE ticks (bid REAL)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_ticks" not in _get_names(conn, "view")
assert "Skipping grafana_ticks" in caplog.text
def test_grafana_history_deals_skipped_when_time_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_history_deals is skipped when history_deals.time is missing."""
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_history_deals" not in _get_names(conn, "view")
assert "Skipping grafana_history_deals" in caplog.text
def test_grafana_history_orders_skipped_when_time_setup_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_history_orders is skipped when time_setup is absent."""
conn.execute("CREATE TABLE history_orders (symbol TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_history_orders" not in _get_names(conn, "view")
assert "Skipping grafana_history_orders" in caplog.text
def test_grafana_trade_deals_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_trade_deals is skipped when history_deals missing time/type."""
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_trade_deals" not in _get_names(conn, "view")
def test_grafana_cash_events_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_cash_events is skipped when history_deals missing time/type."""
conn.execute("CREATE TABLE history_deals (symbol TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_cash_events" not in _get_names(conn, "view")
def test_grafana_realized_pnl_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_realized_pnl is skipped when history_deals missing required cols."""
conn.execute("CREATE TABLE history_deals (time TEXT, type INTEGER)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_realized_pnl" not in _get_names(conn, "view")
assert "Skipping grafana_realized_pnl" in caplog.text
def test_grafana_realized_pnl_skipped_when_entry_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_realized_pnl is skipped when entry column is absent."""
_make_history_deals_minimal(conn)
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_realized_pnl" not in _get_names(conn, "view")
assert "Skipping grafana_realized_pnl" in caplog.text
def test_grafana_symbol_pnl_skipped_when_required_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_symbol_pnl is skipped when required columns are absent."""
conn.execute("CREATE TABLE history_deals (time TEXT, type INTEGER)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_symbol_pnl" not in _get_names(conn, "view")
assert "Skipping grafana_symbol_pnl" in caplog.text
def test_grafana_symbol_pnl_without_volume_and_price(
self,
conn: sqlite3.Connection,
) -> None:
"""grafana_symbol_pnl is created with only required columns."""
conn.execute(
"CREATE TABLE history_deals"
" (time TEXT, symbol TEXT, profit REAL, type INTEGER, entry INTEGER)"
)
create_grafana_views(conn)
assert "grafana_symbol_pnl" in _get_names(conn, "view")
def test_grafana_symbol_pnl_with_volume_and_price(
self,
conn: sqlite3.Connection,
) -> None:
"""grafana_symbol_pnl includes volume and price columns when present."""
_make_history_deals_full(conn)
create_grafana_views(conn)
assert "grafana_symbol_pnl" in _get_names(conn, "view")
# View columns include volume and price
cols = {row[1] for row in conn.execute("PRAGMA table_info(grafana_symbol_pnl)")}
assert "volume" in cols
assert "price" in cols
def test_grafana_trade_stats_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""grafana_trade_stats is skipped when history_deals missing required cols."""
conn.execute("CREATE TABLE history_deals (time TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
assert "grafana_trade_stats" not in _get_names(conn, "view")
assert "Skipping grafana_trade_stats" in caplog.text
def test_grafana_trade_stats_without_entry_col(
self,
conn: sqlite3.Connection,
) -> None:
"""grafana_trade_stats is a static summary view with no time column."""
_make_history_deals_minimal(conn)
create_grafana_views(conn)
assert "grafana_trade_stats" in _get_names(conn, "view")
cols = {
row[1] for row in conn.execute("PRAGMA table_info(grafana_trade_stats)")
}
assert "time" not in cols
assert "symbol" in cols
def test_grafana_trade_stats_with_entry_col(
self,
conn: sqlite3.Connection,
) -> None:
"""grafana_trade_stats is a static summary view with no time column."""
_make_history_deals_full(conn)
create_grafana_views(conn)
assert "grafana_trade_stats" in _get_names(conn, "view")
cols = {
row[1] for row in conn.execute("PRAGMA table_info(grafana_trade_stats)")
}
assert "time" not in cols
assert "symbol" in cols
def test_snapshot_views_skipped_when_snapshot_tables_absent(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Snapshot views are skipped when snapshot tables are not created."""
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
create_grafana_views(conn)
views = _get_names(conn, "view")
assert "grafana_account_snapshots" not in views
assert "grafana_position_snapshots" not in views
assert "grafana_order_snapshots" not in views
assert "grafana_terminal_snapshots" not in views
def test_build_snapshot_view_with_only_run_id_col(
self,
conn: sqlite3.Connection,
) -> None:
"""_build_snapshot_view exposes time and run_id when table has only run_id."""
create_snapshot_tables(conn)
conn.execute("CREATE TABLE only_run (run_id INTEGER NOT NULL)")
run_id = start_snapshot_run(conn, 1000)
record_snapshot_run(conn, run_id, "ok")
conn.execute("INSERT INTO only_run (run_id) VALUES (?)", (run_id,))
_build_snapshot_view(conn, "test_view", "only_run")
assert "test_view" in _get_names(conn, "view")
cols = {row[1] for row in conn.execute("PRAGMA table_info(test_view)")}
assert "time" in cols
assert "run_id" in cols
def test_build_snapshot_view_skips_when_snapshot_runs_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""_build_snapshot_view skips view when snapshot_runs has wrong columns."""
conn.execute("CREATE TABLE only_run (run_id INTEGER NOT NULL)")
conn.execute("CREATE TABLE snapshot_runs (foo TEXT)")
_build_snapshot_view(conn, "test_view", "only_run")
views = _get_names(conn, "view")
assert "test_view" not in views
def test_build_snapshot_view_skips_when_run_id_col_missing(
self,
conn: sqlite3.Connection,
caplog: pytest.LogCaptureFixture,
) -> None:
"""_build_snapshot_view skips view when the table lacks run_id."""
create_snapshot_tables(conn)
conn.execute("CREATE TABLE no_run_id (symbol TEXT)")
with caplog.at_level(logging.WARNING, logger="mt5cli.grafana"):
_build_snapshot_view(conn, "test_view", "no_run_id")
assert "test_view" not in _get_names(conn, "view")
assert "missing run_id column" in caplog.text
def test_snapshot_view_excludes_failed_run_rows(
self,
conn: sqlite3.Connection,
) -> None:
"""Snapshot views hide rows from failed runs."""
create_snapshot_tables(conn)
run_id = start_snapshot_run(conn, 1000)
conn.execute(
"INSERT INTO account_snapshots"
" (run_id, login, balance, equity, margin, margin_free, profit)"
" VALUES (?, 12345, 10000.0, 9800.0, 200.0, 9600.0, -200.0)",
(run_id,),
)
record_snapshot_run(conn, run_id, "error", "terminal offline")
create_grafana_views(conn)
rows = conn.execute("SELECT * FROM grafana_account_snapshots").fetchall()
assert rows == []
def test_snapshot_view_includes_ok_run_rows(
self,
conn: sqlite3.Connection,
) -> None:
"""Snapshot views show rows from successful runs and expose run_id."""
create_snapshot_tables(conn)
run_id = start_snapshot_run(conn, 2000)
conn.execute(
"INSERT INTO account_snapshots"
" (run_id, login, balance, equity, margin, margin_free, profit)"
" VALUES (?, 12345, 10000.0, 9800.0, 200.0, 9600.0, -200.0)",
(run_id,),
)
record_snapshot_run(conn, run_id, "ok")
create_grafana_views(conn)
rows = conn.execute(
"SELECT time, run_id, login FROM grafana_account_snapshots"
).fetchall()
assert rows == [(2000, run_id, 12345)]
cols = {
row[1]
for row in conn.execute("PRAGMA table_info(grafana_account_snapshots)")
}
assert "run_id" in cols
def test_snapshot_view_same_second_ok_and_error_no_cross_contamination(
self,
conn: sqlite3.Connection,
) -> None:
"""An ok and error run sharing observed_at expose only the ok run's rows."""
create_snapshot_tables(conn)
run_err = start_snapshot_run(conn, 3000)
conn.execute(
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 99)",
(run_err,),
)
record_snapshot_run(conn, run_err, "error")
run_ok = start_snapshot_run(conn, 3000)
conn.execute(
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 12345)",
(run_ok,),
)
record_snapshot_run(conn, run_ok, "ok")
create_grafana_views(conn)
rows = conn.execute("SELECT login FROM grafana_account_snapshots").fetchall()
assert rows == [(12345,)]
def test_snapshot_view_two_ok_runs_same_second_no_duplication(
self,
conn: sqlite3.Connection,
) -> None:
"""Two ok runs sharing observed_at each produce exactly one row in the view."""
create_snapshot_tables(conn)
run1 = start_snapshot_run(conn, 4000)
conn.execute(
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 1)",
(run1,),
)
record_snapshot_run(conn, run1, "ok")
run2 = start_snapshot_run(conn, 4000)
conn.execute(
"INSERT INTO account_snapshots (run_id, login) VALUES (?, 2)",
(run2,),
)
record_snapshot_run(conn, run2, "ok")
create_grafana_views(conn)
rows = conn.execute("SELECT login FROM grafana_account_snapshots").fetchall()
assert len(rows) == 2
# ---------------------------------------------------------------------------
# TestGrafanaIndexes
# ---------------------------------------------------------------------------
class TestGrafanaIndexes:
"""Tests for create_grafana_indexes."""
def test_all_indexes_created_with_full_schema(
self,
conn: sqlite3.Connection,
) -> None:
"""All 9 indexes are created when all source tables are present."""
_make_rates_table(conn)
_make_ticks_table(conn)
_make_history_deals_full(conn)
_make_history_orders_table(conn)
create_snapshot_tables(conn)
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_rates_time_symbol_timeframe" in indexes
assert "idx_ticks_time_symbol" in indexes
assert "idx_history_deals_time_symbol" in indexes
assert "idx_history_deals_symbol_time" in indexes
assert "idx_history_orders_time_setup_symbol" in indexes
assert "idx_account_snapshots_time_login" in indexes
assert "idx_position_snapshots_time_symbol" in indexes
assert "idx_order_snapshots_time_symbol" in indexes
assert "idx_snapshot_runs_time_status" in indexes
def test_no_indexes_created_when_tables_absent(
self,
conn: sqlite3.Connection,
) -> None:
"""No indexes are created when tables are absent."""
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert not any(name.startswith("idx_") for name in indexes)
def test_indexes_for_snapshot_tables_skipped_when_absent(
self,
conn: sqlite3.Connection,
) -> None:
"""Snapshot table indexes are skipped when snapshot tables don't exist."""
_make_history_deals_full(conn)
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_account_snapshots_time_login" not in indexes
assert "idx_position_snapshots_time_symbol" not in indexes
assert "idx_order_snapshots_time_symbol" not in indexes
assert "idx_snapshot_runs_time_status" not in indexes
def test_rates_index_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""Rates index is skipped when required columns are absent."""
conn.execute("CREATE TABLE rates (open REAL)")
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_rates_time_symbol_timeframe" not in indexes
def test_ticks_index_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""Ticks index is skipped when required columns are absent."""
conn.execute("CREATE TABLE ticks (bid REAL)")
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_ticks_time_symbol" not in indexes
def test_deals_indexes_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""history_deals indexes are skipped when required columns are absent."""
conn.execute("CREATE TABLE history_deals (ticket INTEGER)")
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_history_deals_time_symbol" not in indexes
def test_orders_index_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""history_orders index is skipped when required columns are absent."""
conn.execute("CREATE TABLE history_orders (ticket INTEGER)")
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_history_orders_time_setup_symbol" not in indexes
def test_snapshot_indexes_skipped_when_cols_missing(
self,
conn: sqlite3.Connection,
) -> None:
"""Snapshot table indexes are skipped when required columns are absent."""
conn.execute("CREATE TABLE account_snapshots (foo TEXT)")
conn.execute("CREATE TABLE position_snapshots (foo TEXT)")
conn.execute("CREATE TABLE order_snapshots (foo TEXT)")
conn.execute("CREATE TABLE snapshot_runs (foo TEXT)")
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_account_snapshots_time_login" not in indexes
assert "idx_position_snapshots_time_symbol" not in indexes
assert "idx_order_snapshots_time_symbol" not in indexes
assert "idx_snapshot_runs_time_status" not in indexes
def test_indexes_are_idempotent(self, conn: sqlite3.Connection) -> None:
"""Creating indexes twice does not raise (IF NOT EXISTS)."""
_make_rates_table(conn)
create_grafana_indexes(conn)
create_grafana_indexes(conn)
indexes = _get_names(conn, "index")
assert "idx_rates_time_symbol_timeframe" in indexes
# ---------------------------------------------------------------------------
# TestEnsureGrafanaSchema
# ---------------------------------------------------------------------------
class TestEnsureGrafanaSchema:
"""Tests for ensure_grafana_schema."""
def test_creates_all_tables_views_and_indexes(
self,
conn: sqlite3.Connection,
) -> None:
"""ensure_grafana_schema creates snapshot tables, views, and indexes."""
_make_rates_table(conn)
_make_history_deals_full(conn)
ensure_grafana_schema(conn)
tables = _get_names(conn, "table")
assert "snapshot_runs" in tables
assert "account_snapshots" in tables
views = _get_names(conn, "view")
assert "grafana_rates" in views
assert "grafana_account_snapshots" in views
indexes = _get_names(conn, "index")
assert "idx_rates_time_symbol_timeframe" in indexes
def test_is_idempotent(self, conn: sqlite3.Connection) -> None:
"""Calling ensure_grafana_schema twice does not raise."""
ensure_grafana_schema(conn)
ensure_grafana_schema(conn)
# ---------------------------------------------------------------------------
# TestSnapshotInserts
# ---------------------------------------------------------------------------
class TestSnapshotInserts:
"""Tests for snapshot insert helpers."""
@pytest.fixture(autouse=True)
def setup_tables(self, conn: sqlite3.Connection) -> None:
"""Create snapshot tables before each insert test."""
create_snapshot_tables(conn)
def test_insert_account_snapshot(self, conn: sqlite3.Connection) -> None:
"""insert_account_snapshot appends a row with correct values."""
run_id = start_snapshot_run(conn, 1700000000)
row: dict[str, object] = {
"login": 12345,
"currency": "USD",
"balance": 10000.0,
"equity": 9800.0,
"margin": 200.0,
"margin_free": 9800.0,
"margin_level": 4900.0,
"profit": -200.0,
"leverage": 100,
}
insert_account_snapshot(conn, run_id, row)
result = conn.execute(
"SELECT login, currency, balance FROM account_snapshots"
).fetchone()
assert result == (12345, "USD", 10000.0)
def test_insert_account_snapshot_partial_row(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_account_snapshot works when some fields are missing (uses None)."""
run_id = start_snapshot_run(conn, 1700000000)
insert_account_snapshot(conn, run_id, {"login": 1})
result = conn.execute(
"SELECT login, currency FROM account_snapshots"
).fetchone()
assert result == (1, None)
def test_insert_position_snapshots_with_rows(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_position_snapshots appends each position row."""
run_id = start_snapshot_run(conn, 1700000000)
rows: list[dict[str, object]] = [
{"ticket": 1, "symbol": "EURUSD", "volume": 0.1, "profit": 10.0},
{"ticket": 2, "symbol": "GBPUSD", "volume": 0.2, "profit": -5.0},
]
insert_position_snapshots(conn, run_id, 12345, rows)
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[0]
assert count == 2
def test_insert_position_snapshots_noop_when_empty(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_position_snapshots is a no-op when rows is empty."""
run_id = start_snapshot_run(conn, 1700000000)
insert_position_snapshots(conn, run_id, 12345, [])
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[0]
assert count == 0
def test_insert_order_snapshots_with_rows(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_order_snapshots appends each order row."""
run_id = start_snapshot_run(conn, 1700000000)
rows: list[dict[str, object]] = [
{"ticket": 10, "symbol": "EURUSD", "type": 2, "volume_current": 0.1},
]
insert_order_snapshots(conn, run_id, 12345, rows)
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
assert count == 1
def test_insert_order_snapshots_noop_when_empty(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_order_snapshots is a no-op when rows is empty."""
run_id = start_snapshot_run(conn, 1700000000)
insert_order_snapshots(conn, run_id, 12345, [])
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
assert count == 0
def test_insert_order_snapshots_normalizes_timestamp_time_setup(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_order_snapshots converts pd.Timestamp time_setup to epoch int."""
run_id = start_snapshot_run(conn, 1700000000)
ts = pd.Timestamp("2024-01-15 10:30:00", tz="UTC")
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": ts}]
insert_order_snapshots(conn, run_id, 12345, rows)
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
assert stored == int(ts.timestamp())
def test_insert_order_snapshots_stores_int_time_setup(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_order_snapshots stores an integer time_setup as-is."""
run_id = start_snapshot_run(conn, 1700000000)
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": 1705314600}]
insert_order_snapshots(conn, run_id, 12345, rows)
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
assert stored == 1705314600
def test_insert_order_snapshots_stores_null_for_unknown_time_setup_type(
self,
conn: sqlite3.Connection,
) -> None:
"""insert_order_snapshots stores NULL for an unrecognized time_setup type."""
run_id = start_snapshot_run(conn, 1700000000)
rows: list[dict[str, object]] = [{"ticket": 10, "time_setup": "not_a_time"}]
insert_order_snapshots(conn, run_id, 12345, rows)
stored = conn.execute("SELECT time_setup FROM order_snapshots").fetchone()[0]
assert stored is None
def test_insert_terminal_snapshot(self, conn: sqlite3.Connection) -> None:
"""insert_terminal_snapshot appends a terminal info row."""
run_id = start_snapshot_run(conn, 1700000000)
row: dict[str, object] = {
"name": "MetaTrader 5",
"connected": 1,
"community_account": 0,
"trade_allowed": 1,
"trade_expert": 1,
"path": "/mt5",
"company": "Broker",
"language": "en",
}
insert_terminal_snapshot(conn, run_id, row)
result = conn.execute(
"SELECT name, connected FROM terminal_snapshots"
).fetchone()
assert result == ("MetaTrader 5", 1)
def test_start_snapshot_run_returns_incrementing_ids(
self,
conn: sqlite3.Connection,
) -> None:
"""start_snapshot_run returns a unique run_id for each call."""
run1 = start_snapshot_run(conn, 1700000000)
run2 = start_snapshot_run(conn, 1700000000)
assert run1 != run2
def test_record_snapshot_run_with_detail(
self,
conn: sqlite3.Connection,
) -> None:
"""record_snapshot_run stores status and detail text."""
run_id = start_snapshot_run(conn, 1700000000)
record_snapshot_run(conn, run_id, "error", "RuntimeError: boom")
row = conn.execute("SELECT status, detail FROM snapshot_runs").fetchone()
assert row == ("error", "RuntimeError: boom")
def test_record_snapshot_run_without_detail(
self,
conn: sqlite3.Connection,
) -> None:
"""record_snapshot_run stores None for detail when omitted."""
run_id = start_snapshot_run(conn, 1700000000)
record_snapshot_run(conn, run_id, "ok")
row = conn.execute("SELECT status, detail FROM snapshot_runs").fetchone()
assert row == ("ok", None)
# ---------------------------------------------------------------------------
# TestPublishGrafanaCopy
# ---------------------------------------------------------------------------
def _make_source_db(path: Path) -> None:
"""Create a minimal source SQLite database with snapshot tables."""
with sqlite3.connect(path) as conn:
conn.execute("PRAGMA journal_mode=WAL")
create_snapshot_tables(conn)
conn.execute(
"INSERT INTO snapshot_runs (observed_at, status) VALUES (?, 'ok')",
(1700000000,),
)
class TestPublishGrafanaCopy:
"""Tests for publish_grafana_copy."""
def test_publish_to_fresh_target(self, tmp_path: Path) -> None:
"""publish_grafana_copy creates the target file."""
source = tmp_path / "src.db"
target = tmp_path / "out" / "grafana.db"
_make_source_db(source)
result = publish_grafana_copy(source, target)
assert target.exists()
assert result == target.resolve()
def test_overwrite_existing_target(self, tmp_path: Path) -> None:
"""publish_grafana_copy replaces an existing target without error."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
target.write_bytes(b"stale")
publish_grafana_copy(source, target)
# Target must now be a valid SQLite file from source
with sqlite3.connect(target) as conn:
tables = {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
}
assert "snapshot_runs" in tables
def test_target_contains_source_tables(self, tmp_path: Path) -> None:
"""Published target contains the same tables as the source."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
publish_grafana_copy(source, target)
with sqlite3.connect(target) as conn:
tables = {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
}
assert {"snapshot_runs", "account_snapshots"}.issubset(tables)
def test_target_can_be_opened_readonly(self, tmp_path: Path) -> None:
"""Published target can be opened with uri=True in read-only mode."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
publish_grafana_copy(source, target)
uri = f"file:{target}?mode=ro"
with sqlite3.connect(uri, uri=True) as conn:
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
assert row == ("ok",)
def test_same_path_raises(self, tmp_path: Path) -> None:
"""publish_grafana_copy raises ValueError when source equals target."""
db = tmp_path / "history.db"
_make_source_db(db)
with pytest.raises(ValueError, match="must differ from the source"):
publish_grafana_copy(db, db)
def test_source_not_found_raises(self, tmp_path: Path) -> None:
"""publish_grafana_copy raises FileNotFoundError when source is absent."""
with pytest.raises(FileNotFoundError):
publish_grafana_copy(tmp_path / "missing.db", tmp_path / "out.db")
def test_preserve_old_target_on_backup_failure(self, tmp_path: Path) -> None:
"""Old target is preserved when the backup fails."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
original_content = b"original_data"
target.write_bytes(original_content)
with patch("sqlite3.connect") as mock_connect:
mock_src = MagicMock()
mock_src.__enter__ = MagicMock(return_value=mock_src)
mock_src.__exit__ = MagicMock(return_value=False)
mock_src.backup.side_effect = sqlite3.OperationalError("backup failed")
mock_connect.return_value = mock_src
with pytest.raises(sqlite3.OperationalError, match="backup failed"):
publish_grafana_copy(source, target)
assert target.read_bytes() == original_content
def test_temp_file_cleaned_up_on_failure(self, tmp_path: Path) -> None:
"""Temporary file is removed when backup raises an exception."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
with patch("sqlite3.connect") as mock_connect:
mock_src = MagicMock()
mock_src.__enter__ = MagicMock(return_value=mock_src)
mock_src.__exit__ = MagicMock(return_value=False)
mock_src.backup.side_effect = sqlite3.OperationalError("fail")
mock_connect.return_value = mock_src
with pytest.raises(sqlite3.OperationalError):
publish_grafana_copy(source, target)
tmp_files = list(tmp_path.glob("grafana.db.*.tmp"))
assert not tmp_files, "Temp file should be cleaned up on failure"
def test_returns_path_object(self, tmp_path: Path) -> None:
"""publish_grafana_copy returns a Path instance."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
result = publish_grafana_copy(source, target)
assert isinstance(result, Path)
def test_fresh_target_has_readable_permissions(self, tmp_path: Path) -> None:
"""Published copy is readable by the owner."""
import stat as _stat # noqa: PLC0415
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
publish_grafana_copy(source, target)
mode = target.stat().st_mode & 0o777
assert bool(mode & _stat.S_IRUSR), "owner must be able to read"
@pytest.mark.skipif(
__import__("sys").platform == "win32",
reason="Windows does not support Unix-style group/other permission bits",
)
def test_overwrite_preserves_existing_target_mode(self, tmp_path: Path) -> None:
"""Overwriting an existing target preserves that target's file mode."""
source = tmp_path / "src.db"
target = tmp_path / "grafana.db"
_make_source_db(source)
target.write_bytes(b"old")
target.chmod(0o640)
publish_grafana_copy(source, target)
mode = target.stat().st_mode & 0o777
assert mode == 0o640
+515
View File
@@ -61,6 +61,8 @@ from mt5cli.sdk import (
terminal_info,
update_history,
update_history_with_config,
update_observability,
update_observability_with_config,
version,
)
from mt5cli.utils import Dataset, IfExists, coerce_login
@@ -2746,3 +2748,516 @@ class TestSubstituteMappingValues:
result = substitute_mapping_values(data, keys={"mt5_login"})
# tuple is returned as-is; inner dict is NOT visited
assert result == {"accounts": ({"mt5_login": "${MT5_LOGIN}"},)}
class TestUpdateObservability:
"""Tests for update_observability and update_observability_with_config."""
@pytest.fixture
def mock_client(self) -> MagicMock:
"""Mock client returning minimal valid frames."""
client = MagicMock()
client.account_info_as_df.return_value = pd.DataFrame([
{
"login": 12345,
"currency": "USD",
"balance": 10000.0,
"equity": 10000.0,
"margin": 0.0,
"margin_free": 10000.0,
"margin_level": 0.0,
"profit": 0.0,
"leverage": 100,
}
])
client.positions_get_as_df.return_value = pd.DataFrame()
client.orders_get_as_df.return_value = pd.DataFrame()
client.terminal_info_as_df.return_value = pd.DataFrame([
{
"name": "MetaTrader 5",
"connected": 1,
"community_account": 0,
"trade_allowed": 1,
"trade_expert": 1,
"path": "/mt5",
"company": "Broker",
"language": "en",
}
])
return client
def test_update_observability_creates_snapshot_tables(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""Snapshot tables are created in the output database."""
output = tmp_path / "obs.db"
update_observability(client=mock_client, output=output)
with sqlite3.connect(output) as conn:
tables = {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
}
assert "snapshot_runs" in tables
assert "account_snapshots" in tables
assert "position_snapshots" in tables
def test_update_observability_records_ok_on_success(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""snapshot_runs records 'ok' status on a successful run."""
output = tmp_path / "obs.db"
update_observability(client=mock_client, output=output)
with sqlite3.connect(output) as conn:
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
assert row == ("ok",)
def test_update_observability_records_error_on_failure(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""snapshot_runs records 'error' and re-raises when a snapshot fails."""
mock_client.account_info_as_df.side_effect = RuntimeError("boom")
output = tmp_path / "obs.db"
with pytest.raises(RuntimeError, match="boom"):
update_observability(client=mock_client, output=output)
with sqlite3.connect(output) as conn:
row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
assert row == ("error",)
def test_update_observability_skips_ensure_grafana_schema_when_disabled(
self,
mock_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""with_grafana_schema=False does not call ensure_grafana_schema."""
spy = mocker.spy(sdk, "ensure_grafana_schema")
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
with_grafana_schema=False,
)
spy.assert_not_called()
def test_update_observability_calls_ensure_grafana_schema_by_default(
self,
mock_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""with_grafana_schema=True calls ensure_grafana_schema."""
spy = mocker.spy(sdk, "ensure_grafana_schema")
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
with_grafana_schema=True,
)
spy.assert_called_once()
def test_update_observability_skips_account_when_disabled(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""include_account=False does not call account_info_as_df."""
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
include_account=False,
)
mock_client.account_info_as_df.assert_not_called()
def test_update_observability_skips_positions_when_disabled(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""include_positions=False does not call positions_get_as_df."""
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
include_positions=False,
)
mock_client.positions_get_as_df.assert_not_called()
def test_update_observability_skips_orders_when_disabled(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""include_orders=False does not call orders_get_as_df."""
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
include_orders=False,
)
mock_client.orders_get_as_df.assert_not_called()
def test_update_observability_skips_terminal_when_disabled(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""include_terminal=False does not call terminal_info_as_df."""
update_observability(
client=mock_client,
output=tmp_path / "obs.db",
include_terminal=False,
)
mock_client.terminal_info_as_df.assert_not_called()
def test_update_observability_with_positions_rows(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""Non-empty positions are written to position_snapshots."""
mock_client.positions_get_as_df.return_value = pd.DataFrame([
{
"ticket": 1,
"position_id": 1,
"symbol": "EURUSD",
"type": 0,
"volume": 0.1,
"price_open": 1.1,
"price_current": 1.1,
"profit": 0.0,
"swap": 0.0,
"comment": "",
"magic": 0,
}
])
output = tmp_path / "obs.db"
update_observability(client=mock_client, output=output)
with sqlite3.connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
0
]
assert count == 1
def test_update_observability_with_order_rows(
self,
mock_client: MagicMock,
tmp_path: Path,
) -> None:
"""Non-empty orders are written to order_snapshots."""
mock_client.orders_get_as_df.return_value = pd.DataFrame([
{
"ticket": 10,
"symbol": "EURUSD",
"type": 2,
"volume_current": 0.1,
"price_open": 1.2,
"price_current": 1.1,
"state": 1,
"comment": "",
"magic": 0,
"time_setup": 1700000000,
}
])
output = tmp_path / "obs.db"
update_observability(client=mock_client, output=output)
with sqlite3.connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
assert count == 1
def test_update_observability_symbol_filter_positions(
self,
tmp_path: Path,
) -> None:
"""Symbol filter fetches all positions in one call and filters client-side."""
client = MagicMock()
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
# All positions; only EURUSD matches the filter
client.positions_get_as_df.return_value = pd.DataFrame([
{"ticket": 1, "symbol": "EURUSD", "volume": 0.1, "profit": 0.0},
{"ticket": 2, "symbol": "USDJPY", "volume": 0.2, "profit": 0.0},
])
client.orders_get_as_df.return_value = pd.DataFrame()
client.terminal_info_as_df.return_value = pd.DataFrame()
output = tmp_path / "obs.db"
update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
assert client.positions_get_as_df.call_count == 1
with sqlite3.connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
0
]
assert count == 1
def test_update_observability_symbol_filter_orders(
self,
tmp_path: Path,
) -> None:
"""Symbol filter fetches all orders in one call and filters client-side."""
client = MagicMock()
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
client.positions_get_as_df.return_value = pd.DataFrame()
# All orders; only EURUSD matches the filter
client.orders_get_as_df.return_value = pd.DataFrame([
{"ticket": 10, "symbol": "EURUSD", "volume_current": 0.1},
{"ticket": 11, "symbol": "USDJPY", "volume_current": 0.5},
])
client.terminal_info_as_df.return_value = pd.DataFrame()
output = tmp_path / "obs.db"
update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
assert client.orders_get_as_df.call_count == 1
with sqlite3.connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
assert count == 1
def test_update_observability_symbol_filter_no_symbol_col(
self,
tmp_path: Path,
) -> None:
"""Symbol filter is skipped when positions df has no symbol column."""
client = MagicMock()
client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
# No symbol column in positions — all rows pass through unfiltered
client.positions_get_as_df.return_value = pd.DataFrame([
{"ticket": 1, "volume": 0.1},
])
client.orders_get_as_df.return_value = pd.DataFrame()
client.terminal_info_as_df.return_value = pd.DataFrame()
output = tmp_path / "obs.db"
update_observability(client=client, output=output, symbols=["EURUSD"])
with sqlite3.connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
0
]
assert count == 1
def test_update_observability_account_none_login(
self,
tmp_path: Path,
) -> None:
"""Account row with no login key returns None login for downstream helpers."""
client = MagicMock()
client.account_info_as_df.return_value = pd.DataFrame([{"balance": 10000.0}])
client.positions_get_as_df.return_value = pd.DataFrame()
client.orders_get_as_df.return_value = pd.DataFrame()
client.terminal_info_as_df.return_value = pd.DataFrame()
output = tmp_path / "obs.db"
update_observability(client=client, output=output)
with sqlite3.connect(output) as conn:
row = conn.execute("SELECT login FROM account_snapshots").fetchone()
assert row is not None
assert row[0] is None
def test_update_observability_empty_account_logs_warning(
self,
mock_client: MagicMock,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Empty account_info_as_df logs a warning and does not write account row."""
mock_client.account_info_as_df.return_value = pd.DataFrame()
with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
update_observability(client=mock_client, output=tmp_path / "obs.db")
assert "account_info_as_df returned empty frame" in caplog.text
with sqlite3.connect(tmp_path / "obs.db") as conn:
count = conn.execute("SELECT COUNT(*) FROM account_snapshots").fetchone()[0]
assert count == 0
def test_update_observability_empty_terminal_logs_warning(
self,
mock_client: MagicMock,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Empty terminal_info_as_df logs a warning and does not write terminal row."""
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
update_observability(client=mock_client, output=tmp_path / "obs.db")
assert "terminal_info_as_df returned empty frame" in caplog.text
with sqlite3.connect(tmp_path / "obs.db") as conn:
count = conn.execute("SELECT COUNT(*) FROM terminal_snapshots").fetchone()[
0
]
assert count == 0
def test_update_observability_with_config_opens_and_closes_connection(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""update_observability_with_config manages the MT5 connection lifecycle."""
mock_client = MagicMock()
mock_client.account_info_as_df.return_value = pd.DataFrame()
mock_client.positions_get_as_df.return_value = pd.DataFrame()
mock_client.orders_get_as_df.return_value = pd.DataFrame()
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
update_observability_with_config(output=tmp_path / "obs.db")
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
def test_update_observability_with_config_passes_symbols(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""update_observability_with_config forwards symbols to update_observability."""
mock_client = MagicMock()
mock_client.account_info_as_df.return_value = pd.DataFrame()
mock_client.positions_get_as_df.return_value = pd.DataFrame()
mock_client.orders_get_as_df.return_value = pd.DataFrame()
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
spy = mocker.patch("mt5cli.sdk.update_observability")
update_observability_with_config(
output=tmp_path / "obs.db",
symbols=["EURUSD"],
include_account=False,
)
spy.assert_called_once()
call_kwargs = spy.call_args.kwargs
assert call_kwargs["symbols"] == ["EURUSD"]
assert call_kwargs["include_account"] is False
def test_update_observability_invokes_snapshot_telemetry(
self,
mock_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""update_observability calls record_snapshot_update on the global metrics."""
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_snapshot_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_observability(client=mock_client, output=tmp_path / "obs.db")
mock_metrics.record_snapshot_update.assert_called_once()
def test_update_observability_emits_account_metrics(
self,
mock_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""_snapshot_account emits account gauges via get_metrics."""
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_snapshot_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_observability(client=mock_client, output=tmp_path / "obs.db")
mock_metrics.record_account_state.assert_called_once()
def test_update_observability_emits_terminal_metrics(
self,
mock_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""_snapshot_terminal emits connected/trade gauges via get_metrics."""
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_snapshot_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_observability(client=mock_client, output=tmp_path / "obs.db")
mock_metrics.record_terminal_state.assert_called_once_with(
connected=1.0, trade_allowed=1.0, trade_expert=1.0
)
def test_update_observability_aggregates_same_symbol_positions(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Same-symbol positions are summed before emitting gauges (hedging)."""
mock_client = MagicMock()
mock_client.account_info_as_df.return_value = pd.DataFrame([
{
"login": 1,
"server": "demo",
"balance": 1000.0,
"equity": 1000.0,
"margin": 0.0,
"margin_free": 1000.0,
"margin_level": 0.0,
}
])
mock_client.positions_get_as_df.return_value = pd.DataFrame([
{"ticket": 1, "symbol": "EURUSD", "profit": 10.0, "volume": 0.1},
{"ticket": 2, "symbol": "EURUSD", "profit": -5.0, "volume": 0.2},
{"ticket": 3, "symbol": "GBPUSD", "profit": 3.0, "volume": 0.05},
])
mock_client.orders_get_as_df.return_value = pd.DataFrame()
mock_client.terminal_info_as_df.return_value = pd.DataFrame()
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_snapshot_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_observability(client=mock_client, output=tmp_path / "obs.db")
calls = mock_metrics.record_position_state.call_args_list
# Two EURUSD positions should be collapsed to one call; GBPUSD is one call.
assert len(calls) == 2
by_symbol = {c.kwargs["symbol"]: c.kwargs for c in calls}
assert abs(float(by_symbol["EURUSD"]["profit"]) - 5.0) < 1e-9
assert abs(float(by_symbol["EURUSD"]["volume"]) - 0.3) < 1e-9
assert abs(float(by_symbol["GBPUSD"]["profit"]) - 3.0) < 1e-9
assert abs(float(by_symbol["GBPUSD"]["volume"]) - 0.05) < 1e-9
class TestUpdateHistoryTelemetry:
"""Tests for telemetry hooks in update_history."""
def test_update_history_invokes_history_telemetry(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""update_history wraps write_incremental_datasets with telemetry."""
mock_client = MagicMock()
mock_client.copy_rates_range_as_df.return_value = pd.DataFrame()
mock_client.history_orders_get_as_df.return_value = pd.DataFrame()
mock_client.history_deals_get_as_df.return_value = pd.DataFrame()
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_history_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_history(
client=mock_client,
output=tmp_path / "hist.db",
symbols=["EURUSD"],
)
mock_metrics.record_history_update.assert_called_once_with(dataset="history")
def test_update_history_emits_history_rows(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""update_history calls add_history_rows with the SQLite change delta."""
mock_client = MagicMock()
mock_client.copy_rates_range_as_df.return_value = pd.DataFrame()
mock_client.history_orders_get_as_df.return_value = pd.DataFrame()
mock_client.history_deals_get_as_df.return_value = pd.DataFrame()
mock_metrics = MagicMock()
mock_cm = MagicMock()
mock_cm.__enter__ = MagicMock(return_value=None)
mock_cm.__exit__ = MagicMock(return_value=False)
mock_metrics.record_history_update.return_value = mock_cm
mocker.patch("mt5cli.sdk.get_metrics", return_value=mock_metrics)
update_history(
client=mock_client,
output=tmp_path / "hist.db",
symbols=["EURUSD"],
)
mock_metrics.add_history_rows.assert_called_once_with(0, dataset="history")
+249
View File
@@ -0,0 +1,249 @@
"""Tests for mt5cli.telemetry module."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from mt5cli.telemetry import (
_OTEL_AVAILABLE, # type: ignore[reportPrivateUsage]
_Mt5Metrics, # type: ignore[reportPrivateUsage]
_NoOp, # type: ignore[reportPrivateUsage]
configure_metrics,
enable_otel_metrics,
get_metrics,
)
class TestNoOp:
"""Tests for _NoOp no-op instrument."""
def test_add_is_noop(self) -> None:
"""_NoOp.add accepts amount and optional attributes without error."""
noop = _NoOp()
noop.add(1.0)
noop.add(1.0, {"key": "val"})
def test_set_is_noop(self) -> None:
"""_NoOp.set accepts amount and optional attributes without error."""
noop = _NoOp()
noop.set(2.0)
noop.set(2.0, {"key": "val"})
def test_record_is_noop(self) -> None:
"""_NoOp.record accepts amount and optional attributes without error."""
noop = _NoOp()
noop.record(3.0)
noop.record(3.0, {"key": "val"})
class TestMt5Metrics:
"""Tests for _Mt5Metrics."""
def test_default_instruments_are_noop(self) -> None:
"""Default _Mt5Metrics methods do not raise before configure is called."""
m = _Mt5Metrics()
m.record_account_state(
login="123",
server="demo",
balance=1000.0,
equity=1050.0,
margin=100.0,
margin_free=950.0,
margin_level=1050.0,
)
def test_configure_calls_meter(self) -> None:
"""configure() calls create_histogram, create_counter, create_gauge on meter."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
assert meter.create_histogram.called
assert meter.create_counter.called
assert meter.create_gauge.called
def test_record_history_update_success(self) -> None:
"""record_history_update records duration and timestamp on success."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
with m.record_history_update(dataset="rates"):
pass
m._history_duration.record.assert_called_once() # type: ignore[reportPrivateUsage]
m._last_successful_update.set.assert_called_once() # type: ignore[reportPrivateUsage]
m._history_failures.add.assert_not_called() # type: ignore[reportPrivateUsage]
def test_record_history_update_failure(self) -> None:
"""record_history_update increments failure counter and re-raises on error."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
exc = ValueError("boom")
with (
pytest.raises(ValueError, match="boom"),
m.record_history_update(dataset="rates"),
):
raise exc
m._history_failures.add.assert_called_once_with( # type: ignore[reportPrivateUsage]
1, {"dataset": "rates"}
)
m._history_duration.record.assert_not_called() # type: ignore[reportPrivateUsage]
def test_add_history_rows(self) -> None:
"""add_history_rows increments the rows-written counter."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
m.add_history_rows(42, dataset="rates")
m._history_rows.add.assert_called_once_with( # type: ignore[reportPrivateUsage]
42, {"dataset": "rates"}
)
def test_record_snapshot_update_success(self) -> None:
"""record_snapshot_update records duration on success."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
with m.record_snapshot_update():
pass
m._snapshot_duration.record.assert_called_once() # type: ignore[reportPrivateUsage]
m._snapshot_failures.add.assert_not_called() # type: ignore[reportPrivateUsage]
def test_record_snapshot_update_failure(self) -> None:
"""record_snapshot_update increments failure counter and re-raises on error."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
exc = RuntimeError("snap fail")
with (
pytest.raises(RuntimeError, match="snap fail"),
m.record_snapshot_update(),
):
raise exc
m._snapshot_failures.add.assert_called_once_with(1, {}) # type: ignore[reportPrivateUsage]
m._snapshot_duration.record.assert_not_called() # type: ignore[reportPrivateUsage]
def test_record_position_state(self) -> None:
"""record_position_state emits profit and volume gauges."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
m.record_position_state(
login="42",
server="demo",
symbol="EURUSD",
profit=12.5,
volume=0.01,
)
# Both profit and volume share the same gauge mock via create_gauge.
# Verify that set was called exactly twice (once each).
assert m._position_profit.set.call_count == 2 # type: ignore[reportPrivateUsage]
def test_record_terminal_state(self) -> None:
"""record_terminal_state emits connected, trade_allowed, trade_expert gauges."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
m.record_terminal_state(connected=1.0, trade_allowed=1.0, trade_expert=0.0)
# All three terminal gauges share the same mock; set is called 3 times.
assert m._terminal_connected.set.call_count == 3 # type: ignore[reportPrivateUsage]
def test_record_account_state_after_configure(self) -> None:
"""record_account_state emits all five account gauges."""
meter = MagicMock()
m = _Mt5Metrics()
m.configure(meter)
m.record_account_state(
login="99",
server="live",
balance=5000.0,
equity=5100.0,
margin=200.0,
margin_free=4800.0,
margin_level=2550.0,
)
# All five account gauges share the same gauge mock; set is called 5 times.
assert m._account_balance.set.call_count == 5 # type: ignore[reportPrivateUsage]
def test_record_history_update_noop_before_configure(self) -> None:
"""record_history_update works without configure (no-op instruments)."""
m = _Mt5Metrics()
with m.record_history_update(dataset="ticks"):
pass
def test_record_snapshot_update_noop_before_configure(self) -> None:
"""record_snapshot_update works without configure (no-op instruments)."""
m = _Mt5Metrics()
with m.record_snapshot_update():
pass
class TestConfigureMetrics:
"""Tests for configure_metrics and get_metrics."""
def test_configure_metrics_updates_global(self) -> None:
"""configure_metrics wires up the global singleton."""
meter = MagicMock()
configure_metrics(meter)
assert get_metrics() is get_metrics()
def test_get_metrics_returns_mt5metrics(self) -> None:
"""get_metrics returns the global _Mt5Metrics instance."""
assert isinstance(get_metrics(), _Mt5Metrics)
class TestEnableOtelMetrics:
"""Tests for enable_otel_metrics."""
def test_enable_raises_when_unavailable(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""enable_otel_metrics raises ImportError when OTel is not installed."""
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", False)
with pytest.raises(ImportError, match="opentelemetry-api"):
enable_otel_metrics()
def test_enable_configures_sdk_pipeline_with_readers(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""enable_otel_metrics wires up an SDK MeterProvider with supplied readers."""
mock_mod = MagicMock()
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
reader = InMemoryMetricReader()
enable_otel_metrics("my-service", readers=[reader])
mock_mod.set_meter_provider.assert_called_once()
provider = mock_mod.set_meter_provider.call_args[0][0]
assert provider.get_meter("my-service") is not None
def test_enable_default_readers_uses_otlp(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""enable_otel_metrics with no readers creates an OTLP pipeline by default."""
mock_mod = MagicMock()
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
monkeypatch.setattr("mt5cli.telemetry._OtelOTLPExporter", MagicMock())
enable_otel_metrics("my-service")
mock_mod.set_meter_provider.assert_called_once()
def test_enable_default_readers_raises_when_otlp_missing(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""enable_otel_metrics raises ImportError when the OTLP exporter is missing."""
mock_mod = MagicMock()
monkeypatch.setattr("mt5cli.telemetry._OTEL_AVAILABLE", True)
monkeypatch.setattr("mt5cli.telemetry._otel_metrics_mod", mock_mod)
monkeypatch.setattr("mt5cli.telemetry._OtelOTLPExporter", None)
with pytest.raises(ImportError, match="opentelemetry-exporter-otlp-proto-http"):
enable_otel_metrics()
def test_otel_available_flag_is_bool(self) -> None:
"""_OTEL_AVAILABLE is a boolean."""
assert isinstance(_OTEL_AVAILABLE, bool)
Generated
+123 -3
View File
@@ -223,6 +223,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" },
]
[[package]]
name = "googleapis-common-protos"
version = "1.75.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/b5/c8/f439cffde755cffa462bfbb156278fa6f9d09119719af9814b858fd4f81f/googleapis_common_protos-1.75.0.tar.gz", hash = "sha256:53a062ff3c32552fbd62c11fe23768b78e4ddf0494d5e5fd97d3f4689c75fbbd", size = 151035, upload-time = "2026-05-07T08:04:49.423Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e7/c8/e2645aa8ed02fd4c7a2f59d68783b65b1f3cbdfe39a6308e156509d1fee8/googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed", size = 300631, upload-time = "2026-05-07T08:03:30.345Z" },
]
[[package]]
name = "griffelib"
version = "2.0.2"
@@ -358,7 +370,7 @@ name = "metatrader5"
version = "5.0.5640"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "numpy" },
{ name = "numpy", marker = "sys_platform == 'win32'" },
]
wheels = [
{ url = "https://files.pythonhosted.org/packages/ef/a0/3b764c6743ef601ff12f7d8d62ca5768eb25e90d3758ac7e1d08e667af85/metatrader5-5.0.5640-cp311-cp311-win_amd64.whl", hash = "sha256:4057255f2d63138a3ea1a5d492715038a71d3177cad793fca97a5c58771b4eb1", size = 48091, upload-time = "2026-02-20T23:31:12.289Z" },
@@ -487,7 +499,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "1.0.3"
version = "1.1.0"
source = { editable = "." }
dependencies = [
{ name = "click" },
@@ -496,6 +508,11 @@ dependencies = [
]
[package.optional-dependencies]
otel = [
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp-proto-http" },
{ name = "opentelemetry-sdk" },
]
parquet = [
{ name = "pyarrow" },
]
@@ -505,6 +522,8 @@ dev = [
{ name = "mkdocs" },
{ name = "mkdocs-material" },
{ name = "mkdocstrings", extra = ["python"] },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-sdk" },
{ name = "pandas-stubs" },
{ name = "pyarrow" },
{ name = "pymdown-extensions" },
@@ -518,17 +537,22 @@ dev = [
[package.metadata]
requires-dist = [
{ name = "click", specifier = ">=8.1.0" },
{ name = "opentelemetry-api", marker = "extra == 'otel'" },
{ name = "opentelemetry-exporter-otlp-proto-http", marker = "extra == 'otel'" },
{ name = "opentelemetry-sdk", marker = "extra == 'otel'" },
{ name = "pdmt5", specifier = ">=1.0.0" },
{ name = "pyarrow", marker = "extra == 'parquet'", specifier = ">=19.0.0" },
{ name = "typer", specifier = ">=0.15.0" },
]
provides-extras = ["parquet"]
provides-extras = ["parquet", "otel"]
[package.metadata.requires-dev]
dev = [
{ name = "mkdocs", specifier = ">=1.6.1" },
{ name = "mkdocs-material", specifier = ">=9.7.6" },
{ name = "mkdocstrings", extras = ["python"], specifier = ">=1.0.4" },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-sdk" },
{ name = "pandas-stubs", specifier = ">=2.2.3.250527" },
{ name = "pyarrow", specifier = ">=19.0.0" },
{ name = "pymdown-extensions", specifier = ">=10.21.2" },
@@ -606,6 +630,87 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/57/a7/b35835e278c18b85206834b3aa3abe68e77a98769c59233d1f6300284781/numpy-2.4.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:4b42639cdde6d24e732ff823a3fa5b701d8acad89c4142bc1d0bd6dc85200ba5", size = 12504685, upload-time = "2026-03-09T07:58:50.525Z" },
]
[[package]]
name = "opentelemetry-api"
version = "1.43.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/ae/cc/e4c9584181f86494df0f6bdec1a4f3280c50db44704dc2a407e994fc87bb/opentelemetry_api-1.43.0.tar.gz", hash = "sha256:107d0d03857ea8fc7c5fcbbbd83f800c281f0d560553d61c1d675fccfd1761c1", size = 73476, upload-time = "2026-06-24T15:19:55.323Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/17/83/6dba32b85f31868400440dc7ad2ca1eab94cbbf3a7b0459ed39f8311a9e2/opentelemetry_api-1.43.0-py3-none-any.whl", hash = "sha256:20acf45e9b21851926835292e4045d290acade1edd2ff3de86d2f069687ba1fd", size = 61912, upload-time = "2026-06-24T15:19:35.434Z" },
]
[[package]]
name = "opentelemetry-exporter-otlp-proto-common"
version = "1.43.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-proto" },
]
sdist = { url = "https://files.pythonhosted.org/packages/55/c1/e8098490ab15abf116dcaf9fa89ededcb35547c7d08d4b5a62f573dc1e63/opentelemetry_exporter_otlp_proto_common-1.43.0.tar.gz", hash = "sha256:c4e32ba6d6b13bdb2b8f6764c4fd28d00192826561aa04f6d14eedfce7ac076f", size = 20197, upload-time = "2026-06-24T15:20:00.247Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d0/b2/41ebc74ae1d5859901f1b69305de58724bf043381103d6ef413521cbc35a/opentelemetry_exporter_otlp_proto_common-1.43.0-py3-none-any.whl", hash = "sha256:123c3f9cc87218562490c63b36f497bf3a722faf174a515d1443f31ababa6264", size = 17048, upload-time = "2026-06-24T15:19:41.264Z" },
]
[[package]]
name = "opentelemetry-exporter-otlp-proto-http"
version = "1.43.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "googleapis-common-protos" },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp-proto-common" },
{ name = "opentelemetry-proto" },
{ name = "opentelemetry-sdk" },
{ name = "requests" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fc/92/0b9f56412483a8891d4843890294796c9df8ab42417bd9bad8035d840cb3/opentelemetry_exporter_otlp_proto_http-1.43.0.tar.gz", hash = "sha256:fa8a42bb7d00ee5391f4c0b04d8e6a46c03caa437903296ab73a81dc11ba118f", size = 25406, upload-time = "2026-06-24T15:20:01.515Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b3/20/b685ed7af2e17c29ffc8af56f1fa8bc2033258fc30fb0d2b722f49d13ba0/opentelemetry_exporter_otlp_proto_http-1.43.0-py3-none-any.whl", hash = "sha256:647f603aa8efdbdb4dbff842e0729d0406a6fff26b295a72d3d60e7d963b2610", size = 21795, upload-time = "2026-06-24T15:19:43.164Z" },
]
[[package]]
name = "opentelemetry-proto"
version = "1.43.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e0/b9/d357faefb40bda1d4799913e6af611171ff22a2dedcb93576bc92242d056/opentelemetry_proto-1.43.0.tar.gz", hash = "sha256:224778df17e1f3fafeaaa21d874236ca5f6ffc2f86e0899298ec7351aac27924", size = 46481, upload-time = "2026-06-24T15:20:07.625Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ed/a7/3e5308cf548b8f72529c7db1afdb3a404211982376a12927fd7759f77bf3/opentelemetry_proto-1.43.0-py3-none-any.whl", hash = "sha256:c58f1f7ef84bc7dc2834016c0c37fe0081dde7ca9f6339be1970fbf9cdaaa90d", size = 72489, upload-time = "2026-06-24T15:19:51.164Z" },
]
[[package]]
name = "opentelemetry-sdk"
version = "1.43.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-api" },
{ name = "opentelemetry-semantic-conventions" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3e/eb/5041074274ac0956b03637cc039d434569112468e875eddfcc9a0674ce06/opentelemetry_sdk-1.43.0.tar.gz", hash = "sha256:d8187c81c162df9913e4003dd6485f7390d9a24fc17026ec7387b8b8218b08e9", size = 254744, upload-time = "2026-06-24T15:20:08.467Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/49/e3/b17be23af124201c9f52eececd4cc8ddfed1597d37b4ee771895d325805c/opentelemetry_sdk-1.43.0-py3-none-any.whl", hash = "sha256:d1323a547c1ce69d6a069a17a44b7da82bb8b332051ecb074041f87642c86823", size = 178852, upload-time = "2026-06-24T15:19:52.169Z" },
]
[[package]]
name = "opentelemetry-semantic-conventions"
version = "0.64b0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-api" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/5a/30/5f26df29509eccd86b99b481ac9ffa39da49ba9577cc69071c552ae30447/opentelemetry_semantic_conventions-0.64b0.tar.gz", hash = "sha256:72f76fb2d1582d9d033dd1fcd84532e961e6ff3d90d24ba6fabc72975a83864c", size = 148340, upload-time = "2026-06-24T15:20:09.267Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f2/ca/23ba87a221b574a7c5a99d48849d80bfe8b047624681357e2b002e566187/opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl", hash = "sha256:ea77e85e354b8f604ddbe5f3d9135216f982fa4d77e5859ac30f6d8a50505aa6", size = 203713, upload-time = "2026-06-24T15:19:53.339Z" },
]
[[package]]
name = "packaging"
version = "26.0"
@@ -721,6 +826,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
]
[[package]]
name = "protobuf"
version = "7.35.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/da/01/9ef0afd7999eb9badb3a768b4aedd78c86d4c65cfaf1958ab276199e76b4/protobuf-7.35.1.tar.gz", hash = "sha256:ce115a26fe0c39a2c29973d914d327e516a6455464489fe3cd1e51a1b354f81a", size = 458717, upload-time = "2026-06-11T21:55:40.257Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/10/03/8aeeb7458d22546bf64b5250ca1daeb5ff757d900e8e4a7476c6f0db843e/protobuf-7.35.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:24f857477359a85c0c235261b8ba905fd51b2562f4a64ca1df5473f29850cbf6", size = 433226, upload-time = "2026-06-11T21:55:31.719Z" },
{ url = "https://files.pythonhosted.org/packages/37/4b/dfb89eb0e652a1ff073c39a59fb5e3a83cfe9b57a2c83fa6d78270101767/protobuf-7.35.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:11d6b0ec246892d85215b0a13ca6e0233cf5284b68f0ac02646427f4ff88a799", size = 328847, upload-time = "2026-06-11T21:55:34.035Z" },
{ url = "https://files.pythonhosted.org/packages/0f/58/dc12f2cd484951524af6e3382c785869b9b3fb5e52ee95ae23add53ee8f9/protobuf-7.35.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:b73f9489a4b8b1c9cb1f8ed951c736392592edb24b9d6819f36d2e10b171d5b4", size = 344030, upload-time = "2026-06-11T21:55:34.941Z" },
{ url = "https://files.pythonhosted.org/packages/e4/be/5b3cfe508bfab6761414ff944e3366eb13be4fd71efcd69450f89ba39f43/protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:74758715c53d7158fb76caf4f0cfdacc5329a4b1bb994f865d6cf302d413a1c4", size = 327130, upload-time = "2026-06-11T21:55:35.921Z" },
{ url = "https://files.pythonhosted.org/packages/d8/bc/6d6c7ba8709c85f8f2c390b2b118d6fb08a783676a572271851bf45a7d22/protobuf-7.35.1-cp310-abi3-win32.whl", hash = "sha256:353652e4efd0bca5b5fc2656abf8307ef351f0cf938c9eba09f0e09c20a25c30", size = 428945, upload-time = "2026-06-11T21:55:37.034Z" },
{ url = "https://files.pythonhosted.org/packages/0a/19/8d0cb6f20a1ef7b18f1c8986ad5783f22f84cce39c6ce9a6e645ea55192e/protobuf-7.35.1-cp310-abi3-win_amd64.whl", hash = "sha256:230a75ddfc2de4806e56696ce9640c1cdfdb6543b7cfce98d42a4c0a0e7bdb87", size = 439996, upload-time = "2026-06-11T21:55:38.123Z" },
{ url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
]
[[package]]
name = "pyarrow"
version = "23.0.1"