feat: add Grafana-ready SQLite observability (#86)
* feat: add Grafana-ready SQLite observability (#79, #80, #81) New `mt5cli/grafana.py` module with idempotent DDL helpers: - `create_snapshot_tables` — five SQLite tables for time-series account, position, order, terminal, and run-status snapshots - `create_grafana_views` — 13 `grafana_*` views with integer epoch-second `time` columns; missing source tables emit warnings and are skipped - `create_grafana_indexes` — 9 performance indexes guarded by column checks - `ensure_grafana_schema` — convenience wrapper calling all three above - Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run` New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`: - `update_observability` — appends a timestamped snapshot to a SQLite db using an already-connected `Mt5DataClient`; never places orders - `update_observability_with_config` — standalone wrapper that opens and closes the MT5 connection automatically New CLI commands (Collection panel): - `grafana-schema` — idempotent schema setup, no MT5 connection required - `snapshot` — append account/position/order/terminal rows; supports `--symbol`, `--with-account/--no-account`, and equivalent flags All public modules maintain 100 % branch coverage; 968 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply mdformat to docs after grafana observability additions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: bump version to 1.1.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address PR #86 review feedback - Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX) - Replace O(N) per-symbol API calls with single call + client-side filter - Eliminate double create_snapshot_tables when with_grafana_schema=True - Move grafana imports to module level in sdk.py (remove PLC0415 noqa) - Default with_grafana_schema to False (run grafana-schema once for setup) - Fix README position example to use snapshot_runs for latest snapshot - Update tests to reflect new behavior and correct patch targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply ruff formatting and sync lock file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address owner review feedback on PR #86 - Filter grafana_*_snapshots views to only expose rows from successful runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot visibility gap raised in PRRT_kwDORzI_286MvDJ6 - Add issubset column guards for snapshot table index creation, consistent with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1) - Fix README example queries: views expose 'time' not 'observed_at' (PRRT_kwDORzI_286MvUxw) - Correct public-contract.md default for with_grafana_schema (False, not True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add run_id to snapshot schema and fix stale-positions README query Replace second-level observed_at as the join key between snapshot_runs and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs in the same second now get distinct run_ids, preventing view duplication and cross-contamination from a failed run. Update README example to use snapshot_runs for latest-snapshot lookup so zero-position runs return an empty result instead of stale rows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: format markdown tables Align table column widths in README and public-contract documentation. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: close sqlite connections deterministically * fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats grafana_realized_pnl now requires the entry column and filters to close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl and grafana_trade_stats. grafana_trade_stats now requires the time column and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana view contract (integer epoch-second time column throughout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots orders_get_as_df() returns datetime-converted columns by default, so time_setup is a pd.Timestamp in normal use. Passing it directly to sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper that converts datetime.datetime subclasses (including pd.Timestamp) and raw int/float values to integer epoch seconds, returning None for other types. Regression tests cover all four input paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: pre-drop all grafana_* views at start of create_grafana_views Previously, a builder that skipped due to a missing source table or column would not drop the view it owned, leaving stale views referencing gone tables. Now create_grafana_views drops all 13 known grafana_* views before calling any builder, so a schema refresh always removes views whose source has disappeared. Regression test covers the create → drop-source → refresh cycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: expose run_id in snapshot views and drop time from summary views Grafana snapshot views now expose run_id so latest-state queries can use MAX(run_id) instead of the ambiguous second-level MAX(observed_at). grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column and are reclassified as static summary views; their all-time aggregates are not filterable by Grafana time-range selectors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: guard snapshot views against missing run_id and fix README view docs _build_snapshot_view now skips with a warning when the underlying snapshot table exists but lacks a run_id column, preventing a broken view that fails at query time. Adds a regression test for that path. README Grafana section now qualifies that grafana_realized_pnl and grafana_trade_stats are static summary views (no time column) and splits the view table to match docs/api/public-contract.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Daichi Narushima
@@ -124,6 +124,63 @@ sending requests. Failed, malformed, or unknown broker retcodes are fail-closed
|
||||
and returned as `status="failed"` with normalized `request` / `response` details;
|
||||
`dry_run=True` never calls `ensure_symbol_selected()` or `order_send()`.
|
||||
|
||||
### Grafana observability (SQLite read model)
|
||||
|
||||
These helpers prepare a SQLite database as a Grafana datasource. All DDL is
|
||||
idempotent (`CREATE TABLE IF NOT EXISTS`, `DROP VIEW IF EXISTS` + `CREATE
|
||||
VIEW`, `CREATE INDEX IF NOT EXISTS`). Missing source tables are skipped with a
|
||||
warning rather than raising an error.
|
||||
|
||||
| Symbol | Role |
|
||||
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `update_observability` | Append one timestamped snapshot row per data type; accepts an already-connected `Mt5DataClient` |
|
||||
| `update_observability_with_config` | Standalone wrapper: opens/closes MT5 connection automatically around `update_observability` |
|
||||
|
||||
Both functions write to the SQLite path given by `output=`. The optional
|
||||
`symbols` parameter filters `positions_get` / `orders_get` by symbol.
|
||||
`with_grafana_schema=False` (default) skips Grafana view/index setup; run
|
||||
`grafana-schema` once to set up the schema, then call `snapshot` repeatedly
|
||||
without this flag.
|
||||
|
||||
**Snapshot tables** (created by `create_snapshot_tables` in `mt5cli.grafana`):
|
||||
|
||||
| Table | Content |
|
||||
| -------------------- | ----------------------------------------- |
|
||||
| `account_snapshots` | Balance, equity, margin, free-margin, P&L |
|
||||
| `position_snapshots` | Open positions: symbol, volume, profit, … |
|
||||
| `order_snapshots` | Active orders: symbol, type, price, … |
|
||||
| `terminal_snapshots` | Terminal connectivity and build info |
|
||||
| `snapshot_runs` | Per-run status (`ok` / `error`) timestamp |
|
||||
|
||||
**Grafana time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
|
||||
|
||||
| View | Source |
|
||||
| ---------------------------- | -------------------------------- |
|
||||
| `grafana_rates` | `rates` table |
|
||||
| `grafana_ticks` | `ticks` table |
|
||||
| `grafana_history_deals` | `history_deals` |
|
||||
| `grafana_history_orders` | `history_orders` |
|
||||
| `grafana_trade_deals` | `history_deals` trade types only |
|
||||
| `grafana_cash_events` | `history_deals` non-trade events |
|
||||
| `grafana_symbol_pnl` | Per-close-deal P&L per symbol |
|
||||
| `grafana_account_snapshots` | `account_snapshots` |
|
||||
| `grafana_position_snapshots` | `position_snapshots` |
|
||||
| `grafana_order_snapshots` | `order_snapshots` |
|
||||
| `grafana_terminal_snapshots` | `terminal_snapshots` |
|
||||
|
||||
**Grafana static summary views** (no `time` column; use for table/stat panels, not time-series):
|
||||
|
||||
| View | Source |
|
||||
| ---------------------- | ------------------------------------- |
|
||||
| `grafana_realized_pnl` | Cumulative realized PnL per symbol |
|
||||
| `grafana_trade_stats` | Win/loss counts and profit per symbol |
|
||||
|
||||
Lower-level helpers (`ensure_grafana_schema`, `create_grafana_views`,
|
||||
`create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`,
|
||||
`insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`,
|
||||
`insert_terminal_snapshot`, `record_snapshot_run`) are available directly from
|
||||
`mt5cli.grafana` and are not part of the package-root stable surface.
|
||||
|
||||
### Errors
|
||||
|
||||
| Symbol | Role |
|
||||
@@ -135,14 +192,15 @@ and returned as `status="failed"` with normalized `request` / `response` details
|
||||
Lower-level helpers are available from their owning modules and are not part
|
||||
of the package-root stable surface. Import them directly when needed:
|
||||
|
||||
| Module | Examples |
|
||||
| ------------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| `mt5cli.history` | `resolve_rate_view_name`, `resolve_rate_tables`, `load_rate_data`, `build_rate_view_name` |
|
||||
| `mt5cli.sdk` | `copy_rates_from`, `copy_ticks_from`, `account_info`, `symbols`, `mt5_summary`, `latest_rates` |
|
||||
| `mt5cli.schemas` | `DataKind`, `normalize_dataframe`, `validate_schema`, `DEDUP_KEYS` |
|
||||
| `mt5cli.utils` | `Dataset`, `IfExists`, `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
|
||||
| `mt5cli.converters` | `normalize_symbol`, `ensure_utc`, `parse_date_range`, `granularity_name` |
|
||||
| `mt5cli.exceptions` | `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` |
|
||||
| Module | Examples |
|
||||
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `mt5cli.grafana` | `ensure_grafana_schema`, `create_grafana_views`, `create_grafana_indexes`, `create_snapshot_tables`, `start_snapshot_run`, `insert_account_snapshot`, `record_snapshot_run` |
|
||||
| `mt5cli.history` | `resolve_rate_view_name`, `resolve_rate_tables`, `load_rate_data`, `build_rate_view_name` |
|
||||
| `mt5cli.sdk` | `copy_rates_from`, `copy_ticks_from`, `account_info`, `symbols`, `mt5_summary`, `latest_rates` |
|
||||
| `mt5cli.schemas` | `DataKind`, `normalize_dataframe`, `validate_schema`, `DEDUP_KEYS` |
|
||||
| `mt5cli.utils` | `Dataset`, `IfExists`, `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
|
||||
| `mt5cli.converters` | `normalize_symbol`, `ensure_utc`, `parse_date_range`, `granularity_name` |
|
||||
| `mt5cli.exceptions` | `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` |
|
||||
|
||||
## CLI commands
|
||||
|
||||
@@ -155,6 +213,15 @@ The Typer application in `mt5cli.cli` exposes file-export commands documented in
|
||||
- Delegate to the same Python APIs described here; they are not duplicated
|
||||
business logic.
|
||||
|
||||
`grafana-schema` initializes Grafana views, indexes, and snapshot tables in the
|
||||
target SQLite database without connecting to MT5. It is idempotent and safe to
|
||||
run repeatedly.
|
||||
|
||||
`snapshot` appends one timestamped row per enabled data type
|
||||
(`--with-account`, `--with-positions`, `--with-orders`, `--with-terminal`) and
|
||||
never places orders or modifies trading state. Both commands require
|
||||
`-o/--output` to point at a `.db` / SQLite file.
|
||||
|
||||
`order-send` is the expert raw-request path; it requires `--yes` and a fully
|
||||
constructed request payload. `close-positions` is the safer high-level helper
|
||||
that closes open positions by `--symbol` or `--ticket` using
|
||||
|
||||
Reference in New Issue
Block a user