gavindiaz/mt5cli
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Grafana-ready SQLite observability (#86)

Browse Source 
* feat: add Grafana-ready SQLite observability (#79, #80, #81)

New `mt5cli/grafana.py` module with idempotent DDL helpers:
- `create_snapshot_tables` — five SQLite tables for time-series account,
  position, order, terminal, and run-status snapshots
- `create_grafana_views` — 13 `grafana_*` views with integer epoch-second
  `time` columns; missing source tables emit warnings and are skipped
- `create_grafana_indexes` — 9 performance indexes guarded by column checks
- `ensure_grafana_schema` — convenience wrapper calling all three above
- Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`,
  `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run`

New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`:
- `update_observability` — appends a timestamped snapshot to a SQLite db
  using an already-connected `Mt5DataClient`; never places orders
- `update_observability_with_config` — standalone wrapper that opens and
  closes the MT5 connection automatically

New CLI commands (Collection panel):
- `grafana-schema` — idempotent schema setup, no MT5 connection required
- `snapshot` — append account/position/order/terminal rows; supports
  `--symbol`, `--with-account/--no-account`, and equivalent flags

All public modules maintain 100 % branch coverage; 968 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply mdformat to docs after grafana observability additions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: bump version to 1.1.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address PR #86 review feedback

- Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX)
- Replace O(N) per-symbol API calls with single call + client-side filter
- Eliminate double create_snapshot_tables when with_grafana_schema=True
- Move grafana imports to module level in sdk.py (remove PLC0415 noqa)
- Default with_grafana_schema to False (run grafana-schema once for setup)
- Fix README position example to use snapshot_runs for latest snapshot
- Update tests to reflect new behavior and correct patch targets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply ruff formatting and sync lock file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address owner review feedback on PR #86

- Filter grafana_*_snapshots views to only expose rows from successful
  runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot
  visibility gap raised in PRRT_kwDORzI_286MvDJ6
- Add issubset column guards for snapshot table index creation, consistent
  with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1)
- Fix README example queries: views expose 'time' not 'observed_at'
  (PRRT_kwDORzI_286MvUxw)
- Correct public-contract.md default for with_grafana_schema (False, not
  True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: add run_id to snapshot schema and fix stale-positions README query

Replace second-level observed_at as the join key between snapshot_runs
and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs
in the same second now get distinct run_ids, preventing view duplication
and cross-contamination from a failed run. Update README example to use
snapshot_runs for latest-snapshot lookup so zero-position runs return an
empty result instead of stale rows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: format markdown tables

Align table column widths in README and public-contract documentation.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: close sqlite connections deterministically

* fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats

grafana_realized_pnl now requires the entry column and filters to
close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl
and grafana_trade_stats. grafana_trade_stats now requires the time column
and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana
view contract (integer epoch-second time column throughout).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots

orders_get_as_df() returns datetime-converted columns by default, so
time_setup is a pd.Timestamp in normal use. Passing it directly to
sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper
that converts datetime.datetime subclasses (including pd.Timestamp) and
raw int/float values to integer epoch seconds, returning None for other
types. Regression tests cover all four input paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: pre-drop all grafana_* views at start of create_grafana_views

Previously, a builder that skipped due to a missing source table or column
would not drop the view it owned, leaving stale views referencing gone
tables. Now create_grafana_views drops all 13 known grafana_* views before
calling any builder, so a schema refresh always removes views whose source
has disappeared. Regression test covers the create → drop-source → refresh
cycle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: expose run_id in snapshot views and drop time from summary views

Grafana snapshot views now expose run_id so latest-state queries can use
MAX(run_id) instead of the ambiguous second-level MAX(observed_at).
grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column
and are reclassified as static summary views; their all-time aggregates
are not filterable by Grafana time-range selectors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: guard snapshot views against missing run_id and fix README view docs

_build_snapshot_view now skips with a warning when the underlying
snapshot table exists but lacks a run_id column, preventing a broken
view that fails at query time. Adds a regression test for that path.

README Grafana section now qualifies that grafana_realized_pnl and
grafana_trade_stats are static summary views (no time column) and splits
the view table to match docs/api/public-contract.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Daichi Narushima
2026-06-28 07:55:26 +09:00
committed by GitHub
parent 4bc36d09d2
commit d27da02f3f
14 changed files with 2494 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+100
View File
@@ -185,6 +185,8 @@ python -m mt5cli -o account.csv account-info
| `order-send` | Send a raw trade request to the trade server (`--yes` required; expert path) |
| `close-positions` | Close open positions by `--symbol` or `--ticket` (`--yes` required for live; `--dry-run` available) |
| `collect-history` | Collect rates, history-orders, and history-deals for one or more symbols into a single SQLite database (ticks opt-in via `--dataset ticks`) |
| `grafana-schema` | Create or refresh Grafana-ready views and indexes in an existing SQLite database (idempotent, no MT5 connection) |
| `snapshot` | Snapshot current account, position, order, and terminal state into SQLite for live Grafana dashboards |
Use `order-check` to validate a request payload before running `order-send --yes`.
`close-positions` is the safer high-level alternative that builds correct close
@@ -204,6 +206,104 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
### Grafana-ready SQLite dashboards
mt5cli can prepare a SQLite database for use as a Grafana datasource (via the [SQLite plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/) or similar). Most `grafana_*` views expose an integer epoch-second `time` column for use in Grafana time-series panels. Two views (`grafana_realized_pnl`, `grafana_trade_stats`) are static symbol-level summaries with no `time` column — use them in table or stat panels.
#### Prepare the schema (idempotent, no MT5 connection needed)
```bash
mt5cli -o history.db grafana-schema
```
This creates snapshot tables (`account_snapshots`, `position_snapshots`, `order_snapshots`, `terminal_snapshots`, `snapshot_runs`) and all `grafana_*` views and indexes in the SQLite database. Safe to run repeatedly — all operations are idempotent.
#### Snapshot current account state
```bash
mt5cli -o history.db snapshot \
--symbol JP225 --symbol HK50 --symbol NL25 \
--with-account --with-positions --with-orders --with-terminal \
--with-grafana-schema
```
Appends one timestamped row per data type. Never places orders or modifies trading state. Run periodically (e.g. from a cron job or a loop) to build a time-series account history.
#### SDK usage
```python
from pdmt5 import Mt5DataClient, Mt5Config
from mt5cli import update_observability, update_observability_with_config
# Reuse an already-connected client
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
update_observability(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"], # optional position/order filter
include_account=True,
include_positions=True,
include_orders=True,
include_terminal=True,
with_grafana_schema=True,
)
finally:
client.shutdown()
# Standalone wrapper that opens/closes MT5 automatically
update_observability_with_config(
output="history.db",
config=Mt5Config(login=12345),
)
```
#### Available Grafana views
**Time-series views** (integer epoch-second `time` column; snapshot views also expose `run_id`):
| View | Source | Description |
| ---------------------------- | -------------------- | ---------------------------------------------------------- |
| `grafana_rates` | `rates` | OHLCV bars with integer epoch `time` |
| `grafana_ticks` | `ticks` | Tick data with integer epoch `time` |
| `grafana_history_deals` | `history_deals` | All deals with epoch `time` |
| `grafana_history_orders` | `history_orders` | All historical orders; adds epoch `time` from `time_setup` |
| `grafana_trade_deals` | `history_deals` | Trade deals only (`type IN (0,1)`) |
| `grafana_cash_events` | `history_deals` | Non-trade deals (deposits, dividends, etc.) |
| `grafana_symbol_pnl` | `history_deals` | Per-close-deal profit/loss per symbol |
| `grafana_account_snapshots` | `account_snapshots` | Account balance/equity/margin time series |
| `grafana_position_snapshots` | `position_snapshots` | Open position snapshots over time |
| `grafana_order_snapshots` | `order_snapshots` | Active order snapshots over time |
| `grafana_terminal_snapshots` | `terminal_snapshots` | Terminal connectivity snapshots |
**Static summary views** (no `time` column; use in table or stat panels, not time-series):
| View | Source | Description |
| ---------------------- | --------------- | ------------------------------------- |
| `grafana_realized_pnl` | `history_deals` | Cumulative realized PnL per symbol |
| `grafana_trade_stats` | `history_deals` | Win/loss counts and profit per symbol |
#### Example Grafana queries
```sql
-- Equity curve over time
SELECT time, equity FROM grafana_account_snapshots ORDER BY time;
-- Rolling balance by account login
SELECT time, login, balance FROM grafana_account_snapshots
WHERE login = $login ORDER BY time;
-- Open positions at latest successful snapshot
SELECT symbol, volume, profit FROM grafana_position_snapshots
WHERE run_id = (SELECT MAX(run_id) FROM snapshot_runs WHERE status = 'ok');
-- Realized PnL by symbol
SELECT symbol, total_profit FROM grafana_trade_stats ORDER BY total_profit DESC;
```
> **Note**: OpenTelemetry integration is intentionally not part of this release and is tracked separately.
### Incremental history SDK
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges: