Daichi Narushima
1ffac45d57
* feat: Grafana copy publishing, dashboard examples, and optional OTel metrics Implements three observability improvements: #82 — publish_grafana_copy(): Uses SQLite online backup API (WAL-safe) to atomically publish a consistent read-only copy beside the target. Adds --publish-copy option to grafana-schema and snapshot CLI commands. #83 — examples/grafana/: Minimal working Grafana setup with docker-compose, provisioning datasource/dashboard YAML, and three dashboard JSON files (mt5cli-overview, mt5cli-trades, mt5cli-market). All queries use grafana_* views; no credentials or private paths included. #84 — mt5cli/telemetry.py: Optional OTel metrics behind mt5cli[otel] extra. Base install is unaffected. Adds _Mt5Metrics singleton (no-op until configure_metrics() is called), wraps update_history() and update_observability() with record_history_update / record_snapshot_update context managers, and emits account/position gauges from snapshots. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace ambiguous multiplication sign in comment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: normalize markdown formatting in grafana README Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve file mode on Grafana copy and fix unsupported time macro - publish_grafana_copy: chmod temp file to match the existing target's permissions (or 0o644 when no prior target exists) before atomic replace, so Grafana running as a different OS user (e.g. UID 472 in Docker) can read the published database - mt5cli-market.json: replace unsupported \$__timeFilter(time) with the epoch-based filter supported by frser-sqlite-datasource: "time" >= \$__from / 1000 AND "time" < \$__to / 1000 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: skip Windows-incompatible mode test, rename compose file to compose.yaml - Skip test_overwrite_preserves_existing_target_mode on win32 since Windows chmod does not preserve Unix group/other permission bits - Simplify test_fresh_target_has_readable_permissions to check owner read bit only (portable across platforms) - Rename docker-compose.yml -> compose.yaml (modern Compose convention) - Update README and test reference to match new filename Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: rename *.yaml to *.yml in examples/grafana Renames compose.yaml, mt5cli-sqlite.yaml, and mt5cli.yaml to .yml; updates README and test references accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: format Grafana dashboards and expand qa script to include JSON - Update qa.sh prettier pattern to format JSON files alongside markdown - Reformat Grafana dashboard JSONs with consistent spacing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address owner review comments before merge - qa.sh: fix Prettier glob from `{,d,json}` to `{md,json}` so Markdown files are actually formatted by local QA (P2) - compose.yml: add GF_INSTALL_PLUGINS env var so the frser-sqlite-datasource plugin is installed at container start (P1) - telemetry.py: replace no-op get_meter() call with a real SDK MeterProvider pipeline; add optional `readers` kwarg so callers can inject custom readers (e.g. InMemoryMetricReader in tests) without needing the OTLP package (P1) - sdk.py: aggregate profit and volume by symbol before emitting gauge values so hedging accounts with multiple same-symbol positions emit one point per symbol instead of overwriting with each row (P2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: emit mt5_history_update_rows_total via conn.total_changes delta The counter was registered but never incremented, making the advertised history-update throughput metric permanently zero. Add add_history_rows() to _Mt5Metrics and call it in update_history() using the SQLite total_changes delta measured around write_incremental_datasets(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address three owner review comments - compose.yml: replace soft fallback with :? error expansion so Compose refuses to start when MT5CLI_DB_PATH is unset or empty (P1) - README.md: tell native Windows users to copy only the datasource provisioning file; the dashboards yml contains a Docker-specific path that is invalid on Windows (P2) - telemetry.py / sdk.py: emit mt5_terminal_connected, mt5_terminal_trade_allowed, and mt5_terminal_trade_expert gauges via a new record_terminal_state() method called from _snapshot_terminal(), completing the connection-status metric surface from issue #84 (P2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add snapshot freshness panel and win-rate column to dashboards - mt5cli-overview.json: add a full-width "Last Snapshot" stat panel (dateTimeFromNow unit) below the account stats, querying MAX(time)*1000 from grafana_account_snapshots so users can tell whether Grafana is reading a current published copy (#83) - mt5cli-trades.json: add win_rate_pct computed column to the Trade Statistics by Symbol table via 100.0 * winning_deals / NULLIF( total_deals, 0), with a percent unit override and "Win Rate (%)" display label (#83) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: reject same source and target path in publish_grafana_copy Adds an early same-path guard to publish_grafana_copy: resolves both paths before any I/O and raises ValueError if they are identical, preventing the function from overwriting the live source database with its own backup copy. Also adds a unit test for the rejected case. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address ruff EM102/TRY003/E501 in same-path guard Assigns the ValueError message to a variable before raising and shortens the test docstring to stay within the 88-char line limit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: apply ruff format to publish_grafana_copy error message Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove grafana_ticks panel from default market dashboard The Tick Bid/Ask panel queried grafana_ticks which only exists when users collect tick data (opt-in). Users following the default OHLCV-only setup path hit "no such table: grafana_ticks" on dashboard load. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: close SQLite connections before atomic replace in publish_grafana_copy Wrap both src and dst connections with contextlib.closing() so they are explicitly closed before tmp_path.replace(target_path) runs. Without this, sqlite3.Connection's context manager only commits/rolls back but leaves the file handle open, which can cause PermissionError on Windows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rename history.grafana.db to history.mt5cli.db in Grafana examples frser-sqlite-datasource blocks paths containing "grafana.db" via its internal blocklist. Rename the recommended published filename in the README, compose comment, and datasource provisioning comment to avoid a blocked/denied datasource for native Windows users. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update Docker Compose quick-start to pass MT5CLI_DB_PATH The compose.yml already required MT5CLI_DB_PATH via ${MT5CLI_DB_PATH:?...}, but the README still showed bare `docker compose up -d`. Update the section to show the env-var-prefixed invocation and document the .env file alternative. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
96 lines
3.4 KiB
Markdown
96 lines
3.4 KiB
Markdown
# Grafana Integration for mt5cli
|
|
|
|
This directory contains example configuration and dashboard files for visualising
|
|
mt5cli SQLite data in [Grafana](https://grafana.com/) using the
|
|
[Grafana SQLite datasource plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/).
|
|
|
|
## Prerequisites
|
|
|
|
- mt5cli installed and able to connect to MetaTrader 5
|
|
- Grafana 10+ with the `frser-sqlite-datasource` plugin installed
|
|
- (Optional) Docker and Docker Compose for the containerised setup
|
|
|
|
## Generating the SQLite database
|
|
|
|
Collect historical data and snapshot current account state:
|
|
|
|
```sh
|
|
# Collect OHLCV history
|
|
mt5cli -o history.db collect-history --symbol EURUSD --date-from 2024-01-01 --date-to 2024-12-31
|
|
|
|
# Create Grafana-ready views and indexes
|
|
mt5cli -o history.db grafana-schema
|
|
|
|
# Snapshot current account, positions, and orders
|
|
mt5cli -o history.db snapshot --with-grafana-schema
|
|
```
|
|
|
|
## Publishing a Grafana-readable copy
|
|
|
|
Grafana reads the SQLite file directly. To avoid read/write conflicts, publish
|
|
a consistent copy after each update:
|
|
|
|
```sh
|
|
mt5cli -o history.db grafana-schema --publish-copy history.mt5cli.db
|
|
mt5cli -o history.db snapshot --publish-copy history.mt5cli.db
|
|
```
|
|
|
|
The `--publish-copy` option uses the SQLite online backup API, which is safe
|
|
even when the source database uses WAL journal mode.
|
|
|
|
## Configuring the datasource path
|
|
|
|
Edit `provisioning/datasources/mt5cli-sqlite.yml` and set the `path` field
|
|
to the absolute path of your published `.db` file:
|
|
|
|
```yaml
|
|
jsonData:
|
|
path: /absolute/path/to/history.mt5cli.db
|
|
```
|
|
|
|
## Running Grafana on Windows (native)
|
|
|
|
1. Download and install Grafana from <https://grafana.com/grafana/download/>.
|
|
2. Install the SQLite plugin: `grafana-cli plugins install frser-sqlite-datasource`.
|
|
3. Copy `provisioning/datasources/mt5cli-sqlite.yml` into
|
|
`%ProgramFiles%\GrafanaLabs\grafana\conf\provisioning\datasources\`.
|
|
Do not copy `provisioning/dashboards/mt5cli.yml` — it contains a
|
|
Docker-specific dashboard path that is not valid on Windows.
|
|
4. Import the dashboards from `dashboards/` via the Grafana UI
|
|
(Dashboards → Import → Upload JSON file).
|
|
|
|
## Running with Docker Compose
|
|
|
|
Set `MT5CLI_DB_PATH` to the absolute path of your published `.db` file, then
|
|
start the stack:
|
|
|
|
```sh
|
|
# From the examples/grafana directory
|
|
MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db docker compose up -d
|
|
```
|
|
|
|
Alternatively, create a `.env` file in `examples/grafana/` containing
|
|
`MT5CLI_DB_PATH=/absolute/path/to/history.mt5cli.db` and run
|
|
`docker compose up -d`. Compose refuses to start if the variable is unset or
|
|
empty.
|
|
|
|
Then open <http://localhost:3000> (default credentials: admin / admin).
|
|
|
|
## Dashboard overview
|
|
|
|
| Dashboard | Description |
|
|
| ---------------------- | ------------------------------------------------------- |
|
|
| `mt5cli-overview.json` | Account balance, equity, margin, and snapshot freshness |
|
|
| `mt5cli-trades.json` | Trade P/L, win rate, symbol breakdown |
|
|
| `mt5cli-market.json` | OHLCV rates, spreads, and tick volume |
|
|
|
|
All panel queries use the `grafana_*` views; they do not read internal storage
|
|
tables directly.
|
|
|
|
## Importing dashboards
|
|
|
|
1. Open Grafana and navigate to **Dashboards → Import**.
|
|
2. Click **Upload JSON file** and select one of the files in `dashboards/`.
|
|
3. Select the `mt5cli-SQLite` datasource when prompted.
|
|
4. Click **Import**.
|