feat: Grafana copy publishing, dashboard examples, and optional OTel metrics

Implements three observability improvements: #82 — publish_grafana_copy(): Uses SQLite online backup API (WAL-safe) to atomically publish a consistent read-only copy beside the target. Adds --publish-copy option to grafana-schema and snapshot CLI commands. #83 — examples/grafana/: Minimal working Grafana setup with docker-compose, provisioning datasource/dashboard YAML, and three dashboard JSON files (mt5cli-overview, mt5cli-trades, mt5cli-market). All queries use grafana_* views; no credentials or private paths included. #84 — mt5cli/telemetry.py: Optional OTel metrics behind mt5cli[otel] extra. Base install is unaffected. Adds _Mt5Metrics singleton (no-op until configure_metrics() is called), wraps update_history() and update_observability() with record_history_update / record_snapshot_update context managers, and emits account/position gauges from snapshots. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-27 23:17:13 +00:00
parent d27da02f3f
commit 4c2e25af8a
18 changed files with 1609 additions and 35 deletions
@@ -0,0 +1,87 @@
+# Grafana Integration for mt5cli
+
+This directory contains example configuration and dashboard files for visualising
+mt5cli SQLite data in [Grafana](https://grafana.com/) using the
+[Grafana SQLite datasource plugin](https://grafana.com/grafana/plugins/frser-sqlite-datasource/).
+
+## Prerequisites
+
+- mt5cli installed and able to connect to MetaTrader 5
+- Grafana 10+ with the `frser-sqlite-datasource` plugin installed
+- (Optional) Docker and Docker Compose for the containerised setup
+
+## Generating the SQLite database
+
+Collect historical data and snapshot current account state:
+
+```sh
+# Collect OHLCV history
+mt5cli -o history.db collect-history --symbol EURUSD --date-from 2024-01-01 --date-to 2024-12-31
+
+# Create Grafana-ready views and indexes
+mt5cli -o history.db grafana-schema
+
+# Snapshot current account, positions, and orders
+mt5cli -o history.db snapshot --with-grafana-schema
+```
+
+## Publishing a Grafana-readable copy
+
+Grafana reads the SQLite file directly. To avoid read/write conflicts, publish
+a consistent copy after each update:
+
+```sh
+mt5cli -o history.db grafana-schema --publish-copy history.grafana.db
+mt5cli -o history.db snapshot --publish-copy history.grafana.db
+```
+
+The `--publish-copy` option uses the SQLite online backup API, which is safe
+even when the source database uses WAL journal mode.
+
+## Configuring the datasource path
+
+Edit `provisioning/datasources/mt5cli-sqlite.yaml` and set the `path` field
+to the absolute path of your published `.db` file:
+
+```yaml
+jsonData:
+  path: /absolute/path/to/history.grafana.db
+```
+
+## Running Grafana on Windows (native)
+
+1. Download and install Grafana from <https://grafana.com/grafana/download/>.
+2. Install the SQLite plugin: `grafana-cli plugins install frser-sqlite-datasource`.
+3. Copy `provisioning/` into `%ProgramFiles%\GrafanaLabs\grafana\conf\provisioning\`.
+4. Import the dashboards from `dashboards/` via the Grafana UI
+   (Dashboards → Import → Upload JSON file).
+
+## Running with Docker Compose
+
+```sh
+# From the examples/grafana directory
+docker compose up -d
+```
+
+Then open <http://localhost:3000> (default credentials: admin / admin).
+
+The Compose file mounts this directory and the SQLite file into the container.
+Edit `docker-compose.yml` to point `MT5CLI_DB_PATH` at your `.db` file.
+
+## Dashboard overview
+
+| Dashboard | Description |
+|---|---|
+| `mt5cli-overview.json` | Account balance, equity, margin, and snapshot freshness |
+| `mt5cli-trades.json` | Trade P/L, win rate, symbol breakdown |
+| `mt5cli-market.json` | OHLCV rates, spreads, and tick volume |
+
+All panel queries use the `grafana_*` views; they do not read internal storage
+tables directly.
+
+## Importing dashboards
+
+1. Open Grafana and navigate to **Dashboards → Import**.
+2. Click **Upload JSON file** and select one of the files in `dashboards/`.
+3. Select the `mt5cli-SQLite` datasource when prompted.
+4. Click **Import**.
@@ -0,0 +1,113 @@
+{
+  "__inputs": [
+    {
+      "name": "DS_MT5CLI_SQLITE",
+      "label": "mt5cli-SQLite",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "frser-sqlite-datasource",
+      "pluginName": "SQLite"
+    }
+  ],
+  "__requires": [
+    {
+      "type": "datasource",
+      "id": "frser-sqlite-datasource",
+      "name": "SQLite",
+      "version": "1.0.0"
+    }
+  ],
+  "annotations": {"list": []},
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 0},
+      "id": 1,
+      "title": "Close Price Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"symbol\", \"close\" FROM grafana_rates WHERE $__timeFilter(time) ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
+      "id": 2,
+      "title": "Spread Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"symbol\", \"spread\" FROM grafana_rates WHERE $__timeFilter(time) ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8},
+      "id": 3,
+      "title": "Tick Volume Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"symbol\", \"tick_volume\" FROM grafana_rates WHERE $__timeFilter(time) ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 16},
+      "id": 4,
+      "title": "Tick Bid/Ask Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"symbol\", \"bid\", \"ask\" FROM grafana_ticks WHERE $__timeFilter(time) ORDER BY time LIMIT 5000",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    }
+  ],
+  "refresh": "1m",
+  "schemaVersion": 36,
+  "tags": ["mt5cli", "market"],
+  "templating": {
+    "list": [
+      {
+        "current": {},
+        "hide": 0,
+        "includeAll": false,
+        "label": "Data Source",
+        "multi": false,
+        "name": "DS_MT5CLI_SQLITE",
+        "options": [],
+        "query": "frser-sqlite-datasource",
+        "refresh": 1,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {"from": "now-24h", "to": "now"},
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "MT5CLI - Market Data",
+  "uid": "mt5cli-market",
+  "version": 1
+}
@@ -0,0 +1,159 @@
+{
+  "__inputs": [
+    {
+      "name": "DS_MT5CLI_SQLITE",
+      "label": "mt5cli-SQLite",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "frser-sqlite-datasource",
+      "pluginName": "SQLite"
+    }
+  ],
+  "__requires": [
+    {
+      "type": "datasource",
+      "id": "frser-sqlite-datasource",
+      "name": "SQLite",
+      "version": "1.0.0"
+    }
+  ],
+  "annotations": {"list": []},
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {
+        "defaults": {"unit": "currencyUSD"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 1,
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}},
+      "title": "Balance",
+      "type": "stat",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {
+        "defaults": {"unit": "currencyUSD"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 2,
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}},
+      "title": "Equity",
+      "type": "stat",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {
+        "defaults": {"unit": "currencyUSD"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 3,
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}},
+      "title": "Free Margin",
+      "type": "stat",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"margin_free\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {
+        "defaults": {"unit": "percent"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 4,
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}},
+      "title": "Margin Level",
+      "type": "stat",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"margin_level\" FROM grafana_account_snapshots ORDER BY time DESC LIMIT 1",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 4},
+      "id": 5,
+      "title": "Account Balance Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"balance\" FROM grafana_account_snapshots ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 12},
+      "id": 6,
+      "title": "Equity Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"equity\" FROM grafana_account_snapshots ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    }
+  ],
+  "refresh": "1m",
+  "schemaVersion": 36,
+  "tags": ["mt5cli", "account"],
+  "templating": {
+    "list": [
+      {
+        "current": {},
+        "hide": 0,
+        "includeAll": false,
+        "label": "Data Source",
+        "multi": false,
+        "name": "DS_MT5CLI_SQLITE",
+        "options": [],
+        "query": "frser-sqlite-datasource",
+        "refresh": 1,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {"from": "now-7d", "to": "now"},
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "MT5CLI - Account Overview",
+  "uid": "mt5cli-overview",
+  "version": 1
+}
@@ -0,0 +1,113 @@
+{
+  "__inputs": [
+    {
+      "name": "DS_MT5CLI_SQLITE",
+      "label": "mt5cli-SQLite",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "frser-sqlite-datasource",
+      "pluginName": "SQLite"
+    }
+  ],
+  "__requires": [
+    {
+      "type": "datasource",
+      "id": "frser-sqlite-datasource",
+      "name": "SQLite",
+      "version": "1.0.0"
+    }
+  ],
+  "annotations": {"list": []},
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
+      "id": 1,
+      "title": "Realized P/L by Symbol",
+      "type": "table",
+      "targets": [
+        {
+          "rawSql": "SELECT \"symbol\", \"cumulative_pnl\", \"deal_count\" FROM grafana_realized_pnl ORDER BY cumulative_pnl DESC",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
+      "id": 2,
+      "title": "Trade Statistics by Symbol",
+      "type": "table",
+      "targets": [
+        {
+          "rawSql": "SELECT \"symbol\", \"total_deals\", \"winning_deals\", \"losing_deals\", \"total_profit\", \"avg_profit\" FROM grafana_trade_stats ORDER BY total_profit DESC",
+          "format": "table",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 8},
+      "id": 3,
+      "title": "Open Position Profit Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"symbol\", SUM(\"profit\") AS profit FROM grafana_position_snapshots GROUP BY time, symbol ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": "${DS_MT5CLI_SQLITE}",
+      "fieldConfig": {"defaults": {}, "overrides": []},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 16},
+      "id": 4,
+      "title": "Cash Events Over Time",
+      "type": "timeseries",
+      "targets": [
+        {
+          "rawSql": "SELECT \"time\" AS time, \"profit\" FROM grafana_cash_events ORDER BY time",
+          "format": "time_series",
+          "refId": "A"
+        }
+      ]
+    }
+  ],
+  "refresh": "5m",
+  "schemaVersion": 36,
+  "tags": ["mt5cli", "trades"],
+  "templating": {
+    "list": [
+      {
+        "current": {},
+        "hide": 0,
+        "includeAll": false,
+        "label": "Data Source",
+        "multi": false,
+        "name": "DS_MT5CLI_SQLITE",
+        "options": [],
+        "query": "frser-sqlite-datasource",
+        "refresh": 1,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {"from": "now-30d", "to": "now"},
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "MT5CLI - Trade Analytics",
+  "uid": "mt5cli-trades",
+  "version": 1
+}
@@ -0,0 +1,24 @@
+# Docker Compose for Grafana with mt5cli SQLite datasource.
+#
+# Set MT5CLI_DB_PATH to the absolute host path of your published .db file
+# before running `docker compose up -d`.
+#
+# Example:
+#   MT5CLI_DB_PATH=/home/user/history.grafana.db docker compose up -d
+
+services:
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000"
+    environment:
+      GF_PATHS_PROVISIONING: /etc/grafana/provisioning
+    volumes:
+      - ./provisioning:/etc/grafana/provisioning:ro
+      - ./dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-storage:/var/lib/grafana
+      - ${MT5CLI_DB_PATH:-/tmp/mt5cli-placeholder.db}:/data/mt5cli.db:ro
+    user: "472"
+
+volumes:
+  grafana-storage:
@@ -0,0 +1,13 @@
+# Grafana dashboard provisioning for mt5cli dashboards.
+
+apiVersion: 1
+
+providers:
+  - name: mt5cli
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: false
@@ -0,0 +1,16 @@
+# Grafana datasource provisioning for mt5cli SQLite.
+#
+# Requires the frser-sqlite-datasource plugin:
+#   grafana-cli plugins install frser-sqlite-datasource
+#
+# Set `path` to the absolute path of your published history.grafana.db file.
+
+apiVersion: 1
+
+datasources:
+  - name: mt5cli-SQLite
+    type: frser-sqlite-datasource
+    access: proxy
+    isDefault: true
+    jsonData:
+      path: /data/mt5cli.db