feat: add Grafana-ready SQLite observability (#86)

* feat: add Grafana-ready SQLite observability (#79, #80, #81) New `mt5cli/grafana.py` module with idempotent DDL helpers: - `create_snapshot_tables` — five SQLite tables for time-series account, position, order, terminal, and run-status snapshots - `create_grafana_views` — 13 `grafana_*` views with integer epoch-second `time` columns; missing source tables emit warnings and are skipped - `create_grafana_indexes` — 9 performance indexes guarded by column checks - `ensure_grafana_schema` — convenience wrapper calling all three above - Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run` New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`: - `update_observability` — appends a timestamped snapshot to a SQLite db using an already-connected `Mt5DataClient`; never places orders - `update_observability_with_config` — standalone wrapper that opens and closes the MT5 connection automatically New CLI commands (Collection panel): - `grafana-schema` — idempotent schema setup, no MT5 connection required - `snapshot` — append account/position/order/terminal rows; supports `--symbol`, `--with-account/--no-account`, and equivalent flags All public modules maintain 100 % branch coverage; 968 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply mdformat to docs after grafana observability additions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: bump version to 1.1.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address PR #86 review feedback - Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX) - Replace O(N) per-symbol API calls with single call + client-side filter - Eliminate double create_snapshot_tables when with_grafana_schema=True - Move grafana imports to module level in sdk.py (remove PLC0415 noqa) - Default with_grafana_schema to False (run grafana-schema once for setup) - Fix README position example to use snapshot_runs for latest snapshot - Update tests to reflect new behavior and correct patch targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply ruff formatting and sync lock file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address owner review feedback on PR #86 - Filter grafana_*_snapshots views to only expose rows from successful runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot visibility gap raised in PRRT_kwDORzI_286MvDJ6 - Add issubset column guards for snapshot table index creation, consistent with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1) - Fix README example queries: views expose 'time' not 'observed_at' (PRRT_kwDORzI_286MvUxw) - Correct public-contract.md default for with_grafana_schema (False, not True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add run_id to snapshot schema and fix stale-positions README query Replace second-level observed_at as the join key between snapshot_runs and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs in the same second now get distinct run_ids, preventing view duplication and cross-contamination from a failed run. Update README example to use snapshot_runs for latest-snapshot lookup so zero-position runs return an empty result instead of stale rows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: format markdown tables Align table column widths in README and public-contract documentation. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: close sqlite connections deterministically * fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats grafana_realized_pnl now requires the entry column and filters to close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl and grafana_trade_stats. grafana_trade_stats now requires the time column and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana view contract (integer epoch-second time column throughout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots orders_get_as_df() returns datetime-converted columns by default, so time_setup is a pd.Timestamp in normal use. Passing it directly to sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper that converts datetime.datetime subclasses (including pd.Timestamp) and raw int/float values to integer epoch seconds, returning None for other types. Regression tests cover all four input paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: pre-drop all grafana_* views at start of create_grafana_views Previously, a builder that skipped due to a missing source table or column would not drop the view it owned, leaving stale views referencing gone tables. Now create_grafana_views drops all 13 known grafana_* views before calling any builder, so a schema refresh always removes views whose source has disappeared. Regression test covers the create → drop-source → refresh cycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: expose run_id in snapshot views and drop time from summary views Grafana snapshot views now expose run_id so latest-state queries can use MAX(run_id) instead of the ambiguous second-level MAX(observed_at). grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column and are reclassified as static summary views; their all-time aggregates are not filterable by Grafana time-range selectors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: guard snapshot views against missing run_id and fix README view docs _build_snapshot_view now skips with a warning when the underlying snapshot table exists but lacks a run_id column, preventing a broken view that fails at query time. Adds a regression test for that path. README Grafana section now qualifies that grafana_realized_pnl and grafana_trade_stats are static summary views (no time column) and splits the view table to match docs/api/public-contract.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 07:55:26 +09:00
parent 4bc36d09d2
commit d27da02f3f
14 changed files with 2494 additions and 14 deletions
@@ -61,6 +61,8 @@ from mt5cli.sdk import (
    terminal_info,
    update_history,
    update_history_with_config,
+    update_observability,
+    update_observability_with_config,
    version,
 )
 from mt5cli.utils import Dataset, IfExists, coerce_login
@@ -2746,3 +2748,375 @@ class TestSubstituteMappingValues:
        result = substitute_mapping_values(data, keys={"mt5_login"})
        # tuple is returned as-is; inner dict is NOT visited
        assert result == {"accounts": ({"mt5_login": "${MT5_LOGIN}"},)}
+
+
+class TestUpdateObservability:
+    """Tests for update_observability and update_observability_with_config."""
+
+    @pytest.fixture
+    def mock_client(self) -> MagicMock:
+        """Mock client returning minimal valid frames."""
+        client = MagicMock()
+        client.account_info_as_df.return_value = pd.DataFrame([
+            {
+                "login": 12345,
+                "currency": "USD",
+                "balance": 10000.0,
+                "equity": 10000.0,
+                "margin": 0.0,
+                "margin_free": 10000.0,
+                "margin_level": 0.0,
+                "profit": 0.0,
+                "leverage": 100,
+            }
+        ])
+        client.positions_get_as_df.return_value = pd.DataFrame()
+        client.orders_get_as_df.return_value = pd.DataFrame()
+        client.terminal_info_as_df.return_value = pd.DataFrame([
+            {
+                "name": "MetaTrader 5",
+                "connected": 1,
+                "community_account": 0,
+                "trade_allowed": 1,
+                "trade_expert": 1,
+                "path": "/mt5",
+                "company": "Broker",
+                "language": "en",
+            }
+        ])
+        return client
+
+    def test_update_observability_creates_snapshot_tables(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """Snapshot tables are created in the output database."""
+        output = tmp_path / "obs.db"
+        update_observability(client=mock_client, output=output)
+        with sqlite3.connect(output) as conn:
+            tables = {
+                row[0]
+                for row in conn.execute(
+                    "SELECT name FROM sqlite_master WHERE type='table'"
+                ).fetchall()
+            }
+        assert "snapshot_runs" in tables
+        assert "account_snapshots" in tables
+        assert "position_snapshots" in tables
+
+    def test_update_observability_records_ok_on_success(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """snapshot_runs records 'ok' status on a successful run."""
+        output = tmp_path / "obs.db"
+        update_observability(client=mock_client, output=output)
+        with sqlite3.connect(output) as conn:
+            row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
+        assert row == ("ok",)
+
+    def test_update_observability_records_error_on_failure(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """snapshot_runs records 'error' and re-raises when a snapshot fails."""
+        mock_client.account_info_as_df.side_effect = RuntimeError("boom")
+        output = tmp_path / "obs.db"
+        with pytest.raises(RuntimeError, match="boom"):
+            update_observability(client=mock_client, output=output)
+        with sqlite3.connect(output) as conn:
+            row = conn.execute("SELECT status FROM snapshot_runs").fetchone()
+        assert row == ("error",)
+
+    def test_update_observability_skips_ensure_grafana_schema_when_disabled(
+        self,
+        mock_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """with_grafana_schema=False does not call ensure_grafana_schema."""
+        spy = mocker.spy(sdk, "ensure_grafana_schema")
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            with_grafana_schema=False,
+        )
+        spy.assert_not_called()
+
+    def test_update_observability_calls_ensure_grafana_schema_by_default(
+        self,
+        mock_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """with_grafana_schema=True calls ensure_grafana_schema."""
+        spy = mocker.spy(sdk, "ensure_grafana_schema")
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            with_grafana_schema=True,
+        )
+        spy.assert_called_once()
+
+    def test_update_observability_skips_account_when_disabled(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """include_account=False does not call account_info_as_df."""
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            include_account=False,
+        )
+        mock_client.account_info_as_df.assert_not_called()
+
+    def test_update_observability_skips_positions_when_disabled(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """include_positions=False does not call positions_get_as_df."""
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            include_positions=False,
+        )
+        mock_client.positions_get_as_df.assert_not_called()
+
+    def test_update_observability_skips_orders_when_disabled(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """include_orders=False does not call orders_get_as_df."""
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            include_orders=False,
+        )
+        mock_client.orders_get_as_df.assert_not_called()
+
+    def test_update_observability_skips_terminal_when_disabled(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """include_terminal=False does not call terminal_info_as_df."""
+        update_observability(
+            client=mock_client,
+            output=tmp_path / "obs.db",
+            include_terminal=False,
+        )
+        mock_client.terminal_info_as_df.assert_not_called()
+
+    def test_update_observability_with_positions_rows(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """Non-empty positions are written to position_snapshots."""
+        mock_client.positions_get_as_df.return_value = pd.DataFrame([
+            {
+                "ticket": 1,
+                "position_id": 1,
+                "symbol": "EURUSD",
+                "type": 0,
+                "volume": 0.1,
+                "price_open": 1.1,
+                "price_current": 1.1,
+                "profit": 0.0,
+                "swap": 0.0,
+                "comment": "",
+                "magic": 0,
+            }
+        ])
+        output = tmp_path / "obs.db"
+        update_observability(client=mock_client, output=output)
+        with sqlite3.connect(output) as conn:
+            count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
+                0
+            ]
+        assert count == 1
+
+    def test_update_observability_with_order_rows(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """Non-empty orders are written to order_snapshots."""
+        mock_client.orders_get_as_df.return_value = pd.DataFrame([
+            {
+                "ticket": 10,
+                "symbol": "EURUSD",
+                "type": 2,
+                "volume_current": 0.1,
+                "price_open": 1.2,
+                "price_current": 1.1,
+                "state": 1,
+                "comment": "",
+                "magic": 0,
+                "time_setup": 1700000000,
+            }
+        ])
+        output = tmp_path / "obs.db"
+        update_observability(client=mock_client, output=output)
+        with sqlite3.connect(output) as conn:
+            count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
+        assert count == 1
+
+    def test_update_observability_symbol_filter_positions(
+        self,
+        tmp_path: Path,
+    ) -> None:
+        """Symbol filter fetches all positions in one call and filters client-side."""
+        client = MagicMock()
+        client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
+        # All positions; only EURUSD matches the filter
+        client.positions_get_as_df.return_value = pd.DataFrame([
+            {"ticket": 1, "symbol": "EURUSD", "volume": 0.1, "profit": 0.0},
+            {"ticket": 2, "symbol": "USDJPY", "volume": 0.2, "profit": 0.0},
+        ])
+        client.orders_get_as_df.return_value = pd.DataFrame()
+        client.terminal_info_as_df.return_value = pd.DataFrame()
+        output = tmp_path / "obs.db"
+        update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
+        assert client.positions_get_as_df.call_count == 1
+        with sqlite3.connect(output) as conn:
+            count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
+                0
+            ]
+        assert count == 1
+
+    def test_update_observability_symbol_filter_orders(
+        self,
+        tmp_path: Path,
+    ) -> None:
+        """Symbol filter fetches all orders in one call and filters client-side."""
+        client = MagicMock()
+        client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
+        client.positions_get_as_df.return_value = pd.DataFrame()
+        # All orders; only EURUSD matches the filter
+        client.orders_get_as_df.return_value = pd.DataFrame([
+            {"ticket": 10, "symbol": "EURUSD", "volume_current": 0.1},
+            {"ticket": 11, "symbol": "USDJPY", "volume_current": 0.5},
+        ])
+        client.terminal_info_as_df.return_value = pd.DataFrame()
+        output = tmp_path / "obs.db"
+        update_observability(client=client, output=output, symbols=["EURUSD", "GBPUSD"])
+        assert client.orders_get_as_df.call_count == 1
+        with sqlite3.connect(output) as conn:
+            count = conn.execute("SELECT COUNT(*) FROM order_snapshots").fetchone()[0]
+        assert count == 1
+
+    def test_update_observability_symbol_filter_no_symbol_col(
+        self,
+        tmp_path: Path,
+    ) -> None:
+        """Symbol filter is skipped when positions df has no symbol column."""
+        client = MagicMock()
+        client.account_info_as_df.return_value = pd.DataFrame([{"login": 1}])
+        # No symbol column in positions — all rows pass through unfiltered
+        client.positions_get_as_df.return_value = pd.DataFrame([
+            {"ticket": 1, "volume": 0.1},
+        ])
+        client.orders_get_as_df.return_value = pd.DataFrame()
+        client.terminal_info_as_df.return_value = pd.DataFrame()
+        output = tmp_path / "obs.db"
+        update_observability(client=client, output=output, symbols=["EURUSD"])
+        with sqlite3.connect(output) as conn:
+            count = conn.execute("SELECT COUNT(*) FROM position_snapshots").fetchone()[
+                0
+            ]
+        assert count == 1
+
+    def test_update_observability_account_none_login(
+        self,
+        tmp_path: Path,
+    ) -> None:
+        """Account row with no login key returns None login for downstream helpers."""
+        client = MagicMock()
+        client.account_info_as_df.return_value = pd.DataFrame([{"balance": 10000.0}])
+        client.positions_get_as_df.return_value = pd.DataFrame()
+        client.orders_get_as_df.return_value = pd.DataFrame()
+        client.terminal_info_as_df.return_value = pd.DataFrame()
+        output = tmp_path / "obs.db"
+        update_observability(client=client, output=output)
+        with sqlite3.connect(output) as conn:
+            row = conn.execute("SELECT login FROM account_snapshots").fetchone()
+        assert row is not None
+        assert row[0] is None
+
+    def test_update_observability_empty_account_logs_warning(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+        caplog: pytest.LogCaptureFixture,
+    ) -> None:
+        """Empty account_info_as_df logs a warning and does not write account row."""
+        mock_client.account_info_as_df.return_value = pd.DataFrame()
+        with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
+            update_observability(client=mock_client, output=tmp_path / "obs.db")
+        assert "account_info_as_df returned empty frame" in caplog.text
+        with sqlite3.connect(tmp_path / "obs.db") as conn:
+            count = conn.execute("SELECT COUNT(*) FROM account_snapshots").fetchone()[0]
+        assert count == 0
+
+    def test_update_observability_empty_terminal_logs_warning(
+        self,
+        mock_client: MagicMock,
+        tmp_path: Path,
+        caplog: pytest.LogCaptureFixture,
+    ) -> None:
+        """Empty terminal_info_as_df logs a warning and does not write terminal row."""
+        mock_client.terminal_info_as_df.return_value = pd.DataFrame()
+        with caplog.at_level(logging.WARNING, logger="mt5cli.sdk"):
+            update_observability(client=mock_client, output=tmp_path / "obs.db")
+        assert "terminal_info_as_df returned empty frame" in caplog.text
+        with sqlite3.connect(tmp_path / "obs.db") as conn:
+            count = conn.execute("SELECT COUNT(*) FROM terminal_snapshots").fetchone()[
+                0
+            ]
+        assert count == 0
+
+    def test_update_observability_with_config_opens_and_closes_connection(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """update_observability_with_config manages the MT5 connection lifecycle."""
+        mock_client = MagicMock()
+        mock_client.account_info_as_df.return_value = pd.DataFrame()
+        mock_client.positions_get_as_df.return_value = pd.DataFrame()
+        mock_client.orders_get_as_df.return_value = pd.DataFrame()
+        mock_client.terminal_info_as_df.return_value = pd.DataFrame()
+        mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
+        update_observability_with_config(output=tmp_path / "obs.db")
+        mock_client.initialize_and_login_mt5.assert_called_once()
+        mock_client.shutdown.assert_called_once()
+
+    def test_update_observability_with_config_passes_symbols(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """update_observability_with_config forwards symbols to update_observability."""
+        mock_client = MagicMock()
+        mock_client.account_info_as_df.return_value = pd.DataFrame()
+        mock_client.positions_get_as_df.return_value = pd.DataFrame()
+        mock_client.orders_get_as_df.return_value = pd.DataFrame()
+        mock_client.terminal_info_as_df.return_value = pd.DataFrame()
+        mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
+        spy = mocker.patch("mt5cli.sdk.update_observability")
+        update_observability_with_config(
+            output=tmp_path / "obs.db",
+            symbols=["EURUSD"],
+            include_account=False,
+        )
+        spy.assert_called_once()
+        call_kwargs = spy.call_args.kwargs
+        assert call_kwargs["symbols"] == ["EURUSD"]
+        assert call_kwargs["include_account"] is False