1ffac45d57
* feat: Grafana copy publishing, dashboard examples, and optional OTel metrics Implements three observability improvements: #82 — publish_grafana_copy(): Uses SQLite online backup API (WAL-safe) to atomically publish a consistent read-only copy beside the target. Adds --publish-copy option to grafana-schema and snapshot CLI commands. #83 — examples/grafana/: Minimal working Grafana setup with docker-compose, provisioning datasource/dashboard YAML, and three dashboard JSON files (mt5cli-overview, mt5cli-trades, mt5cli-market). All queries use grafana_* views; no credentials or private paths included. #84 — mt5cli/telemetry.py: Optional OTel metrics behind mt5cli[otel] extra. Base install is unaffected. Adds _Mt5Metrics singleton (no-op until configure_metrics() is called), wraps update_history() and update_observability() with record_history_update / record_snapshot_update context managers, and emits account/position gauges from snapshots. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace ambiguous multiplication sign in comment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: normalize markdown formatting in grafana README Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve file mode on Grafana copy and fix unsupported time macro - publish_grafana_copy: chmod temp file to match the existing target's permissions (or 0o644 when no prior target exists) before atomic replace, so Grafana running as a different OS user (e.g. UID 472 in Docker) can read the published database - mt5cli-market.json: replace unsupported \$__timeFilter(time) with the epoch-based filter supported by frser-sqlite-datasource: "time" >= \$__from / 1000 AND "time" < \$__to / 1000 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: skip Windows-incompatible mode test, rename compose file to compose.yaml - Skip test_overwrite_preserves_existing_target_mode on win32 since Windows chmod does not preserve Unix group/other permission bits - Simplify test_fresh_target_has_readable_permissions to check owner read bit only (portable across platforms) - Rename docker-compose.yml -> compose.yaml (modern Compose convention) - Update README and test reference to match new filename Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: rename *.yaml to *.yml in examples/grafana Renames compose.yaml, mt5cli-sqlite.yaml, and mt5cli.yaml to .yml; updates README and test references accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: format Grafana dashboards and expand qa script to include JSON - Update qa.sh prettier pattern to format JSON files alongside markdown - Reformat Grafana dashboard JSONs with consistent spacing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address owner review comments before merge - qa.sh: fix Prettier glob from `{,d,json}` to `{md,json}` so Markdown files are actually formatted by local QA (P2) - compose.yml: add GF_INSTALL_PLUGINS env var so the frser-sqlite-datasource plugin is installed at container start (P1) - telemetry.py: replace no-op get_meter() call with a real SDK MeterProvider pipeline; add optional `readers` kwarg so callers can inject custom readers (e.g. InMemoryMetricReader in tests) without needing the OTLP package (P1) - sdk.py: aggregate profit and volume by symbol before emitting gauge values so hedging accounts with multiple same-symbol positions emit one point per symbol instead of overwriting with each row (P2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: emit mt5_history_update_rows_total via conn.total_changes delta The counter was registered but never incremented, making the advertised history-update throughput metric permanently zero. Add add_history_rows() to _Mt5Metrics and call it in update_history() using the SQLite total_changes delta measured around write_incremental_datasets(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address three owner review comments - compose.yml: replace soft fallback with :? error expansion so Compose refuses to start when MT5CLI_DB_PATH is unset or empty (P1) - README.md: tell native Windows users to copy only the datasource provisioning file; the dashboards yml contains a Docker-specific path that is invalid on Windows (P2) - telemetry.py / sdk.py: emit mt5_terminal_connected, mt5_terminal_trade_allowed, and mt5_terminal_trade_expert gauges via a new record_terminal_state() method called from _snapshot_terminal(), completing the connection-status metric surface from issue #84 (P2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add snapshot freshness panel and win-rate column to dashboards - mt5cli-overview.json: add a full-width "Last Snapshot" stat panel (dateTimeFromNow unit) below the account stats, querying MAX(time)*1000 from grafana_account_snapshots so users can tell whether Grafana is reading a current published copy (#83) - mt5cli-trades.json: add win_rate_pct computed column to the Trade Statistics by Symbol table via 100.0 * winning_deals / NULLIF( total_deals, 0), with a percent unit override and "Win Rate (%)" display label (#83) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: reject same source and target path in publish_grafana_copy Adds an early same-path guard to publish_grafana_copy: resolves both paths before any I/O and raises ValueError if they are identical, preventing the function from overwriting the live source database with its own backup copy. Also adds a unit test for the rejected case. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address ruff EM102/TRY003/E501 in same-path guard Assigns the ValueError message to a variable before raising and shortens the test docstring to stay within the 88-char line limit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: apply ruff format to publish_grafana_copy error message Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove grafana_ticks panel from default market dashboard The Tick Bid/Ask panel queried grafana_ticks which only exists when users collect tick data (opt-in). Users following the default OHLCV-only setup path hit "no such table: grafana_ticks" on dashboard load. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: close SQLite connections before atomic replace in publish_grafana_copy Wrap both src and dst connections with contextlib.closing() so they are explicitly closed before tmp_path.replace(target_path) runs. Without this, sqlite3.Connection's context manager only commits/rolls back but leaves the file handle open, which can cause PermissionError on Windows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rename history.grafana.db to history.mt5cli.db in Grafana examples frser-sqlite-datasource blocks paths containing "grafana.db" via its internal blocklist. Rename the recommended published filename in the README, compose comment, and datasource provisioning comment to avoid a blocked/denied datasource for native Windows users. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update Docker Compose quick-start to pass MT5CLI_DB_PATH The compose.yml already required MT5CLI_DB_PATH via ${MT5CLI_DB_PATH:?...}, but the README still showed bare `docker compose up -d`. Update the section to show the env-var-prefixed invocation and document the .env file alternative. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
355 lines
12 KiB
Python
355 lines
12 KiB
Python
"""Optional OpenTelemetry metrics for MT5 history and snapshot observability."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import time
|
|
from contextlib import contextmanager
|
|
from typing import TYPE_CHECKING, Any
|
|
|
|
if TYPE_CHECKING:
|
|
from collections.abc import Iterator
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
_otel_available_flag = False
|
|
|
|
try:
|
|
import opentelemetry.metrics as _otel_metrics_mod
|
|
from opentelemetry.sdk.metrics import MeterProvider as _OtelMeterProvider
|
|
from opentelemetry.sdk.metrics.export import (
|
|
PeriodicExportingMetricReader as _OtelPeriodicReader,
|
|
)
|
|
from opentelemetry.sdk.resources import Resource as _OtelResource
|
|
|
|
_otel_available_flag = True
|
|
except ImportError: # pragma: no cover
|
|
_otel_metrics_mod = None # type: ignore[assignment]
|
|
_OtelMeterProvider = None # type: ignore[assignment]
|
|
_OtelPeriodicReader = None # type: ignore[assignment]
|
|
_OtelResource = None # type: ignore[assignment]
|
|
|
|
_OTEL_AVAILABLE: bool = _otel_available_flag
|
|
|
|
try:
|
|
from opentelemetry.exporter.otlp.proto.http.metric_exporter import ( # type: ignore[import]
|
|
OTLPMetricExporter as _OtelOTLPExporter, # type: ignore[reportUnknownVariableType]
|
|
)
|
|
except ImportError: # pragma: no cover
|
|
_OtelOTLPExporter = None # type: ignore[assignment, misc]
|
|
|
|
|
|
class _NoOp:
|
|
"""No-op instrument that silently ignores all calls."""
|
|
|
|
def add(
|
|
self,
|
|
amount: float,
|
|
attributes: dict[str, str] | None = None,
|
|
) -> None:
|
|
"""No-op add."""
|
|
|
|
def set(
|
|
self,
|
|
amount: float,
|
|
attributes: dict[str, str] | None = None,
|
|
) -> None:
|
|
"""No-op set."""
|
|
|
|
def record(
|
|
self,
|
|
amount: float,
|
|
attributes: dict[str, str] | None = None,
|
|
) -> None:
|
|
"""No-op record."""
|
|
|
|
|
|
_NOOP: _NoOp = _NoOp()
|
|
|
|
|
|
class _Mt5Metrics:
|
|
"""MT5 metric instrument registry.
|
|
|
|
Holds references to OTel instruments. All instruments are no-op until
|
|
:meth:`configure` is called with a compatible meter object.
|
|
"""
|
|
|
|
def __init__(self) -> None:
|
|
self._history_duration: Any = _NOOP
|
|
self._history_rows: Any = _NOOP
|
|
self._history_failures: Any = _NOOP
|
|
self._snapshot_duration: Any = _NOOP
|
|
self._snapshot_failures: Any = _NOOP
|
|
self._account_balance: Any = _NOOP
|
|
self._account_equity: Any = _NOOP
|
|
self._account_margin: Any = _NOOP
|
|
self._account_margin_free: Any = _NOOP
|
|
self._account_margin_level: Any = _NOOP
|
|
self._position_profit: Any = _NOOP
|
|
self._position_volume: Any = _NOOP
|
|
self._terminal_connected: Any = _NOOP
|
|
self._terminal_trade_allowed: Any = _NOOP
|
|
self._terminal_trade_expert: Any = _NOOP
|
|
self._last_successful_update: Any = _NOOP
|
|
|
|
def configure(self, meter: Any) -> None: # noqa: ANN401
|
|
"""Set up metric instruments from a meter object.
|
|
|
|
Args:
|
|
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object
|
|
that supports ``create_counter``, ``create_histogram``, and
|
|
``create_gauge``.
|
|
"""
|
|
self._history_duration = meter.create_histogram(
|
|
"mt5_history_update_duration_seconds",
|
|
unit="s",
|
|
description="Duration of incremental history update operations.",
|
|
)
|
|
self._history_rows = meter.create_counter(
|
|
"mt5_history_update_rows_total",
|
|
description="Rows written during incremental history updates.",
|
|
)
|
|
self._history_failures = meter.create_counter(
|
|
"mt5_history_update_failures_total",
|
|
description="Number of incremental history update failures.",
|
|
)
|
|
self._snapshot_duration = meter.create_histogram(
|
|
"mt5_snapshot_update_duration_seconds",
|
|
unit="s",
|
|
description="Duration of snapshot update operations.",
|
|
)
|
|
self._snapshot_failures = meter.create_counter(
|
|
"mt5_snapshot_update_failures_total",
|
|
description="Number of snapshot update failures.",
|
|
)
|
|
self._account_balance = meter.create_gauge(
|
|
"mt5_account_balance",
|
|
description="Account balance.",
|
|
)
|
|
self._account_equity = meter.create_gauge(
|
|
"mt5_account_equity",
|
|
description="Account equity.",
|
|
)
|
|
self._account_margin = meter.create_gauge(
|
|
"mt5_account_margin",
|
|
description="Account margin used.",
|
|
)
|
|
self._account_margin_free = meter.create_gauge(
|
|
"mt5_account_margin_free",
|
|
description="Account free margin.",
|
|
)
|
|
self._account_margin_level = meter.create_gauge(
|
|
"mt5_account_margin_level",
|
|
description="Account margin level as a percentage.",
|
|
)
|
|
self._position_profit = meter.create_gauge(
|
|
"mt5_position_profit",
|
|
description="Floating profit for an open position.",
|
|
)
|
|
self._position_volume = meter.create_gauge(
|
|
"mt5_position_volume",
|
|
description="Volume of an open position.",
|
|
)
|
|
self._terminal_connected = meter.create_gauge(
|
|
"mt5_terminal_connected",
|
|
description="1 if the terminal is connected to the broker, 0 otherwise.",
|
|
)
|
|
self._terminal_trade_allowed = meter.create_gauge(
|
|
"mt5_terminal_trade_allowed",
|
|
description="1 if trading is allowed by the broker server, 0 otherwise.",
|
|
)
|
|
self._terminal_trade_expert = meter.create_gauge(
|
|
"mt5_terminal_trade_expert",
|
|
description="1 if Expert Advisor trading is enabled, 0 otherwise.",
|
|
)
|
|
self._last_successful_update = meter.create_gauge(
|
|
"mt5_last_successful_update_timestamp",
|
|
description="Unix timestamp of the last successful history update.",
|
|
)
|
|
|
|
@contextmanager
|
|
def record_history_update(
|
|
self,
|
|
*,
|
|
dataset: str,
|
|
) -> Iterator[None]:
|
|
"""Context manager recording history update duration and failures.
|
|
|
|
Args:
|
|
dataset: Dataset label (e.g. ``"rates"``).
|
|
|
|
Yields:
|
|
None inside the update operation.
|
|
"""
|
|
attrs = {"dataset": dataset}
|
|
start = time.monotonic()
|
|
try:
|
|
yield
|
|
self._history_duration.record(time.monotonic() - start, attrs)
|
|
self._last_successful_update.set(time.time(), attrs)
|
|
except Exception:
|
|
self._history_failures.add(1, attrs)
|
|
raise
|
|
|
|
def add_history_rows(self, count: int, *, dataset: str) -> None:
|
|
"""Increment the history rows-written counter.
|
|
|
|
Args:
|
|
count: Number of rows written during this update.
|
|
dataset: Dataset label (e.g. ``"rates"``).
|
|
"""
|
|
self._history_rows.add(count, {"dataset": dataset})
|
|
|
|
@contextmanager
|
|
def record_snapshot_update(self) -> Iterator[None]:
|
|
"""Context manager recording snapshot update duration and failures.
|
|
|
|
Yields:
|
|
None inside the snapshot operation.
|
|
"""
|
|
start = time.monotonic()
|
|
try:
|
|
yield
|
|
self._snapshot_duration.record(time.monotonic() - start, {})
|
|
except Exception:
|
|
self._snapshot_failures.add(1, {})
|
|
raise
|
|
|
|
def record_account_state(
|
|
self,
|
|
*,
|
|
login: str,
|
|
server: str,
|
|
balance: float,
|
|
equity: float,
|
|
margin: float,
|
|
margin_free: float,
|
|
margin_level: float,
|
|
) -> None:
|
|
"""Emit account metric gauges.
|
|
|
|
Args:
|
|
login: Account login number (as string; not a password or secret).
|
|
server: Broker server name.
|
|
balance: Account balance.
|
|
equity: Account equity.
|
|
margin: Margin used.
|
|
margin_free: Free margin.
|
|
margin_level: Margin level percentage.
|
|
"""
|
|
attrs: dict[str, str] = {"login": login, "server": server}
|
|
self._account_balance.set(balance, attrs)
|
|
self._account_equity.set(equity, attrs)
|
|
self._account_margin.set(margin, attrs)
|
|
self._account_margin_free.set(margin_free, attrs)
|
|
self._account_margin_level.set(margin_level, attrs)
|
|
|
|
def record_position_state(
|
|
self,
|
|
*,
|
|
login: str,
|
|
server: str,
|
|
symbol: str,
|
|
profit: float,
|
|
volume: float,
|
|
) -> None:
|
|
"""Emit position metric gauges.
|
|
|
|
Args:
|
|
login: Account login number (as string).
|
|
server: Broker server name.
|
|
symbol: Position symbol.
|
|
profit: Floating profit/loss.
|
|
volume: Position volume.
|
|
"""
|
|
attrs: dict[str, str] = {"login": login, "server": server, "symbol": symbol}
|
|
self._position_profit.set(profit, attrs)
|
|
self._position_volume.set(volume, attrs)
|
|
|
|
def record_terminal_state(
|
|
self,
|
|
*,
|
|
connected: float,
|
|
trade_allowed: float,
|
|
trade_expert: float,
|
|
) -> None:
|
|
"""Emit terminal connection and trading status gauges.
|
|
|
|
Args:
|
|
connected: 1.0 if connected to the broker, 0.0 otherwise.
|
|
trade_allowed: 1.0 if broker server allows trading, 0.0 otherwise.
|
|
trade_expert: 1.0 if Expert Advisor trading is enabled, 0.0 otherwise.
|
|
"""
|
|
self._terminal_connected.set(connected, {})
|
|
self._terminal_trade_allowed.set(trade_allowed, {})
|
|
self._terminal_trade_expert.set(trade_expert, {})
|
|
|
|
|
|
_metrics = _Mt5Metrics()
|
|
|
|
|
|
def configure_metrics(meter: Any) -> None: # noqa: ANN401
|
|
"""Configure MT5 metrics using the provided meter.
|
|
|
|
Args:
|
|
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object.
|
|
"""
|
|
_metrics.configure(meter)
|
|
|
|
|
|
def enable_otel_metrics(
|
|
service_name: str = "mt5cli",
|
|
readers: list[Any] | None = None,
|
|
) -> None:
|
|
"""Enable OTel metrics by wiring up an SDK ``MeterProvider`` pipeline.
|
|
|
|
Requires the ``otel`` optional dependency group:
|
|
``pip install "mt5cli[otel]"``.
|
|
|
|
Args:
|
|
service_name: OTel meter/service name used for the ``Resource`` and
|
|
the meter itself.
|
|
readers: Optional list of metric readers. When *None* (the default),
|
|
a :class:`~opentelemetry.sdk.metrics.export.PeriodicExportingMetricReader`
|
|
backed by an OTLP HTTP exporter is created automatically
|
|
(reads the endpoint from ``OTEL_EXPORTER_OTLP_ENDPOINT``).
|
|
Pass a custom list (e.g. ``InMemoryMetricReader`` for tests)
|
|
to override.
|
|
|
|
Raises:
|
|
ImportError: If ``opentelemetry-api`` is not installed, or if
|
|
``readers`` is *None* and
|
|
``opentelemetry-exporter-otlp-proto-http`` is not installed.
|
|
"""
|
|
if not _OTEL_AVAILABLE:
|
|
msg = (
|
|
"opentelemetry-api is not installed. "
|
|
'Install it with: pip install "mt5cli[otel]"'
|
|
)
|
|
raise ImportError(msg)
|
|
if readers is None:
|
|
if _OtelOTLPExporter is None:
|
|
msg = (
|
|
"opentelemetry-exporter-otlp-proto-http is required for the "
|
|
"default OTLP export pipeline. "
|
|
'Install it with: pip install "mt5cli[otel]" or pass a '
|
|
"custom readers list."
|
|
)
|
|
raise ImportError(msg)
|
|
readers = [_OtelPeriodicReader(_OtelOTLPExporter())] # type: ignore[misc]
|
|
resource = _OtelResource.create({"service.name": service_name}) # type: ignore[union-attr]
|
|
provider = _OtelMeterProvider(resource=resource, metric_readers=readers) # type: ignore[misc]
|
|
_otel_metrics_mod.set_meter_provider(provider) # type: ignore[union-attr]
|
|
meter = provider.get_meter(service_name)
|
|
configure_metrics(meter)
|
|
|
|
|
|
def get_metrics() -> _Mt5Metrics:
|
|
"""Return the global :class:`_Mt5Metrics` instance.
|
|
|
|
Returns:
|
|
The global metric registry (no-op until :func:`configure_metrics` is
|
|
called).
|
|
"""
|
|
return _metrics
|