Files
mt5cli/mt5cli/telemetry.py
T
Daichi Narushima 1ffac45d57 feat: add publish_grafana_copy, Grafana examples, and optional OTel metrics (#89)
* feat: Grafana copy publishing, dashboard examples, and optional OTel metrics

Implements three observability improvements:

#82 — publish_grafana_copy(): Uses SQLite online backup API (WAL-safe) to
atomically publish a consistent read-only copy beside the target. Adds
--publish-copy option to grafana-schema and snapshot CLI commands.

#83 — examples/grafana/: Minimal working Grafana setup with docker-compose,
provisioning datasource/dashboard YAML, and three dashboard JSON files
(mt5cli-overview, mt5cli-trades, mt5cli-market). All queries use grafana_*
views; no credentials or private paths included.

#84 — mt5cli/telemetry.py: Optional OTel metrics behind mt5cli[otel] extra.
Base install is unaffected. Adds _Mt5Metrics singleton (no-op until
configure_metrics() is called), wraps update_history() and
update_observability() with record_history_update / record_snapshot_update
context managers, and emits account/position gauges from snapshots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: replace ambiguous multiplication sign in comment

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: normalize markdown formatting in grafana README

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: preserve file mode on Grafana copy and fix unsupported time macro

- publish_grafana_copy: chmod temp file to match the existing target's
  permissions (or 0o644 when no prior target exists) before atomic
  replace, so Grafana running as a different OS user (e.g. UID 472 in
  Docker) can read the published database
- mt5cli-market.json: replace unsupported \$__timeFilter(time) with the
  epoch-based filter supported by frser-sqlite-datasource:
  "time" >= \$__from / 1000 AND "time" < \$__to / 1000

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: skip Windows-incompatible mode test, rename compose file to compose.yaml

- Skip test_overwrite_preserves_existing_target_mode on win32 since
  Windows chmod does not preserve Unix group/other permission bits
- Simplify test_fresh_target_has_readable_permissions to check owner
  read bit only (portable across platforms)
- Rename docker-compose.yml -> compose.yaml (modern Compose convention)
- Update README and test reference to match new filename

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: rename *.yaml to *.yml in examples/grafana

Renames compose.yaml, mt5cli-sqlite.yaml, and mt5cli.yaml to .yml;
updates README and test references accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: format Grafana dashboards and expand qa script to include JSON

- Update qa.sh prettier pattern to format JSON files alongside markdown
- Reformat Grafana dashboard JSONs with consistent spacing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address owner review comments before merge

- qa.sh: fix Prettier glob from `{,d,json}` to `{md,json}` so Markdown
  files are actually formatted by local QA (P2)
- compose.yml: add GF_INSTALL_PLUGINS env var so the frser-sqlite-datasource
  plugin is installed at container start (P1)
- telemetry.py: replace no-op get_meter() call with a real SDK MeterProvider
  pipeline; add optional `readers` kwarg so callers can inject custom readers
  (e.g. InMemoryMetricReader in tests) without needing the OTLP package (P1)
- sdk.py: aggregate profit and volume by symbol before emitting gauge values
  so hedging accounts with multiple same-symbol positions emit one point per
  symbol instead of overwriting with each row (P2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: emit mt5_history_update_rows_total via conn.total_changes delta

The counter was registered but never incremented, making the advertised
history-update throughput metric permanently zero. Add add_history_rows()
to _Mt5Metrics and call it in update_history() using the SQLite
total_changes delta measured around write_incremental_datasets().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address three owner review comments

- compose.yml: replace soft fallback with :? error expansion so Compose
  refuses to start when MT5CLI_DB_PATH is unset or empty (P1)
- README.md: tell native Windows users to copy only the datasource
  provisioning file; the dashboards yml contains a Docker-specific path
  that is invalid on Windows (P2)
- telemetry.py / sdk.py: emit mt5_terminal_connected,
  mt5_terminal_trade_allowed, and mt5_terminal_trade_expert gauges via a
  new record_terminal_state() method called from _snapshot_terminal(),
  completing the connection-status metric surface from issue #84 (P2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: add snapshot freshness panel and win-rate column to dashboards

- mt5cli-overview.json: add a full-width "Last Snapshot" stat panel
  (dateTimeFromNow unit) below the account stats, querying
  MAX(time)*1000 from grafana_account_snapshots so users can tell
  whether Grafana is reading a current published copy (#83)
- mt5cli-trades.json: add win_rate_pct computed column to the Trade
  Statistics by Symbol table via 100.0 * winning_deals / NULLIF(
  total_deals, 0), with a percent unit override and "Win Rate (%)"
  display label (#83)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: reject same source and target path in publish_grafana_copy

Adds an early same-path guard to publish_grafana_copy: resolves both
paths before any I/O and raises ValueError if they are identical,
preventing the function from overwriting the live source database with
its own backup copy. Also adds a unit test for the rejected case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address ruff EM102/TRY003/E501 in same-path guard

Assigns the ValueError message to a variable before raising and
shortens the test docstring to stay within the 88-char line limit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: apply ruff format to publish_grafana_copy error message

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: remove grafana_ticks panel from default market dashboard

The Tick Bid/Ask panel queried grafana_ticks which only exists when users
collect tick data (opt-in). Users following the default OHLCV-only setup
path hit "no such table: grafana_ticks" on dashboard load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: close SQLite connections before atomic replace in publish_grafana_copy

Wrap both src and dst connections with contextlib.closing() so they are
explicitly closed before tmp_path.replace(target_path) runs. Without
this, sqlite3.Connection's context manager only commits/rolls back but
leaves the file handle open, which can cause PermissionError on Windows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: rename history.grafana.db to history.mt5cli.db in Grafana examples

frser-sqlite-datasource blocks paths containing "grafana.db" via its
internal blocklist. Rename the recommended published filename in the
README, compose comment, and datasource provisioning comment to avoid
a blocked/denied datasource for native Windows users.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update Docker Compose quick-start to pass MT5CLI_DB_PATH

The compose.yml already required MT5CLI_DB_PATH via ${MT5CLI_DB_PATH:?...},
but the README still showed bare `docker compose up -d`. Update the section
to show the env-var-prefixed invocation and document the .env file alternative.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 17:14:17 +09:00

355 lines
12 KiB
Python

"""Optional OpenTelemetry metrics for MT5 history and snapshot observability."""
from __future__ import annotations
import logging
import time
from contextlib import contextmanager
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Iterator
logger = logging.getLogger(__name__)
_otel_available_flag = False
try:
import opentelemetry.metrics as _otel_metrics_mod
from opentelemetry.sdk.metrics import MeterProvider as _OtelMeterProvider
from opentelemetry.sdk.metrics.export import (
PeriodicExportingMetricReader as _OtelPeriodicReader,
)
from opentelemetry.sdk.resources import Resource as _OtelResource
_otel_available_flag = True
except ImportError: # pragma: no cover
_otel_metrics_mod = None # type: ignore[assignment]
_OtelMeterProvider = None # type: ignore[assignment]
_OtelPeriodicReader = None # type: ignore[assignment]
_OtelResource = None # type: ignore[assignment]
_OTEL_AVAILABLE: bool = _otel_available_flag
try:
from opentelemetry.exporter.otlp.proto.http.metric_exporter import ( # type: ignore[import]
OTLPMetricExporter as _OtelOTLPExporter, # type: ignore[reportUnknownVariableType]
)
except ImportError: # pragma: no cover
_OtelOTLPExporter = None # type: ignore[assignment, misc]
class _NoOp:
"""No-op instrument that silently ignores all calls."""
def add(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op add."""
def set(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op set."""
def record(
self,
amount: float,
attributes: dict[str, str] | None = None,
) -> None:
"""No-op record."""
_NOOP: _NoOp = _NoOp()
class _Mt5Metrics:
"""MT5 metric instrument registry.
Holds references to OTel instruments. All instruments are no-op until
:meth:`configure` is called with a compatible meter object.
"""
def __init__(self) -> None:
self._history_duration: Any = _NOOP
self._history_rows: Any = _NOOP
self._history_failures: Any = _NOOP
self._snapshot_duration: Any = _NOOP
self._snapshot_failures: Any = _NOOP
self._account_balance: Any = _NOOP
self._account_equity: Any = _NOOP
self._account_margin: Any = _NOOP
self._account_margin_free: Any = _NOOP
self._account_margin_level: Any = _NOOP
self._position_profit: Any = _NOOP
self._position_volume: Any = _NOOP
self._terminal_connected: Any = _NOOP
self._terminal_trade_allowed: Any = _NOOP
self._terminal_trade_expert: Any = _NOOP
self._last_successful_update: Any = _NOOP
def configure(self, meter: Any) -> None: # noqa: ANN401
"""Set up metric instruments from a meter object.
Args:
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object
that supports ``create_counter``, ``create_histogram``, and
``create_gauge``.
"""
self._history_duration = meter.create_histogram(
"mt5_history_update_duration_seconds",
unit="s",
description="Duration of incremental history update operations.",
)
self._history_rows = meter.create_counter(
"mt5_history_update_rows_total",
description="Rows written during incremental history updates.",
)
self._history_failures = meter.create_counter(
"mt5_history_update_failures_total",
description="Number of incremental history update failures.",
)
self._snapshot_duration = meter.create_histogram(
"mt5_snapshot_update_duration_seconds",
unit="s",
description="Duration of snapshot update operations.",
)
self._snapshot_failures = meter.create_counter(
"mt5_snapshot_update_failures_total",
description="Number of snapshot update failures.",
)
self._account_balance = meter.create_gauge(
"mt5_account_balance",
description="Account balance.",
)
self._account_equity = meter.create_gauge(
"mt5_account_equity",
description="Account equity.",
)
self._account_margin = meter.create_gauge(
"mt5_account_margin",
description="Account margin used.",
)
self._account_margin_free = meter.create_gauge(
"mt5_account_margin_free",
description="Account free margin.",
)
self._account_margin_level = meter.create_gauge(
"mt5_account_margin_level",
description="Account margin level as a percentage.",
)
self._position_profit = meter.create_gauge(
"mt5_position_profit",
description="Floating profit for an open position.",
)
self._position_volume = meter.create_gauge(
"mt5_position_volume",
description="Volume of an open position.",
)
self._terminal_connected = meter.create_gauge(
"mt5_terminal_connected",
description="1 if the terminal is connected to the broker, 0 otherwise.",
)
self._terminal_trade_allowed = meter.create_gauge(
"mt5_terminal_trade_allowed",
description="1 if trading is allowed by the broker server, 0 otherwise.",
)
self._terminal_trade_expert = meter.create_gauge(
"mt5_terminal_trade_expert",
description="1 if Expert Advisor trading is enabled, 0 otherwise.",
)
self._last_successful_update = meter.create_gauge(
"mt5_last_successful_update_timestamp",
description="Unix timestamp of the last successful history update.",
)
@contextmanager
def record_history_update(
self,
*,
dataset: str,
) -> Iterator[None]:
"""Context manager recording history update duration and failures.
Args:
dataset: Dataset label (e.g. ``"rates"``).
Yields:
None inside the update operation.
"""
attrs = {"dataset": dataset}
start = time.monotonic()
try:
yield
self._history_duration.record(time.monotonic() - start, attrs)
self._last_successful_update.set(time.time(), attrs)
except Exception:
self._history_failures.add(1, attrs)
raise
def add_history_rows(self, count: int, *, dataset: str) -> None:
"""Increment the history rows-written counter.
Args:
count: Number of rows written during this update.
dataset: Dataset label (e.g. ``"rates"``).
"""
self._history_rows.add(count, {"dataset": dataset})
@contextmanager
def record_snapshot_update(self) -> Iterator[None]:
"""Context manager recording snapshot update duration and failures.
Yields:
None inside the snapshot operation.
"""
start = time.monotonic()
try:
yield
self._snapshot_duration.record(time.monotonic() - start, {})
except Exception:
self._snapshot_failures.add(1, {})
raise
def record_account_state(
self,
*,
login: str,
server: str,
balance: float,
equity: float,
margin: float,
margin_free: float,
margin_level: float,
) -> None:
"""Emit account metric gauges.
Args:
login: Account login number (as string; not a password or secret).
server: Broker server name.
balance: Account balance.
equity: Account equity.
margin: Margin used.
margin_free: Free margin.
margin_level: Margin level percentage.
"""
attrs: dict[str, str] = {"login": login, "server": server}
self._account_balance.set(balance, attrs)
self._account_equity.set(equity, attrs)
self._account_margin.set(margin, attrs)
self._account_margin_free.set(margin_free, attrs)
self._account_margin_level.set(margin_level, attrs)
def record_position_state(
self,
*,
login: str,
server: str,
symbol: str,
profit: float,
volume: float,
) -> None:
"""Emit position metric gauges.
Args:
login: Account login number (as string).
server: Broker server name.
symbol: Position symbol.
profit: Floating profit/loss.
volume: Position volume.
"""
attrs: dict[str, str] = {"login": login, "server": server, "symbol": symbol}
self._position_profit.set(profit, attrs)
self._position_volume.set(volume, attrs)
def record_terminal_state(
self,
*,
connected: float,
trade_allowed: float,
trade_expert: float,
) -> None:
"""Emit terminal connection and trading status gauges.
Args:
connected: 1.0 if connected to the broker, 0.0 otherwise.
trade_allowed: 1.0 if broker server allows trading, 0.0 otherwise.
trade_expert: 1.0 if Expert Advisor trading is enabled, 0.0 otherwise.
"""
self._terminal_connected.set(connected, {})
self._terminal_trade_allowed.set(trade_allowed, {})
self._terminal_trade_expert.set(trade_expert, {})
_metrics = _Mt5Metrics()
def configure_metrics(meter: Any) -> None: # noqa: ANN401
"""Configure MT5 metrics using the provided meter.
Args:
meter: An OpenTelemetry ``Meter`` or duck-typed compatible object.
"""
_metrics.configure(meter)
def enable_otel_metrics(
service_name: str = "mt5cli",
readers: list[Any] | None = None,
) -> None:
"""Enable OTel metrics by wiring up an SDK ``MeterProvider`` pipeline.
Requires the ``otel`` optional dependency group:
``pip install "mt5cli[otel]"``.
Args:
service_name: OTel meter/service name used for the ``Resource`` and
the meter itself.
readers: Optional list of metric readers. When *None* (the default),
a :class:`~opentelemetry.sdk.metrics.export.PeriodicExportingMetricReader`
backed by an OTLP HTTP exporter is created automatically
(reads the endpoint from ``OTEL_EXPORTER_OTLP_ENDPOINT``).
Pass a custom list (e.g. ``InMemoryMetricReader`` for tests)
to override.
Raises:
ImportError: If ``opentelemetry-api`` is not installed, or if
``readers`` is *None* and
``opentelemetry-exporter-otlp-proto-http`` is not installed.
"""
if not _OTEL_AVAILABLE:
msg = (
"opentelemetry-api is not installed. "
'Install it with: pip install "mt5cli[otel]"'
)
raise ImportError(msg)
if readers is None:
if _OtelOTLPExporter is None:
msg = (
"opentelemetry-exporter-otlp-proto-http is required for the "
"default OTLP export pipeline. "
'Install it with: pip install "mt5cli[otel]" or pass a '
"custom readers list."
)
raise ImportError(msg)
readers = [_OtelPeriodicReader(_OtelOTLPExporter())] # type: ignore[misc]
resource = _OtelResource.create({"service.name": service_name}) # type: ignore[union-attr]
provider = _OtelMeterProvider(resource=resource, metric_readers=readers) # type: ignore[misc]
_otel_metrics_mod.set_meter_provider(provider) # type: ignore[union-attr]
meter = provider.get_meter(service_name)
configure_metrics(meter)
def get_metrics() -> _Mt5Metrics:
"""Return the global :class:`_Mt5Metrics` instance.
Returns:
The global metric registry (no-op until :func:`configure_metrics` is
called).
"""
return _metrics