Files
mt5cli/pyproject.toml
T
Daichi Narushima d27da02f3f feat: add Grafana-ready SQLite observability (#86)
* feat: add Grafana-ready SQLite observability (#79, #80, #81)

New `mt5cli/grafana.py` module with idempotent DDL helpers:
- `create_snapshot_tables` — five SQLite tables for time-series account,
  position, order, terminal, and run-status snapshots
- `create_grafana_views` — 13 `grafana_*` views with integer epoch-second
  `time` columns; missing source tables emit warnings and are skipped
- `create_grafana_indexes` — 9 performance indexes guarded by column checks
- `ensure_grafana_schema` — convenience wrapper calling all three above
- Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`,
  `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run`

New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`:
- `update_observability` — appends a timestamped snapshot to a SQLite db
  using an already-connected `Mt5DataClient`; never places orders
- `update_observability_with_config` — standalone wrapper that opens and
  closes the MT5 connection automatically

New CLI commands (Collection panel):
- `grafana-schema` — idempotent schema setup, no MT5 connection required
- `snapshot` — append account/position/order/terminal rows; supports
  `--symbol`, `--with-account/--no-account`, and equivalent flags

All public modules maintain 100 % branch coverage; 968 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply mdformat to docs after grafana observability additions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: bump version to 1.1.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address PR #86 review feedback

- Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX)
- Replace O(N) per-symbol API calls with single call + client-side filter
- Eliminate double create_snapshot_tables when with_grafana_schema=True
- Move grafana imports to module level in sdk.py (remove PLC0415 noqa)
- Default with_grafana_schema to False (run grafana-schema once for setup)
- Fix README position example to use snapshot_runs for latest snapshot
- Update tests to reflect new behavior and correct patch targets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: apply ruff formatting and sync lock file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address owner review feedback on PR #86

- Filter grafana_*_snapshots views to only expose rows from successful
  runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot
  visibility gap raised in PRRT_kwDORzI_286MvDJ6
- Add issubset column guards for snapshot table index creation, consistent
  with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1)
- Fix README example queries: views expose 'time' not 'observed_at'
  (PRRT_kwDORzI_286MvUxw)
- Correct public-contract.md default for with_grafana_schema (False, not
  True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: add run_id to snapshot schema and fix stale-positions README query

Replace second-level observed_at as the join key between snapshot_runs
and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs
in the same second now get distinct run_ids, preventing view duplication
and cross-contamination from a failed run. Update README example to use
snapshot_runs for latest-snapshot lookup so zero-position runs return an
empty result instead of stale rows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: format markdown tables

Align table column widths in README and public-contract documentation.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix: close sqlite connections deterministically

* fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats

grafana_realized_pnl now requires the entry column and filters to
close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl
and grafana_trade_stats. grafana_trade_stats now requires the time column
and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana
view contract (integer epoch-second time column throughout).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots

orders_get_as_df() returns datetime-converted columns by default, so
time_setup is a pd.Timestamp in normal use. Passing it directly to
sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper
that converts datetime.datetime subclasses (including pd.Timestamp) and
raw int/float values to integer epoch seconds, returning None for other
types. Regression tests cover all four input paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: pre-drop all grafana_* views at start of create_grafana_views

Previously, a builder that skipped due to a missing source table or column
would not drop the view it owned, leaving stale views referencing gone
tables. Now create_grafana_views drops all 13 known grafana_* views before
calling any builder, so a schema refresh always removes views whose source
has disappeared. Regression test covers the create → drop-source → refresh
cycle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: expose run_id in snapshot views and drop time from summary views

Grafana snapshot views now expose run_id so latest-state queries can use
MAX(run_id) instead of the ambiguous second-level MAX(observed_at).
grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column
and are reclassified as static summary views; their all-time aggregates
are not filterable by Grafana time-range selectors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: guard snapshot views against missing run_id and fix README view docs

_build_snapshot_view now skips with a warning when the underlying
snapshot table exists but lacks a run_id column, preventing a broken
view that fails at query time. Adds a regression test for that path.

README Grafana section now qualifies that grafana_realized_pnl and
grafana_trade_stats are static summary views (no time column) and splits
the view table to match docs/api/public-contract.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 07:55:26 +09:00

189 lines
5.0 KiB
TOML

[project]
name = "mt5cli"
version = "1.1.0"
description = "Generic MT5 data and execution infrastructure for Python applications"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
license = "MIT"
license-files = ["LICENSE"]
readme = "README.md"
requires-python = ">= 3.11, < 3.14"
dependencies = [
"pdmt5>=1.0.0",
"click >= 8.1.0",
"typer >= 0.15.0",
]
classifiers = [
"Development Status :: 3 - Alpha",
"Environment :: Console",
"License :: OSI Approved :: MIT License",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Intended Audience :: Financial and Insurance Industry",
"Topic :: Office/Business :: Financial :: Investment",
]
[project.optional-dependencies]
parquet = ["pyarrow >= 19.0.0"]
[project.scripts]
mt5cli = "mt5cli.cli:main"
[project.urls]
Repository = "https://github.com/dceoy/mt5cli.git"
[tool.uv]
required-environments = ["platform_system == 'Windows'"]
[dependency-groups]
dev = [
"ruff >= 0.11.0",
"pyright >= 1.1.407",
"pytest>=9.0.3",
"pytest-mock >= 3.12.0",
"pytest-cov >= 5.0.0",
"pandas-stubs >= 2.2.3.250527",
"pyarrow >= 19.0.0",
"mkdocs >= 1.6.1",
"mkdocs-material >= 9.7.6",
"mkdocstrings[python] >= 1.0.4",
"pymdown-extensions >= 10.21.2",
]
[tool.ruff]
line-length = 88
exclude = ["build", ".venv"]
preview = true
[tool.ruff.lint]
select = [
"F", # Pyflakes (F)
"E", # pycodestyle error (E)
"W", # pycodestyle warning (W)
"C90", # mccabe (C90)
"I", # isort (I)
"N", # pep8-naming (N)
"D", # pydocstyle (D)
"UP", # pyupgrade (UP)
"S", # flake8-bandit (S)
"B", # flake8-bugbear (B)
"C4", # flake8-comprehensions (C4)
"SIM", # flake8-simplify (SIM)
"ARG", # flake8-unused-arguments (ARG)
"PD", # pandas-vet (PD)
"PLC", # Pylint convention (PLC)
"PLE", # Pylint error (PLE)
"PLR", # Pylint refactor (PLR)
"PLW", # Pylint warning (PLW)
"FLY", # flynt (FLY)
"NPY", # NumPy-specific rules (NPY)
"PERF", # Perflint (PERF)
"FURB", # refurb (FURB)
"RUF", # Ruff-specific rules (RUF)
"YTT", # flake8-2020 (YTT)
"ANN", # flake8-annotations (ANN)
"ASYNC", # flake8-async (ASYNC)
"BLE", # flake8-blind-except (BLE)
"FBT", # flake8-boolean-trap (FBT)
"A", # flake8-builtins (A)
"COM", # flake8-commas (COM)
"DTZ", # flake8-datetimez (DTZ)
"T10", # flake8-debugger (T10)
"DJ", # flake8-django (DJ)
"EM", # flake8-errmsg (EM)
"EXE", # flake8-executable (EXE)
"FA", # flake8-future-annotations (FA)
"ISC", # flake8-implicit-str-concat (ISC)
"ICN", # flake8-import-conventions (ICN)
"LOG", # flake8-logging (LOG)
"G", # flake8-logging-format (G)
"INP", # flake8-no-pep420 (INP)
"PIE", # flake8-pie (PIE)
"T20", # flake8-print (T20)
"PYI", # flake8-pyi (PYI)
"PT", # flake8-pytest-style (PT)
"Q", # flake8-quotes (Q)
"RSE", # flake8-raise (RSE)
"SLF", # flake8-self (SLF)
"SLOT", # flake8-slots (SLOT)
"TID", # flake8-tidy-imports (TID)
"TCH", # flake8-type-checking (TCH)
"INT", # flake8-gettext (INT)
"PTH", # flake8-use-pathlib (PTH)
"TD", # flake8-todos (TD)
"FIX", # flake8-fixme (FIX)
"ERA", # eradicate (ERA)
"PGH", # pygrep-hooks (PGH)
"TRY", # tryceratops (TRY)
"FAST", # FastAPI (FAST)
"AIR", # Airflow (AIR)
"DOC", # pydoclint (DOC)
]
ignore = [
"COM812", # missing-trailing-comma
"FBT001", # boolean-type-hint-positional-argument
"FBT002", # boolean-default-value-positional-argument
]
[tool.ruff.lint.per-file-ignores]
"tests/**/*.py" = [
"DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring
"PLC2701", # Private name import
"PLR0904", # Too many public methods
"PLR2004", # Magic value used in comparison
"PLR6301", # Method could be function/static/classmethod
"S101", # Use of assert (acceptable in tests)
"S106", # Possible hardcoded password
"SLF001", # Private member accessed
]
[tool.ruff.lint.pydocstyle]
convention = "google"
[tool.ruff.lint.pylint]
max-args = 10
max-public-methods = 40
[tool.pyright]
exclude = ["build", ".venv"]
venvPath = "."
venv = ".venv"
typeCheckingMode = "strict"
reportMissingTypeStubs = false
[tool.pytest.ini_options]
addopts = [
"--cov=mt5cli",
"--cov-branch",
"--doctest-modules",
"--capture=no",
]
pythonpath = ["."]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
minversion = "6.0"
[tool.coverage.run]
source = ["mt5cli"]
omit = [
"**/__init__.py",
"**/__main__.py",
"tests/**",
]
[tool.coverage.report]
show_missing = true
fail_under = 100
exclude_also = [
"if TYPE_CHECKING:",
"^\\s+\\.\\.\\.$",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"