d27da02f3f305b5095c4ff53acee2a8b7ec9eff2
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Daichi Narushima
|
d27da02f3f |
feat: add Grafana-ready SQLite observability (#86)
* feat: add Grafana-ready SQLite observability (#79, #80, #81) New `mt5cli/grafana.py` module with idempotent DDL helpers: - `create_snapshot_tables` — five SQLite tables for time-series account, position, order, terminal, and run-status snapshots - `create_grafana_views` — 13 `grafana_*` views with integer epoch-second `time` columns; missing source tables emit warnings and are skipped - `create_grafana_indexes` — 9 performance indexes guarded by column checks - `ensure_grafana_schema` — convenience wrapper calling all three above - Insert helpers: `insert_account_snapshot`, `insert_position_snapshots`, `insert_order_snapshots`, `insert_terminal_snapshot`, `record_snapshot_run` New stable SDK exports in `mt5cli.__init__` and `mt5cli.contract`: - `update_observability` — appends a timestamped snapshot to a SQLite db using an already-connected `Mt5DataClient`; never places orders - `update_observability_with_config` — standalone wrapper that opens and closes the MT5 connection automatically New CLI commands (Collection panel): - `grafana-schema` — idempotent schema setup, no MT5 connection required - `snapshot` — append account/position/order/terminal rows; supports `--symbol`, `--with-account/--no-account`, and equivalent flags All public modules maintain 100 % branch coverage; 968 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply mdformat to docs after grafana observability additions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: bump version to 1.1.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address PR #86 review feedback - Fix unaggregated time in grafana_realized_pnl GROUP BY query (MAX) - Replace O(N) per-symbol API calls with single call + client-side filter - Eliminate double create_snapshot_tables when with_grafana_schema=True - Move grafana imports to module level in sdk.py (remove PLC0415 noqa) - Default with_grafana_schema to False (run grafana-schema once for setup) - Fix README position example to use snapshot_runs for latest snapshot - Update tests to reflect new behavior and correct patch targets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply ruff formatting and sync lock file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address owner review feedback on PR #86 - Filter grafana_*_snapshots views to only expose rows from successful runs (JOIN snapshot_runs WHERE status='ok'), closing the partial-snapshot visibility gap raised in PRRT_kwDORzI_286MvDJ6 - Add issubset column guards for snapshot table index creation, consistent with the rest of create_grafana_indexes (PRRT_kwDORzI_286MvUx1) - Fix README example queries: views expose 'time' not 'observed_at' (PRRT_kwDORzI_286MvUxw) - Correct public-contract.md default for with_grafana_schema (False, not True) and point to grafana-schema command (PRRT_kwDORzI_286MvUxy) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add run_id to snapshot schema and fix stale-positions README query Replace second-level observed_at as the join key between snapshot_runs and snapshot tables with a stable run_id INTEGER PRIMARY KEY. Two runs in the same second now get distinct run_ids, preventing view duplication and cross-contamination from a failed run. Update README example to use snapshot_runs for latest-snapshot lookup so zero-position runs return an empty result instead of stale rows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: format markdown tables Align table column widths in README and public-contract documentation. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: close sqlite connections deterministically * fix: require entry filter in grafana_realized_pnl and add time to grafana_trade_stats grafana_realized_pnl now requires the entry column and filters to close-side deals (entry IN (1, 2, 3)), consistent with grafana_symbol_pnl and grafana_trade_stats. grafana_trade_stats now requires the time column and emits MAX(time_expr) AS "time" so it satisfies the documented Grafana view contract (integer epoch-second time column throughout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: normalize pd.Timestamp time_setup to epoch int in insert_order_snapshots orders_get_as_df() returns datetime-converted columns by default, so time_setup is a pd.Timestamp in normal use. Passing it directly to sqlite3.executemany raises ProgrammingError. Added _to_epoch_int helper that converts datetime.datetime subclasses (including pd.Timestamp) and raw int/float values to integer epoch seconds, returning None for other types. Regression tests cover all four input paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: pre-drop all grafana_* views at start of create_grafana_views Previously, a builder that skipped due to a missing source table or column would not drop the view it owned, leaving stale views referencing gone tables. Now create_grafana_views drops all 13 known grafana_* views before calling any builder, so a schema refresh always removes views whose source has disappeared. Regression test covers the create → drop-source → refresh cycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: expose run_id in snapshot views and drop time from summary views Grafana snapshot views now expose run_id so latest-state queries can use MAX(run_id) instead of the ambiguous second-level MAX(observed_at). grafana_realized_pnl and grafana_trade_stats lose their MAX(time) column and are reclassified as static summary views; their all-time aggregates are not filterable by Grafana time-range selectors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: guard snapshot views against missing run_id and fix README view docs _build_snapshot_view now skips with a warning when the underlying snapshot table exists but lacks a run_id column, preventing a broken view that fails at query time. Adds a regression test for that path. README Grafana section now qualifies that grafana_realized_pnl and grafana_trade_stats are static summary views (no time column) and splits the view table to match docs/api/public-contract.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Daichi Narushima
|
80c3f3f65e |
Make ticks dataset opt-in for collect-history (#87)
* feat: make SQLite tick history opt-in for collect-history
Ticks can grow SQLite databases quickly, so they are excluded from the
default dataset selection. The new DEFAULT_HISTORY_DATASETS constant
(rates, history-orders, history-deals) drives resolve_history_datasets(None),
collect_history(), and update_history(). Callers must pass
--dataset ticks (CLI) or datasets={Dataset.ticks} (SDK) to include ticks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALK71tg75JWrrCKaiShb7b
* chore: reformat docs/index.md table column widths
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALK71tg75JWrrCKaiShb7b
* fix: update stale docstrings and tighten CLI None check
- update_history and ThrottledHistoryUpdater.__init__ docstrings now
state that ticks are opt-in, matching collect_history's wording
- cli.py collect-history uses `is not None` for explicit empty-list safety
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALK71tg75JWrrCKaiShb7b
* docs: update README collect-history to reflect ticks opt-in default
The command table and section intro previously stated ticks were
collected by default ("all four", "rates, ticks, history-orders, and
history-deals"). Both now reflect the new default (rates, history-orders,
history-deals) and note that --dataset ticks is required to include ticks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ALK71tg75JWrrCKaiShb7b
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Daichi Narushima
|
43f632bc40 |
Reorganize CLI help text and command grouping for data/execution clarity (#85)
* feat: clarify CLI/docs scope as generic MT5 data and execution infrastructure - Update app help text and module docstring to describe mt5cli as MT5 data and execution utilities rather than export-only tooling - Group CLI commands under rich_help_panel sections: Data / Export, Execution, and Collection; command names are unchanged for compatibility - Expand order-send docstring to explicitly flag it as the expert raw-request live-trading path; preserve --yes gate - Split docs/index.md Trading section into "Trading State" (read-only) and "Execution (live / mutating)" with close-positions now documented - Add TestHelpText tests verifying top-level panel grouping, order-send expert/live language, and close-positions safety gate coverage Closes #78 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 * chore: trim trailing whitespace in docs/index.md table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 * chore: bump version to 1.0.1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 * chore: update uv.lock for version 1.0.1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 * fix: address review feedback on CLI/docs scope PR - Remove dead help invocation in test_order_send_help_mentions_expert_and_raw (the result was immediately overwritten by result2) - Strengthen assertion from `or` to `and`; both "raw" and "expert" are present in the docstring so disjunction masked a potential regression - Split into two `assert` statements to satisfy PT018 (ruff) - Fix docs/index.md inaccuracy: order-check has no --yes gate; clarify that only order-send and close-positions require confirmation for live execution - Move order-check from "Execution" rich_help_panel to "Data / Export" so the Execution panel name is truthful (order-check is read-only) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 * docs: move order-check out of Execution section into Trading State order-check is read-only and now lives in the CLI's Data / Export panel, so documenting it under "Execution (live / mutating)" was inconsistent. Moved it to the Trading State table. The Execution section now only lists order-send and close-positions, both of which require --yes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yH6esaqc5D1cmo1dK2Ur9 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Daichi Narushima
|
dfe80ce500 |
feat: add close-positions CLI and replace_symbol projection mode (#65 #66) (#67)
* feat: add close-positions CLI command and replace_symbol projection mode (#65 #66) Part 1 — close-positions CLI (#65): - Add `close-positions` subcommand delegating to `close_open_positions()`. - Accepts repeated `--symbol` and `--ticket` filters (AND semantics). - Supports `--dry-run` (no `--yes` required); live execution requires `--yes`. - Fails closed with `BadParameter` when neither `--symbol` nor `--ticket` is given. - Exports normalized `OrderExecutionResult` list as a DataFrame (request/response serialized as JSON strings for clean CSV/JSON/Parquet/SQLite output). - `order-send` remains the raw expert path; `close-positions` is the safer high-level helper that builds correct close requests automatically. Part 2 — ProjectionMode and replace_symbol (#66): - Add `ProjectionMode = Literal["add", "replace_symbol"]` type alias. - Add optional `projection_mode` parameter to `calculate_symbol_group_margin_ratio`. Default `"add"` preserves existing additive behavior. `"replace_symbol"` subtracts current margin for `new_symbol`, then adds candidate margin — the subtraction and addition are atomic (suppressed together). - Export `ProjectionMode` from `mt5cli` and add to `STABLE_SDK_EXPORTS`. - No mteor-specific strategy, risk-threshold, or policy logic added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: remove unused ProjectionMode import in test_contracts.py The parametrized test_stable_exports_are_importable_from_package_root already covers ProjectionMode via hasattr(mt5cli, name). Ruff correctly flagged the explicit top-level import as unused (F401). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Bump version to v0.9.6 * fix: address PR #67 review feedback - Floor replace_symbol margin subtraction at zero to prevent negative ratio - Serialize response unconditionally via json.dumps (null for dry-run rows) - Return a schema-preserving empty DataFrame when results list is empty - Add test: --dry-run --yes precedence (dry-run wins, no order_send) - Add test: zero-match filter produces empty JSON array with exit 0 - Move projection_mode prose to stable trading section in docs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add runtime validation for projection_mode in calculate_symbol_group_margin_ratio Unsupported values previously silently fell through as "add". The new _validate_projection_mode helper raises ValueError with a message that names the bad value and the two accepted modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: agent <agent@localhost> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Daichi Narushima
|
9356d5dcdf |
Consolidate duplicated export and history streaming helpers (#29)
* Consolidate duplicated export and history streaming helpers. Reduce repeated CLI export plumbing, shared per-symbol SQLite writes, and test mock setup without changing public behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * Bump version from 0.7.0 to 0.7.1. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daichi Narushima
|
0fad55d609 |
Refactor MT5 constant parsing to delegate to pdmt5 >= 0.3.0 (#28)
* Refactor MT5 constant parsing to delegate to pdmt5 >= 0.3.0 Replace local TIMEFRAME_MAP, TICK_FLAG_MAP, and parser helpers with thin compatibility wrappers around pdmt5. COPY_TICKS flags now use real MT5 values (ALL=-1, INFO=1, TRADE=2). Click parameter types validate all inputs through the wrappers. Update tests and docs to describe the pdmt5/mt5cli/mt5api layering. Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com> * Fix timeframe defaults and COPY_TICKS flag defaults after pdmt5 migration Use short timeframe aliases for default history collection and granularity naming via pdmt5.get_timeframe_name. Set CLI/SDK default tick flags to ALL (-1) instead of the legacy mt5cli-only value. Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com> * Address CI lint failure and PR review feedback Fix ruff import ordering in history.py. Use ALL string defaults for CLI tick flags, isolate TICK_FLAG_MAP as a dict snapshot, derive flag names from pdmt5, reuse TIMEFRAME_NAMES for default history timeframes, and add tests for prefix stripping and TIMEFRAME_ key filtering. Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com> * Bump version to 0.7.0 Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com> |
||
|
Daichi Narushima
|
9957b0a1de |
[codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader * Fix MT5 latest rates connection reuse * Make MT5 summary export safe * Address PR review feedback for SDK and SQLite rate loader. Reuse parse_sqlite_timestamp for rate time parsing, document empty-table errors, tighten tests, and align docs with require_existing=True. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daichi Narushima
|
b2bb2ad0a0 |
Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names. Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules. Co-authored-by: Cursor <cursoragent@cursor.com> * Add reusable export, tick-window, and margin helpers for downstream tools. Expose SQLite append/dedup export, recent tick retrieval, and minimum margin summary through the SDK and CLI so projects like mteor can depend on mt5cli instead of duplicating MT5 data plumbing. Co-authored-by: Cursor <cursoragent@cursor.com> * Bump version to 0.4.3. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback for rate view resolution and SDK helpers. Harden SQLite read-only connections, tighten view discovery, improve recent_ticks fetch efficiency, default SQLite export to append, and expand tests and docs. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix read-only SQLite URI construction on Windows. Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daichi Narushima
|
c4232bf44d |
Add incremental SQLite history SDK (#16)
* Add incremental SQLite history SDK for automated pipelines. Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix incremental history deals and stale rate view cleanup. Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix incremental deal filtering edge cases Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback for incremental SQLite history. Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers. Co-authored-by: Cursor <cursoragent@cursor.com> * Document collect-history SQLite schema with ER diagram. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix account-event filtering and drop legacy rates resume. Account events must follow only account_event_start, not per-symbol trade cursors. Require normalized rates schema and fail fast when timeframe is missing. Co-authored-by: Cursor <cursoragent@cursor.com> * Validate normalized rates schema before incremental resume. Require symbol, timeframe, and time on existing rates tables with clear ValueError messages, and add regression tests for malformed schemas. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daichi Narushima
|
5b44318d55 |
Add programmatic SDK and refactor mt5cli into cli, sdk, and utils (#15)
* Refactor cli.py into cli and utils modules Extract constants, enums, Click parameter types, and parse/export utility functions into a new mt5cli/utils.py module, keeping the typer app, commands, and collect-history SQLite helpers in cli.py. https://claude.ai/code/session_016JwSEhPyq6phXySktQ1FGU * Address review comments * Add programmatic SDK layer for read-only MT5 data collection. Expose Mt5CliClient and collect_history through the package API while keeping CLI commands as thin adapters over the SDK. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden SDK connection lifecycle and scope internal helpers as private. Co-authored-by: Cursor <cursoragent@cursor.com> * Export build_config in the public API and bump version to 0.4.0. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove duplicate scripts/ in favor of local-qa skill script. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daichi Narushima
|
da74c11087 |
Add collect-history command for bulk data collection (#14)
* Add collect-history command for bulk SQLite export Bundles rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database. Tick collection uses copy_ticks_range_as_df with a --flags option defaulting to ALL. With --with-views, optional cash_events and positions_reconstructed views are derived from history_deals when the required columns are present. * Extend collect-history with datasets, if-exists, timeframe, view fixes - Fetch history-orders and history-deals per symbol so --symbol applies consistently across all four datasets. - Add repeatable --dataset (rates, ticks, history-orders, history-deals) so ticks are no longer required and any subset can be collected. - Add --if-exists append|replace|fail to control SQLite table conflict behavior instead of hard-coding replace. - Record the requested timeframe in a timeframe column on the rates table so appended runs at different timeframes stay distinguishable. - Fix positions_reconstructed to exclude positions with no closing deals, use volume-weighted open/close prices, and report reversal deals (DEAL_ENTRY_INOUT) via volume_reversal / reversal_count without contributing to weighted prices. - Update tests, README, docs, and skill to match. * Address collect-history review feedback * Stream collect-history writes per symbol * Address PR cleanup for collect-history --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Daichi Narushima
|
b8ce76c9d6 |
Add CLI commands for remaining pdmt5.dataframe methods (#9)
* Add CLI commands for remaining pdmt5.dataframe methods Add the following subcommands so the CLI fully covers pdmt5.dataframe's public *_as_df methods: - version - last-error - symbol-info-tick - market-book - order-check (request via inline JSON or @path/to/file.json) - order-send (request via inline JSON or @path/to/file.json) Also export a new parse_request helper for parsing JSON order requests, add tests for every new command, and update the README and docs command tables. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version to 0.2.0 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Address PR review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden CLI error assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com> |
||
|
Claude
|
527f9213da |
Add mt5cli package with CLI for MetaTrader 5 data export
Create a standalone CLI package that uses pdmt5 as a dependency to export MetaTrader 5 data to CSV, JSON, Parquet, and SQLite3 formats. Includes 12 subcommands for rates, ticks, account/terminal info, symbols, orders, positions, and trading history, along with comprehensive test coverage. https://claude.ai/code/session_01YW3YHru8wRH9dvHnBX7xf1 |