Compare commits

...

30 Commits

Author SHA1 Message Date
Daichi Narushima 96c75f7852 Add account-wide projected margin ratio helper (#60)
* feat: add account projected margin ratio helper

* Bump version to v0.9.4

* fix: address account margin ratio review feedback

* fix: simplify account margin ratio errors
2026-06-24 03:43:52 +09:00
Daichi Narushima 292fac899a Add generic trading helpers and reduce public API tiers (#58)
* feat: add generic trading helpers and API tiers

* Bump version to v0.9.3

* fix: require symbol digits for trailing stops

* fix: allow side-specific trailing stop ticks

* test: enforce complete public export tiers

* docs: align public contract tiers

* refactor: remove legacy public supports
2026-06-24 01:58:32 +09:00
Daichi Narushima 9ac3b885c3 test: add explicit unit tests for calculate_positions_margin_by_symbol and calculate_positions_margin_safe (#50) (#53)
* test: add explicit unit tests for calculate_positions_margin_by_symbol and calculate_positions_margin_safe (#50)

Covers all acceptance criteria: partial failure with warning log, all-fail,
empty symbol list with no-broker-call assertion, duplicate deduplication,
successful aggregation with first-seen key order, suppress_errors=False
propagation, and three calculate_positions_margin_safe cases (partial skip,
all-fail → 0.0, empty list → 0.0).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: fix warning log assertion and parametrize suppress_errors=False test

- Use record.getMessage() + levelno check instead of record.message, which
  is only populated after formatting and can return an empty string.
- Parametrize test_one_symbol_fails_suppress_errors_false over all three
  exception types caught by the implementation (Mt5TradingError,
  Mt5RuntimeError, AttributeError) so any future narrowing of the except
  tuple would be caught by tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: shorten docstring to fit 88-char line limit

* style: shorten docstring to fit 88-char line limit

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 21:41:30 +09:00
Daichi Narushima 823cb5b0a4 Revert "Bump version to v0.9.3 (#51)" (#52)
This reverts commit f1ada55bce.
2026-06-23 19:17:42 +09:00
agent 1c57be5c44 fix: centralize tick price validation in calculate_spread_ratio and determine_order_limits (#52)
Replaces manual isinstance/<=0 checks in calculate_spread_ratio() and
determine_order_limits() with _valid_tick_price(), ensuring NaN, inf,
-inf, zero, negative, bool, and invalid-string tick values are
consistently rejected across all trading helpers.

Adds regression tests covering numeric-string acceptance and every
invalid-value category for both functions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 09:48:13 +00:00
Daichi Narushima f1ada55bce Bump version to v0.9.3 (#51) 2026-06-23 18:30:18 +09:00
agent d292fbb9d9 feat: centralize tick price validation and add resilient position margin helpers (#49, #50)
Add _valid_tick_price() internal helper that returns a positive finite float
from a tick dict or None for any invalid value (missing, None, NaN, infinite,
zero, negative, or unsupported type). Refactor five existing bid/ask validation
sites in trading.py to use it, removing duplicated isinstance/isfinite checks.

Add calculate_positions_margin_by_symbol() which computes margin per unique
symbol independently using the existing strict calculate_positions_margin(),
with first-seen deduplication and configurable error suppression
(Mt5TradingError, Mt5RuntimeError, AttributeError) via suppress_errors=.

Add calculate_positions_margin_safe() as a thin sum wrapper with
suppress_errors=True, returning 0.0 on empty or fully-failed inputs.

Both new helpers are exported from mt5cli, added to STABLE_SDK_EXPORTS, and
documented in docs/api/public-contract.md. Existing strict behavior of
calculate_positions_margin() is unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 07:30:02 +00:00
Daichi Narushima 8e53212a24 fix: always use mt5cli calculate_volume_by_margin to prevent LACK OF FUNDS (#48) 2026-06-23 14:01:46 +09:00
Daichi Narushima b878a61c07 fix: re-verify normalized volume margin in calculate_volume_by_margin (#46)
* fix: re-verify normalized volume margin before returning from calculate_volume_by_margin

For CFDs, index products, and tiered-margin instruments, the initial
min-lot margin estimate can be optimistic; the normalized stepped volume
may require more margin than available_margin.  After computing the
normalized volume, step down by volume_step until order_calc_margin
confirms affordability, or return 0.0 if no step is affordable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: fix ruff line-length violations in calculate_volume_by_margin tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: use integer step index and add actual>0 guard in calculate_volume_by_margin

Replace float-subtraction loop with integer step index to eliminate
accumulation rounding error and add `actual > 0` guard so a broker
returning zero/negative margin is never accepted as affordable.
Inline `capped` to keep local-variable count within Ruff PLR0914 limit.
Update docstring to reflect re-verification behaviour and 0.0 fallback.
Tighten test assertion from `volume > 0` to the symbol's valid range.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Bump version to v0.9.1

* perf: replace linear step-down scan with binary search in calculate_volume_by_margin

Resolves the P2 review finding: the previous O(n) loop called
order_calc_margin once per volume step, making sizing appear hung for
symbols with a large step range or small volume_step.

Binary search over the integer step index finds the largest affordable
step in O(log n) IPC calls (≈17 for a 99 999-step range vs up to 99 999
in the worst case). Monotonicity of broker margin with volume is assumed,
which holds for standard linear margin schedules.

To stay within the PLR0914 local-variable limit the steps variable is
inlined into hi and the tick temporary is eliminated by accessing the
snapshot dict directly. Error messages still go via msg to satisfy EM102.

Two existing tests are updated to match the binary-search call sequence.
A new regression test (volume_min=0.01, volume_max=1000.0) configures
a tiered-margin mock with its threshold at step 50000 and asserts that
the total order_calc_margin call count does not exceed 25.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: remove obsolete TC003 per-file-ignore for history.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: agent <agent@localhost>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 04:40:00 +09:00
dceoy 0610ea732c fix: handle NumPy object rate timestamps 2026-06-22 23:01:31 +09:00
Daichi Narushima 82a39731ed feat: add fetch_latest_closed_rates_indexed and allow_whole_dollar_env opt-in (#45)
* feat: add fetch_latest_closed_rates_indexed and allow_whole_dollar_env opt-in (#43, #44)

Closes #43: add fetch_latest_closed_rates_indexed(client, *, symbol,
granularity, count) -> pd.DataFrame to mt5cli/trading.py. Internally
reuses fetch_latest_closed_rates_for_trading_client(), converts the
"time" column to a UTC-aware DatetimeIndex named "time", and drops the
original column. Exported from trading.__all__, mt5cli.__init__, and
STABLE_SDK_EXPORTS.

Closes #44: extend substitute_env_placeholders() with opt-in
allow_whole_dollar_env=False that expands whole-value $ENV_NAME strings
(entire string must be exactly $IDENTIFIER). Threaded through
build_config(), resolve_account_spec(), and resolve_account_specs() with
the same default=False. Partial strings like "plan$pass", "abc$ENV", or
"$ENV-suffix" are never expanded.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: align Markdown table columns in docs and skill file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: treat numeric (float64) epoch seconds as UTC in _rate_time_to_utc

After DataFrame concat or NA upcast the time column becomes float64, which
is still epoch seconds. Using is_numeric_dtype instead of is_integer_dtype
fixes the silent misalignment. Using series.to_numpy() before passing to
pd.to_datetime avoids the redundant pd.DatetimeIndex() wrapper and aligns
with how existing rate-time normalization in schemas.py handles numeric
timestamps.

Add test_converts_float_epoch_seconds_to_utc_datetime_index to cover the
regression. Add a doc note clarifying that build_config cannot expand
login since that parameter is int | None.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: reject NaT values after rate timestamp conversion in _rate_time_to_utc

pd.to_datetime() silently produces NaT for None/NaN inputs rather than
raising, so the function could return a DatetimeIndex containing NaT
despite documenting invalid timestamps as a ValueError. Check any(idx.isna())
after conversion and raise with a clear message.

Add test_raises_on_nat_time_column to cover the regression.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Bump version to v0.9.0

* fix: handle object numeric rate timestamps

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-22 22:52:19 +09:00
Daichi Narushima c4a4253fbc feat: stable SDK helpers for volume, margin, and closed bars (#39–#41) (#42)
* feat: add stable SDK helpers for volume, margin, and closed bars (#39, #40, #41)

Expose generic trading utilities in the stable downstream SDK so applications
like mteor can drop local MT5 adapter code:

- normalize_order_volume() for broker step/min/max sizing
- estimate_order_margin() and calculate_positions_margin() for margin totals
- fetch_latest_closed_rates_for_trading_client() for closed bars from Mt5TradingClient

Update STABLE_SDK_EXPORTS, package-root exports, docs, and unit tests.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* chore: bump version to 0.8.3

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address PR review feedback on volume cap, rate time, and margin grouping

- Re-apply volume_max after step normalization in normalize_order_volume()
- Drop misleading non-time index reset branch in _ensure_rate_time_column()
- Group positions by (symbol, side) before margin estimation
- Add branch-coverage tests for tick price validation and volume cap edge case

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address remaining PR review threads on docs and DatetimeIndex

- Rename unnamed DatetimeIndex column to time after reset_index()
- Guard estimate_order_margin example on positive normalized volume
- Document calculate_positions_margin skip vs error propagation behavior
- Add test for unnamed DatetimeIndex branch coverage

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: harden stable SDK margin, rate fetch, and volume normalization

- Wrap order_calc_margin conversion and reject None/non-numeric results
- Validate fetched rate objects are DataFrames before time normalization
- Return 0.0 for non-finite volume inputs and constraints in normalize_order_volume

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: reject non-finite volumes in margin estimation helpers

Use _is_positive_finite_number() in estimate_order_margin() and
calculate_positions_margin() so NaN/inf volumes never reach broker calls.
Add focused tests and document non-finite volume skipping in trading.md.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: guard symbol filter in calculate_positions_margin for empty frames

Return 0.0 before filtering when positions are empty or lack a symbol column.
Add regression tests for filtered calls on malformed position frames.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-19 01:02:18 +09:00
dceoy 9f2968cc98 Update .agents/skills/pr-feedback-triage/SKILL.md 2026-06-19 00:21:33 +09:00
Daichi Narushima 7de3ce0b7a feat: add injectable update_backend to ThrottledHistoryUpdater (#38)
* feat: add injectable update_backend to ThrottledHistoryUpdater

Allow downstream applications to substitute the history update backend via
the ThrottledHistoryUpdater constructor without monkey-patching
mt5cli.sdk.update_history. Defaults to update_history for backward
compatibility.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* chore: fix lint and format after QA

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* chore: bump version to 0.8.2

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: use explicit None check for ThrottledHistoryUpdater backend

Only None selects the default update_history backend so falsy callable
objects with __bool__ returning False are preserved as custom backends.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-18 22:58:09 +09:00
Daichi Narushima 897f7f0a0d docs: stable SDK contract and strategy-neutral order helpers (#37) 2026-06-18 19:12:11 +09:00
Daichi Narushima d156dd7176 [codex] fix mt5 adapter APIs (#36)
* fix mt5 adapter APIs

* address PR feedback

* fix zero ratio minimum volume sizing

* Bump version to v0.8.0
2026-06-15 02:47:05 +09:00
dceoy 307d6f5320 docs: restructure AGENTS.md with concise repository guidance
Align agent instructions with the streamlined project structure, QA workflow, and security notes used elsewhere in the repo.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-14 23:07:13 +09:00
Daichi Narushima 8031389a67 Add GitHub CodeQL analysis to CI workflow (#35)
* chore: add GitHub CodeQL analysis to CI workflow

Enable automated security scanning with GitHub CodeQL to detect potential vulnerabilities in Python code.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* chore: run CodeQL analysis on pull requests

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add checks and statuses read permissions for dependabot auto-merge.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-14 22:54:28 +09:00
dceoy fdf5e08d31 Add .agents/skills/pr-feedback-triage/SKILL.md 2026-06-14 21:16:52 +09:00
dceoy 254c159ad5 Bump version to v0.7.2 2026-06-13 01:34:03 +09:00
Daichi Narushima 78c49238cf feat: stable MT5Client public API and infrastructure layer (#30)
* feat: add stable MT5Client public API and infrastructure layer

Introduce a reusable public API for downstream trading applications:

- MT5Client as the primary client abstraction with order_check/order_send
- schemas module with DataKind contracts, validation, and normalization
- converters, exceptions, retry, and storage facade modules
- CLI order commands now route through MT5Client
- connected_client made public; retry logic centralized
- Contract tests for API surface, schemas, and storage round-trips
- README and docs updated with Python API usage examples

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: correct time coercion, broker-safe symbols, and execution docs

- Normalize MT5 time columns with correct second/millisecond units
- Coerce all present known MT5 time fields, including optional order times
- Preserve broker symbol casing in normalize_symbol()
- Document order_send() as a live execution primitive with clear scope boundaries
- Add contract tests for timestamp and symbol normalization behavior

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-13 01:32:03 +09:00
Daichi Narushima 9356d5dcdf Consolidate duplicated export and history streaming helpers (#29)
* Consolidate duplicated export and history streaming helpers.

Reduce repeated CLI export plumbing, shared per-symbol SQLite writes, and test mock setup without changing public behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version from 0.7.0 to 0.7.1.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 23:14:34 +09:00
Daichi Narushima 0fad55d609 Refactor MT5 constant parsing to delegate to pdmt5 >= 0.3.0 (#28)
* Refactor MT5 constant parsing to delegate to pdmt5 >= 0.3.0

Replace local TIMEFRAME_MAP, TICK_FLAG_MAP, and parser helpers with thin
compatibility wrappers around pdmt5. COPY_TICKS flags now use real MT5 values
(ALL=-1, INFO=1, TRADE=2). Click parameter types validate all inputs through
the wrappers. Update tests and docs to describe the pdmt5/mt5cli/mt5api layering.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Fix timeframe defaults and COPY_TICKS flag defaults after pdmt5 migration

Use short timeframe aliases for default history collection and granularity
naming via pdmt5.get_timeframe_name. Set CLI/SDK default tick flags to ALL
(-1) instead of the legacy mt5cli-only value.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Address CI lint failure and PR review feedback

Fix ruff import ordering in history.py. Use ALL string defaults for CLI tick
flags, isolate TICK_FLAG_MAP as a dict snapshot, derive flag names from pdmt5,
reuse TIMEFRAME_NAMES for default history timeframes, and add tests for prefix
stripping and TIMEFRAME_ key filtering.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Bump version to 0.7.0

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-11 23:22:50 +09:00
dceoy d654b82f9d Bump version from 0.6.0 to 0.6.1.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 19:36:34 +09:00
Daichi Narushima b5e82e71c7 Add trading session helpers and extend ThrottledHistoryUpdater (#25)
* Add trading session helpers and extend ThrottledHistoryUpdater

Introduce mt5cli.trading with mt5_trading_session() for Mt5TradingClient
lifecycle management and reusable operational helpers for position-side
detection, margin/volume sizing, and protective order price derivation.

Extend ThrottledHistoryUpdater to validate inputs before updates and to
optionally suppress ValueError, OSError, and missing-method errors without
advancing the throttle timestamp.

Export the new helpers from mt5cli.__init__, add unit tests with mocked
clients, and document migration guidance for downstream projects such as
mteor.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Narrow ThrottledHistoryUpdater suppress_errors handling (#27)

* Narrow ThrottledHistoryUpdater suppress_errors for MT5 capability only

Remove broad AttributeError/TypeError handling from recoverable errors.
Add _is_mt5_client_capability_error() to detect missing history API methods
or non-callable client attributes by message and attribute name.

Generic AttributeError/TypeError values always propagate even when
suppress_errors=True. Update docs and tests accordingly.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Detect non-callable history client methods in suppress_errors

Address review feedback: when a history API attribute exists but is not
callable, Python raises a generic TypeError. Inspect the traceback for
mt5cli.history client call sites so these capability mismatches are still
suppressed without matching all TypeError values.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Address PR review feedback on trading helpers

- Resolve history module path once at import time
- Only treat non-callable TypeErrors as capability errors at the raise site
- Validate SL/TP ratios in determine_order_limits
- Add tests for margin_free edge cases, body-raise shutdown, and internal TypeError propagation
- Clarify ThrottledHistoryUpdater suppress_errors docs
- Split README migration example into trading vs read-only history sessions

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Tighten protective ratio validation and clamp negative margin_free

Add _require_protective_ratio enforcing 0 <= ratio < 1 for SL/TP limits so
a ratio of 1.0 cannot produce zero protective prices. Clamp negative
margin_free to 0.0 in calculate_margin_and_volume before sizing.

Add boundary and negative-margin tests; document constraints in trading API
docs.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-11 19:32:52 +09:00
Daichi Narushima 18df96872b Add closed-bar rate helpers (v0.6.0) (#26)
* Add closed-bar rate helpers and bump version to 0.6.0.

Expose drop_forming_rate_bar and multi-account collectors so downstream apps no longer need count+1 fetches and manual bar trimming.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump pygments to 2.20.0 to fix CVE-2026-4539 ReDoS advisory.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback on closed-bar rate collection.

Validate count and start_pos before MT5 fetches, avoid redundant frame copies, clarify empty-series errors, and expand test coverage.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Include symbol and timeframe in empty closed-rate error messages.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 02:30:48 +09:00
Daichi Narushima 5b1d54bfe9 Add resilient multi-account orchestration helpers (#22)
* Add SDK orchestration helpers for resilient multi-account collection

- collect_latest_rates_for_accounts_with_retries(): exponential-backoff
  retries around collect_latest_rates_for_accounts(), retrying only
  Mt5TradingError/Mt5RuntimeError and re-raising on exhaustion.
- resolve_account_spec()/resolve_account_specs() and
  substitute_env_placeholders(): merge explicit overrides over AccountSpec
  fields and expand ${ENV_VAR} placeholders, raising ValueError on missing
  variables.
- ThrottledHistoryUpdater: monotonic-clock throttled wrapper around
  update_history() with should_update()/update() and opt-in suppress_errors.
- load_rate_series_by_granularity(): rate-series loader keyed by
  (symbol | None, granularity_name).
- Export new APIs, add unit tests (100% coverage), and document in README
  and docs/api.

* chore: bump version from 0.5.1 to 0.5.3 (#24)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: resolve leftover merge conflict markers in version files

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address PR review feedback on SDK orchestration helpers

- Use single-pass env substitution to avoid TOCTOU KeyError
- Apply backoff_base to all retry delays (backoff_base ** (attempt + 1))
- Preserve integer logins in resolve_account_spec; hide login in repr
- Fix docs examples (env ordering, while True loop, backoff comment)
- Parametrize suppress_errors tests for MT5 and SQLite errors

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-10 00:15:07 +09:00
Daichi Narushima ad9e513253 [codex] Guard dedup scopes by written columns (#23)
* Guard dedup scopes by written columns

* Address dedup scope review feedback

* Remove legacy dedup scope support

* Remove stale legacy descriptions

* chore: bump version from 0.5.1 to 0.5.2
2026-06-09 23:27:54 +09:00
Daichi Narushima 334f01b647 chore: bump version from 0.5.0 to 0.5.1 (#21) 2026-06-09 15:52:32 +09:00
Daichi Narushima 1b69e8f08e Add generic MT5 rate-loading SDK APIs for downstream reuse (#20) 2026-06-09 15:37:24 +09:00
38 changed files with 12000 additions and 532 deletions
+201
View File
@@ -0,0 +1,201 @@
---
name: pr-feedback-triage
description: Triage pull request review comments into fixes, replies, clarification requests, or open follow-ups while respecting safe execution modes.
---
# PR Feedback Triage
Triage pull request review feedback, decide what action each thread needs, make focused fixes when allowed, and report or resolve only what is actually handled.
## When to Use
- A PR has review comments, requested changes, unresolved review threads, or bot review findings.
- The user asks to address, respond to, or resolve PR feedback.
- The user provides a PR URL/number, a branch with an associated PR, or copied comments.
Do not use this skill for a first-pass code review with no existing feedback; use a code review skill instead.
## Inputs
- Pull request URL or number, or a current branch that has an associated pull request.
- Repository checkout or platform access sufficient to inspect the PR diff and review feedback.
- Optional reviewer priorities from the user, such as "only address blocking comments" or "do not reply on the PR platform".
- Optional operating mode flags: `dry_run`, `no_push`, and `no_reply`.
If no PR or review comments are identifiable, ask for the target PR or the copied comments before proceeding.
## Modes
- `dry_run`: inspect review feedback and report the triage only. Do not edit files, run write-mode formatters, commit, push, post replies, or resolve review threads.
- `no_push`: local edits and verification are allowed, but do not push commits or otherwise update the remote branch. Report the local diff or local commits that still need to be pushed. Do not resolve threads whose resolution depends on unpushed local edits.
- `no_reply`: do not post replies, submit reviews, or resolve review threads. Provide suggested replies and resolution actions in the final report instead.
When a mode disables an action, skip that destructive or externally visible action even if normal workflow text would otherwise allow it.
## Preflight
1. Identify the current branch and target PR.
2. Check tracked local changes with `git diff --name-only` and `git diff --cached --name-only`. Ignore untracked files unless the review feedback explicitly concerns them.
3. Check unpushed commits before relying on remote review feedback.
4. If tracked local changes or unpushed commits exist, warn that existing PR comments may not cover the latest local state. In `normal` mode, push only when the user request or repository workflow allows it; otherwise continue with a clearly reported limitation.
## Feedback Collection
Gather the complete feedback set before editing:
- Fetch unresolved review threads, requested-change reviews, PR-level summary comments, and copied comments.
- Use platform-native APIs/CLI when available. Paginate results; do not inspect only the first page of threads or comments.
- For bot reviewers that post both summary comments and inline comments, collect both. Summary comments often contain severity, rationale, and fix instructions; inline comments contain the exact file and line context.
- Preserve every thread/comment identifier needed to reply or resolve later.
- Compare each comment with the current diff and file contents because review lines can become outdated.
## Deduplication and Ordering
Build one triage record per distinct finding:
- Prefer exact review-thread identity when available.
- For duplicate bot findings appearing in both summary and inline comments, merge by exact issue title first, then by file path plus line range as a fallback.
- Prefer inline comments for location and current code context.
- Prefer summary comments for severity, category, rationale, and detailed agent prompts.
- Preserve the reviewers exact issue title and original wording where practical. Do not rename findings in a way that would make replies hard to map back to comments.
- Preserve the reviewers original ordering unless the user asks for priority reordering. Many review bots already order findings by severity.
Each triage record should track: original title, reviewer, source IDs, location, current applicability, severity/priority if available, disposition, planned action, verification, reply text if any, resolution decision, platform action attempted, and final platform state.
## Resolution Policy
In normal mode, `Resolve conversation` is the default action for any review thread that has been fully handled. A thread is handled when the requested change is implemented and verified, the current code already satisfies the comment, the comment is outdated and no longer applies, or a deliberate deferral/won't-fix response has been posted with a clear reason.
Keep a thread open only when it still needs reviewer, maintainer, or product input, the fix is local-only and not pushed, verification is missing for a material change, or the user explicitly requested `dry_run`, `no_push`, or `no_reply` behavior that prevents resolution.
When resolving a thread, add a concise reply first only if it provides useful context, such as what changed, why no code change was needed, why a finding was intentionally deferred, or why the original comment is now outdated. Do not add noisy replies for self-evident fixes unless project norms require them.
## Platform Action Contract
Do not treat triage as complete until every collected source ID reaches an explicit terminal state:
- `resolved`: a platform resolve action succeeded, or a re-check shows the thread is already resolved.
- `replied_left_open`: a reply or question was posted and the thread is intentionally left unresolved.
- `not_resolvable`: the source is a PR-level summary comment or copied comment that has no platform-level resolve action; reply or post a PR summary when useful.
- `skipped_by_mode`: `dry_run`, `no_push`, or `no_reply` prevented the external action.
- `failed_action`: a reply or resolve action was attempted and failed; include the attempted action and failure in the final summary.
In normal mode, build and execute a platform action queue after fixes are verified and pushed when needed:
- `reply_then_resolve`: use for handled threads where the reviewer needs context before resolution.
- `resolve_only`: use for self-evident fixes and already-addressed or outdated threads where an extra reply would add noise.
- `reply_leave_open`: use only for clarification requests, blocked work, or intentionally open follow-ups.
- `reply_only`: use for PR-level comments or summaries that cannot be resolved as review threads.
For duplicate findings, execute the terminal action for every source thread ID, not only the primary triage record. If one finding is represented by three unresolved inline threads, all three must be resolved or explicitly left open.
## GitHub Action Guidance
Prefer platform-native APIs or `gh` commands that expose review-thread resolution state. For GitHub inline review threads, use the thread node ID and the GraphQL `resolveReviewThread` mutation rather than assuming that a reply resolves the conversation.
A reliable pattern is:
1. Re-fetch review threads and comments immediately before acting.
2. Reply to the thread when the action queue says a reply is needed.
3. Resolve the review thread by node ID when the terminal state should be `resolved`.
4. Re-fetch unresolved review threads after the action queue completes.
5. Retry any expected-to-be-resolved thread that is still unresolved once; if it still remains unresolved, mark it `failed_action` instead of claiming completion.
Example GraphQL mutation shape:
```graphql
mutation ($threadId: ID!) {
resolveReviewThread(input: { threadId: $threadId }) {
thread {
id
isResolved
}
}
}
```
A posted reply alone is sufficient only for `reply_leave_open`, `reply_only`, or `not_resolvable` sources. For handled inline review threads, reply and resolve are separate actions.
## Flow
```mermaid
flowchart TD
A[Identify PR and branch state] --> B[Collect all review feedback]
B --> C[Deduplicate and preserve source IDs]
C --> D[Inspect current diff and code]
D --> E{Classify each triage record}
E -->|Fix| F[Implement minimal change]
E -->|Answer| G[Prepare concise reply]
E -->|Clarify| H[Prepare question and leave open]
E -->|Already addressed or Outdated| I[Prepare evidence]
E -->|Defer or Won't fix| J[Document reason]
F --> K[Verify]
G --> L{Mode}
H --> L
I --> L
J --> L
K --> L
L -->|dry_run| M[Report triage only]
L -->|no_push| N[Report local diff or commits]
L -->|no_reply| O[Report suggested replies/actions]
L -->|normal| P[Commit/push if changed]
P --> R[Execute reply/resolve action queue]
R --> S[Re-fetch threads and retry unresolved handled threads once]
M --> Q[Final summary]
N --> Q
O --> Q
S --> Q
```
## Compact Workflow
1. **Collect all relevant feedback**
- Identify the PR and gather unresolved review threads, requested-change reviews, PR-level summaries, inline comments, and copied comments.
- Paginate all platform calls and keep comment/thread IDs for later replies and resolution.
- For bot reviews, collect both summary and inline comments, then merge duplicates rather than fixing the same finding twice.
2. **Classify each triage record**
- **Fix**: Valid requested change; make the smallest focused edit when not in `dry_run`.
- **Answer**: No code change needed; prepare a concise explanation.
- **Clarify**: Ambiguous, conflicting, or missing context; reply with the question and leave unresolved.
- **Already addressed**: Current code already satisfies it; prepare evidence.
- **Outdated**: Commented code or issue no longer exists; prepare evidence.
- **Defer / Won't fix**: Valid concern intentionally not changed now; document a specific reason.
3. **Act according to the classification and mode**
- Keep edits scoped to the review feedback.
- Follow reviewer-provided fix instructions literally when they are still applicable; deviate only when the current code proves the instruction is stale or unsafe.
- In `dry_run`, stop at triage, proposed fixes, suggested replies, and verification plan.
- In `no_push`, local edits are allowed, but do not push or resolve threads whose fix is only local. Reply or resolve non-code, already-addressed, or outdated threads only when the action does not depend on unpushed work and `no_reply` is not set.
- In `no_reply`, do not post replies or resolve threads; report suggested replies/actions instead.
- In normal mode, commit and push changed code when appropriate, then execute the platform action queue for every collected source ID.
4. **Verify before claiming completion**
- For fixes, run appropriate checks or explain why they could not run.
- Re-inspect the updated diff and comment context to confirm the concern is resolved.
- Re-fetch review threads after reply/resolve actions and confirm all expected-to-be-resolved thread IDs are resolved.
- Do not mark a thread resolved if it still needs reviewer, maintainer, or product input.
- If a resolve or reply operation fails, retry once when safe; then report `failed_action` with the affected source ID and reason.
5. **Finish**
- Normal mode: commit/push changes when appropriate, post useful replies or a summary, resolve all handled threads by default, and reconcile the final unresolved set.
- Safe modes: report the local state and the exact replies/resolution actions a human could take.
## Reply Guidance
- Keep inline replies short and tied to the original title or concern.
- For fixed findings, mention the concrete change or commit if useful.
- For already-addressed or outdated findings, cite the current code path or behavior that makes the finding no longer applicable.
- For deferred or won't-fix findings, provide the reason and any follow-up issue or owner if known.
- If a reply or resolve operation fails, continue with the remaining threads and report the failure in the final summary.
## Final Summary Checklist
- Mode used: `normal`, `dry_run`, `no_push`, or `no_reply`
- Counts by disposition: fixed, answered, clarified/left open, already addressed, outdated, deferred/won't-fix
- Counts by platform terminal state: resolved, replied-left-open, not-resolvable, skipped-by-mode, failed-action
- Threads resolved, intentionally left open, already resolved, or resolution actions skipped by mode
- Any expected-to-be-resolved thread that remained unresolved after retry
- Verification run or planned
- Commits pushed, local diff/commits, or "none"
- Remaining open items and who needs to respond
+15
View File
@@ -62,6 +62,19 @@ jobs:
runs-on: ubuntu-slim
secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
github-codeql-analysis:
if: >
github.event_name == 'push'
|| github.event_name == 'pull_request'
|| (github.event_name == 'workflow_dispatch' && inputs.workflow == 'lint-and-test')
permissions:
contents: read
security-events: write
actions: read
uses: dceoy/gh-actions-for-devops/.github/workflows/github-codeql-analysis.yml@main # zizmor: ignore[unpinned-uses]
with:
language: >
["python"]
dependabot-auto-merge:
if: >
github.event_name == 'pull_request' && github.actor == 'dependabot[bot]'
@@ -73,5 +86,7 @@ jobs:
contents: write
pull-requests: write
actions: read
checks: read
statuses: read
with:
unconditional: true
+20 -64
View File
@@ -1,81 +1,37 @@
# Repository Guidelines
## Commands
## Project Structure & Module Organization
### Development Setup
`mt5cli/` contains the package source. Important modules include `cli.py` for the Typer command-line app, `client.py` and `sdk.py` for public MT5 client/session APIs, `history.py` for SQLite history collection, `storage.py` and `converters.py` for export behavior, and `schemas.py` for normalized dataset contracts. `tests/` holds pytest coverage for CLI behavior, SDK contracts, trading helpers, history, and utilities. `docs/` and `mkdocs.yml` define the MkDocs site and API reference. `skills/mt5cli/SKILL.md` documents the mt5cli agent skill.
```bash
uv sync
```
## Build, Test, and Development Commands
### Code Quality and Documentation
- `uv sync` installs runtime and development dependencies from `pyproject.toml` and `uv.lock`.
- `uv run mt5cli --help` runs the local CLI entry point.
- `uv run ruff format .` formats Python files.
- `uv run ruff check --fix .` lints and applies safe fixes.
- `uv run pyright .` runs strict type checking.
- `uv run pytest` runs doctests, branch coverage, and the test suite.
- `uv run mkdocs serve` previews documentation locally; `uv run mkdocs build` validates the docs build.
**Important**: Run these before committing or creating a PR.
Use `.agents/skills/local-qa/SKILL.md` for pre-handoff QA. It runs `.agents/skills/local-qa/scripts/qa.sh`, which formats, lints, type-checks, tests, formats Markdown, and checks GitHub workflows.
1. **format, lint, and test**: Use `local-qa` skill.
2. **Documentation build** (if any public API changes): `uv run mkdocs build`
## Coding Style & Naming Conventions
## Architecture
Target Python `>=3.11,<3.14`. Use Ruffs configured 88-character line length and Google-style docstrings. Pyright is strict, so prefer explicit public type annotations and narrow exception handling. Keep module, function, and variable names in `snake_case`; classes and enums use `PascalCase`. Preserve the packages small, typed helper style rather than adding broad abstractions.
### Key Dependencies
## Design Principles
- **pdmt5**: Pandas-based data handler for MetaTrader 5 (core library)
- **typer**: CLI framework for building command-line interfaces
- **click**: Parameter type customization for CLI options
- **pandas**: Core data manipulation and analysis
Apply KISS, DRY, and YAGNI when changing code. Prefer the simplest implementation that satisfies the current CLI/API contract. Remove duplication when shared behavior is already proven by at least two concrete call sites, but avoid generic helpers for speculative reuse. Do not add configuration flags, extension hooks, or alternate backends until a real repository use case requires them.
### Package Structure
## Testing Guidelines
- `mt5cli/`: Main package directory
- `__init__.py`: Package initialization and exports (`detect_format`, `export_dataframe`)
- `cli.py`: CLI application with typer-based commands for data export
- `utils.py`: Constants, enums, parameter types, parsers, and export utilities
- `__main__.py`: Entry point for `python -m mt5cli`
- `tests/`: Comprehensive test suite (pytest-based)
- `test_cli.py`: Tests for CLI commands and collect-history behavior
- `test_utils.py`: Tests for utility constants, parameter types, parsers, and export functions
- `docs/`: MkDocs documentation with API reference
- `docs/index.md`: Main documentation
- `docs/api/`: Auto-generated API documentation for all modules
- Modern Python packaging with `pyproject.toml` and uv dependency management
### Quality Standards
- Type hints required (pyright strict mode)
- Comprehensive linting with 35+ rule categories (ruff)
- Test coverage tracking with 100% (pytest-cov)
- Parametrized tests for input/result matrices using `pytest.mark.parametrize` (pytest)
- Test doubles (mocks, stubs) using `pytest_mock` for external dependencies (pytest-mock)
- Pydantic models for data validation and configuration
### Documentation workflow
1. Add Google-style docstrings to functions/classes
2. Local preview: `uv run mkdocs serve`
3. Build: `uv run mkdocs build`
4. Deploy: `uv run mkdocs gh-deploy`
Tests use pytest, pytest-mock, doctests, and pytest-cov. Test files should match `tests/test_*.py`, classes `Test*`, and functions `test_*`. Coverage is configured with `fail_under = 100`, so add focused tests for every behavior change. Mock MT5/pdmt5 boundaries; do not require a live MetaTrader terminal in unit tests.
## Commit & Pull Request Guidelines
- Run QA checks using `local-qa` skill before committing or creating a PR.
- Branch names use appropriate prefixes on creation (e.g., `feature/...`, `bugfix/...`, `refactor/...`, `docs/...`, `chore/...`).
- When instructed to create a PR, create it as a draft with appropriate labels by default.
Recent history uses concise imperative commits, sometimes with conventional prefixes such as `feat:` or `chore:` and PR numbers appended by GitHub. Keep commits scoped to one logical change. Pull requests should describe behavior changes, note tests run, link related issues, and call out MT5/live-trading risk where relevant.
## Code Design Principles
## Security & Configuration Tips
Always prefer the simplest design that works.
- **KISS**: Choose straightforward solutions and avoid unnecessary abstraction.
- **DRY**: Remove duplication when it improves clarity and maintainability.
- **YAGNI**: Do not add features, hooks, or flexibility until they are needed.
- **SOLID/Clean Code**: Apply these as tools, only when they keep the design simpler and easier to change.
## Development Methodology
Keep delivery incremental, test-backed, and easy to review.
- Make small, safe, reversible changes.
- Prefer `Red -> Green -> Refactor`.
- Do not mix feature work and refactoring in the same commit.
- Refactor when it improves clarity or removes real duplication (Rule of Three).
- Keep tests fast, focused, and self-validating.
Never commit account credentials, broker passwords, exported private data, or local `.venv` contents. Treat `order_send` and CLI `order-send --yes` as live execution paths; gate examples and tests so they cannot place real trades accidentally.
+179 -4
View File
@@ -2,10 +2,18 @@
[![CI/CD](https://github.com/dceoy/mt5cli/actions/workflows/ci.yml/badge.svg)](https://github.com/dceoy/mt5cli/actions/workflows/ci.yml)
Command-line tool for exporting MetaTrader 5 data to CSV, JSON, Parquet, and SQLite3.
Generic MT5 data and execution infrastructure for Python applications. Export from the CLI or import a small, stable Python API in downstream packages.
The [Public API Contract](docs/api/public-contract.md) lists stable SDK exports (`mt5cli.STABLE_SDK_EXPORTS`), CLI commands, internal helpers, and responsibilities that remain out of scope (strategy logic, backtests, optimization).
Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
## Architecture
- **pdmt5** — canonical MT5 client, DataFrame/trading primitives, and MT5 constant parsing (`TIMEFRAME_*`, `COPY_TICKS_*`, order types).
- **mt5cli** — public `MT5Client` API, standardized dataset schemas, storage helpers, CLI commands, and SQLite history collection built on pdmt5.
- **mt5api** — sibling HTTP adapter for remote MT5 access; not a dependency of mt5cli.
## Features
- **Multi-format export**: CSV, JSON, Parquet, and SQLite3 output formats
@@ -21,7 +29,96 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
pip install -U mt5cli MetaTrader5
```
## Usage
## Python API (downstream packages)
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
DataKind,
Dataset,
MT5Client,
build_config,
collect_history,
export_dataframe,
mt5_session,
normalize_dataframe,
update_history_with_config,
)
# Persistent session for multiple calls
with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
rates = client.copy_rates_range(
"EURUSD",
timeframe="H1",
date_from="2024-01-01",
date_to="2024-02-01",
)
positions = client.positions()
check = client.order_check({"action": 1, "symbol": "EURUSD", "volume": 0.1})
# Normalize MT5 frames to the public schema contract before storage
closed_rates = normalize_dataframe(
rates, DataKind.rates, symbol="EURUSD", timeframe="H1"
)
export_dataframe(closed_rates, Path("rates.csv"), "csv")
# Bulk SQLite history (same behavior as collect-history CLI command)
collect_history(
Path("history.db"),
symbols=["EURUSD"],
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
datasets={Dataset.rates, Dataset.history_deals},
)
# Incremental append for automated pipelines
update_history_with_config(
output="history.db",
symbols=["EURUSD"],
config=build_config(login=12345),
)
```
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly.
### Trading lifecycle and state helpers
Trading applications can depend on `mt5cli` imports only; terminal path,
credentials, server, and timeout are forwarded to `pdmt5.Mt5Config`, numeric
login strings are coerced to integers, and empty login strings are treated as
unset.
```python
from mt5cli import (
calculate_spread_ratio,
create_trading_client,
get_account_snapshot,
mt5_trading_session,
)
with mt5_trading_session(
path=r"C:\Program Files\MetaTrader 5\terminal64.exe",
login="12345",
password="from-env-or-secret-store",
server="Broker-Demo",
) as client:
account = get_account_snapshot(client)
spread = calculate_spread_ratio(client, "EURUSD")
client = create_trading_client(login=12345, server="Broker-Demo")
try:
positions = client.positions_get_as_df(symbol="EURUSD")
finally:
client.shutdown()
```
## CLI usage
```bash
# Export account information to CSV
@@ -132,9 +229,30 @@ update_history_with_config(
- **`collect-history`**: explicit date-range export into SQLite.
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline downstream tools.
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
```python
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
```
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`, `sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability errors for history API methods without advancing the throttle (other `AttributeError` / `TypeError` values always propagate). Pass `update_backend` to inject a custom history update callable (same keyword arguments as `update_history`) instead of monkey-patching `mt5cli.sdk.update_history`.
- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable `pdmt5.Mt5TradingClient` that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. Keep read-only collection on `mt5_session()` / `MT5Client`.
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `MT5Client` that shuts down on exit.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
@@ -144,6 +262,63 @@ update_history_with_config(
- Windows OS (MetaTrader 5 requirement)
- MetaTrader 5 platform installed
### Migration note for downstream trading apps
Replace local MT5 lifecycle and trading helper code with mt5cli imports:
```python
# Before (local application helpers)
# with local_mt5_trading_session(config) as client:
# side = local_detect_position_side(client, symbol)
# sizing = local_calculate_margin_and_volume(client, symbol, unit_ratio, preserved_ratio)
# limits = local_determine_order_limits(client, symbol, side, sl_ratio, tp_ratio)
# After (mt5cli shared layer)
from pdmt5 import Mt5Config
from mt5cli import (
calculate_margin_and_volume,
detect_position_side,
determine_order_limits,
mt5_trading_session,
)
with mt5_trading_session(
Mt5Config(path=terminal_path, login=login), retry_count=2
) as client:
side = detect_position_side(client, symbol)
sizing = calculate_margin_and_volume(
client, symbol, unit_margin_ratio=0.5, preserved_margin_ratio=0.2
)
if side is not None:
limits = determine_order_limits(
client,
symbol,
side,
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.02,
)
```
Throttled history updates use a separate read-only session:
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db", interval_seconds=60, suppress_errors=True
)
client = Mt5DataClient(config=Mt5Config(login=login))
client.initialize_and_login_mt5()
try:
updater.update(client, ["EURUSD"])
finally:
client.shutdown()
```
Read-only collectors can keep using `mt5_session()` and `MT5Client`.
## Development
```bash
+3
View File
@@ -0,0 +1,3 @@
# Client
::: mt5cli.client
+3
View File
@@ -0,0 +1,3 @@
# Converters
::: mt5cli.converters
+3
View File
@@ -0,0 +1,3 @@
# Exceptions
::: mt5cli.exceptions
+75 -5
View File
@@ -133,8 +133,8 @@ The `update_history` SDK path uses the same base tables and optional
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing legacy
naming schemes:
existing SQLite history database without creating files or guessing naming
schemes:
```python
from pathlib import Path
@@ -167,19 +167,89 @@ Resolution rules:
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
The canonical normalized rate table is `rates`; compatibility views are named
with `rate_<symbol>__<timeframe>` for single-timeframe symbols or
`rate_<symbol>__<granularity>_<timeframe>` when a symbol has multiple stored
timeframes. `resolve_rate_table_name()` returns `rates`, while
`resolve_rate_view_name()` returns the per-symbol compatibility view name.
Use `load_rate_data()` or `load_rate_series_from_sqlite(..., table=...)` to load
a single table or view from a SQLite path. Use
`load_rate_series_by_granularity()` to load multiple instrument/granularity
targets without hard-coding view names:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli import (
load_rate_data,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_rate_table_name,
)
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
same_rates = load_rate_series_from_sqlite(Path("history.db"), table=view, count=1000)
table = resolve_rate_table_name("EURUSD", "M1") # "rates"
series = load_rate_series_by_granularity(
Path("history.db"),
symbols=["EURUSD", "GBPUSD"],
granularities=["M1", "H1"],
count=500,
)
```
`count` returns the latest rows while preserving chronological order. Missing
tables/views and mismatched `explicit_tables` lengths raise `ValueError` with
the requested database target in the message.
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
### Multi-series rate loading
For loading many rate series at once, build neutral `RateTarget` pairs and load
them from SQLite in one call. View names are resolved via the same
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
```python
from pathlib import Path
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
```
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
row-major order, normalizing timeframe names such as `"M1"` to their integer
values; set `allow_missing_symbol=True` to address series solely by
`explicit_tables` (targets carry `symbol=None`).
- `resolve_rate_tables()` maps targets to table or view names and validates that
any `explicit_tables` count matches the target count. Pass
`require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or managed view is missing. When
`explicit_tables` is provided, names are returned as-is and
`require_existing` is ignored.
- `load_rate_series_from_sqlite()` returns a mapping keyed by
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
requires existing managed `rate_*` compatibility views and raises
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
are rejected.
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
loads the series, and rekeys the result by granularity name to avoid
converting integer timeframes downstream:
```python
from mt5cli import load_rate_series_by_granularity
series = load_rate_series_by_granularity(
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
)
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
```
+46 -105
View File
@@ -1,120 +1,61 @@
# API Reference
This section contains the complete API documentation for mt5cli.
This section documents the mt5cli public Python API and CLI modules.
## Modules
Start with the [Public API Contract](public-contract.md) for the stable
downstream SDK surface, CLI boundary, internal modules, and out-of-scope strategy
responsibilities.
The mt5cli package consists of the following modules:
## Public API layers
### [CLI](cli.md)
| Module | Purpose |
| ----------------------------------------- | ------------------------------------------------------------------------- |
| [Public API Contract](public-contract.md) | Stable downstream SDK exports, CLI boundary, and out-of-scope items |
| [Client](client.md) | `MT5Client` session abstraction for data access and order primitives |
| [Schemas](schemas.md) | Canonical DataFrame contracts and normalization helpers |
| [Storage](storage.md) | CSV/JSON/Parquet/SQLite export and history collection helpers |
| [Converters](converters.md) | Symbol, timeframe, timezone, and date-range utilities |
| [Exceptions](exceptions.md) | Stable mt5cli exception types and MT5 error normalization |
| [SDK](sdk.md) | Module-level fetch helpers, multi-account collectors, incremental history |
| [Trading](trading.md) | Trading-capable sessions and operational helpers |
| [History Collection (SQLite)](history.md) | SQLite schema, incremental writes, dedup, and rate views |
| [CLI](cli.md) | Typer commands that delegate to the Python API |
| [Utils](utils.md) | Parsing helpers and Click parameter types |
Command-line interface module providing typer-based commands for exporting MetaTrader 5 data to CSV, JSON, Parquet, and SQLite3 formats.
## Architecture overview
### [Utils](utils.md)
Utility module providing constants, enums, Click parameter types, and helper functions for parsing and exporting data.
### [SDK](sdk.md)
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
### [History Collection (SQLite)](history.md)
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
## Architecture Overview
The package follows a simple architecture built on top of pdmt5:
1. **CLI Layer** (`cli.py`): Typer application with subcommands that delegate to the SDK and export results.
2. **SDK Layer** (`sdk.py`): Read-only data access functions, `Mt5CliClient`, and `collect_history` orchestration.
3. **Utils Layer** (`utils.py`): Constants, enums, custom Click parameter types, parsing helpers, and format detection/export utilities.
4. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient` and `Mt5Config` from the pdmt5 package for all MetaTrader 5 data access.
## Usage Guidelines
All modules follow these conventions:
- **Type Safety**: All functions include comprehensive type hints
- **Error Handling**: User-friendly error messages via typer
- **Documentation**: Google-style docstrings with examples
- **Validation**: Custom Click parameter types for input validation
## Quick Start
```bash
# Export account information to CSV
mt5cli -o account.csv account-info
# Export EURUSD H1 rates to Parquet
mt5cli -o rates.parquet rates-from --symbol EURUSD --timeframe H1 \
--date-from 2024-01-01 --count 1000
# Export ticks to JSON
mt5cli -o ticks.json ticks-from --symbol EURUSD \
--date-from 2024-01-01 --count 500 --flags ALL
# Export to SQLite3 with custom table name
mt5cli -o data.db --table symbols symbols --group "*USD*"
```mermaid
flowchart TD
App["Downstream application"] --> Client["MT5Client"]
CLI["mt5cli CLI"] --> Client
Client --> SDK["sdk / pdmt5"]
Client --> Schemas["schemas"]
Storage["storage"] --> History["history SQLite"]
Storage --> Utils["utils export"]
SDK --> PDMT5["pdmt5.Mt5DataClient"]
```
## Python API
Downstream packages should depend on the package root exports documented in the
[Public API Contract](public-contract.md) (`MT5Client`,
`DataKind`, `normalize_dataframe`, `collect_history`, `load_rate_data`,
`resolve_rate_view_name`, etc.) rather than private modules.
`MT5Client.order_send()` is a live execution primitive that can place real trades. mt5cli exposes minimal execution helpers only; strategy logic, signals, backtests, and optimization remain out of scope and must be implemented downstream with explicit execution gating.
## Quick start
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import MT5Client, build_config, mt5_session
from mt5cli import (
Dataset,
IfExists,
Mt5CliClient,
collect_history,
copy_rates_range,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically
rates = copy_rates_range(
"EURUSD",
timeframe="H1",
date_from="2024-01-01",
date_to="2024-02-01",
)
# Detect output format from file extension
fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite
collect_history(
Path("history.db"),
symbols=["EURUSD"],
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
)
with mt5_session(build_config(login=12345)) as client:
rates = client.copy_rates_range("EURUSD", "H1", "2024-01-01", "2024-02-01")
positions = client.positions()
```
## Examples
```bash
mt5cli -o account.csv account-info
mt5cli -o rates.parquet rates-range --symbol EURUSD --timeframe H1 \
--date-from 2024-01-01 --date-to 2024-02-01
```
See individual module pages for detailed usage examples and code samples.
See individual module pages for detailed usage examples.
+222
View File
@@ -0,0 +1,222 @@
# Public API Contract
mt5cli is the generic MT5 data and execution infrastructure layer for downstream
Python applications. The intended dependency direction is:
```text
downstream app -> mt5cli -> pdmt5 -> MetaTrader 5
```
Downstream packages should import from the package root (`from mt5cli import
...`) and use the public tier sets in `mt5cli.contract` to distinguish API
stability. CLI commands mirror the same behavior but are not importable Python
APIs.
## Public API tiers
mt5cli classifies package-root imports by intended downstream use:
| Tier | Contract set | Meaning |
| ---------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Stable core | `STABLE_SDK_EXPORTS` | Preferred SDK surface for downstream MT5 infrastructure adapters. Changes require a deliberate compatibility path. |
| Secondary public | `SECONDARY_PUBLIC_EXPORTS` | Public helpers for CLI/export/schema integrations and lower-level MT5 wrappers. Importable, but less central to the downstream trading SDK. |
## Stable downstream SDK API
These names are exported from `mt5cli` and covered by the contract in
`mt5cli.STABLE_SDK_EXPORTS` (defined in `mt5cli.contract`).
### Session lifecycle and configuration
| Symbol | Role |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `MT5Client` | Read-only data client with optional `order_check` / `order_send` |
| `build_config` | Build `pdmt5.Mt5Config` from connection fields |
| `mt5_session` | Context manager: initialize, login, yield client, shutdown |
| `create_trading_client`, `mt5_trading_session` | Trading-capable `pdmt5.Mt5TradingClient` lifecycle |
| `AccountSpec` | Generic account group: symbols plus optional credentials |
| `resolve_account_spec`, `resolve_account_specs` | Merge overrides and expand `${ENV_VAR}` placeholders; opt-in `allow_whole_dollar_env` for bare `$NAME` |
| `substitute_env_placeholders` | Replace `${NAME}` substrings from the environment; opt-in `allow_whole_dollar_env` for whole-value `$NAME` |
Credential resolution is generic: any environment variable name may appear inside
`${...}`. mt5cli does not hard-code application-specific keys such as
`mt5_login` or `mt5_exe`.
Pass `allow_whole_dollar_env=True` to `substitute_env_placeholders()`,
`resolve_account_spec()`, `resolve_account_specs()`, and `build_config()` to
additionally expand strings whose entire value is a bare `$ENV_NAME` identifier.
Partial strings such as `"plan$pass"`, `"abc$ENV"`, or `"$ENV-suffix"` are
**never** expanded — only an exact `$IDENTIFIER` whole-string match qualifies.
Default is `False` to preserve backward compatibility.
### Closed-bar rate helpers
MetaTrader 5 returns the still-forming bar as the last row when
`start_pos=0`. Use these helpers instead of reimplementing bar trimming or
timestamp normalization in downstream apps.
| Symbol | Role |
| ------------------------------------------------ | ------------------------------------------------------------------------------- |
| `drop_forming_rate_bar` | Remove the last row from chronologically ordered rate data |
| `fetch_latest_closed_rates` | Single connected client: fetch `count + 1`, drop forming bar |
| `fetch_latest_closed_rates_for_trading_client` | Closed bars from an active `Mt5TradingClient` session; returns RangeIndex |
| `fetch_latest_closed_rates_indexed` | Same as above but returns a UTC `DatetimeIndex` named `"time"` (no time column) |
| `collect_latest_closed_rates_for_accounts` | Multi-account closed bars with optional retry wrapper |
| `collect_latest_closed_rates_by_granularity` | Same data keyed by `(symbol, granularity_name)` |
| `collect_latest_rates_for_accounts_with_retries` | Bounded exponential backoff for transient MT5 errors |
### SQLite history collection and rate loading
| Symbol | Role |
| ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `collect_history` | One-shot date-range export into SQLite |
| `update_history`, `update_history_with_config` | Incremental append from `MAX(time)` cursors |
| `ThrottledHistoryUpdater` | Minimum interval between successful incremental updates; optional `update_backend` injection |
| `resolve_history_datasets`, `resolve_history_timeframes`, `resolve_history_tick_flags` | History pipeline configuration |
| `build_rate_view_name`, `resolve_rate_table_name`, `resolve_rate_view_name`, `resolve_rate_view_names`, `resolve_rate_tables` | Map symbols/timeframes to mt5cli-managed table or view names |
| `RateTarget`, `build_rate_targets` | Neutral `(symbol, timeframe)` series descriptors |
| `load_rate_data`, `load_rate_data_from_connection` | Load one table/view into a time-indexed DataFrame |
| `load_rate_series_from_sqlite`, `load_rate_series_by_granularity` | Load one or many series; fail clearly when managed views are missing |
Pass `require_existing=True` to rate view resolution helpers when downstream
code must fail instead of receiving a best-guess view name. Multi-series loaders
require existing managed `rate_*__*` views unless `explicit_tables` is supplied.
See [History Collection (SQLite)](history.md) for schema, view naming, and ER
diagrams.
### Trading and sizing primitives (generic)
These helpers implement broker-facing calculations only. They do not encode
strategy entries, exits, Kelly sizing, or signal logic.
| Symbol | Role |
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| `get_account_snapshot`, `get_symbol_snapshot`, `get_tick_snapshot`, `get_positions_frame` | Normalized account/symbol/tick/position views |
| `extract_tick_price` | Positive finite bid/ask extraction from tick mappings |
| `detect_position_side` | Net long / short / flat from open positions |
| `calculate_spread_ratio` | Relative bid-ask spread |
| `calculate_margin_and_volume`, `calculate_volume_by_margin`, `calculate_new_position_margin_ratio` | Margin budget and volume sizing |
| `normalize_order_volume`, `estimate_order_margin`, `calculate_positions_margin` | Broker volume normalization and margin totals |
| `calculate_positions_margin_by_symbol` | Per-symbol margin map (resilient, first-seen order) |
| `calculate_positions_margin_safe` | Summed total margin across symbols (failed symbols skipped) |
| `calculate_projected_margin_ratio` | Estimated symbol-scoped margin/equity after optional new exposure |
| `calculate_account_projected_margin_ratio` | Account snapshot margin/equity after optional new exposure |
| `calculate_symbol_group_margin_ratio` | Estimated symbol-group margin/equity with optional exposure |
| `determine_order_limits` | SL/TP price levels from ratios |
| `calculate_trailing_stop_updates` | Per-ticket generic trailing stop-loss update plan |
| `ensure_symbol_selected` | Select/verify Market Watch visibility |
| `place_market_order`, `close_open_positions`, `update_sltp_for_open_positions`, `update_trailing_stop_loss_for_open_positions` | Order execution helpers (`dry_run` supported) |
| `MarginVolume`, `OrderLimits`, `OrderExecutionResult` | Typed return contracts for order helpers |
| `OrderSide`, `OrderFillingMode`, `OrderTimeMode`, `PositionSide`, `ExecutionStatus` | Typed enums for order helpers |
`MT5Client.order_send()` and CLI `order-send --yes` are live execution paths.
Order helpers validate broker stop-level distance in `determine_order_limits()` and
raise `Mt5TradingError` when computed SL/TP prices are too close to the entry
quote. Validation uses `trade_stops_level * point` from the current quote and
symbol metadata as a pre-check only; it does not guarantee live order acceptance
after price movement and does not inspect `trade_freeze_level`. Live
`place_market_order()` and SL/TP updates call
`ensure_symbol_selected()` so hidden symbols are added to Market Watch before
sending requests. Failed, malformed, or unknown broker retcodes are fail-closed
and returned as `status="failed"` with normalized `request` / `response` details;
`dry_run=True` never calls `ensure_symbol_selected()` or `order_send()`.
### Errors and MT5 type re-exports
| Symbol | Role |
| ------------------------------------------------------------------------------------ | ----------------------------------------------- |
| `Mt5CliError`, `Mt5ConnectionError`, `Mt5OperationError`, `Mt5SchemaError` | Stable mt5cli exception types |
| `normalize_mt5_exception`, `call_with_normalized_errors`, `is_recoverable_mt5_error` | Error normalization and retry classification |
| `Mt5Config`, `Mt5RuntimeError`, `Mt5TradingClient`, `Mt5TradingError` | Re-exported pdmt5 types for adapter convenience |
## Secondary public exports
These names remain importable from `mt5cli` and are covered by
`SECONDARY_PUBLIC_EXPORTS`, but they are oriented toward CLI/export/schema
integrations, parsing, and lower-level MT5 access rather than the stable core
SDK surface. Prefer the stable symbols above for downstream infrastructure
adapters.
### Read-only MT5 data wrappers
Module-level helpers open a transient connection per call. Prefer `mt5_session`
or `MT5Client` when making many requests in one process.
| Area | Symbols |
| -------------------- | ---------------------------------------------------------------------------------------------------- |
| Rates | `copy_rates_from`, `copy_rates_from_pos`, `copy_rates_range`, `latest_rates`, `collect_latest_rates` |
| Ticks | `copy_ticks_from`, `copy_ticks_range`, `recent_ticks` |
| Account / terminal | `account_info`, `terminal_info`, `mt5_version`, `last_error`, `mt5_summary`, `mt5_summary_as_df` |
| Symbols / market | `symbols`, `symbol_info`, `symbol_info_tick`, `market_book`, `minimum_margins` |
| Trading state (read) | `orders`, `positions`, `history_orders`, `history_deals`, `recent_history_deals` |
| Multi-account rates | `collect_latest_rates_for_accounts` |
Use `mt5_version` for MetaTrader 5 terminal version data. The name `version` at
the package root refers to `importlib.metadata.version` (package metadata), not
the MT5 SDK helper.
### Schema, export, and parser helpers
| Area | Symbols |
| -------------------- | ------------------------------------------------------------------------------------------------------------- |
| Dataset contracts | `DataKind`, `Dataset`, `IfExists`, `DEDUP_KEYS`, `REQUIRED_COLUMNS`, `TIME_COLUMNS`, `KNOWN_MT5_TIME_COLUMNS` |
| Schema normalization | `normalize_dataframe`, `normalize_time_columns`, `schema_columns`, `validate_schema` |
| Export helpers | `detect_format`, `export_dataframe`, `export_dataframe_to_sqlite` |
| Symbol parsing | `normalize_symbol`, `normalize_symbols` |
| Time parsing | `ensure_utc`, `parse_date_range`, `parse_datetime`, `recent_window` |
| MT5 parsing maps | `granularity_name`, `parse_tick_flags`, `parse_timeframe`, `TICK_FLAG_MAP`, `TIMEFRAME_MAP` |
| Trading data shapes | `POSITION_COLUMNS` |
## CLI commands
The Typer application in `mt5cli.cli` exposes file-export commands documented in
[CLI Module](cli.md) and the project README. CLI commands:
- Require `-o/--output` and write CSV, JSON, Parquet, or SQLite.
- Accept global MT5 connection options (`--login`, `--password`, `--server`,
`--path`, `--timeout`).
- Delegate to the same Python APIs described here; they are not duplicated
business logic.
`order-send` requires `--yes` before placing live trades.
## Internal helpers (not stable)
Do not import these for downstream contracts; they may change without a semver
notice:
| Module | Examples |
| ------------------------ | ------------------------------------------------------------------------- |
| `mt5cli.sdk` | `connected_client`, `_run_with_client`, private coercion helpers |
| `mt5cli.history` | `write_*_dataset`, `deduplicate_history_tables`, `parse_sqlite_timestamp` |
| `mt5cli.retry` | `retry_with_backoff` |
| `mt5cli.cli` | Typer command handlers and Click parameter types |
| Leading-underscore names | Any `_`-prefixed function or method |
Use the package-root stable exports instead of reaching into submodule
internals.
## Explicitly out of scope
mt5cli must **not** implement downstream strategy or research responsibilities.
The following belong in consuming applications, not in mt5cli:
- Signal detection (for example AR-GARCH or other model-specific triggers)
- Backtesting, walk-forward analysis, or parameter optimization
- Strategy-specific risk policy, position sizing systems, or Kelly fractions
- Entry/exit decision logic or YAML strategy semantics
- Application-specific credential schema keys wired into mt5cli internals
mt5cli provides connection lifecycle, normalized data access, SQLite history
machinery, closed-bar helpers, generic margin/volume/spread/SL/TP utilities, and
optional order primitives so downstream apps can focus on strategy code behind
their own adapter layer.
## Contract verification
`tests/test_contracts.py` asserts that every name in the stable and secondary
tier sets is importable from `mt5cli`, documents key closed-bar, rate-view,
SQLite loading, account-resolution, and trading-session behaviors, and keeps the
tier sets aligned with `__all__`.
+3
View File
@@ -0,0 +1,3 @@
# Schemas
::: mt5cli.schemas
+171
View File
@@ -1,3 +1,174 @@
# SDK Module
::: mt5cli.sdk
## Resilient multi-account orchestration
The SDK ships strategy-agnostic helpers for building long-running collectors on
top of the read-only client. None of them depend on a particular trading
application.
### Retrying transient rate collection
`collect_latest_rates_for_accounts_with_retries()` wraps
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
failure is re-raised once `retry_count` is exhausted.
```python
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
rates = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1", "H1"],
count=500,
retry_count=3,
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
)
```
### Latest closed rate bars
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
row. `fetch_latest_closed_rates()` handles one connected `MT5Client`; use
`fetch_latest_closed_rates_for_trading_client()` from an active
`Mt5TradingClient` session. Multi-account helpers fetch `count + 1` bars, drop
that row with `drop_forming_rate_bar()`, and validate each series is non-empty. Returned frames are ordered
oldest-to-newest and may contain fewer than `count` rows only when MT5 returns
fewer closed bars.
```python
from mt5cli import (
AccountSpec,
collect_latest_closed_rates_by_granularity,
fetch_latest_closed_rates,
)
closed = fetch_latest_closed_rates(
client,
symbol="EURUSD",
granularity="M1",
count=500,
)
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
closed_m1 = rates["EURUSD", "M1"]
```
Use `collect_latest_closed_rates_by_granularity()` when callers prefer keys such
as `("EURUSD", "M1")` instead of integer timeframes.
### Resolving credentials and `${ENV_VAR}` placeholders
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
secrets out of plan/config files. A missing environment variable raises
`ValueError`.
```python
import os
from mt5cli import AccountSpec, resolve_account_specs
os.environ["MT5_LOGIN"] = "12345"
os.environ["MT5_PASSWORD"] = "secret"
accounts = [
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
]
resolved = resolve_account_specs(accounts, server="Broker-Demo")
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
```
Pass `allow_whole_dollar_env=True` to also expand strings whose **entire value**
is a bare `$ENV_NAME` identifier (no braces). This opt-in covers
`substitute_env_placeholders()`, `resolve_account_spec()`,
`resolve_account_specs()`, and `build_config()`. Note: `build_config` cannot
expand `login` because that parameter is `int | None`; use
`resolve_account_spec` for a string `login` placeholder. Partial strings such as
`"plan$pass"`, `"abc$ENV"`, or `"$ENV-suffix"` are never expanded — only an
exact `$IDENTIFIER` whole-string match qualifies. The default is `False` to
preserve backward compatibility.
```python
import os
from mt5cli import AccountSpec, resolve_account_specs
os.environ["MT5_PASSWORD"] = "secret"
accounts = [AccountSpec(symbols=["EURUSD"], password="$MT5_PASSWORD")]
resolved = resolve_account_specs(accounts, allow_whole_dollar_env=True)
# resolved[0].password == "secret"
```
### Throttled incremental history updates
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
between successful runs (using a monotonic clock), so an application loop can
call it every iteration without over-fetching.
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db",
datasets={Dataset.rates},
timeframes=["M1"],
interval_seconds=60, # <= 0 updates on every call
)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
while True:
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
# ... do other work; break when shutting down ...
finally:
client.shutdown()
```
Pass `update_backend` to substitute the default `update_history` implementation
without monkey-patching `mt5cli.sdk.update_history`. The callable receives the
same keyword arguments as `update_history` (`client`, `output`, `symbols`,
`datasets`, `timeframes`, `flags`, `lookback_hours`, `with_views`,
`include_account_events`). The resolved backend is stored on
`updater.update_backend` for inspection or subclassing.
```python
from mt5cli import ThrottledHistoryUpdater, update_history
def app_update_history(**kwargs) -> None:
update_history(**kwargs) # or delegate to application-specific logic
updater = ThrottledHistoryUpdater(
output="history.db",
interval_seconds=60,
update_backend=app_update_history,
)
```
By default recoverable errors (`Mt5TradingError`, `Mt5RuntimeError`,
`sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability
`AttributeError` / `TypeError` for history API methods) propagate so the caller
controls logging; pass `suppress_errors=True` to swallow them and return
`False` without advancing the throttle. Other `AttributeError` / `TypeError`
values always propagate. Input validation (`_resolve_update_history_request`)
runs before any MT5 or SQLite calls, but when `suppress_errors=True` the
resulting `ValueError` is suppressed along with other recoverable errors.
## Trading-capable sessions
For order placement and trading calculations, use the dedicated
[Trading module](trading.md). Use `mt5_session()` / `MT5Client` for read-only
collection.
+3
View File
@@ -0,0 +1,3 @@
# Storage
::: mt5cli.storage
+199
View File
@@ -0,0 +1,199 @@
# Trading Module
::: mt5cli.trading
## Trading-capable MT5 sessions
`create_trading_client()` and `mt5_trading_session()` complement the read-only
`mt5_session()` helper in `sdk.py`. They return or yield an initialized
`pdmt5.Mt5TradingClient`, use `Mt5Config.path` to launch the terminal when
configured, and `mt5_trading_session()` always calls `shutdown()` on exit.
```python
from mt5cli import create_trading_client, mt5_trading_session
with mt5_trading_session(
path=r"C:\Program Files\MetaTrader 5\terminal64.exe",
login="12345",
password="secret",
server="Broker-Demo",
retry_count=2,
) as client:
positions = client.positions_get_as_df(symbol="EURUSD")
client = create_trading_client(login=12345, server="Broker-Demo")
try:
account = client.account_info_as_dict()
finally:
client.shutdown()
```
`login` accepts `int`, numeric `str`, or an empty string; empty strings are
treated as unset. `path`, `password`, `server`, and `timeout` are forwarded to
`pdmt5.Mt5Config`, and omitted `timeout` values keep the lower-level default.
Use `mt5_session()` / `MT5Client` for read-only data collection.
## State and order helpers
These helpers are strategy-agnostic and do not depend on signal detection,
betting logic, or scheduling code in downstream applications.
```python
from mt5cli import (
calculate_positions_margin,
calculate_spread_ratio,
calculate_margin_and_volume,
close_open_positions,
detect_position_side,
determine_order_limits,
estimate_order_margin,
fetch_latest_closed_rates_for_trading_client,
fetch_latest_closed_rates_indexed,
get_account_snapshot,
get_positions_frame,
get_symbol_snapshot,
get_tick_snapshot,
normalize_order_volume,
place_market_order,
)
account = get_account_snapshot(client)
symbol = get_symbol_snapshot(client, "EURUSD")
tick = get_tick_snapshot(client, "EURUSD")
positions = get_positions_frame(client, "EURUSD")
side = detect_position_side(client, "EURUSD")
spread_ratio = calculate_spread_ratio(client, "EURUSD")
volume = normalize_order_volume(
0.15,
volume_min=symbol["volume_min"],
volume_max=symbol["volume_max"],
volume_step=symbol["volume_step"],
)
buy_margin = (
estimate_order_margin(client, "EURUSD", "BUY", volume) if volume > 0 else 0.0
)
open_margin = calculate_positions_margin(client, symbols=["EURUSD"])
closed_bars = fetch_latest_closed_rates_for_trading_client(
client,
symbol="EURUSD",
granularity="M1",
count=100,
)
# Or fetch with a UTC DatetimeIndex instead of a "time" column:
indexed_bars = fetch_latest_closed_rates_indexed(
client,
symbol="EURUSD",
granularity="M1",
count=100,
)
# indexed_bars.index is a UTC-aware DatetimeIndex named "time"
sizing = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
limits = determine_order_limits(
client,
"EURUSD",
side="long",
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.02,
)
preview = place_market_order(
client,
symbol="EURUSD",
volume=sizing["buy_volume"],
order_side="BUY",
sl=limits["stop_loss"],
tp=limits["take_profit"],
dry_run=True,
)
closed = close_open_positions(client, symbols="EURUSD", dry_run=True)
```
`detect_position_side()` returns `long` for buy-only exposure, `short` for
sell-only exposure, and `None` for no positions or mixed long/short exposure.
`calculate_spread_ratio()` uses `(ask - bid) / ((ask + bid) / 2)` and raises
`Mt5TradingError` when bid or ask is missing or non-positive.
`normalize_order_volume()` returns `0.0` for invalid constraints or
sub-minimum requests; check the result before calling `estimate_order_margin()`,
which requires a positive finite volume. `calculate_positions_margin()` silently
skips rows with missing symbols, non-positive volumes, non-finite volumes, or
unsupported position types, but propagates `Mt5TradingError` from `estimate_order_margin()` when a valid row
encounters invalid tick data or margin results from the broker.
SL/TP ratios for `determine_order_limits()` must satisfy `0 <= ratio < 1`; `0`
omits that level. SL/TP prices are rounded with symbol `digits` metadata when
available. `determine_order_limits()` pre-validates computed SL/TP prices against
available `trade_stops_level * point` metadata when present; violations raise
`Mt5TradingError`. This is a planning helper only: it does not guarantee broker
acceptance because live validation can still depend on price movement, bid/ask
side, freeze levels, and server-side rules, and it does not validate
`trade_freeze_level`. When symbol metadata cannot be loaded, protective prices
still round with `digits=8` and stop-level validation is skipped.
`unit_margin_ratio` and `preserved_margin_ratio` for `calculate_margin_and_volume()`
accept `0 <= ratio <= 1`; `unit_margin_ratio=0` requests one minimum valid unit
when the post-reserve margin can afford it. Negative `margin_free` is clamped to
`0.0` before sizing. Execution helpers return normalized `OrderExecutionResult`
dictionaries containing the request, response, status, retcode, and `dry_run`
flag; `dry_run=True` never sends an order or mutates Market Watch visibility.
`ensure_symbol_selected()` adds hidden symbols to Market Watch before live order
placement and SL/TP updates. Failed, malformed, or unknown broker retcodes are
fail-closed and returned as `status="failed"` while keeping the normalized
response for inspection.
## Order planning return contracts
```python
from mt5cli import MarginVolume, OrderLimits, OrderExecutionResult
sizing: MarginVolume = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
limits: OrderLimits = determine_order_limits(
client,
"EURUSD",
side="long",
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.02,
)
preview: OrderExecutionResult = place_market_order(
client,
symbol="EURUSD",
volume=sizing["buy_volume"],
order_side="BUY",
sl=limits["stop_loss"],
tp=limits["take_profit"],
dry_run=True,
)
updates: list[OrderExecutionResult] = update_sltp_for_open_positions(
client,
symbol="EURUSD",
stop_loss=limits["stop_loss"],
dry_run=True,
)
```
Closes issue #33: strategy-neutral order planning and execution helpers exposed
through the stable package root without embedding entry/exit policy.
## Migration from application-local helpers
| Application-local concern | mt5cli replacement |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Manual terminal spawn/kill around trading code | `mt5_trading_session()` |
| Local position-side detection | `detect_position_side()` |
| Local margin/volume sizing | `calculate_margin_and_volume()` |
| Local broker volume step normalization | `normalize_order_volume()` |
| Local order or position margin estimation | `estimate_order_margin()`, `calculate_positions_margin()` |
| Local closed-bar fetch from a trading session | `fetch_latest_closed_rates_for_trading_client()`, `fetch_latest_closed_rates_indexed()` |
| Local SL/TP price derivation | `determine_order_limits()` |
| Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)` |
Keep read-only data collection on `mt5_session()` / `MT5Client`; use
`mt5_trading_session()` only where order placement or trading calculations are
required.
+39 -31
View File
@@ -1,10 +1,16 @@
# mt5cli
Command-line tool for MetaTrader 5 data export.
Generic MT5 data and execution infrastructure for Python applications.
## Overview
mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple file formats. It is built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
mt5cli provides a stable `MT5Client` Python API, standardized dataset schemas, storage helpers, and a CLI for exporting MetaTrader 5 data. It is built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
## Architecture
- **pdmt5** — canonical MT5 client, DataFrame/trading primitives, and MT5 constant parsing (`TIMEFRAME_*`, `COPY_TICKS_*`, order types).
- **mt5cli** — public `MT5Client` API, schema contracts, storage helpers, CLI commands, and SQLite history collection built on pdmt5.
- **mt5api** — sibling HTTP adapter for remote MT5 access; not a dependency of mt5cli.
## Features
@@ -21,66 +27,68 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
pip install mt5cli
```
## Programmatic usage / SDK usage
## Python API for downstream packages
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
Mt5CliClient,
DataKind,
Dataset,
MT5Client,
build_config,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
mt5_session,
normalize_dataframe,
recent_ticks,
resolve_rate_view_name,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers
rates = copy_rates_range(
"EURUSD",
timeframe="H1",
date_from="2024-01-01",
date_to="2024-02-01",
# Persistent session for multiple calls
with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
rates = client.copy_rates_range(
"EURUSD",
timeframe="H1",
date_from="2024-01-01",
date_to="2024-02-01",
)
positions = client.positions()
check = client.order_check({"action": 1, "symbol": "EURUSD", "volume": 0.1})
# Normalize MT5 frames to the public schema contract before storage
closed_rates = normalize_dataframe(
rates, DataKind.rates, symbol="EURUSD", timeframe="H1"
)
export_dataframe(rates, Path("rates.csv"), "csv")
export_dataframe(closed_rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
# Offline rate loading from mt5cli-managed SQLite history
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
# One-off helpers still work without instantiating a client
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
Path("history.db"),
symbols=["EURUSD", "GBPUSD"],
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
timeframe="M1",
flags="ALL",
with_views=True,
datasets={Dataset.rates, Dataset.history_deals},
)
```
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly (the CLI requires `--yes` for `order-send`).
`MT5Client.mt5_summary()` returns structured nested Python values. Use `MT5Client.mt5_summary_as_df()` when you need a one-row DataFrame for export.
## Quick Start
+8 -1
View File
@@ -1,5 +1,5 @@
site_name: mt5cli API Documentation
site_description: Command-line tool for MetaTrader 5
site_description: Generic MT5 data and execution infrastructure for Python
site_author: dceoy
site_url: https://github.com/dceoy/mt5cli
@@ -56,8 +56,15 @@ nav:
- Home: index.md
- API Reference:
- Overview: api/index.md
- Public API Contract: api/public-contract.md
- Client: api/client.md
- Schemas: api/schemas.md
- Storage: api/storage.md
- Converters: api/converters.md
- Exceptions: api/exceptions.md
- CLI: api/cli.md
- SDK: api/sdk.md
- Trading: api/trading.md
- History Collection (SQLite): api/history.md
- Utils: api/utils.md
+215 -6
View File
@@ -1,19 +1,82 @@
"""mt5cli: Command-line tool and SDK for MetaTrader 5."""
"""mt5cli: Generic MT5 data and execution infrastructure for Python applications.
Downstream packages should import from this module (``from mt5cli import ...``)
rather than private submodule helpers. See ``docs/api/public-contract.md`` for
the stable SDK contract, CLI surface, internal modules, and out-of-scope
strategy responsibilities.
"""
from importlib.metadata import version
from .history import load_rate_data, load_rate_data_from_connection
from pdmt5 import Mt5Config, Mt5RuntimeError, Mt5TradingClient, Mt5TradingError
from .client import MT5Client, build_config, mt5_session
from .contract import (
PUBLIC_EXPORT_TIERS,
SECONDARY_PUBLIC_EXPORTS,
STABLE_SDK_EXPORTS,
)
from .converters import (
ensure_utc,
granularity_name,
normalize_symbol,
normalize_symbols,
parse_date_range,
recent_window,
)
from .exceptions import (
Mt5CliError,
Mt5ConnectionError,
Mt5OperationError,
Mt5SchemaError,
call_with_normalized_errors,
is_recoverable_mt5_error,
normalize_mt5_exception,
)
from .history import (
RateTarget,
build_rate_targets,
build_rate_view_name,
drop_forming_rate_bar,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_table_name,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
)
from .schemas import (
DEDUP_KEYS,
KNOWN_MT5_TIME_COLUMNS,
REQUIRED_COLUMNS,
TIME_COLUMNS,
DataKind,
normalize_dataframe,
normalize_time_columns,
schema_columns,
validate_schema,
)
from .sdk import (
Mt5CliClient,
AccountSpec,
ThrottledHistoryUpdater,
account_info,
build_config,
collect_history,
collect_latest_closed_rates_by_granularity,
collect_latest_closed_rates_for_accounts,
collect_latest_rates,
collect_latest_rates_for_accounts,
collect_latest_rates_for_accounts_with_retries,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
copy_ticks_from,
copy_ticks_range,
fetch_latest_closed_rates,
history_deals,
history_orders,
last_error,
@@ -26,6 +89,9 @@ from .sdk import (
positions,
recent_history_deals,
recent_ticks,
resolve_account_spec,
resolve_account_specs,
substitute_env_placeholders,
symbol_info,
symbol_info_tick,
symbols,
@@ -36,51 +102,194 @@ from .sdk import (
from .sdk import (
version as mt5_version,
)
from .utils import (
from .storage import (
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
)
from .trading import (
POSITION_COLUMNS,
ExecutionStatus,
MarginVolume,
OrderExecutionResult,
OrderFillingMode,
OrderLimits,
OrderSide,
OrderTimeMode,
PositionSide,
calculate_account_projected_margin_ratio,
calculate_margin_and_volume,
calculate_new_position_margin_ratio,
calculate_positions_margin,
calculate_positions_margin_by_symbol,
calculate_positions_margin_safe,
calculate_projected_margin_ratio,
calculate_spread_ratio,
calculate_symbol_group_margin_ratio,
calculate_trailing_stop_updates,
calculate_volume_by_margin,
close_open_positions,
create_trading_client,
detect_position_side,
determine_order_limits,
ensure_symbol_selected,
estimate_order_margin,
extract_tick_price,
fetch_latest_closed_rates_for_trading_client,
fetch_latest_closed_rates_indexed,
get_account_snapshot,
get_positions_frame,
get_symbol_snapshot,
get_tick_snapshot,
mt5_trading_session,
normalize_order_volume,
place_market_order,
update_sltp_for_open_positions,
update_trailing_stop_loss_for_open_positions,
)
from .utils import (
TICK_FLAG_MAP,
TIMEFRAME_MAP,
parse_datetime,
parse_tick_flags,
parse_timeframe,
)
__version__ = version(__package__) if __package__ else None
__all__ = [
"DEDUP_KEYS",
"KNOWN_MT5_TIME_COLUMNS",
"POSITION_COLUMNS",
"PUBLIC_EXPORT_TIERS",
"REQUIRED_COLUMNS",
"SECONDARY_PUBLIC_EXPORTS",
"STABLE_SDK_EXPORTS",
"TICK_FLAG_MAP",
"TIMEFRAME_MAP",
"TIME_COLUMNS",
"AccountSpec",
"DataKind",
"Dataset",
"ExecutionStatus",
"IfExists",
"Mt5CliClient",
"MT5Client",
"MarginVolume",
"Mt5CliError",
"Mt5Config",
"Mt5ConnectionError",
"Mt5OperationError",
"Mt5RuntimeError",
"Mt5SchemaError",
"Mt5TradingClient",
"Mt5TradingError",
"OrderExecutionResult",
"OrderFillingMode",
"OrderLimits",
"OrderSide",
"OrderTimeMode",
"PositionSide",
"RateTarget",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"build_rate_targets",
"build_rate_view_name",
"calculate_account_projected_margin_ratio",
"calculate_margin_and_volume",
"calculate_new_position_margin_ratio",
"calculate_positions_margin",
"calculate_positions_margin_by_symbol",
"calculate_positions_margin_safe",
"calculate_projected_margin_ratio",
"calculate_spread_ratio",
"calculate_symbol_group_margin_ratio",
"calculate_trailing_stop_updates",
"calculate_volume_by_margin",
"call_with_normalized_errors",
"close_open_positions",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
"copy_ticks_from",
"copy_ticks_range",
"create_trading_client",
"detect_format",
"detect_position_side",
"determine_order_limits",
"drop_forming_rate_bar",
"ensure_symbol_selected",
"ensure_utc",
"estimate_order_margin",
"export_dataframe",
"export_dataframe_to_sqlite",
"extract_tick_price",
"fetch_latest_closed_rates",
"fetch_latest_closed_rates_for_trading_client",
"fetch_latest_closed_rates_indexed",
"get_account_snapshot",
"get_positions_frame",
"get_symbol_snapshot",
"get_tick_snapshot",
"granularity_name",
"history_deals",
"history_orders",
"is_recoverable_mt5_error",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"mt5_trading_session",
"mt5_version",
"normalize_dataframe",
"normalize_mt5_exception",
"normalize_order_volume",
"normalize_symbol",
"normalize_symbols",
"normalize_time_columns",
"orders",
"parse_date_range",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"place_market_order",
"positions",
"recent_history_deals",
"recent_ticks",
"recent_window",
"resolve_account_spec",
"resolve_account_specs",
"resolve_history_datasets",
"resolve_history_tick_flags",
"resolve_history_timeframes",
"resolve_rate_table_name",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"schema_columns",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
"terminal_info",
"update_history",
"update_history_with_config",
"update_sltp_for_open_positions",
"update_trailing_stop_loss_for_open_positions",
"validate_schema",
]
+60 -74
View File
@@ -12,6 +12,7 @@ import typer
from pdmt5 import Mt5Config
from . import sdk
from .client import MT5Client
from .utils import (
DATETIME_TYPE,
REQUEST_TYPE,
@@ -91,9 +92,18 @@ def _execute_export(
)
def _sdk_client(ctx: typer.Context) -> sdk.Mt5CliClient:
def _sdk_client(ctx: typer.Context) -> MT5Client:
export_ctx = _get_export_context(ctx)
return sdk.Mt5CliClient(config=export_ctx.config)
return MT5Client(config=export_ctx.config)
def _export_command(
ctx: typer.Context,
fetch_fn: Callable[[MT5Client], pd.DataFrame],
) -> None:
"""Create an SDK client, fetch a DataFrame, and export it."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: fetch_fn(client))
@app.callback()
@@ -193,10 +203,9 @@ def rates_from(
count: Annotated[int, typer.Option(help="Number of records.")],
) -> None:
"""Export rates from a start date."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.copy_rates_from(symbol, timeframe, date_from, count),
lambda client: client.copy_rates_from(symbol, timeframe, date_from, count),
)
@@ -215,10 +224,14 @@ def rates_from_pos(
count: Annotated[int, typer.Option(help="Number of records.")],
) -> None:
"""Export rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.copy_rates_from_pos(symbol, timeframe, start_pos, count),
lambda client: client.copy_rates_from_pos(
symbol,
timeframe,
start_pos,
count,
),
)
@@ -240,10 +253,14 @@ def latest_rates(
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
lambda client: client.latest_rates(
symbol,
timeframe,
count,
start_pos=start_pos,
),
)
@@ -268,10 +285,9 @@ def rates_range(
],
) -> None:
"""Export rates for a date range."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.copy_rates_range(symbol, timeframe, date_from, date_to),
lambda client: client.copy_rates_range(symbol, timeframe, date_from, date_to),
)
@@ -293,10 +309,9 @@ def ticks_from(
],
) -> None:
"""Export ticks from a start date."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.copy_ticks_from(symbol, date_from, count, flags),
lambda client: client.copy_ticks_from(symbol, date_from, count, flags),
)
@@ -318,10 +333,9 @@ def ticks_range(
],
) -> None:
"""Export ticks for a date range."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.copy_ticks_range(symbol, date_from, date_to, flags),
lambda client: client.copy_ticks_range(symbol, date_from, date_to, flags),
)
@@ -347,13 +361,12 @@ def ticks_recent(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
] = "ALL", # pyright: ignore[reportArgumentType]
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.recent_ticks(
lambda client: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
@@ -366,13 +379,13 @@ def ticks_recent(
@app.command()
def account_info(ctx: typer.Context) -> None:
"""Export account information."""
_execute_export(ctx, _sdk_client(ctx).account_info)
_export_command(ctx, lambda client: client.account_info())
@app.command()
def terminal_info(ctx: typer.Context) -> None:
"""Export terminal information."""
_execute_export(ctx, _sdk_client(ctx).terminal_info)
_export_command(ctx, lambda client: client.terminal_info())
@app.command()
@@ -384,8 +397,7 @@ def symbols(
] = None,
) -> None:
"""Export symbol list."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.symbols(group=group))
_export_command(ctx, lambda client: client.symbols(group=group))
@app.command()
@@ -394,8 +406,7 @@ def symbol_info(
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export symbol details."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.symbol_info(symbol))
_export_command(ctx, lambda client: client.symbol_info(symbol))
@app.command()
@@ -404,8 +415,7 @@ def minimum_margins(
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
_export_command(ctx, lambda client: client.minimum_margins(symbol))
@app.command()
@@ -416,10 +426,9 @@ def orders(
ticket: Annotated[int | None, typer.Option(help="Ticket filter.")] = None,
) -> None:
"""Export active orders."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.orders(symbol=symbol, group=group, ticket=ticket),
lambda client: client.orders(symbol=symbol, group=group, ticket=ticket),
)
@@ -431,10 +440,9 @@ def positions(
ticket: Annotated[int | None, typer.Option(help="Ticket filter.")] = None,
) -> None:
"""Export open positions."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.positions(symbol=symbol, group=group, ticket=ticket),
lambda client: client.positions(symbol=symbol, group=group, ticket=ticket),
)
@@ -455,10 +463,9 @@ def history_orders(
position: Annotated[int | None, typer.Option(help="Position ticket.")] = None,
) -> None:
"""Export historical orders."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.history_orders(
lambda client: client.history_orders(
date_from=date_from,
date_to=date_to,
group=group,
@@ -486,10 +493,9 @@ def history_deals(
position: Annotated[int | None, typer.Option(help="Position ticket.")] = None,
) -> None:
"""Export historical deals."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.history_deals(
lambda client: client.history_deals(
date_from=date_from,
date_to=date_to,
group=group,
@@ -512,10 +518,9 @@ def recent_history_deals(
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
_export_command(
ctx,
lambda: client.recent_history_deals(
lambda client: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
@@ -527,20 +532,19 @@ def recent_history_deals(
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
_export_command(ctx, lambda client: client.mt5_summary_as_df())
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
_execute_export(ctx, _sdk_client(ctx).version)
_export_command(ctx, lambda client: client.version())
@app.command()
def last_error(ctx: typer.Context) -> None:
"""Export the last error information."""
_execute_export(ctx, _sdk_client(ctx).last_error)
_export_command(ctx, lambda client: client.last_error())
@app.command()
@@ -549,8 +553,7 @@ def symbol_info_tick(
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export the last tick for a symbol."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.symbol_info_tick(symbol))
_export_command(ctx, lambda client: client.symbol_info_tick(symbol))
@app.command()
@@ -559,8 +562,7 @@ def market_book(
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export market depth (order book) for a symbol."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.market_book(symbol))
_export_command(ctx, lambda client: client.market_book(symbol))
@app.command()
@@ -572,15 +574,7 @@ def order_check(
],
) -> None:
"""Check funds sufficiency for a trading operation."""
export_ctx = _get_export_context(ctx)
def _fetch() -> pd.DataFrame:
return sdk._run_with_client( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
export_ctx.config,
lambda c: c.order_check_as_df(request=request),
)
_execute_export(ctx, _fetch)
_export_command(ctx, lambda client: client.order_check(request))
@app.command()
@@ -603,15 +597,7 @@ def order_send(
if not yes:
msg = "Pass --yes to send a live trade request."
raise typer.BadParameter(msg, param_hint="--yes")
export_ctx = _get_export_context(ctx)
def _fetch() -> pd.DataFrame:
return sdk._run_with_client( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
export_ctx.config,
lambda c: c.order_send_as_df(request=request),
)
_execute_export(ctx, _fetch)
_export_command(ctx, lambda client: client.order_send(request))
@app.command()
@@ -656,7 +642,7 @@ def collect_history(
click_type=TICK_FLAGS_TYPE,
help="Tick copy flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
] = "ALL", # pyright: ignore[reportArgumentType]
if_exists: Annotated[
IfExists,
typer.Option(
+86
View File
@@ -0,0 +1,86 @@
"""Stable public client abstraction for MT5 data and execution operations."""
from __future__ import annotations
from contextlib import contextmanager
from typing import TYPE_CHECKING, Any, Self
from .sdk import Mt5CliClient, build_config, connected_client
if TYPE_CHECKING:
from collections.abc import Iterator
import pandas as pd
from pdmt5 import Mt5Config, Mt5DataClient
__all__ = [
"MT5Client",
"build_config",
"mt5_session",
]
class MT5Client(Mt5CliClient):
"""Public client for generic MT5 data access and order primitives.
Extends the read-only SDK client with optional order check/send helpers and
exposes the same connection lifecycle as :func:`mt5_session`.
mt5cli intentionally exposes minimal execution primitives only. Trading
decisions, signals, strategies, backtests, and optimization remain the
responsibility of downstream applications.
"""
def order_check(self, request: dict[str, Any]) -> pd.DataFrame:
"""Check funds sufficiency for a trade request.
Args:
request: MT5 order request dictionary.
Returns:
One-row DataFrame with the order-check result.
"""
return self._fetch(lambda client: client.order_check_as_df(request=request))
def order_send(self, request: dict[str, Any]) -> pd.DataFrame:
"""Send a live trade request to the MT5 trade server.
Warning:
This is a live execution primitive. A successful call can place,
modify, or close real trades on the connected account. Downstream
applications must gate usage explicitly (for example behind manual
confirmation or application-specific risk controls). mt5cli does
not implement strategy logic, signal generation, or trade sizing.
Args:
request: MT5 order request dictionary.
Returns:
One-row DataFrame with the order-send result.
"""
return self._fetch(lambda client: client.order_send_as_df(request=request))
@classmethod
def from_connected_client(cls, client: Mt5DataClient) -> Self:
"""Bind to an already-connected ``Mt5DataClient`` without owning it.
Returns:
Client wrapper bound to the injected connection.
"""
return cls(client=client)
@contextmanager
def mt5_session(config: Mt5Config | None = None) -> Iterator[MT5Client]:
"""Open an MT5 terminal session and yield a connected :class:`MT5Client`.
Args:
config: MT5 connection configuration. Defaults to an empty config that
attaches to a running terminal.
Yields:
Connected :class:`MT5Client` bound to the session.
"""
mt5_config = config or build_config()
with connected_client(mt5_config) as client:
yield MT5Client.from_connected_client(client)
+149
View File
@@ -0,0 +1,149 @@
"""Downstream SDK export tiers for mt5cli."""
from __future__ import annotations
STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
"AccountSpec",
"MT5Client",
"Mt5CliError",
"Mt5Config",
"Mt5ConnectionError",
"Mt5OperationError",
"Mt5RuntimeError",
"Mt5SchemaError",
"Mt5TradingClient",
"Mt5TradingError",
"OrderFillingMode",
"OrderSide",
"OrderTimeMode",
"PositionSide",
"ExecutionStatus",
"MarginVolume",
"OrderExecutionResult",
"OrderLimits",
"RateTarget",
"ThrottledHistoryUpdater",
"build_config",
"build_rate_targets",
"build_rate_view_name",
"calculate_account_projected_margin_ratio",
"calculate_margin_and_volume",
"calculate_new_position_margin_ratio",
"calculate_projected_margin_ratio",
"calculate_positions_margin",
"calculate_positions_margin_by_symbol",
"calculate_positions_margin_safe",
"calculate_spread_ratio",
"calculate_symbol_group_margin_ratio",
"calculate_trailing_stop_updates",
"calculate_volume_by_margin",
"call_with_normalized_errors",
"close_open_positions",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"create_trading_client",
"detect_position_side",
"determine_order_limits",
"drop_forming_rate_bar",
"ensure_symbol_selected",
"estimate_order_margin",
"extract_tick_price",
"fetch_latest_closed_rates",
"fetch_latest_closed_rates_for_trading_client",
"fetch_latest_closed_rates_indexed",
"get_account_snapshot",
"get_positions_frame",
"get_symbol_snapshot",
"get_tick_snapshot",
"is_recoverable_mt5_error",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"mt5_session",
"mt5_trading_session",
"normalize_mt5_exception",
"normalize_order_volume",
"place_market_order",
"resolve_account_spec",
"resolve_account_specs",
"resolve_history_datasets",
"resolve_history_tick_flags",
"resolve_history_timeframes",
"resolve_rate_table_name",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"substitute_env_placeholders",
"update_history",
"update_history_with_config",
"update_sltp_for_open_positions",
"update_trailing_stop_loss_for_open_positions",
})
SECONDARY_PUBLIC_EXPORTS: frozenset[str] = frozenset({
"DEDUP_KEYS",
"DataKind",
"Dataset",
"IfExists",
"KNOWN_MT5_TIME_COLUMNS",
"POSITION_COLUMNS",
"REQUIRED_COLUMNS",
"TICK_FLAG_MAP",
"TIMEFRAME_MAP",
"TIME_COLUMNS",
"account_info",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
"copy_ticks_from",
"copy_ticks_range",
"detect_format",
"ensure_utc",
"export_dataframe",
"export_dataframe_to_sqlite",
"granularity_name",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"market_book",
"minimum_margins",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version",
"normalize_dataframe",
"normalize_symbol",
"normalize_symbols",
"normalize_time_columns",
"orders",
"parse_date_range",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"positions",
"recent_history_deals",
"recent_ticks",
"recent_window",
"schema_columns",
"symbol_info",
"symbol_info_tick",
"symbols",
"terminal_info",
"validate_schema",
})
PUBLIC_EXPORT_TIERS: dict[str, frozenset[str]] = {
"stable": STABLE_SDK_EXPORTS,
"secondary": SECONDARY_PUBLIC_EXPORTS,
}
__all__ = [
"PUBLIC_EXPORT_TIERS",
"SECONDARY_PUBLIC_EXPORTS",
"STABLE_SDK_EXPORTS",
]
+162
View File
@@ -0,0 +1,162 @@
"""Shared conversion helpers for MT5 symbols, timeframes, and date ranges."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from pdmt5 import get_timeframe_name as _get_timeframe_name
from .utils import parse_datetime, parse_tick_flags, parse_timeframe
if TYPE_CHECKING:
from collections.abc import Sequence
__all__ = [
"ensure_utc",
"granularity_name",
"normalize_symbol",
"normalize_symbols",
"parse_date_range",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"recent_window",
]
def normalize_symbol(symbol: str) -> str:
"""Normalize a broker symbol name for MT5 API calls.
Strips surrounding whitespace while preserving broker-specific casing and
suffixes (for example ``XAUUSDm``, ``US500.cash``, or ``EURUSD.r``).
Args:
symbol: Raw symbol name.
Returns:
Normalized symbol string.
Raises:
ValueError: If the symbol is empty after normalization.
"""
normalized = symbol.strip()
if not normalized:
msg = "Symbol must not be empty."
raise ValueError(msg)
return normalized
def normalize_symbols(symbols: Sequence[str]) -> list[str]:
"""Normalize a sequence of broker symbol names.
Args:
symbols: Raw symbol names.
Returns:
List of normalized, de-duplicated symbols preserving first-seen order.
"""
seen: set[str] = set()
resolved: list[str] = []
for symbol in symbols:
normalized = normalize_symbol(symbol)
if normalized not in seen:
seen.add(normalized)
resolved.append(normalized)
return resolved
def ensure_utc(value: datetime | str) -> datetime:
"""Return a timezone-aware UTC datetime.
Args:
value: Datetime instance or ISO 8601 string.
Returns:
UTC-aware datetime.
"""
if isinstance(value, str):
return parse_datetime(value)
if value.tzinfo is None:
return value.replace(tzinfo=UTC)
return value.astimezone(UTC)
def parse_date_range(
date_from: datetime | str,
date_to: datetime | str,
) -> tuple[datetime, datetime]:
"""Parse and validate an inclusive UTC date range.
Args:
date_from: Range start as datetime or ISO 8601 string.
date_to: Range end as datetime or ISO 8601 string.
Returns:
Tuple of UTC-aware ``(start, end)`` datetimes.
Raises:
ValueError: If ``date_from`` is after ``date_to``.
"""
start = ensure_utc(date_from)
end = ensure_utc(date_to)
if start > end:
msg = (
f"date_from ({start.isoformat()}) must not be after "
f"date_to ({end.isoformat()})."
)
raise ValueError(msg)
return start, end
def recent_window(
*,
hours: float | None = None,
seconds: float | None = None,
date_to: datetime | str | None = None,
) -> tuple[datetime, datetime]:
"""Build a trailing UTC window ending at ``date_to`` or now.
Exactly one of ``hours`` or ``seconds`` must be provided.
Args:
hours: Trailing window length in hours.
seconds: Trailing window length in seconds.
date_to: Window end. Defaults to current UTC time.
Returns:
Tuple of UTC-aware ``(start, end)`` datetimes.
Raises:
ValueError: If neither or both window lengths are provided, or if a
length is not positive.
"""
if (hours is None) == (seconds is None):
msg = "Provide exactly one of hours or seconds."
raise ValueError(msg)
if hours is not None:
length = timedelta(hours=hours)
else:
length = timedelta(seconds=seconds if seconds is not None else 0)
if length.total_seconds() <= 0:
msg = "Window length must be positive."
raise ValueError(msg)
end = ensure_utc(date_to) if date_to is not None else datetime.now(UTC)
return end - length, end
def granularity_name(timeframe: int | str) -> str:
"""Return a short granularity label for a timeframe integer or name.
Args:
timeframe: MT5 timeframe as integer or name (for example ``M1``).
Returns:
Short name such as ``M1`` or the stringified integer when unknown.
"""
tf = parse_timeframe(timeframe)
try:
name = _get_timeframe_name(tf)
except ValueError:
return str(tf)
return name.removeprefix("TIMEFRAME_")
+90
View File
@@ -0,0 +1,90 @@
"""Normalized exception types for MT5 and mt5cli operations."""
from __future__ import annotations
from typing import TYPE_CHECKING, TypeVar
from pdmt5 import Mt5RuntimeError, Mt5TradingError
if TYPE_CHECKING:
from collections.abc import Callable
T = TypeVar("T")
__all__ = [
"Mt5CliError",
"Mt5ConnectionError",
"Mt5OperationError",
"Mt5SchemaError",
"call_with_normalized_errors",
"is_recoverable_mt5_error",
"normalize_mt5_exception",
]
_RECOVERABLE_MT5_ERRORS: tuple[type[BaseException], ...] = (
Mt5TradingError,
Mt5RuntimeError,
)
class Mt5CliError(Exception):
"""Base exception for mt5cli public API errors."""
class Mt5ConnectionError(Mt5CliError):
"""Raised when MT5 initialization, login, or shutdown fails."""
class Mt5OperationError(Mt5CliError):
"""Raised when an MT5 data or trading operation fails."""
class Mt5SchemaError(Mt5CliError):
"""Raised when a DataFrame does not match an expected dataset schema."""
def is_recoverable_mt5_error(exc: BaseException) -> bool:
"""Return whether an exception is a transient MT5 failure worth retrying.
Args:
exc: Exception raised by MT5 or pdmt5.
Returns:
True for ``Mt5RuntimeError`` and ``Mt5TradingError``.
"""
return isinstance(exc, _RECOVERABLE_MT5_ERRORS)
def normalize_mt5_exception(exc: BaseException) -> Mt5CliError:
"""Map pdmt5/MT5 exceptions to stable mt5cli exception types.
Args:
exc: Original exception from MT5 or pdmt5.
Returns:
``Mt5ConnectionError`` for runtime failures, ``Mt5OperationError`` for
trading failures, or the original exception when it is not recognized.
"""
if isinstance(exc, Mt5TradingError):
return Mt5OperationError(str(exc))
if isinstance(exc, Mt5RuntimeError):
return Mt5ConnectionError(str(exc))
if isinstance(exc, Mt5CliError):
return exc
return Mt5CliError(str(exc))
def call_with_normalized_errors(fn: Callable[[], T]) -> T:
"""Run ``fn`` and map recoverable MT5 errors to mt5cli types.
Args:
fn: Callable performing MT5 work.
Returns:
Value returned by ``fn``.
"""
try:
return fn()
except _RECOVERABLE_MT5_ERRORS as exc:
normalized = normalize_mt5_exception(exc)
raise normalized from exc
+504 -77
View File
@@ -4,14 +4,17 @@ from __future__ import annotations
import logging
import sqlite3
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import TYPE_CHECKING, Literal, cast
from typing import TYPE_CHECKING, Literal, cast, overload
import pandas as pd
from pdmt5 import get_timeframe_name as _get_timeframe_name
from .schemas import DEDUP_KEYS, DataKind
from .utils import (
TIMEFRAME_MAP,
TIMEFRAME_NAMES,
Dataset,
IfExists,
parse_datetime,
@@ -20,19 +23,19 @@ from .utils import (
)
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from collections.abc import Callable, Mapping, Sequence
from pdmt5 import Mt5DataClient
logger = logging.getLogger(__name__)
DEFAULT_HISTORY_TIMEFRAMES: tuple[str, ...] = tuple(TIMEFRAME_MAP)
DEFAULT_HISTORY_TIMEFRAMES: tuple[str, ...] = TIMEFRAME_NAMES
_HISTORY_DEDUP_KEYS: dict[Dataset, tuple[tuple[str, ...], ...]] = {
Dataset.rates: (("symbol", "timeframe", "time"), ("symbol", "time")),
Dataset.ticks: (("symbol", "time_msc"), ("symbol", "time")),
Dataset.history_orders: (("ticket",), ("symbol", "time", "type")),
Dataset.history_deals: (("ticket",), ("symbol", "time", "type", "entry")),
Dataset.rates: DEDUP_KEYS[DataKind.rates],
Dataset.ticks: DEDUP_KEYS[DataKind.ticks],
Dataset.history_orders: DEDUP_KEYS[DataKind.history_orders],
Dataset.history_deals: DEDUP_KEYS[DataKind.history_deals],
}
_TRADE_DEAL_TYPES: tuple[int, int] = (0, 1)
@@ -79,7 +82,7 @@ def resolve_history_timeframes(
seen: set[int] = set()
resolved: list[int] = []
for value in raw:
tf = value if isinstance(value, int) else parse_timeframe(str(value))
tf = parse_timeframe(value)
if tf not in seen:
seen.add(tf)
resolved.append(tf)
@@ -92,17 +95,33 @@ def resolve_history_tick_flags(flags: int | str) -> int:
Returns:
Integer tick flag value.
"""
if isinstance(flags, int):
return flags
return parse_tick_flags(flags)
def resolve_granularity_name(timeframe: int) -> str:
"""Return a granularity name for a timeframe integer when known."""
for name, value in TIMEFRAME_MAP.items():
if value == timeframe:
return name
return str(timeframe)
try:
name = _get_timeframe_name(timeframe)
except ValueError:
return str(timeframe)
return name.removeprefix("TIMEFRAME_")
def drop_forming_rate_bar(df_rate: pd.DataFrame) -> pd.DataFrame:
"""Return closed bars from chronologically ordered MT5 rate data.
MetaTrader 5 ``copy_rates_from_pos(start_pos=0)`` includes the still-forming
current bar as the last row. Slice it off so downstream logic only sees
completed bars. Empty frames and single-row frames return empty results.
Args:
df_rate: Rate data ordered oldest-to-newest with the forming bar last.
Returns:
A new DataFrame with all rows except the last. Index and columns are
preserved. The input frame is not modified.
"""
return df_rate.iloc[:-1].copy()
def build_rate_view_name(
@@ -123,6 +142,26 @@ def build_rate_view_name(
return f"rate_{symbol}__{granularity}_{timeframe}"
def resolve_rate_table_name(symbol: str, granularity: str) -> str:
"""Return the canonical normalized SQLite rate table name.
The normalized history table stores all symbols and timeframes in
``rates``; use :func:`resolve_rate_view_name` for per-symbol compatibility
view names.
Returns:
Canonical normalized rates table name.
Raises:
ValueError: If ``symbol`` or ``granularity`` is invalid.
"""
parse_timeframe(granularity)
if not symbol.strip():
msg = "symbol must not be empty."
raise ValueError(msg)
return Dataset.rates.table_name
SqliteConnOrPath = sqlite3.Connection | Path | str
@@ -135,14 +174,17 @@ def _require_non_empty_identifier(identifier: str, kind: str) -> str:
def _open_history_connection(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
) -> tuple[sqlite3.Connection | None, bool]:
"""Open a read-only SQLite connection when given a path.
Returns:
A connection and whether the caller should close it. When the path does
not exist, returns ``(None, False)`` without creating a database file.
A connection and whether the caller should close it. When ``conn_or_path``
is None or the path does not exist, returns ``(None, False)`` without
creating a database file.
"""
if conn_or_path is None:
return None, False
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
@@ -376,7 +418,7 @@ def _resolve_rate_view_name_from_context(
def resolve_rate_view_name(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
symbol: str,
granularity: str,
*,
@@ -385,7 +427,9 @@ def resolve_rate_view_name(
"""Resolve the mt5cli-managed rate compatibility view name.
Args:
conn_or_path: SQLite database path or open connection.
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, the deterministic
default view name is returned without creating a database file.
symbol: Symbol stored in the normalized ``rates`` table.
granularity: Timeframe name (for example ``M1``) or integer string.
require_existing: When True, require the database and a managed view to exist.
@@ -429,7 +473,7 @@ def resolve_rate_view_name(
def resolve_rate_view_names(
conn_or_path: SqliteConnOrPath,
conn_or_path: SqliteConnOrPath | None,
symbols: Sequence[str],
granularities: Sequence[str],
*,
@@ -438,7 +482,9 @@ def resolve_rate_view_names(
"""Resolve rate compatibility view names for symbol and granularity pairs.
Args:
conn_or_path: SQLite database path or open connection.
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, deterministic
default view names are returned without creating a database file.
symbols: Symbols stored in the normalized ``rates`` table.
granularities: Timeframe names (for example ``M1``) or integer strings.
require_existing: When True, require the database and managed views to exist.
@@ -482,6 +528,313 @@ def resolve_rate_view_names(
conn.close()
@dataclass(frozen=True)
class RateTarget:
"""A single rate series identified by symbol and timeframe.
Attributes:
symbol: MT5 symbol name, or None when the rate series is addressed only
by an explicit table (for example a custom SQLite view).
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
"""
symbol: str | None
timeframe: int | str
def __post_init__(self) -> None:
"""Normalize accepted timeframe aliases to the stored integer value."""
if not isinstance(self.timeframe, int):
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
@property
def timeframe_int(self) -> int:
"""Return the timeframe as its integer MT5 value."""
return cast("int", self.timeframe)
def build_rate_targets(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
allow_missing_symbol: bool = False,
) -> list[RateTarget]:
"""Build rate targets for every symbol and timeframe combination.
Args:
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each timeframe instead of raising.
Returns:
Targets in row-major order: every timeframe for the first symbol, then
every timeframe for the next symbol, and so on.
Raises:
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
``allow_missing_symbol`` is False.
"""
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if not symbols:
if not allow_missing_symbol:
msg = "At least one symbol is required."
raise ValueError(msg)
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
return [
RateTarget(symbol=symbol, timeframe=tf)
for symbol in symbols
for tf in timeframes
]
def resolve_rate_tables(
conn_or_path: SqliteConnOrPath | None,
targets: Sequence[RateTarget],
explicit_tables: Sequence[str] | None = None,
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve SQLite table or view names for rate targets.
Args:
conn_or_path: SQLite database path or open connection. May be None when
``explicit_tables`` is provided, or when ``require_existing`` is
False and deterministic default view names are sufficient.
targets: Rate targets to resolve.
explicit_tables: Optional explicit table or view names. When provided,
they are used as-is and must match the number of targets.
require_existing: When True, require the database and managed views to
exist for each symbol target. Ignored when ``explicit_tables`` is
provided.
Returns:
Table or view names aligned with ``targets``.
Raises:
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
match the target count, a target without a symbol is resolved
without an explicit table, or ``require_existing`` is True and the
database or a managed view is missing.
"""
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is not None:
tables = list(explicit_tables)
if len(tables) != len(target_list):
msg = (
f"Expected {len(target_list)} explicit table(s) "
f"to match the targets, got {len(tables)}."
)
raise ValueError(msg)
return tables
if any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
timeframe_counts = None
existing_views: set[str] = set()
else:
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for target in target_list:
symbol = cast("str", target.symbol)
timeframe = target.timeframe_int
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
if TYPE_CHECKING:
@overload
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: None = None,
count: int | None = None,
explicit_tables: None = None,
*,
table: str,
) -> pd.DataFrame: ...
@overload
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: None = None,
count: int | None = None,
explicit_tables: Sequence[str] | None = None,
*,
table: None = None,
) -> dict[tuple[str | None, int], pd.DataFrame]: ...
@overload
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget],
count: int,
explicit_tables: Sequence[str] | None = None,
*,
table: None = None,
) -> dict[tuple[str | None, int], pd.DataFrame]: ...
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget] | None = None,
count: int | None = None,
explicit_tables: Sequence[str] | None = None,
*,
table: str | None = None,
) -> dict[tuple[str | None, int], pd.DataFrame] | pd.DataFrame:
"""Load one table/view or multiple rate series from a SQLite database.
Args:
conn_or_path: SQLite database path or open connection.
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair must
be unique. Omit when loading a single explicit ``table``.
count: Optional number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching targets.
When omitted, managed ``rate_*`` compatibility views must already
exist in the database.
table: Optional single table or view name to load directly.
Returns:
A DataFrame when ``table`` is provided, otherwise a mapping keyed by
``(symbol, timeframe_int)`` to each rate DataFrame.
Raises:
ValueError: If ``count`` is not positive, targets are empty, duplicate
``(symbol, timeframe_int)`` pairs are present, or table resolution
fails.
"""
if table is not None:
return load_rate_data(conn_or_path, table, count=count)
if count is None or count <= 0:
msg = "count must be positive."
raise ValueError(msg)
if targets is None:
msg = "targets are required when table is not provided."
raise ValueError(msg)
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is None and any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
seen_keys: set[tuple[str | None, int]] = set()
for target in target_list:
key = (target.symbol, target.timeframe_int)
if key in seen_keys:
symbol_repr = repr(target.symbol)
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
raise ValueError(msg)
seen_keys.add(key)
tables = (
resolve_rate_tables(None, target_list, explicit_tables)
if explicit_tables is not None
else None
)
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
resolved_tables = tables or resolve_rate_tables(
conn,
target_list,
require_existing=True,
)
return {
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
conn,
table,
count=count,
)
for target, table in zip(target_list, resolved_tables, strict=True)
}
finally:
if should_close:
conn.close()
def load_rate_series_by_granularity(
conn_or_path: SqliteConnOrPath,
symbols: Sequence[str],
granularities: Sequence[int | str],
count: int,
*,
explicit_tables: Sequence[str] | None = None,
allow_missing_symbol: bool = False,
) -> dict[tuple[str | None, str], pd.DataFrame]:
"""Load rate series keyed by symbol and string granularity name.
Builds targets with :func:`build_rate_targets` and loads them with
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
name (for example ``M1``) instead of the integer timeframe to reduce
downstream boilerplate.
Args:
conn_or_path: SQLite database path or open connection.
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching the
built targets in row-major order. Required when symbols are omitted.
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each granularity instead of raising.
Returns:
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
table resolution fails, or duplicate targets are present.
"""
targets = build_rate_targets(
symbols,
granularities,
allow_missing_symbol=allow_missing_symbol,
)
series = load_rate_series_from_sqlite(
conn_or_path,
targets,
count,
explicit_tables=explicit_tables,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in series.items()
}
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
"""Return existing SQLite columns for a table."""
quoted_table = quote_sqlite_identifier(table)
@@ -765,7 +1118,20 @@ def drop_duplicates_in_table(
)
DedupScope = tuple[str, tuple[object, ...]]
@dataclass(frozen=True)
class DedupScope:
"""Scoped deduplication predicate and the columns it references.
Attributes:
where: SQL predicate appended to the duplicate-removal query.
params: Parameters bound to the scope predicate.
required_columns: Columns that must be present in the written table for
the scope to run.
"""
where: str
params: tuple[object, ...]
required_columns: frozenset[str]
def _record_dedup_scope(
@@ -773,17 +1139,25 @@ def _record_dedup_scope(
dataset: Dataset,
scope_where: str,
scope_params: tuple[object, ...],
required_columns: frozenset[str],
) -> None:
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
dedup_scopes.setdefault(dataset, []).append(
DedupScope(scope_where, scope_params, required_columns),
)
def deduplicate_history_tables(
conn: sqlite3.Connection,
written_columns: dict[Dataset, set[str]],
written_tables: set[Dataset],
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
) -> None:
"""Deduplicate appended history tables by stable identifiers."""
"""Deduplicate appended history tables by stable identifiers.
Scopes whose required columns are not present in the written table are
skipped. If all scopes for a dataset are skipped, the table receives one
unscoped deduplication pass instead.
"""
cursor = conn.cursor()
for dataset in written_tables:
columns = written_columns.get(dataset, set())
@@ -802,16 +1176,19 @@ def deduplicate_history_tables(
table,
)
continue
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
raw_scopes: Sequence[DedupScope] = (
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
)
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
if scopes:
for scope_where, scope_params in scopes:
for scope in scopes:
drop_duplicates_in_table(
cursor,
table,
list(keys),
keep="last",
scope_where=scope_where,
scope_params=scope_params,
scope_where=scope.where,
scope_params=scope.params,
)
continue
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
@@ -1018,6 +1395,50 @@ def create_rate_compatibility_views(conn: sqlite3.Connection) -> None:
)
def _stream_symbol_frames(
conn: sqlite3.Connection,
symbols: Sequence[str],
dataset: Dataset,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
fetch_frame: Callable[[str], pd.DataFrame],
) -> bool:
"""Stream per-symbol frames into SQLite.
Returns:
True if the dataset table was written.
"""
table_exists = False
for sym in symbols:
table_exists = write_streamed_frame(
conn,
fetch_frame(sym),
dataset,
table_exists,
if_exists,
written_columns,
)
return table_exists
def _record_symbol_time_dedup(
dedup_scopes: dict[Dataset, list[DedupScope]],
written_tables: set[Dataset],
dataset: Dataset,
symbol: str,
start_date: datetime,
) -> None:
"""Record a symbol-scoped deduplication window after an incremental write."""
written_tables.add(dataset)
_record_dedup_scope(
dedup_scopes,
dataset,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
def write_rates_dataset(
conn: sqlite3.Connection,
client: Mt5DataClient,
@@ -1033,8 +1454,8 @@ def write_rates_dataset(
Returns:
True if the rates table was written.
"""
table_exists = False
for sym in symbols:
def _fetch_rates_frame(sym: str) -> pd.DataFrame:
frame = client.copy_rates_range_as_df(
symbol=sym,
timeframe=timeframe,
@@ -1044,15 +1465,16 @@ def write_rates_dataset(
if len(frame.columns) != 0:
frame.insert(0, "symbol", sym)
frame.insert(1, "timeframe", timeframe)
table_exists = write_streamed_frame(
conn,
frame,
Dataset.rates,
table_exists,
if_exists,
written_columns,
)
return table_exists
return frame
return _stream_symbol_frames(
conn,
symbols,
Dataset.rates,
if_exists,
written_columns,
_fetch_rates_frame,
)
def write_ticks_dataset(
@@ -1070,8 +1492,8 @@ def write_ticks_dataset(
Returns:
True if the ticks table was written.
"""
table_exists = False
for sym in symbols:
def _fetch_ticks_frame(sym: str) -> pd.DataFrame:
frame = client.copy_ticks_range_as_df(
symbol=sym,
date_from=date_from,
@@ -1080,15 +1502,16 @@ def write_ticks_dataset(
).drop(columns=["symbol"], errors="ignore")
if len(frame.columns) != 0:
frame.insert(0, "symbol", sym)
table_exists = write_streamed_frame(
conn,
frame,
Dataset.ticks,
table_exists,
if_exists,
written_columns,
)
return table_exists
return frame
return _stream_symbol_frames(
conn,
symbols,
Dataset.ticks,
if_exists,
written_columns,
_fetch_ticks_frame,
)
def write_history_dataset(
@@ -1123,22 +1546,22 @@ def write_history_dataset(
if_exists,
written_columns,
)
for sym in symbols:
frame = fetch(date_from=date_from, date_to=date_to, symbol=sym)
frame = filter_trade_history_frame(
frame,
def _fetch_history_frame(sym: str) -> pd.DataFrame:
return filter_trade_history_frame(
fetch(date_from=date_from, date_to=date_to, symbol=sym),
[sym],
include_account_events=False,
)
table_exists = write_streamed_frame(
conn,
frame,
dataset,
table_exists,
if_exists,
written_columns,
)
return table_exists
return _stream_symbol_frames(
conn,
symbols,
dataset,
if_exists,
written_columns,
_fetch_history_frame,
)
def _write_incremental_rates(
@@ -1178,6 +1601,7 @@ def _write_incremental_rates(
Dataset.rates,
"symbol = ? AND timeframe = ? AND time >= ?",
(symbol, timeframe, start_date),
frozenset({"symbol", "timeframe", "time"}),
)
@@ -1210,12 +1634,12 @@ def _write_incremental_ticks(
IfExists.APPEND,
written_columns,
):
written_tables.add(Dataset.ticks)
_record_dedup_scope(
_record_symbol_time_dedup(
dedup_scopes,
written_tables,
Dataset.ticks,
"symbol = ? AND time >= ?",
(symbol, start_date),
symbol,
start_date,
)
@@ -1248,12 +1672,12 @@ def _write_incremental_history_orders(
written_columns,
include_account_events=False,
):
written_tables.add(Dataset.history_orders)
_record_dedup_scope(
_record_symbol_time_dedup(
dedup_scopes,
written_tables,
Dataset.history_orders,
"symbol = ? AND time >= ?",
(symbol, start_date),
symbol,
start_date,
)
@@ -1307,6 +1731,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_by_symbol[symbol, None]),
frozenset({"symbol", "time"}),
)
if "type" in columns:
_record_dedup_scope(
@@ -1314,6 +1739,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
(account_event_start,),
frozenset({"type", "time"}),
)
if "type" not in columns and "symbol" in columns:
_record_dedup_scope(
@@ -1321,6 +1747,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"(symbol IS NULL OR symbol = '') AND time >= ?",
(account_event_start,),
frozenset({"symbol", "time"}),
)
return
start_by_symbol = load_incremental_start_datetimes(
@@ -1342,12 +1769,12 @@ def _write_incremental_history_deals(
written_columns,
include_account_events=False,
):
written_tables.add(Dataset.history_deals)
_record_dedup_scope(
_record_symbol_time_dedup(
dedup_scopes,
written_tables,
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_date),
symbol,
start_date,
)
+64
View File
@@ -0,0 +1,64 @@
"""Retry and reconnect helpers for transient MT5 failures."""
from __future__ import annotations
import logging
import time
from typing import TYPE_CHECKING, TypeVar
from .exceptions import is_recoverable_mt5_error
if TYPE_CHECKING:
from collections.abc import Callable
T = TypeVar("T")
logger = logging.getLogger(__name__)
__all__ = [
"retry_with_backoff",
]
def retry_with_backoff(
fn: Callable[[], T],
*,
retry_count: int = 0,
backoff_base: float = 2.0,
operation: str = "MT5 operation",
) -> T:
"""Call ``fn`` with bounded exponential backoff on recoverable MT5 errors.
Only ``pdmt5.Mt5RuntimeError`` and ``pdmt5.Mt5TradingError`` are retried.
Other exceptions propagate immediately. The final failure is re-raised once
retries are exhausted.
Args:
fn: Callable performing MT5 work.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff. The delay before retry
attempt ``n`` (1-indexed) is ``backoff_base ** n`` seconds.
operation: Label used in warning logs.
Returns:
Value returned by ``fn`` on success.
"""
attempts = max(retry_count, 0) + 1
for attempt in range(attempts - 1):
try:
return fn()
except Exception as exc:
if not is_recoverable_mt5_error(exc):
raise
delay = backoff_base ** (attempt + 1)
logger.warning(
"%s failed (attempt %d/%d): %s; retrying in %.1fs",
operation,
attempt + 1,
attempts,
exc,
delay,
)
time.sleep(delay)
return fn()
+291
View File
@@ -0,0 +1,291 @@
"""Canonical DataFrame schemas for MT5 market and account datasets."""
from __future__ import annotations
from enum import StrEnum
from typing import TYPE_CHECKING, Final
import pandas as pd
from .converters import normalize_symbol, parse_timeframe
from .exceptions import Mt5SchemaError
if TYPE_CHECKING:
from collections.abc import Iterable
__all__ = [
"DEDUP_KEYS",
"KNOWN_MT5_TIME_COLUMNS",
"REQUIRED_COLUMNS",
"TIME_COLUMNS",
"DataKind",
"normalize_dataframe",
"normalize_time_columns",
"schema_columns",
"validate_schema",
]
KNOWN_MT5_TIME_COLUMNS: Final[frozenset[str]] = frozenset({
"time",
"time_setup",
"time_setup_msc",
"time_done",
"time_done_msc",
"time_msc",
})
_TIME_COLUMN_NAMES = KNOWN_MT5_TIME_COLUMNS
class DataKind(StrEnum):
"""Supported MT5 dataset kinds with canonical column contracts."""
rates = "rates"
ticks = "ticks"
orders = "orders"
positions = "positions"
history_orders = "history_orders"
history_deals = "history_deals"
REQUIRED_COLUMNS: dict[DataKind, frozenset[str]] = {
DataKind.rates: frozenset({
"time",
"open",
"high",
"low",
"close",
"tick_volume",
"spread",
"real_volume",
}),
DataKind.ticks: frozenset({
"time",
"bid",
"ask",
"last",
"volume",
"time_msc",
"flags",
"volume_real",
}),
DataKind.orders: frozenset({
"ticket",
"time_setup",
"type",
"state",
"symbol",
"volume_current",
"price_open",
}),
DataKind.positions: frozenset({
"ticket",
"time",
"type",
"symbol",
"volume",
"price_open",
"price_current",
"profit",
}),
DataKind.history_orders: frozenset({
"ticket",
"time_setup",
"type",
"state",
"symbol",
"volume_initial",
"price_open",
}),
DataKind.history_deals: frozenset({
"ticket",
"order",
"time",
"type",
"entry",
"symbol",
"volume",
"price",
"profit",
}),
}
_OPTIONAL_TIME_COLUMNS_BY_KIND: dict[DataKind, frozenset[str]] = {
DataKind.orders: frozenset({
"time_setup_msc",
"time_done",
"time_done_msc",
}),
DataKind.history_orders: frozenset({
"time_setup_msc",
"time_done",
"time_done_msc",
}),
DataKind.positions: frozenset({"time_msc"}),
}
TIME_COLUMNS: dict[DataKind, frozenset[str]] = {
kind: (REQUIRED_COLUMNS[kind] & _TIME_COLUMN_NAMES)
| _OPTIONAL_TIME_COLUMNS_BY_KIND.get(kind, frozenset())
for kind in DataKind
}
DEDUP_KEYS: dict[DataKind, tuple[tuple[str, ...], ...]] = {
DataKind.rates: (("symbol", "timeframe", "time"), ("symbol", "time")),
DataKind.ticks: (("symbol", "time_msc"), ("symbol", "time")),
DataKind.history_orders: (("ticket",), ("symbol", "time", "type")),
DataKind.history_deals: (("ticket",), ("symbol", "time", "type", "entry")),
}
def schema_columns(kind: DataKind) -> frozenset[str]:
"""Return required column names for a dataset kind.
Args:
kind: Dataset kind.
Returns:
Required column names for ``kind``.
"""
return REQUIRED_COLUMNS[kind]
def validate_schema(
frame: pd.DataFrame,
kind: DataKind,
*,
extra_required: Iterable[str] | None = None,
) -> None:
"""Validate that a DataFrame includes required columns for a dataset kind.
Args:
frame: DataFrame to validate.
kind: Expected dataset kind.
extra_required: Additional columns that must be present (for example
``symbol`` and ``timeframe`` on stored rate history).
Raises:
Mt5SchemaError: If required columns are missing.
"""
if frame.empty and len(frame.columns) == 0:
return
required = set(REQUIRED_COLUMNS[kind])
if extra_required is not None:
required.update(extra_required)
missing = required - set(frame.columns)
if missing:
msg = (
f"{kind.value} schema is missing required columns: "
f"{', '.join(sorted(missing))}."
)
raise Mt5SchemaError(msg)
def _coerce_mt5_time_column(series: pd.Series, column: str) -> pd.Series:
"""Coerce one MT5 time column to UTC-aware datetimes.
Returns:
Series with UTC-aware datetime values.
"""
if pd.api.types.is_datetime64_any_dtype(series):
return pd.to_datetime(series, utc=True, errors="coerce")
if pd.api.types.is_numeric_dtype(series):
unit = "ms" if column.endswith("_msc") else "s"
return pd.to_datetime(series, unit=unit, utc=True, errors="coerce")
return pd.to_datetime(series, utc=True, errors="coerce")
def normalize_time_columns(frame: pd.DataFrame, kind: DataKind) -> pd.DataFrame:
"""Coerce dataset time columns to UTC-aware datetimes when present.
Any column in :data:`KNOWN_MT5_TIME_COLUMNS` that is present in ``frame``
is normalized. Numeric MT5 epoch values use seconds for ``time``,
``time_setup``, and ``time_done``, and milliseconds for ``*_msc`` columns.
Args:
frame: Source DataFrame from MT5 or pdmt5.
kind: Dataset kind (retained for API compatibility).
Returns:
DataFrame copy with normalized time columns.
"""
del kind
normalized = frame.copy()
for column in normalized.columns:
if column not in _TIME_COLUMN_NAMES:
continue
normalized[column] = _coerce_mt5_time_column(normalized[column], column)
return normalized
def normalize_dataframe(
frame: pd.DataFrame,
kind: DataKind,
*,
symbol: str | None = None,
timeframe: int | str | None = None,
sort: bool = True,
) -> pd.DataFrame:
"""Normalize MT5 DataFrame columns, timestamps, and storage metadata.
Ensures UTC timestamps, optionally injects ``symbol`` / ``timeframe`` for
storage-oriented datasets, and sorts chronologically when a ``time`` column
exists.
Args:
frame: Source DataFrame from MT5 or pdmt5.
kind: Dataset kind guiding normalization rules.
symbol: Optional symbol to inject when missing.
timeframe: Optional timeframe integer or name to inject for rates.
sort: Whether to sort by ``time`` or ``time_msc`` when present.
Returns:
Normalized DataFrame copy.
"""
if frame.empty and len(frame.columns) == 0:
return frame.copy()
normalized = normalize_time_columns(frame, kind)
if symbol is not None and "symbol" not in normalized.columns:
normalized.insert(0, "symbol", normalize_symbol(symbol))
if timeframe is not None and kind is DataKind.rates:
tf = parse_timeframe(timeframe)
if "timeframe" not in normalized.columns:
insert_at = 1 if "symbol" in normalized.columns else 0
normalized.insert(insert_at, "timeframe", tf)
validate_schema(normalized, kind)
if sort:
if "time" in normalized.columns:
normalized = normalized.sort_values("time", kind="stable")
elif "time_msc" in normalized.columns:
normalized = normalized.sort_values("time_msc", kind="stable")
normalized = normalized.reset_index(drop=True)
return normalized
def ensure_utc_columns(frame: pd.DataFrame, columns: Iterable[str]) -> pd.DataFrame:
"""Return a copy with selected columns coerced to UTC datetimes.
Args:
frame: Source DataFrame.
columns: Column names to coerce.
Returns:
DataFrame copy with UTC-aware datetime columns.
"""
normalized = frame.copy()
for column in columns:
if column not in normalized.columns:
continue
if column in _TIME_COLUMN_NAMES:
normalized[column] = _coerce_mt5_time_column(normalized[column], column)
else:
normalized[column] = pd.to_datetime(
normalized[column], utc=True, errors="coerce"
)
return normalized
+762 -12
View File
@@ -4,26 +4,32 @@ from __future__ import annotations
import json
import logging
import os
import re
import sqlite3
import time
from contextlib import contextmanager
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import TYPE_CHECKING, Self, TypeVar, cast
import pandas as pd
from pdmt5 import Mt5Config, Mt5DataClient
from pdmt5 import Mt5Config, Mt5DataClient, Mt5RuntimeError, Mt5TradingError
from .history import (
create_cash_events_view,
create_history_indexes,
create_positions_reconstructed_view,
drop_forming_rate_bar,
resolve_granularity_name,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
write_collected_datasets,
write_incremental_datasets,
)
from .retry import retry_with_backoff
from .utils import (
Dataset,
IfExists,
@@ -31,37 +37,111 @@ from .utils import (
parse_tick_flags,
parse_timeframe,
)
from .utils import coerce_login as _coerce_login
if TYPE_CHECKING:
from collections.abc import Callable, Iterator, Sequence
UpdateHistoryBackend = Callable[..., None]
T = TypeVar("T")
logger = logging.getLogger(__name__)
_RECOVERABLE_HISTORY_UPDATE_ERRORS: tuple[type[BaseException], ...] = (
Mt5TradingError,
Mt5RuntimeError,
sqlite3.Error,
ValueError,
OSError,
)
_MT5_CLIENT_CAPABILITY_METHODS: frozenset[str] = frozenset({
"copy_rates_range_as_df",
"copy_ticks_range_as_df",
"history_deals_get_as_df",
"history_orders_get_as_df",
})
_MT5_HISTORY_MODULE = Path(__file__).with_name("history.py").resolve()
_MT5_HISTORY_CLIENT_CALL_FUNCTIONS: frozenset[str] = frozenset({
"write_rates_dataset",
"write_ticks_dataset",
"write_history_dataset",
"_write_incremental_history_deals",
"_fetch_rates_frame",
"_fetch_ticks_frame",
"_fetch_history_frame",
})
_NON_CALLABLE_TYPE_ERROR = re.compile(r"^'[^']+' object is not callable$")
def _is_non_callable_history_client_type_error(exc: TypeError) -> bool:
"""Return whether a TypeError came from calling a history client API attribute."""
if not _NON_CALLABLE_TYPE_ERROR.match(str(exc)):
return False
tb = exc.__traceback__
if tb is None:
return False
while tb.tb_next is not None:
tb = tb.tb_next
frame = tb.tb_frame
return (
frame.f_code.co_name in _MT5_HISTORY_CLIENT_CALL_FUNCTIONS
and Path(frame.f_code.co_filename).resolve() == _MT5_HISTORY_MODULE
)
def _is_mt5_client_capability_error(exc: BaseException) -> bool:
"""Return whether an error indicates an incompatible MT5 client API surface."""
if isinstance(exc, AttributeError):
msg = str(exc)
if msg.startswith("MT5 client is missing required method:"):
return True
name = getattr(exc, "name", None)
return isinstance(name, str) and name in _MT5_CLIENT_CAPABILITY_METHODS
if isinstance(exc, TypeError):
msg = str(exc)
if msg.startswith("MT5 client attribute is not callable:"):
return True
return _is_non_callable_history_client_type_error(exc)
return False
__all__ = [
"AccountSpec",
"Mt5CliClient",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"connected_client",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
"copy_ticks_from",
"copy_ticks_range",
"fetch_latest_closed_rates",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"orders",
"positions",
"recent_history_deals",
"recent_ticks",
"resolve_account_spec",
"resolve_account_specs",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
@@ -73,14 +153,10 @@ __all__ = [
def _coerce_timeframe(timeframe: int | str) -> int:
if isinstance(timeframe, int):
return timeframe
return parse_timeframe(timeframe)
def _coerce_tick_flags(flags: int | str) -> int:
if isinstance(flags, int):
return flags
return parse_tick_flags(flags)
@@ -118,6 +194,12 @@ def _require_positive(value: float, name: str) -> None:
raise ValueError(msg)
def _require_non_negative(value: int, name: str) -> None:
if value < 0:
msg = f"{name} must be non-negative."
raise ValueError(msg)
def _call_required_client_method(client: Mt5DataClient, name: str) -> object:
try:
method = getattr(client, name)
@@ -227,12 +309,33 @@ def build_config(
password: str | None = None,
server: str | None = None,
timeout: int | None = None,
allow_whole_dollar_env: bool = False,
) -> Mt5Config:
"""Build an ``Mt5Config`` from optional connection parameters.
Args:
path: Optional terminal executable path.
login: Optional trading account login.
password: Optional trading account password.
server: Optional trading server name.
timeout: Optional connection timeout in milliseconds.
allow_whole_dollar_env: When ``True``, string parameters that are
exactly ``$ENV_NAME`` are expanded from the environment. Applies
to ``path``, ``password``, and ``server``. Default ``False``
preserves existing behavior.
Returns:
Configured ``Mt5Config`` instance.
"""
if allow_whole_dollar_env:
if path is not None:
path = substitute_env_placeholders(path, allow_whole_dollar_env=True)
if password is not None:
password = substitute_env_placeholders(
password, allow_whole_dollar_env=True
)
if server is not None:
server = substitute_env_placeholders(server, allow_whole_dollar_env=True)
return Mt5Config(
path=path,
login=login,
@@ -243,7 +346,7 @@ def build_config(
@contextmanager
def _connected_client(config: Mt5Config) -> Iterator[Mt5DataClient]:
def connected_client(config: Mt5Config) -> Iterator[Mt5DataClient]:
"""Initialize MT5, yield a connected client, and always shut down.
Args:
@@ -273,10 +376,30 @@ def _run_with_client(
Returns:
Value returned by ``fetch_fn``.
"""
with _connected_client(config) as client:
with connected_client(config) as client:
return fetch_fn(client)
@contextmanager
def mt5_session(config: Mt5Config | None = None) -> Iterator[Mt5CliClient]:
"""Open an MT5 terminal session and yield a connected client.
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` (when set),
logs in, yields a connected :class:`Mt5CliClient`, and always shuts the
terminal down on exit.
Args:
config: MT5 connection configuration. Defaults to an empty config that
attaches to a running terminal.
Yields:
Connected ``Mt5CliClient`` bound to the session.
"""
mt5_config = config or build_config()
with connected_client(mt5_config) as client:
yield Mt5CliClient.from_connected_client(client)
class Mt5CliClient:
"""Programmatic client for read-only MetaTrader 5 data access."""
@@ -288,6 +411,7 @@ class Mt5CliClient:
password: str | None = None,
server: str | None = None,
timeout: int | None = None,
retry_count: int = 3,
config: Mt5Config | None = None,
client: Mt5DataClient | None = None,
) -> None:
@@ -299,6 +423,8 @@ class Mt5CliClient:
password: Trading account password.
server: Trading server name.
timeout: Connection timeout in milliseconds.
retry_count: Number of MT5 initialization retries for sessions
opened by this client.
config: Optional pre-built ``Mt5Config`` (overrides other args).
client: Optional already-connected ``Mt5DataClient``. Injected
clients are reused as-is and are not initialized or shut down.
@@ -310,6 +436,7 @@ class Mt5CliClient:
server=server,
timeout=timeout,
)
self._retry_count = retry_count
self._client = client
self._owns_client = client is None
@@ -338,7 +465,7 @@ class Mt5CliClient:
"""
if self._client is not None:
return self
client = Mt5DataClient(config=self._config)
client = Mt5DataClient(config=self._config, retry_count=self._retry_count)
try:
client.initialize_and_login_mt5()
except Exception:
@@ -920,7 +1047,7 @@ def update_history_with_config( # noqa: PLR0913
if request is None:
return
mt5_config = config or build_config()
with _connected_client(mt5_config) as client:
with connected_client(mt5_config) as client:
update_history(
client=client,
output=output,
@@ -937,6 +1064,151 @@ def update_history_with_config( # noqa: PLR0913
)
class ThrottledHistoryUpdater:
"""Throttled incremental SQLite history updater for long-running apps.
Wraps :func:`update_history` (or a custom ``update_backend``) with a minimum
interval between successful updates, so a tight application loop can call
:meth:`update` every iteration without re-fetching MT5 history more often
than desired. Timing uses a monotonic clock, so it is unaffected by
wall-clock changes.
Downstream applications may pass ``update_backend`` to substitute the
default :func:`update_history` implementationfor example to add
application-specific logging, metrics, or test doubleswithout monkey-
patching ``mt5cli.sdk.update_history``.
"""
def __init__(
self,
*,
output: Path | str,
datasets: set[Dataset] | None = None,
timeframes: Sequence[int | str] | None = None,
flags: int | str = "ALL",
lookback_hours: float = 24.0,
with_views: bool = False,
include_account_events: bool = True,
interval_seconds: float = 0.0,
suppress_errors: bool = False,
update_backend: UpdateHistoryBackend | None = None,
) -> None:
"""Initialize the throttled updater.
Args:
output: SQLite database path.
datasets: Datasets to include (defaults to all).
timeframes: Rate timeframes to update (defaults to all fixed MT5
timeframes).
flags: Tick copy flags as integer or name (e.g. ``ALL``).
lookback_hours: First-run lookback when a table has no prior rows.
with_views: Create ``cash_events`` and ``positions_reconstructed``
views.
include_account_events: Include account-level cash events.
interval_seconds: Minimum seconds between successful updates. Values
``<= 0`` update on every call.
suppress_errors: When True, recoverable errors (``Mt5TradingError``,
``Mt5RuntimeError``, ``sqlite3.Error``, ``ValueError``,
``OSError``, and MT5 client capability ``AttributeError`` /
``TypeError`` for history API methods) raised during an update
are swallowed and :meth:`update` returns False without advancing
the throttle. Other ``AttributeError`` / ``TypeError`` values
always propagate. When False (default), recoverable errors
propagate so callers control logging.
update_backend: Callable invoked instead of :func:`update_history`
when :meth:`update` runs. Receives the same keyword arguments as
:func:`update_history` (``client``, ``output``, ``symbols``,
``datasets``, ``timeframes``, ``flags``, ``lookback_hours``,
``with_views``, ``include_account_events``). Defaults to
:func:`update_history`.
"""
self.output = output
self.datasets = datasets
self.timeframes = timeframes
self.flags = flags
self.lookback_hours = lookback_hours
self.with_views = with_views
self.include_account_events = include_account_events
self.interval_seconds = interval_seconds
self.suppress_errors = suppress_errors
self.update_backend = (
update_history if update_backend is None else update_backend
)
self._last_update_monotonic: float | None = None
@property
def last_update_monotonic(self) -> float | None:
"""Return the monotonic timestamp of the last successful update."""
return self._last_update_monotonic
def should_update(self) -> bool:
"""Return whether enough time has elapsed to run another update.
Returns:
True when ``interval_seconds <= 0``, when no update has succeeded
yet, or when at least ``interval_seconds`` have elapsed since the
last successful update.
"""
if self.interval_seconds <= 0 or self._last_update_monotonic is None:
return True
return (time.monotonic() - self._last_update_monotonic) >= self.interval_seconds
def update(self, client: Mt5DataClient, symbols: Sequence[str]) -> bool:
"""Run a throttled incremental history update.
Args:
client: Connected MT5 data client.
symbols: Symbols to update.
Returns:
True if an update ran successfully, False if it was throttled or
(when ``suppress_errors`` is True) failed with a recoverable error.
When ``suppress_errors`` is False, recoverable update failures
propagate to the caller.
Raises:
AttributeError: MT5 client capability mismatch when
``suppress_errors`` is False, or any other attribute error.
TypeError: MT5 client capability mismatch when ``suppress_errors``
is False, or any other type error.
"""
if not self.should_update():
return False
try:
_resolve_update_history_request(
output=self.output,
symbols=symbols,
datasets=self.datasets,
timeframes=self.timeframes,
flags=self.flags,
lookback_hours=self.lookback_hours,
date_to=None,
)
self.update_backend(
client=client,
output=self.output,
symbols=symbols,
datasets=self.datasets,
timeframes=self.timeframes,
flags=self.flags,
lookback_hours=self.lookback_hours,
with_views=self.with_views,
include_account_events=self.include_account_events,
)
except _RECOVERABLE_HISTORY_UPDATE_ERRORS:
if self.suppress_errors:
logger.warning("Suppressed history update error", exc_info=True)
return False
raise
except (AttributeError, TypeError) as exc:
if self.suppress_errors and _is_mt5_client_capability_error(exc):
logger.warning("Suppressed history update error", exc_info=True)
return False
raise
self._last_update_monotonic = time.monotonic()
return True
def collect_history(
output: Path,
symbols: list[str],
@@ -945,7 +1217,7 @@ def collect_history(
*,
datasets: set[Dataset] | None = None,
timeframe: int | str = 1,
flags: int | str = 1,
flags: int | str = "ALL",
if_exists: IfExists = IfExists.FAIL,
with_views: bool = False,
config: Mt5Config | None = None,
@@ -970,7 +1242,7 @@ def collect_history(
tf = _coerce_timeframe(timeframe)
tick_flags = _coerce_tick_flags(flags)
mt5_config = config or build_config()
with _connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
with connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
written_tables, written_columns = write_collected_datasets(
@@ -1058,6 +1330,33 @@ def latest_rates(
)
def fetch_latest_closed_rates(
client: Mt5CliClient,
*,
symbol: str,
granularity: str,
count: int,
) -> pd.DataFrame:
"""Fetch up to ``count`` most recent closed bars, oldest to newest.
Returns:
Closed rate bars ordered oldest to newest.
Raises:
ValueError: If ``count`` is not positive or no closed bars are returned.
"""
_require_positive(count, "count")
frame = client.latest_rates(symbol, granularity, count + 1, start_pos=0)
closed = drop_forming_rate_bar(frame)
if closed.empty:
msg = (
f"Rate data is empty for {symbol!r} at granularity {granularity!r} "
f"with count {count}."
)
raise ValueError(msg)
return closed.tail(count).reset_index(drop=True)
def collect_latest_rates(
symbols: Sequence[str],
timeframes: Sequence[int | str],
@@ -1075,6 +1374,457 @@ def collect_latest_rates(
)
@dataclass(frozen=True)
class AccountSpec:
"""Connection parameters and symbols for one MT5 account group.
Attributes:
symbols: Symbols to load latest rates for under this account.
login: Trading account login. String values are coerced to int when
non-empty.
password: Trading account password.
server: Trading server name.
path: Path to the MetaTrader5 terminal EXE file.
timeout: Connection timeout in milliseconds.
"""
symbols: Sequence[str]
login: int | str | None = field(default=None, repr=False)
password: str | None = field(default=None, repr=False)
server: str | None = None
path: str | None = None
timeout: int | None = None
_ENV_PLACEHOLDER_PATTERN = re.compile(r"\$\{(?P<name>[A-Za-z_][A-Za-z0-9_]*)\}")
_WHOLE_DOLLAR_PATTERN = re.compile(r"^\$(?P<name>[A-Za-z_][A-Za-z0-9_]*)$")
def substitute_env_placeholders(
value: str,
*,
allow_whole_dollar_env: bool = False,
) -> str:
"""Replace ``${ENV_VAR}`` placeholders in a string with environment values.
Args:
value: String that may contain one or more ``${ENV_VAR}`` placeholders.
allow_whole_dollar_env: When ``True``, a string that is exactly
``$ENV_NAME`` (the whole value and nothing else) is also expanded
from the environment. Partial occurrences such as ``"plan$pass"``
or ``"$ENV-suffix"`` are left unchanged.
Returns:
The string with every placeholder replaced by its environment value.
Raises:
ValueError: If a referenced environment variable is not set.
"""
if allow_whole_dollar_env:
m = _WHOLE_DOLLAR_PATTERN.match(value)
if m:
name = m.group("name")
if name not in os.environ:
msg = f"Environment variable {name!r} is not set."
raise ValueError(msg)
return os.environ[name]
parts: list[str] = []
last_end = 0
for match in _ENV_PLACEHOLDER_PATTERN.finditer(value):
parts.append(value[last_end : match.start()])
name = match.group("name")
if name not in os.environ:
msg = f"Environment variable {name!r} is not set."
raise ValueError(msg)
parts.append(os.environ[name])
last_end = match.end()
parts.append(value[last_end:])
return "".join(parts)
def _resolve_field(
override: str | None,
account_value: str | None,
*,
allow_whole_dollar_env: bool = False,
) -> str | None:
"""Resolve a string field from an override or account value with env subst.
Returns:
The explicit override when provided, otherwise the account value, with
any ``${ENV_VAR}`` placeholders substituted.
"""
value = override if override is not None else account_value
if value is None:
return None
return substitute_env_placeholders(
value, allow_whole_dollar_env=allow_whole_dollar_env
)
def _resolve_login(
override: int | str | None,
account_login: int | str | None,
*,
allow_whole_dollar_env: bool = False,
) -> int | str | None:
"""Resolve a login from an override or account value with env substitution.
Returns:
The explicit override when provided, otherwise the account login.
Integer values are preserved; string values have ``${ENV_VAR}``
placeholders substituted.
"""
if override is not None:
if isinstance(override, int):
return override
return substitute_env_placeholders(
override, allow_whole_dollar_env=allow_whole_dollar_env
)
if account_login is None or isinstance(account_login, int):
return account_login
return substitute_env_placeholders(
account_login, allow_whole_dollar_env=allow_whole_dollar_env
)
def resolve_account_spec(
account: AccountSpec,
*,
login: int | str | None = None,
password: str | None = None,
server: str | None = None,
path: str | None = None,
timeout: int | None = None,
allow_whole_dollar_env: bool = False,
) -> AccountSpec:
"""Resolve an account's credentials from overrides and ``${ENV_VAR}`` values.
Explicit override arguments take precedence over the corresponding
:class:`AccountSpec` fields. The resolved string fields (``login``,
``password``, ``server``, ``path``) have any ``${ENV_VAR}`` placeholders
substituted from the environment.
Args:
account: Source account specification.
login: Optional explicit login override.
password: Optional explicit password override.
server: Optional explicit server override.
path: Optional explicit terminal path override.
timeout: Optional explicit connection timeout override.
allow_whole_dollar_env: When ``True``, string fields that are exactly
``$ENV_NAME`` are also expanded from the environment. Default
``False`` preserves existing behavior.
Returns:
A new :class:`AccountSpec` with resolved credentials and the original
symbols preserved. Raises ``ValueError`` (via
:func:`substitute_env_placeholders`) if a referenced environment
variable is not set.
"""
return AccountSpec(
symbols=account.symbols,
login=_resolve_login(
login, account.login, allow_whole_dollar_env=allow_whole_dollar_env
),
password=_resolve_field(
password, account.password, allow_whole_dollar_env=allow_whole_dollar_env
),
server=_resolve_field(
server, account.server, allow_whole_dollar_env=allow_whole_dollar_env
),
path=_resolve_field(
path, account.path, allow_whole_dollar_env=allow_whole_dollar_env
),
timeout=timeout if timeout is not None else account.timeout,
)
def resolve_account_specs(
accounts: Sequence[AccountSpec],
*,
login: int | str | None = None,
password: str | None = None,
server: str | None = None,
path: str | None = None,
timeout: int | None = None,
allow_whole_dollar_env: bool = False,
) -> list[AccountSpec]:
"""Resolve credentials for multiple accounts.
Applies the same overrides and ``${ENV_VAR}`` substitution as
:func:`resolve_account_spec` to every account.
Args:
accounts: Source account specifications.
login: Optional explicit login override applied to each account.
password: Optional explicit password override applied to each account.
server: Optional explicit server override applied to each account.
path: Optional explicit terminal path override applied to each account.
timeout: Optional explicit timeout override applied to each account.
allow_whole_dollar_env: When ``True``, string fields that are exactly
``$ENV_NAME`` are also expanded from the environment. Default
``False`` preserves existing behavior.
Returns:
Resolved account specifications in the original order. Raises
``ValueError`` (via :func:`substitute_env_placeholders`) if a referenced
environment variable is not set.
"""
return [
resolve_account_spec(
account,
login=login,
password=password,
server=server,
path=path,
timeout=timeout,
allow_whole_dollar_env=allow_whole_dollar_env,
)
for account in accounts
]
def _build_account_config(
account: AccountSpec,
base_config: Mt5Config | None,
) -> Mt5Config:
"""Build an ``Mt5Config`` for an account, falling back to ``base_config``.
Returns:
Merged MT5 configuration for the account.
"""
login = _coerce_login(account.login)
if login is None and base_config is not None:
login = base_config.login
return build_config(
path=account.path or (base_config.path if base_config else None),
login=login,
password=account.password or (base_config.password if base_config else None),
server=account.server or (base_config.server if base_config else None),
timeout=account.timeout
if account.timeout is not None
else (base_config.timeout if base_config else None),
)
def collect_latest_rates_for_accounts(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest rates across multiple MT5 account groups.
Each account is connected in turn, its symbols are read for every
timeframe, and the resulting frames are merged into a single mapping.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent bars to read per symbol/timeframe.
start_pos: Initial bar position offset.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``. When accounts share a
symbol/timeframe pair, the last account processed wins.
Raises:
ValueError: If ``accounts``, ``timeframes``, or any account's symbols are
empty, or ``count`` is not positive.
"""
account_list = list(accounts)
if not account_list:
msg = "At least one account is required."
raise ValueError(msg)
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if any(not account.symbols for account in account_list):
msg = "Each account requires at least one symbol."
raise ValueError(msg)
_require_positive(count, "count")
result: dict[tuple[str, int], pd.DataFrame] = {}
for account in account_list:
config = _build_account_config(account, base_config)
with Mt5CliClient(config=config) as client:
result.update(
client.collect_latest_rates(
account.symbols,
timeframes,
count=count,
start_pos=start_pos,
),
)
return result
def collect_latest_rates_for_accounts_with_retries(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest rates across accounts, retrying transient MT5 failures.
Wraps :func:`collect_latest_rates_for_accounts` with bounded exponential
backoff. Only ``pdmt5.Mt5TradingError`` and ``pdmt5.Mt5RuntimeError`` are
retried; other exceptions propagate immediately. The final failure is
re-raised once retries are exhausted.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent bars to read per symbol/timeframe.
start_pos: Initial bar position offset.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff. The delay before retry
attempt ``n`` (1-indexed) is ``backoff_base ** n`` seconds.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``. Propagates ``ValueError``
for invalid inputs (see :func:`collect_latest_rates_for_accounts`) and
re-raises the last ``pdmt5.Mt5TradingError`` or ``pdmt5.Mt5RuntimeError``
once retries are exhausted.
"""
def _collect() -> dict[tuple[str, int], pd.DataFrame]:
return collect_latest_rates_for_accounts(
accounts,
timeframes,
count,
start_pos=start_pos,
base_config=base_config,
)
return retry_with_backoff(
_collect,
retry_count=retry_count,
backoff_base=backoff_base,
operation="Rate collection",
)
def collect_latest_closed_rates_for_accounts(
accounts: Sequence[AccountSpec],
timeframes: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Collect latest closed rate bars across multiple MT5 account groups.
When ``start_pos`` is ``0`` (the default), MetaTrader 5 includes the
still-forming current bar as the last row. This helper fetches
``count + 1`` bars, drops that bar with :func:`drop_forming_rate_bar`, and
validates that each resulting frame is non-empty. When ``start_pos`` is
greater than zero the forming bar is not in range, so only ``count`` bars
are fetched and no row is dropped.
Wraps :func:`collect_latest_rates_for_accounts_with_retries` for transient
MT5 error handling.
Args:
accounts: Account groups to read. Each must define at least one symbol.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
count: Number of closed bars to return per symbol/timeframe.
start_pos: Initial bar position offset passed to the underlying collector.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff between retry attempts.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``.
Raises:
ValueError: If inputs are invalid, or any series is empty (after
dropping the still-forming bar when ``start_pos`` is ``0``).
"""
_require_positive(count, "count")
_require_non_negative(start_pos, "start_pos")
fetch_count = count + 1 if start_pos == 0 else count
loaded = collect_latest_rates_for_accounts_with_retries(
accounts,
timeframes,
fetch_count,
start_pos=start_pos,
base_config=base_config,
retry_count=retry_count,
backoff_base=backoff_base,
)
result: dict[tuple[str, int], pd.DataFrame] = {}
for key, df_rate in loaded.items():
closed = drop_forming_rate_bar(df_rate) if start_pos == 0 else df_rate
if closed.empty:
symbol, timeframe = key
msg = f"Rate data is empty for {symbol!r} at timeframe {timeframe}."
raise ValueError(msg)
result[key] = closed
return result
def collect_latest_closed_rates_by_granularity(
accounts: Sequence[AccountSpec],
granularities: Sequence[int | str],
count: int,
*,
start_pos: int = 0,
base_config: Mt5Config | None = None,
retry_count: int = 0,
backoff_base: float = 2.0,
) -> dict[tuple[str, str], pd.DataFrame]:
"""Collect latest closed rate bars keyed by symbol and granularity name.
Thin wrapper around :func:`collect_latest_closed_rates_for_accounts` that
rekeys the result by granularity name (for example ``M1``) instead of the
integer timeframe.
Args:
accounts: Account groups to read. Each must define at least one symbol.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of closed bars to return per symbol/timeframe.
start_pos: Initial bar position offset passed to the underlying collector.
base_config: Optional base configuration whose fields fill any value not
set on an individual account.
retry_count: Maximum number of retries after the first attempt. ``0``
disables retries.
backoff_base: Base for exponential backoff between retry attempts.
Returns:
Mapping keyed by ``(symbol, granularity_name)``. Propagates
``ValueError`` from :func:`collect_latest_closed_rates_for_accounts`.
"""
loaded = collect_latest_closed_rates_for_accounts(
accounts,
granularities,
count,
start_pos=start_pos,
base_config=base_config,
retry_count=retry_count,
backoff_base=backoff_base,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in loaded.items()
}
def copy_rates_range(
symbol: str,
timeframe: int | str,
+49
View File
@@ -0,0 +1,49 @@
"""Generic storage helpers for MT5 market and account history."""
from __future__ import annotations
from .history import (
RateTarget,
build_rate_targets,
build_rate_view_name,
drop_forming_rate_bar,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
)
from .sdk import collect_history, update_history, update_history_with_config
from .utils import (
Dataset,
IfExists,
OutputFormat,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
)
__all__ = [
"Dataset",
"IfExists",
"OutputFormat",
"RateTarget",
"build_rate_targets",
"build_rate_view_name",
"collect_history",
"detect_format",
"drop_forming_rate_bar",
"export_dataframe",
"export_dataframe_to_sqlite",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"update_history",
"update_history_with_config",
]
+1696
View File
File diff suppressed because it is too large Load Diff
+45 -50
View File
@@ -10,6 +10,9 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard
import click
from pdmt5 import COPY_TICKS_MAP, TIMEFRAME_MAP
from pdmt5 import parse_copy_ticks as _parse_copy_ticks
from pdmt5 import parse_timeframe as _parse_timeframe
if TYPE_CHECKING:
from collections.abc import Sequence
@@ -20,35 +23,15 @@ if TYPE_CHECKING:
# Constants
# ---------------------------------------------------------------------------
TIMEFRAME_MAP: dict[str, int] = {
"M1": 1,
"M2": 2,
"M3": 3,
"M4": 4,
"M5": 5,
"M6": 6,
"M10": 10,
"M12": 12,
"M15": 15,
"M20": 20,
"M30": 30,
"H1": 16385,
"H2": 16386,
"H3": 16387,
"H4": 16388,
"H6": 16390,
"H8": 16392,
"H12": 16396,
"D1": 16408,
"W1": 32769,
"MN1": 49153,
}
# Backward-compatible snapshot; prefer ``COPY_TICKS_MAP`` from pdmt5 directly.
TICK_FLAG_MAP: dict[str, int] = dict(COPY_TICKS_MAP)
TICK_FLAG_MAP: dict[str, int] = {
"ALL": 1,
"INFO": 2,
"TRADE": 4,
}
TIMEFRAME_NAMES: tuple[str, ...] = tuple(
name for name in TIMEFRAME_MAP if not name.startswith("TIMEFRAME_")
)
_TICK_FLAG_NAMES: tuple[str, ...] = tuple(
name for name in COPY_TICKS_MAP if not name.startswith("COPY_TICKS_")
)
_FORMAT_EXTENSIONS: dict[str, str] = {
".csv": "csv",
@@ -160,10 +143,8 @@ class _TimeframeType(click.ParamType):
Returns:
Integer timeframe value.
"""
if isinstance(value, int):
return value
try:
return parse_timeframe(str(value))
return parse_timeframe(value)
except ValueError as exc:
self.fail(str(exc), param, ctx)
@@ -189,10 +170,8 @@ class _TickFlagsType(click.ParamType):
Returns:
Integer tick flag value.
"""
if isinstance(value, int):
return value
try:
return parse_tick_flags(str(value))
return parse_tick_flags(value)
except ValueError as exc:
self.fail(str(exc), param, ctx)
@@ -262,6 +241,20 @@ def detect_format(
raise ValueError(msg)
def coerce_login(login: int | str | None) -> int | None:
"""Coerce a login value to int, treating empty strings as unset.
Returns:
Integer login, or None when unset or an empty string.
"""
if login is None or isinstance(login, int):
return login
text = login.strip()
if not text:
return None
return int(text)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
@@ -370,7 +363,7 @@ def parse_datetime(value: str) -> datetime:
return dt
def parse_timeframe(value: str) -> int:
def parse_timeframe(value: object) -> int:
"""Parse a timeframe string or integer value.
Args:
@@ -382,37 +375,39 @@ def parse_timeframe(value: str) -> int:
Raises:
ValueError: If the timeframe is invalid.
"""
upper = value.upper()
if upper in TIMEFRAME_MAP:
return TIMEFRAME_MAP[upper]
try:
return int(value)
return _parse_timeframe(value)
except ValueError:
valid = ", ".join(TIMEFRAME_MAP)
msg = f"Invalid timeframe: '{value}'. Use one of: {valid}, or an integer."
display = value if isinstance(value, str) else repr(value)
valid = ", ".join(TIMEFRAME_NAMES)
msg = (
f"Invalid timeframe: '{display}'. "
f"Use one of: {valid}, or a supported integer."
)
raise ValueError(msg) from None
def parse_tick_flags(value: str) -> int:
def parse_tick_flags(value: object) -> int:
"""Parse tick flags string or integer value.
Args:
value: Tick flag name (ALL, INFO, TRADE) or integer value.
value: Tick flag name (ALL, INFO, TRADE, COPY_TICKS_*) or integer value.
Returns:
Integer tick flag value.
Integer tick flag value compatible with MetaTrader 5 ``COPY_TICKS_*``.
Raises:
ValueError: If the flag is invalid.
"""
upper = value.upper()
if upper in TICK_FLAG_MAP:
return TICK_FLAG_MAP[upper]
try:
return int(value)
return _parse_copy_ticks(value)
except ValueError:
valid = ", ".join(TICK_FLAG_MAP)
msg = f"Invalid tick flags: '{value}'. Use one of: {valid}, or an integer."
display = value if isinstance(value, str) else repr(value)
valid = ", ".join(_TICK_FLAG_NAMES)
msg = (
f"Invalid tick flags: '{display}'. "
f"Use one of: {valid}, or a supported integer."
)
raise ValueError(msg) from None
+3 -4
View File
@@ -1,7 +1,7 @@
[project]
name = "mt5cli"
version = "0.5.0"
description = "Command-line tool for MetaTrader 5"
version = "0.9.4"
description = "Generic MT5 data and execution infrastructure for Python applications"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
license = "MIT"
@@ -9,7 +9,7 @@ license-files = ["LICENSE"]
readme = "README.md"
requires-python = ">= 3.11, < 3.14"
dependencies = [
"pdmt5 >= 0.2.3",
"pdmt5>=0.3.0",
"click >= 8.1.0",
"pyarrow >= 19.0.0",
"typer >= 0.15.0",
@@ -124,7 +124,6 @@ ignore = [
]
[tool.ruff.lint.per-file-ignores]
"mt5cli/history.py" = ["TC003"]
"tests/**/*.py" = [
"DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring
+52
View File
@@ -0,0 +1,52 @@
"""Shared pytest fixtures for mt5cli tests."""
from __future__ import annotations
from unittest.mock import MagicMock
import pandas as pd
import pytest
from pytest_mock import MockerFixture # noqa: TC002
_DATAFRAME_METHODS = (
"copy_rates_from_as_df",
"copy_rates_from_pos_as_df",
"copy_rates_range_as_df",
"copy_ticks_from_as_df",
"copy_ticks_range_as_df",
"account_info_as_df",
"terminal_info_as_df",
"symbols_get_as_df",
"symbol_info_as_df",
"orders_get_as_df",
"positions_get_as_df",
"history_orders_get_as_df",
"history_deals_get_as_df",
"version_as_df",
"last_error_as_df",
"symbol_info_tick_as_df",
"market_book_get_as_df",
"order_check_as_df",
"order_send_as_df",
)
def build_mock_mt5_data_client() -> MagicMock:
"""Return a MagicMock Mt5DataClient with common DataFrame stubs."""
client = MagicMock()
sample_df = pd.DataFrame({"col": [1]})
for method_name in _DATAFRAME_METHODS:
getattr(client, method_name).return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
return client
@pytest.fixture
def mock_client(mocker: MockerFixture) -> MagicMock:
"""Create and patch a mock Mt5DataClient for CLI and SDK tests."""
client = build_mock_mt5_data_client()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
+5 -37
View File
@@ -69,38 +69,6 @@ class TestExecuteExport:
# ---------------------------------------------------------------------------
@pytest.fixture
def mock_client(mocker: MockerFixture) -> MagicMock:
"""Create and patch a mock Mt5DataClient for CLI tests."""
client = MagicMock()
sample_df = pd.DataFrame({"col": [1]})
client.copy_rates_from_as_df.return_value = sample_df
client.copy_rates_from_pos_as_df.return_value = sample_df
client.copy_rates_range_as_df.return_value = sample_df
client.copy_ticks_from_as_df.return_value = sample_df
client.copy_ticks_range_as_df.return_value = sample_df
client.account_info_as_df.return_value = sample_df
client.terminal_info_as_df.return_value = sample_df
client.symbols_get_as_df.return_value = sample_df
client.symbol_info_as_df.return_value = sample_df
client.orders_get_as_df.return_value = sample_df
client.positions_get_as_df.return_value = sample_df
client.history_orders_get_as_df.return_value = sample_df
client.history_deals_get_as_df.return_value = sample_df
client.version_as_df.return_value = sample_df
client.last_error_as_df.return_value = sample_df
client.symbol_info_tick_as_df.return_value = sample_df
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
class TestCommands:
"""Tests for all CLI subcommands via CliRunner."""
@@ -317,7 +285,7 @@ class TestCommands:
symbol="EURUSD",
date_from=datetime(2024, 1, 1, tzinfo=UTC),
count=100,
flags=1,
flags=-1,
)
def test_ticks_range(
@@ -348,7 +316,7 @@ class TestCommands:
symbol="EURUSD",
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
flags=2,
flags=1,
)
def test_ticks_recent(
@@ -381,7 +349,7 @@ class TestCommands:
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
flags=-1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
@@ -1000,7 +968,7 @@ class TestCollectHistory:
symbol="EURUSD",
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
flags=1,
flags=-1,
)
with sqlite3.connect(output) as conn:
tables = {
@@ -1213,7 +1181,7 @@ class TestCollectHistory:
symbol="EURUSD",
date_from=datetime(2024, 1, 1, tzinfo=UTC),
date_to=datetime(2024, 2, 1, tzinfo=UTC),
flags=1,
flags=-1,
)
def test_collect_history_with_views(
+840
View File
@@ -0,0 +1,840 @@
"""Contract tests for the mt5cli public API and dataset schemas."""
from __future__ import annotations
import re
import sqlite3
from datetime import UTC, datetime
from pathlib import Path
from typing import get_type_hints
from unittest.mock import MagicMock
import pandas as pd
import pytest
from pdmt5 import Mt5RuntimeError, Mt5TradingError
from pytest_mock import MockerFixture # noqa: TC002
import mt5cli
from mt5cli import (
DEDUP_KEYS,
PUBLIC_EXPORT_TIERS,
REQUIRED_COLUMNS,
SECONDARY_PUBLIC_EXPORTS,
STABLE_SDK_EXPORTS,
TIME_COLUMNS,
AccountSpec,
DataKind,
Dataset,
ExecutionStatus,
MarginVolume,
MT5Client,
Mt5CliError,
Mt5ConnectionError,
Mt5OperationError,
Mt5SchemaError,
OrderExecutionResult,
OrderLimits,
RateTarget,
build_config,
build_rate_targets,
calculate_account_projected_margin_ratio,
calculate_margin_and_volume,
calculate_positions_margin,
calculate_projected_margin_ratio,
calculate_symbol_group_margin_ratio,
calculate_trailing_stop_updates,
call_with_normalized_errors,
detect_format,
drop_forming_rate_bar,
ensure_symbol_selected,
ensure_utc,
export_dataframe,
export_dataframe_to_sqlite,
extract_tick_price,
fetch_latest_closed_rates,
fetch_latest_closed_rates_for_trading_client,
fetch_latest_closed_rates_indexed,
granularity_name,
is_recoverable_mt5_error,
load_rate_data,
load_rate_series_from_sqlite,
mt5_session,
mt5_trading_session,
normalize_dataframe,
normalize_mt5_exception,
normalize_order_volume,
normalize_symbol,
normalize_symbols,
parse_date_range,
place_market_order,
recent_window,
resolve_account_spec,
resolve_account_specs,
resolve_rate_view_name,
schema_columns,
validate_schema,
)
from mt5cli.history import create_rate_compatibility_views
from mt5cli.retry import retry_with_backoff
from mt5cli.schemas import ensure_utc_columns, normalize_time_columns
def _sample_frame(kind: DataKind) -> pd.DataFrame:
if kind is DataKind.rates:
return pd.DataFrame({
"time": [datetime(2024, 1, 1, tzinfo=UTC)],
"open": [1.1],
"high": [1.2],
"low": [1.0],
"close": [1.15],
"tick_volume": [10],
"spread": [1],
"real_volume": [0],
})
if kind is DataKind.ticks:
return pd.DataFrame({
"time": [datetime(2024, 1, 1, tzinfo=UTC)],
"bid": [1.1],
"ask": [1.11],
"last": [1.105],
"volume": [1],
"time_msc": [datetime(2024, 1, 1, tzinfo=UTC)],
"flags": [2],
"volume_real": [0.0],
})
if kind is DataKind.orders:
return pd.DataFrame({
"ticket": [1],
"time_setup": [datetime(2024, 1, 1, tzinfo=UTC)],
"type": [0],
"state": [1],
"symbol": ["EURUSD"],
"volume_current": [0.1],
"price_open": [1.1],
})
if kind is DataKind.positions:
return pd.DataFrame({
"ticket": [1],
"time": [datetime(2024, 1, 1, tzinfo=UTC)],
"type": [0],
"symbol": ["EURUSD"],
"volume": [0.1],
"price_open": [1.1],
"price_current": [1.11],
"profit": [1.0],
})
if kind is DataKind.history_orders:
return pd.DataFrame({
"ticket": [1],
"time_setup": [datetime(2024, 1, 1, tzinfo=UTC)],
"type": [0],
"state": [3],
"symbol": ["EURUSD"],
"volume_initial": [0.1],
"price_open": [1.1],
})
return pd.DataFrame({
"ticket": [1],
"order": [2],
"time": [datetime(2024, 1, 1, tzinfo=UTC)],
"type": [0],
"entry": [0],
"symbol": ["EURUSD"],
"volume": [0.1],
"price": [1.1],
"profit": [0.0],
})
@pytest.mark.parametrize("kind", list(DataKind))
def test_required_columns_contract(kind: DataKind) -> None:
"""Each dataset kind exposes a non-empty required column contract."""
assert REQUIRED_COLUMNS[kind]
validate_schema(_sample_frame(kind), kind)
@pytest.mark.parametrize("kind", list(DataKind))
def test_normalize_dataframe_injects_storage_metadata(kind: DataKind) -> None:
"""Normalization accepts MT5 frames and optional storage metadata."""
frame = _sample_frame(kind)
normalized = normalize_dataframe(
frame,
kind,
symbol="eurusd",
timeframe="M1" if kind is DataKind.rates else None,
)
if kind is DataKind.rates:
assert normalized.loc[0, "symbol"] == "eurusd"
assert normalized.loc[0, "timeframe"] == 1
validate_schema(normalized, kind)
def test_validate_schema_raises_for_missing_columns() -> None:
"""Schema validation fails fast on missing required columns."""
with pytest.raises(Mt5SchemaError, match="missing required columns"):
validate_schema(pd.DataFrame({"time": [1]}), DataKind.rates)
def test_history_dedup_keys_match_schema_contract() -> None:
"""SQLite history dedup keys stay aligned with schema contracts."""
assert DEDUP_KEYS[DataKind.rates][0] == ("symbol", "timeframe", "time")
assert DEDUP_KEYS[DataKind.ticks][0] == ("symbol", "time_msc")
assert Dataset.rates.table_name == "rates"
@pytest.mark.parametrize(
("raw", "expected"),
[
(" eurusd ", "eurusd"),
("GbpJpy", "GbpJpy"),
("XAUUSDm", "XAUUSDm"),
("US500.cash", "US500.cash"),
("EURUSD.r", "EURUSD.r"),
],
)
def test_normalize_symbol(raw: str, expected: str) -> None:
"""Symbol normalization trims whitespace and preserves broker casing."""
assert normalize_symbol(raw) == expected
def test_normalize_symbols_deduplicates() -> None:
"""Symbol lists are normalized and de-duplicated in order."""
assert normalize_symbols(["XAUUSDm", " XAUUSDm ", "EURUSD.r", "eurusd"]) == [
"XAUUSDm",
"EURUSD.r",
"eurusd",
]
def test_parse_date_range_rejects_inverted_bounds() -> None:
"""Date ranges must not be inverted."""
with pytest.raises(ValueError, match="must not be after"):
parse_date_range("2024-02-01", "2024-01-01")
def test_recent_window_builds_trailing_bounds() -> None:
"""Recent windows end at the provided timestamp."""
end = datetime(2024, 1, 2, tzinfo=UTC)
start, resolved_end = recent_window(hours=24, date_to=end)
assert resolved_end == end
assert start < end
def test_granularity_name_maps_timeframe_alias() -> None:
"""Granularity labels resolve MT5 timeframe aliases."""
assert granularity_name("M1") == "M1"
@pytest.mark.parametrize(
"exc",
[Mt5RuntimeError("init failed"), Mt5TradingError("trade failed")],
)
def test_is_recoverable_mt5_error(exc: Exception) -> None:
"""Recoverable MT5 errors are classified consistently."""
assert is_recoverable_mt5_error(exc)
def test_normalize_mt5_exception_maps_types() -> None:
"""MT5 exceptions map to stable mt5cli types."""
assert isinstance(
normalize_mt5_exception(Mt5RuntimeError("x")),
Mt5ConnectionError,
)
assert isinstance(
normalize_mt5_exception(Mt5TradingError("x")),
Mt5OperationError,
)
def test_call_with_normalized_errors_reraises_mapped_type() -> None:
"""Normalized error helper re-raises mapped mt5cli exceptions."""
def _raise() -> None:
message = "boom"
raise Mt5RuntimeError(message)
with pytest.raises(Mt5ConnectionError):
call_with_normalized_errors(_raise)
def test_retry_with_backoff_retries_recoverable_errors(
mocker: MockerFixture,
) -> None:
"""Retry helper retries recoverable MT5 failures."""
calls = {"count": 0}
def _flaky() -> str:
calls["count"] += 1
if calls["count"] == 1:
message = "transient"
raise Mt5RuntimeError(message)
return "ok"
mocker.patch("mt5cli.retry.time.sleep")
assert retry_with_backoff(_flaky, retry_count=1) == "ok"
assert calls["count"] == 2
def test_public_api_exports_mt5_client() -> None:
"""MT5Client is the primary importable client abstraction."""
client = MT5Client(config=build_config())
assert isinstance(client, MT5Client)
assert isinstance(client, MT5Client.__mro__[1])
def test_mt5_client_order_primitives_use_connected_client(
mock_client: object,
) -> None:
"""Order check/send route through the same client fetch path as exports."""
request = {"action": 1}
client = MT5Client()
client.order_check(request)
client.order_send(request)
assert mock_client.order_check_as_df.call_count == 1 # type: ignore[attr-defined]
assert mock_client.order_send_as_df.call_count == 1 # type: ignore[attr-defined]
def test_storage_export_round_trip_csv(tmp_path: Path) -> None:
"""Storage helpers export normalized rate frames to CSV."""
frame = normalize_dataframe(
_sample_frame(DataKind.rates),
DataKind.rates,
symbol="EURUSD",
timeframe="M1",
)
output = tmp_path / "rates.csv"
export_dataframe(frame, output, detect_format(output))
loaded = pd.read_csv(output)
assert len(loaded) == 1
assert "close" in loaded.columns
def test_normalize_symbol_rejects_empty_value() -> None:
"""Empty symbols are rejected after trimming."""
with pytest.raises(ValueError, match="must not be empty"):
normalize_symbol(" ")
def test_ensure_utc_handles_naive_and_aware_datetimes() -> None:
"""UTC coercion accepts naive and timezone-aware datetimes."""
naive = datetime(2024, 1, 1, tzinfo=UTC).replace(tzinfo=None)
aware = datetime(2024, 1, 1, tzinfo=UTC)
assert ensure_utc(naive).tzinfo == UTC
assert ensure_utc(aware).tzinfo == UTC
assert ensure_utc("2024-01-01T00:00:00+00:00").tzinfo == UTC
def test_recent_window_validation_errors() -> None:
"""Recent window helpers validate mutually exclusive length arguments."""
with pytest.raises(ValueError, match="exactly one"):
recent_window()
with pytest.raises(ValueError, match="exactly one"):
recent_window(hours=1, seconds=1)
with pytest.raises(ValueError, match="positive"):
recent_window(hours=0)
def test_recent_window_supports_seconds_argument() -> None:
"""Recent windows can be built from a seconds-based length."""
end = datetime(2024, 1, 2, tzinfo=UTC)
start, resolved_end = recent_window(seconds=3600, date_to=end)
assert resolved_end == end
assert start < end
def test_parse_date_range_returns_ordered_bounds() -> None:
"""Valid date ranges return UTC-aware bounds."""
start, end = parse_date_range("2024-01-01", "2024-02-01")
assert start < end
def test_granularity_name_falls_back_for_unknown_timeframe(
mocker: MockerFixture,
) -> None:
"""Unknown timeframe integers stringify as granularity labels."""
mocker.patch(
"mt5cli.converters._get_timeframe_name",
side_effect=ValueError("unknown"),
)
assert granularity_name(1) == "1"
def test_normalize_mt5_exception_passthrough_and_generic() -> None:
"""Normalization preserves mt5cli errors and wraps unknown exceptions."""
original = Mt5CliError("known")
assert normalize_mt5_exception(original) is original
assert isinstance(normalize_mt5_exception(ValueError("x")), Mt5CliError)
def test_schema_columns_and_extra_required_validation() -> None:
"""Schema helpers expose contracts and honor extra required columns."""
assert schema_columns(DataKind.rates) == REQUIRED_COLUMNS[DataKind.rates]
validate_schema(pd.DataFrame(), DataKind.rates)
frame = _sample_frame(DataKind.rates)
with pytest.raises(Mt5SchemaError, match="storage_symbol"):
validate_schema(frame, DataKind.rates, extra_required=["storage_symbol"])
def test_normalize_dataframe_empty_and_tick_sort_paths() -> None:
"""Normalization handles empty frames and tick time_msc sorting."""
empty = pd.DataFrame()
assert normalize_dataframe(empty, DataKind.rates).empty
ticks = _sample_frame(DataKind.ticks)
ticks = pd.concat([ticks, ticks], ignore_index=True)
sorted_ticks = normalize_dataframe(ticks, DataKind.ticks, sort=True)
assert len(sorted_ticks) == 2
unsorted_ticks = normalize_dataframe(ticks, DataKind.ticks, sort=False)
assert len(unsorted_ticks) == 2
def test_normalize_dataframe_rate_timeframe_without_symbol() -> None:
"""Rate normalization can inject timeframe without symbol metadata."""
frame = _sample_frame(DataKind.rates)
normalized = normalize_dataframe(frame, DataKind.rates, timeframe="M1")
assert "timeframe" in normalized.columns
def test_normalize_dataframe_keeps_existing_symbol_and_timeframe() -> None:
"""Normalization does not duplicate existing storage metadata columns."""
frame = normalize_dataframe(
_sample_frame(DataKind.rates),
DataKind.rates,
symbol="EURUSD",
timeframe="M1",
)
normalized = normalize_dataframe(
frame,
DataKind.rates,
symbol="GBPUSD",
timeframe="H1",
)
assert normalized.loc[0, "symbol"] == "EURUSD"
assert normalized.loc[0, "timeframe"] == 1
def test_normalize_time_columns_skips_absent_time_fields() -> None:
"""Time normalization ignores absent optional time columns."""
frame = pd.DataFrame({"open": [1.0]})
result = normalize_time_columns(frame, DataKind.rates)
assert list(result.columns) == ["open"]
def test_normalize_time_columns_converts_unix_seconds() -> None:
"""Numeric MT5 ``time`` values are interpreted as Unix seconds."""
frame = pd.DataFrame({"time": [1704067200]})
result = normalize_time_columns(frame, DataKind.rates)
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
def test_normalize_time_columns_converts_unix_milliseconds() -> None:
"""Numeric MT5 ``time_msc`` values are interpreted as Unix milliseconds."""
frame = pd.DataFrame({"time_msc": [1704067200000]})
result = normalize_time_columns(frame, DataKind.ticks)
assert result.loc[0, "time_msc"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
def test_normalize_time_columns_preserves_utc_datetimes() -> None:
"""Already-converted datetime values remain UTC-normalized."""
aware = datetime(2024, 1, 1, tzinfo=UTC)
frame = pd.DataFrame({"time": [aware]})
result = normalize_time_columns(frame, DataKind.rates)
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
def test_normalize_time_columns_handles_optional_order_times() -> None:
"""Optional order/history time columns are normalized when present."""
frame = pd.DataFrame({
"time_setup": [1704067200],
"time_setup_msc": [1704067200000],
"time_done": [1704153600],
"time_done_msc": [1704153600000],
})
result = normalize_time_columns(frame, DataKind.orders)
assert result.loc[0, "time_setup"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
assert result.loc[0, "time_setup_msc"] == pd.Timestamp(
"2024-01-01T00:00:00+00:00",
)
assert result.loc[0, "time_done"] == pd.Timestamp("2024-01-02T00:00:00+00:00")
assert result.loc[0, "time_done_msc"] == pd.Timestamp(
"2024-01-02T00:00:00+00:00",
)
def test_time_columns_include_optional_order_fields() -> None:
"""Schema contracts document optional MT5 time columns per dataset kind."""
assert "time_done" in TIME_COLUMNS[DataKind.orders]
assert "time_setup_msc" in TIME_COLUMNS[DataKind.history_orders]
def test_normalize_dataframe_sorts_ticks_by_time_msc(
mocker: MockerFixture,
) -> None:
"""Tick frames without ``time`` can still sort on ``time_msc``."""
mocker.patch("mt5cli.schemas.validate_schema")
ticks = pd.concat([_sample_frame(DataKind.ticks)] * 2, ignore_index=True).drop(
columns=["time"],
)
ticks.loc[0, "time_msc"] = datetime(2024, 1, 1, tzinfo=UTC)
ticks.loc[1, "time_msc"] = datetime(2024, 1, 2, tzinfo=UTC)
ticks = pd.concat([ticks.iloc[[1]], ticks.iloc[[0]]], ignore_index=True)
normalized = normalize_dataframe(ticks, DataKind.ticks, sort=True)
assert normalized.iloc[0]["time_msc"] <= normalized.iloc[1]["time_msc"]
def test_ensure_utc_columns_skips_missing_columns() -> None:
"""UTC column coercion ignores absent columns."""
frame = _sample_frame(DataKind.rates)
result = ensure_utc_columns(frame, ["time", "missing"])
assert "time" in result.columns
def test_normalize_time_columns_coerces_string_timestamps() -> None:
"""String timestamps are parsed with timezone-aware datetime coercion."""
frame = pd.DataFrame({"time": ["2024-01-01T00:00:00+00:00"]})
result = normalize_time_columns(frame, DataKind.rates)
assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
def test_ensure_utc_columns_coerces_non_mt5_columns() -> None:
"""Non-MT5 columns still coerce to UTC datetimes."""
frame = pd.DataFrame({"created_at": ["2024-01-01T00:00:00+00:00"]})
result = ensure_utc_columns(frame, ["created_at"])
assert result.loc[0, "created_at"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
def test_mt5_session_yields_connected_client(mocker: MockerFixture) -> None:
"""Public mt5_session yields an MT5Client bound to a connected session."""
connected = mocker.MagicMock()
context = mocker.MagicMock()
context.__enter__.return_value = connected
context.__exit__.return_value = False
mocker.patch("mt5cli.client.connected_client", return_value=context)
with mt5_session(build_config()) as client:
assert isinstance(client, MT5Client)
def test_retry_with_backoff_reraises_non_recoverable_errors() -> None:
"""Non-MT5 errors are not retried."""
def _raise() -> None:
message = "fatal"
raise ValueError(message)
with pytest.raises(ValueError, match="fatal"):
retry_with_backoff(_raise, retry_count=2)
def test_storage_export_round_trip_sqlite(tmp_path: Path) -> None:
"""Storage helpers append deduplicated frames to SQLite."""
frame = normalize_dataframe(
_sample_frame(DataKind.rates),
DataKind.rates,
symbol="EURUSD",
timeframe="M1",
)
output = tmp_path / "rates.db"
export_dataframe_to_sqlite(
frame,
output,
"rates",
deduplicate_on=DEDUP_KEYS[DataKind.rates][0],
)
with __import__("sqlite3").connect(output) as conn:
count = conn.execute("SELECT COUNT(*) FROM rates").fetchone()[0]
assert count == 1
class TestStableSdkContract:
"""Tests for the documented stable downstream SDK contract."""
def test_stable_exports_are_subset_of_all(self) -> None:
"""Every stable export is also listed in the package __all__."""
missing = sorted(STABLE_SDK_EXPORTS - set(mt5cli.__all__))
assert not missing, f"STABLE_SDK_EXPORTS missing from __all__: {missing}"
def test_public_export_tiers_are_disjoint_and_complete(self) -> None:
"""Documented public tiers do not overlap and classify root exports."""
assert PUBLIC_EXPORT_TIERS == {
"stable": STABLE_SDK_EXPORTS,
"secondary": SECONDARY_PUBLIC_EXPORTS,
}
assert not (STABLE_SDK_EXPORTS & SECONDARY_PUBLIC_EXPORTS)
tiered_exports = STABLE_SDK_EXPORTS | SECONDARY_PUBLIC_EXPORTS
root_exports = set(mt5cli.__all__)
missing_from_root = sorted(tiered_exports - root_exports)
assert not missing_from_root, (
f"Tiered exports missing from __all__: {missing_from_root}"
)
tier_metadata_exports = {
"PUBLIC_EXPORT_TIERS",
"SECONDARY_PUBLIC_EXPORTS",
"STABLE_SDK_EXPORTS",
}
unclassified_root_exports = sorted(
root_exports - tiered_exports - tier_metadata_exports,
)
assert not unclassified_root_exports, (
f"Root exports missing from public API tiers: {unclassified_root_exports}"
)
def test_stable_docs_do_not_document_nonstable_exports(self) -> None:
"""Stable docs do not promote secondary root exports."""
docs_path = Path("docs/api/public-contract.md")
docs = docs_path.read_text(encoding="utf-8")
stable_section = docs.split("## Stable downstream SDK API", maxsplit=1)[
1
].split(
"## Secondary public exports",
maxsplit=1,
)[0]
documented_symbols = set(
re.findall(r"`([A-Za-z_][A-Za-z0-9_]*)`", stable_section)
)
nonstable_exports = SECONDARY_PUBLIC_EXPORTS
wrongly_stable = sorted(documented_symbols & nonstable_exports)
assert not wrongly_stable, (
f"Non-stable exports documented in stable section: {wrongly_stable}"
)
@pytest.mark.parametrize("name", sorted(STABLE_SDK_EXPORTS))
def test_stable_exports_are_importable_from_package_root(self, name: str) -> None:
"""Stable SDK names resolve through ``from mt5cli import ...``."""
assert hasattr(mt5cli, name), f"{name!r} missing from mt5cli package root"
@pytest.mark.parametrize(
"name",
sorted(SECONDARY_PUBLIC_EXPORTS),
)
def test_secondary_exports_are_importable(
self,
name: str,
) -> None:
"""Non-stable public names remain available from the package root."""
assert hasattr(mt5cli, name), f"{name!r} missing from mt5cli package root"
def test_drop_forming_rate_bar_from_package_root(self) -> None:
"""Closed-bar trimming is available from the stable package surface."""
frame = pd.DataFrame({"time": [1, 2, 3], "close": [1.0, 1.1, 1.2]})
closed = drop_forming_rate_bar(frame)
assert list(closed["close"]) == [1.0, 1.1]
assert len(closed) == 2
def test_fetch_latest_closed_rates_from_package_root(self) -> None:
"""Single-client closed-bar helper drops the forming row."""
client = MagicMock()
client.latest_rates.return_value = pd.DataFrame(
{"time": [1, 2, 3], "close": [1.0, 1.1, 1.2]},
)
result = fetch_latest_closed_rates(
client,
symbol="EURUSD",
granularity="M1",
count=2,
)
client.latest_rates.assert_called_once_with("EURUSD", "M1", 3, start_pos=0)
assert list(result["close"]) == [1.0, 1.1]
def test_fetch_latest_closed_rates_for_trading_client_from_package_root(
self,
) -> None:
"""Trading-client closed-bar helper is importable from the stable surface."""
client = MagicMock()
client.fetch_latest_rates_as_df.return_value = pd.DataFrame(
{"time": [1, 2, 3], "close": [1.0, 1.1, 1.2]},
)
result = fetch_latest_closed_rates_for_trading_client(
client,
symbol="EURUSD",
granularity="M1",
count=2,
)
assert list(result["close"]) == [1.0, 1.1]
def test_normalize_order_volume_from_package_root(self) -> None:
"""Volume normalization helper is importable from the stable surface."""
result = normalize_order_volume(
0.25,
volume_min=0.1,
volume_max=1.0,
volume_step=0.1,
)
assert abs(result - 0.2) < 1e-9
def test_calculate_positions_margin_from_package_root(self) -> None:
"""Position margin helper is importable from the stable surface."""
client = MagicMock()
client.mt5.POSITION_TYPE_BUY = 0
client.mt5.POSITION_TYPE_SELL = 1
client.mt5.ORDER_TYPE_BUY = 10
client.mt5.ORDER_TYPE_SELL = 11
client.positions_get_as_df.return_value = pd.DataFrame()
assert calculate_positions_margin(client) == 0
def test_generic_trading_helpers_from_package_root(self) -> None:
"""New generic trading helpers resolve through the stable surface."""
price = extract_tick_price({"bid": "1.2"}, "bid")
assert price is not None
assert abs(price - 1.2) < 1e-9
assert callable(calculate_trailing_stop_updates)
assert callable(calculate_account_projected_margin_ratio)
assert callable(calculate_projected_margin_ratio)
assert callable(calculate_symbol_group_margin_ratio)
def test_resolve_rate_view_name_from_package_root(self, tmp_path: Path) -> None:
"""Rate view resolution is importable and honors require_existing."""
db_path = tmp_path / "rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
missing = tmp_path / "missing.db"
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_name(missing, "EURUSD", "M1", require_existing=True)
def test_load_rate_data_from_package_root(self, tmp_path: Path) -> None:
"""SQLite rate loading normalizes timestamps through the stable API."""
db_path = tmp_path / "view.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
'CREATE VIEW "rate_EURUSD__1" AS'
" SELECT '2024-01-01T00:00:00+00:00' AS time, 1.1 AS close",
)
frame = load_rate_data(db_path, "rate_EURUSD__1")
assert frame.index.name == "time"
assert abs(float(frame.iloc[0]["close"]) - 1.1) < 1e-9
def test_load_rate_series_from_sqlite_requires_managed_views(
self,
tmp_path: Path,
) -> None:
"""Multi-series loading fails clearly when managed views are absent."""
db_path = tmp_path / "empty-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="No rate compatibility view exists"):
load_rate_series_from_sqlite(db_path, targets, count=10)
assert targets == [RateTarget(symbol="EURUSD", timeframe=1)]
def test_resolve_account_spec_from_package_root(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Account credential resolution uses generic ${ENV_VAR} placeholders."""
monkeypatch.setenv("APP_MT5_LOGIN", "555")
monkeypatch.setenv("APP_MT5_PASSWORD", "secret")
account = AccountSpec(
symbols=["EURUSD"],
login="${APP_MT5_LOGIN}",
password="${APP_MT5_PASSWORD}",
server="Broker-Demo",
)
resolved = resolve_account_spec(account, timeout=3000)
assert resolved.login == "555"
assert resolved.password == "secret" # noqa: S105
assert resolved.timeout == 3000
batch = resolve_account_specs([account], server="Override")
assert batch[0].server == "Override"
def test_mt5_trading_session_lifecycle_from_package_root(
self,
mocker: MockerFixture,
) -> None:
"""Trading session helper initializes and always shuts down."""
mock_client = MagicMock()
mocker.patch(
"mt5cli.trading.Mt5TradingClient",
return_value=mock_client,
)
with mt5_trading_session(login=12345, server="Broker-Demo") as client:
assert client is mock_client
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
def test_trading_order_helpers_importable_from_package_root(self) -> None:
"""Order planning helpers resolve through the stable package surface."""
assert callable(calculate_margin_and_volume)
assert callable(ensure_symbol_selected)
assert callable(place_market_order)
margin_hints = get_type_hints(MarginVolume)
limits_hints = get_type_hints(OrderLimits)
execution_hints = get_type_hints(OrderExecutionResult)
assert margin_hints["buy_volume"] is float
assert limits_hints["stop_loss"] == float | None
assert execution_hints["status"] == ExecutionStatus
def test_mt5_trading_session_shuts_down_on_exception(
self,
mocker: MockerFixture,
) -> None:
"""Trading session helper shuts down even when the body raises."""
mock_client = MagicMock()
mocker.patch(
"mt5cli.trading.Mt5TradingClient",
return_value=mock_client,
)
message = "strategy error"
with (
pytest.raises(RuntimeError, match=message),
mt5_trading_session(login=12345, server="Broker-Demo"),
):
raise RuntimeError(message)
mock_client.shutdown.assert_called_once()
def test_fetch_latest_closed_rates_indexed_from_package_root(
self,
mocker: MockerFixture,
) -> None:
"""Indexed closed-bar helper returns a UTC DatetimeIndex named 'time'."""
client = MagicMock()
mocker.patch(
"mt5cli.trading.fetch_latest_closed_rates_for_trading_client",
return_value=pd.DataFrame(
{
"time": [1704067200, 1704153600, 1704240000],
"close": [1.0, 1.1, 1.2],
},
),
)
result = fetch_latest_closed_rates_indexed(
client,
symbol="EURUSD",
granularity="M1",
count=2,
)
assert isinstance(result.index, pd.DatetimeIndex)
assert result.index.name == "time"
assert result.index.tz is not None
assert "time" not in result.columns
assert "close" in result.columns
+604 -4
View File
@@ -10,14 +10,19 @@ from unittest.mock import MagicMock
import pandas as pd
import pytest
from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from pathlib import Path
from mt5cli import history
from mt5cli.history import (
DEFAULT_HISTORY_TIMEFRAMES,
DedupScope,
RateTarget,
append_dataframe,
augment_written_columns_from_sqlite,
build_rate_targets,
build_rate_view_name,
create_cash_events_view,
create_history_indexes,
@@ -25,6 +30,7 @@ from mt5cli.history import (
create_rate_compatibility_views,
deduplicate_history_tables,
drop_duplicates_in_table,
drop_forming_rate_bar,
filter_incremental_history_deals_frame,
filter_trade_history_frame,
get_history_deals_account_event_start_datetime,
@@ -33,6 +39,8 @@ from mt5cli.history import (
load_incremental_start_datetimes,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
parse_sqlite_timestamp,
quote_sqlite_identifier,
record_written_columns,
@@ -40,6 +48,8 @@ from mt5cli.history import (
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_table_name,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
write_collected_datasets,
@@ -54,12 +64,36 @@ from mt5cli.utils import TIMEFRAME_MAP, Dataset, IfExists
class TestResolveRateViewName:
"""Tests for resolve_rate_view_name and resolve_rate_view_names."""
def test_resolve_rate_table_name_returns_normalized_table(self) -> None:
"""Test canonical normalized rates table name is stable."""
assert resolve_rate_table_name("EURUSD", "M1") == "rates"
def test_resolve_rate_table_name_rejects_empty_symbol(self) -> None:
"""Test canonical rate table resolution validates symbols."""
with pytest.raises(ValueError, match="symbol must not be empty"):
resolve_rate_table_name(" ", "M1")
def test_missing_database_path_does_not_create_file(self, tmp_path: Path) -> None:
"""Test resolving against a missing path does not create a database."""
db_path = tmp_path / "missing.db"
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
assert not db_path.exists()
def test_none_path_returns_default_name(self) -> None:
"""Test a None connection or path returns the deterministic default."""
assert resolve_rate_view_name(None, "EURUSD", "M1") == "rate_EURUSD__1"
assert resolve_rate_view_names(None, ["EURUSD"], ["M1", "H1"]) == [
"rate_EURUSD__1",
"rate_EURUSD__16385",
]
def test_none_path_with_require_existing_raises(self) -> None:
"""Test a None path under strict mode raises a clear error."""
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_name(None, "EURUSD", "M1", require_existing=True)
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_names(None, ["EURUSD"], ["M1"], require_existing=True)
def test_no_rates_table_falls_back_to_single_timeframe_name(
self,
tmp_path: Path,
@@ -392,6 +426,32 @@ class TestLoadRateData:
frame = load_rate_data_from_connection(conn, "rate_view")
assert list(frame["close"]) == [1.0]
def test_load_rate_series_from_sqlite_table_style(
self,
tmp_path: Path,
) -> None:
"""Test public table-style loader returns one rate DataFrame."""
db_path = tmp_path / "table-style.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.executemany(
"INSERT INTO rates(time, close) VALUES (?, ?)",
[
("2024-01-01T00:00:00+00:00", 1.0),
("2024-01-01T00:01:00+00:00", 1.1),
],
)
frame = load_rate_series_from_sqlite(db_path, table="rates", count=1)
assert isinstance(frame, pd.DataFrame)
assert list(frame["close"]) == [1.1]
def test_load_rate_series_from_sqlite_requires_targets_without_table(self) -> None:
"""Test multi-series loading requires targets when table is omitted."""
with pytest.raises(ValueError, match="targets are required"):
load_rate_series_from_sqlite("unused.db", count=1)
def test_loads_quoted_identifier(self, tmp_path: Path) -> None:
"""Test table names are quoted safely."""
db_path = tmp_path / "quoted.db"
@@ -493,6 +553,9 @@ class TestResolveHistorySettings:
"""Test default timeframes include all fixed MT5 values."""
resolved = resolve_history_timeframes(None)
assert len(resolved) == len(DEFAULT_HISTORY_TIMEFRAMES)
assert not any(
name.startswith("TIMEFRAME_") for name in DEFAULT_HISTORY_TIMEFRAMES
)
assert 1 in resolved
assert TIMEFRAME_MAP["H1"] in resolved
@@ -502,7 +565,7 @@ class TestResolveHistorySettings:
def test_resolve_history_tick_flags(self) -> None:
"""Test tick flag resolution."""
assert resolve_history_tick_flags("ALL") == 1
assert resolve_history_tick_flags("ALL") == -1
assert resolve_history_tick_flags(2) == 2
def test_resolve_granularity_name_falls_back_to_integer(self) -> None:
@@ -510,6 +573,57 @@ class TestResolveHistorySettings:
assert resolve_granularity_name(999) == "999"
assert resolve_granularity_name(1) == "M1"
def test_resolve_granularity_name_strips_official_prefix(
self,
mocker: MockerFixture,
) -> None:
"""Test official pdmt5 timeframe names are normalized to short aliases."""
mocker.patch(
"mt5cli.history._get_timeframe_name",
return_value="TIMEFRAME_H1",
)
assert resolve_granularity_name(16385) == "H1"
class TestDropFormingRateBar:
"""Tests for drop_forming_rate_bar."""
def test_drops_still_forming_last_bar(self) -> None:
"""Test the still-forming last bar is removed."""
df_rate = pd.DataFrame(
{"time": [1, 2, 3], "close": [1.1, 1.2, 1.3]},
index=pd.Index(["a", "b", "c"], name="idx"),
)
result = drop_forming_rate_bar(df_rate)
pd.testing.assert_frame_equal(
result,
pd.DataFrame(
{"time": [1, 2], "close": [1.1, 1.2]},
index=pd.Index(["a", "b"], name="idx"),
),
)
assert df_rate.shape == (3, 2)
def test_returns_empty_frame_when_input_empty(self) -> None:
"""Test empty frames stay empty."""
df_rate = pd.DataFrame(columns=["time", "close"])
result = drop_forming_rate_bar(df_rate)
assert result.empty
assert list(result.columns) == ["time", "close"]
def test_returns_empty_frame_when_only_forming_bar_present(self) -> None:
"""Test a single-bar frame becomes empty after dropping the forming bar."""
df_rate = pd.DataFrame({"time": [1], "close": [1.1]})
result = drop_forming_rate_bar(df_rate)
assert result.empty
assert list(result.columns) == ["time", "close"]
class TestParseSqliteTimestamp:
"""Tests for parse_sqlite_timestamp."""
@@ -597,7 +711,7 @@ class TestIncrementalStart:
) -> None:
"""Test rates tables without timeframe fail fast during incremental resume."""
fallback = datetime(2024, 1, 1, tzinfo=UTC)
with sqlite3.connect(tmp_path / "legacy-rates.db") as conn:
with sqlite3.connect(tmp_path / "rates-without-timeframe.db") as conn:
conn.execute("CREATE TABLE rates(symbol TEXT, time TEXT, open REAL)")
conn.execute(
"INSERT INTO rates(symbol, time, open) VALUES (?, ?, ?)",
@@ -866,9 +980,10 @@ class TestDeduplication:
{Dataset.rates},
{
Dataset.rates: [
(
DedupScope(
"symbol = ? AND timeframe = ? AND time >= ?",
("EURUSD", 1, boundary),
frozenset({"symbol", "timeframe", "time"}),
),
],
},
@@ -881,6 +996,89 @@ class TestDeduplication:
("2024-01-02T00:00:00+00:00", 9.9),
]
def test_unusable_scope_falls_back_to_table_dedup(self, tmp_path: Path) -> None:
"""Test scopes with missing columns do not break stable-key dedup."""
boundary = datetime(2024, 1, 1, tzinfo=UTC)
with sqlite3.connect(tmp_path / "orders-without-time.db") as conn:
conn.execute(
"CREATE TABLE history_orders("
" ticket INTEGER, symbol TEXT, time_setup TEXT, type INTEGER)",
)
conn.executemany(
"INSERT INTO history_orders(ticket, symbol, time_setup, type)"
" VALUES (?, ?, ?, ?)",
[
(1, "EURUSD", "2024-01-01T00:00:00+00:00", 0),
(1, "EURUSD", "2024-01-01T00:00:01+00:00", 1),
],
)
deduplicate_history_tables(
conn,
{Dataset.history_orders: {"ticket", "symbol", "time_setup", "type"}},
{Dataset.history_orders},
{
Dataset.history_orders: [
DedupScope(
"symbol = ? AND time >= ?",
("EURUSD", boundary),
frozenset({"symbol", "time"}),
),
],
},
)
rows = conn.execute(
"SELECT ticket, time_setup, type FROM history_orders",
).fetchall()
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
def test_partially_unusable_scopes_only_run_usable_scopes(
self,
tmp_path: Path,
) -> None:
"""Test mixed scope filtering skips only scopes with missing columns."""
boundary = datetime(2024, 1, 2, tzinfo=UTC)
with sqlite3.connect(tmp_path / "partial-scope-filter.db") as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, open REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, open) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 2.0),
("EURUSD", 1, "2024-01-02T00:00:00+00:00", 9.9),
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 100.0),
("USDJPY", 1, "2024-01-02T00:00:00+00:00", 101.0),
],
)
deduplicate_history_tables(
conn,
{Dataset.rates: {"symbol", "timeframe", "time", "open"}},
{Dataset.rates},
{
Dataset.rates: [
DedupScope(
"symbol = ? AND timeframe = ? AND time >= ?",
("EURUSD", 1, boundary),
frozenset({"symbol", "timeframe", "time"}),
),
DedupScope(
"symbol = ? AND timeframe = ? AND broker = ?",
("USDJPY", 1, "demo"),
frozenset({"symbol", "timeframe", "broker"}),
),
],
},
)
rows = conn.execute(
"SELECT symbol, open FROM rates ORDER BY symbol, open",
).fetchall()
assert rows == [
("EURUSD", 9.9),
("USDJPY", 100.0),
("USDJPY", 101.0),
]
class TestRateCompatibilityViews:
"""Tests for rate compatibility view creation."""
@@ -1341,6 +1539,54 @@ class TestIncrementalIntegration:
"rate_EURUSD_M1__1",
}
def test_incremental_orders_without_time_deduplicate_by_ticket(
self,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test incremental history_orders without time deduplicate safely."""
def history_orders_get_as_df(**kwargs: object) -> pd.DataFrame:
if kwargs["symbol"] == "GBPUSD":
return pd.DataFrame()
return pd.DataFrame({
"ticket": [1, 1],
"symbol": ["EURUSD", "EURUSD"],
"time_setup": [
"2024-01-01T00:00:00+00:00",
"2024-01-01T00:00:01+00:00",
],
"type": [0, 1],
})
client = MagicMock()
client.history_orders_get_as_df.side_effect = history_orders_get_as_df
start = datetime(2024, 1, 1, tzinfo=UTC)
end = datetime(2024, 1, 2, tzinfo=UTC)
with (
sqlite3.connect(tmp_path / "incremental-orders-without-time.db") as conn,
caplog.at_level(logging.WARNING, logger="mt5cli.history"),
):
write_incremental_datasets(
conn,
client,
["EURUSD", "GBPUSD"],
{Dataset.history_orders},
[],
0,
start,
end,
deduplicate=True,
create_rate_views=False,
with_views=False,
include_account_events=False,
)
rows = conn.execute(
"SELECT ticket, time_setup, type FROM history_orders",
).fetchall()
assert rows == [(1, "2024-01-01T00:00:01+00:00", 1)]
assert "Skipping history_orders: dataset returned no columns" in caplog.text
def test_write_collected_datasets_and_edge_branches(
self,
tmp_path: Path,
@@ -1558,6 +1804,8 @@ class TestIncrementalIntegration:
"""Test invalid tick flags raise ValueError."""
with pytest.raises(ValueError, match="Invalid tick flags"):
resolve_history_tick_flags("BAD")
with pytest.raises(ValueError, match="Invalid tick flags"):
resolve_history_tick_flags(7)
def test_resolve_history_timeframes_invalid(self) -> None:
"""Test invalid timeframes raise ValueError."""
@@ -1716,7 +1964,7 @@ class TestIncrementalHistoryDeals:
})
start = datetime(2024, 1, 1, tzinfo=UTC)
end = datetime(2024, 1, 3, tzinfo=UTC)
with sqlite3.connect(tmp_path / "legacy-deals.db") as conn:
with sqlite3.connect(tmp_path / "deals-without-type.db") as conn:
conn.execute(
"CREATE TABLE history_deals( ticket INTEGER, symbol TEXT, time TEXT)",
)
@@ -1915,3 +2163,355 @@ class TestWriteHelpers:
)
assert get_table_columns(conn, "rates") == {"time", "open"}
create_history_indexes(conn, written_columns)
class TestRateSourceHelpers:
"""Tests for generic rate-source SDK helpers."""
def test_rate_target_timeframe_int(self) -> None:
"""Test RateTarget resolves named and integer timeframes."""
target = RateTarget(symbol="EURUSD", timeframe="M1")
assert target.timeframe == 1
assert target.timeframe_int == 1
assert RateTarget(symbol="EURUSD", timeframe=16385).timeframe_int == 16385
def test_build_rate_targets_row_major(self) -> None:
"""Test targets are built in row-major symbol/timeframe order."""
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
assert [(t.symbol, t.timeframe) for t in targets] == [
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
]
def test_build_rate_targets_allows_missing_symbol(self) -> None:
"""Test missing symbols produce None-symbol targets when allowed."""
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
assert [(t.symbol, t.timeframe) for t in targets] == [
(None, 1),
(None, 16385),
]
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
(["EURUSD"], [], "At least one timeframe"),
([], ["M1"], "At least one symbol"),
],
)
def test_build_rate_targets_rejects_empty(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test target building input validation."""
with pytest.raises(ValueError, match=match):
build_rate_targets(symbols, timeframes)
def test_resolve_rate_tables_uses_explicit_tables(self) -> None:
"""Test explicit tables bypass view resolution when counts match."""
targets = build_rate_targets([], ["M1", "H1"], allow_missing_symbol=True)
assert resolve_rate_tables(None, targets, ["t1", "t2"]) == ["t1", "t2"]
def test_resolve_rate_tables_rejects_mismatched_explicit_count(self) -> None:
"""Test explicit table count must match the number of targets."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="Expected 1 explicit table"):
resolve_rate_tables(None, targets, ["t1", "t2"])
def test_resolve_rate_tables_rejects_empty_targets(self) -> None:
"""Test resolving requires at least one target."""
with pytest.raises(ValueError, match="At least one rate target"):
resolve_rate_tables(None, [])
def test_resolve_rate_tables_requires_symbol_without_explicit(self) -> None:
"""Test None-symbol targets require explicit tables."""
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
with pytest.raises(ValueError, match="without a symbol"):
resolve_rate_tables(None, targets)
def test_resolve_rate_tables_resolves_view_names(self) -> None:
"""Test symbol targets resolve to default view names without a database."""
targets = build_rate_targets(["EURUSD"], ["M1", "H1"])
assert resolve_rate_tables(None, targets) == [
"rate_EURUSD__1",
"rate_EURUSD__16385",
]
def test_resolve_rate_tables_none_path_with_require_existing_raises(self) -> None:
"""Test strict mode rejects a missing database path."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_tables(None, targets, require_existing=True)
def test_resolve_rate_tables_missing_db_with_require_existing_raises(
self,
tmp_path: Path,
) -> None:
"""Test strict mode rejects a non-existing database path."""
db_path = tmp_path / "missing.db"
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_tables(db_path, targets, require_existing=True)
def test_resolve_rate_tables_missing_view_with_require_existing_raises(
self,
tmp_path: Path,
) -> None:
"""Test strict mode rejects databases without managed rate views."""
db_path = tmp_path / "no-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="No rate compatibility view exists"):
resolve_rate_tables(db_path, targets, require_existing=True)
def test_resolve_rate_tables_with_require_existing_resolves_views(
self,
tmp_path: Path,
) -> None:
"""Test strict mode resolves existing managed rate views."""
db_path = tmp_path / "strict-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
targets = build_rate_targets(["EURUSD"], ["M1"])
assert resolve_rate_tables(db_path, targets, require_existing=True) == [
"rate_EURUSD__1",
]
def test_resolve_rate_tables_batches_sqlite_metadata(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Test resolving multiple targets loads SQLite metadata once."""
db_path = tmp_path / "batch-rate-tables.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 16385, "2024-01-01T01:00:00+00:00", 1.1),
("GBPUSD", 1, "2024-01-01T00:00:00+00:00", 1.2),
],
)
create_rate_compatibility_views(conn)
counts_spy = mocker.spy(history, "_load_rates_timeframe_counts")
views_spy = mocker.spy(history, "_load_existing_rate_views")
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
assert resolve_rate_tables(db_path, targets) == [
"rate_EURUSD__M1_1",
"rate_EURUSD__H1_16385",
"rate_GBPUSD__1",
"rate_GBPUSD__16385",
]
assert counts_spy.call_count == 1
assert views_spy.call_count == 1
def test_load_rate_series_from_sqlite(self, tmp_path: Path) -> None:
"""Test loading multiple rate series keyed by symbol and timeframe."""
db_path = tmp_path / "series.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 1, "2024-01-01T00:01:00+00:00", 1.1),
],
)
create_rate_compatibility_views(conn)
targets = build_rate_targets(["EURUSD"], ["M1"])
result = load_rate_series_from_sqlite(db_path, targets, count=2)
assert set(result) == {("EURUSD", 1)}
assert len(result["EURUSD", 1]) == 2
def test_load_rate_series_by_granularity(self, tmp_path: Path) -> None:
"""Test loading rate series keyed by symbol and granularity name."""
db_path = tmp_path / "granularity.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", 16385, "2024-01-01T00:00:00+00:00", 1.1),
],
)
create_rate_compatibility_views(conn)
result = load_rate_series_by_granularity(
db_path,
["EURUSD"],
["M1", "H1"],
count=1,
)
assert set(result) == {("EURUSD", "M1"), ("EURUSD", "H1")}
def test_load_rate_series_by_granularity_explicit_tables(
self,
tmp_path: Path,
) -> None:
"""Test explicit tables with None-symbol targets key by granularity."""
db_path = tmp_path / "granularity-explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
result = load_rate_series_by_granularity(
db_path,
[],
["M1"],
count=1,
explicit_tables=["custom_view"],
allow_missing_symbol=True,
)
assert set(result) == {(None, "M1")}
def test_load_rate_series_reuses_path_connection(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""Test loading from a path opens SQLite once for resolve and reads."""
db_path = tmp_path / "single-open-series.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
connect_spy = mocker.spy(history.sqlite3, "connect")
result = load_rate_series_from_sqlite(
db_path,
build_rate_targets(["EURUSD"], ["M1"]),
count=1,
)
assert set(result) == {("EURUSD", 1)}
assert connect_spy.call_count == 1
def test_load_rate_series_with_explicit_tables(self, tmp_path: Path) -> None:
"""Test explicit tables and None-symbol targets load series."""
db_path = tmp_path / "explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
result = load_rate_series_from_sqlite(
db_path,
targets,
count=1,
explicit_tables=["custom_view"],
)
assert set(result) == {(None, 1)}
def test_load_rate_series_rejects_non_positive_count(self) -> None:
"""Test loading requires a positive count."""
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="count must be positive"):
load_rate_series_from_sqlite("unused.db", targets, count=0)
def test_load_rate_series_rejects_empty_targets(self) -> None:
"""Test loading requires at least one target before opening SQLite."""
with pytest.raises(ValueError, match="At least one rate target"):
load_rate_series_from_sqlite("unused.db", [], count=1)
def test_load_rate_series_requires_symbol_without_explicit_tables(self) -> None:
"""Test None-symbol targets require explicit tables before opening SQLite."""
targets = build_rate_targets([], ["M1"], allow_missing_symbol=True)
with pytest.raises(ValueError, match="without a symbol"):
load_rate_series_from_sqlite("unused.db", targets, count=1)
def test_load_rate_series_requires_existing_managed_views(
self,
tmp_path: Path,
) -> None:
"""Test loading without explicit tables requires managed rate views."""
db_path = tmp_path / "no-managed-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
targets = build_rate_targets(["EURUSD"], ["M1"])
with pytest.raises(ValueError, match="No rate compatibility view exists"):
load_rate_series_from_sqlite(db_path, targets, count=1)
def test_load_rate_series_rejects_duplicate_targets(self) -> None:
"""Test duplicate (symbol, timeframe) targets are rejected."""
targets = [
RateTarget("EURUSD", 1),
RateTarget("EURUSD", "M1"),
]
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
load_rate_series_from_sqlite("unused.db", targets, count=1)
def test_load_rate_series_rejects_duplicate_targets_with_explicit_tables(
self,
tmp_path: Path,
) -> None:
"""Test duplicate targets are rejected even with explicit tables."""
db_path = tmp_path / "duplicate-explicit.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE custom_view(time TEXT, close REAL)")
conn.execute(
"INSERT INTO custom_view(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
targets = [
RateTarget("EURUSD", 1),
RateTarget("EURUSD", 1),
]
with pytest.raises(ValueError, match=r"Duplicate rate target: \('EURUSD', 1\)"):
load_rate_series_from_sqlite(
db_path,
targets,
count=1,
explicit_tables=["custom_view", "custom_view"],
)
+1224 -36
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+51 -14
View File
@@ -274,8 +274,14 @@ class TestParseTimeframe:
assert parse_timeframe(value) == expected
def test_integer_timeframe(self) -> None:
"""Test parsing integer timeframe."""
assert parse_timeframe("42") == 42
"""Test parsing supported integer timeframes."""
assert parse_timeframe("1") == 1
assert parse_timeframe(16385) == 16385
def test_unsupported_integer_timeframe_raises(self) -> None:
"""Test that unsupported integer timeframes raise ValueError."""
with pytest.raises(ValueError, match="Invalid timeframe"):
parse_timeframe("42")
def test_invalid_timeframe_raises(self) -> None:
"""Test that invalid timeframe raises ValueError."""
@@ -288,15 +294,21 @@ class TestParseTickFlags:
@pytest.mark.parametrize(
("value", "expected"),
[("ALL", 1), ("info", 2), ("TRADE", 4)],
[("ALL", -1), ("info", 1), ("TRADE", 2), ("COPY_TICKS_ALL", -1)],
)
def test_named_flag(self, value: str, expected: int) -> None:
"""Test parsing named tick flags."""
assert parse_tick_flags(value) == expected
def test_integer_flag(self) -> None:
"""Test parsing integer tick flag."""
assert parse_tick_flags("7") == 7
"""Test parsing supported integer tick flags."""
assert parse_tick_flags("-1") == -1
assert parse_tick_flags(2) == 2
def test_unsupported_integer_flag_raises(self) -> None:
"""Test that unsupported integer tick flags raise ValueError."""
with pytest.raises(ValueError, match="Invalid tick flags"):
parse_tick_flags("7")
def test_invalid_flag_raises(self) -> None:
"""Test that invalid flag raises ValueError."""
@@ -355,8 +367,11 @@ class TestConstants:
assert key in TIMEFRAME_MAP
def test_tick_flag_map_has_expected_keys(self) -> None:
"""Test that TICK_FLAG_MAP contains standard flags."""
assert set(TICK_FLAG_MAP) == {"ALL", "INFO", "TRADE"}
"""Test that TICK_FLAG_MAP contains standard flags with MT5 values."""
assert {"ALL", "INFO", "TRADE"} <= set(TICK_FLAG_MAP)
assert TICK_FLAG_MAP["ALL"] == -1
assert TICK_FLAG_MAP["INFO"] == 1
assert TICK_FLAG_MAP["TRADE"] == 2
@pytest.mark.parametrize(
("dataset", "expected"),
@@ -403,26 +418,48 @@ class TestTimeframeType:
"""Test converting a string to timeframe integer."""
assert TIMEFRAME_TYPE.convert("H1", None, None) == 16385
def test_convert_int_passthrough(self) -> None:
"""Test that integer values pass through unchanged."""
assert TIMEFRAME_TYPE.convert(42, None, None) == 42
def test_convert_int(self) -> None:
"""Test converting supported integer timeframe values."""
assert TIMEFRAME_TYPE.convert(16385, None, None) == 16385
def test_convert_unsupported_int(self) -> None:
"""Test that unsupported integer values raise BadParameter."""
with pytest.raises(Exception, match="Invalid timeframe"):
TIMEFRAME_TYPE.convert(42, None, None)
def test_convert_invalid(self) -> None:
"""Test that invalid values raise BadParameter."""
with pytest.raises(Exception, match="Invalid timeframe"):
TIMEFRAME_TYPE.convert("bad", None, None)
@pytest.mark.parametrize("value", [True, False, None, 1.5])
def test_convert_invalid_types(self, value: object) -> None:
"""Test that bool, float, and None values raise BadParameter."""
with pytest.raises(Exception, match="Invalid timeframe"):
TIMEFRAME_TYPE.convert(value, None, None)
class TestTickFlagsType:
"""Tests for _TickFlagsType."""
def test_convert_string(self) -> None:
"""Test converting a string to tick flags integer."""
assert TICK_FLAGS_TYPE.convert("ALL", None, None) == 1
assert TICK_FLAGS_TYPE.convert("ALL", None, None) == -1
def test_convert_int_passthrough(self) -> None:
"""Test that integer values pass through unchanged."""
assert TICK_FLAGS_TYPE.convert(7, None, None) == 7
def test_convert_int(self) -> None:
"""Test converting supported integer tick flag values."""
assert TICK_FLAGS_TYPE.convert(2, None, None) == 2
def test_convert_unsupported_int(self) -> None:
"""Test that unsupported integer values raise BadParameter."""
with pytest.raises(Exception, match="Invalid tick flags"):
TICK_FLAGS_TYPE.convert(7, None, None)
@pytest.mark.parametrize("value", [True, False, None, 1.5])
def test_convert_invalid_types(self, value: object) -> None:
"""Test that bool, float, and None values raise BadParameter."""
with pytest.raises(Exception, match="Invalid tick flags"):
TICK_FLAGS_TYPE.convert(value, None, None)
def test_convert_invalid(self) -> None:
"""Test that invalid values raise BadParameter."""
Generated
+8 -8
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.5.0"
version = "0.9.4"
source = { editable = "." }
dependencies = [
{ name = "click" },
@@ -513,7 +513,7 @@ dev = [
[package.metadata]
requires-dist = [
{ name = "click", specifier = ">=8.1.0" },
{ name = "pdmt5", specifier = ">=0.2.3" },
{ name = "pdmt5", specifier = ">=0.3.0" },
{ name = "pyarrow", specifier = ">=19.0.0" },
{ name = "typer", specifier = ">=0.15.0" },
]
@@ -684,16 +684,16 @@ wheels = [
[[package]]
name = "pdmt5"
version = "0.2.3"
version = "0.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "metatrader5", marker = "sys_platform == 'win32'" },
{ name = "pandas" },
{ name = "pydantic" },
]
sdist = { url = "https://files.pythonhosted.org/packages/02/25/52d9d954504ccdd0fe91f715ab74c424d61234b237cc4160d3ebe20070f1/pdmt5-0.2.3.tar.gz", hash = "sha256:21384f5826fb0125fee3f93c90b108340f55ab53b1c819d229ceac162289d2ec", size = 226665, upload-time = "2026-02-05T13:28:21.071Z" }
sdist = { url = "https://files.pythonhosted.org/packages/bf/cc/c8fa3a01e0e34178fec8527992f7bb8eda5881477ce23aaacaa9b2ef7bec/pdmt5-0.3.0.tar.gz", hash = "sha256:bb612d5c2695eafac9b2a7b74756e13bd383d7e5517bd90c9a2efa92492c484c", size = 215100, upload-time = "2026-06-11T13:26:46.976Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c1/75/c5e52a9cf459b85b2dd52f83e70857571b1b45805c9fe610b3959a26ac15/pdmt5-0.2.3-py3-none-any.whl", hash = "sha256:f92246a05cfc3b7feb3ab0cc5b48768a4d84aad6b02e7a68060948f5828718a1", size = 22967, upload-time = "2026-02-05T13:28:19.523Z" },
{ url = "https://files.pythonhosted.org/packages/f2/03/b12cc4c9db983d971c9172b3765161b6d91136d0624e6718a04dd815e7a1/pdmt5-0.3.0-py3-none-any.whl", hash = "sha256:5388b406cc583202600cfe22c9d781679b1d931b1ed5a2b5dcf37c566149b49f", size = 26250, upload-time = "2026-06-11T13:26:45.689Z" },
]
[[package]]
@@ -836,11 +836,11 @@ wheels = [
[[package]]
name = "pygments"
version = "2.19.2"
version = "2.20.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
]
[[package]]