Compare commits

...

9 Commits

Author SHA1 Message Date
dceoy d654b82f9d Bump version from 0.6.0 to 0.6.1.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 19:36:34 +09:00
Daichi Narushima b5e82e71c7 Add trading session helpers and extend ThrottledHistoryUpdater (#25)
* Add trading session helpers and extend ThrottledHistoryUpdater

Introduce mt5cli.trading with mt5_trading_session() for Mt5TradingClient
lifecycle management and reusable operational helpers for position-side
detection, margin/volume sizing, and protective order price derivation.

Extend ThrottledHistoryUpdater to validate inputs before updates and to
optionally suppress ValueError, OSError, and missing-method errors without
advancing the throttle timestamp.

Export the new helpers from mt5cli.__init__, add unit tests with mocked
clients, and document migration guidance for downstream projects such as
mteor.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Narrow ThrottledHistoryUpdater suppress_errors handling (#27)

* Narrow ThrottledHistoryUpdater suppress_errors for MT5 capability only

Remove broad AttributeError/TypeError handling from recoverable errors.
Add _is_mt5_client_capability_error() to detect missing history API methods
or non-callable client attributes by message and attribute name.

Generic AttributeError/TypeError values always propagate even when
suppress_errors=True. Update docs and tests accordingly.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Detect non-callable history client methods in suppress_errors

Address review feedback: when a history API attribute exists but is not
callable, Python raises a generic TypeError. Inspect the traceback for
mt5cli.history client call sites so these capability mismatches are still
suppressed without matching all TypeError values.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Address PR review feedback on trading helpers

- Resolve history module path once at import time
- Only treat non-callable TypeErrors as capability errors at the raise site
- Validate SL/TP ratios in determine_order_limits
- Add tests for margin_free edge cases, body-raise shutdown, and internal TypeError propagation
- Clarify ThrottledHistoryUpdater suppress_errors docs
- Split README migration example into trading vs read-only history sessions

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* Tighten protective ratio validation and clamp negative margin_free

Add _require_protective_ratio enforcing 0 <= ratio < 1 for SL/TP limits so
a ratio of 1.0 cannot produce zero protective prices. Clamp negative
margin_free to 0.0 in calculate_margin_and_volume before sizing.

Add boundary and negative-margin tests; document constraints in trading API
docs.

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-11 19:32:52 +09:00
Daichi Narushima 18df96872b Add closed-bar rate helpers (v0.6.0) (#26)
* Add closed-bar rate helpers and bump version to 0.6.0.

Expose drop_forming_rate_bar and multi-account collectors so downstream apps no longer need count+1 fetches and manual bar trimming.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump pygments to 2.20.0 to fix CVE-2026-4539 ReDoS advisory.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback on closed-bar rate collection.

Validate count and start_pos before MT5 fetches, avoid redundant frame copies, clarify empty-series errors, and expand test coverage.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Include symbol and timeframe in empty closed-rate error messages.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 02:30:48 +09:00
Daichi Narushima 5b1d54bfe9 Add resilient multi-account orchestration helpers (#22)
* Add SDK orchestration helpers for resilient multi-account collection

- collect_latest_rates_for_accounts_with_retries(): exponential-backoff
  retries around collect_latest_rates_for_accounts(), retrying only
  Mt5TradingError/Mt5RuntimeError and re-raising on exhaustion.
- resolve_account_spec()/resolve_account_specs() and
  substitute_env_placeholders(): merge explicit overrides over AccountSpec
  fields and expand ${ENV_VAR} placeholders, raising ValueError on missing
  variables.
- ThrottledHistoryUpdater: monotonic-clock throttled wrapper around
  update_history() with should_update()/update() and opt-in suppress_errors.
- load_rate_series_by_granularity(): rate-series loader keyed by
  (symbol | None, granularity_name).
- Export new APIs, add unit tests (100% coverage), and document in README
  and docs/api.

* chore: bump version from 0.5.1 to 0.5.3 (#24)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: resolve leftover merge conflict markers in version files

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: address PR review feedback on SDK orchestration helpers

- Use single-pass env substitution to avoid TOCTOU KeyError
- Apply backoff_base to all retry delays (backoff_base ** (attempt + 1))
- Preserve integer logins in resolve_account_spec; hide login in repr
- Fix docs examples (env ordering, while True loop, backoff comment)
- Parametrize suppress_errors tests for MT5 and SQLite errors

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>
2026-06-10 00:15:07 +09:00
Daichi Narushima ad9e513253 [codex] Guard dedup scopes by written columns (#23)
* Guard dedup scopes by written columns

* Address dedup scope review feedback

* Remove legacy dedup scope support

* Remove stale legacy descriptions

* chore: bump version from 0.5.1 to 0.5.2
2026-06-09 23:27:54 +09:00
Daichi Narushima 334f01b647 chore: bump version from 0.5.0 to 0.5.1 (#21) 2026-06-09 15:52:32 +09:00
Daichi Narushima 1b69e8f08e Add generic MT5 rate-loading SDK APIs for downstream reuse (#20) 2026-06-09 15:37:24 +09:00
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
Daichi Narushima b2bb2ad0a0 Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names.

Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add reusable export, tick-window, and margin helpers for downstream tools.

Expose SQLite append/dedup export, recent tick retrieval, and minimum margin
summary through the SDK and CLI so projects like mteor can depend on mt5cli
instead of duplicating MT5 data plumbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version to 0.4.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for rate view resolution and SDK helpers.

Harden SQLite read-only connections, tighten view discovery, improve recent_ticks
fetch efficiency, default SQLite export to append, and expand tests and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix read-only SQLite URI construction on Windows.

Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 03:29:03 +09:00
20 changed files with 5552 additions and 82 deletions
+110 -22
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -127,6 +133,31 @@ update_history_with_config(
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
```python
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
eurusd_m1 = rates["EURUSD", "M1"] # closed bars only
```
- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`, `sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability errors for history API methods without advancing the throttle (other `AttributeError` / `TypeError` values always propagate).
- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable `pdmt5.Mt5TradingClient` that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. The read-only `mt5_session()` / `Mt5CliClient` SDK is unchanged.
- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
## Requirements
@@ -134,6 +165,63 @@ update_history_with_config(
- Windows OS (MetaTrader 5 requirement)
- MetaTrader 5 platform installed
### Migration note for mteor
Replace local MT5 lifecycle and trading helper code with mt5cli imports:
```python
# Before (local mteor helpers)
# with local_mt5_trading_session(config) as client:
# side = local_detect_position_side(client, symbol)
# sizing = local_calculate_margin_and_volume(client, symbol, unit_ratio, preserved_ratio)
# limits = local_determine_order_limits(client, symbol, side, sl_ratio, tp_ratio)
# After (mt5cli shared layer)
from pdmt5 import Mt5Config
from mt5cli import (
calculate_margin_and_volume,
detect_position_side,
determine_order_limits,
mt5_trading_session,
)
with mt5_trading_session(
Mt5Config(path=terminal_path, login=login), retry_count=2
) as client:
side = detect_position_side(client, symbol)
sizing = calculate_margin_and_volume(
client, symbol, unit_margin_ratio=0.5, preserved_margin_ratio=0.2
)
if side is not None:
limits = determine_order_limits(
client,
symbol,
side,
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.02,
)
```
Throttled history updates use a separate read-only session:
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db", interval_seconds=60, suppress_errors=True
)
client = Mt5DataClient(config=Mt5Config(login=login))
client.initialize_and_login_mt5()
try:
updater.update(client, ["EURUSD"])
finally:
client.shutdown()
```
Read-only collectors can keep using `mt5_session()` and `Mt5CliClient` without changes.
## Development
```bash
+98
View File
@@ -129,3 +129,101 @@ when required columns are missing.
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing naming
schemes:
```python
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
```
Resolution rules:
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
are stored for the same symbol.
- When multiple naming candidates apply, prefers an existing managed
`rate_*__*` view from the candidate list.
- Falls back to single-timeframe naming when the database path is missing or
`rates` metadata is unavailable.
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
### Multi-series rate loading
For loading many rate series at once, build neutral `RateTarget` pairs and load
them from SQLite in one call. View names are resolved via the same
compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
```python
from pathlib import Path
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
```
- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
row-major order, normalizing timeframe names such as `"M1"` to their integer
values; set `allow_missing_symbol=True` to address series solely by
`explicit_tables` (targets carry `symbol=None`).
- `resolve_rate_tables()` maps targets to table or view names and validates that
any `explicit_tables` count matches the target count. Pass
`require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or managed view is missing. When
`explicit_tables` is provided, names are returned as-is and
`require_existing` is ignored.
- `load_rate_series_from_sqlite()` returns a mapping keyed by
`(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
requires existing managed `rate_*` compatibility views and raises
`ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
are rejected.
- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
loads the series, and rekeys the result by granularity name to avoid
converting integer timeframes downstream:
```python
from mt5cli import load_rate_series_by_granularity
series = load_rate_series_by_granularity(
"history.db", ["EURUSD"], ["M1", "H1"], count=1000
)
frame = series["EURUSD", "M1"] # keyed by (symbol | None, granularity_name)
```
+27 -2
View File
@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
### [Trading](trading.md)
Trading-capable session management and operational helpers built on `pdmt5.Mt5TradingClient`. Complements the read-only SDK without changing existing `Mt5CliClient` behavior.
### [History Collection (SQLite)](history.md)
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
@@ -28,8 +32,9 @@ The package follows a simple architecture built on top of pdmt5:
1. **CLI Layer** (`cli.py`): Typer application with subcommands that delegate to the SDK and export results.
2. **SDK Layer** (`sdk.py`): Read-only data access functions, `Mt5CliClient`, and `collect_history` orchestration.
3. **Utils Layer** (`utils.py`): Constants, enums, custom Click parameter types, parsing helpers, and format detection/export utilities.
4. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient` and `Mt5Config` from the pdmt5 package for all MetaTrader 5 data access.
3. **Trading Layer** (`trading.py`): Trading-capable sessions and operational helpers on `Mt5TradingClient`.
4. **Utils Layer** (`utils.py`): Constants, enums, custom Click parameter types, parsing helpers, and format detection/export utilities.
5. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient`, `Mt5TradingClient`, and `Mt5Config` from the pdmt5 package for MetaTrader 5 access.
## Usage Guidelines
@@ -65,12 +70,18 @@ from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
Dataset,
IfExists,
Mt5CliClient,
collect_history,
copy_rates_range,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically
rates = copy_rates_range(
@@ -86,6 +97,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite
collect_history(
Path("history.db"),
+111
View File
@@ -1,3 +1,114 @@
# SDK Module
::: mt5cli.sdk
## Resilient multi-account orchestration
The SDK ships strategy-agnostic helpers for building long-running collectors on
top of the read-only client. None of them depend on a particular trading
application.
### Retrying transient rate collection
`collect_latest_rates_for_accounts_with_retries()` wraps
`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
failure is re-raised once `retry_count` is exhausted.
```python
from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
rates = collect_latest_rates_for_accounts_with_retries(
accounts,
["M1", "H1"],
count=500,
retry_count=3,
backoff_base=2, # sleeps 2s, 4s, 8s between attempts
)
```
### Latest closed rate bars
MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
row. `collect_latest_closed_rates_for_accounts()` fetches `count + 1` bars,
drops that row with `drop_forming_rate_bar()`, and validates each series is
non-empty. Use `collect_latest_closed_rates_by_granularity()` when callers
prefer keys such as `("EURUSD", "M1")` instead of integer timeframes.
```python
from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
rates = collect_latest_closed_rates_by_granularity(
[AccountSpec(symbols=["EURUSD"], login=12345)],
["M1", "H1"],
count=500,
retry_count=3,
)
closed_m1 = rates["EURUSD", "M1"]
```
### Resolving credentials and `${ENV_VAR}` placeholders
`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
secrets out of plan/config files. A missing environment variable raises
`ValueError`.
```python
import os
from mt5cli import AccountSpec, resolve_account_specs
os.environ["MT5_LOGIN"] = "12345"
os.environ["MT5_PASSWORD"] = "secret"
accounts = [
AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
]
resolved = resolve_account_specs(accounts, server="Broker-Demo")
# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
```
### Throttled incremental history updates
`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
between successful runs (using a monotonic clock), so an application loop can
call it every iteration without over-fetching.
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, ThrottledHistoryUpdater
updater = ThrottledHistoryUpdater(
output="history.db",
datasets={Dataset.rates},
timeframes=["M1"],
interval_seconds=60, # <= 0 updates on every call
)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
while True:
updater.update(client, ["EURUSD", "GBPUSD"]) # no-op until 60s elapse
# ... do other work; break when shutting down ...
finally:
client.shutdown()
```
By default recoverable errors (`Mt5TradingError`, `Mt5RuntimeError`,
`sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability
`AttributeError` / `TypeError` for history API methods) propagate so the caller
controls logging; pass `suppress_errors=True` to swallow them and return
`False` without advancing the throttle. Other `AttributeError` / `TypeError`
values always propagate. Input validation (`_resolve_update_history_request`)
runs before any MT5 or SQLite calls, but when `suppress_errors=True` the
resulting `ValueError` is suppressed along with other recoverable errors.
## Trading-capable sessions
For order placement and trading calculations, use the dedicated
[Trading module](trading.md). The read-only `Mt5CliClient` and `mt5_session()`
helpers in this module are unchanged.
+70
View File
@@ -0,0 +1,70 @@
# Trading Module
::: mt5cli.trading
## Trading-capable MT5 sessions
`mt5_trading_session()` complements the read-only `mt5_session()` helper in
`sdk.py`. It yields a connected `pdmt5.Mt5TradingClient`, uses
`Mt5Config.path` to launch the terminal when configured, and always calls
`shutdown()` on exit.
```python
from pdmt5 import Mt5Config
from mt5cli import mt5_trading_session
with mt5_trading_session(
Mt5Config(path=r"C:\Program Files\MetaTrader 5\terminal64.exe", login=12345),
retry_count=2,
) as client:
positions = client.positions_get_as_df(symbol="EURUSD")
```
The read-only `Mt5CliClient` / `mt5_session()` API is unchanged.
## Operational trading helpers
These helpers are strategy-agnostic and do not depend on signal detection,
betting logic, or scheduling code in downstream applications.
```python
from mt5cli import (
calculate_margin_and_volume,
detect_position_side,
determine_order_limits,
)
side = detect_position_side(client, "EURUSD")
sizing = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
limits = determine_order_limits(
client,
"EURUSD",
side="long",
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.02,
)
```
Protective ratios must satisfy `0 <= ratio < 1`; `0` omits that level.
`calculate_margin_and_volume()` clamps negative `margin_free` to `0.0`
before sizing.
## Migration from mteor-local helpers
| mteor-local concern | mt5cli replacement |
| -------------------------------------------------------- | ----------------------------------------------- |
| Manual terminal spawn/kill around trading code | `mt5_trading_session()` |
| Local position-side detection | `detect_position_side()` |
| Local margin/volume sizing | `calculate_margin_and_volume()` |
| Local SL/TP price derivation | `determine_order_limits()` |
| Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)` |
Keep read-only data collection on `mt5_session()` / `Mt5CliClient`; use
`mt5_trading_session()` only where order placement or trading calculations are
required.
+43 -14
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -22,13 +23,23 @@ pip install mt5cli
## Programmatic usage / SDK usage
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results.
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe
from mt5cli import (
Mt5CliClient,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers
rates = copy_rates_range(
@@ -39,10 +50,21 @@ rates = copy_rates_range(
)
export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
@@ -58,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start
```bash
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range |
### Ticks
| Command | Description |
| ------------- | ------------------------------ |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| Command | Description |
| -------------- | ----------------------------------- |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a trailing window |
### Information
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume margin summary |
| `market-book` | Export market depth (order book) |
### Trading
| Command | Description |
| ---------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| Command | Description |
| ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`.
+1
View File
@@ -58,6 +58,7 @@ nav:
- Overview: api/index.md
- CLI: api/cli.md
- SDK: api/sdk.md
- Trading: api/trading.md
- History Collection (SQLite): api/history.md
- Utils: api/utils.md
+92 -1
View File
@@ -2,11 +2,34 @@
from importlib.metadata import version
from .history import (
RateTarget,
build_rate_targets,
build_rate_view_name,
drop_forming_rate_bar,
load_rate_data,
load_rate_data_from_connection,
load_rate_series_by_granularity,
load_rate_series_from_sqlite,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_tables,
resolve_rate_view_name,
resolve_rate_view_names,
)
from .sdk import (
AccountSpec,
Mt5CliClient,
ThrottledHistoryUpdater,
account_info,
build_config,
collect_history,
collect_latest_closed_rates_by_granularity,
collect_latest_closed_rates_for_accounts,
collect_latest_rates,
collect_latest_rates_for_accounts,
collect_latest_rates_for_accounts_with_retries,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -15,9 +38,19 @@ from .sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_session,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
resolve_account_spec,
resolve_account_specs,
substitute_env_placeholders,
symbol_info,
symbol_info_tick,
symbols,
@@ -28,31 +61,89 @@ from .sdk import (
from .sdk import (
version as mt5_version,
)
from .utils import Dataset, IfExists, detect_format, export_dataframe
from .trading import (
calculate_margin_and_volume,
detect_position_side,
determine_order_limits,
mt5_trading_session,
)
from .utils import (
TICK_FLAG_MAP,
TIMEFRAME_MAP,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_tick_flags,
parse_timeframe,
)
__version__ = version(__package__) if __package__ else None
__all__ = [
"TICK_FLAG_MAP",
"TIMEFRAME_MAP",
"AccountSpec",
"Dataset",
"IfExists",
"Mt5CliClient",
"RateTarget",
"ThrottledHistoryUpdater",
"account_info",
"build_config",
"build_rate_targets",
"build_rate_view_name",
"calculate_margin_and_volume",
"collect_history",
"collect_latest_closed_rates_by_granularity",
"collect_latest_closed_rates_for_accounts",
"collect_latest_rates",
"collect_latest_rates_for_accounts",
"collect_latest_rates_for_accounts_with_retries",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
"copy_ticks_from",
"copy_ticks_range",
"detect_format",
"detect_position_side",
"determine_order_limits",
"drop_forming_rate_bar",
"export_dataframe",
"export_dataframe_to_sqlite",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"load_rate_series_by_granularity",
"load_rate_series_from_sqlite",
"market_book",
"minimum_margins",
"mt5_session",
"mt5_summary",
"mt5_summary_as_df",
"mt5_trading_session",
"mt5_version",
"orders",
"parse_datetime",
"parse_tick_flags",
"parse_timeframe",
"positions",
"recent_history_deals",
"recent_ticks",
"resolve_account_spec",
"resolve_account_specs",
"resolve_history_datasets",
"resolve_history_tick_flags",
"resolve_history_timeframes",
"resolve_rate_tables",
"resolve_rate_view_name",
"resolve_rate_view_names",
"substitute_env_placeholders",
"symbol_info",
"symbol_info_tick",
"symbols",
+104
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
)
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command()
def rates_range(
ctx: typer.Context,
@@ -300,6 +325,44 @@ def ticks_range(
)
@app.command()
def ticks_recent(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
seconds: Annotated[
float,
typer.Option(help="Lookback window in seconds."),
],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
count: Annotated[
int,
typer.Option(help="Maximum number of ticks to return."),
] = 10000,
flags: Annotated[
int,
typer.Option(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
),
)
@app.command()
def account_info(ctx: typer.Context) -> None:
"""Export account information."""
@@ -335,6 +398,16 @@ def symbol_info(
_execute_export(ctx, lambda: client.symbol_info(symbol))
@app.command()
def minimum_margins(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
@app.command()
def orders(
ctx: typer.Context,
@@ -427,6 +500,37 @@ def history_deals(
)
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
+693 -11
View File
@@ -4,8 +4,10 @@ from __future__ import annotations
import logging
import sqlite3
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Literal
from pathlib import Path
from typing import TYPE_CHECKING, Literal, cast
import pandas as pd
@@ -19,7 +21,7 @@ from .utils import (
)
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from collections.abc import Callable, Mapping, Sequence
from pdmt5 import Mt5DataClient
@@ -104,6 +106,23 @@ def resolve_granularity_name(timeframe: int) -> str:
return str(timeframe)
def drop_forming_rate_bar(df_rate: pd.DataFrame) -> pd.DataFrame:
"""Return closed bars from chronologically ordered MT5 rate data.
MetaTrader 5 ``copy_rates_from_pos(start_pos=0)`` includes the still-forming
current bar as the last row. Slice it off so downstream logic only sees
completed bars. Empty frames and single-row frames return empty results.
Args:
df_rate: Rate data ordered oldest-to-newest with the forming bar last.
Returns:
A new DataFrame with all rows except the last. Index and columns are
preserved. The input frame is not modified.
"""
return df_rate.iloc[:-1].copy()
def build_rate_view_name(
*,
symbol: str,
@@ -122,9 +141,641 @@ def build_rate_view_name(
return f"rate_{symbol}__{granularity}_{timeframe}"
SqliteConnOrPath = sqlite3.Connection | Path | str
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
value = identifier.strip()
if not value:
msg = f"SQLite {kind} name must not be empty."
raise ValueError(msg)
return value
def _open_history_connection(
conn_or_path: SqliteConnOrPath | None,
) -> tuple[sqlite3.Connection | None, bool]:
"""Open a read-only SQLite connection when given a path.
Returns:
A connection and whether the caller should close it. When ``conn_or_path``
is None or the path does not exist, returns ``(None, False)`` without
creating a database file.
"""
if conn_or_path is None:
return None, False
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
return None, False
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _open_existing_sqlite_database(
conn_or_path: SqliteConnOrPath,
) -> tuple[sqlite3.Connection, bool]:
"""Open a read-only SQLite database or reuse an existing connection.
Returns:
Tuple of connection and whether the caller should close it.
Raises:
ValueError: If the database path does not exist or is not a file.
"""
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
if not path.is_file():
msg = f"SQLite database path is not a file: {path}"
raise ValueError(msg)
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _validate_rate_load_request(table: str, count: int | None) -> str:
table_name = _require_non_empty_identifier(table, "table or view")
if count is not None and count <= 0:
msg = "count must be positive when provided."
raise ValueError(msg)
return table_name
def _ensure_rate_columns(columns: set[str], table: str) -> None:
if not columns:
msg = f"SQLite table or view not found: {table}"
raise ValueError(msg)
if "time" not in columns:
msg = f"SQLite table or view {table!r} must include a time column."
raise ValueError(msg)
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
msg = (
f"SQLite table or view {table!r} must include close, "
"or both ask and bid columns."
)
raise ValueError(msg)
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
parsed = frame["time"].map(parse_sqlite_timestamp)
if parsed.isna().any():
msg = f"SQLite table or view {table!r} contains unparsable time values."
raise ValueError(msg)
result = frame.drop(columns=["time"])
result.index = pd.DatetimeIndex(parsed, name="time")
return result.sort_index(kind="stable")
def load_rate_data_from_connection(
connection: sqlite3.Connection,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite table or view.
Args:
connection: Open SQLite connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
Raises:
ValueError: If inputs, schema, timestamps are invalid, or the table
or view contains no rows.
"""
table_name = _validate_rate_load_request(table, count)
columns = get_table_columns(connection, table_name)
_ensure_rate_columns(columns, table_name)
quoted_table = quote_sqlite_identifier(table_name)
if count is None:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
connection,
),
)
else:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
connection,
params=(count,),
),
)
if frame.empty:
msg = f"SQLite table or view {table_name!r} contains no rows."
raise ValueError(msg)
return _parse_rate_time_index(frame, table_name)
def load_rate_data(
conn_or_path: SqliteConnOrPath,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite database path or connection.
Args:
conn_or_path: SQLite database path or open connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
"""
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
return load_rate_data_from_connection(conn, table, count=count)
finally:
if should_close:
conn.close()
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
"""Return distinct timeframe counts per symbol from the normalized rates table."""
columns = get_table_columns(conn, Dataset.rates.table_name)
if not {"symbol", "timeframe"}.issubset(columns):
return None
rows = conn.execute(
"SELECT symbol, COUNT(DISTINCT timeframe) FROM rates GROUP BY symbol",
).fetchall()
return {str(symbol): int(count) for symbol, count in rows}
def _load_existing_rate_views(conn: sqlite3.Connection) -> set[str]:
"""Return mt5cli-managed ``rate_*__*`` compatibility view names."""
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'view' AND name GLOB 'rate_*__*'",
).fetchall()
return {str(row[0]) for row in rows}
def _rate_view_name_candidates(
*,
symbol: str,
granularity: str,
granularity_count: int,
timeframe: int,
) -> list[str]:
"""Return candidate view names in preference order."""
single = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=1,
timeframe=timeframe,
)
if granularity_count <= 1:
return [single]
multi = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=granularity_count,
timeframe=timeframe,
)
return [multi, single]
def _resolve_rate_view_name_from_context(
*,
symbol: str,
timeframe: int,
granularity_name: str,
timeframe_counts: dict[str, int] | None,
existing_views: set[str],
require_existing: bool = False,
) -> str:
"""Resolve one rate view name using preloaded SQLite metadata.
Returns:
Preferred mt5cli-managed rate compatibility view name.
Raises:
ValueError: If ``require_existing`` is True and no managed view exists.
"""
if timeframe_counts is None or symbol not in timeframe_counts:
candidates = [
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
),
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=2,
timeframe=timeframe,
),
]
else:
candidates = _rate_view_name_candidates(
symbol=symbol,
granularity=granularity_name,
granularity_count=timeframe_counts[symbol],
timeframe=timeframe,
)
for candidate in candidates:
if candidate in existing_views:
return candidate
if require_existing:
msg = (
f"No rate compatibility view exists for symbol {symbol!r} "
f"and granularity {granularity_name!r}; "
f"candidates: {', '.join(candidates)}."
)
raise ValueError(msg)
return candidates[0]
def resolve_rate_view_name(
conn_or_path: SqliteConnOrPath | None,
symbol: str,
granularity: str,
*,
require_existing: bool = False,
) -> str:
"""Resolve the mt5cli-managed rate compatibility view name.
Args:
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, the deterministic
default view name is returned without creating a database file.
symbol: Symbol stored in the normalized ``rates`` table.
granularity: Timeframe name (for example ``M1``) or integer string.
require_existing: When True, require the database and a managed view to exist.
Returns:
View name such as ``rate_EURUSD__1`` or ``rate_EURUSD__M1_1``.
Raises:
ValueError: If ``require_existing`` is True and the database or view is missing.
"""
timeframe = parse_timeframe(granularity)
granularity_name = resolve_granularity_name(timeframe)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
return build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
)
return _resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=granularity_name,
timeframe_counts=_load_rates_timeframe_counts(conn),
existing_views=_load_existing_rate_views(conn),
require_existing=require_existing,
)
finally:
if should_close and conn is not None:
conn.close()
def resolve_rate_view_names(
conn_or_path: SqliteConnOrPath | None,
symbols: Sequence[str],
granularities: Sequence[str],
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve rate compatibility view names for symbol and granularity pairs.
Args:
conn_or_path: SQLite database path or open connection. When None or a
non-existing path and ``require_existing`` is False, deterministic
default view names are returned without creating a database file.
symbols: Symbols stored in the normalized ``rates`` table.
granularities: Timeframe names (for example ``M1``) or integer strings.
require_existing: When True, require the database and managed views to exist.
Returns:
View names in row-major order: every ``granularity`` for the first
symbol, then every granularity for the next symbol, and so on.
"""
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
return [
resolve_rate_view_name(
conn_or_path,
symbol,
granularity,
require_existing=require_existing,
)
for symbol in symbols
for granularity in granularities
]
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for symbol in symbols:
for granularity in granularities:
timeframe = parse_timeframe(granularity)
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
@dataclass(frozen=True)
class RateTarget:
"""A single rate series identified by symbol and timeframe.
Attributes:
symbol: MT5 symbol name, or None when the rate series is addressed only
by an explicit table (for example a custom SQLite view).
timeframe: MT5 timeframe as an integer or name (for example ``M1``).
"""
symbol: str | None
timeframe: int | str
def __post_init__(self) -> None:
"""Normalize accepted timeframe aliases to the stored integer value."""
if not isinstance(self.timeframe, int):
object.__setattr__(self, "timeframe", parse_timeframe(self.timeframe))
@property
def timeframe_int(self) -> int:
"""Return the timeframe as its integer MT5 value."""
return cast("int", self.timeframe)
def build_rate_targets(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
allow_missing_symbol: bool = False,
) -> list[RateTarget]:
"""Build rate targets for every symbol and timeframe combination.
Args:
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
timeframes: MT5 timeframes as integers or names (for example ``M1``).
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each timeframe instead of raising.
Returns:
Targets in row-major order: every timeframe for the first symbol, then
every timeframe for the next symbol, and so on.
Raises:
ValueError: If ``timeframes`` is empty, or ``symbols`` is empty and
``allow_missing_symbol`` is False.
"""
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
if not symbols:
if not allow_missing_symbol:
msg = "At least one symbol is required."
raise ValueError(msg)
return [RateTarget(symbol=None, timeframe=tf) for tf in timeframes]
return [
RateTarget(symbol=symbol, timeframe=tf)
for symbol in symbols
for tf in timeframes
]
def resolve_rate_tables(
conn_or_path: SqliteConnOrPath | None,
targets: Sequence[RateTarget],
explicit_tables: Sequence[str] | None = None,
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve SQLite table or view names for rate targets.
Args:
conn_or_path: SQLite database path or open connection. May be None when
``explicit_tables`` is provided, or when ``require_existing`` is
False and deterministic default view names are sufficient.
targets: Rate targets to resolve.
explicit_tables: Optional explicit table or view names. When provided,
they are used as-is and must match the number of targets.
require_existing: When True, require the database and managed views to
exist for each symbol target. Ignored when ``explicit_tables`` is
provided.
Returns:
Table or view names aligned with ``targets``.
Raises:
ValueError: If ``targets`` is empty, ``explicit_tables`` length does not
match the target count, a target without a symbol is resolved
without an explicit table, or ``require_existing`` is True and the
database or a managed view is missing.
"""
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is not None:
tables = list(explicit_tables)
if len(tables) != len(target_list):
msg = (
f"Expected {len(target_list)} explicit table(s) "
f"to match the targets, got {len(tables)}."
)
raise ValueError(msg)
return tables
if any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
timeframe_counts = None
existing_views: set[str] = set()
else:
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for target in target_list:
symbol = cast("str", target.symbol)
timeframe = target.timeframe_int
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
def load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget],
count: int,
explicit_tables: Sequence[str] | None = None,
) -> dict[tuple[str | None, int], pd.DataFrame]:
"""Load multiple rate series from a SQLite database.
Args:
conn_or_path: SQLite database path or open connection.
targets: Rate targets to load. Each ``(symbol, timeframe_int)`` pair
must be unique.
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching targets.
When omitted, managed ``rate_*`` compatibility views must already
exist in the database.
Returns:
Mapping keyed by ``(symbol, timeframe_int)`` to each rate DataFrame.
Raises:
ValueError: If ``count`` is not positive, targets are empty, duplicate
``(symbol, timeframe_int)`` pairs are present, or table resolution
fails.
"""
if count <= 0:
msg = "count must be positive."
raise ValueError(msg)
target_list = list(targets)
if not target_list:
msg = "At least one rate target is required."
raise ValueError(msg)
if explicit_tables is None and any(target.symbol is None for target in target_list):
msg = (
"Cannot resolve a rate table for a target without a symbol; "
"provide explicit_tables."
)
raise ValueError(msg)
seen_keys: set[tuple[str | None, int]] = set()
for target in target_list:
key = (target.symbol, target.timeframe_int)
if key in seen_keys:
symbol_repr = repr(target.symbol)
msg = f"Duplicate rate target: ({symbol_repr}, {target.timeframe_int})"
raise ValueError(msg)
seen_keys.add(key)
tables = (
resolve_rate_tables(None, target_list, explicit_tables)
if explicit_tables is not None
else None
)
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
resolved_tables = tables or resolve_rate_tables(
conn,
target_list,
require_existing=True,
)
return {
(target.symbol, target.timeframe_int): load_rate_data_from_connection(
conn,
table,
count=count,
)
for target, table in zip(target_list, resolved_tables, strict=True)
}
finally:
if should_close:
conn.close()
def load_rate_series_by_granularity(
conn_or_path: SqliteConnOrPath,
symbols: Sequence[str],
granularities: Sequence[int | str],
count: int,
*,
explicit_tables: Sequence[str] | None = None,
allow_missing_symbol: bool = False,
) -> dict[tuple[str | None, str], pd.DataFrame]:
"""Load rate series keyed by symbol and string granularity name.
Builds targets with :func:`build_rate_targets` and loads them with
:func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
name (for example ``M1``) instead of the integer timeframe to reduce
downstream boilerplate.
Args:
conn_or_path: SQLite database path or open connection.
symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
granularities: MT5 timeframes as integers or names (for example ``M1``).
count: Number of most recent rows to load per series.
explicit_tables: Optional explicit table or view names matching the
built targets in row-major order. Required when symbols are omitted.
allow_missing_symbol: When True and ``symbols`` is empty, build targets
with ``symbol=None`` for each granularity instead of raising.
Returns:
Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
:func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
table resolution fails, or duplicate targets are present.
"""
targets = build_rate_targets(
symbols,
granularities,
allow_missing_symbol=allow_missing_symbol,
)
series = load_rate_series_from_sqlite(
conn_or_path,
targets,
count,
explicit_tables=explicit_tables,
)
return {
(symbol, resolve_granularity_name(timeframe)): frame
for (symbol, timeframe), frame in series.items()
}
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
"""Return existing SQLite columns for a table."""
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
quoted_table = quote_sqlite_identifier(table)
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
return {str(row[1]) for row in rows}
@@ -404,7 +1055,20 @@ def drop_duplicates_in_table(
)
DedupScope = tuple[str, tuple[object, ...]]
@dataclass(frozen=True)
class DedupScope:
"""Scoped deduplication predicate and the columns it references.
Attributes:
where: SQL predicate appended to the duplicate-removal query.
params: Parameters bound to the scope predicate.
required_columns: Columns that must be present in the written table for
the scope to run.
"""
where: str
params: tuple[object, ...]
required_columns: frozenset[str]
def _record_dedup_scope(
@@ -412,17 +1076,25 @@ def _record_dedup_scope(
dataset: Dataset,
scope_where: str,
scope_params: tuple[object, ...],
required_columns: frozenset[str],
) -> None:
dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
dedup_scopes.setdefault(dataset, []).append(
DedupScope(scope_where, scope_params, required_columns),
)
def deduplicate_history_tables(
conn: sqlite3.Connection,
written_columns: dict[Dataset, set[str]],
written_tables: set[Dataset],
dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
) -> None:
"""Deduplicate appended history tables by stable identifiers."""
"""Deduplicate appended history tables by stable identifiers.
Scopes whose required columns are not present in the written table are
skipped. If all scopes for a dataset are skipped, the table receives one
unscoped deduplication pass instead.
"""
cursor = conn.cursor()
for dataset in written_tables:
columns = written_columns.get(dataset, set())
@@ -441,16 +1113,19 @@ def deduplicate_history_tables(
table,
)
continue
scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
raw_scopes: Sequence[DedupScope] = (
dedup_scopes.get(dataset, ()) if dedup_scopes else ()
)
scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
if scopes:
for scope_where, scope_params in scopes:
for scope in scopes:
drop_duplicates_in_table(
cursor,
table,
list(keys),
keep="last",
scope_where=scope_where,
scope_params=scope_params,
scope_where=scope.where,
scope_params=scope.params,
)
continue
drop_duplicates_in_table(cursor, table, list(keys), keep="last")
@@ -817,6 +1492,7 @@ def _write_incremental_rates(
Dataset.rates,
"symbol = ? AND timeframe = ? AND time >= ?",
(symbol, timeframe, start_date),
frozenset({"symbol", "timeframe", "time"}),
)
@@ -855,6 +1531,7 @@ def _write_incremental_ticks(
Dataset.ticks,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -893,6 +1570,7 @@ def _write_incremental_history_orders(
Dataset.history_orders,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
@@ -946,6 +1624,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_by_symbol[symbol, None]),
frozenset({"symbol", "time"}),
)
if "type" in columns:
_record_dedup_scope(
@@ -953,6 +1632,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
(account_event_start,),
frozenset({"type", "time"}),
)
if "type" not in columns and "symbol" in columns:
_record_dedup_scope(
@@ -960,6 +1640,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"(symbol IS NULL OR symbol = '') AND time >= ?",
(account_event_start,),
frozenset({"symbol", "time"}),
)
return
start_by_symbol = load_incremental_start_datetimes(
@@ -987,6 +1668,7 @@ def _write_incremental_history_deals(
Dataset.history_deals,
"symbol = ? AND time >= ?",
(symbol, start_date),
frozenset({"symbol", "time"}),
)
+1059 -8
View File
File diff suppressed because it is too large Load Diff
+210
View File
@@ -0,0 +1,210 @@
"""Trading-capable MetaTrader 5 session helpers and operational utilities."""
from __future__ import annotations
from contextlib import contextmanager
from typing import TYPE_CHECKING, Literal
from pdmt5 import Mt5Config, Mt5TradingClient
from .sdk import build_config
if TYPE_CHECKING:
from collections.abc import Iterator
import pandas as pd
PositionSide = Literal["long", "short"]
OrderSide = Literal["long", "short"]
__all__ = [
"OrderSide",
"PositionSide",
"calculate_margin_and_volume",
"detect_position_side",
"determine_order_limits",
"mt5_trading_session",
]
def _require_unit_ratio(value: float, name: str) -> None:
if not 0.0 <= value <= 1.0:
msg = f"{name} must be between 0 and 1 inclusive."
raise ValueError(msg)
def _require_protective_ratio(value: float, name: str) -> None:
if not 0.0 <= value < 1.0:
msg = f"{name} must be at least 0 and less than 1."
raise ValueError(msg)
def _sum_position_volume(positions: pd.DataFrame, position_type: object) -> float:
matched = positions.loc[positions["type"] == position_type, "volume"]
if matched.empty:
return 0.0
return float(matched.to_numpy(dtype=float).sum())
def _normalize_order_side(side: str) -> OrderSide:
normalized = side.lower()
if normalized in {"long", "buy"}:
return "long"
if normalized in {"short", "sell"}:
return "short"
msg = (
f"Unsupported order side: {side!r}. Expected 'long', 'short', 'buy', or 'sell'."
)
raise ValueError(msg)
def detect_position_side(
client: Mt5TradingClient,
symbol: str,
) -> PositionSide | None:
"""Detect the net open position side for a symbol.
Args:
client: Connected ``Mt5TradingClient`` instance.
symbol: Symbol to inspect.
Returns:
``"long"`` when net buy volume exceeds sell volume, ``"short"`` when
net sell volume exceeds buy volume, or ``None`` when no positions exist
or buy/sell volumes are exactly balanced.
"""
positions = client.positions_get_as_df(symbol=symbol)
if positions.empty:
return None
buy_type = client.mt5.POSITION_TYPE_BUY
sell_type = client.mt5.POSITION_TYPE_SELL
buy_volume = _sum_position_volume(positions, buy_type)
sell_volume = _sum_position_volume(positions, sell_type)
net_volume = buy_volume - sell_volume
if net_volume > 0:
return "long"
if net_volume < 0:
return "short"
return None
def calculate_margin_and_volume(
client: Mt5TradingClient,
symbol: str,
unit_margin_ratio: float,
preserved_margin_ratio: float,
) -> dict[str, float]:
"""Calculate tradable margin and volumes from account free margin.
Applies ``preserved_margin_ratio`` to keep a reserve off ``margin_free``,
then allocates ``unit_margin_ratio`` of the remainder as the margin budget
for volume sizing on both buy and sell sides.
Args:
client: Connected ``Mt5TradingClient`` instance.
symbol: Symbol used for minimum-lot margin and volume calculations.
unit_margin_ratio: Fraction of post-reserve margin to allocate per unit.
preserved_margin_ratio: Fraction of ``margin_free`` to preserve.
Returns:
Dictionary with ``margin_free``, ``available_margin``, ``trade_margin``,
``buy_volume``, and ``sell_volume``. Negative ``margin_free`` values are
clamped to ``0.0`` before sizing.
"""
_require_unit_ratio(unit_margin_ratio, "unit_margin_ratio")
_require_unit_ratio(preserved_margin_ratio, "preserved_margin_ratio")
account = client.account_info_as_dict()
margin_free = max(0.0, float(account.get("margin_free") or 0.0))
available_margin = margin_free * (1.0 - preserved_margin_ratio)
trade_margin = available_margin * unit_margin_ratio
buy_volume = client.calculate_volume_by_margin(symbol, trade_margin, "BUY")
sell_volume = client.calculate_volume_by_margin(symbol, trade_margin, "SELL")
return {
"margin_free": margin_free,
"available_margin": available_margin,
"trade_margin": trade_margin,
"buy_volume": buy_volume,
"sell_volume": sell_volume,
}
def determine_order_limits(
client: Mt5TradingClient,
symbol: str,
side: OrderSide | str,
stop_loss_limit_ratio: float,
take_profit_limit_ratio: float,
) -> dict[str, float | None]:
"""Derive entry and protective order prices from current market quotes.
Args:
client: Connected ``Mt5TradingClient`` instance.
symbol: Symbol used for the quote lookup.
side: Position side as ``"long"``/``"short"`` (``"buy"``/``"sell"``
aliases are accepted).
stop_loss_limit_ratio: Relative distance from entry for stop loss in
``[0, 1)``. A value of ``0`` omits the stop loss.
take_profit_limit_ratio: Relative distance from entry for take profit in
``[0, 1)``. A value of ``0`` omits the take profit.
Returns:
Dictionary with ``entry``, ``stop_loss``, and ``take_profit`` keys.
Omitted protective levels are returned as ``None``.
"""
_require_protective_ratio(stop_loss_limit_ratio, "stop_loss_limit_ratio")
_require_protective_ratio(take_profit_limit_ratio, "take_profit_limit_ratio")
normalized_side = _normalize_order_side(side)
tick = client.symbol_info_tick_as_dict(symbol=symbol)
entry = float(tick["ask"] if normalized_side == "long" else tick["bid"])
stop_loss: float | None = None
if stop_loss_limit_ratio > 0:
if normalized_side == "long":
stop_loss = entry * (1.0 - stop_loss_limit_ratio)
else:
stop_loss = entry * (1.0 + stop_loss_limit_ratio)
take_profit: float | None = None
if take_profit_limit_ratio > 0:
if normalized_side == "long":
take_profit = entry * (1.0 + take_profit_limit_ratio)
else:
take_profit = entry * (1.0 - take_profit_limit_ratio)
return {
"entry": entry,
"stop_loss": stop_loss,
"take_profit": take_profit,
}
@contextmanager
def mt5_trading_session(
config: Mt5Config | None = None,
retry_count: int = 0,
) -> Iterator[Mt5TradingClient]:
"""Open a trading-capable MT5 session and always shut down safely.
Launches the MetaTrader 5 terminal using ``Mt5Config.path`` when set,
initializes and logs in via ``initialize_and_login_mt5()``, yields a
connected :class:`~pdmt5.Mt5TradingClient`, and calls ``shutdown()`` on
exit even when an error is raised inside the context.
Args:
config: MT5 connection configuration. Defaults to an empty config that
attaches to a running terminal.
retry_count: Number of initialization retries passed to
``Mt5TradingClient``.
Yields:
Connected ``Mt5TradingClient`` bound to the session.
"""
mt5_config = config or build_config()
client = Mt5TradingClient(config=mt5_config, retry_count=retry_count)
try:
client.initialize_and_login_mt5()
yield client
finally:
client.shutdown()
+55 -10
View File
@@ -2,16 +2,18 @@
from __future__ import annotations
import importlib
import json
import sqlite3
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard, cast
from typing import TYPE_CHECKING, Any, TypeGuard
import click
if TYPE_CHECKING:
from collections.abc import Sequence
import pandas as pd
# ---------------------------------------------------------------------------
@@ -260,6 +262,50 @@ def detect_format(
raise ValueError(msg)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
table_name: str = "data",
*,
if_exists: IfExists = IfExists.APPEND,
index: bool = False,
index_label: str | None = None,
deduplicate_on: Sequence[str] | None = None,
) -> None:
"""Write a DataFrame to SQLite with configurable append and deduplication.
Args:
df: DataFrame to export.
output_path: SQLite database path.
table_name: Target table name.
if_exists: Conflict behavior when the table already exists.
index: Whether to write the DataFrame index as a column.
index_label: Column name for the index when ``index=True``.
deduplicate_on: Optional key columns to deduplicate after writing,
keeping the latest ``ROWID`` per key group. Deduplication scans the
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=index,
index_label=index_label,
)
if deduplicate_on:
from .history import drop_duplicates_in_table # noqa: PLC0415
drop_duplicates_in_table(
conn.cursor(),
table_name,
list(deduplicate_on),
keep="last",
)
conn.commit()
def export_dataframe(
df: pd.DataFrame,
output_path: Path,
@@ -289,14 +335,13 @@ def export_dataframe(
elif output_format == "parquet":
df.to_parquet(output_path, index=False)
elif output_format == "sqlite3":
sqlite3 = cast("Any", importlib.import_module("sqlite3"))
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists="replace",
index=False,
)
export_dataframe_to_sqlite(
df,
output_path,
table_name,
if_exists=IfExists.REPLACE,
index=False,
)
else:
msg = f"Unsupported output format: {output_format}"
raise ValueError(msg)
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.2"
version = "0.6.1"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
+173 -1
View File
@@ -6,7 +6,7 @@ import json
import logging
import re
import sqlite3
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50,
)
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range(
self,
tmp_path: Path,
@@ -316,6 +351,65 @@ class TestCommands:
flags=2,
)
def test_ticks_recent(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test ticks-recent command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"ticks-recent",
"--symbol",
"EURUSD",
"--seconds",
"120",
"--date-to",
"2024-01-02",
"--count",
"500",
"--flags",
"ALL",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
def test_minimum_margins(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test minimum-margins command."""
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
mock_client.symbol_info.return_value = sym
mock_client.account_info.return_value = account
mock_client.symbol_info_tick.return_value = tick
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
mock_client.mt5.ORDER_TYPE_BUY = 0
mock_client.mt5.ORDER_TYPE_SELL = 1
output = tmp_path / "out.csv"
result = runner.invoke(
app,
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
)
assert result.exit_code == 0, result.output
mock_client.symbol_info.assert_called_once_with("EURUSD")
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
def test_orders(
self,
tmp_path: Path,
@@ -392,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version(
self,
tmp_path: Path,
+971 -3
View File
File diff suppressed because it is too large Load Diff
+1266 -5
View File
File diff suppressed because it is too large Load Diff
+356
View File
@@ -0,0 +1,356 @@
"""Tests for trading session helpers and operational utilities."""
from __future__ import annotations
from unittest.mock import MagicMock
import pandas as pd
import pytest
from pdmt5 import Mt5RuntimeError
from pytest_mock import MockerFixture # noqa: TC002
from mt5cli.sdk import build_config
from mt5cli.trading import (
calculate_margin_and_volume,
detect_position_side,
determine_order_limits,
mt5_trading_session,
)
class TestDetectPositionSide:
"""Tests for detect_position_side."""
def test_returns_none_when_no_positions(self) -> None:
"""Test None is returned when no open positions exist."""
client = MagicMock()
client.positions_get_as_df.return_value = pd.DataFrame()
assert detect_position_side(client, "EURUSD") is None
def test_returns_long_for_net_buy_volume(self) -> None:
"""Test long is returned when buy volume exceeds sell volume."""
client = MagicMock()
client.mt5.POSITION_TYPE_BUY = 0
client.mt5.POSITION_TYPE_SELL = 1
client.positions_get_as_df.return_value = pd.DataFrame(
{
"type": [0, 0, 1],
"volume": [0.2, 0.1, 0.05],
},
)
assert detect_position_side(client, "EURUSD") == "long"
def test_returns_short_for_net_sell_volume(self) -> None:
"""Test short is returned when sell volume exceeds buy volume."""
client = MagicMock()
client.mt5.POSITION_TYPE_BUY = 0
client.mt5.POSITION_TYPE_SELL = 1
client.positions_get_as_df.return_value = pd.DataFrame(
{
"type": [1, 1],
"volume": [0.3, 0.1],
},
)
assert detect_position_side(client, "EURUSD") == "short"
def test_returns_none_for_balanced_hedged_positions(self) -> None:
"""Test None is returned when buy and sell volumes net to zero."""
client = MagicMock()
client.mt5.POSITION_TYPE_BUY = 0
client.mt5.POSITION_TYPE_SELL = 1
client.positions_get_as_df.return_value = pd.DataFrame(
{
"type": [0, 1],
"volume": [0.2, 0.2],
},
)
assert detect_position_side(client, "EURUSD") is None
class TestCalculateMarginAndVolume:
"""Tests for calculate_margin_and_volume."""
def test_calculates_margin_budget_and_volumes(self) -> None:
"""Test margin budget and buy/sell volumes are derived from ratios."""
client = MagicMock()
client.account_info_as_dict.return_value = {"margin_free": 1000.0}
client.calculate_volume_by_margin.side_effect = [0.3, 0.2]
result = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
assert result == {
"margin_free": 1000.0,
"available_margin": 800.0,
"trade_margin": 400.0,
"buy_volume": 0.3,
"sell_volume": 0.2,
}
client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "BUY")
client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "SELL")
@pytest.mark.parametrize(
("account_dict", "expected_margin_free"),
[
({"margin_free": 0.0}, 0.0),
({}, 0.0),
({"margin_free": None}, 0.0),
],
)
def test_zero_or_missing_margin_free(
self,
account_dict: dict[str, float | None],
expected_margin_free: float,
) -> None:
"""Test missing or zero margin_free yields zero trade margin."""
client = MagicMock()
client.account_info_as_dict.return_value = account_dict
client.calculate_volume_by_margin.return_value = 0.0
result = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
assert result["margin_free"] == expected_margin_free
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
def test_clamps_negative_margin_free_to_zero(self) -> None:
"""Test negative margin_free is clamped to zero before sizing."""
client = MagicMock()
client.account_info_as_dict.return_value = {"margin_free": -500.0}
client.calculate_volume_by_margin.return_value = 0.0
result = calculate_margin_and_volume(
client,
"EURUSD",
unit_margin_ratio=0.5,
preserved_margin_ratio=0.2,
)
expected_margin_free = 0.0
assert result["margin_free"] == expected_margin_free
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
@pytest.mark.parametrize(
("unit_ratio", "preserved_ratio"),
[
(-0.1, 0.0),
(1.1, 0.0),
(0.5, -0.1),
(0.5, 1.1),
],
)
def test_rejects_invalid_ratios(
self,
unit_ratio: float,
preserved_ratio: float,
) -> None:
"""Test invalid ratio values raise ValueError."""
with pytest.raises(ValueError, match="must be between 0 and 1"):
calculate_margin_and_volume(
MagicMock(),
"EURUSD",
unit_margin_ratio=unit_ratio,
preserved_margin_ratio=preserved_ratio,
)
class TestDetermineOrderLimits:
"""Tests for determine_order_limits."""
@pytest.mark.parametrize(
("side", "expected_entry_key"),
[
("long", "ask"),
("short", "bid"),
("buy", "ask"),
("sell", "bid"),
],
)
def test_uses_expected_quote_for_entry(
self,
side: str,
expected_entry_key: str,
) -> None:
"""Test entry price is taken from ask for long/buy and bid for short/sell."""
client = MagicMock()
client.symbol_info_tick_as_dict.return_value = {"ask": 1.1010, "bid": 1.1000}
result = determine_order_limits(
client,
"EURUSD",
side,
stop_loss_limit_ratio=0.0,
take_profit_limit_ratio=0.0,
)
assert (
result["entry"]
== client.symbol_info_tick_as_dict.return_value[expected_entry_key]
)
assert result["stop_loss"] is None
assert result["take_profit"] is None
def test_calculates_long_protective_levels(self) -> None:
"""Test long stop loss and take profit are placed below/above entry."""
client = MagicMock()
client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
result = determine_order_limits(
client,
"EURUSD",
"long",
stop_loss_limit_ratio=0.02,
take_profit_limit_ratio=0.03,
)
assert result == {
"entry": 100.0,
"stop_loss": 98.0,
"take_profit": 103.0,
}
def test_calculates_short_protective_levels(self) -> None:
"""Test short stop loss and take profit are placed above/below entry."""
client = MagicMock()
client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
result = determine_order_limits(
client,
"EURUSD",
"short",
stop_loss_limit_ratio=0.02,
take_profit_limit_ratio=0.03,
)
assert result == {
"entry": 99.0,
"stop_loss": 100.98,
"take_profit": 96.03,
}
def test_rejects_unknown_side(self) -> None:
"""Test unsupported side values raise ValueError."""
with pytest.raises(ValueError, match="Unsupported order side"):
determine_order_limits(
MagicMock(),
"EURUSD",
"flat",
stop_loss_limit_ratio=0.01,
take_profit_limit_ratio=0.01,
)
@pytest.mark.parametrize(
("stop_loss_ratio", "take_profit_ratio"),
[
(-0.05, 0.01),
(0.01, 2.0),
],
)
def test_rejects_invalid_protective_ratios(
self,
stop_loss_ratio: float,
take_profit_ratio: float,
) -> None:
"""Test out-of-range protective ratios raise ValueError."""
with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
determine_order_limits(
MagicMock(),
"EURUSD",
"long",
stop_loss_limit_ratio=stop_loss_ratio,
take_profit_limit_ratio=take_profit_ratio,
)
@pytest.mark.parametrize(
("field", "ratio"),
[
("stop_loss_limit_ratio", 1.0),
("take_profit_limit_ratio", 1.0),
],
)
def test_rejects_unit_boundary_protective_ratios(
self,
field: str,
ratio: float,
) -> None:
"""Test protective ratios of exactly 1.0 are rejected."""
kwargs = {
"stop_loss_limit_ratio": 0.01,
"take_profit_limit_ratio": 0.01,
field: ratio,
}
with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
determine_order_limits(
MagicMock(),
"EURUSD",
"long",
**kwargs,
)
class TestMt5TradingSession:
"""Tests for the mt5_trading_session context manager."""
def test_yields_connected_client_and_shuts_down(
self,
mocker: MockerFixture,
) -> None:
"""Test mt5_trading_session connects, yields a client, and shuts down."""
mock_client = MagicMock()
trading_client = mocker.patch(
"mt5cli.trading.Mt5TradingClient",
return_value=mock_client,
)
with mt5_trading_session(
build_config(path="/opt/mt5/terminal64.exe"),
retry_count=2,
) as client:
mock_client.initialize_and_login_mt5.assert_called_once()
assert client is mock_client
trading_client.assert_called_once()
assert trading_client.call_args.kwargs["retry_count"] == 2
assert (
trading_client.call_args.kwargs["config"].path == "/opt/mt5/terminal64.exe"
)
mock_client.shutdown.assert_called_once()
def test_shuts_down_when_initialize_raises(
self,
mocker: MockerFixture,
) -> None:
"""Test shutdown is called when initialization fails."""
mock_client = MagicMock()
mock_client.initialize_and_login_mt5.side_effect = Mt5RuntimeError("boom")
mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
with pytest.raises(Mt5RuntimeError, match="boom"), mt5_trading_session():
pass
mock_client.shutdown.assert_called_once()
def test_shuts_down_when_body_raises(self, mocker: MockerFixture) -> None:
"""Test shutdown is called when the context body raises."""
mock_client = MagicMock()
mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
body_error = "body error"
with pytest.raises(RuntimeError, match=body_error), mt5_trading_session():
raise RuntimeError(body_error)
mock_client.shutdown.assert_called_once()
+108
View File
@@ -21,8 +21,10 @@ from mt5cli.utils import (
TIMEFRAME_MAP,
TIMEFRAME_TYPE,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_request,
parse_tick_flags,
@@ -130,6 +132,112 @@ class TestExportDataframe:
export_dataframe(sample_df, tmp_path / "out.txt", "xml")
class TestExportDataframeToSqlite:
"""Tests for export_dataframe_to_sqlite."""
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
"""Test append mode keeps prior rows in the SQLite table."""
output = tmp_path / "append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
"""Test deduplication keeps the latest ROWID for key columns."""
output = tmp_path / "dedup.db"
first = pd.DataFrame({
"symbol": ["EURUSD", "EURUSD"],
"time": ["2024-01-01", "2024-01-01"],
"bid": [1.0, 1.1],
})
second = pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
})
export_dataframe_to_sqlite(
first,
output,
"ticks",
if_exists=IfExists.REPLACE,
deduplicate_on=("symbol", "time"),
)
export_dataframe_to_sqlite(
second,
output,
"ticks",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "time"),
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, time, bid FROM ticks",
conn,
)
pd.testing.assert_frame_equal(
result.reset_index(drop=True),
pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
}),
)
def test_default_if_exists_appends_without_dropping_rows(
self,
tmp_path: Path,
) -> None:
"""Test the default append mode keeps prior rows."""
output = tmp_path / "default-append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items")
export_dataframe_to_sqlite(second, output, "items")
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_writes_index_with_label(self, tmp_path: Path) -> None:
"""Test optional index export with a custom label."""
output = tmp_path / "index.db"
frame = pd.DataFrame(
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
)
export_dataframe_to_sqlite(
frame,
output,
"margins",
if_exists=IfExists.REPLACE,
index=True,
index_label="symbol",
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, value FROM margins",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
)
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
Generated
+4 -4
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.2"
version = "0.6.1"
source = { editable = "." }
dependencies = [
{ name = "click" },
@@ -836,11 +836,11 @@ wheels = [
[[package]]
name = "pygments"
version = "2.19.2"
version = "2.20.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
]
[[package]]