Compare commits

...

3 Commits

Author SHA1 Message Date
Daichi Narushima 9957b0a1de [codex] Add generic MT5 SDK and SQLite rate loader (#19)
* Add generic MT5 SDK and SQLite rate loader

* Fix MT5 latest rates connection reuse

* Make MT5 summary export safe

* Address PR review feedback for SDK and SQLite rate loader.

Reuse parse_sqlite_timestamp for rate time parsing, document empty-table
errors, tighten tests, and align docs with require_existing=True.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 11:27:29 +09:00
Daichi Narushima b2bb2ad0a0 Add rate view resolution and downstream SDK helpers (#18)
* Add public helpers to resolve rate compatibility view names.

Expose resolve_rate_view_name and resolve_rate_view_names in mt5cli.history so consumers can derive mt5cli-managed SQLite view names from stored rates metadata without reimplementing the naming rules.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add reusable export, tick-window, and margin helpers for downstream tools.

Expose SQLite append/dedup export, recent tick retrieval, and minimum margin
summary through the SDK and CLI so projects like mteor can depend on mt5cli
instead of duplicating MT5 data plumbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump version to 0.4.3.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for rate view resolution and SDK helpers.

Harden SQLite read-only connections, tighten view discovery, improve recent_ticks
fetch efficiency, default SQLite export to append, and expand tests and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix read-only SQLite URI construction on Windows.

Use Path.as_uri() so encoded file URIs work cross-platform with mode=ro.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 03:29:03 +09:00
Daichi Narushima 756faf747b Rename sqlite_history module to history (#17)
* Rename sqlite_history module to history.

Drop the sqlite-specific prefix now that history collection is the primary module name across SDK, tests, and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for history module rename.

Add a sqlite_history compatibility shim, clarify docs naming, and align the
module docstring with the collect-history SQLite scope.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove sqlite_history compatibility shim.

The rename to mt5cli.history is intentionally breaking; downstream code
should update imports rather than rely on a deprecated re-export path.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-09 02:40:25 +09:00
16 changed files with 2262 additions and 73 deletions
+32 -22
View File
@@ -13,6 +13,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -50,28 +51,33 @@ python -m mt5cli -o account.csv account-info
## Commands
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates from a start position |
| `rates-range` | Export rates for a date range |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a recent trailing window |
| `account-info` | Export account information |
| `terminal-info` | Export terminal information |
| `version` | Export MetaTrader 5 version information |
| `last-error` | Export the last error information |
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume buy and sell margin requirements |
| `market-book` | Export market depth (order book) |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a recent trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| `collect-history` | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -127,6 +133,10 @@ update_history_with_config(
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
- **Rate view resolution**: use `mt5cli.history.resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases.
- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
## Requirements
@@ -1,6 +1,6 @@
# SQLite History Module
# History Collection (SQLite)
::: mt5cli.sqlite_history
::: mt5cli.history
## `collect-history` schema
@@ -129,3 +129,57 @@ when required columns are missing.
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
### Rate view resolution
Downstream tools can resolve mt5cli-managed compatibility view names from an
existing SQLite history database without creating files or guessing legacy
naming schemes:
```python
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
```
Resolution rules:
- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
are stored for the same symbol.
- When multiple naming candidates apply, prefers an existing managed
`rate_*__*` view from the candidate list.
- Falls back to single-timeframe naming when the database path is missing or
`rates` metadata is unavailable.
- Pass `require_existing=True` to raise `ValueError` instead of returning a
best-guess name when the database or view is missing.
- Accepts either a SQLite path or an open `sqlite3.Connection`.
### Rate data loading
Use `load_rate_data()` to load a table or view from a SQLite path, or
`load_rate_data_from_connection()` when you already have a connection:
```python
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
```
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that `time` exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending `DatetimeIndex` named `time`.
+24
View File
@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
### [History Collection (SQLite)](history.md)
SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
## Architecture Overview
The package follows a simple architecture built on top of pdmt5:
@@ -61,12 +65,18 @@ from datetime import UTC, datetime
from pathlib import Path
from mt5cli import (
Dataset,
IfExists,
Mt5CliClient,
collect_history,
copy_rates_range,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# Fetch rates programmatically
rates = copy_rates_range(
@@ -82,6 +92,20 @@ fmt = detect_format(Path("output.parquet")) # Returns "parquet"
# Export a DataFrame
export_dataframe(rates, Path("output.csv"), "csv")
# Append to SQLite with deduplication
export_dataframe_to_sqlite(
rates,
Path("history.db"),
"rates",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "timeframe", "time"),
)
# Resolve rate compatibility views and fetch recent ticks
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Collect history into SQLite
collect_history(
Path("history.db"),
+44 -15
View File
@@ -13,6 +13,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
- **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
- **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
- **Connection management**: Optional credentials, server, and timeout configuration
- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
## Installation
@@ -22,13 +23,23 @@ pip install mt5cli
## Programmatic usage / SDK usage
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` when you need to persist results.
mt5cli can be used as a small Python SDK for read-only MetaTrader 5 data collection. SDK functions return pandas DataFrames without writing files. Use `export_dataframe` or `export_dataframe_to_sqlite` when you need to persist results.
```python
from datetime import UTC, datetime
from pathlib import Path
from mt5cli import Mt5CliClient, collect_history, copy_rates_range, export_dataframe
from mt5cli import (
Mt5CliClient,
collect_history,
copy_rates_range,
export_dataframe,
export_dataframe_to_sqlite,
load_rate_data,
minimum_margins,
recent_ticks,
)
from mt5cli.history import resolve_rate_view_name
# One-off fetch with module-level helpers
rates = copy_rates_range(
@@ -39,10 +50,21 @@ rates = copy_rates_range(
)
export_dataframe(rates, Path("rates.csv"), "csv")
# Resolve SQLite rate compatibility views for downstream tools
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
offline_rates = load_rate_data(Path("history.db"), view, count=1000)
# Recent tick window and minimum margin summary
ticks = recent_ticks("EURUSD", seconds=300)
margins = minimum_margins("EURUSD")
# Reuse one MT5 connection for multiple calls
with Mt5CliClient(login=12345, password="secret", server="Broker-Demo") as client:
account = client.account_info()
positions = client.positions()
latest = client.latest_rates("EURUSD", "M1", count=100)
summary = client.mt5_summary()
summary_table = client.mt5_summary_as_df()
# Bulk SQLite collection (same behavior as the collect-history CLI command)
collect_history(
@@ -58,6 +80,8 @@ collect_history(
Timeframes, tick flags, and ISO 8601 date strings are accepted wherever noted in the SDK API.
`Mt5CliClient.mt5_summary()` returns the SDK structured form as plain nested Python values. Use `Mt5CliClient.mt5_summary_as_df()` when you need a one-row DataFrame for export. The `mt5-summary` CLI command uses this tabular form, so nested terminal/account fields are JSON-encoded strings that are safe for CSV, JSON, Parquet, and SQLite output.
## Quick Start
```bash
@@ -88,14 +112,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| ---------------- | ---------------------------------- |
| `rates-from` | Export rates from a start date |
| `rates-from-pos` | Export rates from a start position |
| `latest-rates` | Export latest rates |
| `rates-range` | Export rates for a date range |
### Ticks
| Command | Description |
| ------------- | ------------------------------ |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| Command | Description |
| -------------- | ----------------------------------- |
| `ticks-from` | Export ticks from a start date |
| `ticks-range` | Export ticks for a date range |
| `ticks-recent` | Export ticks from a trailing window |
### Information
@@ -108,18 +134,21 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
| `symbols` | Export symbol list |
| `symbol-info` | Export symbol details |
| `symbol-info-tick` | Export the last tick for a symbol |
| `minimum-margins` | Export minimum-volume margin summary |
| `market-book` | Export market depth (order book) |
### Trading
| Command | Description |
| ---------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
| Command | Description |
| ---------------------- | ----------------------------------------------------------- |
| `orders` | Export active orders |
| `positions` | Export open positions |
| `history-orders` | Export historical orders |
| `history-deals` | Export historical deals |
| `recent-history-deals` | Export historical deals from a trailing window |
| `mt5-summary` | Export terminal/account status summary |
| `order-check` | Check funds sufficiency for a trade request |
| `order-send` | Send a trade request to the trade server (`--yes` required) |
Use `order-check` to validate a request payload before running `order-send --yes`.
@@ -152,7 +181,7 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
See the [SQLite History schema diagram](api/sqlite_history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
See the [History schema diagram](api/history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
## Global Options
+1 -1
View File
@@ -58,7 +58,7 @@ nav:
- Overview: api/index.md
- CLI: api/cli.md
- SDK: api/sdk.md
- SQLite History: api/sqlite_history.md
- History Collection (SQLite): api/history.md
- Utils: api/utils.md
markdown_extensions:
+25 -1
View File
@@ -2,11 +2,13 @@
from importlib.metadata import version
from .history import load_rate_data, load_rate_data_from_connection
from .sdk import (
Mt5CliClient,
account_info,
build_config,
collect_history,
collect_latest_rates,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -15,9 +17,15 @@ from .sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
symbol_info,
symbol_info_tick,
symbols,
@@ -28,7 +36,13 @@ from .sdk import (
from .sdk import (
version as mt5_version,
)
from .utils import Dataset, IfExists, detect_format, export_dataframe
from .utils import (
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
)
__version__ = version(__package__) if __package__ else None
@@ -39,6 +53,7 @@ __all__ = [
"account_info",
"build_config",
"collect_history",
"collect_latest_rates",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
@@ -46,13 +61,22 @@ __all__ = [
"copy_ticks_range",
"detect_format",
"export_dataframe",
"export_dataframe_to_sqlite",
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"load_rate_data",
"load_rate_data_from_connection",
"market_book",
"minimum_margins",
"mt5_summary",
"mt5_summary_as_df",
"mt5_version",
"orders",
"positions",
"recent_history_deals",
"recent_ticks",
"symbol_info",
"symbol_info_tick",
"symbols",
+104
View File
@@ -222,6 +222,31 @@ def rates_from_pos(
)
@app.command()
def latest_rates(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
timeframe: Annotated[
int,
typer.Option(
click_type=TIMEFRAME_TYPE,
help="Timeframe.",
),
],
count: Annotated[int, typer.Option(help="Number of records.")],
start_pos: Annotated[
int,
typer.Option(help="Start position (0 = current bar)."),
] = 0,
) -> None:
"""Export latest rates from a start position."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.latest_rates(symbol, timeframe, count, start_pos=start_pos),
)
@app.command()
def rates_range(
ctx: typer.Context,
@@ -300,6 +325,44 @@ def ticks_range(
)
@app.command()
def ticks_recent(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
seconds: Annotated[
float,
typer.Option(help="Lookback window in seconds."),
],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
count: Annotated[
int,
typer.Option(help="Maximum number of ticks to return."),
] = 10000,
flags: Annotated[
int,
typer.Option(
click_type=TICK_FLAGS_TYPE,
help="Tick flags (ALL, INFO, TRADE, or integer).",
),
] = 1,
) -> None:
"""Export ticks from a recent time window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
),
)
@app.command()
def account_info(ctx: typer.Context) -> None:
"""Export account information."""
@@ -335,6 +398,16 @@ def symbol_info(
_execute_export(ctx, lambda: client.symbol_info(symbol))
@app.command()
def minimum_margins(
ctx: typer.Context,
symbol: Annotated[str, typer.Option(help="Symbol name.")],
) -> None:
"""Export minimum-volume buy and sell margin requirements."""
client = _sdk_client(ctx)
_execute_export(ctx, lambda: client.minimum_margins(symbol))
@app.command()
def orders(
ctx: typer.Context,
@@ -427,6 +500,37 @@ def history_deals(
)
@app.command()
def recent_history_deals(
ctx: typer.Context,
hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
date_to: Annotated[
datetime | None,
typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
] = None,
group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
) -> None:
"""Export historical deals from a recent trailing window."""
client = _sdk_client(ctx)
_execute_export(
ctx,
lambda: client.recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
),
)
@app.command()
def mt5_summary(ctx: typer.Context) -> None:
"""Export a compact terminal/account status summary."""
client = _sdk_client(ctx)
_execute_export(ctx, client.mt5_summary_as_df)
@app.command()
def version(ctx: typer.Context) -> None:
"""Export MetaTrader5 version information."""
+364 -3
View File
@@ -1,11 +1,12 @@
"""SQLite helpers for incremental MT5 history collection."""
"""SQLite storage helpers for the ``collect-history`` incremental data pipeline."""
from __future__ import annotations
import logging
import sqlite3
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Literal
from pathlib import Path
from typing import TYPE_CHECKING, Literal, cast
import pandas as pd
@@ -122,9 +123,369 @@ def build_rate_view_name(
return f"rate_{symbol}__{granularity}_{timeframe}"
SqliteConnOrPath = sqlite3.Connection | Path | str
def _require_non_empty_identifier(identifier: str, kind: str) -> str:
value = identifier.strip()
if not value:
msg = f"SQLite {kind} name must not be empty."
raise ValueError(msg)
return value
def _open_history_connection(
conn_or_path: SqliteConnOrPath,
) -> tuple[sqlite3.Connection | None, bool]:
"""Open a read-only SQLite connection when given a path.
Returns:
A connection and whether the caller should close it. When the path does
not exist, returns ``(None, False)`` without creating a database file.
"""
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
return None, False
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _open_existing_sqlite_database(
conn_or_path: SqliteConnOrPath,
) -> tuple[sqlite3.Connection, bool]:
"""Open a read-only SQLite database or reuse an existing connection.
Returns:
Tuple of connection and whether the caller should close it.
Raises:
ValueError: If the database path does not exist or is not a file.
"""
if isinstance(conn_or_path, sqlite3.Connection):
return conn_or_path, False
path = Path(conn_or_path)
if not path.exists():
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
if not path.is_file():
msg = f"SQLite database path is not a file: {path}"
raise ValueError(msg)
conn = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
return conn, True
def _validate_rate_load_request(table: str, count: int | None) -> str:
table_name = _require_non_empty_identifier(table, "table or view")
if count is not None and count <= 0:
msg = "count must be positive when provided."
raise ValueError(msg)
return table_name
def _ensure_rate_columns(columns: set[str], table: str) -> None:
if not columns:
msg = f"SQLite table or view not found: {table}"
raise ValueError(msg)
if "time" not in columns:
msg = f"SQLite table or view {table!r} must include a time column."
raise ValueError(msg)
if "close" not in columns and not {"ask", "bid"}.issubset(columns):
msg = (
f"SQLite table or view {table!r} must include close, "
"or both ask and bid columns."
)
raise ValueError(msg)
def _parse_rate_time_index(frame: pd.DataFrame, table: str) -> pd.DataFrame:
parsed = frame["time"].map(parse_sqlite_timestamp)
if parsed.isna().any():
msg = f"SQLite table or view {table!r} contains unparsable time values."
raise ValueError(msg)
result = frame.drop(columns=["time"])
result.index = pd.DatetimeIndex(parsed, name="time")
return result.sort_index(kind="stable")
def load_rate_data_from_connection(
connection: sqlite3.Connection,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite table or view.
Args:
connection: Open SQLite connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
Raises:
ValueError: If inputs, schema, timestamps are invalid, or the table
or view contains no rows.
"""
table_name = _validate_rate_load_request(table, count)
columns = get_table_columns(connection, table_name)
_ensure_rate_columns(columns, table_name)
quoted_table = quote_sqlite_identifier(table_name)
if count is None:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time ASC", # noqa: S608
connection,
),
)
else:
frame = cast(
"pd.DataFrame",
pd.read_sql_query( # type: ignore[reportUnknownMemberType]
f"SELECT * FROM {quoted_table} ORDER BY time DESC LIMIT ?", # noqa: S608
connection,
params=(count,),
),
)
if frame.empty:
msg = f"SQLite table or view {table_name!r} contains no rows."
raise ValueError(msg)
return _parse_rate_time_index(frame, table_name)
def load_rate_data(
conn_or_path: SqliteConnOrPath,
table: str,
count: int | None = None,
) -> pd.DataFrame:
"""Load rate-like data from a SQLite database path or connection.
Args:
conn_or_path: SQLite database path or open connection.
table: Source table or view name.
count: Optional number of most recent rows to load.
Returns:
DataFrame indexed by ascending ``time``.
"""
conn, should_close = _open_existing_sqlite_database(conn_or_path)
try:
return load_rate_data_from_connection(conn, table, count=count)
finally:
if should_close:
conn.close()
def _load_rates_timeframe_counts(conn: sqlite3.Connection) -> dict[str, int] | None:
"""Return distinct timeframe counts per symbol from the normalized rates table."""
columns = get_table_columns(conn, Dataset.rates.table_name)
if not {"symbol", "timeframe"}.issubset(columns):
return None
rows = conn.execute(
"SELECT symbol, COUNT(DISTINCT timeframe) FROM rates GROUP BY symbol",
).fetchall()
return {str(symbol): int(count) for symbol, count in rows}
def _load_existing_rate_views(conn: sqlite3.Connection) -> set[str]:
"""Return mt5cli-managed ``rate_*__*`` compatibility view names."""
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'view' AND name GLOB 'rate_*__*'",
).fetchall()
return {str(row[0]) for row in rows}
def _rate_view_name_candidates(
*,
symbol: str,
granularity: str,
granularity_count: int,
timeframe: int,
) -> list[str]:
"""Return candidate view names in preference order."""
single = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=1,
timeframe=timeframe,
)
if granularity_count <= 1:
return [single]
multi = build_rate_view_name(
symbol=symbol,
granularity=granularity,
granularity_count=granularity_count,
timeframe=timeframe,
)
return [multi, single]
def _resolve_rate_view_name_from_context(
*,
symbol: str,
timeframe: int,
granularity_name: str,
timeframe_counts: dict[str, int] | None,
existing_views: set[str],
require_existing: bool = False,
) -> str:
"""Resolve one rate view name using preloaded SQLite metadata.
Returns:
Preferred mt5cli-managed rate compatibility view name.
Raises:
ValueError: If ``require_existing`` is True and no managed view exists.
"""
if timeframe_counts is None or symbol not in timeframe_counts:
candidates = [
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
),
build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=2,
timeframe=timeframe,
),
]
else:
candidates = _rate_view_name_candidates(
symbol=symbol,
granularity=granularity_name,
granularity_count=timeframe_counts[symbol],
timeframe=timeframe,
)
for candidate in candidates:
if candidate in existing_views:
return candidate
if require_existing:
msg = (
f"No rate compatibility view exists for symbol {symbol!r} "
f"and granularity {granularity_name!r}; "
f"candidates: {', '.join(candidates)}."
)
raise ValueError(msg)
return candidates[0]
def resolve_rate_view_name(
conn_or_path: SqliteConnOrPath,
symbol: str,
granularity: str,
*,
require_existing: bool = False,
) -> str:
"""Resolve the mt5cli-managed rate compatibility view name.
Args:
conn_or_path: SQLite database path or open connection.
symbol: Symbol stored in the normalized ``rates`` table.
granularity: Timeframe name (for example ``M1``) or integer string.
require_existing: When True, require the database and a managed view to exist.
Returns:
View name such as ``rate_EURUSD__1`` or ``rate_EURUSD__M1_1``.
Raises:
ValueError: If ``require_existing`` is True and the database or view is missing.
"""
timeframe = parse_timeframe(granularity)
granularity_name = resolve_granularity_name(timeframe)
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
if require_existing:
path = (
conn_or_path
if isinstance(conn_or_path, (Path, str))
else "database"
)
msg = f"SQLite database not found: {path}"
raise ValueError(msg)
return build_rate_view_name(
symbol=symbol,
granularity=granularity_name,
granularity_count=1,
timeframe=timeframe,
)
return _resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=granularity_name,
timeframe_counts=_load_rates_timeframe_counts(conn),
existing_views=_load_existing_rate_views(conn),
require_existing=require_existing,
)
finally:
if should_close and conn is not None:
conn.close()
def resolve_rate_view_names(
conn_or_path: SqliteConnOrPath,
symbols: Sequence[str],
granularities: Sequence[str],
*,
require_existing: bool = False,
) -> list[str]:
"""Resolve rate compatibility view names for symbol and granularity pairs.
Args:
conn_or_path: SQLite database path or open connection.
symbols: Symbols stored in the normalized ``rates`` table.
granularities: Timeframe names (for example ``M1``) or integer strings.
require_existing: When True, require the database and managed views to exist.
Returns:
View names in row-major order: every ``granularity`` for the first
symbol, then every granularity for the next symbol, and so on.
"""
conn, should_close = _open_history_connection(conn_or_path)
try:
if conn is None:
return [
resolve_rate_view_name(
conn_or_path,
symbol,
granularity,
require_existing=require_existing,
)
for symbol in symbols
for granularity in granularities
]
timeframe_counts = _load_rates_timeframe_counts(conn)
existing_views = _load_existing_rate_views(conn)
resolved: list[str] = []
for symbol in symbols:
for granularity in granularities:
timeframe = parse_timeframe(granularity)
resolved.append(
_resolve_rate_view_name_from_context(
symbol=symbol,
timeframe=timeframe,
granularity_name=resolve_granularity_name(timeframe),
timeframe_counts=timeframe_counts,
existing_views=existing_views,
require_existing=require_existing,
),
)
return resolved
finally:
if should_close and conn is not None:
conn.close()
def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
"""Return existing SQLite columns for a table."""
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
quoted_table = quote_sqlite_identifier(table)
rows = conn.execute(f"PRAGMA table_info({quoted_table})").fetchall()
return {str(row[1]) for row in rows}
+405 -7
View File
@@ -2,17 +2,19 @@
from __future__ import annotations
import json
import logging
import sqlite3
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import TYPE_CHECKING, Self, TypeVar
from typing import TYPE_CHECKING, Self, TypeVar, cast
import pandas as pd
from pdmt5 import Mt5Config, Mt5DataClient
from .sqlite_history import (
from .history import (
create_cash_events_view,
create_history_indexes,
create_positions_reconstructed_view,
@@ -33,8 +35,6 @@ from .utils import (
if TYPE_CHECKING:
from collections.abc import Callable, Iterator, Sequence
import pandas as pd
T = TypeVar("T")
logger = logging.getLogger(__name__)
@@ -44,6 +44,7 @@ __all__ = [
"account_info",
"build_config",
"collect_history",
"collect_latest_rates",
"copy_rates_from",
"copy_rates_from_pos",
"copy_rates_range",
@@ -52,9 +53,15 @@ __all__ = [
"history_deals",
"history_orders",
"last_error",
"latest_rates",
"market_book",
"minimum_margins",
"mt5_summary",
"mt5_summary_as_df",
"orders",
"positions",
"recent_history_deals",
"recent_ticks",
"symbol_info",
"symbol_info_tick",
"symbols",
@@ -77,6 +84,22 @@ def _coerce_tick_flags(flags: int | str) -> int:
return parse_tick_flags(flags)
def _plain_mt5_value(value: object) -> object:
asdict = getattr(value, "_asdict", None)
if callable(asdict):
return _plain_mt5_value(asdict())
if isinstance(value, dict):
typed_value = cast("dict[object, object]", value)
return {key: _plain_mt5_value(item) for key, item in typed_value.items()}
if isinstance(value, tuple):
typed_value = cast("tuple[object, ...]", value)
return [_plain_mt5_value(item) for item in typed_value]
if isinstance(value, list):
typed_value = cast("list[object]", value)
return [_plain_mt5_value(item) for item in typed_value]
return value
def _require_datetime(value: datetime | str) -> datetime:
if isinstance(value, datetime):
return value
@@ -89,6 +112,114 @@ def _coerce_datetime(value: datetime | str | None) -> datetime | None:
return parse_datetime(value)
def _require_positive(value: float, name: str) -> None:
if value <= 0:
msg = f"{name} must be positive."
raise ValueError(msg)
def _call_required_client_method(client: Mt5DataClient, name: str) -> object:
try:
method = getattr(client, name)
except AttributeError as exc:
msg = f"MT5 client is missing required method: {name}"
raise AttributeError(msg) from exc
if not callable(method):
msg = f"MT5 client attribute is not callable: {name}"
raise TypeError(msg)
return method()
def _mt5_summary_export_value(value: object) -> object:
plain_value = _plain_mt5_value(value)
if isinstance(plain_value, dict | list):
return json.dumps(plain_value, sort_keys=True, separators=(",", ":"))
return plain_value
def _coerce_tick_time(value: object) -> datetime:
if isinstance(value, datetime):
return value
if isinstance(value, str):
return parse_datetime(value)
if isinstance(value, (int, float)):
return datetime.fromtimestamp(value, tz=UTC)
msg = f"Unsupported tick time value: {value!r}"
raise TypeError(msg)
def _filter_ticks_to_end(frame: pd.DataFrame, end: datetime) -> pd.DataFrame:
if frame.empty or "time" not in frame.columns:
return frame
times = pd.to_datetime(frame["time"], utc=True)
return frame.loc[times <= end].reset_index(drop=True)
def _fetch_recent_ticks(
client: Mt5DataClient,
symbol: str,
seconds: float,
date_to: datetime | None,
count: int,
flags: int,
) -> pd.DataFrame:
if date_to is not None:
end = date_to
else:
tick = client.symbol_info_tick(symbol)
end = _coerce_tick_time(tick.time)
start = end - timedelta(seconds=seconds)
if count > 0:
from_frame = _filter_ticks_to_end(
client.copy_ticks_from_as_df(
symbol=symbol,
date_from=start,
count=count,
flags=flags,
),
end,
)
if len(from_frame) < count:
return from_frame
frame = client.copy_ticks_range_as_df(
symbol=symbol,
date_from=start,
date_to=end,
flags=flags,
)
if count > 0 and len(frame) > count:
return frame.tail(count).reset_index(drop=True)
return frame
def _fetch_minimum_margins(client: Mt5DataClient, symbol: str) -> pd.DataFrame:
sym = client.symbol_info(symbol)
account = client.account_info()
tick = client.symbol_info_tick(symbol)
volume_min = sym.volume_min
buy_margin = client.order_calc_margin(
client.mt5.ORDER_TYPE_BUY,
symbol,
volume_min,
tick.ask,
)
sell_margin = client.order_calc_margin(
client.mt5.ORDER_TYPE_SELL,
symbol,
volume_min,
tick.bid,
)
return pd.DataFrame([
{
"symbol": symbol,
"account_currency": account.currency,
"volume_min": volume_min,
"buy_margin": buy_margin,
"sell_margin": sell_margin,
}
])
def build_config(
*,
path: str | None = None,
@@ -158,6 +289,7 @@ class Mt5CliClient:
server: str | None = None,
timeout: int | None = None,
config: Mt5Config | None = None,
client: Mt5DataClient | None = None,
) -> None:
"""Initialize the SDK client.
@@ -168,6 +300,8 @@ class Mt5CliClient:
server: Trading server name.
timeout: Connection timeout in milliseconds.
config: Optional pre-built ``Mt5Config`` (overrides other args).
client: Optional already-connected ``Mt5DataClient``. Injected
clients are reused as-is and are not initialized or shut down.
"""
self._config = config or build_config(
path=path,
@@ -176,7 +310,20 @@ class Mt5CliClient:
server=server,
timeout=timeout,
)
self._client: Mt5DataClient | None = None
self._client = client
self._owns_client = client is None
@classmethod
def from_connected_client(cls, client: Mt5DataClient) -> Self:
"""Bind to an already-connected ``Mt5DataClient`` without owning it.
The returned ``Mt5CliClient`` never initializes or shuts down the
injected client, including when used as a context manager.
Returns:
Client wrapper bound to the injected connection.
"""
return cls(client=client)
@property
def config(self) -> Mt5Config:
@@ -189,6 +336,8 @@ class Mt5CliClient:
Returns:
This client instance.
"""
if self._client is not None:
return self
client = Mt5DataClient(config=self._config)
try:
client.initialize_and_login_mt5()
@@ -196,6 +345,7 @@ class Mt5CliClient:
client.shutdown()
raise
self._client = client
self._owns_client = True # only set when this method created the client
return self
def __exit__(
@@ -205,15 +355,18 @@ class Mt5CliClient:
tb: object,
) -> None:
"""Shut down the persistent MT5 connection."""
if self._client is not None:
if self._client is not None and self._owns_client:
self._client.shutdown()
self._client = None
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
def _fetch_value(self, fetch_fn: Callable[[Mt5DataClient], T]) -> T:
if self._client is not None:
return fetch_fn(self._client)
return _run_with_client(self._config, fetch_fn)
def _fetch(self, fetch_fn: Callable[[Mt5DataClient], pd.DataFrame]) -> pd.DataFrame:
return self._fetch_value(fetch_fn)
def copy_rates_from(
self,
symbol: str,
@@ -251,6 +404,54 @@ class Mt5CliClient:
),
)
def latest_rates(
self,
symbol: str,
timeframe: int | str,
count: int,
start_pos: int = 0,
) -> pd.DataFrame:
"""Return the latest rates from a bar position."""
_require_positive(count, "count")
return self.copy_rates_from_pos(symbol, timeframe, start_pos, count)
def collect_latest_rates(
self,
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
count: int,
start_pos: int = 0,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Return latest rates for each symbol/timeframe pair.
Returns:
Mapping keyed by ``(symbol, timeframe_int)``.
Raises:
ValueError: If ``count`` is not positive or inputs are empty.
"""
_require_positive(count, "count")
if not symbols:
msg = "At least one symbol is required."
raise ValueError(msg)
if not timeframes:
msg = "At least one timeframe is required."
raise ValueError(msg)
resolved_timeframes = [_coerce_timeframe(timeframe) for timeframe in timeframes]
return self._fetch_value(
lambda c: {
(symbol, timeframe): c.copy_rates_from_pos_as_df(
symbol=symbol,
timeframe=timeframe,
start_pos=start_pos,
count=count,
)
for symbol in symbols
for timeframe in resolved_timeframes
},
)
def copy_rates_range(
self,
symbol: str,
@@ -402,6 +603,24 @@ class Mt5CliClient:
),
)
def recent_history_deals(
self,
hours: float,
date_to: datetime | str | None = None,
group: str | None = None,
symbol: str | None = None,
) -> pd.DataFrame:
"""Return historical deals from a recent trailing window."""
_require_positive(hours, "hours")
end = _require_datetime(date_to) if date_to is not None else datetime.now(UTC)
start = end - timedelta(hours=hours)
return self.history_deals(
date_from=start,
date_to=end,
group=group,
symbol=symbol,
)
def version(self) -> pd.DataFrame:
"""Return MetaTrader5 version information."""
return self._fetch(lambda c: c.version_as_df())
@@ -418,6 +637,90 @@ class Mt5CliClient:
"""Return market depth for a symbol."""
return self._fetch(lambda c: c.market_book_get_as_df(symbol=symbol))
def recent_ticks(
self,
symbol: str,
seconds: float,
*,
date_to: datetime | str | None = None,
count: int = 10000,
flags: int | str = "ALL",
) -> pd.DataFrame:
"""Return ticks from a recent time window.
Args:
symbol: Symbol name.
seconds: Lookback window in seconds ending at ``date_to``.
date_to: Window end time. When ``None``, uses the latest
``symbol_info_tick().time`` rather than wall-clock now.
count: Maximum ticks to return. Values ``<= 0`` return the full
window without trimming. Positive values keep the most recent
ticks; when the window is sparse, ``copy_ticks_from`` avoids
fetching the entire range.
flags: Tick flags as ``ALL``, ``INFO``, ``TRADE``, or an integer.
Returns:
Tick DataFrame with MT5 tick columns such as ``time``, ``bid``,
``ask``, ``last``, and ``volume``.
"""
tick_flags = _coerce_tick_flags(flags)
end = _coerce_datetime(date_to)
return self._fetch(
lambda c: _fetch_recent_ticks(
c,
symbol,
seconds,
end,
count,
tick_flags,
),
)
def minimum_margins(self, symbol: str) -> pd.DataFrame:
"""Return minimum-volume buy and sell margin requirements.
Args:
symbol: Symbol name.
Returns:
One-row DataFrame with columns ``symbol``, ``account_currency``,
``volume_min``, ``buy_margin``, and ``sell_margin``.
"""
return self._fetch(lambda c: _fetch_minimum_margins(c, symbol))
def mt5_summary(self) -> dict[str, object]:
"""Return a compact terminal/account status summary."""
def _summary(client: Mt5DataClient) -> dict[str, object]:
return {
"version": _plain_mt5_value(
_call_required_client_method(client, "version"),
),
"terminal_info": _plain_mt5_value(
_call_required_client_method(client, "terminal_info"),
),
"account_info": _plain_mt5_value(
_call_required_client_method(client, "account_info"),
),
"symbols_total": _plain_mt5_value(
_call_required_client_method(client, "symbols_total"),
),
}
return self._fetch_value(_summary)
def mt5_summary_as_df(self) -> pd.DataFrame:
"""Return an export-safe one-row terminal/account summary DataFrame."""
summary = self.mt5_summary()
return pd.DataFrame(
[
{
key: _mt5_summary_export_value(value)
for key, value in summary.items()
},
],
)
def _resolve_incremental_settings(
selected_datasets: set[Dataset],
@@ -738,6 +1041,40 @@ def copy_rates_from_pos(
)
def latest_rates(
symbol: str,
timeframe: int | str,
count: int,
start_pos: int = 0,
*,
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return the latest rates from a bar position."""
return _make_client(config=config).latest_rates(
symbol,
timeframe,
count,
start_pos=start_pos,
)
def collect_latest_rates(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
count: int,
start_pos: int = 0,
config: Mt5Config | None = None,
) -> dict[tuple[str, int], pd.DataFrame]:
"""Return latest rates for each symbol/timeframe pair."""
return _make_client(config=config).collect_latest_rates(
symbols,
timeframes,
count=count,
start_pos=start_pos,
)
def copy_rates_range(
symbol: str,
timeframe: int | str,
@@ -889,6 +1226,23 @@ def history_deals(
)
def recent_history_deals(
hours: float,
date_to: datetime | str | None = None,
group: str | None = None,
symbol: str | None = None,
*,
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return historical deals from a recent trailing window."""
return _make_client(config=config).recent_history_deals(
hours,
date_to=date_to,
group=group,
symbol=symbol,
)
def version(*, config: Mt5Config | None = None) -> pd.DataFrame:
"""Return MetaTrader5 version information."""
return _make_client(config=config).version()
@@ -915,3 +1269,47 @@ def market_book(
) -> pd.DataFrame:
"""Return market depth for a symbol."""
return _make_client(config=config).market_book(symbol)
def recent_ticks(
symbol: str,
seconds: float,
*,
date_to: datetime | str | None = None,
count: int = 10000,
flags: int | str = "ALL",
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return ticks from a recent time window ending at ``date_to`` or now.
See ``Mt5CliClient.recent_ticks`` for parameter and return details.
"""
return _make_client(config=config).recent_ticks(
symbol,
seconds,
date_to=date_to,
count=count,
flags=flags,
)
def minimum_margins(
symbol: str,
*,
config: Mt5Config | None = None,
) -> pd.DataFrame:
"""Return minimum-volume buy and sell margin requirements.
See ``Mt5CliClient.minimum_margins`` for return details.
"""
return _make_client(config=config).minimum_margins(symbol)
def mt5_summary(*, config: Mt5Config | None = None) -> dict[str, object]:
"""Return a compact terminal/account status summary."""
return _make_client(config=config).mt5_summary()
def mt5_summary_as_df(*, config: Mt5Config | None = None) -> pd.DataFrame:
"""Return an export-safe terminal/account status summary DataFrame."""
return _make_client(config=config).mt5_summary_as_df()
+55 -10
View File
@@ -2,16 +2,18 @@
from __future__ import annotations
import importlib
import json
import sqlite3
from datetime import UTC, datetime
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING, Any, TypeGuard, cast
from typing import TYPE_CHECKING, Any, TypeGuard
import click
if TYPE_CHECKING:
from collections.abc import Sequence
import pandas as pd
# ---------------------------------------------------------------------------
@@ -260,6 +262,50 @@ def detect_format(
raise ValueError(msg)
def export_dataframe_to_sqlite(
df: pd.DataFrame,
output_path: Path,
table_name: str = "data",
*,
if_exists: IfExists = IfExists.APPEND,
index: bool = False,
index_label: str | None = None,
deduplicate_on: Sequence[str] | None = None,
) -> None:
"""Write a DataFrame to SQLite with configurable append and deduplication.
Args:
df: DataFrame to export.
output_path: SQLite database path.
table_name: Target table name.
if_exists: Conflict behavior when the table already exists.
index: Whether to write the DataFrame index as a column.
index_label: Column name for the index when ``index=True``.
deduplicate_on: Optional key columns to deduplicate after writing,
keeping the latest ``ROWID`` per key group. Deduplication scans the
full table, so repeated appends cost O(table size); index the key
columns when appending frequently.
"""
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=index,
index_label=index_label,
)
if deduplicate_on:
from .history import drop_duplicates_in_table # noqa: PLC0415
drop_duplicates_in_table(
conn.cursor(),
table_name,
list(deduplicate_on),
keep="last",
)
conn.commit()
def export_dataframe(
df: pd.DataFrame,
output_path: Path,
@@ -289,14 +335,13 @@ def export_dataframe(
elif output_format == "parquet":
df.to_parquet(output_path, index=False)
elif output_format == "sqlite3":
sqlite3 = cast("Any", importlib.import_module("sqlite3"))
with sqlite3.connect(output_path) as conn:
df.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists="replace",
index=False,
)
export_dataframe_to_sqlite(
df,
output_path,
table_name,
if_exists=IfExists.REPLACE,
index=False,
)
else:
msg = f"Unsupported output format: {output_format}"
raise ValueError(msg)
+2 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.1"
version = "0.5.0"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -124,7 +124,7 @@ ignore = [
]
[tool.ruff.lint.per-file-ignores]
"mt5cli/sqlite_history.py" = ["TC003"]
"mt5cli/history.py" = ["TC003"]
"tests/**/*.py" = [
"DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring
+173 -1
View File
@@ -6,7 +6,7 @@ import json
import logging
import re
import sqlite3
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
@@ -93,6 +93,10 @@ def mock_client(mocker: MockerFixture) -> MagicMock:
client.market_book_get_as_df.return_value = sample_df
client.order_check_as_df.return_value = sample_df
client.order_send_as_df.return_value = sample_df
client.version.return_value = (5, 0, 1)
client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
client.symbols_total.return_value = 42
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
return client
@@ -223,6 +227,37 @@ class TestCommands:
count=50,
)
def test_latest_rates(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test latest-rates command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"latest-rates",
"--symbol",
"GBPUSD",
"--timeframe",
"H1",
"--count",
"50",
"--start-pos",
"2",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="GBPUSD",
timeframe=16385,
start_pos=2,
count=50,
)
def test_rates_range(
self,
tmp_path: Path,
@@ -316,6 +351,65 @@ class TestCommands:
flags=2,
)
def test_ticks_recent(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test ticks-recent command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"ticks-recent",
"--symbol",
"EURUSD",
"--seconds",
"120",
"--date-to",
"2024-01-02",
"--count",
"500",
"--flags",
"ALL",
],
)
assert result.exit_code == 0, result.output
mock_client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=datetime(2024, 1, 2, tzinfo=UTC) - timedelta(seconds=120),
count=500,
flags=1,
)
mock_client.copy_ticks_range_as_df.assert_not_called()
def test_minimum_margins(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test minimum-margins command."""
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
mock_client.symbol_info.return_value = sym
mock_client.account_info.return_value = account
mock_client.symbol_info_tick.return_value = tick
mock_client.order_calc_margin.side_effect = [12.5, 12.4]
mock_client.mt5.ORDER_TYPE_BUY = 0
mock_client.mt5.ORDER_TYPE_SELL = 1
output = tmp_path / "out.csv"
result = runner.invoke(
app,
["-o", str(output), "minimum-margins", "--symbol", "EURUSD"],
)
assert result.exit_code == 0, result.output
mock_client.symbol_info.assert_called_once_with("EURUSD")
mock_client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
mock_client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
def test_orders(
self,
tmp_path: Path,
@@ -392,6 +486,84 @@ class TestCommands:
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once()
def test_recent_history_deals(
self,
tmp_path: Path,
mock_client: MagicMock,
) -> None:
"""Test recent-history-deals command."""
output = tmp_path / "out.csv"
result = runner.invoke(
app,
[
"-o",
str(output),
"recent-history-deals",
"--hours",
"6",
"--date-to",
"2024-01-02",
"--symbol",
"EURUSD",
],
)
assert result.exit_code == 0, result.output
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group=None,
symbol="EURUSD",
ticket=None,
position=None,
)
@pytest.mark.parametrize(
("filename", "reader"),
[
("summary.csv", "csv"),
("summary.json", "json"),
("summary.db", "sqlite3"),
("summary.parquet", "parquet"),
],
)
def test_mt5_summary_export_formats(
self,
tmp_path: Path,
mock_client: MagicMock,
filename: str,
reader: str,
) -> None:
"""Test mt5-summary writes export-safe files for supported formats."""
output = tmp_path / filename
result = runner.invoke(app, ["-o", str(output), "mt5-summary"])
assert result.exit_code == 0, result.output
assert output.exists()
mock_client.version.assert_called_once()
mock_client.terminal_info.assert_called_once()
mock_client.account_info.assert_called_once()
mock_client.symbols_total.assert_called_once()
if reader == "csv":
frame = pd.read_csv(output)
elif reader == "json":
with output.open() as f:
records = json.load(f)
frame = pd.DataFrame(records)
elif reader == "sqlite3":
with sqlite3.connect(output) as conn:
frame = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT * FROM data",
conn,
)
else:
frame = pd.read_parquet(output)
assert len(frame) == 1
assert frame.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"paths":["terminal.exe"]}',
"account_info": '{"limits":{"modes":["demo"]},"login":123}',
"symbols_total": 42,
}
def test_version(
self,
tmp_path: Path,
@@ -1,4 +1,4 @@
"""Tests for mt5cli.sqlite_history module."""
"""Tests for mt5cli.history module."""
from __future__ import annotations
@@ -14,7 +14,7 @@ import pytest
if TYPE_CHECKING:
from pathlib import Path
from mt5cli.sqlite_history import (
from mt5cli.history import (
DEFAULT_HISTORY_TIMEFRAMES,
append_dataframe,
augment_written_columns_from_sqlite,
@@ -31,6 +31,8 @@ from mt5cli.sqlite_history import (
get_incremental_start_datetime,
get_table_columns,
load_incremental_start_datetimes,
load_rate_data,
load_rate_data_from_connection,
parse_sqlite_timestamp,
quote_sqlite_identifier,
record_written_columns,
@@ -38,6 +40,8 @@ from mt5cli.sqlite_history import (
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
resolve_rate_view_name,
resolve_rate_view_names,
write_collected_datasets,
write_history_dataset,
write_incremental_datasets,
@@ -47,6 +51,281 @@ from mt5cli.sqlite_history import (
from mt5cli.utils import TIMEFRAME_MAP, Dataset, IfExists
class TestResolveRateViewName:
"""Tests for resolve_rate_view_name and resolve_rate_view_names."""
def test_missing_database_path_does_not_create_file(self, tmp_path: Path) -> None:
"""Test resolving against a missing path does not create a database."""
db_path = tmp_path / "missing.db"
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
assert not db_path.exists()
def test_no_rates_table_falls_back_to_single_timeframe_name(
self,
tmp_path: Path,
) -> None:
"""Test databases without a rates table use single-timeframe naming."""
db_path = tmp_path / "no-rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(symbol TEXT, time TEXT)")
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
def test_single_timeframe_for_one_symbol(self, tmp_path: Path) -> None:
"""Test one stored timeframe resolves to the short view name."""
db_path = tmp_path / "single-timeframe.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
def test_multiple_timeframes_for_one_symbol(self, tmp_path: Path) -> None:
"""Test multiple stored timeframes resolve to disambiguated view names."""
db_path = tmp_path / "multi-timeframe.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", TIMEFRAME_MAP["H1"], "2024-01-01T01:00:00+00:00", 1.1),
],
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__M1_1"
assert (
resolve_rate_view_name(db_path, "EURUSD", "H1") == "rate_EURUSD__H1_16385"
)
def test_prefers_multi_name_when_both_candidate_views_exist(
self,
tmp_path: Path,
) -> None:
"""Test multi-timeframe metadata wins over stale single-timeframe views."""
db_path = tmp_path / "stale-and-current-views.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", TIMEFRAME_MAP["H1"], "2024-01-01T01:00:00+00:00", 1.1),
],
)
conn.execute(
'CREATE VIEW "rate_EURUSD__1" AS'
" SELECT time, close FROM rates"
" WHERE symbol = 'EURUSD' AND timeframe = 1",
)
conn.execute(
'CREATE VIEW "rate_EURUSD__M1_1" AS'
" SELECT time, close FROM rates"
" WHERE symbol = 'EURUSD' AND timeframe = 1",
)
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__M1_1"
def test_prefers_existing_view_when_metadata_unavailable(
self,
tmp_path: Path,
) -> None:
"""Test an existing managed view is preferred without rates metadata."""
db_path = tmp_path / "view-only.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(symbol TEXT, time TEXT)")
conn.execute('CREATE VIEW "rate_EURUSD__M1_1" AS SELECT 1 AS close')
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__M1_1"
def test_symbol_absent_from_rates_metadata_uses_candidate_pair(
self,
tmp_path: Path,
) -> None:
"""Test symbols missing from rates metadata still resolve known views."""
db_path = tmp_path / "other-symbol-only.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("GBPUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
conn.execute(
'CREATE VIEW "rate_EURUSD__1" AS'
" SELECT time, close FROM rates"
" WHERE symbol = 'EURUSD' AND timeframe = 1",
)
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
def test_ignores_non_compatibility_rate_views(self, tmp_path: Path) -> None:
"""Test unrelated rate_* views without the __ separator are ignored."""
db_path = tmp_path / "summary-view.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(symbol TEXT, time TEXT)")
conn.execute('CREATE VIEW "rate_summary" AS SELECT 1 AS close')
assert resolve_rate_view_name(db_path, "EURUSD", "M1") == "rate_EURUSD__1"
def test_invalid_granularity_propagates_value_error(self, tmp_path: Path) -> None:
"""Test invalid granularities raise ValueError from parse_timeframe."""
with pytest.raises(ValueError, match="Invalid timeframe"):
resolve_rate_view_name(tmp_path / "unused.db", "EURUSD", "BAD")
with pytest.raises(ValueError, match="Invalid timeframe"):
resolve_rate_view_names(tmp_path / "unused.db", ["EURUSD"], ["BAD"])
def test_resolve_rate_view_names_for_multiple_pairs(self, tmp_path: Path) -> None:
"""Test batch resolution returns row-major symbol/granularity pairs."""
db_path = tmp_path / "batch-resolve.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.executemany(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
[
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
("EURUSD", TIMEFRAME_MAP["H1"], "2024-01-01T01:00:00+00:00", 1.1),
("GBPUSD", 1, "2024-01-01T00:00:00+00:00", 1.2),
],
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_names(
db_path,
["EURUSD", "GBPUSD"],
["M1", "H1"],
) == [
"rate_EURUSD__M1_1",
"rate_EURUSD__H1_16385",
"rate_GBPUSD__1",
"rate_GBPUSD__16385",
]
@pytest.mark.parametrize(
"symbol",
["EUR/USD", "US500.cash", "#US500"],
)
def test_supports_broker_specific_symbols(
self,
tmp_path: Path,
symbol: str,
) -> None:
"""Test broker-specific symbols resolve to safely created view names."""
db_path = tmp_path / "broker-symbol-resolve.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
(symbol, 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_name(db_path, symbol, "M1") == build_rate_view_name(
symbol=symbol,
granularity="M1",
granularity_count=1,
timeframe=1,
)
def test_accepts_open_sqlite_connection(self, tmp_path: Path) -> None:
"""Test resolver accepts an already-open SQLite connection."""
db_path = tmp_path / "open-connection.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
assert resolve_rate_view_name(conn, "EURUSD", "M1") == "rate_EURUSD__1"
def test_require_existing_raises_when_database_missing(
self,
tmp_path: Path,
) -> None:
"""Test strict mode rejects missing database paths."""
db_path = tmp_path / "missing.db"
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_name(
db_path,
"EURUSD",
"M1",
require_existing=True,
)
with pytest.raises(ValueError, match="SQLite database not found"):
resolve_rate_view_names(
db_path,
["EURUSD"],
["M1"],
require_existing=True,
)
def test_require_existing_raises_when_view_missing(self, tmp_path: Path) -> None:
"""Test strict mode rejects databases without matching rate views."""
db_path = tmp_path / "no-view.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(symbol TEXT, time TEXT)")
with pytest.raises(ValueError, match="No rate compatibility view exists"):
resolve_rate_view_name(
db_path,
"EURUSD",
"M1",
require_existing=True,
)
with pytest.raises(ValueError, match="No rate compatibility view exists"):
resolve_rate_view_names(
db_path,
["EURUSD"],
["M1"],
require_existing=True,
)
def test_require_existing_returns_existing_view(self, tmp_path: Path) -> None:
"""Test strict mode returns a view when one exists."""
db_path = tmp_path / "existing-view.db"
with sqlite3.connect(db_path) as conn:
conn.execute(
"CREATE TABLE rates("
" symbol TEXT, timeframe INTEGER, time TEXT, close REAL)",
)
conn.execute(
"INSERT INTO rates(symbol, timeframe, time, close) VALUES (?, ?, ?, ?)",
("EURUSD", 1, "2024-01-01T00:00:00+00:00", 1.0),
)
create_rate_compatibility_views(conn)
assert (
resolve_rate_view_name(
db_path,
"EURUSD",
"M1",
require_existing=True,
)
== "rate_EURUSD__1"
)
assert resolve_rate_view_names(
db_path,
["EURUSD"],
["M1"],
require_existing=True,
) == ["rate_EURUSD__1"]
class TestQuoteSqliteIdentifier:
"""Tests for quote_sqlite_identifier."""
@@ -61,6 +340,147 @@ class TestQuoteSqliteIdentifier:
assert quoted.endswith('"')
class TestLoadRateData:
"""Tests for SQLite rate-like table and view loading."""
def test_loads_close_rates_from_path_with_count(self, tmp_path: Path) -> None:
"""Test loading the latest close-based rates in ascending time order."""
db_path = tmp_path / "rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.executemany(
"INSERT INTO rates(time, close) VALUES (?, ?)",
[
("2024-01-01T00:00:00+00:00", 1.0),
("2024-01-01T00:02:00+00:00", 1.2),
("2024-01-01T00:01:00+00:00", 1.1),
],
)
frame = load_rate_data(db_path, "rates", count=2)
assert list(frame["close"]) == [1.1, 1.2]
assert isinstance(frame.index, pd.DatetimeIndex)
assert frame.index.name == "time"
assert frame.index.is_monotonic_increasing
def test_loads_ask_bid_tick_like_rates_from_connection(
self,
tmp_path: Path,
) -> None:
"""Test loading tick-like tables with bid and ask columns."""
db_path = tmp_path / "ticks.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE ticks(time TEXT, bid REAL, ask REAL)")
conn.execute(
"INSERT INTO ticks(time, bid, ask) VALUES (?, ?, ?)",
("2024-01-01T00:00:00+00:00", 1.0, 1.1),
)
frame = load_rate_data_from_connection(conn, "ticks")
path_frame = load_rate_data(conn, "ticks")
assert frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
assert path_frame.iloc[0].to_dict() == {"bid": 1.0, "ask": 1.1}
def test_loads_from_view(self, tmp_path: Path) -> None:
"""Test loading from a SQLite view."""
db_path = tmp_path / "view.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.execute(
"INSERT INTO rates(time, close) VALUES (?, ?)",
("2024-01-01T00:00:00+00:00", 1.0),
)
conn.execute("CREATE VIEW rate_view AS SELECT time, close FROM rates")
frame = load_rate_data_from_connection(conn, "rate_view")
assert list(frame["close"]) == [1.0]
def test_loads_quoted_identifier(self, tmp_path: Path) -> None:
"""Test table names are quoted safely."""
db_path = tmp_path / "quoted.db"
table = 'rate "quoted"'
quoted = quote_sqlite_identifier(table)
with sqlite3.connect(db_path) as conn:
conn.execute(f"CREATE TABLE {quoted}(time TEXT, close REAL)")
conn.execute(
f"INSERT INTO {quoted}(time, close) VALUES (?, ?)", # noqa: S608
("2024-01-01T00:00:00+00:00", 1.0),
)
frame = load_rate_data_from_connection(conn, table)
assert list(frame["close"]) == [1.0]
def test_rejects_missing_database_and_non_file(self, tmp_path: Path) -> None:
"""Test path validation for SQLite database inputs."""
with pytest.raises(ValueError, match="SQLite database not found"):
load_rate_data(tmp_path / "missing.db", "rates")
with pytest.raises(ValueError, match="not a file"):
load_rate_data(tmp_path, "rates")
@pytest.mark.parametrize(
("table", "count", "match"),
[
("", None, "must not be empty"),
("rates", 0, "count must be positive"),
("rates", -1, "count must be positive"),
],
)
def test_rejects_invalid_inputs(
self,
tmp_path: Path,
table: str,
count: int | None,
match: str,
) -> None:
"""Test request validation."""
db_path = tmp_path / "invalid-inputs.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
with pytest.raises(ValueError, match=match):
load_rate_data_from_connection(conn, table, count=count)
@pytest.mark.parametrize(
("ddl", "match"),
[
("CREATE TABLE rates(time TEXT, close REAL)", "contains no rows"),
("CREATE TABLE rates(close REAL)", "time column"),
("CREATE TABLE rates(time TEXT, open REAL)", "close, or both ask and bid"),
],
)
def test_rejects_invalid_tables(
self,
tmp_path: Path,
ddl: str,
match: str,
) -> None:
"""Test missing table, empty table, and invalid schemas."""
db_path = tmp_path / "invalid-tables.db"
with sqlite3.connect(db_path) as conn:
conn.execute(ddl)
with pytest.raises(ValueError, match=match):
load_rate_data_from_connection(conn, "rates")
with pytest.raises(ValueError, match="not found"):
load_rate_data_from_connection(conn, "missing")
def test_rejects_invalid_timestamp(self, tmp_path: Path) -> None:
"""Test unparsable timestamps fail clearly."""
db_path = tmp_path / "invalid-time.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time TEXT, close REAL)")
conn.execute("INSERT INTO rates(time, close) VALUES (?, ?)", ("bad", 1.0))
with pytest.raises(ValueError, match="unparsable time"):
load_rate_data_from_connection(conn, "rates")
def test_loads_numeric_mt5_epoch_seconds(self, tmp_path: Path) -> None:
"""Test MT5-native integer timestamps are parsed as epoch seconds."""
db_path = tmp_path / "epoch-rates.db"
with sqlite3.connect(db_path) as conn:
conn.execute("CREATE TABLE rates(time INTEGER, close REAL)")
conn.execute(
"INSERT INTO rates(time, close) VALUES (?, ?)",
(1_704_067_200, 1.0),
)
frame = load_rate_data_from_connection(conn, "rates")
assert frame.index[0] == pd.Timestamp("2024-01-01", tz="UTC")
assert list(frame["close"]) == [1.0]
class TestResolveHistorySettings:
"""Tests for history dataset and timeframe resolution."""
@@ -1024,7 +1444,7 @@ class TestIncrementalIntegration:
sqlite3.connect(tmp_path / "no-keys.db") as conn,
caplog.at_level(
logging.WARNING,
logger="mt5cli.sqlite_history",
logger="mt5cli.history",
),
):
deduplicate_history_tables(conn, {Dataset.ticks: {"time"}}, {Dataset.ticks})
@@ -1059,7 +1479,7 @@ class TestIncrementalIntegration:
client = MagicMock()
client.copy_rates_range_as_df.return_value = pd.DataFrame()
with (
caplog.at_level(logging.WARNING, logger="mt5cli.sqlite_history"),
caplog.at_level(logging.WARNING, logger="mt5cli.history"),
sqlite3.connect(tmp_path / "views-warning.db") as conn,
):
write_incremental_datasets(
+444 -4
View File
@@ -4,9 +4,9 @@ from __future__ import annotations
import logging
import sqlite3
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, NamedTuple, cast
from unittest.mock import MagicMock, call
import pandas as pd
import pytest
@@ -15,12 +15,16 @@ from pytest_mock import MockerFixture # noqa: TC002
if TYPE_CHECKING:
from pathlib import Path
from pdmt5 import Mt5DataClient
from mt5cli import sdk
from mt5cli.history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.sdk import (
Mt5CliClient,
account_info,
build_config,
collect_history,
collect_latest_rates,
copy_rates_from,
copy_rates_from_pos,
copy_rates_range,
@@ -29,9 +33,15 @@ from mt5cli.sdk import (
history_deals,
history_orders,
last_error,
latest_rates,
market_book,
minimum_margins,
mt5_summary,
mt5_summary_as_df,
orders,
positions,
recent_history_deals,
recent_ticks,
symbol_info,
symbol_info_tick,
symbols,
@@ -40,9 +50,34 @@ from mt5cli.sdk import (
update_history_with_config,
version,
)
from mt5cli.sqlite_history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.utils import Dataset
class _TerminalInfo(NamedTuple):
connected: bool
path: str
class _AccountInfo(NamedTuple):
login: int
limits: dict[str, object]
class _MissingSummaryMethodClient:
def version(self) -> tuple[int, int, int]:
return (5, 0, 1)
def terminal_info(self) -> dict[str, bool]:
return {"connected": True}
def symbols_total(self) -> int:
return 42
class _NonCallableSummaryMethodClient:
version = (5, 0, 1)
_DEALS_FIXTURE: dict[str, list[object]] = {
"ticket": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
"position_id": [100, 100, 100, 0, 200, 200, 300, 400, 400, 500, 500, 600, 600, 600],
@@ -230,6 +265,32 @@ class TestConnectionLifecycle:
client = Mt5CliClient()
client.__exit__(None, None, None)
def test_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test injected connected clients are not initialized or shut down."""
connected = MagicMock()
connected.account_info_as_df.return_value = pd.DataFrame({"a": [1]})
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
with Mt5CliClient.from_connected_client(connected) as client:
result = client.account_info()
assert result.to_dict("list") == {"a": [1]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
connected.account_info_as_df.assert_called_once()
after_exit = client.terminal_info()
assert after_exit.to_dict("list") == {"b": [2]}
connected.terminal_info_as_df.assert_called_once()
def test_constructor_injected_client_is_reused_and_not_shutdown(self) -> None:
"""Test constructor injection has the same non-owning lifecycle."""
connected = MagicMock()
connected.terminal_info_as_df.return_value = pd.DataFrame({"b": [2]})
client = Mt5CliClient(client=connected)
with client:
result = client.terminal_info()
assert result.to_dict("list") == {"b": [2]}
connected.initialize_and_login_mt5.assert_not_called()
connected.shutdown.assert_not_called()
class TestModuleFunctions:
"""Tests for module-level SDK wrappers."""
@@ -269,6 +330,7 @@ class TestModuleFunctions:
(last_error, (), "last_error_as_df"),
(symbol_info_tick, ("EURUSD",), "symbol_info_tick_as_df"),
(market_book, ("EURUSD",), "market_book_get_as_df"),
(latest_rates, ("EURUSD", "M1", 10), "copy_rates_from_pos_as_df"),
],
)
def test_module_functions_delegate(
@@ -352,6 +414,214 @@ class TestMt5CliClient:
assert isinstance(df, pd.DataFrame)
mock_client.copy_rates_range_as_df.assert_called_once()
def test_latest_rates_delegates_to_copy_rates_from_pos(
self,
mock_client: MagicMock,
) -> None:
"""Test latest_rates is a convenience wrapper for positional rates."""
Mt5CliClient().latest_rates("EURUSD", "M1", 5, start_pos=2)
mock_client.copy_rates_from_pos_as_df.assert_called_once_with(
symbol="EURUSD",
timeframe=1,
start_pos=2,
count=5,
)
def test_latest_rates_rejects_non_positive_count(self) -> None:
"""Test latest_rates validates count."""
with pytest.raises(ValueError, match="count must be positive"):
Mt5CliClient().latest_rates("EURUSD", "M1", 0)
def test_collect_latest_rates_returns_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test multi-target latest rate collection."""
result = collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
assert set(result) == {
("EURUSD", 1),
("EURUSD", 16385),
("GBPUSD", 1),
("GBPUSD", 16385),
}
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
def test_collect_latest_rates_uses_single_transient_connection(
self,
mock_client: MagicMock,
mocker: MockerFixture,
) -> None:
"""Test module helper opens one connection for all target pairs."""
mt5_data_client = mocker.patch(
"mt5cli.sdk.Mt5DataClient",
return_value=mock_client,
)
collect_latest_rates(["EURUSD", "GBPUSD"], ["M1", "H1"], count=3)
mt5_data_client.assert_called_once()
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
assert mock_client.copy_rates_from_pos_as_df.call_count == 4
mock_client.copy_rates_from_pos_as_df.assert_has_calls(
[
call(symbol="EURUSD", timeframe=1, start_pos=0, count=3),
call(symbol="EURUSD", timeframe=16385, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=1, start_pos=0, count=3),
call(symbol="GBPUSD", timeframe=16385, start_pos=0, count=3),
],
)
@pytest.mark.parametrize(
("symbols", "timeframes", "match"),
[
([], ["M1"], "At least one symbol"),
(["EURUSD"], [], "At least one timeframe"),
],
)
def test_collect_latest_rates_rejects_empty_inputs(
self,
symbols: list[str],
timeframes: list[str],
match: str,
) -> None:
"""Test multi-target latest rate input validation."""
with pytest.raises(ValueError, match=match):
Mt5CliClient().collect_latest_rates(symbols, timeframes, count=1)
def test_recent_history_deals_uses_trailing_window(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals calculates date_from from hours."""
result = recent_history_deals(
6,
date_to="2024-01-02T00:00:00+00:00",
group="*",
symbol="EURUSD",
)
assert isinstance(result, pd.DataFrame)
mock_client.history_deals_get_as_df.assert_called_once_with(
date_from=datetime(2024, 1, 1, 18, tzinfo=UTC),
date_to=datetime(2024, 1, 2, tzinfo=UTC),
group="*",
symbol="EURUSD",
ticket=None,
position=None,
)
def test_recent_history_deals_defaults_date_to_now(
self,
mock_client: MagicMock,
) -> None:
"""Test recent_history_deals uses current UTC time when date_to is omitted."""
before = datetime.now(UTC)
recent_history_deals(1.0)
after = datetime.now(UTC)
call_kwargs = mock_client.history_deals_get_as_df.call_args.kwargs
assert before <= call_kwargs["date_to"] <= after
assert call_kwargs["date_from"] == call_kwargs["date_to"] - timedelta(hours=1)
def test_recent_history_deals_rejects_non_positive_hours(self) -> None:
"""Test recent_history_deals validates hours."""
with pytest.raises(ValueError, match="hours must be positive"):
Mt5CliClient().recent_history_deals(0)
def test_mt5_summary_returns_status_mapping(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary calls raw terminal/account status methods."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = {"connected": True}
mock_client.account_info.return_value = {"login": 123}
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True},
"account_info": {"login": 123},
"symbols_total": 42,
}
def test_mt5_summary_normalizes_namedtuple_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary returns structured plain Python values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
assert mt5_summary() == {
"version": [5, 0, 1],
"terminal_info": {"connected": True, "path": "terminal.exe"},
"account_info": {
"login": 123,
"limits": {"modes": ["netting", "hedging"], "servers": ["demo"]},
},
"symbols_total": 42,
}
def test_mt5_summary_as_df_stringifies_nested_values(
self,
mock_client: MagicMock,
) -> None:
"""Test mt5_summary_as_df returns export-safe tabular values."""
mock_client.version.return_value = (5, 0, 1)
mock_client.terminal_info.return_value = _TerminalInfo(
connected=True,
path="terminal.exe",
)
mock_client.account_info.return_value = _AccountInfo(
login=123,
limits={"modes": ("netting", "hedging"), "servers": ["demo"]},
)
mock_client.symbols_total.return_value = 42
result = mt5_summary_as_df()
assert len(result) == 1
assert result.iloc[0].to_dict() == {
"version": "[5,0,1]",
"terminal_info": '{"connected":true,"path":"terminal.exe"}',
"account_info": (
'{"limits":{"modes":["netting","hedging"],'
'"servers":["demo"]},"login":123}'
),
"symbols_total": 42,
}
def test_mt5_summary_missing_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is missing."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _MissingSummaryMethodClient()),
)
with pytest.raises(
AttributeError,
match="MT5 client is missing required method: account_info",
):
client.mt5_summary()
def test_mt5_summary_non_callable_method_raises_clear_error(self) -> None:
"""Test mt5_summary fails clearly when a required method is not callable."""
client = Mt5CliClient(
client=cast("Mt5DataClient", _NonCallableSummaryMethodClient()),
)
with pytest.raises(
TypeError,
match="MT5 client attribute is not callable: version",
):
client.mt5_summary()
class TestCollectHistory:
"""Tests for collect_history SDK function."""
@@ -808,3 +1078,173 @@ class TestUpdateHistory:
)
after = datetime.now(UTC)
assert before <= captured["end"] <= after
class TestRecentTicks:
"""Tests for recent_ticks helper."""
def test_recent_ticks_uses_explicit_date_to_window(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks fetches the requested trailing window."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [end],
"bid": [1.0],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=100,
flags="INFO",
config=build_config(login=123),
)
assert isinstance(result, pd.DataFrame)
client.copy_ticks_from_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
count=100,
flags=2,
)
client.copy_ticks_range_as_df.assert_not_called()
def test_recent_ticks_uses_latest_tick_when_date_to_omitted(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks anchors the window on the latest tick time."""
client = MagicMock()
tick = MagicMock()
tick.time = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.symbol_info_tick.return_value = tick
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [1, 2],
"bid": [1.0, 1.1],
})
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = Mt5CliClient().recent_ticks("EURUSD", 30, count=2, flags="ALL")
assert len(result) == 2
client.symbol_info_tick.assert_called_once_with("EURUSD")
client.copy_ticks_from_as_df.assert_called_once()
_, kwargs = client.copy_ticks_range_as_df.call_args
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_to"] == tick.time
assert kwargs["date_from"] == tick.time - timedelta(seconds=30)
assert kwargs["flags"] == 1
def test_recent_ticks_rejects_unsupported_tick_time(
self,
mocker: MockerFixture,
) -> None:
"""Test recent_ticks raises when the latest tick time is unsupported."""
client = MagicMock()
tick = MagicMock()
tick.time = object()
client.symbol_info_tick.return_value = tick
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
with pytest.raises(TypeError, match="Unsupported tick time value"):
Mt5CliClient().recent_ticks("EURUSD", 30)
@pytest.mark.parametrize(
"tick_time",
[
"2024-01-02T12:00:00+00:00",
1704196800,
],
)
def test_recent_ticks_coerces_string_and_unix_tick_times(
self,
mocker: MockerFixture,
tick_time: str | int,
) -> None:
"""Test recent_ticks accepts string and unix tick timestamps."""
client = MagicMock()
tick = MagicMock()
tick.time = tick_time
client.symbol_info_tick.return_value = tick
expected_end = (
datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
if isinstance(tick_time, str)
else datetime.fromtimestamp(tick_time, tz=UTC)
)
client.copy_ticks_from_as_df.return_value = pd.DataFrame({
"time": [expected_end],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
Mt5CliClient().recent_ticks("EURUSD", 30)
_, kwargs = client.copy_ticks_from_as_df.call_args
assert kwargs["date_from"] == expected_end - timedelta(seconds=30)
def test_recent_ticks_returns_full_frame_when_count_not_positive(
self,
mocker: MockerFixture,
) -> None:
"""Test non-positive count returns the full range without trimming."""
client = MagicMock()
end = datetime(2024, 1, 2, 12, 0, 0, tzinfo=UTC)
client.copy_ticks_range_as_df.return_value = pd.DataFrame({
"time": [1, 2, 3],
"bid": [1.0, 1.1, 1.2],
})
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = recent_ticks(
"EURUSD",
60,
date_to=end,
count=0,
config=build_config(login=123),
)
assert len(result) == 3
client.copy_ticks_from_as_df.assert_not_called()
client.copy_ticks_range_as_df.assert_called_once_with(
symbol="EURUSD",
date_from=end - timedelta(seconds=60),
date_to=end,
flags=1,
)
class TestMinimumMargins:
"""Tests for minimum_margins helper."""
def test_minimum_margins_shape(
self,
mocker: MockerFixture,
) -> None:
"""Test minimum_margins returns the expected summary columns."""
client = MagicMock()
sym = MagicMock(volume_min=0.01)
account = MagicMock(currency="USD")
tick = MagicMock(ask=1.1010, bid=1.1000)
client.symbol_info.return_value = sym
client.account_info.return_value = account
client.symbol_info_tick.return_value = tick
client.order_calc_margin.side_effect = [12.5, 12.4]
client.mt5.ORDER_TYPE_BUY = 0
client.mt5.ORDER_TYPE_SELL = 1
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
result = minimum_margins("EURUSD", config=build_config(login=123))
pd.testing.assert_frame_equal(
result,
pd.DataFrame([
{
"symbol": "EURUSD",
"account_currency": "USD",
"volume_min": 0.01,
"buy_margin": 12.5,
"sell_margin": 12.4,
}
]),
)
client.order_calc_margin.assert_any_call(0, "EURUSD", 0.01, 1.1010)
client.order_calc_margin.assert_any_call(1, "EURUSD", 0.01, 1.1000)
+108
View File
@@ -21,8 +21,10 @@ from mt5cli.utils import (
TIMEFRAME_MAP,
TIMEFRAME_TYPE,
Dataset,
IfExists,
detect_format,
export_dataframe,
export_dataframe_to_sqlite,
parse_datetime,
parse_request,
parse_tick_flags,
@@ -130,6 +132,112 @@ class TestExportDataframe:
export_dataframe(sample_df, tmp_path / "out.txt", "xml")
class TestExportDataframeToSqlite:
"""Tests for export_dataframe_to_sqlite."""
def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
"""Test append mode keeps prior rows in the SQLite table."""
output = tmp_path / "append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
"""Test deduplication keeps the latest ROWID for key columns."""
output = tmp_path / "dedup.db"
first = pd.DataFrame({
"symbol": ["EURUSD", "EURUSD"],
"time": ["2024-01-01", "2024-01-01"],
"bid": [1.0, 1.1],
})
second = pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
})
export_dataframe_to_sqlite(
first,
output,
"ticks",
if_exists=IfExists.REPLACE,
deduplicate_on=("symbol", "time"),
)
export_dataframe_to_sqlite(
second,
output,
"ticks",
if_exists=IfExists.APPEND,
deduplicate_on=("symbol", "time"),
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, time, bid FROM ticks",
conn,
)
pd.testing.assert_frame_equal(
result.reset_index(drop=True),
pd.DataFrame({
"symbol": ["EURUSD"],
"time": ["2024-01-01"],
"bid": [1.2],
}),
)
def test_default_if_exists_appends_without_dropping_rows(
self,
tmp_path: Path,
) -> None:
"""Test the default append mode keeps prior rows."""
output = tmp_path / "default-append.db"
first = pd.DataFrame({"id": [1], "value": ["a"]})
second = pd.DataFrame({"id": [2], "value": ["b"]})
export_dataframe_to_sqlite(first, output, "items")
export_dataframe_to_sqlite(second, output, "items")
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT id, value FROM items ORDER BY id",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
)
def test_writes_index_with_label(self, tmp_path: Path) -> None:
"""Test optional index export with a custom label."""
output = tmp_path / "index.db"
frame = pd.DataFrame(
{"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
)
export_dataframe_to_sqlite(
frame,
output,
"margins",
if_exists=IfExists.REPLACE,
index=True,
index_label="symbol",
)
with sqlite3.connect(output) as conn:
result = pd.read_sql( # type: ignore[reportUnknownMemberType]
"SELECT symbol, value FROM margins",
conn,
)
pd.testing.assert_frame_equal(
result,
pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
)
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
Generated
+1 -1
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.1"
version = "0.5.0"
source = { editable = "." }
dependencies = [
{ name = "click" },