gavindiaz/mt5cli
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add incremental SQLite history SDK (#16)

Browse Source 
* Add incremental SQLite history SDK for automated pipelines.

Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental history deals and stale rate view cleanup.

Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental deal filtering edge cases

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for incremental SQLite history.

Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document collect-history SQLite schema with ER diagram.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix account-event filtering and drop legacy rates resume.

Account events must follow only account_event_start, not per-symbol trade
cursors. Require normalized rates schema and fail fast when timeframe is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Validate normalized rates schema before incremental resume.

Require symbol, timeframe, and time on existing rates tables with clear
ValueError messages, and add regression tests for malformed schemas.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Daichi Narushima
2026-06-09 01:28:22 +09:00
committed by GitHub
parent 5b44318d55
commit c4232bf44d
12 changed files with 3427 additions and 335 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+40 -1
View File
@@ -87,7 +87,46 @@ mt5cli -o history.db collect-history \
--timeframe M1 --flags ALL --if-exists append --with-views
```
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing deals, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns and do not contribute to the weighted prices.
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
### Incremental history SDK
For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
```python
from pdmt5 import Mt5Config, Mt5DataClient
from mt5cli import Dataset, update_history, update_history_with_config
# Reuse an already-connected pdmt5 client (does not open/close MT5)
client = Mt5DataClient(config=Mt5Config(login=12345))
client.initialize_and_login_mt5()
try:
update_history(
client=client,
output="history.db",
symbols=["EURUSD", "GBPUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1", "H1"], # default: all fixed MT5 timeframes
lookback_hours=24,
create_rate_views=True,
with_views=True,
include_account_events=True,
)
finally:
client.shutdown()
# Standalone wrapper that opens and closes MT5 for you
update_history_with_config(
output="history.db",
symbols=["EURUSD"],
config=Mt5Config(login=12345),
)
```
- **`collect-history`**: explicit date-range export into SQLite.
- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
## Requirements
docs/api/sqlite_history.md
+131
View File
@@ -0,0 +1,131 @@
# SQLite History Module
::: mt5cli.sqlite_history
## `collect-history` schema
The `collect-history` command (and the matching `collect_history` SDK function) writes
selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
- `symbol` is prepended on every table.
- `timeframe` is prepended on `rates` so appended runs at different bar sizes stay
distinguishable.
SQLite does not declare foreign keys. Rows are linked logically by `symbol`, time
windows, and (for deals) `position_id` / `order`. Duplicate rows are removed on
append using dataset-specific keys (for example `ticket` on history tables, or
`(symbol, timeframe, time)` on rates).
Optional views are created when `--with-views` is set and the `history-deals` dataset
was written.
### Entity-relationship diagram
Sample layout for a full collection with `--with-views`:
```mermaid
erDiagram
rates {
TEXT symbol "dedup key"
INTEGER timeframe "dedup key"
TEXT time "dedup key"
REAL open
REAL high
REAL low
REAL close
INTEGER tick_volume
INTEGER spread
INTEGER real_volume
}
ticks {
TEXT symbol "dedup key"
TEXT time "dedup key"
INTEGER time_msc "dedup key (preferred)"
REAL bid
REAL ask
REAL last
INTEGER volume
INTEGER flags
REAL volume_real
}
history_orders {
INTEGER ticket "dedup key"
TEXT symbol
TEXT time
INTEGER type
INTEGER state
REAL volume_initial
REAL price_open
REAL price_current
INTEGER magic
}
history_deals {
INTEGER ticket "dedup key"
INTEGER order
INTEGER position_id "groups position view"
TEXT symbol
TEXT time
INTEGER type "0/1 trade, else cash event"
INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
REAL volume
REAL price
REAL profit
REAL commission
REAL swap
REAL fee
}
cash_events {
INTEGER ticket
TEXT symbol
TEXT time
INTEGER type
REAL profit
}
positions_reconstructed {
INTEGER position_id
TEXT symbol
TEXT open_time
TEXT close_time
INTEGER direction
REAL volume_open
REAL volume_close
REAL volume_reversal
REAL open_price
REAL close_price
REAL total_profit
INTEGER reversal_count
INTEGER deals_count
}
rates ||--o{ history_deals : "symbol (logical)"
ticks ||--o{ history_deals : "symbol (logical)"
history_orders ||--o{ history_deals : "order ~ ticket (logical)"
history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
```
### Tables and views
| Object | Kind | Source | Notes |
| ------------------------- | ----- | -------------------- | ------------------------------------------------------------------------------------------- |
| `rates` | table | `copy_rates_range` | Indexed on `(symbol, timeframe, time)` when columns exist. |
| `ticks` | table | `copy_ticks_range` | Indexed on `(symbol, time)` when columns exist. |
| `history_orders` | table | `history_orders_get` | Fetched per `--symbol`, then concatenated. |
| `history_deals` | table | `history_deals_get` | Fetched per `--symbol`, then concatenated. Indexed on `(position_id, symbol)` when present. |
| `cash_events` | view | `history_deals` | Non-trade deal types (deposits, balance ops, etc.). Requires `type` column. |
| `positions_reconstructed` | view | `history_deals` | One row per closed `position_id`; volume-weighted prices and reversal stats. |
Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning
when required columns are missing.
### Incremental collection
The `update_history` SDK path uses the same base tables and optional
`cash_events` / `positions_reconstructed` views. It additionally maintains
`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
docs/index.md
+2
View File
@@ -152,6 +152,8 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
See the [SQLite History schema diagram](api/sqlite_history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
## Global Options
| Option | Description |
mkdocs.yml
+2
View File
@@ -24,6 +24,7 @@ theme:
features:
- content.code.annotate
- content.code.copy
- content.code.mermaid
- navigation.indexes
- navigation.sections
- navigation.tabs
@@ -57,6 +58,7 @@ nav:
- Overview: api/index.md
- CLI: api/cli.md
- SDK: api/sdk.md
- SQLite History: api/sqlite_history.md
- Utils: api/utils.md
markdown_extensions:
mt5cli/__init__.py
+7 -1
View File
@@ -22,15 +22,19 @@ from .sdk import (
symbol_info_tick,
symbols,
terminal_info,
update_history,
update_history_with_config,
)
from .sdk import (
version as mt5_version,
)
from .utils import detect_format, export_dataframe
from .utils import Dataset, IfExists, detect_format, export_dataframe
__version__ = version(__package__) if __package__ else None
__all__ = [
"Dataset",
"IfExists",
"Mt5CliClient",
"account_info",
"build_config",
@@ -53,4 +57,6 @@ __all__ = [
"symbol_info_tick",
"symbols",
"terminal_info",
"update_history",
"update_history_with_config",
]
mt5cli/sdk.py
+222 -328
View File
@@ -5,12 +5,23 @@ from __future__ import annotations
import logging
import sqlite3
from contextlib import contextmanager
from datetime import datetime
from pathlib import Path # noqa: TC003
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import TYPE_CHECKING, Self, TypeVar
from pdmt5 import Mt5Config, Mt5DataClient
from .sqlite_history import (
create_cash_events_view,
create_history_indexes,
create_positions_reconstructed_view,
resolve_history_datasets,
resolve_history_tick_flags,
resolve_history_timeframes,
write_collected_datasets,
write_incremental_datasets,
)
from .utils import (
Dataset,
IfExists,
@@ -20,7 +31,7 @@ from .utils import (
)
if TYPE_CHECKING:
from collections.abc import Callable, Iterator
from collections.abc import Callable, Iterator, Sequence
import pandas as pd
@@ -48,22 +59,11 @@ __all__ = [
"symbol_info_tick",
"symbols",
"terminal_info",
"update_history",
"update_history_with_config",
"version",
]
_TRADE_DEAL_TYPES: tuple[int, int] = (0, 1)
_TRADE_DEAL_TYPES_SQL = f"({', '.join(str(value) for value in _TRADE_DEAL_TYPES)})"
_POSITIONS_VIEW_REQUIRED_COLUMNS: frozenset[str] = frozenset({
"position_id",
"symbol",
"time",
"type",
"entry",
"volume",
"price",
"profit",
})
def _coerce_timeframe(timeframe: int | str) -> int:
if isinstance(timeframe, int):
@@ -419,325 +419,219 @@ class Mt5CliClient:
return self._fetch(lambda c: c.market_book_get_as_df(symbol=symbol))
def _create_cash_events_view(
conn: sqlite3.Connection,
deals_columns: set[str],
) -> bool:
"""Create the cash_events SQLite view derived from history_deals.
def _resolve_incremental_settings(
selected_datasets: set[Dataset],
timeframes: Sequence[int | str] | None,
flags: int | str,
) -> tuple[list[int], int]:
"""Resolve dataset-specific incremental update settings.
Returns:
True if the view was created, False if required columns are missing.
Tuple of resolved rate timeframes and tick copy flags.
Raises:
ValueError: If timeframe or tick flag values are invalid.
"""
if "type" not in deals_columns:
logger.warning("Skipping cash_events view: history_deals.type is missing")
return False
conn.execute("DROP VIEW IF EXISTS cash_events")
conn.execute(
"CREATE VIEW cash_events AS" # noqa: S608
f" SELECT * FROM history_deals WHERE type NOT IN {_TRADE_DEAL_TYPES_SQL}",
)
return True
resolved_timeframes: list[int] = []
if Dataset.rates in selected_datasets:
try:
resolved_timeframes = resolve_history_timeframes(timeframes)
except ValueError as exc:
msg = str(exc)
raise ValueError(msg) from exc
resolved_tick_flags = 0
if Dataset.ticks in selected_datasets:
try:
resolved_tick_flags = resolve_history_tick_flags(flags)
except ValueError as exc:
msg = str(exc)
raise ValueError(msg) from exc
return resolved_timeframes, resolved_tick_flags
def _create_positions_reconstructed_view(
conn: sqlite3.Connection,
deals_columns: set[str],
) -> bool:
"""Create the positions_reconstructed SQLite view derived from history_deals.
@dataclass(frozen=True)
class _UpdateHistoryRequest:
selected: set[Dataset]
end: datetime
fallback_start: datetime
resolved_timeframes: list[int]
resolved_tick_flags: int
output_path: Path
def _resolve_update_history_request(
*,
output: Path | str,
symbols: Sequence[str],
datasets: set[Dataset] | None,
timeframes: Sequence[int | str] | None,
flags: int | str,
lookback_hours: float,
date_to: datetime | str | None,
) -> _UpdateHistoryRequest | None:
"""Validate and resolve incremental history update inputs.
Returns:
True if the view was created, False if required columns are missing.
Resolved request parameters, or None when no datasets are selected.
Raises:
ValueError: If symbols are empty, lookback_hours is not positive, or
timeframe/flag values are invalid.
"""
if not _POSITIONS_VIEW_REQUIRED_COLUMNS.issubset(deals_columns):
missing = ", ".join(sorted(_POSITIONS_VIEW_REQUIRED_COLUMNS - deals_columns))
logger.warning(
"Skipping positions_reconstructed view: history_deals missing columns: %s",
missing,
)
return False
conn.execute("DROP VIEW IF EXISTS positions_reconstructed")
conn.execute(
"CREATE VIEW positions_reconstructed AS" # noqa: S608
" SELECT"
" position_id,"
" symbol,"
" MIN(CASE WHEN entry = 0 THEN time END) AS open_time,"
" MAX(CASE WHEN entry IN (1, 2, 3) THEN time END) AS close_time,"
" MIN(CASE WHEN entry = 0 THEN type END) AS direction,"
" SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END) AS volume_open,"
" SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END) AS volume_close,"
" SUM(CASE WHEN entry = 2 THEN volume ELSE 0 END) AS volume_reversal,"
" CASE"
" WHEN SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END) > 0"
" THEN SUM(CASE WHEN entry = 0 THEN price * volume ELSE 0 END)"
" / SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END)"
" END AS open_price,"
" CASE"
" WHEN SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END) > 0"
" THEN SUM(CASE WHEN entry IN (1, 3) THEN price * volume ELSE 0 END)"
" / SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END)"
" END AS close_price,"
" SUM(profit) AS total_profit,"
" SUM(CASE WHEN entry = 2 THEN 1 ELSE 0 END) AS reversal_count,"
" COUNT(*) AS deals_count"
" FROM history_deals"
f" WHERE type IN {_TRADE_DEAL_TYPES_SQL} AND position_id != 0"
" GROUP BY position_id, symbol"
" HAVING SUM(CASE WHEN entry IN (1, 3) THEN 1 ELSE 0 END) > 0",
)
return True
if lookback_hours <= 0:
msg = "lookback_hours must be positive."
raise ValueError(msg)
selected = resolve_history_datasets(datasets)
if not selected:
logger.info("Skipping SQLite history update: no datasets selected.")
return None
if not symbols:
msg = "At least one symbol is required."
raise ValueError(msg)
def _write_frame_to_sqlite(
conn: sqlite3.Connection,
frame: pd.DataFrame,
table_name: str,
if_exists: IfExists,
) -> bool:
"""Write a non-empty-schema frame to SQLite.
Returns:
True if a table was written, False if the frame had no columns.
"""
if len(frame.columns) == 0:
logger.warning("Skipping %s: dataset returned no columns", table_name)
return False
frame.to_sql( # type: ignore[reportUnknownMemberType]
table_name,
conn,
if_exists=if_exists.value,
index=False,
chunksize=50_000,
method="multi",
)
return True
def _create_collect_history_indexes(
conn: sqlite3.Connection,
written_columns: dict[Dataset, set[str]],
) -> None:
"""Create useful indexes for collected history tables when present."""
if {"symbol", "time"}.issubset(written_columns.get(Dataset.rates, set())):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_rates_symbol_time ON rates(symbol, time)",
)
if {"symbol", "time"}.issubset(written_columns.get(Dataset.ticks, set())):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_ticks_symbol_time ON ticks(symbol, time)",
)
if {"position_id", "symbol"}.issubset(
written_columns.get(Dataset.history_deals, set())
):
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_history_deals_position_symbol"
" ON history_deals(position_id, symbol)",
)
def _record_written_columns(
written_columns: dict[Dataset, set[str]],
dataset: Dataset,
frame: pd.DataFrame,
) -> None:
"""Remember columns for datasets written during streaming collection."""
columns = set(frame.columns)
if dataset in written_columns:
written_columns[dataset].update(columns)
if date_to is not None:
resolved_end = _coerce_datetime(date_to)
else:
written_columns[dataset] = columns
def _write_streamed_frame(
conn: sqlite3.Connection,
frame: pd.DataFrame,
dataset: Dataset,
table_exists: bool,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool:
"""Write one streamed dataset frame and track table state.
Returns:
True if the dataset table exists after this write attempt.
"""
write_mode = IfExists.APPEND if table_exists else if_exists
if _write_frame_to_sqlite(
conn,
frame,
dataset.table_name,
write_mode,
):
_record_written_columns(written_columns, dataset, frame)
return True
return table_exists
def _write_rates_dataset(
conn: sqlite3.Connection,
client: Mt5DataClient,
symbols: list[str],
timeframe: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool:
"""Stream rates frames into SQLite.
Returns:
True if the rates table was written.
"""
table_exists = False
for sym in symbols:
frame = client.copy_rates_range_as_df(
symbol=sym,
timeframe=timeframe,
date_from=date_from,
date_to=date_to,
)
frame.insert(0, "symbol", sym)
frame.insert(1, "timeframe", timeframe)
table_exists = _write_streamed_frame(
conn,
frame,
Dataset.rates,
table_exists,
if_exists,
written_columns,
)
return table_exists
def _write_ticks_dataset(
conn: sqlite3.Connection,
client: Mt5DataClient,
symbols: list[str],
flags: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool:
"""Stream ticks frames into SQLite.
Returns:
True if the ticks table was written.
"""
table_exists = False
for sym in symbols:
frame = client.copy_ticks_range_as_df(
symbol=sym,
date_from=date_from,
date_to=date_to,
flags=flags,
)
frame.insert(0, "symbol", sym)
table_exists = _write_streamed_frame(
conn,
frame,
Dataset.ticks,
table_exists,
if_exists,
written_columns,
)
return table_exists
def _write_history_dataset(
conn: sqlite3.Connection,
fetch: Callable[..., pd.DataFrame],
dataset: Dataset,
symbols: list[str],
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool:
"""Stream a history dataset into SQLite with exact symbol filtering.
Returns:
True if the history table was written.
"""
table_exists = False
for sym in symbols:
frame = fetch(date_from=date_from, date_to=date_to, symbol=sym)
if "symbol" in frame.columns:
frame = frame[frame["symbol"] == sym]
table_exists = _write_streamed_frame(
conn,
frame,
dataset,
table_exists,
if_exists,
written_columns,
)
return table_exists
def _write_collected_datasets(
conn: sqlite3.Connection,
client: Mt5DataClient,
symbols: list[str],
datasets: set[Dataset],
timeframe: int,
flags: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
"""Collect selected datasets and stream each symbol frame into SQLite.
Returns:
Written datasets and their columns.
"""
written_columns: dict[Dataset, set[str]] = {}
written_tables: set[Dataset] = set()
if Dataset.rates in datasets and _write_rates_dataset(
conn,
client,
symbols,
timeframe,
date_from,
date_to,
if_exists,
written_columns,
):
written_tables.add(Dataset.rates)
if Dataset.ticks in datasets and _write_ticks_dataset(
conn,
client,
symbols,
resolved_end = datetime.now(UTC)
end = resolved_end if resolved_end is not None else datetime.now(UTC)
fallback_start = end - timedelta(hours=lookback_hours)
resolved_timeframes, resolved_tick_flags = _resolve_incremental_settings(
selected,
timeframes,
flags,
date_from,
date_to,
if_exists,
written_columns,
):
written_tables.add(Dataset.ticks)
if Dataset.history_orders in datasets and _write_history_dataset(
conn,
client.history_orders_get_as_df,
Dataset.history_orders,
symbols,
date_from,
date_to,
if_exists,
written_columns,
):
written_tables.add(Dataset.history_orders)
if Dataset.history_deals in datasets and _write_history_dataset(
conn,
client.history_deals_get_as_df,
Dataset.history_deals,
symbols,
date_from,
date_to,
if_exists,
written_columns,
):
written_tables.add(Dataset.history_deals)
return written_tables, written_columns
)
return _UpdateHistoryRequest(
selected=selected,
end=end,
fallback_start=fallback_start,
resolved_timeframes=resolved_timeframes,
resolved_tick_flags=resolved_tick_flags,
output_path=Path(output),
)
def update_history( # noqa: PLR0913
*,
client: Mt5DataClient,
output: Path | str,
symbols: Sequence[str],
datasets: set[Dataset] | None = None,
timeframes: Sequence[int | str] | None = None,
flags: int | str = "ALL",
lookback_hours: float = 24.0,
date_to: datetime | str | None = None,
deduplicate: bool = True,
create_rate_views: bool = True,
with_views: bool = False,
include_account_events: bool = True,
) -> None:
"""Incrementally append MT5 history into a SQLite database.
Uses an already-connected ``Mt5DataClient`` and does not create or close
the MT5 connection. For first-time tables, data is fetched from
``date_to - lookback_hours``. Subsequent runs resume from existing
``MAX(time)`` per symbol (and timeframe for rates); when
``include_account_events=True``, account-level deals use a separate cursor
over ``type NOT IN (0, 1)`` / empty-symbol rows.
Args:
client: Connected MT5 data client.
output: SQLite database path.
symbols: Symbols to update.
datasets: Datasets to include (defaults to all).
timeframes: Rate timeframes to update (defaults to all fixed MT5
timeframes when None).
flags: Tick copy flags as integer or name (e.g. ``ALL``).
lookback_hours: First-run lookback when a table has no prior rows.
date_to: Optional update end datetime. Defaults to now (UTC).
deduplicate: Remove duplicate rows after append, keeping latest ROWID.
create_rate_views: Create ``rate_<symbol>__<timeframe>`` views.
with_views: Create ``cash_events`` and ``positions_reconstructed`` views.
include_account_events: Include account-level cash events in
``history_deals`` when True.
"""
request = _resolve_update_history_request(
output=output,
symbols=symbols,
datasets=datasets,
timeframes=timeframes,
flags=flags,
lookback_hours=lookback_hours,
date_to=date_to,
)
if request is None:
return
logger.info(
"Updating history in SQLite: symbols=%s, datasets=%s, path=%s",
list(symbols),
sorted(dataset.value for dataset in request.selected),
request.output_path,
)
with sqlite3.connect(request.output_path) as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
write_incremental_datasets(
conn,
client,
symbols,
request.selected,
request.resolved_timeframes,
request.resolved_tick_flags,
request.fallback_start,
request.end,
deduplicate=deduplicate,
create_rate_views=create_rate_views,
with_views=with_views,
include_account_events=include_account_events,
)
def update_history_with_config( # noqa: PLR0913
*,
output: Path | str,
symbols: Sequence[str],
config: Mt5Config | None = None,
datasets: set[Dataset] | None = None,
timeframes: Sequence[int | str] | None = None,
flags: int | str = "ALL",
lookback_hours: float = 24.0,
date_to: datetime | str | None = None,
deduplicate: bool = True,
create_rate_views: bool = True,
with_views: bool = False,
include_account_events: bool = True,
) -> None:
"""Incrementally append MT5 history, opening and closing the MT5 connection.
Convenience wrapper around :func:`update_history` for standalone use.
"""
request = _resolve_update_history_request(
output=output,
symbols=symbols,
datasets=datasets,
timeframes=timeframes,
flags=flags,
lookback_hours=lookback_hours,
date_to=date_to,
)
if request is None:
return
mt5_config = config or build_config()
with _connected_client(mt5_config) as client:
update_history(
client=client,
output=output,
symbols=symbols,
datasets=datasets,
timeframes=timeframes,
flags=flags,
lookback_hours=lookback_hours,
date_to=date_to,
deduplicate=deduplicate,
create_rate_views=create_rate_views,
with_views=with_views,
include_account_events=include_account_events,
)
def collect_history(
@@ -776,7 +670,7 @@ def collect_history(
with _connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
written_tables, written_columns = _write_collected_datasets(
written_tables, written_columns = write_collected_datasets(
conn,
client,
symbols,
@@ -787,10 +681,10 @@ def collect_history(
end,
if_exists,
)
_create_collect_history_indexes(conn, written_columns)
create_history_indexes(conn, written_columns)
if with_views and Dataset.history_deals in written_tables:
_create_cash_events_view(conn, written_columns[Dataset.history_deals])
_create_positions_reconstructed_view(
create_cash_events_view(conn, written_columns[Dataset.history_deals])
create_positions_reconstructed_view(
conn,
written_columns[Dataset.history_deals],
)
mt5cli/sqlite_history.py
+1176
View File
File diff suppressed because it is too large Load Diff
pyproject.toml
+2 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "mt5cli"
version = "0.4.0"
version = "0.4.1"
description = "Command-line tool for MetaTrader 5"
authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -124,6 +124,7 @@ ignore = [
]
[tool.ruff.lint.per-file-ignores]
"mt5cli/sqlite_history.py" = ["TC003"]
"tests/**/*.py" = [
"DOC201", # Missing return documentation
"DOC501", # Raised exception missing from docstring
tests/test_cli.py
+3 -3
View File
@@ -1089,7 +1089,7 @@ class TestCollectHistory:
assert all(row[0] not in {0, 1} for row in cash)
# Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed.
# Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed.
# Position 300 (open-only) and 400 (reversal-only) are excluded.
# Position 400 (reversal-only with non-trade deal type) stays excluded.
assert set(positions) == {100, 200, 500, 600}
pos_100 = positions[100]
tol = 1e-9
@@ -1106,10 +1106,10 @@ class TestCollectHistory:
assert abs(pos_500[5] - 1.05) < tol
pos_600 = positions[600]
assert abs(pos_600[1] - 3.0) < tol
assert abs(pos_600[2] - 3.0) < tol
assert abs(pos_600[2] - 4.0) < tol # reversal + close volumes
assert abs(pos_600[3] - 1.0) < tol
assert abs(pos_600[4] - 1.10) < tol
assert abs(pos_600[5] - 1.40) < tol
assert abs(pos_600[5] - 3.5475) < tol
assert pos_600[6] == 1
def test_collect_history_filters_history_symbols_exactly(
tests/test_sdk.py
+344
View File
@@ -36,8 +36,11 @@ from mt5cli.sdk import (
symbol_info_tick,
symbols,
terminal_info,
update_history,
update_history_with_config,
version,
)
from mt5cli.sqlite_history import DEFAULT_HISTORY_TIMEFRAMES
from mt5cli.utils import Dataset
_DEALS_FIXTURE: dict[str, list[object]] = {
@@ -464,3 +467,344 @@ class TestCollectHistory:
}
assert "cash_events" not in views
assert "positions_reconstructed" not in views
class TestUpdateHistory:
"""Tests for update_history SDK functions."""
@pytest.fixture
def connected_client(self) -> MagicMock:
"""Create a connected mock client without MT5 lifecycle patching."""
return MagicMock()
def test_update_history_appends_incrementally(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test sequential SQLite history updates use existing max timestamps."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
first_expected_start = datetime(2024, 1, 1, tzinfo=UTC)
second_expected_start = datetime(2024, 1, 1, 12, tzinfo=UTC)
rate_starts: list[datetime] = []
deal_starts: list[datetime] = []
def make_rates(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["timeframe"] == 1
assert kwargs["date_to"] == date_to
rate_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"open": [1.0 + len(rate_starts) / 10],
})
def make_deals(**kwargs: object) -> pd.DataFrame:
assert kwargs["date_to"] == date_to
deal_starts.append(kwargs["date_from"]) # type: ignore[arg-type]
return pd.DataFrame({
"ticket": [10],
"position_id": [100],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
"entry": [0],
"volume": [1.0],
"price": [1.1],
"profit": [0.0],
})
connected_client.copy_rates_range_as_df.side_effect = make_rates
connected_client.history_deals_get_as_df.side_effect = make_deals
mocker.patch("mt5cli.sdk.Mt5DataClient")
output = tmp_path / "incremental-history.db"
for _ in range(2):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates, Dataset.history_deals},
timeframes=["M1"],
lookback_hours=24,
date_to=date_to,
with_views=True,
)
assert rate_starts == [first_expected_start, second_expected_start]
assert deal_starts == [first_expected_start, first_expected_start]
connected_client.initialize_and_login_mt5.assert_not_called()
connected_client.shutdown.assert_not_called()
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM rates").fetchone() == (1,)
assert conn.execute("SELECT open FROM rates").fetchone() == (1.2,)
assert conn.execute(
"SELECT COUNT(*) FROM history_deals",
).fetchone() == (1,)
assert conn.execute(
"SELECT name FROM sqlite_master WHERE name = 'cash_events'",
).fetchone() == ("cash_events",)
def test_update_history_rejects_invalid_inputs(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test validation errors for incremental history updates."""
output = tmp_path / "invalid-update.db"
with pytest.raises(ValueError, match="At least one symbol"):
update_history(
client=connected_client,
output=output,
symbols=[],
)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
lookback_hours=0,
)
with pytest.raises(ValueError, match="Invalid timeframe"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["BAD"],
)
with pytest.raises(ValueError, match="Invalid tick flags"):
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks},
flags="BAD",
)
def test_update_history_noops_for_empty_datasets(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 and SQLite writes."""
writer = mocker.patch("mt5cli.sdk.write_incremental_datasets")
connect = mocker.patch("mt5cli.sdk.sqlite3.connect")
update_history(
client=connected_client,
output=tmp_path / "empty-datasets.db",
symbols=["EURUSD"],
datasets=set(),
)
writer.assert_not_called()
connect.assert_not_called()
def test_update_history_uses_all_default_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test that timeframes=None writes rates for all default MT5 timeframes."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "default-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=None,
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert len(timeframes_written) == len(DEFAULT_HISTORY_TIMEFRAMES)
def test_update_history_uses_specified_timeframes(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test explicit timeframes limit rate updates."""
timeframes_written: list[int] = []
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
timeframes_written.extend(args[4]) # type: ignore[arg-type]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
update_history(
client=connected_client,
output=tmp_path / "specific-timeframes.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1", "H1"],
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
)
assert timeframes_written == [1, 16385]
def test_update_history_updates_ticks_and_orders(
self,
connected_client: MagicMock,
tmp_path: Path,
) -> None:
"""Test incremental update writes selected ticks and orders datasets."""
date_to = datetime(2024, 1, 2, tzinfo=UTC)
expected_start = datetime(2024, 1, 1, tzinfo=UTC)
def make_ticks(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
assert kwargs["flags"] == 1
return pd.DataFrame({
"time": ["2024-01-01T12:00:00+00:00"],
"time_msc": [1_704_110_400_000],
"bid": [1.1],
})
def make_orders(**kwargs: object) -> pd.DataFrame:
assert kwargs["symbol"] == "EURUSD"
assert kwargs["date_from"] == expected_start
assert kwargs["date_to"] == date_to
return pd.DataFrame({
"ticket": [1],
"symbol": ["EURUSD"],
"time": ["2024-01-01T12:00:00+00:00"],
"type": [0],
})
connected_client.copy_ticks_range_as_df.side_effect = make_ticks
connected_client.history_orders_get_as_df.side_effect = make_orders
output = tmp_path / "ticks-orders.db"
update_history(
client=connected_client,
output=output,
symbols=["EURUSD"],
datasets={Dataset.ticks, Dataset.history_orders},
lookback_hours=24,
date_to=date_to,
)
with sqlite3.connect(output) as conn:
assert conn.execute("SELECT COUNT(*) FROM ticks").fetchone() == (1,)
assert conn.execute(
"SELECT COUNT(*) FROM history_orders",
).fetchone() == (1,)
def test_update_history_with_config_opens_and_closes_connection(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history_with_config manages MT5 connection lifecycle."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "config-wrapper.db",
symbols=["EURUSD"],
datasets={Dataset.history_deals},
timeframes=["M1"],
flags="ALL",
lookback_hours=1,
date_to=datetime(2024, 1, 1, tzinfo=UTC),
deduplicate=False,
create_rate_views=False,
with_views=True,
include_account_events=False,
)
mock_client.initialize_and_login_mt5.assert_called_once()
mock_client.shutdown.assert_called_once()
updater.assert_called_once()
assert updater.call_args.kwargs == {
"client": mock_client,
"output": tmp_path / "config-wrapper.db",
"symbols": ["EURUSD"],
"datasets": {Dataset.history_deals},
"timeframes": ["M1"],
"flags": "ALL",
"lookback_hours": 1,
"date_to": datetime(2024, 1, 1, tzinfo=UTC),
"deduplicate": False,
"create_rate_views": False,
"with_views": True,
"include_account_events": False,
}
def test_update_history_with_config_validates_before_connecting(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test invalid inputs fail before MT5 is initialized."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
with pytest.raises(ValueError, match="lookback_hours must be positive"):
update_history_with_config(
output=tmp_path / "invalid-config.db",
symbols=["EURUSD"],
lookback_hours=0,
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
def test_update_history_with_config_noops_for_empty_datasets(
self,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test empty dataset selection skips MT5 initialization."""
mock_client = MagicMock()
mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
updater = mocker.patch("mt5cli.sdk.update_history")
update_history_with_config(
output=tmp_path / "empty-config.db",
symbols=["EURUSD"],
datasets=set(),
)
mock_client.initialize_and_login_mt5.assert_not_called()
mock_client.shutdown.assert_not_called()
updater.assert_not_called()
def test_update_history_defaults_date_to_now(
self,
connected_client: MagicMock,
mocker: MockerFixture,
tmp_path: Path,
) -> None:
"""Test update_history uses current UTC time when date_to is omitted."""
captured: dict[str, datetime] = {}
def capture(
*args: object,
**_kwargs: object,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
captured["end"] = args[7] # type: ignore[assignment]
return set(), {}
mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
before = datetime.now(UTC)
update_history(
client=connected_client,
output=tmp_path / "now-default.db",
symbols=["EURUSD"],
datasets={Dataset.rates},
timeframes=["M1"],
lookback_hours=12,
)
after = datetime.now(UTC)
assert before <= captured["end"] <= after
tests/test_sqlite_history.py
+1497
View File
File diff suppressed because it is too large Load Diff
uv.lock
Generated
+1 -1
View File
@@ -487,7 +487,7 @@ wheels = [
[[package]]
name = "mt5cli"
version = "0.4.0"
version = "0.4.1"
source = { editable = "." }
dependencies = [
{ name = "click" },