Rename sqlite_history module to history (#17)

* Rename sqlite_history module to history.

Drop the sqlite-specific prefix now that history collection is the primary module name across SDK, tests, and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for history module rename.

Add a sqlite_history compatibility shim, clarify docs naming, and align the
module docstring with the collect-history SQLite scope.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove sqlite_history compatibility shim.

The rename to mt5cli.history is intentionally breaking; downstream code
should update imports rather than rely on a deprecated re-export path.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

README.md

@@ -87,7 +87,46 @@ mt5cli -o history.db collect-history \
   --timeframe M1 --flags ALL --if-exists append --with-views
 ```
 
-History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing deals, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns and do not contribute to the weighted prices.
+History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
+
+### Incremental history SDK
+
+For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
+
+```python
+from pdmt5 import Mt5Config, Mt5DataClient
+from mt5cli import Dataset, update_history, update_history_with_config
+
+# Reuse an already-connected pdmt5 client (does not open/close MT5)
+client = Mt5DataClient(config=Mt5Config(login=12345))
+client.initialize_and_login_mt5()
+try:
+    update_history(
+        client=client,
+        output="history.db",
+        symbols=["EURUSD", "GBPUSD"],
+        datasets={Dataset.rates, Dataset.history_deals},
+        timeframes=["M1", "H1"],  # default: all fixed MT5 timeframes
+        lookback_hours=24,
+        create_rate_views=True,
+        with_views=True,
+        include_account_events=True,
+    )
+finally:
+    client.shutdown()
+
+# Standalone wrapper that opens and closes MT5 for you
+update_history_with_config(
+    output="history.db",
+    symbols=["EURUSD"],
+    config=Mt5Config(login=12345),
+)
+```
+
+- **`collect-history`**: explicit date-range export into SQLite.
+- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
+- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
+- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
 
 ## Requirements
 

docs/api/history.md

@@ -0,0 +1,131 @@
+# History Collection (SQLite)
+
+::: mt5cli.history
+
+## `collect-history` schema
+
+The `collect-history` command (and the matching `collect_history` SDK function) writes
+selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
+names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
+
+- `symbol` is prepended on every table.
+- `timeframe` is prepended on `rates` so appended runs at different bar sizes stay
+  distinguishable.
+
+SQLite does not declare foreign keys. Rows are linked logically by `symbol`, time
+windows, and (for deals) `position_id` / `order`. Duplicate rows are removed on
+append using dataset-specific keys (for example `ticket` on history tables, or
+`(symbol, timeframe, time)` on rates).
+
+Optional views are created when `--with-views` is set and the `history-deals` dataset
+was written.
+
+### Entity-relationship diagram
+
+Sample layout for a full collection with `--with-views`:
+
+```mermaid
+erDiagram
+    rates {
+        TEXT symbol "dedup key"
+        INTEGER timeframe "dedup key"
+        TEXT time "dedup key"
+        REAL open
+        REAL high
+        REAL low
+        REAL close
+        INTEGER tick_volume
+        INTEGER spread
+        INTEGER real_volume
+    }
+
+    ticks {
+        TEXT symbol "dedup key"
+        TEXT time "dedup key"
+        INTEGER time_msc "dedup key (preferred)"
+        REAL bid
+        REAL ask
+        REAL last
+        INTEGER volume
+        INTEGER flags
+        REAL volume_real
+    }
+
+    history_orders {
+        INTEGER ticket "dedup key"
+        TEXT symbol
+        TEXT time
+        INTEGER type
+        INTEGER state
+        REAL volume_initial
+        REAL price_open
+        REAL price_current
+        INTEGER magic
+    }
+
+    history_deals {
+        INTEGER ticket "dedup key"
+        INTEGER order
+        INTEGER position_id "groups position view"
+        TEXT symbol
+        TEXT time
+        INTEGER type "0/1 trade, else cash event"
+        INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
+        REAL volume
+        REAL price
+        REAL profit
+        REAL commission
+        REAL swap
+        REAL fee
+    }
+
+    cash_events {
+        INTEGER ticket
+        TEXT symbol
+        TEXT time
+        INTEGER type
+        REAL profit
+    }
+
+    positions_reconstructed {
+        INTEGER position_id
+        TEXT symbol
+        TEXT open_time
+        TEXT close_time
+        INTEGER direction
+        REAL volume_open
+        REAL volume_close
+        REAL volume_reversal
+        REAL open_price
+        REAL close_price
+        REAL total_profit
+        INTEGER reversal_count
+        INTEGER deals_count
+    }
+
+    rates ||--o{ history_deals : "symbol (logical)"
+    ticks ||--o{ history_deals : "symbol (logical)"
+    history_orders ||--o{ history_deals : "order ~ ticket (logical)"
+    history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
+    history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
+```
+
+### Tables and views
+
+| Object                    | Kind  | Source               | Notes                                                                                       |
+| ------------------------- | ----- | -------------------- | ------------------------------------------------------------------------------------------- |
+| `rates`                   | table | `copy_rates_range`   | Indexed on `(symbol, timeframe, time)` when columns exist.                                  |
+| `ticks`                   | table | `copy_ticks_range`   | Indexed on `(symbol, time)` when columns exist.                                             |
+| `history_orders`          | table | `history_orders_get` | Fetched per `--symbol`, then concatenated.                                                  |
+| `history_deals`           | table | `history_deals_get`  | Fetched per `--symbol`, then concatenated. Indexed on `(position_id, symbol)` when present. |
+| `cash_events`             | view  | `history_deals`      | Non-trade deal types (deposits, balance ops, etc.). Requires `type` column.                 |
+| `positions_reconstructed` | view  | `history_deals`      | One row per closed `position_id`; volume-weighted prices and reversal stats.                |
+
+Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning
+when required columns are missing.
+
+### Incremental collection
+
+The `update_history` SDK path uses the same base tables and optional
+`cash_events` / `positions_reconstructed` views. It additionally maintains
+`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.

docs/api/index.md

@@ -18,6 +18,10 @@ Utility module providing constants, enums, Click parameter types, and helper fun
 
 Programmatic SDK for read-only MetaTrader 5 data collection. Returns pandas DataFrames and provides `collect_history` for SQLite bulk collection.
 
+### [History Collection (SQLite)](history.md)
+
+SQLite storage helpers for the `collect-history` command schema, incremental updates, deduplication, indexes, and optional views.
+
 ## Architecture Overview
 
 The package follows a simple architecture built on top of pdmt5:

docs/index.md

@@ -152,6 +152,8 @@ mt5cli -o history.db collect-history \
 
 History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
 
+See the [History schema diagram](api/history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
+
 ## Global Options
 
 | Option         | Description                                             |

mkdocs.yml

@@ -24,6 +24,7 @@ theme:
   features:
     - content.code.annotate
     - content.code.copy
+    - content.code.mermaid
     - navigation.indexes
     - navigation.sections
     - navigation.tabs
@@ -57,6 +58,7 @@ nav:
       - Overview: api/index.md
       - CLI: api/cli.md
       - SDK: api/sdk.md
+      - History Collection (SQLite): api/history.md
       - Utils: api/utils.md
 
 markdown_extensions:

mt5cli/__init__.py

@@ -22,15 +22,19 @@ from .sdk import (
     symbol_info_tick,
     symbols,
     terminal_info,
+    update_history,
+    update_history_with_config,
 )
 from .sdk import (
     version as mt5_version,
 )
-from .utils import detect_format, export_dataframe
+from .utils import Dataset, IfExists, detect_format, export_dataframe
 
 __version__ = version(__package__) if __package__ else None
 
 __all__ = [
+    "Dataset",
+    "IfExists",
     "Mt5CliClient",
     "account_info",
     "build_config",
@@ -53,4 +57,6 @@ __all__ = [
     "symbol_info_tick",
     "symbols",
     "terminal_info",
+    "update_history",
+    "update_history_with_config",
 ]

mt5cli/sdk.py

@@ -5,12 +5,23 @@ from __future__ import annotations
 import logging
 import sqlite3
 from contextlib import contextmanager
-from datetime import datetime
+from dataclasses import dataclass
-from pathlib import Path  # noqa: TC003
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
 from typing import TYPE_CHECKING, Self, TypeVar
 
 from pdmt5 import Mt5Config, Mt5DataClient
 
+from .history import (
+    create_cash_events_view,
+    create_history_indexes,
+    create_positions_reconstructed_view,
+    resolve_history_datasets,
+    resolve_history_tick_flags,
+    resolve_history_timeframes,
+    write_collected_datasets,
+    write_incremental_datasets,
+)
 from .utils import (
     Dataset,
     IfExists,
@@ -20,7 +31,7 @@ from .utils import (
 )
 
 if TYPE_CHECKING:
-    from collections.abc import Callable, Iterator
+    from collections.abc import Callable, Iterator, Sequence
 
     import pandas as pd
 
@@ -48,22 +59,11 @@ __all__ = [
     "symbol_info_tick",
     "symbols",
     "terminal_info",
+    "update_history",
+    "update_history_with_config",
     "version",
 ]
 
-_TRADE_DEAL_TYPES: tuple[int, int] = (0, 1)
-_TRADE_DEAL_TYPES_SQL = f"({', '.join(str(value) for value in _TRADE_DEAL_TYPES)})"
-_POSITIONS_VIEW_REQUIRED_COLUMNS: frozenset[str] = frozenset({
-    "position_id",
-    "symbol",
-    "time",
-    "type",
-    "entry",
-    "volume",
-    "price",
-    "profit",
-})
-
 
 def _coerce_timeframe(timeframe: int | str) -> int:
     if isinstance(timeframe, int):
@@ -419,325 +419,219 @@ class Mt5CliClient:
         return self._fetch(lambda c: c.market_book_get_as_df(symbol=symbol))
 
 
-def _create_cash_events_view(
+def _resolve_incremental_settings(
-    conn: sqlite3.Connection,
+    selected_datasets: set[Dataset],
-    deals_columns: set[str],
+    timeframes: Sequence[int | str] | None,
-) -> bool:
+    flags: int | str,
-    """Create the cash_events SQLite view derived from history_deals.
+) -> tuple[list[int], int]:
+    """Resolve dataset-specific incremental update settings.
 
     Returns:
-        True if the view was created, False if required columns are missing.
+        Tuple of resolved rate timeframes and tick copy flags.
+
+    Raises:
+        ValueError: If timeframe or tick flag values are invalid.
     """
-    if "type" not in deals_columns:
+    resolved_timeframes: list[int] = []
-        logger.warning("Skipping cash_events view: history_deals.type is missing")
+    if Dataset.rates in selected_datasets:
-        return False
+        try:
-    conn.execute("DROP VIEW IF EXISTS cash_events")
+            resolved_timeframes = resolve_history_timeframes(timeframes)
-    conn.execute(
+        except ValueError as exc:
-        "CREATE VIEW cash_events AS"  # noqa: S608
+            msg = str(exc)
-        f" SELECT * FROM history_deals WHERE type NOT IN {_TRADE_DEAL_TYPES_SQL}",
+            raise ValueError(msg) from exc
-    )
+    resolved_tick_flags = 0
-    return True
+    if Dataset.ticks in selected_datasets:
+        try:
+            resolved_tick_flags = resolve_history_tick_flags(flags)
+        except ValueError as exc:
+            msg = str(exc)
+            raise ValueError(msg) from exc
+    return resolved_timeframes, resolved_tick_flags
 
 
-def _create_positions_reconstructed_view(
+@dataclass(frozen=True)
-    conn: sqlite3.Connection,
+class _UpdateHistoryRequest:
-    deals_columns: set[str],
+    selected: set[Dataset]
-) -> bool:
+    end: datetime
-    """Create the positions_reconstructed SQLite view derived from history_deals.
+    fallback_start: datetime
+    resolved_timeframes: list[int]
+    resolved_tick_flags: int
+    output_path: Path
+
+
+def _resolve_update_history_request(
+    *,
+    output: Path | str,
+    symbols: Sequence[str],
+    datasets: set[Dataset] | None,
+    timeframes: Sequence[int | str] | None,
+    flags: int | str,
+    lookback_hours: float,
+    date_to: datetime | str | None,
+) -> _UpdateHistoryRequest | None:
+    """Validate and resolve incremental history update inputs.
 
     Returns:
-        True if the view was created, False if required columns are missing.
+        Resolved request parameters, or None when no datasets are selected.
+
+    Raises:
+        ValueError: If symbols are empty, lookback_hours is not positive, or
+            timeframe/flag values are invalid.
     """
-    if not _POSITIONS_VIEW_REQUIRED_COLUMNS.issubset(deals_columns):
+    if lookback_hours <= 0:
-        missing = ", ".join(sorted(_POSITIONS_VIEW_REQUIRED_COLUMNS - deals_columns))
+        msg = "lookback_hours must be positive."
-        logger.warning(
+        raise ValueError(msg)
-            "Skipping positions_reconstructed view: history_deals missing columns: %s",
+    selected = resolve_history_datasets(datasets)
-            missing,
+    if not selected:
-        )
+        logger.info("Skipping SQLite history update: no datasets selected.")
-        return False
+        return None
-    conn.execute("DROP VIEW IF EXISTS positions_reconstructed")
+    if not symbols:
-    conn.execute(
+        msg = "At least one symbol is required."
-        "CREATE VIEW positions_reconstructed AS"  # noqa: S608
+        raise ValueError(msg)
-        " SELECT"
-        " position_id,"
-        " symbol,"
-        " MIN(CASE WHEN entry = 0 THEN time END) AS open_time,"
-        " MAX(CASE WHEN entry IN (1, 2, 3) THEN time END) AS close_time,"
-        " MIN(CASE WHEN entry = 0 THEN type END) AS direction,"
-        " SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END) AS volume_open,"
-        " SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END) AS volume_close,"
-        " SUM(CASE WHEN entry = 2 THEN volume ELSE 0 END) AS volume_reversal,"
-        " CASE"
-        " WHEN SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END) > 0"
-        " THEN SUM(CASE WHEN entry = 0 THEN price * volume ELSE 0 END)"
-        " / SUM(CASE WHEN entry = 0 THEN volume ELSE 0 END)"
-        " END AS open_price,"
-        " CASE"
-        " WHEN SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END) > 0"
-        " THEN SUM(CASE WHEN entry IN (1, 3) THEN price * volume ELSE 0 END)"
-        " / SUM(CASE WHEN entry IN (1, 3) THEN volume ELSE 0 END)"
-        " END AS close_price,"
-        " SUM(profit) AS total_profit,"
-        " SUM(CASE WHEN entry = 2 THEN 1 ELSE 0 END) AS reversal_count,"
-        " COUNT(*) AS deals_count"
-        " FROM history_deals"
-        f" WHERE type IN {_TRADE_DEAL_TYPES_SQL} AND position_id != 0"
-        " GROUP BY position_id, symbol"
-        " HAVING SUM(CASE WHEN entry IN (1, 3) THEN 1 ELSE 0 END) > 0",
-    )
-    return True
 
-
+    if date_to is not None:
-def _write_frame_to_sqlite(
+        resolved_end = _coerce_datetime(date_to)
-    conn: sqlite3.Connection,
-    frame: pd.DataFrame,
-    table_name: str,
-    if_exists: IfExists,
-) -> bool:
-    """Write a non-empty-schema frame to SQLite.
-
-    Returns:
-        True if a table was written, False if the frame had no columns.
-    """
-    if len(frame.columns) == 0:
-        logger.warning("Skipping %s: dataset returned no columns", table_name)
-        return False
-    frame.to_sql(  # type: ignore[reportUnknownMemberType]
-        table_name,
-        conn,
-        if_exists=if_exists.value,
-        index=False,
-        chunksize=50_000,
-        method="multi",
-    )
-    return True
-
-
-def _create_collect_history_indexes(
-    conn: sqlite3.Connection,
-    written_columns: dict[Dataset, set[str]],
-) -> None:
-    """Create useful indexes for collected history tables when present."""
-    if {"symbol", "time"}.issubset(written_columns.get(Dataset.rates, set())):
-        conn.execute(
-            "CREATE INDEX IF NOT EXISTS idx_rates_symbol_time ON rates(symbol, time)",
-        )
-    if {"symbol", "time"}.issubset(written_columns.get(Dataset.ticks, set())):
-        conn.execute(
-            "CREATE INDEX IF NOT EXISTS idx_ticks_symbol_time ON ticks(symbol, time)",
-        )
-    if {"position_id", "symbol"}.issubset(
-        written_columns.get(Dataset.history_deals, set())
-    ):
-        conn.execute(
-            "CREATE INDEX IF NOT EXISTS idx_history_deals_position_symbol"
-            " ON history_deals(position_id, symbol)",
-        )
-
-
-def _record_written_columns(
-    written_columns: dict[Dataset, set[str]],
-    dataset: Dataset,
-    frame: pd.DataFrame,
-) -> None:
-    """Remember columns for datasets written during streaming collection."""
-    columns = set(frame.columns)
-    if dataset in written_columns:
-        written_columns[dataset].update(columns)
     else:
-        written_columns[dataset] = columns
+        resolved_end = datetime.now(UTC)
-
+    end = resolved_end if resolved_end is not None else datetime.now(UTC)
-
+    fallback_start = end - timedelta(hours=lookback_hours)
-def _write_streamed_frame(
+    resolved_timeframes, resolved_tick_flags = _resolve_incremental_settings(
-    conn: sqlite3.Connection,
+        selected,
-    frame: pd.DataFrame,
+        timeframes,
-    dataset: Dataset,
-    table_exists: bool,
-    if_exists: IfExists,
-    written_columns: dict[Dataset, set[str]],
-) -> bool:
-    """Write one streamed dataset frame and track table state.
-
-    Returns:
-        True if the dataset table exists after this write attempt.
-    """
-    write_mode = IfExists.APPEND if table_exists else if_exists
-    if _write_frame_to_sqlite(
-        conn,
-        frame,
-        dataset.table_name,
-        write_mode,
-    ):
-        _record_written_columns(written_columns, dataset, frame)
-        return True
-    return table_exists
-
-
-def _write_rates_dataset(
-    conn: sqlite3.Connection,
-    client: Mt5DataClient,
-    symbols: list[str],
-    timeframe: int,
-    date_from: datetime,
-    date_to: datetime,
-    if_exists: IfExists,
-    written_columns: dict[Dataset, set[str]],
-) -> bool:
-    """Stream rates frames into SQLite.
-
-    Returns:
-        True if the rates table was written.
-    """
-    table_exists = False
-    for sym in symbols:
-        frame = client.copy_rates_range_as_df(
-            symbol=sym,
-            timeframe=timeframe,
-            date_from=date_from,
-            date_to=date_to,
-        )
-        frame.insert(0, "symbol", sym)
-        frame.insert(1, "timeframe", timeframe)
-        table_exists = _write_streamed_frame(
-            conn,
-            frame,
-            Dataset.rates,
-            table_exists,
-            if_exists,
-            written_columns,
-        )
-    return table_exists
-
-
-def _write_ticks_dataset(
-    conn: sqlite3.Connection,
-    client: Mt5DataClient,
-    symbols: list[str],
-    flags: int,
-    date_from: datetime,
-    date_to: datetime,
-    if_exists: IfExists,
-    written_columns: dict[Dataset, set[str]],
-) -> bool:
-    """Stream ticks frames into SQLite.
-
-    Returns:
-        True if the ticks table was written.
-    """
-    table_exists = False
-    for sym in symbols:
-        frame = client.copy_ticks_range_as_df(
-            symbol=sym,
-            date_from=date_from,
-            date_to=date_to,
-            flags=flags,
-        )
-        frame.insert(0, "symbol", sym)
-        table_exists = _write_streamed_frame(
-            conn,
-            frame,
-            Dataset.ticks,
-            table_exists,
-            if_exists,
-            written_columns,
-        )
-    return table_exists
-
-
-def _write_history_dataset(
-    conn: sqlite3.Connection,
-    fetch: Callable[..., pd.DataFrame],
-    dataset: Dataset,
-    symbols: list[str],
-    date_from: datetime,
-    date_to: datetime,
-    if_exists: IfExists,
-    written_columns: dict[Dataset, set[str]],
-) -> bool:
-    """Stream a history dataset into SQLite with exact symbol filtering.
-
-    Returns:
-        True if the history table was written.
-    """
-    table_exists = False
-    for sym in symbols:
-        frame = fetch(date_from=date_from, date_to=date_to, symbol=sym)
-        if "symbol" in frame.columns:
-            frame = frame[frame["symbol"] == sym]
-        table_exists = _write_streamed_frame(
-            conn,
-            frame,
-            dataset,
-            table_exists,
-            if_exists,
-            written_columns,
-        )
-    return table_exists
-
-
-def _write_collected_datasets(
-    conn: sqlite3.Connection,
-    client: Mt5DataClient,
-    symbols: list[str],
-    datasets: set[Dataset],
-    timeframe: int,
-    flags: int,
-    date_from: datetime,
-    date_to: datetime,
-    if_exists: IfExists,
-) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
-    """Collect selected datasets and stream each symbol frame into SQLite.
-
-    Returns:
-        Written datasets and their columns.
-    """
-    written_columns: dict[Dataset, set[str]] = {}
-    written_tables: set[Dataset] = set()
-    if Dataset.rates in datasets and _write_rates_dataset(
-        conn,
-        client,
-        symbols,
-        timeframe,
-        date_from,
-        date_to,
-        if_exists,
-        written_columns,
-    ):
-        written_tables.add(Dataset.rates)
-    if Dataset.ticks in datasets and _write_ticks_dataset(
-        conn,
-        client,
-        symbols,
         flags,
-        date_from,
+    )
-        date_to,
+    return _UpdateHistoryRequest(
-        if_exists,
+        selected=selected,
-        written_columns,
+        end=end,
-    ):
+        fallback_start=fallback_start,
-        written_tables.add(Dataset.ticks)
+        resolved_timeframes=resolved_timeframes,
-    if Dataset.history_orders in datasets and _write_history_dataset(
+        resolved_tick_flags=resolved_tick_flags,
-        conn,
+        output_path=Path(output),
-        client.history_orders_get_as_df,
+    )
-        Dataset.history_orders,
+
-        symbols,
+
-        date_from,
+def update_history(  # noqa: PLR0913
-        date_to,
+    *,
-        if_exists,
+    client: Mt5DataClient,
-        written_columns,
+    output: Path | str,
-    ):
+    symbols: Sequence[str],
-        written_tables.add(Dataset.history_orders)
+    datasets: set[Dataset] | None = None,
-    if Dataset.history_deals in datasets and _write_history_dataset(
+    timeframes: Sequence[int | str] | None = None,
-        conn,
+    flags: int | str = "ALL",
-        client.history_deals_get_as_df,
+    lookback_hours: float = 24.0,
-        Dataset.history_deals,
+    date_to: datetime | str | None = None,
-        symbols,
+    deduplicate: bool = True,
-        date_from,
+    create_rate_views: bool = True,
-        date_to,
+    with_views: bool = False,
-        if_exists,
+    include_account_events: bool = True,
-        written_columns,
+) -> None:
-    ):
+    """Incrementally append MT5 history into a SQLite database.
-        written_tables.add(Dataset.history_deals)
+
-    return written_tables, written_columns
+    Uses an already-connected ``Mt5DataClient`` and does not create or close
+    the MT5 connection. For first-time tables, data is fetched from
+    ``date_to - lookback_hours``. Subsequent runs resume from existing
+    ``MAX(time)`` per symbol (and timeframe for rates); when
+    ``include_account_events=True``, account-level deals use a separate cursor
+    over ``type NOT IN (0, 1)`` / empty-symbol rows.
+
+    Args:
+        client: Connected MT5 data client.
+        output: SQLite database path.
+        symbols: Symbols to update.
+        datasets: Datasets to include (defaults to all).
+        timeframes: Rate timeframes to update (defaults to all fixed MT5
+            timeframes when None).
+        flags: Tick copy flags as integer or name (e.g. ``ALL``).
+        lookback_hours: First-run lookback when a table has no prior rows.
+        date_to: Optional update end datetime. Defaults to now (UTC).
+        deduplicate: Remove duplicate rows after append, keeping latest ROWID.
+        create_rate_views: Create ``rate_<symbol>__<timeframe>`` views.
+        with_views: Create ``cash_events`` and ``positions_reconstructed`` views.
+        include_account_events: Include account-level cash events in
+            ``history_deals`` when True.
+    """
+    request = _resolve_update_history_request(
+        output=output,
+        symbols=symbols,
+        datasets=datasets,
+        timeframes=timeframes,
+        flags=flags,
+        lookback_hours=lookback_hours,
+        date_to=date_to,
+    )
+    if request is None:
+        return
+    logger.info(
+        "Updating history in SQLite: symbols=%s, datasets=%s, path=%s",
+        list(symbols),
+        sorted(dataset.value for dataset in request.selected),
+        request.output_path,
+    )
+    with sqlite3.connect(request.output_path) as conn:
+        conn.execute("PRAGMA journal_mode=WAL")
+        conn.execute("PRAGMA synchronous=NORMAL")
+        write_incremental_datasets(
+            conn,
+            client,
+            symbols,
+            request.selected,
+            request.resolved_timeframes,
+            request.resolved_tick_flags,
+            request.fallback_start,
+            request.end,
+            deduplicate=deduplicate,
+            create_rate_views=create_rate_views,
+            with_views=with_views,
+            include_account_events=include_account_events,
+        )
+
+
+def update_history_with_config(  # noqa: PLR0913
+    *,
+    output: Path | str,
+    symbols: Sequence[str],
+    config: Mt5Config | None = None,
+    datasets: set[Dataset] | None = None,
+    timeframes: Sequence[int | str] | None = None,
+    flags: int | str = "ALL",
+    lookback_hours: float = 24.0,
+    date_to: datetime | str | None = None,
+    deduplicate: bool = True,
+    create_rate_views: bool = True,
+    with_views: bool = False,
+    include_account_events: bool = True,
+) -> None:
+    """Incrementally append MT5 history, opening and closing the MT5 connection.
+
+    Convenience wrapper around :func:`update_history` for standalone use.
+    """
+    request = _resolve_update_history_request(
+        output=output,
+        symbols=symbols,
+        datasets=datasets,
+        timeframes=timeframes,
+        flags=flags,
+        lookback_hours=lookback_hours,
+        date_to=date_to,
+    )
+    if request is None:
+        return
+    mt5_config = config or build_config()
+    with _connected_client(mt5_config) as client:
+        update_history(
+            client=client,
+            output=output,
+            symbols=symbols,
+            datasets=datasets,
+            timeframes=timeframes,
+            flags=flags,
+            lookback_hours=lookback_hours,
+            date_to=date_to,
+            deduplicate=deduplicate,
+            create_rate_views=create_rate_views,
+            with_views=with_views,
+            include_account_events=include_account_events,
+        )
 
 
 def collect_history(
@@ -776,7 +670,7 @@ def collect_history(
     with _connected_client(mt5_config) as client, sqlite3.connect(output) as conn:
         conn.execute("PRAGMA journal_mode=WAL")
         conn.execute("PRAGMA synchronous=NORMAL")
-        written_tables, written_columns = _write_collected_datasets(
+        written_tables, written_columns = write_collected_datasets(
             conn,
             client,
             symbols,
@@ -787,10 +681,10 @@ def collect_history(
             end,
             if_exists,
         )
-        _create_collect_history_indexes(conn, written_columns)
+        create_history_indexes(conn, written_columns)
         if with_views and Dataset.history_deals in written_tables:
-            _create_cash_events_view(conn, written_columns[Dataset.history_deals])
+            create_cash_events_view(conn, written_columns[Dataset.history_deals])
-            _create_positions_reconstructed_view(
+            create_positions_reconstructed_view(
                 conn,
                 written_columns[Dataset.history_deals],
             )

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mt5cli"
-version = "0.4.0"
+version = "0.4.2"
 description = "Command-line tool for MetaTrader 5"
 authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
 maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
@@ -124,6 +124,7 @@ ignore = [
 ]
 
 [tool.ruff.lint.per-file-ignores]
+"mt5cli/history.py" = ["TC003"]
 "tests/**/*.py" = [
   "DOC201",   # Missing return documentation
   "DOC501",   # Raised exception missing from docstring

tests/test_cli.py

@@ -1089,7 +1089,7 @@ class TestCollectHistory:
         assert all(row[0] not in {0, 1} for row in cash)
         # Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed.
         # Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed.
-        # Position 300 (open-only) and 400 (reversal-only) are excluded.
+        # Position 400 (reversal-only with non-trade deal type) stays excluded.
         assert set(positions) == {100, 200, 500, 600}
         pos_100 = positions[100]
         tol = 1e-9
@@ -1106,10 +1106,10 @@ class TestCollectHistory:
         assert abs(pos_500[5] - 1.05) < tol
         pos_600 = positions[600]
         assert abs(pos_600[1] - 3.0) < tol
-        assert abs(pos_600[2] - 3.0) < tol
+        assert abs(pos_600[2] - 4.0) < tol  # reversal + close volumes
         assert abs(pos_600[3] - 1.0) < tol
         assert abs(pos_600[4] - 1.10) < tol
-        assert abs(pos_600[5] - 1.40) < tol
+        assert abs(pos_600[5] - 3.5475) < tol
         assert pos_600[6] == 1
 
     def test_collect_history_filters_history_symbols_exactly(

tests/test_sdk.py

@@ -16,6 +16,7 @@ if TYPE_CHECKING:
     from pathlib import Path
 
 from mt5cli import sdk
+from mt5cli.history import DEFAULT_HISTORY_TIMEFRAMES
 from mt5cli.sdk import (
     Mt5CliClient,
     account_info,
@@ -36,6 +37,8 @@ from mt5cli.sdk import (
     symbol_info_tick,
     symbols,
     terminal_info,
+    update_history,
+    update_history_with_config,
     version,
 )
 from mt5cli.utils import Dataset
@@ -464,3 +467,344 @@ class TestCollectHistory:
             }
         assert "cash_events" not in views
         assert "positions_reconstructed" not in views
+
+
+class TestUpdateHistory:
+    """Tests for update_history SDK functions."""
+
+    @pytest.fixture
+    def connected_client(self) -> MagicMock:
+        """Create a connected mock client without MT5 lifecycle patching."""
+        return MagicMock()
+
+    def test_update_history_appends_incrementally(
+        self,
+        connected_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test sequential SQLite history updates use existing max timestamps."""
+        date_to = datetime(2024, 1, 2, tzinfo=UTC)
+        first_expected_start = datetime(2024, 1, 1, tzinfo=UTC)
+        second_expected_start = datetime(2024, 1, 1, 12, tzinfo=UTC)
+        rate_starts: list[datetime] = []
+        deal_starts: list[datetime] = []
+
+        def make_rates(**kwargs: object) -> pd.DataFrame:
+            assert kwargs["symbol"] == "EURUSD"
+            assert kwargs["timeframe"] == 1
+            assert kwargs["date_to"] == date_to
+            rate_starts.append(kwargs["date_from"])  # type: ignore[arg-type]
+            return pd.DataFrame({
+                "time": ["2024-01-01T12:00:00+00:00"],
+                "open": [1.0 + len(rate_starts) / 10],
+            })
+
+        def make_deals(**kwargs: object) -> pd.DataFrame:
+            assert kwargs["date_to"] == date_to
+            deal_starts.append(kwargs["date_from"])  # type: ignore[arg-type]
+            return pd.DataFrame({
+                "ticket": [10],
+                "position_id": [100],
+                "symbol": ["EURUSD"],
+                "time": ["2024-01-01T12:00:00+00:00"],
+                "type": [0],
+                "entry": [0],
+                "volume": [1.0],
+                "price": [1.1],
+                "profit": [0.0],
+            })
+
+        connected_client.copy_rates_range_as_df.side_effect = make_rates
+        connected_client.history_deals_get_as_df.side_effect = make_deals
+        mocker.patch("mt5cli.sdk.Mt5DataClient")
+        output = tmp_path / "incremental-history.db"
+
+        for _ in range(2):
+            update_history(
+                client=connected_client,
+                output=output,
+                symbols=["EURUSD"],
+                datasets={Dataset.rates, Dataset.history_deals},
+                timeframes=["M1"],
+                lookback_hours=24,
+                date_to=date_to,
+                with_views=True,
+            )
+
+        assert rate_starts == [first_expected_start, second_expected_start]
+        assert deal_starts == [first_expected_start, first_expected_start]
+        connected_client.initialize_and_login_mt5.assert_not_called()
+        connected_client.shutdown.assert_not_called()
+        with sqlite3.connect(output) as conn:
+            assert conn.execute("SELECT COUNT(*) FROM rates").fetchone() == (1,)
+            assert conn.execute("SELECT open FROM rates").fetchone() == (1.2,)
+            assert conn.execute(
+                "SELECT COUNT(*) FROM history_deals",
+            ).fetchone() == (1,)
+            assert conn.execute(
+                "SELECT name FROM sqlite_master WHERE name = 'cash_events'",
+            ).fetchone() == ("cash_events",)
+
+    def test_update_history_rejects_invalid_inputs(
+        self,
+        connected_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """Test validation errors for incremental history updates."""
+        output = tmp_path / "invalid-update.db"
+        with pytest.raises(ValueError, match="At least one symbol"):
+            update_history(
+                client=connected_client,
+                output=output,
+                symbols=[],
+            )
+        with pytest.raises(ValueError, match="lookback_hours must be positive"):
+            update_history(
+                client=connected_client,
+                output=output,
+                symbols=["EURUSD"],
+                lookback_hours=0,
+            )
+        with pytest.raises(ValueError, match="Invalid timeframe"):
+            update_history(
+                client=connected_client,
+                output=output,
+                symbols=["EURUSD"],
+                datasets={Dataset.rates},
+                timeframes=["BAD"],
+            )
+        with pytest.raises(ValueError, match="Invalid tick flags"):
+            update_history(
+                client=connected_client,
+                output=output,
+                symbols=["EURUSD"],
+                datasets={Dataset.ticks},
+                flags="BAD",
+            )
+
+    def test_update_history_noops_for_empty_datasets(
+        self,
+        connected_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test empty dataset selection skips MT5 and SQLite writes."""
+        writer = mocker.patch("mt5cli.sdk.write_incremental_datasets")
+        connect = mocker.patch("mt5cli.sdk.sqlite3.connect")
+        update_history(
+            client=connected_client,
+            output=tmp_path / "empty-datasets.db",
+            symbols=["EURUSD"],
+            datasets=set(),
+        )
+        writer.assert_not_called()
+        connect.assert_not_called()
+
+    def test_update_history_uses_all_default_timeframes(
+        self,
+        connected_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test that timeframes=None writes rates for all default MT5 timeframes."""
+        timeframes_written: list[int] = []
+
+        def capture(
+            *args: object,
+            **_kwargs: object,
+        ) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
+            timeframes_written.extend(args[4])  # type: ignore[arg-type]
+            return set(), {}
+
+        mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
+        update_history(
+            client=connected_client,
+            output=tmp_path / "default-timeframes.db",
+            symbols=["EURUSD"],
+            datasets={Dataset.rates},
+            timeframes=None,
+            lookback_hours=1,
+            date_to=datetime(2024, 1, 1, tzinfo=UTC),
+        )
+        assert len(timeframes_written) == len(DEFAULT_HISTORY_TIMEFRAMES)
+
+    def test_update_history_uses_specified_timeframes(
+        self,
+        connected_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test explicit timeframes limit rate updates."""
+        timeframes_written: list[int] = []
+
+        def capture(
+            *args: object,
+            **_kwargs: object,
+        ) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
+            timeframes_written.extend(args[4])  # type: ignore[arg-type]
+            return set(), {}
+
+        mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
+        update_history(
+            client=connected_client,
+            output=tmp_path / "specific-timeframes.db",
+            symbols=["EURUSD"],
+            datasets={Dataset.rates},
+            timeframes=["M1", "H1"],
+            lookback_hours=1,
+            date_to=datetime(2024, 1, 1, tzinfo=UTC),
+        )
+        assert timeframes_written == [1, 16385]
+
+    def test_update_history_updates_ticks_and_orders(
+        self,
+        connected_client: MagicMock,
+        tmp_path: Path,
+    ) -> None:
+        """Test incremental update writes selected ticks and orders datasets."""
+        date_to = datetime(2024, 1, 2, tzinfo=UTC)
+        expected_start = datetime(2024, 1, 1, tzinfo=UTC)
+
+        def make_ticks(**kwargs: object) -> pd.DataFrame:
+            assert kwargs["symbol"] == "EURUSD"
+            assert kwargs["date_from"] == expected_start
+            assert kwargs["date_to"] == date_to
+            assert kwargs["flags"] == 1
+            return pd.DataFrame({
+                "time": ["2024-01-01T12:00:00+00:00"],
+                "time_msc": [1_704_110_400_000],
+                "bid": [1.1],
+            })
+
+        def make_orders(**kwargs: object) -> pd.DataFrame:
+            assert kwargs["symbol"] == "EURUSD"
+            assert kwargs["date_from"] == expected_start
+            assert kwargs["date_to"] == date_to
+            return pd.DataFrame({
+                "ticket": [1],
+                "symbol": ["EURUSD"],
+                "time": ["2024-01-01T12:00:00+00:00"],
+                "type": [0],
+            })
+
+        connected_client.copy_ticks_range_as_df.side_effect = make_ticks
+        connected_client.history_orders_get_as_df.side_effect = make_orders
+        output = tmp_path / "ticks-orders.db"
+        update_history(
+            client=connected_client,
+            output=output,
+            symbols=["EURUSD"],
+            datasets={Dataset.ticks, Dataset.history_orders},
+            lookback_hours=24,
+            date_to=date_to,
+        )
+        with sqlite3.connect(output) as conn:
+            assert conn.execute("SELECT COUNT(*) FROM ticks").fetchone() == (1,)
+            assert conn.execute(
+                "SELECT COUNT(*) FROM history_orders",
+            ).fetchone() == (1,)
+
+    def test_update_history_with_config_opens_and_closes_connection(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test update_history_with_config manages MT5 connection lifecycle."""
+        mock_client = MagicMock()
+        mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
+        updater = mocker.patch("mt5cli.sdk.update_history")
+        update_history_with_config(
+            output=tmp_path / "config-wrapper.db",
+            symbols=["EURUSD"],
+            datasets={Dataset.history_deals},
+            timeframes=["M1"],
+            flags="ALL",
+            lookback_hours=1,
+            date_to=datetime(2024, 1, 1, tzinfo=UTC),
+            deduplicate=False,
+            create_rate_views=False,
+            with_views=True,
+            include_account_events=False,
+        )
+        mock_client.initialize_and_login_mt5.assert_called_once()
+        mock_client.shutdown.assert_called_once()
+        updater.assert_called_once()
+        assert updater.call_args.kwargs == {
+            "client": mock_client,
+            "output": tmp_path / "config-wrapper.db",
+            "symbols": ["EURUSD"],
+            "datasets": {Dataset.history_deals},
+            "timeframes": ["M1"],
+            "flags": "ALL",
+            "lookback_hours": 1,
+            "date_to": datetime(2024, 1, 1, tzinfo=UTC),
+            "deduplicate": False,
+            "create_rate_views": False,
+            "with_views": True,
+            "include_account_events": False,
+        }
+
+    def test_update_history_with_config_validates_before_connecting(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test invalid inputs fail before MT5 is initialized."""
+        mock_client = MagicMock()
+        mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
+        with pytest.raises(ValueError, match="lookback_hours must be positive"):
+            update_history_with_config(
+                output=tmp_path / "invalid-config.db",
+                symbols=["EURUSD"],
+                lookback_hours=0,
+            )
+        mock_client.initialize_and_login_mt5.assert_not_called()
+        mock_client.shutdown.assert_not_called()
+
+    def test_update_history_with_config_noops_for_empty_datasets(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test empty dataset selection skips MT5 initialization."""
+        mock_client = MagicMock()
+        mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=mock_client)
+        updater = mocker.patch("mt5cli.sdk.update_history")
+        update_history_with_config(
+            output=tmp_path / "empty-config.db",
+            symbols=["EURUSD"],
+            datasets=set(),
+        )
+        mock_client.initialize_and_login_mt5.assert_not_called()
+        mock_client.shutdown.assert_not_called()
+        updater.assert_not_called()
+
+    def test_update_history_defaults_date_to_now(
+        self,
+        connected_client: MagicMock,
+        mocker: MockerFixture,
+        tmp_path: Path,
+    ) -> None:
+        """Test update_history uses current UTC time when date_to is omitted."""
+        captured: dict[str, datetime] = {}
+
+        def capture(
+            *args: object,
+            **_kwargs: object,
+        ) -> tuple[set[Dataset], dict[Dataset, set[str]]]:
+            captured["end"] = args[7]  # type: ignore[assignment]
+            return set(), {}
+
+        mocker.patch("mt5cli.sdk.write_incremental_datasets", side_effect=capture)
+        before = datetime.now(UTC)
+        update_history(
+            client=connected_client,
+            output=tmp_path / "now-default.db",
+            symbols=["EURUSD"],
+            datasets={Dataset.rates},
+            timeframes=["M1"],
+            lookback_hours=12,
+        )
+        after = datetime.now(UTC)
+        assert before <= captured["end"] <= after

uv.lock

@@ -487,7 +487,7 @@ wheels = [
 
 [[package]]
 name = "mt5cli"
-version = "0.4.0"
+version = "0.4.2"
 source = { editable = "." }
 dependencies = [
     { name = "click" },