Bump version to v0.7.2

* feat: add stable MT5Client public API and infrastructure layer

Introduce a reusable public API for downstream trading applications:

- MT5Client as the primary client abstraction with order_check/order_send
- schemas module with DataKind contracts, validation, and normalization
- converters, exceptions, retry, and storage facade modules
- CLI order commands now route through MT5Client
- connected_client made public; retry logic centralized
- Contract tests for API surface, schemas, and storage round-trips
- README and docs updated with Python API usage examples

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

* fix: correct time coercion, broker-safe symbols, and execution docs

- Normalize MT5 time columns with correct second/millisecond units
- Coerce all present known MT5 time fields, including optional order times
- Preserve broker symbol casing in normalize_symbol()
- Document order_send() as a live execution primitive with clear scope boundaries
- Add contract tests for timestamp and symbol normalization behavior

Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Daichi Narushima <dceoy@users.noreply.github.com>

.agents/skills/local-qa/SKILL.md

@@ -1,25 +1,22 @@
-# local-qa
+---
+name: local-qa
+description: Run local QA including formatting, linting, and testing for the repository. Use whenever any file has been updated.
+disable-model-invocation: false
+---
 
-Run local QA checks (format, lint, test) on the repository.
+# Local QA (format, lint, and test)
 
-## When to use
+Run the local QA script `scripts/qa.sh` in this skill.
 
-After making changes to repository files, run `scripts/qa.sh` to validate formatting, linting, and tests.
+## Procedure
 
-## Steps
-
-1. Execute `scripts/qa.sh` and capture the results.
-2. Report successes, failures, warnings, and any modified files.
-
-## If tools are missing
-
-Install them following this priority order:
-
-1. Project package managers (`uv`, `poetry`, npm scripts)
-2. System package managers (`brew`, `apt`)
-3. Language-specific installers (`pipx`, `pip`, `npm`, `go install`)
-
-## Constraints
-
-- Only execute QA and tool-installation commands.
-- If installation fails or requires unavailable privileges, report the attempt and exact failure, then stop.
+- Execute the script exactly as shown above when this skill is triggered.
+- Capture and summarize key output (success/failure, major warnings, and any files modified).
+- If the script fails due to missing tooling (`command not found`, missing executable, or equivalent), install the missing tool(s) and rerun `./scripts/qa.sh`.
+- Install tools using this order of preference:
+  1. Use the project's package manager when applicable (`uv`/`poetry` for Python, package manager scripts/dependencies for Node.js).
+  2. Use a system package manager (`brew` on macOS, `apt` on Debian/Ubuntu) when project-local install is not applicable.
+  3. Use language-specific installers as fallback (`pipx`/`pip`, `npm`, `go install`, etc.).
+- If multiple tools are missing, repeat install -> rerun until QA completes or you hit a blocker.
+- If installation fails or requires unavailable privileges, report what was attempted, the exact failure, and stop.
+- Do not run unrelated commands; only run commands needed for QA and missing-tool installation.

.agents/skills/local-qa/scripts/qa.sh

@@ -13,7 +13,14 @@ uv run pytest
 npx -y prettier --write './**/*.md'
 
 # GitHub Actions
-zizmor --fix=safe .github/workflows
-git ls-files -z -- '.github/workflows/*.yml' | xargs -0 -t actionlint
-git ls-files -z -- '.github/workflows/*.yml' | xargs -0 -t yamllint -d '{"extends": "relaxed", "rules": {"line-length": "disable"}}'
-checkov --framework=all --output=github_failed_only --directory=.
+case "${OSTYPE}" in
+  darwin* | linux* )
+    zizmor --fix=safe .github/workflows
+    git ls-files -z -- '.github/workflows/*.yml' | xargs -0 -t actionlint
+    git ls-files -z -- '.github/workflows/*.yml' | xargs -0 -t yamllint -d '{"extends": "relaxed", "rules": {"line-length": "disable"}}'
+    checkov --framework=all --output=github_failed_only --directory=.
+    ;;
+  * )
+    echo "GitHub Actions linting is only supported on Linux and macOS."
+    ;;
+esac

.github/workflows/ci.yml

@@ -35,7 +35,7 @@ jobs:
       || (github.event_name == 'workflow_dispatch' && inputs.workflow == 'lint-and-test')
     permissions:
       contents: read
-    uses: dceoy/gh-actions-for-devops/.github/workflows/python-package-lint-and-scan.yml@main  # zizmor: ignore[unpinned-uses]
+    uses: dceoy/gh-actions-for-devops/.github/workflows/python-package-lint-and-scan.yml@main   # zizmor: ignore[unpinned-uses]
     with:
       package-path: .
       runs-on: windows-latest
@@ -59,7 +59,6 @@ jobs:
     uses: dceoy/gh-actions-for-devops/.github/workflows/python-package-mkdocs-gh-deploy.yml@main  # zizmor: ignore[unpinned-uses]
     with:
       package-path: .
-      mkdocs-theme: material
       runs-on: ubuntu-slim
     secrets:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

AGENTS.md

@@ -29,9 +29,11 @@ uv sync
 - `mt5cli/`: Main package directory
   - `__init__.py`: Package initialization and exports (`detect_format`, `export_dataframe`)
   - `cli.py`: CLI application with typer-based commands for data export
+  - `utils.py`: Constants, enums, parameter types, parsers, and export utilities
   - `__main__.py`: Entry point for `python -m mt5cli`
 - `tests/`: Comprehensive test suite (pytest-based)
-  - `test_cli.py`: Tests for CLI commands, parameter types, and export functions
+  - `test_cli.py`: Tests for CLI commands and collect-history behavior
+  - `test_utils.py`: Tests for utility constants, parameter types, parsers, and export functions
 - `docs/`: MkDocs documentation with API reference
   - `docs/index.md`: Main documentation
   - `docs/api/`: Auto-generated API documentation for all modules

README.md

@@ -2,10 +2,16 @@
 
 [![CI/CD](https://github.com/dceoy/mt5cli/actions/workflows/ci.yml/badge.svg)](https://github.com/dceoy/mt5cli/actions/workflows/ci.yml)
 
-Command-line tool for exporting MetaTrader 5 data to CSV, JSON, Parquet, and SQLite3.
+Generic MT5 data and execution infrastructure for Python applications. Export from the CLI or import a small, stable Python API in downstream packages.
 
 Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
 
+## Architecture
+
+- **pdmt5** — canonical MT5 client, DataFrame/trading primitives, and MT5 constant parsing (`TIMEFRAME_*`, `COPY_TICKS_*`, order types).
+- **mt5cli** — public `MT5Client` API, standardized dataset schemas, storage helpers, CLI commands, and SQLite history collection built on pdmt5.
+- **mt5api** — sibling HTTP adapter for remote MT5 access; not a dependency of mt5cli.
+
 ## Features
 
 - **Multi-format export**: CSV, JSON, Parquet, and SQLite3 output formats
@@ -13,6 +19,7 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
 - **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
 - **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
 - **Connection management**: Optional credentials, server, and timeout configuration
+- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
 
 ## Installation
 
@@ -20,7 +27,65 @@ Built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data han
 pip install -U mt5cli MetaTrader5
 ```
 
-## Usage
+## Python API (downstream packages)
+
+Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives. `Mt5CliClient` remains available as a backward-compatible alias.
+
+```python
+from datetime import UTC, datetime
+from pathlib import Path
+
+from mt5cli import (
+    DataKind,
+    Dataset,
+    MT5Client,
+    build_config,
+    collect_history,
+    export_dataframe,
+    mt5_session,
+    normalize_dataframe,
+    update_history_with_config,
+)
+
+# Persistent session for multiple calls
+with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
+    rates = client.copy_rates_range(
+        "EURUSD",
+        timeframe="H1",
+        date_from="2024-01-01",
+        date_to="2024-02-01",
+    )
+    positions = client.positions()
+    check = client.order_check({"action": 1, "symbol": "EURUSD", "volume": 0.1})
+
+# Normalize MT5 frames to the public schema contract before storage
+closed_rates = normalize_dataframe(
+    rates, DataKind.rates, symbol="EURUSD", timeframe="H1"
+)
+export_dataframe(closed_rates, Path("rates.csv"), "csv")
+
+# Bulk SQLite history (same behavior as collect-history CLI command)
+collect_history(
+    Path("history.db"),
+    symbols=["EURUSD"],
+    date_from=datetime(2024, 1, 1, tzinfo=UTC),
+    date_to=datetime(2024, 2, 1, tzinfo=UTC),
+    datasets={Dataset.rates, Dataset.history_deals},
+)
+
+# Incremental append for automated pipelines
+update_history_with_config(
+    output="history.db",
+    symbols=["EURUSD"],
+    config=build_config(login=12345),
+)
+```
+
+Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
+
+`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly.
+
+## CLI usage
 
 ```bash
 # Export account information to CSV
@@ -50,36 +115,177 @@ python -m mt5cli -o account.csv account-info
 
 ## Commands
 
-| Command            | Description                                 |
-| ------------------ | ------------------------------------------- |
-| `rates-from`       | Export rates from a start date              |
-| `rates-from-pos`   | Export rates from a start position          |
-| `rates-range`      | Export rates for a date range               |
-| `ticks-from`       | Export ticks from a start date              |
-| `ticks-range`      | Export ticks for a date range               |
-| `account-info`     | Export account information                  |
-| `terminal-info`    | Export terminal information                 |
-| `version`          | Export MetaTrader 5 version information     |
-| `last-error`       | Export the last error information           |
-| `symbols`          | Export symbol list                          |
-| `symbol-info`      | Export symbol details                       |
-| `symbol-info-tick` | Export the last tick for a symbol           |
-| `market-book`      | Export market depth (order book)            |
-| `orders`           | Export active orders                        |
-| `positions`        | Export open positions                       |
-| `history-orders`   | Export historical orders                    |
-| `history-deals`    | Export historical deals                     |
-| `order-check`      | Check funds sufficiency for a trade request |
-| `order-send`       | Send a trade request to the trade server (`--yes` required) |
+| Command                | Description                                                                                                  |
+| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
+| `rates-from`           | Export rates from a start date                                                                               |
+| `rates-from-pos`       | Export rates from a start position                                                                           |
+| `latest-rates`         | Export latest rates from a start position                                                                    |
+| `rates-range`          | Export rates for a date range                                                                                |
+| `ticks-from`           | Export ticks from a start date                                                                               |
+| `ticks-range`          | Export ticks for a date range                                                                                |
+| `ticks-recent`         | Export ticks from a recent trailing window                                                                   |
+| `account-info`         | Export account information                                                                                   |
+| `terminal-info`        | Export terminal information                                                                                  |
+| `version`              | Export MetaTrader 5 version information                                                                      |
+| `last-error`           | Export the last error information                                                                            |
+| `symbols`              | Export symbol list                                                                                           |
+| `symbol-info`          | Export symbol details                                                                                        |
+| `symbol-info-tick`     | Export the last tick for a symbol                                                                            |
+| `minimum-margins`      | Export minimum-volume buy and sell margin requirements                                                       |
+| `market-book`          | Export market depth (order book)                                                                             |
+| `orders`               | Export active orders                                                                                         |
+| `positions`            | Export open positions                                                                                        |
+| `history-orders`       | Export historical orders                                                                                     |
+| `history-deals`        | Export historical deals                                                                                      |
+| `recent-history-deals` | Export historical deals from a recent trailing window                                                        |
+| `mt5-summary`          | Export terminal/account status summary                                                                       |
+| `order-check`          | Check funds sufficiency for a trade request                                                                  |
+| `order-send`           | Send a trade request to the trade server (`--yes` required)                                                  |
+| `collect-history`      | Bundle rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database |
 
 Use `order-check` to validate a request payload before running `order-send --yes`.
 
+### `collect-history`
+
+Collect several historical datasets per symbol into one SQLite database in a single MT5 session. Pick datasets with repeatable `--dataset` (default: all four), choose conflict behavior with `--if-exists append|replace|fail` (default: `fail`), and optionally derive `cash_events` / `positions_reconstructed` views from `history_deals` via `--with-views`.
+
+```bash
+mt5cli -o history.db collect-history \
+  --symbol EURUSD --symbol GBPUSD \
+  --date-from 2024-01-01 --date-to 2024-02-01 \
+  --dataset rates --dataset history-deals \
+  --timeframe M1 --flags ALL --if-exists append --with-views
+```
+
+History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `rates` table records the requested `timeframe` so appended runs at different timeframes remain distinguishable. The `positions_reconstructed` view aggregates trade deals by `position_id`, excludes positions without closing-side entries, and uses volume-weighted open/close prices; reversal deals (`DEAL_ENTRY_INOUT`) are reported via `volume_reversal` / `reversal_count` columns.
+
+### Incremental history SDK
+
+For automated pipelines, use the importable incremental API instead of re-fetching fixed date ranges:
+
+```python
+from pdmt5 import Mt5Config, Mt5DataClient
+from mt5cli import Dataset, update_history, update_history_with_config
+
+# Reuse an already-connected pdmt5 client (does not open/close MT5)
+client = Mt5DataClient(config=Mt5Config(login=12345))
+client.initialize_and_login_mt5()
+try:
+    update_history(
+        client=client,
+        output="history.db",
+        symbols=["EURUSD", "GBPUSD"],
+        datasets={Dataset.rates, Dataset.history_deals},
+        timeframes=["M1", "H1"],  # default: all fixed MT5 timeframes
+        lookback_hours=24,
+        create_rate_views=True,
+        with_views=True,
+        include_account_events=True,
+    )
+finally:
+    client.shutdown()
+
+# Standalone wrapper that opens and closes MT5 for you
+update_history_with_config(
+    output="history.db",
+    symbols=["EURUSD"],
+    config=Mt5Config(login=12345),
+)
+```
+
+- **`collect-history`**: explicit date-range export into SQLite.
+- **`update_history`**: incremental append based on existing SQLite `MAX(time)` per symbol (and timeframe for rates); account-level deals use a separate cursor when `include_account_events=True`.
+- **`rates` table**: normalized storage with `symbol` and `timeframe` columns.
+- **Rate compatibility views**: mt5cli manages all `rate_*` views. Naming is `rate_<symbol>__<timeframe>` when a symbol has one timeframe, otherwise `rate_<symbol>__<granularity>_<timeframe>` (for example `rate_EURUSD__M1_1`). Stale `rate_*` views are dropped and recreated when rates change for offline tools such as mteor optimize.
+- **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
+- **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
+- **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
+- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
+- **Latest closed bars**: use `collect_latest_closed_rates_for_accounts()` when downstream logic must exclude the still-forming current bar. It fetches `count + 1` bars at `start_pos=0`, drops the last row with `drop_forming_rate_bar()`, and validates each series is non-empty. `collect_latest_closed_rates_by_granularity()` returns the same data keyed by `(symbol, granularity_name)` such as `("EURUSD", "M1")`.
+
+```python
+from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
+
+rates = collect_latest_closed_rates_by_granularity(
+    [AccountSpec(symbols=["EURUSD", "GBPUSD"], login=12345)],
+    ["M1", "H1"],
+    count=500,
+    retry_count=3,
+)
+eurusd_m1 = rates["EURUSD", "M1"]  # closed bars only
+```
+
+- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
+- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`, `sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability errors for history API methods without advancing the throttle (other `AttributeError` / `TypeError` values always propagate).
+- **Trading session helpers**: use `mt5_trading_session()` for a trading-capable `pdmt5.Mt5TradingClient` that initializes/logs in via `Mt5Config.path` and always shuts down safely. Pair with `detect_position_side()`, `calculate_margin_and_volume()`, and `determine_order_limits()` for generic position and sizing utilities. The read-only `mt5_session()` / `Mt5CliClient` SDK is unchanged.
+- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
+- **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `MT5Client` that shuts down on exit.
+- **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
+- **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.
+
 ## Requirements
 
 - Python 3.11+
 - Windows OS (MetaTrader 5 requirement)
 - MetaTrader 5 platform installed
 
+### Migration note for mteor
+
+Replace local MT5 lifecycle and trading helper code with mt5cli imports:
+
+```python
+# Before (local mteor helpers)
+# with local_mt5_trading_session(config) as client:
+#     side = local_detect_position_side(client, symbol)
+#     sizing = local_calculate_margin_and_volume(client, symbol, unit_ratio, preserved_ratio)
+#     limits = local_determine_order_limits(client, symbol, side, sl_ratio, tp_ratio)
+
+# After (mt5cli shared layer)
+from pdmt5 import Mt5Config
+from mt5cli import (
+    calculate_margin_and_volume,
+    detect_position_side,
+    determine_order_limits,
+    mt5_trading_session,
+)
+
+with mt5_trading_session(
+    Mt5Config(path=terminal_path, login=login), retry_count=2
+) as client:
+    side = detect_position_side(client, symbol)
+    sizing = calculate_margin_and_volume(
+        client, symbol, unit_margin_ratio=0.5, preserved_margin_ratio=0.2
+    )
+    if side is not None:
+        limits = determine_order_limits(
+            client,
+            symbol,
+            side,
+            stop_loss_limit_ratio=0.01,
+            take_profit_limit_ratio=0.02,
+        )
+```
+
+Throttled history updates use a separate read-only session:
+
+```python
+from pdmt5 import Mt5Config, Mt5DataClient
+
+from mt5cli import ThrottledHistoryUpdater
+
+updater = ThrottledHistoryUpdater(
+    output="history.db", interval_seconds=60, suppress_errors=True
+)
+client = Mt5DataClient(config=Mt5Config(login=login))
+client.initialize_and_login_mt5()
+try:
+    updater.update(client, ["EURUSD"])
+finally:
+    client.shutdown()
+```
+
+Read-only collectors can keep using `mt5_session()` and `MT5Client` (or the `Mt5CliClient` alias) without changes.
+
 ## Development
 
 ```bash

docs/api/client.md

@@ -0,0 +1,3 @@
+# Client
+
+::: mt5cli.client

docs/api/converters.md

@@ -0,0 +1,3 @@
+# Converters
+
+::: mt5cli.converters

docs/api/exceptions.md

@@ -0,0 +1,3 @@
+# Exceptions
+
+::: mt5cli.exceptions

docs/api/history.md

@@ -0,0 +1,229 @@
+# History Collection (SQLite)
+
+::: mt5cli.history
+
+## `collect-history` schema
+
+The `collect-history` command (and the matching `collect_history` SDK function) writes
+selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
+names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
+
+- `symbol` is prepended on every table.
+- `timeframe` is prepended on `rates` so appended runs at different bar sizes stay
+  distinguishable.
+
+SQLite does not declare foreign keys. Rows are linked logically by `symbol`, time
+windows, and (for deals) `position_id` / `order`. Duplicate rows are removed on
+append using dataset-specific keys (for example `ticket` on history tables, or
+`(symbol, timeframe, time)` on rates).
+
+Optional views are created when `--with-views` is set and the `history-deals` dataset
+was written.
+
+### Entity-relationship diagram
+
+Sample layout for a full collection with `--with-views`:
+
+```mermaid
+erDiagram
+    rates {
+        TEXT symbol "dedup key"
+        INTEGER timeframe "dedup key"
+        TEXT time "dedup key"
+        REAL open
+        REAL high
+        REAL low
+        REAL close
+        INTEGER tick_volume
+        INTEGER spread
+        INTEGER real_volume
+    }
+
+    ticks {
+        TEXT symbol "dedup key"
+        TEXT time "dedup key"
+        INTEGER time_msc "dedup key (preferred)"
+        REAL bid
+        REAL ask
+        REAL last
+        INTEGER volume
+        INTEGER flags
+        REAL volume_real
+    }
+
+    history_orders {
+        INTEGER ticket "dedup key"
+        TEXT symbol
+        TEXT time
+        INTEGER type
+        INTEGER state
+        REAL volume_initial
+        REAL price_open
+        REAL price_current
+        INTEGER magic
+    }
+
+    history_deals {
+        INTEGER ticket "dedup key"
+        INTEGER order
+        INTEGER position_id "groups position view"
+        TEXT symbol
+        TEXT time
+        INTEGER type "0/1 trade, else cash event"
+        INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
+        REAL volume
+        REAL price
+        REAL profit
+        REAL commission
+        REAL swap
+        REAL fee
+    }
+
+    cash_events {
+        INTEGER ticket
+        TEXT symbol
+        TEXT time
+        INTEGER type
+        REAL profit
+    }
+
+    positions_reconstructed {
+        INTEGER position_id
+        TEXT symbol
+        TEXT open_time
+        TEXT close_time
+        INTEGER direction
+        REAL volume_open
+        REAL volume_close
+        REAL volume_reversal
+        REAL open_price
+        REAL close_price
+        REAL total_profit
+        INTEGER reversal_count
+        INTEGER deals_count
+    }
+
+    rates ||--o{ history_deals : "symbol (logical)"
+    ticks ||--o{ history_deals : "symbol (logical)"
+    history_orders ||--o{ history_deals : "order ~ ticket (logical)"
+    history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
+    history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
+```
+
+### Tables and views
+
+| Object                    | Kind  | Source               | Notes                                                                                       |
+| ------------------------- | ----- | -------------------- | ------------------------------------------------------------------------------------------- |
+| `rates`                   | table | `copy_rates_range`   | Indexed on `(symbol, timeframe, time)` when columns exist.                                  |
+| `ticks`                   | table | `copy_ticks_range`   | Indexed on `(symbol, time)` when columns exist.                                             |
+| `history_orders`          | table | `history_orders_get` | Fetched per `--symbol`, then concatenated.                                                  |
+| `history_deals`           | table | `history_deals_get`  | Fetched per `--symbol`, then concatenated. Indexed on `(position_id, symbol)` when present. |
+| `cash_events`             | view  | `history_deals`      | Non-trade deal types (deposits, balance ops, etc.). Requires `type` column.                 |
+| `positions_reconstructed` | view  | `history_deals`      | One row per closed `position_id`; volume-weighted prices and reversal stats.                |
+
+Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning
+when required columns are missing.
+
+### Incremental collection
+
+The `update_history` SDK path uses the same base tables and optional
+`cash_events` / `positions_reconstructed` views. It additionally maintains
+`rate_<symbol>__<timeframe>` compatibility views when `create_rate_views=True`.
+
+### Rate view resolution
+
+Downstream tools can resolve mt5cli-managed compatibility view names from an
+existing SQLite history database without creating files or guessing naming
+schemes:
+
+```python
+from pathlib import Path
+
+from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
+
+# Single symbol and granularity
+view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
+
+# Batch resolution in row-major order
+views = resolve_rate_view_names(
+    Path("history.db"),
+    ["EURUSD", "GBPUSD"],
+    ["M1", "H1"],
+)
+```
+
+Resolution rules:
+
+- Returns `rate_<symbol>__<timeframe>` when a symbol stores one timeframe.
+- Returns `rate_<symbol>__<granularity>_<timeframe>` when multiple timeframes
+  are stored for the same symbol.
+- When multiple naming candidates apply, prefers an existing managed
+  `rate_*__*` view from the candidate list.
+- Falls back to single-timeframe naming when the database path is missing or
+  `rates` metadata is unavailable.
+- Pass `require_existing=True` to raise `ValueError` instead of returning a
+  best-guess name when the database or view is missing.
+- Accepts either a SQLite path or an open `sqlite3.Connection`.
+
+### Rate data loading
+
+Use `load_rate_data()` to load a table or view from a SQLite path, or
+`load_rate_data_from_connection()` when you already have a connection:
+
+```python
+from pathlib import Path
+
+from mt5cli import load_rate_data
+from mt5cli.history import resolve_rate_view_name
+
+view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
+rates = load_rate_data(Path("history.db"), view, count=1000)
+```
+
+The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
+validates that `time` exists, parses timestamps with pandas, and returns a
+DataFrame indexed by ascending `DatetimeIndex` named `time`.
+
+### Multi-series rate loading
+
+For loading many rate series at once, build neutral `RateTarget` pairs and load
+them from SQLite in one call. View names are resolved via the same
+compatibility-view rules, or you can pass `explicit_tables` to bypass resolution:
+
+```python
+from pathlib import Path
+
+from mt5cli import build_rate_targets, load_rate_series_from_sqlite
+
+targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
+series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
+frame = series["EURUSD", 1]  # keyed by (symbol, integer timeframe)
+```
+
+- `build_rate_targets()` returns `RateTarget(symbol, timeframe)` pairs in
+  row-major order, normalizing timeframe names such as `"M1"` to their integer
+  values; set `allow_missing_symbol=True` to address series solely by
+  `explicit_tables` (targets carry `symbol=None`).
+- `resolve_rate_tables()` maps targets to table or view names and validates that
+  any `explicit_tables` count matches the target count. Pass
+  `require_existing=True` to raise `ValueError` instead of returning a
+  best-guess name when the database or managed view is missing. When
+  `explicit_tables` is provided, names are returned as-is and
+  `require_existing` is ignored.
+- `load_rate_series_from_sqlite()` returns a mapping keyed by
+  `(symbol, integer timeframe)`. Unless `explicit_tables` is supplied, it
+  requires existing managed `rate_*` compatibility views and raises
+  `ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
+  are rejected.
+- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
+  loads the series, and rekeys the result by granularity name to avoid
+  converting integer timeframes downstream:
+
+  ```python
+  from mt5cli import load_rate_series_by_granularity
+
+  series = load_rate_series_by_granularity(
+      "history.db", ["EURUSD"], ["M1", "H1"], count=1000
+  )
+  frame = series["EURUSD", "M1"]  # keyed by (symbol | None, granularity_name)
+  ```

docs/api/index.md

@@ -1,63 +1,53 @@
 # API Reference
 
-This section contains the complete API documentation for mt5cli.
+This section documents the mt5cli public Python API and CLI modules.
 
-## Modules
+## Public API layers
 
-The mt5cli package consists of the following modules:
+| Module                                    | Purpose                                                                   |
+| ----------------------------------------- | ------------------------------------------------------------------------- |
+| [Client](client.md)                       | `MT5Client` session abstraction for data access and order primitives      |
+| [Schemas](schemas.md)                     | Canonical DataFrame contracts and normalization helpers                   |
+| [Storage](storage.md)                     | CSV/JSON/Parquet/SQLite export and history collection helpers             |
+| [Converters](converters.md)               | Symbol, timeframe, timezone, and date-range utilities                     |
+| [Exceptions](exceptions.md)               | Stable mt5cli exception types and MT5 error normalization                 |
+| [SDK](sdk.md)                             | Module-level fetch helpers, multi-account collectors, incremental history |
+| [Trading](trading.md)                     | Trading-capable sessions and operational helpers                          |
+| [History Collection (SQLite)](history.md) | SQLite schema, incremental writes, dedup, and rate views                  |
+| [CLI](cli.md)                             | Typer commands that delegate to the Python API                            |
+| [Utils](utils.md)                         | Parsing helpers and Click parameter types                                 |
 
-### [CLI](cli.md)
+## Architecture overview
 
-Command-line interface module providing typer-based commands for exporting MetaTrader 5 data to CSV, JSON, Parquet, and SQLite3 formats.
-
-## Architecture Overview
-
-The package follows a simple architecture built on top of pdmt5:
-
-1. **CLI Layer** (`cli.py`): Typer application with subcommands for each data type, custom Click parameter types for datetime/timeframe/tick flags parsing, and format detection/export utilities.
-2. **Data Layer** (via `pdmt5`): Uses `Mt5DataClient` and `Mt5Config` from the pdmt5 package for all MetaTrader 5 data access.
-
-## Usage Guidelines
-
-All modules follow these conventions:
-
-- **Type Safety**: All functions include comprehensive type hints
-- **Error Handling**: User-friendly error messages via typer
-- **Documentation**: Google-style docstrings with examples
-- **Validation**: Custom Click parameter types for input validation
-
-## Quick Start
-
-```bash
-# Export account information to CSV
-mt5cli -o account.csv account-info
-
-# Export EURUSD H1 rates to Parquet
-mt5cli -o rates.parquet rates-from --symbol EURUSD --timeframe H1 \
-  --date-from 2024-01-01 --count 1000
-
-# Export ticks to JSON
-mt5cli -o ticks.json ticks-from --symbol EURUSD \
-  --date-from 2024-01-01 --count 500 --flags ALL
-
-# Export to SQLite3 with custom table name
-mt5cli -o data.db --table symbols symbols --group "*USD*"
+```mermaid
+flowchart TD
+    App["Downstream application"] --> Client["MT5Client"]
+    CLI["mt5cli CLI"] --> Client
+    Client --> SDK["sdk / pdmt5"]
+    Client --> Schemas["schemas"]
+    Storage["storage"] --> History["history SQLite"]
+    Storage --> Utils["utils export"]
+    SDK --> PDMT5["pdmt5.Mt5DataClient"]
 ```
 
-## Python API
+Downstream packages should depend on the package root exports (`MT5Client`, `DataKind`, `normalize_dataframe`, `export_dataframe`, `collect_history`, etc.) rather than private modules.
+
+`MT5Client.order_send()` is a live execution primitive that can place real trades. mt5cli exposes minimal execution helpers only; strategy logic, signals, backtests, and optimization remain out of scope and must be implemented downstream with explicit execution gating.
+
+## Quick start
 
 ```python
-from mt5cli import detect_format, export_dataframe
-import pandas as pd
+from mt5cli import MT5Client, build_config, mt5_session
 
-# Detect output format from file extension
-fmt = detect_format(Path("output.parquet"))  # Returns "parquet"
-
-# Export a DataFrame
-df = pd.DataFrame({"symbol": ["EURUSD"], "bid": [1.1234]})
-export_dataframe(df, Path("output.csv"), "csv")
+with mt5_session(build_config(login=12345)) as client:
+    rates = client.copy_rates_range("EURUSD", "H1", "2024-01-01", "2024-02-01")
+    positions = client.positions()
 ```
 
-## Examples
+```bash
+mt5cli -o account.csv account-info
+mt5cli -o rates.parquet rates-range --symbol EURUSD --timeframe H1 \
+  --date-from 2024-01-01 --date-to 2024-02-01
+```
 
-See individual module pages for detailed usage examples and code samples.
+See individual module pages for detailed usage examples.

docs/api/schemas.md

@@ -0,0 +1,3 @@
+# Schemas
+
+::: mt5cli.schemas

docs/api/sdk.md

@@ -0,0 +1,114 @@
+# SDK Module
+
+::: mt5cli.sdk
+
+## Resilient multi-account orchestration
+
+The SDK ships strategy-agnostic helpers for building long-running collectors on
+top of the read-only client. None of them depend on a particular trading
+application.
+
+### Retrying transient rate collection
+
+`collect_latest_rates_for_accounts_with_retries()` wraps
+`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
+`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
+failure is re-raised once `retry_count` is exhausted.
+
+```python
+from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
+
+accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
+rates = collect_latest_rates_for_accounts_with_retries(
+    accounts,
+    ["M1", "H1"],
+    count=500,
+    retry_count=3,
+    backoff_base=2,  # sleeps 2s, 4s, 8s between attempts
+)
+```
+
+### Latest closed rate bars
+
+MetaTrader 5 `start_pos=0` includes the still-forming current bar as the last
+row. `collect_latest_closed_rates_for_accounts()` fetches `count + 1` bars,
+drops that row with `drop_forming_rate_bar()`, and validates each series is
+non-empty. Use `collect_latest_closed_rates_by_granularity()` when callers
+prefer keys such as `("EURUSD", "M1")` instead of integer timeframes.
+
+```python
+from mt5cli import AccountSpec, collect_latest_closed_rates_by_granularity
+
+rates = collect_latest_closed_rates_by_granularity(
+    [AccountSpec(symbols=["EURUSD"], login=12345)],
+    ["M1", "H1"],
+    count=500,
+    retry_count=3,
+)
+closed_m1 = rates["EURUSD", "M1"]
+```
+
+### Resolving credentials and `${ENV_VAR}` placeholders
+
+`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
+values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
+secrets out of plan/config files. A missing environment variable raises
+`ValueError`.
+
+```python
+import os
+
+from mt5cli import AccountSpec, resolve_account_specs
+
+os.environ["MT5_LOGIN"] = "12345"
+os.environ["MT5_PASSWORD"] = "secret"
+accounts = [
+    AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
+]
+
+resolved = resolve_account_specs(accounts, server="Broker-Demo")
+# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
+```
+
+### Throttled incremental history updates
+
+`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
+between successful runs (using a monotonic clock), so an application loop can
+call it every iteration without over-fetching.
+
+```python
+from pdmt5 import Mt5Config, Mt5DataClient
+
+from mt5cli import Dataset, ThrottledHistoryUpdater
+
+updater = ThrottledHistoryUpdater(
+    output="history.db",
+    datasets={Dataset.rates},
+    timeframes=["M1"],
+    interval_seconds=60,  # <= 0 updates on every call
+)
+
+client = Mt5DataClient(config=Mt5Config(login=12345))
+client.initialize_and_login_mt5()
+try:
+    while True:
+        updater.update(client, ["EURUSD", "GBPUSD"])  # no-op until 60s elapse
+        # ... do other work; break when shutting down ...
+finally:
+    client.shutdown()
+```
+
+By default recoverable errors (`Mt5TradingError`, `Mt5RuntimeError`,
+`sqlite3.Error`, `ValueError`, `OSError`, and MT5 client capability
+`AttributeError` / `TypeError` for history API methods) propagate so the caller
+controls logging; pass `suppress_errors=True` to swallow them and return
+`False` without advancing the throttle. Other `AttributeError` / `TypeError`
+values always propagate. Input validation (`_resolve_update_history_request`)
+runs before any MT5 or SQLite calls, but when `suppress_errors=True` the
+resulting `ValueError` is suppressed along with other recoverable errors.
+
+## Trading-capable sessions
+
+For order placement and trading calculations, use the dedicated
+[Trading module](trading.md). The read-only `Mt5CliClient` and `mt5_session()`
+helpers in this module are unchanged.

docs/api/storage.md

@@ -0,0 +1,3 @@
+# Storage
+
+::: mt5cli.storage

docs/api/trading.md

@@ -0,0 +1,70 @@
+# Trading Module
+
+::: mt5cli.trading
+
+## Trading-capable MT5 sessions
+
+`mt5_trading_session()` complements the read-only `mt5_session()` helper in
+`sdk.py`. It yields a connected `pdmt5.Mt5TradingClient`, uses
+`Mt5Config.path` to launch the terminal when configured, and always calls
+`shutdown()` on exit.
+
+```python
+from pdmt5 import Mt5Config
+
+from mt5cli import mt5_trading_session
+
+with mt5_trading_session(
+    Mt5Config(path=r"C:\Program Files\MetaTrader 5\terminal64.exe", login=12345),
+    retry_count=2,
+) as client:
+    positions = client.positions_get_as_df(symbol="EURUSD")
+```
+
+The read-only `Mt5CliClient` / `mt5_session()` API is unchanged.
+
+## Operational trading helpers
+
+These helpers are strategy-agnostic and do not depend on signal detection,
+betting logic, or scheduling code in downstream applications.
+
+```python
+from mt5cli import (
+    calculate_margin_and_volume,
+    detect_position_side,
+    determine_order_limits,
+)
+
+side = detect_position_side(client, "EURUSD")
+sizing = calculate_margin_and_volume(
+    client,
+    "EURUSD",
+    unit_margin_ratio=0.5,
+    preserved_margin_ratio=0.2,
+)
+limits = determine_order_limits(
+    client,
+    "EURUSD",
+    side="long",
+    stop_loss_limit_ratio=0.01,
+    take_profit_limit_ratio=0.02,
+)
+```
+
+Protective ratios must satisfy `0 <= ratio < 1`; `0` omits that level.
+`calculate_margin_and_volume()` clamps negative `margin_free` to `0.0`
+before sizing.
+
+## Migration from mteor-local helpers
+
+| mteor-local concern                                      | mt5cli replacement                              |
+| -------------------------------------------------------- | ----------------------------------------------- |
+| Manual terminal spawn/kill around trading code           | `mt5_trading_session()`                         |
+| Local position-side detection                            | `detect_position_side()`                        |
+| Local margin/volume sizing                               | `calculate_margin_and_volume()`                 |
+| Local SL/TP price derivation                             | `determine_order_limits()`                      |
+| Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)` |
+
+Keep read-only data collection on `mt5_session()` / `Mt5CliClient`; use
+`mt5_trading_session()` only where order placement or trading calculations are
+required.

docs/api/utils.md

@@ -0,0 +1,3 @@
+# Utils Module
+
+::: mt5cli.utils

docs/index.md

@@ -1,10 +1,16 @@
 # mt5cli
 
-Command-line tool for MetaTrader 5 data export.
+Generic MT5 data and execution infrastructure for Python applications.
 
 ## Overview
 
-mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple file formats. It is built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
+mt5cli provides a stable `MT5Client` Python API, standardized dataset schemas, storage helpers, and a CLI for exporting MetaTrader 5 data. It is built on top of [pdmt5](https://github.com/dceoy/pdmt5), a pandas-based data handler for MetaTrader 5.
+
+## Architecture
+
+- **pdmt5** — canonical MT5 client, DataFrame/trading primitives, and MT5 constant parsing (`TIMEFRAME_*`, `COPY_TICKS_*`, order types).
+- **mt5cli** — public `MT5Client` API, schema contracts, storage helpers, CLI commands, and SQLite history collection built on pdmt5.
+- **mt5api** — sibling HTTP adapter for remote MT5 access; not a dependency of mt5cli.
 
 ## Features
 
@@ -13,6 +19,7 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
 - **Comprehensive data access**: Rates, ticks, account info, symbols, orders, positions, and trading history
 - **Flexible timeframes**: Named timeframes (M1, H1, D1, etc.) and numeric values
 - **Connection management**: Optional credentials, server, and timeout configuration
+- **SQLite rate loading**: Load mt5cli-managed rate tables/views for offline workflows
 
 ## Installation
 
@@ -20,6 +27,69 @@ mt5cli is a CLI application that exports MetaTrader 5 trading data to multiple f
 pip install mt5cli
 ```
 
+## Python API for downstream packages
+
+Import `MT5Client` for generic MT5 data access, schema normalization, and optional order primitives. `Mt5CliClient` remains available as a backward-compatible alias.
+
+```python
+from datetime import UTC, datetime
+from pathlib import Path
+
+from mt5cli import (
+    DataKind,
+    Dataset,
+    MT5Client,
+    build_config,
+    collect_history,
+    export_dataframe,
+    load_rate_data,
+    minimum_margins,
+    mt5_session,
+    normalize_dataframe,
+    recent_ticks,
+    resolve_rate_view_name,
+)
+
+# Persistent session for multiple calls
+with mt5_session(build_config(login=12345, server="Broker-Demo")) as client:
+    rates = client.copy_rates_range(
+        "EURUSD",
+        timeframe="H1",
+        date_from="2024-01-01",
+        date_to="2024-02-01",
+    )
+    positions = client.positions()
+    check = client.order_check({"action": 1, "symbol": "EURUSD", "volume": 0.1})
+
+# Normalize MT5 frames to the public schema contract before storage
+closed_rates = normalize_dataframe(
+    rates, DataKind.rates, symbol="EURUSD", timeframe="H1"
+)
+export_dataframe(closed_rates, Path("rates.csv"), "csv")
+
+# Offline rate loading from mt5cli-managed SQLite history
+view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
+offline_rates = load_rate_data(Path("history.db"), view, count=1000)
+
+# One-off helpers still work without instantiating a client
+ticks = recent_ticks("EURUSD", seconds=300)
+margins = minimum_margins("EURUSD")
+
+collect_history(
+    Path("history.db"),
+    symbols=["EURUSD", "GBPUSD"],
+    date_from=datetime(2024, 1, 1, tzinfo=UTC),
+    date_to=datetime(2024, 2, 1, tzinfo=UTC),
+    datasets={Dataset.rates, Dataset.history_deals},
+)
+```
+
+Schema contracts live in `mt5cli.schemas` (`DataKind`, `validate_schema`, `normalize_dataframe`). Storage helpers are re-exported from `mt5cli.storage` and the package root.
+
+`MT5Client.order_send()` is a live execution primitive: it can place real trades on the connected account. mt5cli does not implement strategy logic, signal generation, backtesting, or optimization — downstream applications must gate live execution explicitly (the CLI requires `--yes` for `order-send`).
+
+`MT5Client.mt5_summary()` returns structured nested Python values. Use `MT5Client.mt5_summary_as_df()` when you need a one-row DataFrame for export.
+
 ## Quick Start
 
 ```bash
@@ -50,14 +120,16 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
 | ---------------- | ---------------------------------- |
 | `rates-from`     | Export rates from a start date     |
 | `rates-from-pos` | Export rates from a start position |
+| `latest-rates`   | Export latest rates                |
 | `rates-range`    | Export rates for a date range      |
 
 ### Ticks
 
-| Command       | Description                    |
-| ------------- | ------------------------------ |
-| `ticks-from`  | Export ticks from a start date |
-| `ticks-range` | Export ticks for a date range  |
+| Command        | Description                         |
+| -------------- | ----------------------------------- |
+| `ticks-from`   | Export ticks from a start date      |
+| `ticks-range`  | Export ticks for a date range       |
+| `ticks-recent` | Export ticks from a trailing window |
 
 ### Information
 
@@ -70,21 +142,55 @@ mt5cli --login 12345 --password mypass --server MyBroker-Demo \
 | `symbols`          | Export symbol list                      |
 | `symbol-info`      | Export symbol details                   |
 | `symbol-info-tick` | Export the last tick for a symbol       |
+| `minimum-margins`  | Export minimum-volume margin summary    |
 | `market-book`      | Export market depth (order book)        |
 
 ### Trading
 
-| Command          | Description                                 |
-| ---------------- | ------------------------------------------- |
-| `orders`         | Export active orders                        |
-| `positions`      | Export open positions                       |
-| `history-orders` | Export historical orders                    |
-| `history-deals`  | Export historical deals                     |
-| `order-check`    | Check funds sufficiency for a trade request |
-| `order-send`     | Send a trade request to the trade server (`--yes` required) |
+| Command                | Description                                                 |
+| ---------------------- | ----------------------------------------------------------- |
+| `orders`               | Export active orders                                        |
+| `positions`            | Export open positions                                       |
+| `history-orders`       | Export historical orders                                    |
+| `history-deals`        | Export historical deals                                     |
+| `recent-history-deals` | Export historical deals from a trailing window              |
+| `mt5-summary`          | Export terminal/account status summary                      |
+| `order-check`          | Check funds sufficiency for a trade request                 |
+| `order-send`           | Send a trade request to the trade server (`--yes` required) |
 
 Use `order-check` to validate a request payload before running `order-send --yes`.
 
+### Bulk Collection
+
+| Command           | Description                                                                                                                                        |
+| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `collect-history` | Collect rates, ticks, history-orders, and history-deals for one or more symbols into a single SQLite database (optional cash-event/position views) |
+
+```bash
+mt5cli -o history.db collect-history \
+  --symbol EURUSD --symbol GBPUSD \
+  --date-from 2024-01-01 --date-to 2024-02-01 \
+  --dataset rates --dataset history-deals \
+  --timeframe M1 --flags ALL --if-exists append --with-views
+```
+
+`collect-history` options:
+
+| Option         | Default    | Description                                                                                   |
+| -------------- | ---------- | --------------------------------------------------------------------------------------------- |
+| `--symbol/-s`  | _required_ | Symbol to collect (repeat for multiple).                                                      |
+| `--date-from`  | _required_ | Start date in ISO 8601.                                                                       |
+| `--date-to`    | _required_ | End date in ISO 8601.                                                                         |
+| `--dataset`    | all four   | Repeatable: `rates`, `ticks`, `history-orders`, `history-deals`.                              |
+| `--timeframe`  | `M1`       | Rates timeframe; recorded in a `timeframe` column on the `rates` table.                       |
+| `--flags`      | `ALL`      | Tick copy flags forwarded to `copy_ticks_range`.                                              |
+| `--if-exists`  | `fail`     | `append`, `replace`, or `fail` when a target table already exists.                            |
+| `--with-views` | off        | Add `cash_events` and `positions_reconstructed` views (requires the `history-deals` dataset). |
+
+History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
+
+See the [History schema diagram](api/history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
+
 ## Global Options
 
 | Option         | Description                                             |
@@ -109,7 +215,9 @@ Use `order-check` to validate a request payload before running `order-send --yes
 
 Browse the API documentation for detailed module information:
 
-- [CLI Module](api/cli.md) - CLI application with export commands and utility functions
+- [CLI Module](api/cli.md) - CLI application with export commands
+- [SDK Module](api/sdk.md) - Programmatic read-only data collection API
+- [Utils Module](api/utils.md) - Constants, parameter types, parsers, and export utilities
 
 ## Development
 

mkdocs.yml

@@ -1,5 +1,5 @@
 site_name: mt5cli API Documentation
-site_description: Command-line tool for MetaTrader 5
+site_description: Generic MT5 data and execution infrastructure for Python
 site_author: dceoy
 site_url: https://github.com/dceoy/mt5cli
 
@@ -24,6 +24,7 @@ theme:
   features:
     - content.code.annotate
     - content.code.copy
+    - content.code.mermaid
     - navigation.indexes
     - navigation.sections
     - navigation.tabs
@@ -55,7 +56,16 @@ nav:
   - Home: index.md
   - API Reference:
       - Overview: api/index.md
+      - Client: api/client.md
+      - Schemas: api/schemas.md
+      - Storage: api/storage.md
+      - Converters: api/converters.md
+      - Exceptions: api/exceptions.md
       - CLI: api/cli.md
+      - SDK: api/sdk.md
+      - Trading: api/trading.md
+      - History Collection (SQLite): api/history.md
+      - Utils: api/utils.md
 
 markdown_extensions:
   - admonition

mt5cli/__init__.py

@@ -1,12 +1,205 @@
-"""mt5cli: Command-line tool for MetaTrader 5."""
+"""mt5cli: Generic MT5 data and execution infrastructure for Python applications."""
 
 from importlib.metadata import version
 
-from .cli import detect_format, export_dataframe
+from .client import MT5Client, build_config, mt5_session
+from .converters import (
+    ensure_utc,
+    granularity_name,
+    normalize_symbol,
+    normalize_symbols,
+    parse_date_range,
+    recent_window,
+)
+from .exceptions import (
+    Mt5CliError,
+    Mt5ConnectionError,
+    Mt5OperationError,
+    Mt5SchemaError,
+    call_with_normalized_errors,
+    is_recoverable_mt5_error,
+    normalize_mt5_exception,
+)
+from .history import (
+    RateTarget,
+    build_rate_targets,
+    build_rate_view_name,
+    drop_forming_rate_bar,
+    load_rate_data,
+    load_rate_data_from_connection,
+    load_rate_series_by_granularity,
+    load_rate_series_from_sqlite,
+    resolve_history_datasets,
+    resolve_history_tick_flags,
+    resolve_history_timeframes,
+    resolve_rate_tables,
+    resolve_rate_view_name,
+    resolve_rate_view_names,
+)
+from .schemas import (
+    DEDUP_KEYS,
+    KNOWN_MT5_TIME_COLUMNS,
+    REQUIRED_COLUMNS,
+    TIME_COLUMNS,
+    DataKind,
+    normalize_dataframe,
+    normalize_time_columns,
+    schema_columns,
+    validate_schema,
+)
+from .sdk import (
+    AccountSpec,
+    Mt5CliClient,
+    ThrottledHistoryUpdater,
+    account_info,
+    collect_history,
+    collect_latest_closed_rates_by_granularity,
+    collect_latest_closed_rates_for_accounts,
+    collect_latest_rates,
+    collect_latest_rates_for_accounts,
+    collect_latest_rates_for_accounts_with_retries,
+    copy_rates_from,
+    copy_rates_from_pos,
+    copy_rates_range,
+    copy_ticks_from,
+    copy_ticks_range,
+    history_deals,
+    history_orders,
+    last_error,
+    latest_rates,
+    market_book,
+    minimum_margins,
+    mt5_summary,
+    mt5_summary_as_df,
+    orders,
+    positions,
+    recent_history_deals,
+    recent_ticks,
+    resolve_account_spec,
+    resolve_account_specs,
+    substitute_env_placeholders,
+    symbol_info,
+    symbol_info_tick,
+    symbols,
+    terminal_info,
+    update_history,
+    update_history_with_config,
+)
+from .sdk import (
+    version as mt5_version,
+)
+from .storage import (
+    Dataset,
+    IfExists,
+    detect_format,
+    export_dataframe,
+    export_dataframe_to_sqlite,
+)
+from .trading import (
+    calculate_margin_and_volume,
+    detect_position_side,
+    determine_order_limits,
+    mt5_trading_session,
+)
+from .utils import (
+    TICK_FLAG_MAP,
+    TIMEFRAME_MAP,
+    parse_datetime,
+    parse_tick_flags,
+    parse_timeframe,
+)
 
 __version__ = version(__package__) if __package__ else None
 
 __all__ = [
+    "DEDUP_KEYS",
+    "KNOWN_MT5_TIME_COLUMNS",
+    "REQUIRED_COLUMNS",
+    "TICK_FLAG_MAP",
+    "TIMEFRAME_MAP",
+    "TIME_COLUMNS",
+    "AccountSpec",
+    "DataKind",
+    "Dataset",
+    "IfExists",
+    "MT5Client",
+    "Mt5CliClient",
+    "Mt5CliError",
+    "Mt5ConnectionError",
+    "Mt5OperationError",
+    "Mt5SchemaError",
+    "RateTarget",
+    "ThrottledHistoryUpdater",
+    "account_info",
+    "build_config",
+    "build_rate_targets",
+    "build_rate_view_name",
+    "calculate_margin_and_volume",
+    "call_with_normalized_errors",
+    "collect_history",
+    "collect_latest_closed_rates_by_granularity",
+    "collect_latest_closed_rates_for_accounts",
+    "collect_latest_rates",
+    "collect_latest_rates_for_accounts",
+    "collect_latest_rates_for_accounts_with_retries",
+    "copy_rates_from",
+    "copy_rates_from_pos",
+    "copy_rates_range",
+    "copy_ticks_from",
+    "copy_ticks_range",
     "detect_format",
+    "detect_position_side",
+    "determine_order_limits",
+    "drop_forming_rate_bar",
+    "ensure_utc",
     "export_dataframe",
+    "export_dataframe_to_sqlite",
+    "granularity_name",
+    "history_deals",
+    "history_orders",
+    "is_recoverable_mt5_error",
+    "last_error",
+    "latest_rates",
+    "load_rate_data",
+    "load_rate_data_from_connection",
+    "load_rate_series_by_granularity",
+    "load_rate_series_from_sqlite",
+    "market_book",
+    "minimum_margins",
+    "mt5_session",
+    "mt5_summary",
+    "mt5_summary_as_df",
+    "mt5_trading_session",
+    "mt5_version",
+    "normalize_dataframe",
+    "normalize_mt5_exception",
+    "normalize_symbol",
+    "normalize_symbols",
+    "normalize_time_columns",
+    "orders",
+    "parse_date_range",
+    "parse_datetime",
+    "parse_tick_flags",
+    "parse_timeframe",
+    "positions",
+    "recent_history_deals",
+    "recent_ticks",
+    "recent_window",
+    "resolve_account_spec",
+    "resolve_account_specs",
+    "resolve_history_datasets",
+    "resolve_history_tick_flags",
+    "resolve_history_timeframes",
+    "resolve_rate_tables",
+    "resolve_rate_view_name",
+    "resolve_rate_view_names",
+    "schema_columns",
+    "substitute_env_placeholders",
+    "symbol_info",
+    "symbol_info_tick",
+    "symbols",
+    "terminal_info",
+    "update_history",
+    "update_history_with_config",
+    "validate_schema",
 ]

mt5cli/cli.py

@@ -2,18 +2,29 @@
 
 from __future__ import annotations
 
-import json
 import logging
-import sqlite3
 from dataclasses import dataclass
-from datetime import UTC, datetime
-from enum import StrEnum
-from pathlib import Path
-from typing import TYPE_CHECKING, Annotated, Any, TypeGuard, cast
+from datetime import datetime  # noqa: TC003
+from pathlib import Path  # noqa: TC003
+from typing import TYPE_CHECKING, Annotated, Any, cast
 
-import click
 import typer
-from pdmt5 import Mt5Config, Mt5DataClient
+from pdmt5 import Mt5Config
+
+from . import sdk
+from .client import MT5Client
+from .utils import (
+    DATETIME_TYPE,
+    REQUEST_TYPE,
+    TICK_FLAGS_TYPE,
+    TIMEFRAME_TYPE,
+    Dataset,
+    IfExists,
+    LogLevel,
+    OutputFormat,
+    detect_format,
+    export_dataframe,
+)
 
 if TYPE_CHECKING:
     from collections.abc import Callable
@@ -22,197 +33,6 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
-# ---------------------------------------------------------------------------
-# Constants
-# ---------------------------------------------------------------------------
-
-TIMEFRAME_MAP: dict[str, int] = {
-    "M1": 1,
-    "M2": 2,
-    "M3": 3,
-    "M4": 4,
-    "M5": 5,
-    "M6": 6,
-    "M10": 10,
-    "M12": 12,
-    "M15": 15,
-    "M20": 20,
-    "M30": 30,
-    "H1": 16385,
-    "H2": 16386,
-    "H3": 16387,
-    "H4": 16388,
-    "H6": 16390,
-    "H8": 16392,
-    "H12": 16396,
-    "D1": 16408,
-    "W1": 32769,
-    "MN1": 49153,
-}
-
-TICK_FLAG_MAP: dict[str, int] = {
-    "ALL": 1,
-    "INFO": 2,
-    "TRADE": 4,
-}
-
-_FORMAT_EXTENSIONS: dict[str, str] = {
-    ".csv": "csv",
-    ".json": "json",
-    ".parquet": "parquet",
-    ".pq": "parquet",
-    ".db": "sqlite3",
-    ".sqlite": "sqlite3",
-    ".sqlite3": "sqlite3",
-}
-
-# ---------------------------------------------------------------------------
-# Enums
-# ---------------------------------------------------------------------------
-
-
-class OutputFormat(StrEnum):
-    """Supported output file formats."""
-
-    csv = "csv"
-    json = "json"
-    parquet = "parquet"
-    sqlite3 = "sqlite3"
-
-
-class LogLevel(StrEnum):
-    """Logging verbosity levels."""
-
-    DEBUG = "DEBUG"
-    INFO = "INFO"
-    WARNING = "WARNING"
-    ERROR = "ERROR"
-
-
-# ---------------------------------------------------------------------------
-# Click parameter types
-# ---------------------------------------------------------------------------
-
-
-class _DateTimeType(click.ParamType):
-    """Click parameter type for ISO 8601 datetime strings."""
-
-    name = "DATETIME"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None,
-        ctx: click.Context | None,
-    ) -> datetime:
-        """Convert a string value to a timezone-aware datetime.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Parsed datetime.
-        """
-        if isinstance(value, datetime):
-            return value
-        try:
-            return parse_datetime(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _TimeframeType(click.ParamType):
-    """Click parameter type for MT5 timeframe values."""
-
-    name = "TIMEFRAME"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None,
-        ctx: click.Context | None,
-    ) -> int:
-        """Convert a string or integer value to a timeframe integer.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Integer timeframe value.
-        """
-        if isinstance(value, int):
-            return value
-        try:
-            return parse_timeframe(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _TickFlagsType(click.ParamType):
-    """Click parameter type for MT5 tick copy flags."""
-
-    name = "FLAGS"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None,
-        ctx: click.Context | None,
-    ) -> int:
-        """Convert a string or integer value to a tick flags integer.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Integer tick flag value.
-        """
-        if isinstance(value, int):
-            return value
-        try:
-            return parse_tick_flags(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _RequestType(click.ParamType):
-    """Click parameter type for JSON order requests."""
-
-    name = "REQUEST"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None,
-        ctx: click.Context | None,
-    ) -> dict[str, Any]:
-        """Convert a raw CLI value to an order request dictionary.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Parsed request dictionary.
-        """
-        try:
-            return parse_request(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-DATETIME_TYPE = _DateTimeType()
-TIMEFRAME_TYPE = _TimeframeType()
-TICK_FLAGS_TYPE = _TickFlagsType()
-REQUEST_TYPE = _RequestType()
-
 # ---------------------------------------------------------------------------
 # Export context
 # ---------------------------------------------------------------------------
@@ -228,186 +48,6 @@ class _ExportContext:
     config: Mt5Config
 
 
-# ---------------------------------------------------------------------------
-# Public utility functions
-# ---------------------------------------------------------------------------
-
-
-def detect_format(
-    output_path: Path,
-    explicit_format: str | None = None,
-) -> str:
-    """Detect the output format from a file extension or explicit format string.
-
-    Args:
-        output_path: Path to the output file.
-        explicit_format: Explicitly specified format, if any.
-
-    Returns:
-        The detected format string.
-
-    Raises:
-        ValueError: If the format cannot be determined.
-    """
-    if explicit_format is not None:
-        return explicit_format
-    suffix = output_path.suffix.lower()
-    if suffix in _FORMAT_EXTENSIONS:
-        return _FORMAT_EXTENSIONS[suffix]
-    msg = (
-        f"Cannot detect format from extension '{suffix}'."
-        " Use --format to specify the output format."
-    )
-    raise ValueError(msg)
-
-
-def export_dataframe(
-    df: pd.DataFrame,
-    output_path: Path,
-    output_format: str,
-    table_name: str = "data",
-) -> None:
-    """Export a pandas DataFrame to the specified file format.
-
-    Args:
-        df: DataFrame to export.
-        output_path: Path to the output file.
-        output_format: Output format (csv, json, parquet, or sqlite3).
-        table_name: Table name for SQLite3 output.
-
-    Raises:
-        ValueError: If the output format is not supported.
-    """
-    if output_format == "csv":
-        df.to_csv(output_path, index=False)
-    elif output_format == "json":
-        df.to_json(
-            output_path,
-            orient="records",
-            date_format="iso",
-            indent=2,
-        )
-    elif output_format == "parquet":
-        df.to_parquet(output_path, index=False)
-    elif output_format == "sqlite3":
-        with sqlite3.connect(output_path) as conn:
-            df.to_sql(  # type: ignore[reportUnknownMemberType]
-                table_name,
-                conn,
-                if_exists="replace",
-                index=False,
-            )
-    else:
-        msg = f"Unsupported output format: {output_format}"
-        raise ValueError(msg)
-
-
-def parse_datetime(value: str) -> datetime:
-    """Parse an ISO 8601 datetime string to a timezone-aware datetime.
-
-    Args:
-        value: ISO 8601 datetime string (e.g., '2024-01-01' or
-            '2024-01-01T12:00:00+00:00').
-
-    Returns:
-        Parsed datetime with UTC timezone if no timezone is specified.
-
-    Raises:
-        ValueError: If the string cannot be parsed.
-    """
-    try:
-        dt = datetime.fromisoformat(value)
-    except ValueError:
-        msg = f"Invalid datetime format: '{value}'. Use ISO 8601 format."
-        raise ValueError(msg) from None
-    if dt.tzinfo is None:
-        dt = dt.replace(tzinfo=UTC)
-    return dt
-
-
-def parse_timeframe(value: str) -> int:
-    """Parse a timeframe string or integer value.
-
-    Args:
-        value: Timeframe name (e.g., 'M1', 'H1', 'D1') or integer value.
-
-    Returns:
-        Integer timeframe value.
-
-    Raises:
-        ValueError: If the timeframe is invalid.
-    """
-    upper = value.upper()
-    if upper in TIMEFRAME_MAP:
-        return TIMEFRAME_MAP[upper]
-    try:
-        return int(value)
-    except ValueError:
-        valid = ", ".join(TIMEFRAME_MAP)
-        msg = f"Invalid timeframe: '{value}'. Use one of: {valid}, or an integer."
-        raise ValueError(msg) from None
-
-
-def parse_tick_flags(value: str) -> int:
-    """Parse tick flags string or integer value.
-
-    Args:
-        value: Tick flag name (ALL, INFO, TRADE) or integer value.
-
-    Returns:
-        Integer tick flag value.
-
-    Raises:
-        ValueError: If the flag is invalid.
-    """
-    upper = value.upper()
-    if upper in TICK_FLAG_MAP:
-        return TICK_FLAG_MAP[upper]
-    try:
-        return int(value)
-    except ValueError:
-        valid = ", ".join(TICK_FLAG_MAP)
-        msg = f"Invalid tick flags: '{value}'. Use one of: {valid}, or an integer."
-        raise ValueError(msg) from None
-
-
-def _is_request_dict(value: object) -> TypeGuard[dict[str, Any]]:
-    return isinstance(value, dict)
-
-
-def parse_request(value: str) -> dict[str, Any]:
-    """Parse a JSON-formatted order request string or file reference.
-
-    Args:
-        value: JSON object string, or '@path' to read JSON from a file.
-
-    Returns:
-        Parsed request dictionary.
-
-    Raises:
-        ValueError: If the request file cannot be read or the value is not a
-            JSON object.
-    """
-    if value.startswith("@"):
-        path = Path(value[1:])
-        try:
-            text = path.read_text(encoding="utf-8")
-        except (OSError, UnicodeDecodeError) as exc:
-            msg = f"Failed to read JSON request file '{path}': {exc}"
-            raise ValueError(msg) from exc
-    else:
-        text = value
-    try:
-        parsed: object = json.loads(text)
-    except json.JSONDecodeError as exc:
-        msg = f"Invalid JSON request: {exc}"
-        raise ValueError(msg) from exc
-    if not _is_request_dict(parsed):
-        msg = "Order request must be a JSON object."
-        raise ValueError(msg)
-    return parsed
-
-
 # ---------------------------------------------------------------------------
 # Typer application
 # ---------------------------------------------------------------------------
@@ -428,34 +68,42 @@ def _get_export_context(ctx: typer.Context) -> _ExportContext:
 
 def _execute_export(
     ctx: typer.Context,
-    fetch_fn: Callable[[Mt5DataClient], pd.DataFrame],
+    fetch_fn: Callable[[], pd.DataFrame],
 ) -> None:
-    """Execute the common connect-fetch-export-shutdown workflow.
+    """Execute the common fetch-export workflow.
 
     Args:
         ctx: Typer context carrying shared options.
-        fetch_fn: Callable that receives a connected client and returns a
-            DataFrame.
+        fetch_fn: Callable that returns a DataFrame via the SDK layer.
     """
     export_ctx = _get_export_context(ctx)
-    client = Mt5DataClient(config=export_ctx.config)
-    client.initialize_and_login_mt5()
-    try:
-        df = fetch_fn(client)
-        export_dataframe(
-            df=df,
-            output_path=export_ctx.output,
-            output_format=export_ctx.output_format,
-            table_name=export_ctx.table,
-        )
-        logger.info(
-            "Exported %d rows to %s (%s)",
-            len(df),
-            export_ctx.output,
-            export_ctx.output_format,
-        )
-    finally:
-        client.shutdown()
+    df = fetch_fn()
+    export_dataframe(
+        df=df,
+        output_path=export_ctx.output,
+        output_format=export_ctx.output_format,
+        table_name=export_ctx.table,
+    )
+    logger.info(
+        "Exported %d rows to %s (%s)",
+        len(df),
+        export_ctx.output,
+        export_ctx.output_format,
+    )
+
+
+def _sdk_client(ctx: typer.Context) -> MT5Client:
+    export_ctx = _get_export_context(ctx)
+    return MT5Client(config=export_ctx.config)
+
+
+def _export_command(
+    ctx: typer.Context,
+    fetch_fn: Callable[[MT5Client], pd.DataFrame],
+) -> None:
+    """Create an SDK client, fetch a DataFrame, and export it."""
+    client = _sdk_client(ctx)
+    _execute_export(ctx, lambda: fetch_fn(client))
 
 
 @app.callback()
@@ -555,14 +203,9 @@ def rates_from(
     count: Annotated[int, typer.Option(help="Number of records.")],
 ) -> None:
     """Export rates from a start date."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.copy_rates_from_as_df(
-            symbol=symbol,
-            timeframe=timeframe,
-            date_from=date_from,
-            count=count,
-        ),
+        lambda client: client.copy_rates_from(symbol, timeframe, date_from, count),
     )
 
 
@@ -581,13 +224,42 @@ def rates_from_pos(
     count: Annotated[int, typer.Option(help="Number of records.")],
 ) -> None:
     """Export rates from a start position."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.copy_rates_from_pos_as_df(
-            symbol=symbol,
-            timeframe=timeframe,
+        lambda client: client.copy_rates_from_pos(
+            symbol,
+            timeframe,
+            start_pos,
+            count,
+        ),
+    )
+
+
+@app.command()
+def latest_rates(
+    ctx: typer.Context,
+    symbol: Annotated[str, typer.Option(help="Symbol name.")],
+    timeframe: Annotated[
+        int,
+        typer.Option(
+            click_type=TIMEFRAME_TYPE,
+            help="Timeframe.",
+        ),
+    ],
+    count: Annotated[int, typer.Option(help="Number of records.")],
+    start_pos: Annotated[
+        int,
+        typer.Option(help="Start position (0 = current bar)."),
+    ] = 0,
+) -> None:
+    """Export latest rates from a start position."""
+    _export_command(
+        ctx,
+        lambda client: client.latest_rates(
+            symbol,
+            timeframe,
+            count,
             start_pos=start_pos,
-            count=count,
         ),
     )
 
@@ -613,14 +285,9 @@ def rates_range(
     ],
 ) -> None:
     """Export rates for a date range."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.copy_rates_range_as_df(
-            symbol=symbol,
-            timeframe=timeframe,
-            date_from=date_from,
-            date_to=date_to,
-        ),
+        lambda client: client.copy_rates_range(symbol, timeframe, date_from, date_to),
     )
 
 
@@ -642,14 +309,9 @@ def ticks_from(
     ],
 ) -> None:
     """Export ticks from a start date."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.copy_ticks_from_as_df(
-            symbol=symbol,
-            date_from=date_from,
-            count=count,
-            flags=flags,
-        ),
+        lambda client: client.copy_ticks_from(symbol, date_from, count, flags),
     )
 
 
@@ -671,12 +333,44 @@ def ticks_range(
     ],
 ) -> None:
     """Export ticks for a date range."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.copy_ticks_range_as_df(
-            symbol=symbol,
-            date_from=date_from,
+        lambda client: client.copy_ticks_range(symbol, date_from, date_to, flags),
+    )
+
+
+@app.command()
+def ticks_recent(
+    ctx: typer.Context,
+    symbol: Annotated[str, typer.Option(help="Symbol name.")],
+    seconds: Annotated[
+        float,
+        typer.Option(help="Lookback window in seconds."),
+    ],
+    date_to: Annotated[
+        datetime | None,
+        typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
+    ] = None,
+    count: Annotated[
+        int,
+        typer.Option(help="Maximum number of ticks to return."),
+    ] = 10000,
+    flags: Annotated[
+        int,
+        typer.Option(
+            click_type=TICK_FLAGS_TYPE,
+            help="Tick flags (ALL, INFO, TRADE, or integer).",
+        ),
+    ] = "ALL",  # pyright: ignore[reportArgumentType]
+) -> None:
+    """Export ticks from a recent time window."""
+    _export_command(
+        ctx,
+        lambda client: client.recent_ticks(
+            symbol,
+            seconds,
             date_to=date_to,
+            count=count,
             flags=flags,
         ),
     )
@@ -685,13 +379,13 @@ def ticks_range(
 @app.command()
 def account_info(ctx: typer.Context) -> None:
     """Export account information."""
-    _execute_export(ctx, lambda c: c.account_info_as_df())
+    _export_command(ctx, lambda client: client.account_info())
 
 
 @app.command()
 def terminal_info(ctx: typer.Context) -> None:
     """Export terminal information."""
-    _execute_export(ctx, lambda c: c.terminal_info_as_df())
+    _export_command(ctx, lambda client: client.terminal_info())
 
 
 @app.command()
@@ -703,10 +397,7 @@ def symbols(
     ] = None,
 ) -> None:
     """Export symbol list."""
-    _execute_export(
-        ctx,
-        lambda c: c.symbols_get_as_df(group=group),
-    )
+    _export_command(ctx, lambda client: client.symbols(group=group))
 
 
 @app.command()
@@ -715,10 +406,16 @@ def symbol_info(
     symbol: Annotated[str, typer.Option(help="Symbol name.")],
 ) -> None:
     """Export symbol details."""
-    _execute_export(
-        ctx,
-        lambda c: c.symbol_info_as_df(symbol=symbol),
-    )
+    _export_command(ctx, lambda client: client.symbol_info(symbol))
+
+
+@app.command()
+def minimum_margins(
+    ctx: typer.Context,
+    symbol: Annotated[str, typer.Option(help="Symbol name.")],
+) -> None:
+    """Export minimum-volume buy and sell margin requirements."""
+    _export_command(ctx, lambda client: client.minimum_margins(symbol))
 
 
 @app.command()
@@ -729,13 +426,9 @@ def orders(
     ticket: Annotated[int | None, typer.Option(help="Ticket filter.")] = None,
 ) -> None:
     """Export active orders."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.orders_get_as_df(
-            symbol=symbol,
-            group=group,
-            ticket=ticket,
-        ),
+        lambda client: client.orders(symbol=symbol, group=group, ticket=ticket),
     )
 
 
@@ -747,13 +440,9 @@ def positions(
     ticket: Annotated[int | None, typer.Option(help="Ticket filter.")] = None,
 ) -> None:
     """Export open positions."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.positions_get_as_df(
-            symbol=symbol,
-            group=group,
-            ticket=ticket,
-        ),
+        lambda client: client.positions(symbol=symbol, group=group, ticket=ticket),
     )
 
 
@@ -774,9 +463,9 @@ def history_orders(
     position: Annotated[int | None, typer.Option(help="Position ticket.")] = None,
 ) -> None:
     """Export historical orders."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.history_orders_get_as_df(
+        lambda client: client.history_orders(
             date_from=date_from,
             date_to=date_to,
             group=group,
@@ -804,9 +493,9 @@ def history_deals(
     position: Annotated[int | None, typer.Option(help="Position ticket.")] = None,
 ) -> None:
     """Export historical deals."""
-    _execute_export(
+    _export_command(
         ctx,
-        lambda c: c.history_deals_get_as_df(
+        lambda client: client.history_deals(
             date_from=date_from,
             date_to=date_to,
             group=group,
@@ -817,16 +506,45 @@ def history_deals(
     )
 
 
+@app.command()
+def recent_history_deals(
+    ctx: typer.Context,
+    hours: Annotated[float, typer.Option(help="Lookback window in hours.")],
+    date_to: Annotated[
+        datetime | None,
+        typer.Option(click_type=DATETIME_TYPE, help="Window end date."),
+    ] = None,
+    group: Annotated[str | None, typer.Option(help="Group filter.")] = None,
+    symbol: Annotated[str | None, typer.Option(help="Symbol filter.")] = None,
+) -> None:
+    """Export historical deals from a recent trailing window."""
+    _export_command(
+        ctx,
+        lambda client: client.recent_history_deals(
+            hours,
+            date_to=date_to,
+            group=group,
+            symbol=symbol,
+        ),
+    )
+
+
+@app.command()
+def mt5_summary(ctx: typer.Context) -> None:
+    """Export a compact terminal/account status summary."""
+    _export_command(ctx, lambda client: client.mt5_summary_as_df())
+
+
 @app.command()
 def version(ctx: typer.Context) -> None:
     """Export MetaTrader5 version information."""
-    _execute_export(ctx, lambda c: c.version_as_df())
+    _export_command(ctx, lambda client: client.version())
 
 
 @app.command()
 def last_error(ctx: typer.Context) -> None:
     """Export the last error information."""
-    _execute_export(ctx, lambda c: c.last_error_as_df())
+    _export_command(ctx, lambda client: client.last_error())
 
 
 @app.command()
@@ -835,10 +553,7 @@ def symbol_info_tick(
     symbol: Annotated[str, typer.Option(help="Symbol name.")],
 ) -> None:
     """Export the last tick for a symbol."""
-    _execute_export(
-        ctx,
-        lambda c: c.symbol_info_tick_as_df(symbol=symbol),
-    )
+    _export_command(ctx, lambda client: client.symbol_info_tick(symbol))
 
 
 @app.command()
@@ -847,10 +562,7 @@ def market_book(
     symbol: Annotated[str, typer.Option(help="Symbol name.")],
 ) -> None:
     """Export market depth (order book) for a symbol."""
-    _execute_export(
-        ctx,
-        lambda c: c.market_book_get_as_df(symbol=symbol),
-    )
+    _export_command(ctx, lambda client: client.market_book(symbol))
 
 
 @app.command()
@@ -862,10 +574,7 @@ def order_check(
     ],
 ) -> None:
     """Check funds sufficiency for a trading operation."""
-    _execute_export(
-        ctx,
-        lambda c: c.order_check_as_df(request=request),
-    )
+    _export_command(ctx, lambda client: client.order_check(request))
 
 
 @app.command()
@@ -888,9 +597,103 @@ def order_send(
     if not yes:
         msg = "Pass --yes to send a live trade request."
         raise typer.BadParameter(msg, param_hint="--yes")
-    _execute_export(
-        ctx,
-        lambda c: c.order_send_as_df(request=request),
+    _export_command(ctx, lambda client: client.order_send(request))
+
+
+@app.command()
+def collect_history(
+    ctx: typer.Context,
+    symbol: Annotated[
+        list[str],
+        typer.Option(
+            "--symbol",
+            "-s",
+            help="Symbol to collect (repeat for multiple symbols).",
+        ),
+    ],
+    date_from: Annotated[
+        datetime,
+        typer.Option(click_type=DATETIME_TYPE, help="Start date."),
+    ],
+    date_to: Annotated[
+        datetime,
+        typer.Option(click_type=DATETIME_TYPE, help="End date."),
+    ],
+    dataset: Annotated[
+        list[Dataset] | None,
+        typer.Option(
+            "--dataset",
+            help=(
+                "Dataset to include (repeat for multiple)."
+                " Defaults to all: rates, ticks, history-orders, history-deals."
+            ),
+        ),
+    ] = None,
+    timeframe: Annotated[
+        int,
+        typer.Option(
+            click_type=TIMEFRAME_TYPE,
+            help="Rates timeframe (e.g., M1, H1, D1).",
+        ),
+    ] = 1,
+    flags: Annotated[
+        int,
+        typer.Option(
+            click_type=TICK_FLAGS_TYPE,
+            help="Tick copy flags (ALL, INFO, TRADE, or integer).",
+        ),
+    ] = "ALL",  # pyright: ignore[reportArgumentType]
+    if_exists: Annotated[
+        IfExists,
+        typer.Option(
+            "--if-exists",
+            help="Behavior when a target table already exists.",
+        ),
+    ] = IfExists.FAIL,
+    with_views: Annotated[
+        bool,
+        typer.Option(
+            "--with-views",
+            help=(
+                "Add cash_events and positions_reconstructed SQLite views"
+                " derived from history_deals."
+            ),
+        ),
+    ] = False,
+) -> None:
+    """Collect historical datasets into a single SQLite database.
+
+    Tables written depend on ``--dataset``: ``rates``, ``ticks``,
+    ``history_orders``, ``history_deals``. History datasets are fetched per
+    symbol and concatenated. Rates rows carry the requested ``timeframe`` so
+    appended runs at different timeframes remain distinguishable.
+
+    With ``--with-views`` (requires the ``history-deals`` dataset), optional
+    views ``cash_events`` and ``positions_reconstructed`` are derived from
+    ``history_deals`` when the required columns are present.
+
+    Raises:
+        typer.BadParameter: If the output format is not SQLite3.
+    """
+    export_ctx = _get_export_context(ctx)
+    if export_ctx.output_format != "sqlite3":
+        msg = (
+            "collect-history requires SQLite3 output."
+            " Use a .db/.sqlite/.sqlite3 extension or --format sqlite3."
+        )
+        raise typer.BadParameter(msg)
+    datasets = set(dataset) if dataset else set(Dataset)
+    sdk.collect_history(
+        output=export_ctx.output,
+        symbols=symbol,
+        date_from=date_from,
+        date_to=date_to,
+        datasets=datasets,
+        timeframe=timeframe,
+        flags=flags,
+        if_exists=if_exists,
+        with_views=with_views,
+        config=export_ctx.config,
     )
 
 

mt5cli/client.py

@@ -0,0 +1,88 @@
+"""Stable public client abstraction for MT5 data and execution operations."""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from typing import TYPE_CHECKING, Any, Self
+
+from .sdk import Mt5CliClient, build_config, connected_client
+
+if TYPE_CHECKING:
+    from collections.abc import Iterator
+
+    import pandas as pd
+    from pdmt5 import Mt5Config, Mt5DataClient
+
+__all__ = [
+    "MT5Client",
+    "build_config",
+    "mt5_session",
+]
+
+
+class MT5Client(Mt5CliClient):
+    """Public client for generic MT5 data access and order primitives.
+
+    Extends the read-only SDK client with optional order check/send helpers and
+    exposes the same connection lifecycle as :class:`~mt5cli.sdk.Mt5CliClient`.
+    Downstream applications such as private trading packages should prefer this
+    type over the legacy ``Mt5CliClient`` name.
+
+    mt5cli intentionally exposes minimal execution primitives only. Trading
+    decisions, signals, strategies, backtests, and optimization remain the
+    responsibility of downstream applications.
+    """
+
+    def order_check(self, request: dict[str, Any]) -> pd.DataFrame:
+        """Check funds sufficiency for a trade request.
+
+        Args:
+            request: MT5 order request dictionary.
+
+        Returns:
+            One-row DataFrame with the order-check result.
+        """
+        return self._fetch(lambda client: client.order_check_as_df(request=request))
+
+    def order_send(self, request: dict[str, Any]) -> pd.DataFrame:
+        """Send a live trade request to the MT5 trade server.
+
+        Warning:
+            This is a live execution primitive. A successful call can place,
+            modify, or close real trades on the connected account. Downstream
+            applications must gate usage explicitly (for example behind manual
+            confirmation or application-specific risk controls). mt5cli does
+            not implement strategy logic, signal generation, or trade sizing.
+
+        Args:
+            request: MT5 order request dictionary.
+
+        Returns:
+            One-row DataFrame with the order-send result.
+        """
+        return self._fetch(lambda client: client.order_send_as_df(request=request))
+
+    @classmethod
+    def from_connected_client(cls, client: Mt5DataClient) -> Self:
+        """Bind to an already-connected ``Mt5DataClient`` without owning it.
+
+        Returns:
+            Client wrapper bound to the injected connection.
+        """
+        return cls(client=client)
+
+
+@contextmanager
+def mt5_session(config: Mt5Config | None = None) -> Iterator[MT5Client]:
+    """Open an MT5 terminal session and yield a connected :class:`MT5Client`.
+
+    Args:
+        config: MT5 connection configuration. Defaults to an empty config that
+            attaches to a running terminal.
+
+    Yields:
+        Connected :class:`MT5Client` bound to the session.
+    """
+    mt5_config = config or build_config()
+    with connected_client(mt5_config) as client:
+        yield MT5Client.from_connected_client(client)

mt5cli/converters.py

@@ -0,0 +1,162 @@
+"""Shared conversion helpers for MT5 symbols, timeframes, and date ranges."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+from typing import TYPE_CHECKING
+
+from pdmt5 import get_timeframe_name as _get_timeframe_name
+
+from .utils import parse_datetime, parse_tick_flags, parse_timeframe
+
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
+__all__ = [
+    "ensure_utc",
+    "granularity_name",
+    "normalize_symbol",
+    "normalize_symbols",
+    "parse_date_range",
+    "parse_datetime",
+    "parse_tick_flags",
+    "parse_timeframe",
+    "recent_window",
+]
+
+
+def normalize_symbol(symbol: str) -> str:
+    """Normalize a broker symbol name for MT5 API calls.
+
+    Strips surrounding whitespace while preserving broker-specific casing and
+    suffixes (for example ``XAUUSDm``, ``US500.cash``, or ``EURUSD.r``).
+
+    Args:
+        symbol: Raw symbol name.
+
+    Returns:
+        Normalized symbol string.
+
+    Raises:
+        ValueError: If the symbol is empty after normalization.
+    """
+    normalized = symbol.strip()
+    if not normalized:
+        msg = "Symbol must not be empty."
+        raise ValueError(msg)
+    return normalized
+
+
+def normalize_symbols(symbols: Sequence[str]) -> list[str]:
+    """Normalize a sequence of broker symbol names.
+
+    Args:
+        symbols: Raw symbol names.
+
+    Returns:
+        List of normalized, de-duplicated symbols preserving first-seen order.
+    """
+    seen: set[str] = set()
+    resolved: list[str] = []
+    for symbol in symbols:
+        normalized = normalize_symbol(symbol)
+        if normalized not in seen:
+            seen.add(normalized)
+            resolved.append(normalized)
+    return resolved
+
+
+def ensure_utc(value: datetime | str) -> datetime:
+    """Return a timezone-aware UTC datetime.
+
+    Args:
+        value: Datetime instance or ISO 8601 string.
+
+    Returns:
+        UTC-aware datetime.
+    """
+    if isinstance(value, str):
+        return parse_datetime(value)
+    if value.tzinfo is None:
+        return value.replace(tzinfo=UTC)
+    return value.astimezone(UTC)
+
+
+def parse_date_range(
+    date_from: datetime | str,
+    date_to: datetime | str,
+) -> tuple[datetime, datetime]:
+    """Parse and validate an inclusive UTC date range.
+
+    Args:
+        date_from: Range start as datetime or ISO 8601 string.
+        date_to: Range end as datetime or ISO 8601 string.
+
+    Returns:
+        Tuple of UTC-aware ``(start, end)`` datetimes.
+
+    Raises:
+        ValueError: If ``date_from`` is after ``date_to``.
+    """
+    start = ensure_utc(date_from)
+    end = ensure_utc(date_to)
+    if start > end:
+        msg = (
+            f"date_from ({start.isoformat()}) must not be after "
+            f"date_to ({end.isoformat()})."
+        )
+        raise ValueError(msg)
+    return start, end
+
+
+def recent_window(
+    *,
+    hours: float | None = None,
+    seconds: float | None = None,
+    date_to: datetime | str | None = None,
+) -> tuple[datetime, datetime]:
+    """Build a trailing UTC window ending at ``date_to`` or now.
+
+    Exactly one of ``hours`` or ``seconds`` must be provided.
+
+    Args:
+        hours: Trailing window length in hours.
+        seconds: Trailing window length in seconds.
+        date_to: Window end. Defaults to current UTC time.
+
+    Returns:
+        Tuple of UTC-aware ``(start, end)`` datetimes.
+
+    Raises:
+        ValueError: If neither or both window lengths are provided, or if a
+            length is not positive.
+    """
+    if (hours is None) == (seconds is None):
+        msg = "Provide exactly one of hours or seconds."
+        raise ValueError(msg)
+    if hours is not None:
+        length = timedelta(hours=hours)
+    else:
+        length = timedelta(seconds=seconds if seconds is not None else 0)
+    if length.total_seconds() <= 0:
+        msg = "Window length must be positive."
+        raise ValueError(msg)
+    end = ensure_utc(date_to) if date_to is not None else datetime.now(UTC)
+    return end - length, end
+
+
+def granularity_name(timeframe: int | str) -> str:
+    """Return a short granularity label for a timeframe integer or name.
+
+    Args:
+        timeframe: MT5 timeframe as integer or name (for example ``M1``).
+
+    Returns:
+        Short name such as ``M1`` or the stringified integer when unknown.
+    """
+    tf = parse_timeframe(timeframe)
+    try:
+        name = _get_timeframe_name(tf)
+    except ValueError:
+        return str(tf)
+    return name.removeprefix("TIMEFRAME_")

mt5cli/exceptions.py

@@ -0,0 +1,90 @@
+"""Normalized exception types for MT5 and mt5cli operations."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, TypeVar
+
+from pdmt5 import Mt5RuntimeError, Mt5TradingError
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+T = TypeVar("T")
+
+__all__ = [
+    "Mt5CliError",
+    "Mt5ConnectionError",
+    "Mt5OperationError",
+    "Mt5SchemaError",
+    "call_with_normalized_errors",
+    "is_recoverable_mt5_error",
+    "normalize_mt5_exception",
+]
+
+_RECOVERABLE_MT5_ERRORS: tuple[type[BaseException], ...] = (
+    Mt5TradingError,
+    Mt5RuntimeError,
+)
+
+
+class Mt5CliError(Exception):
+    """Base exception for mt5cli public API errors."""
+
+
+class Mt5ConnectionError(Mt5CliError):
+    """Raised when MT5 initialization, login, or shutdown fails."""
+
+
+class Mt5OperationError(Mt5CliError):
+    """Raised when an MT5 data or trading operation fails."""
+
+
+class Mt5SchemaError(Mt5CliError):
+    """Raised when a DataFrame does not match an expected dataset schema."""
+
+
+def is_recoverable_mt5_error(exc: BaseException) -> bool:
+    """Return whether an exception is a transient MT5 failure worth retrying.
+
+    Args:
+        exc: Exception raised by MT5 or pdmt5.
+
+    Returns:
+        True for ``Mt5RuntimeError`` and ``Mt5TradingError``.
+    """
+    return isinstance(exc, _RECOVERABLE_MT5_ERRORS)
+
+
+def normalize_mt5_exception(exc: BaseException) -> Mt5CliError:
+    """Map pdmt5/MT5 exceptions to stable mt5cli exception types.
+
+    Args:
+        exc: Original exception from MT5 or pdmt5.
+
+    Returns:
+        ``Mt5ConnectionError`` for runtime failures, ``Mt5OperationError`` for
+        trading failures, or the original exception when it is not recognized.
+    """
+    if isinstance(exc, Mt5TradingError):
+        return Mt5OperationError(str(exc))
+    if isinstance(exc, Mt5RuntimeError):
+        return Mt5ConnectionError(str(exc))
+    if isinstance(exc, Mt5CliError):
+        return exc
+    return Mt5CliError(str(exc))
+
+
+def call_with_normalized_errors(fn: Callable[[], T]) -> T:
+    """Run ``fn`` and map recoverable MT5 errors to mt5cli types.
+
+    Args:
+        fn: Callable performing MT5 work.
+
+    Returns:
+        Value returned by ``fn``.
+    """
+    try:
+        return fn()
+    except _RECOVERABLE_MT5_ERRORS as exc:
+        normalized = normalize_mt5_exception(exc)
+        raise normalized from exc

mt5cli/retry.py

@@ -0,0 +1,64 @@
+"""Retry and reconnect helpers for transient MT5 failures."""
+
+from __future__ import annotations
+
+import logging
+import time
+from typing import TYPE_CHECKING, TypeVar
+
+from .exceptions import is_recoverable_mt5_error
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+T = TypeVar("T")
+
+logger = logging.getLogger(__name__)
+
+__all__ = [
+    "retry_with_backoff",
+]
+
+
+def retry_with_backoff(
+    fn: Callable[[], T],
+    *,
+    retry_count: int = 0,
+    backoff_base: float = 2.0,
+    operation: str = "MT5 operation",
+) -> T:
+    """Call ``fn`` with bounded exponential backoff on recoverable MT5 errors.
+
+    Only ``pdmt5.Mt5RuntimeError`` and ``pdmt5.Mt5TradingError`` are retried.
+    Other exceptions propagate immediately. The final failure is re-raised once
+    retries are exhausted.
+
+    Args:
+        fn: Callable performing MT5 work.
+        retry_count: Maximum number of retries after the first attempt. ``0``
+            disables retries.
+        backoff_base: Base for exponential backoff. The delay before retry
+            attempt ``n`` (1-indexed) is ``backoff_base ** n`` seconds.
+        operation: Label used in warning logs.
+
+    Returns:
+        Value returned by ``fn`` on success.
+    """
+    attempts = max(retry_count, 0) + 1
+    for attempt in range(attempts - 1):
+        try:
+            return fn()
+        except Exception as exc:
+            if not is_recoverable_mt5_error(exc):
+                raise
+            delay = backoff_base ** (attempt + 1)
+            logger.warning(
+                "%s failed (attempt %d/%d): %s; retrying in %.1fs",
+                operation,
+                attempt + 1,
+                attempts,
+                exc,
+                delay,
+            )
+            time.sleep(delay)
+    return fn()

mt5cli/schemas.py

@@ -0,0 +1,291 @@
+"""Canonical DataFrame schemas for MT5 market and account datasets."""
+
+from __future__ import annotations
+
+from enum import StrEnum
+from typing import TYPE_CHECKING, Final
+
+import pandas as pd
+
+from .converters import normalize_symbol, parse_timeframe
+from .exceptions import Mt5SchemaError
+
+if TYPE_CHECKING:
+    from collections.abc import Iterable
+
+__all__ = [
+    "DEDUP_KEYS",
+    "KNOWN_MT5_TIME_COLUMNS",
+    "REQUIRED_COLUMNS",
+    "TIME_COLUMNS",
+    "DataKind",
+    "normalize_dataframe",
+    "normalize_time_columns",
+    "schema_columns",
+    "validate_schema",
+]
+
+KNOWN_MT5_TIME_COLUMNS: Final[frozenset[str]] = frozenset({
+    "time",
+    "time_setup",
+    "time_setup_msc",
+    "time_done",
+    "time_done_msc",
+    "time_msc",
+})
+
+_TIME_COLUMN_NAMES = KNOWN_MT5_TIME_COLUMNS
+
+
+class DataKind(StrEnum):
+    """Supported MT5 dataset kinds with canonical column contracts."""
+
+    rates = "rates"
+    ticks = "ticks"
+    orders = "orders"
+    positions = "positions"
+    history_orders = "history_orders"
+    history_deals = "history_deals"
+
+
+REQUIRED_COLUMNS: dict[DataKind, frozenset[str]] = {
+    DataKind.rates: frozenset({
+        "time",
+        "open",
+        "high",
+        "low",
+        "close",
+        "tick_volume",
+        "spread",
+        "real_volume",
+    }),
+    DataKind.ticks: frozenset({
+        "time",
+        "bid",
+        "ask",
+        "last",
+        "volume",
+        "time_msc",
+        "flags",
+        "volume_real",
+    }),
+    DataKind.orders: frozenset({
+        "ticket",
+        "time_setup",
+        "type",
+        "state",
+        "symbol",
+        "volume_current",
+        "price_open",
+    }),
+    DataKind.positions: frozenset({
+        "ticket",
+        "time",
+        "type",
+        "symbol",
+        "volume",
+        "price_open",
+        "price_current",
+        "profit",
+    }),
+    DataKind.history_orders: frozenset({
+        "ticket",
+        "time_setup",
+        "type",
+        "state",
+        "symbol",
+        "volume_initial",
+        "price_open",
+    }),
+    DataKind.history_deals: frozenset({
+        "ticket",
+        "order",
+        "time",
+        "type",
+        "entry",
+        "symbol",
+        "volume",
+        "price",
+        "profit",
+    }),
+}
+
+_OPTIONAL_TIME_COLUMNS_BY_KIND: dict[DataKind, frozenset[str]] = {
+    DataKind.orders: frozenset({
+        "time_setup_msc",
+        "time_done",
+        "time_done_msc",
+    }),
+    DataKind.history_orders: frozenset({
+        "time_setup_msc",
+        "time_done",
+        "time_done_msc",
+    }),
+    DataKind.positions: frozenset({"time_msc"}),
+}
+
+TIME_COLUMNS: dict[DataKind, frozenset[str]] = {
+    kind: (REQUIRED_COLUMNS[kind] & _TIME_COLUMN_NAMES)
+    | _OPTIONAL_TIME_COLUMNS_BY_KIND.get(kind, frozenset())
+    for kind in DataKind
+}
+
+DEDUP_KEYS: dict[DataKind, tuple[tuple[str, ...], ...]] = {
+    DataKind.rates: (("symbol", "timeframe", "time"), ("symbol", "time")),
+    DataKind.ticks: (("symbol", "time_msc"), ("symbol", "time")),
+    DataKind.history_orders: (("ticket",), ("symbol", "time", "type")),
+    DataKind.history_deals: (("ticket",), ("symbol", "time", "type", "entry")),
+}
+
+
+def schema_columns(kind: DataKind) -> frozenset[str]:
+    """Return required column names for a dataset kind.
+
+    Args:
+        kind: Dataset kind.
+
+    Returns:
+        Required column names for ``kind``.
+    """
+    return REQUIRED_COLUMNS[kind]
+
+
+def validate_schema(
+    frame: pd.DataFrame,
+    kind: DataKind,
+    *,
+    extra_required: Iterable[str] | None = None,
+) -> None:
+    """Validate that a DataFrame includes required columns for a dataset kind.
+
+    Args:
+        frame: DataFrame to validate.
+        kind: Expected dataset kind.
+        extra_required: Additional columns that must be present (for example
+            ``symbol`` and ``timeframe`` on stored rate history).
+
+    Raises:
+        Mt5SchemaError: If required columns are missing.
+    """
+    if frame.empty and len(frame.columns) == 0:
+        return
+    required = set(REQUIRED_COLUMNS[kind])
+    if extra_required is not None:
+        required.update(extra_required)
+    missing = required - set(frame.columns)
+    if missing:
+        msg = (
+            f"{kind.value} schema is missing required columns: "
+            f"{', '.join(sorted(missing))}."
+        )
+        raise Mt5SchemaError(msg)
+
+
+def _coerce_mt5_time_column(series: pd.Series, column: str) -> pd.Series:
+    """Coerce one MT5 time column to UTC-aware datetimes.
+
+    Returns:
+        Series with UTC-aware datetime values.
+    """
+    if pd.api.types.is_datetime64_any_dtype(series):
+        return pd.to_datetime(series, utc=True, errors="coerce")
+    if pd.api.types.is_numeric_dtype(series):
+        unit = "ms" if column.endswith("_msc") else "s"
+        return pd.to_datetime(series, unit=unit, utc=True, errors="coerce")
+    return pd.to_datetime(series, utc=True, errors="coerce")
+
+
+def normalize_time_columns(frame: pd.DataFrame, kind: DataKind) -> pd.DataFrame:
+    """Coerce dataset time columns to UTC-aware datetimes when present.
+
+    Any column in :data:`KNOWN_MT5_TIME_COLUMNS` that is present in ``frame``
+    is normalized. Numeric MT5 epoch values use seconds for ``time``,
+    ``time_setup``, and ``time_done``, and milliseconds for ``*_msc`` columns.
+
+    Args:
+        frame: Source DataFrame from MT5 or pdmt5.
+        kind: Dataset kind (retained for API compatibility).
+
+    Returns:
+        DataFrame copy with normalized time columns.
+    """
+    del kind
+    normalized = frame.copy()
+    for column in normalized.columns:
+        if column not in _TIME_COLUMN_NAMES:
+            continue
+        normalized[column] = _coerce_mt5_time_column(normalized[column], column)
+    return normalized
+
+
+def normalize_dataframe(
+    frame: pd.DataFrame,
+    kind: DataKind,
+    *,
+    symbol: str | None = None,
+    timeframe: int | str | None = None,
+    sort: bool = True,
+) -> pd.DataFrame:
+    """Normalize MT5 DataFrame columns, timestamps, and storage metadata.
+
+    Ensures UTC timestamps, optionally injects ``symbol`` / ``timeframe`` for
+    storage-oriented datasets, and sorts chronologically when a ``time`` column
+    exists.
+
+    Args:
+        frame: Source DataFrame from MT5 or pdmt5.
+        kind: Dataset kind guiding normalization rules.
+        symbol: Optional symbol to inject when missing.
+        timeframe: Optional timeframe integer or name to inject for rates.
+        sort: Whether to sort by ``time`` or ``time_msc`` when present.
+
+    Returns:
+        Normalized DataFrame copy.
+    """
+    if frame.empty and len(frame.columns) == 0:
+        return frame.copy()
+
+    normalized = normalize_time_columns(frame, kind)
+
+    if symbol is not None and "symbol" not in normalized.columns:
+        normalized.insert(0, "symbol", normalize_symbol(symbol))
+
+    if timeframe is not None and kind is DataKind.rates:
+        tf = parse_timeframe(timeframe)
+        if "timeframe" not in normalized.columns:
+            insert_at = 1 if "symbol" in normalized.columns else 0
+            normalized.insert(insert_at, "timeframe", tf)
+
+    validate_schema(normalized, kind)
+
+    if sort:
+        if "time" in normalized.columns:
+            normalized = normalized.sort_values("time", kind="stable")
+        elif "time_msc" in normalized.columns:
+            normalized = normalized.sort_values("time_msc", kind="stable")
+        normalized = normalized.reset_index(drop=True)
+
+    return normalized
+
+
+def ensure_utc_columns(frame: pd.DataFrame, columns: Iterable[str]) -> pd.DataFrame:
+    """Return a copy with selected columns coerced to UTC datetimes.
+
+    Args:
+        frame: Source DataFrame.
+        columns: Column names to coerce.
+
+    Returns:
+        DataFrame copy with UTC-aware datetime columns.
+    """
+    normalized = frame.copy()
+    for column in columns:
+        if column not in normalized.columns:
+            continue
+        if column in _TIME_COLUMN_NAMES:
+            normalized[column] = _coerce_mt5_time_column(normalized[column], column)
+        else:
+            normalized[column] = pd.to_datetime(
+                normalized[column], utc=True, errors="coerce"
+            )
+    return normalized

mt5cli/storage.py

@@ -0,0 +1,49 @@
+"""Generic storage helpers for MT5 market and account history."""
+
+from __future__ import annotations
+
+from .history import (
+    RateTarget,
+    build_rate_targets,
+    build_rate_view_name,
+    drop_forming_rate_bar,
+    load_rate_data,
+    load_rate_data_from_connection,
+    load_rate_series_by_granularity,
+    load_rate_series_from_sqlite,
+    resolve_rate_tables,
+    resolve_rate_view_name,
+    resolve_rate_view_names,
+)
+from .sdk import collect_history, update_history, update_history_with_config
+from .utils import (
+    Dataset,
+    IfExists,
+    OutputFormat,
+    detect_format,
+    export_dataframe,
+    export_dataframe_to_sqlite,
+)
+
+__all__ = [
+    "Dataset",
+    "IfExists",
+    "OutputFormat",
+    "RateTarget",
+    "build_rate_targets",
+    "build_rate_view_name",
+    "collect_history",
+    "detect_format",
+    "drop_forming_rate_bar",
+    "export_dataframe",
+    "export_dataframe_to_sqlite",
+    "load_rate_data",
+    "load_rate_data_from_connection",
+    "load_rate_series_by_granularity",
+    "load_rate_series_from_sqlite",
+    "resolve_rate_tables",
+    "resolve_rate_view_name",
+    "resolve_rate_view_names",
+    "update_history",
+    "update_history_with_config",
+]

mt5cli/trading.py

@@ -0,0 +1,210 @@
+"""Trading-capable MetaTrader 5 session helpers and operational utilities."""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from typing import TYPE_CHECKING, Literal
+
+from pdmt5 import Mt5Config, Mt5TradingClient
+
+from .sdk import build_config
+
+if TYPE_CHECKING:
+    from collections.abc import Iterator
+
+    import pandas as pd
+
+PositionSide = Literal["long", "short"]
+OrderSide = Literal["long", "short"]
+
+__all__ = [
+    "OrderSide",
+    "PositionSide",
+    "calculate_margin_and_volume",
+    "detect_position_side",
+    "determine_order_limits",
+    "mt5_trading_session",
+]
+
+
+def _require_unit_ratio(value: float, name: str) -> None:
+    if not 0.0 <= value <= 1.0:
+        msg = f"{name} must be between 0 and 1 inclusive."
+        raise ValueError(msg)
+
+
+def _require_protective_ratio(value: float, name: str) -> None:
+    if not 0.0 <= value < 1.0:
+        msg = f"{name} must be at least 0 and less than 1."
+        raise ValueError(msg)
+
+
+def _sum_position_volume(positions: pd.DataFrame, position_type: object) -> float:
+    matched = positions.loc[positions["type"] == position_type, "volume"]
+    if matched.empty:
+        return 0.0
+    return float(matched.to_numpy(dtype=float).sum())
+
+
+def _normalize_order_side(side: str) -> OrderSide:
+    normalized = side.lower()
+    if normalized in {"long", "buy"}:
+        return "long"
+    if normalized in {"short", "sell"}:
+        return "short"
+    msg = (
+        f"Unsupported order side: {side!r}. Expected 'long', 'short', 'buy', or 'sell'."
+    )
+    raise ValueError(msg)
+
+
+def detect_position_side(
+    client: Mt5TradingClient,
+    symbol: str,
+) -> PositionSide | None:
+    """Detect the net open position side for a symbol.
+
+    Args:
+        client: Connected ``Mt5TradingClient`` instance.
+        symbol: Symbol to inspect.
+
+    Returns:
+        ``"long"`` when net buy volume exceeds sell volume, ``"short"`` when
+        net sell volume exceeds buy volume, or ``None`` when no positions exist
+        or buy/sell volumes are exactly balanced.
+    """
+    positions = client.positions_get_as_df(symbol=symbol)
+    if positions.empty:
+        return None
+
+    buy_type = client.mt5.POSITION_TYPE_BUY
+    sell_type = client.mt5.POSITION_TYPE_SELL
+    buy_volume = _sum_position_volume(positions, buy_type)
+    sell_volume = _sum_position_volume(positions, sell_type)
+    net_volume = buy_volume - sell_volume
+    if net_volume > 0:
+        return "long"
+    if net_volume < 0:
+        return "short"
+    return None
+
+
+def calculate_margin_and_volume(
+    client: Mt5TradingClient,
+    symbol: str,
+    unit_margin_ratio: float,
+    preserved_margin_ratio: float,
+) -> dict[str, float]:
+    """Calculate tradable margin and volumes from account free margin.
+
+    Applies ``preserved_margin_ratio`` to keep a reserve off ``margin_free``,
+    then allocates ``unit_margin_ratio`` of the remainder as the margin budget
+    for volume sizing on both buy and sell sides.
+
+    Args:
+        client: Connected ``Mt5TradingClient`` instance.
+        symbol: Symbol used for minimum-lot margin and volume calculations.
+        unit_margin_ratio: Fraction of post-reserve margin to allocate per unit.
+        preserved_margin_ratio: Fraction of ``margin_free`` to preserve.
+
+    Returns:
+        Dictionary with ``margin_free``, ``available_margin``, ``trade_margin``,
+        ``buy_volume``, and ``sell_volume``. Negative ``margin_free`` values are
+        clamped to ``0.0`` before sizing.
+    """
+    _require_unit_ratio(unit_margin_ratio, "unit_margin_ratio")
+    _require_unit_ratio(preserved_margin_ratio, "preserved_margin_ratio")
+
+    account = client.account_info_as_dict()
+    margin_free = max(0.0, float(account.get("margin_free") or 0.0))
+    available_margin = margin_free * (1.0 - preserved_margin_ratio)
+    trade_margin = available_margin * unit_margin_ratio
+    buy_volume = client.calculate_volume_by_margin(symbol, trade_margin, "BUY")
+    sell_volume = client.calculate_volume_by_margin(symbol, trade_margin, "SELL")
+    return {
+        "margin_free": margin_free,
+        "available_margin": available_margin,
+        "trade_margin": trade_margin,
+        "buy_volume": buy_volume,
+        "sell_volume": sell_volume,
+    }
+
+
+def determine_order_limits(
+    client: Mt5TradingClient,
+    symbol: str,
+    side: OrderSide | str,
+    stop_loss_limit_ratio: float,
+    take_profit_limit_ratio: float,
+) -> dict[str, float | None]:
+    """Derive entry and protective order prices from current market quotes.
+
+    Args:
+        client: Connected ``Mt5TradingClient`` instance.
+        symbol: Symbol used for the quote lookup.
+        side: Position side as ``"long"``/``"short"`` (``"buy"``/``"sell"``
+            aliases are accepted).
+        stop_loss_limit_ratio: Relative distance from entry for stop loss in
+            ``[0, 1)``. A value of ``0`` omits the stop loss.
+        take_profit_limit_ratio: Relative distance from entry for take profit in
+            ``[0, 1)``. A value of ``0`` omits the take profit.
+
+    Returns:
+        Dictionary with ``entry``, ``stop_loss``, and ``take_profit`` keys.
+        Omitted protective levels are returned as ``None``.
+    """
+    _require_protective_ratio(stop_loss_limit_ratio, "stop_loss_limit_ratio")
+    _require_protective_ratio(take_profit_limit_ratio, "take_profit_limit_ratio")
+    normalized_side = _normalize_order_side(side)
+    tick = client.symbol_info_tick_as_dict(symbol=symbol)
+    entry = float(tick["ask"] if normalized_side == "long" else tick["bid"])
+
+    stop_loss: float | None = None
+    if stop_loss_limit_ratio > 0:
+        if normalized_side == "long":
+            stop_loss = entry * (1.0 - stop_loss_limit_ratio)
+        else:
+            stop_loss = entry * (1.0 + stop_loss_limit_ratio)
+
+    take_profit: float | None = None
+    if take_profit_limit_ratio > 0:
+        if normalized_side == "long":
+            take_profit = entry * (1.0 + take_profit_limit_ratio)
+        else:
+            take_profit = entry * (1.0 - take_profit_limit_ratio)
+
+    return {
+        "entry": entry,
+        "stop_loss": stop_loss,
+        "take_profit": take_profit,
+    }
+
+
+@contextmanager
+def mt5_trading_session(
+    config: Mt5Config | None = None,
+    retry_count: int = 0,
+) -> Iterator[Mt5TradingClient]:
+    """Open a trading-capable MT5 session and always shut down safely.
+
+    Launches the MetaTrader 5 terminal using ``Mt5Config.path`` when set,
+    initializes and logs in via ``initialize_and_login_mt5()``, yields a
+    connected :class:`~pdmt5.Mt5TradingClient`, and calls ``shutdown()`` on
+    exit even when an error is raised inside the context.
+
+    Args:
+        config: MT5 connection configuration. Defaults to an empty config that
+            attaches to a running terminal.
+        retry_count: Number of initialization retries passed to
+            ``Mt5TradingClient``.
+
+    Yields:
+        Connected ``Mt5TradingClient`` bound to the session.
+    """
+    mt5_config = config or build_config()
+    client = Mt5TradingClient(config=mt5_config, retry_count=retry_count)
+    try:
+        client.initialize_and_login_mt5()
+        yield client
+    finally:
+        client.shutdown()

mt5cli/utils.py

@@ -0,0 +1,434 @@
+"""Utility constants, types, and functions for the mt5cli package."""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from datetime import UTC, datetime
+from enum import StrEnum
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, TypeGuard
+
+import click
+from pdmt5 import COPY_TICKS_MAP, TIMEFRAME_MAP
+from pdmt5 import parse_copy_ticks as _parse_copy_ticks
+from pdmt5 import parse_timeframe as _parse_timeframe
+
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
+    import pandas as pd
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+# Backward-compatible snapshot; prefer ``COPY_TICKS_MAP`` from pdmt5 directly.
+TICK_FLAG_MAP: dict[str, int] = dict(COPY_TICKS_MAP)
+
+TIMEFRAME_NAMES: tuple[str, ...] = tuple(
+    name for name in TIMEFRAME_MAP if not name.startswith("TIMEFRAME_")
+)
+_TICK_FLAG_NAMES: tuple[str, ...] = tuple(
+    name for name in COPY_TICKS_MAP if not name.startswith("COPY_TICKS_")
+)
+
+_FORMAT_EXTENSIONS: dict[str, str] = {
+    ".csv": "csv",
+    ".json": "json",
+    ".parquet": "parquet",
+    ".pq": "parquet",
+    ".db": "sqlite3",
+    ".sqlite": "sqlite3",
+    ".sqlite3": "sqlite3",
+}
+
+# ---------------------------------------------------------------------------
+# Enums
+# ---------------------------------------------------------------------------
+
+
+class OutputFormat(StrEnum):
+    """Supported output file formats."""
+
+    csv = "csv"
+    json = "json"
+    parquet = "parquet"
+    sqlite3 = "sqlite3"
+
+
+class LogLevel(StrEnum):
+    """Logging verbosity levels."""
+
+    DEBUG = "DEBUG"
+    INFO = "INFO"
+    WARNING = "WARNING"
+    ERROR = "ERROR"
+
+
+class Dataset(StrEnum):
+    """Datasets supported by the ``collect-history`` command."""
+
+    rates = "rates"
+    ticks = "ticks"
+    history_orders = "history-orders"
+    history_deals = "history-deals"
+
+    @property
+    def table_name(self) -> str:
+        """Return the SQLite table name for this dataset."""
+        return self.value.replace("-", "_")
+
+
+class IfExists(StrEnum):
+    """SQLite table conflict behavior for the ``collect-history`` command."""
+
+    APPEND = "append"
+    REPLACE = "replace"
+    FAIL = "fail"
+
+
+# ---------------------------------------------------------------------------
+# Click parameter types
+# ---------------------------------------------------------------------------
+
+
+class _DateTimeType(click.ParamType):
+    """Click parameter type for ISO 8601 datetime strings."""
+
+    name = "DATETIME"
+
+    def convert(
+        self,
+        value: object,
+        param: click.Parameter | None,
+        ctx: click.Context | None,
+    ) -> datetime:
+        """Convert a string value to a timezone-aware datetime.
+
+        Args:
+            value: Raw value from the command line.
+            param: Click parameter instance.
+            ctx: Click context.
+
+        Returns:
+            Parsed datetime.
+        """
+        if isinstance(value, datetime):
+            return value
+        try:
+            return parse_datetime(str(value))
+        except ValueError as exc:
+            self.fail(str(exc), param, ctx)
+
+
+class _TimeframeType(click.ParamType):
+    """Click parameter type for MT5 timeframe values."""
+
+    name = "TIMEFRAME"
+
+    def convert(
+        self,
+        value: object,
+        param: click.Parameter | None,
+        ctx: click.Context | None,
+    ) -> int:
+        """Convert a string or integer value to a timeframe integer.
+
+        Args:
+            value: Raw value from the command line.
+            param: Click parameter instance.
+            ctx: Click context.
+
+        Returns:
+            Integer timeframe value.
+        """
+        try:
+            return parse_timeframe(value)
+        except ValueError as exc:
+            self.fail(str(exc), param, ctx)
+
+
+class _TickFlagsType(click.ParamType):
+    """Click parameter type for MT5 tick copy flags."""
+
+    name = "FLAGS"
+
+    def convert(
+        self,
+        value: object,
+        param: click.Parameter | None,
+        ctx: click.Context | None,
+    ) -> int:
+        """Convert a string or integer value to a tick flags integer.
+
+        Args:
+            value: Raw value from the command line.
+            param: Click parameter instance.
+            ctx: Click context.
+
+        Returns:
+            Integer tick flag value.
+        """
+        try:
+            return parse_tick_flags(value)
+        except ValueError as exc:
+            self.fail(str(exc), param, ctx)
+
+
+class _RequestType(click.ParamType):
+    """Click parameter type for JSON order requests."""
+
+    name = "REQUEST"
+
+    def convert(
+        self,
+        value: object,
+        param: click.Parameter | None,
+        ctx: click.Context | None,
+    ) -> dict[str, Any]:
+        """Convert a raw CLI value to an order request dictionary.
+
+        Args:
+            value: Raw value from the command line.
+            param: Click parameter instance.
+            ctx: Click context.
+
+        Returns:
+            Parsed request dictionary.
+        """
+        try:
+            return parse_request(str(value))
+        except ValueError as exc:
+            self.fail(str(exc), param, ctx)
+
+
+DATETIME_TYPE = _DateTimeType()
+TIMEFRAME_TYPE = _TimeframeType()
+TICK_FLAGS_TYPE = _TickFlagsType()
+REQUEST_TYPE = _RequestType()
+
+# ---------------------------------------------------------------------------
+# Public utility functions
+# ---------------------------------------------------------------------------
+
+
+def detect_format(
+    output_path: Path,
+    explicit_format: str | None = None,
+) -> str:
+    """Detect the output format from a file extension or explicit format string.
+
+    Args:
+        output_path: Path to the output file.
+        explicit_format: Explicitly specified format, if any.
+
+    Returns:
+        The detected format string.
+
+    Raises:
+        ValueError: If the format cannot be determined.
+    """
+    if explicit_format is not None:
+        return explicit_format
+    suffix = output_path.suffix.lower()
+    if suffix in _FORMAT_EXTENSIONS:
+        return _FORMAT_EXTENSIONS[suffix]
+    msg = (
+        f"Cannot detect format from extension '{suffix}'."
+        " Use --format to specify the output format."
+    )
+    raise ValueError(msg)
+
+
+def export_dataframe_to_sqlite(
+    df: pd.DataFrame,
+    output_path: Path,
+    table_name: str = "data",
+    *,
+    if_exists: IfExists = IfExists.APPEND,
+    index: bool = False,
+    index_label: str | None = None,
+    deduplicate_on: Sequence[str] | None = None,
+) -> None:
+    """Write a DataFrame to SQLite with configurable append and deduplication.
+
+    Args:
+        df: DataFrame to export.
+        output_path: SQLite database path.
+        table_name: Target table name.
+        if_exists: Conflict behavior when the table already exists.
+        index: Whether to write the DataFrame index as a column.
+        index_label: Column name for the index when ``index=True``.
+        deduplicate_on: Optional key columns to deduplicate after writing,
+            keeping the latest ``ROWID`` per key group. Deduplication scans the
+            full table, so repeated appends cost O(table size); index the key
+            columns when appending frequently.
+    """
+    with sqlite3.connect(output_path) as conn:
+        df.to_sql(  # type: ignore[reportUnknownMemberType]
+            table_name,
+            conn,
+            if_exists=if_exists.value,
+            index=index,
+            index_label=index_label,
+        )
+        if deduplicate_on:
+            from .history import drop_duplicates_in_table  # noqa: PLC0415
+
+            drop_duplicates_in_table(
+                conn.cursor(),
+                table_name,
+                list(deduplicate_on),
+                keep="last",
+            )
+            conn.commit()
+
+
+def export_dataframe(
+    df: pd.DataFrame,
+    output_path: Path,
+    output_format: str,
+    table_name: str = "data",
+) -> None:
+    """Export a pandas DataFrame to the specified file format.
+
+    Args:
+        df: DataFrame to export.
+        output_path: Path to the output file.
+        output_format: Output format (csv, json, parquet, or sqlite3).
+        table_name: Table name for SQLite3 output.
+
+    Raises:
+        ValueError: If the output format is not supported.
+    """
+    if output_format == "csv":
+        df.to_csv(output_path, index=False)
+    elif output_format == "json":
+        df.to_json(
+            output_path,
+            orient="records",
+            date_format="iso",
+            indent=2,
+        )
+    elif output_format == "parquet":
+        df.to_parquet(output_path, index=False)
+    elif output_format == "sqlite3":
+        export_dataframe_to_sqlite(
+            df,
+            output_path,
+            table_name,
+            if_exists=IfExists.REPLACE,
+            index=False,
+        )
+    else:
+        msg = f"Unsupported output format: {output_format}"
+        raise ValueError(msg)
+
+
+def parse_datetime(value: str) -> datetime:
+    """Parse an ISO 8601 datetime string to a timezone-aware datetime.
+
+    Args:
+        value: ISO 8601 datetime string (e.g., '2024-01-01' or
+            '2024-01-01T12:00:00+00:00').
+
+    Returns:
+        Parsed datetime with UTC timezone if no timezone is specified.
+
+    Raises:
+        ValueError: If the string cannot be parsed.
+    """
+    try:
+        dt = datetime.fromisoformat(value)
+    except ValueError:
+        msg = f"Invalid datetime format: '{value}'. Use ISO 8601 format."
+        raise ValueError(msg) from None
+    if dt.tzinfo is None:
+        dt = dt.replace(tzinfo=UTC)
+    return dt
+
+
+def parse_timeframe(value: object) -> int:
+    """Parse a timeframe string or integer value.
+
+    Args:
+        value: Timeframe name (e.g., 'M1', 'H1', 'D1') or integer value.
+
+    Returns:
+        Integer timeframe value.
+
+    Raises:
+        ValueError: If the timeframe is invalid.
+    """
+    try:
+        return _parse_timeframe(value)
+    except ValueError:
+        display = value if isinstance(value, str) else repr(value)
+        valid = ", ".join(TIMEFRAME_NAMES)
+        msg = (
+            f"Invalid timeframe: '{display}'. "
+            f"Use one of: {valid}, or a supported integer."
+        )
+        raise ValueError(msg) from None
+
+
+def parse_tick_flags(value: object) -> int:
+    """Parse tick flags string or integer value.
+
+    Args:
+        value: Tick flag name (ALL, INFO, TRADE, COPY_TICKS_*) or integer value.
+
+    Returns:
+        Integer tick flag value compatible with MetaTrader 5 ``COPY_TICKS_*``.
+
+    Raises:
+        ValueError: If the flag is invalid.
+    """
+    try:
+        return _parse_copy_ticks(value)
+    except ValueError:
+        display = value if isinstance(value, str) else repr(value)
+        valid = ", ".join(_TICK_FLAG_NAMES)
+        msg = (
+            f"Invalid tick flags: '{display}'. "
+            f"Use one of: {valid}, or a supported integer."
+        )
+        raise ValueError(msg) from None
+
+
+def _is_request_dict(value: object) -> TypeGuard[dict[str, Any]]:
+    return isinstance(value, dict)
+
+
+def parse_request(value: str) -> dict[str, Any]:
+    """Parse a JSON-formatted order request string or file reference.
+
+    Args:
+        value: JSON object string, or '@path' to read JSON from a file.
+
+    Returns:
+        Parsed request dictionary.
+
+    Raises:
+        ValueError: If the request file cannot be read or the value is not a
+            JSON object.
+    """
+    if value.startswith("@"):
+        path = Path(value[1:])
+        try:
+            text = path.read_text(encoding="utf-8")
+        except (OSError, UnicodeDecodeError) as exc:
+            msg = f"Failed to read JSON request file '{path}': {exc}"
+            raise ValueError(msg) from exc
+    else:
+        text = value
+    try:
+        parsed: object = json.loads(text)
+    except json.JSONDecodeError as exc:
+        msg = f"Invalid JSON request: {exc}"
+        raise ValueError(msg) from exc
+    if not _is_request_dict(parsed):
+        msg = "Order request must be a JSON object."
+        raise ValueError(msg)
+    return parsed

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "mt5cli"
-version = "0.2.0"
-description = "Command-line tool for MetaTrader 5"
+version = "0.7.2"
+description = "Generic MT5 data and execution infrastructure for Python applications"
 authors = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
 maintainers = [{name = "dceoy", email = "dceoy@users.noreply.github.com"}]
 license = "MIT"
@@ -9,7 +9,7 @@ license-files = ["LICENSE"]
 readme = "README.md"
 requires-python = ">= 3.11, < 3.14"
 dependencies = [
-  "pdmt5 >= 0.2.3",
+  "pdmt5>=0.3.0",
   "click >= 8.1.0",
   "pyarrow >= 19.0.0",
   "typer >= 0.15.0",
@@ -48,10 +48,6 @@ dev = [
   "pymdown-extensions >= 10.21.2",
 ]
 
-[tool.uv.build-backend]
-source-include = ["mt5cli/**", "LICENSE"]
-source-exclude = ["tests/**"]
-
 [tool.ruff]
 line-length = 88
 exclude = ["build", ".venv"]
@@ -128,6 +124,7 @@ ignore = [
 ]
 
 [tool.ruff.lint.per-file-ignores]
+"mt5cli/history.py" = ["TC003"]
 "tests/**/*.py" = [
   "DOC201",   # Missing return documentation
   "DOC501",   # Raised exception missing from docstring

skills/mt5cli/SKILL.md

@@ -50,21 +50,22 @@ Global options MUST precede the subcommand.
 
 ## Commands
 
-| Command          | Required options                                      | Optional options                                                            |
-| ---------------- | ----------------------------------------------------- | --------------------------------------------------------------------------- |
-| `rates-from`     | `--symbol`, `--timeframe`, `--date-from`, `--count`   | —                                                                           |
-| `rates-from-pos` | `--symbol`, `--timeframe`, `--start-pos`, `--count`   | —                                                                           |
-| `rates-range`    | `--symbol`, `--timeframe`, `--date-from`, `--date-to` | —                                                                           |
-| `ticks-from`     | `--symbol`, `--date-from`, `--count`, `--flags`       | —                                                                           |
-| `ticks-range`    | `--symbol`, `--date-from`, `--date-to`, `--flags`     | —                                                                           |
-| `account-info`   | —                                                     | —                                                                           |
-| `terminal-info`  | —                                                     | —                                                                           |
-| `symbols`        | —                                                     | `--group` (e.g., `*USD*`)                                                   |
-| `symbol-info`    | `--symbol`                                            | —                                                                           |
-| `orders`         | —                                                     | `--symbol`, `--group`, `--ticket`                                           |
-| `positions`      | —                                                     | `--symbol`, `--group`, `--ticket`                                           |
-| `history-orders` | —                                                     | `--date-from`, `--date-to`, `--group`, `--symbol`, `--ticket`, `--position` |
-| `history-deals`  | —                                                     | `--date-from`, `--date-to`, `--group`, `--symbol`, `--ticket`, `--position` |
+| Command           | Required options                                      | Optional options                                                                                                                                                                                                                 |
+| ----------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `rates-from`      | `--symbol`, `--timeframe`, `--date-from`, `--count`   | —                                                                                                                                                                                                                                |
+| `rates-from-pos`  | `--symbol`, `--timeframe`, `--start-pos`, `--count`   | —                                                                                                                                                                                                                                |
+| `rates-range`     | `--symbol`, `--timeframe`, `--date-from`, `--date-to` | —                                                                                                                                                                                                                                |
+| `ticks-from`      | `--symbol`, `--date-from`, `--count`, `--flags`       | —                                                                                                                                                                                                                                |
+| `ticks-range`     | `--symbol`, `--date-from`, `--date-to`, `--flags`     | —                                                                                                                                                                                                                                |
+| `account-info`    | —                                                     | —                                                                                                                                                                                                                                |
+| `terminal-info`   | —                                                     | —                                                                                                                                                                                                                                |
+| `symbols`         | —                                                     | `--group` (e.g., `*USD*`)                                                                                                                                                                                                        |
+| `symbol-info`     | `--symbol`                                            | —                                                                                                                                                                                                                                |
+| `orders`          | —                                                     | `--symbol`, `--group`, `--ticket`                                                                                                                                                                                                |
+| `positions`       | —                                                     | `--symbol`, `--group`, `--ticket`                                                                                                                                                                                                |
+| `history-orders`  | —                                                     | `--date-from`, `--date-to`, `--group`, `--symbol`, `--ticket`, `--position`                                                                                                                                                      |
+| `history-deals`   | —                                                     | `--date-from`, `--date-to`, `--group`, `--symbol`, `--ticket`, `--position`                                                                                                                                                      |
+| `collect-history` | `--symbol` (repeatable), `--date-from`, `--date-to`   | `--dataset` (repeatable; rates/ticks/history-orders/history-deals; default all), `--timeframe` (M1; recorded on rates), `--flags` (ALL), `--if-exists` (append/replace/fail; default fail), `--with-views` (SQLite3 output only) |
 
 ## Examples
 
@@ -85,6 +86,14 @@ mt5cli -o data.db --table symbols symbols --group "*USD*"
 
 # Historical deals filtered by symbol (using an already-logged-in MT5 terminal).
 mt5cli -o deals.csv history-deals --symbol EURUSD --date-from 2024-01-01
+
+# Bundle selected historical datasets into one SQLite db, appending to any
+# existing tables, plus cash_events and positions_reconstructed views.
+mt5cli -o history.db collect-history \
+  --symbol EURUSD --symbol GBPUSD \
+  --date-from 2024-01-01 --date-to 2024-02-01 \
+  --dataset rates --dataset history-deals \
+  --timeframe M1 --flags ALL --if-exists append --with-views
 ```
 
 ## Guidelines

tests/conftest.py

@@ -0,0 +1,52 @@
+"""Shared pytest fixtures for mt5cli tests."""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pandas as pd
+import pytest
+from pytest_mock import MockerFixture  # noqa: TC002
+
+_DATAFRAME_METHODS = (
+    "copy_rates_from_as_df",
+    "copy_rates_from_pos_as_df",
+    "copy_rates_range_as_df",
+    "copy_ticks_from_as_df",
+    "copy_ticks_range_as_df",
+    "account_info_as_df",
+    "terminal_info_as_df",
+    "symbols_get_as_df",
+    "symbol_info_as_df",
+    "orders_get_as_df",
+    "positions_get_as_df",
+    "history_orders_get_as_df",
+    "history_deals_get_as_df",
+    "version_as_df",
+    "last_error_as_df",
+    "symbol_info_tick_as_df",
+    "market_book_get_as_df",
+    "order_check_as_df",
+    "order_send_as_df",
+)
+
+
+def build_mock_mt5_data_client() -> MagicMock:
+    """Return a MagicMock Mt5DataClient with common DataFrame stubs."""
+    client = MagicMock()
+    sample_df = pd.DataFrame({"col": [1]})
+    for method_name in _DATAFRAME_METHODS:
+        getattr(client, method_name).return_value = sample_df
+    client.version.return_value = (5, 0, 1)
+    client.terminal_info.return_value = {"connected": True, "paths": ["terminal.exe"]}
+    client.account_info.return_value = {"login": 123, "limits": {"modes": ["demo"]}}
+    client.symbols_total.return_value = 42
+    return client
+
+
+@pytest.fixture
+def mock_client(mocker: MockerFixture) -> MagicMock:
+    """Create and patch a mock Mt5DataClient for CLI and SDK tests."""
+    client = build_mock_mt5_data_client()
+    mocker.patch("mt5cli.sdk.Mt5DataClient", return_value=client)
+    return client

tests/test_contracts.py

@@ -0,0 +1,512 @@
+"""Contract tests for the mt5cli public API and dataset schemas."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import TYPE_CHECKING
+
+import pandas as pd
+import pytest
+from pdmt5 import Mt5RuntimeError, Mt5TradingError
+from pytest_mock import MockerFixture  # noqa: TC002
+
+from mt5cli import (
+    DEDUP_KEYS,
+    REQUIRED_COLUMNS,
+    TIME_COLUMNS,
+    DataKind,
+    Dataset,
+    MT5Client,
+    Mt5CliError,
+    Mt5ConnectionError,
+    Mt5OperationError,
+    Mt5SchemaError,
+    build_config,
+    call_with_normalized_errors,
+    detect_format,
+    ensure_utc,
+    export_dataframe,
+    export_dataframe_to_sqlite,
+    granularity_name,
+    is_recoverable_mt5_error,
+    mt5_session,
+    normalize_dataframe,
+    normalize_mt5_exception,
+    normalize_symbol,
+    normalize_symbols,
+    parse_date_range,
+    recent_window,
+    schema_columns,
+    validate_schema,
+)
+from mt5cli.retry import retry_with_backoff
+from mt5cli.schemas import ensure_utc_columns, normalize_time_columns
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+
+def _sample_frame(kind: DataKind) -> pd.DataFrame:
+    if kind is DataKind.rates:
+        return pd.DataFrame({
+            "time": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "open": [1.1],
+            "high": [1.2],
+            "low": [1.0],
+            "close": [1.15],
+            "tick_volume": [10],
+            "spread": [1],
+            "real_volume": [0],
+        })
+    if kind is DataKind.ticks:
+        return pd.DataFrame({
+            "time": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "bid": [1.1],
+            "ask": [1.11],
+            "last": [1.105],
+            "volume": [1],
+            "time_msc": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "flags": [2],
+            "volume_real": [0.0],
+        })
+    if kind is DataKind.orders:
+        return pd.DataFrame({
+            "ticket": [1],
+            "time_setup": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "type": [0],
+            "state": [1],
+            "symbol": ["EURUSD"],
+            "volume_current": [0.1],
+            "price_open": [1.1],
+        })
+    if kind is DataKind.positions:
+        return pd.DataFrame({
+            "ticket": [1],
+            "time": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "type": [0],
+            "symbol": ["EURUSD"],
+            "volume": [0.1],
+            "price_open": [1.1],
+            "price_current": [1.11],
+            "profit": [1.0],
+        })
+    if kind is DataKind.history_orders:
+        return pd.DataFrame({
+            "ticket": [1],
+            "time_setup": [datetime(2024, 1, 1, tzinfo=UTC)],
+            "type": [0],
+            "state": [3],
+            "symbol": ["EURUSD"],
+            "volume_initial": [0.1],
+            "price_open": [1.1],
+        })
+    return pd.DataFrame({
+        "ticket": [1],
+        "order": [2],
+        "time": [datetime(2024, 1, 1, tzinfo=UTC)],
+        "type": [0],
+        "entry": [0],
+        "symbol": ["EURUSD"],
+        "volume": [0.1],
+        "price": [1.1],
+        "profit": [0.0],
+    })
+
+
+@pytest.mark.parametrize("kind", list(DataKind))
+def test_required_columns_contract(kind: DataKind) -> None:
+    """Each dataset kind exposes a non-empty required column contract."""
+    assert REQUIRED_COLUMNS[kind]
+    validate_schema(_sample_frame(kind), kind)
+
+
+@pytest.mark.parametrize("kind", list(DataKind))
+def test_normalize_dataframe_injects_storage_metadata(kind: DataKind) -> None:
+    """Normalization accepts MT5 frames and optional storage metadata."""
+    frame = _sample_frame(kind)
+    normalized = normalize_dataframe(
+        frame,
+        kind,
+        symbol="eurusd",
+        timeframe="M1" if kind is DataKind.rates else None,
+    )
+    if kind is DataKind.rates:
+        assert normalized.loc[0, "symbol"] == "eurusd"
+        assert normalized.loc[0, "timeframe"] == 1
+    validate_schema(normalized, kind)
+
+
+def test_validate_schema_raises_for_missing_columns() -> None:
+    """Schema validation fails fast on missing required columns."""
+    with pytest.raises(Mt5SchemaError, match="missing required columns"):
+        validate_schema(pd.DataFrame({"time": [1]}), DataKind.rates)
+
+
+def test_history_dedup_keys_match_schema_contract() -> None:
+    """SQLite history dedup keys stay aligned with schema contracts."""
+    assert DEDUP_KEYS[DataKind.rates][0] == ("symbol", "timeframe", "time")
+    assert DEDUP_KEYS[DataKind.ticks][0] == ("symbol", "time_msc")
+    assert Dataset.rates.table_name == "rates"
+
+
+@pytest.mark.parametrize(
+    ("raw", "expected"),
+    [
+        (" eurusd ", "eurusd"),
+        ("GbpJpy", "GbpJpy"),
+        ("XAUUSDm", "XAUUSDm"),
+        ("US500.cash", "US500.cash"),
+        ("EURUSD.r", "EURUSD.r"),
+    ],
+)
+def test_normalize_symbol(raw: str, expected: str) -> None:
+    """Symbol normalization trims whitespace and preserves broker casing."""
+    assert normalize_symbol(raw) == expected
+
+
+def test_normalize_symbols_deduplicates() -> None:
+    """Symbol lists are normalized and de-duplicated in order."""
+    assert normalize_symbols(["XAUUSDm", " XAUUSDm ", "EURUSD.r", "eurusd"]) == [
+        "XAUUSDm",
+        "EURUSD.r",
+        "eurusd",
+    ]
+
+
+def test_parse_date_range_rejects_inverted_bounds() -> None:
+    """Date ranges must not be inverted."""
+    with pytest.raises(ValueError, match="must not be after"):
+        parse_date_range("2024-02-01", "2024-01-01")
+
+
+def test_recent_window_builds_trailing_bounds() -> None:
+    """Recent windows end at the provided timestamp."""
+    end = datetime(2024, 1, 2, tzinfo=UTC)
+    start, resolved_end = recent_window(hours=24, date_to=end)
+    assert resolved_end == end
+    assert start < end
+
+
+def test_granularity_name_maps_timeframe_alias() -> None:
+    """Granularity labels resolve MT5 timeframe aliases."""
+    assert granularity_name("M1") == "M1"
+
+
+@pytest.mark.parametrize(
+    "exc",
+    [Mt5RuntimeError("init failed"), Mt5TradingError("trade failed")],
+)
+def test_is_recoverable_mt5_error(exc: Exception) -> None:
+    """Recoverable MT5 errors are classified consistently."""
+    assert is_recoverable_mt5_error(exc)
+
+
+def test_normalize_mt5_exception_maps_types() -> None:
+    """MT5 exceptions map to stable mt5cli types."""
+    assert isinstance(
+        normalize_mt5_exception(Mt5RuntimeError("x")),
+        Mt5ConnectionError,
+    )
+    assert isinstance(
+        normalize_mt5_exception(Mt5TradingError("x")),
+        Mt5OperationError,
+    )
+
+
+def test_call_with_normalized_errors_reraises_mapped_type() -> None:
+    """Normalized error helper re-raises mapped mt5cli exceptions."""
+
+    def _raise() -> None:
+        message = "boom"
+        raise Mt5RuntimeError(message)
+
+    with pytest.raises(Mt5ConnectionError):
+        call_with_normalized_errors(_raise)
+
+
+def test_retry_with_backoff_retries_recoverable_errors(
+    mocker: MockerFixture,
+) -> None:
+    """Retry helper retries recoverable MT5 failures."""
+    calls = {"count": 0}
+
+    def _flaky() -> str:
+        calls["count"] += 1
+        if calls["count"] == 1:
+            message = "transient"
+            raise Mt5RuntimeError(message)
+        return "ok"
+
+    mocker.patch("mt5cli.retry.time.sleep")
+    assert retry_with_backoff(_flaky, retry_count=1) == "ok"
+    assert calls["count"] == 2
+
+
+def test_public_api_exports_mt5_client() -> None:
+    """MT5Client is the primary importable client abstraction."""
+    client = MT5Client(config=build_config())
+    assert isinstance(client, MT5Client)
+    assert isinstance(client, MT5Client.__mro__[1])
+
+
+def test_mt5_client_order_primitives_use_connected_client(
+    mock_client: object,
+) -> None:
+    """Order check/send route through the same client fetch path as exports."""
+    request = {"action": 1}
+    client = MT5Client()
+    client.order_check(request)
+    client.order_send(request)
+    assert mock_client.order_check_as_df.call_count == 1  # type: ignore[attr-defined]
+    assert mock_client.order_send_as_df.call_count == 1  # type: ignore[attr-defined]
+
+
+def test_storage_export_round_trip_csv(tmp_path: Path) -> None:
+    """Storage helpers export normalized rate frames to CSV."""
+    frame = normalize_dataframe(
+        _sample_frame(DataKind.rates),
+        DataKind.rates,
+        symbol="EURUSD",
+        timeframe="M1",
+    )
+    output = tmp_path / "rates.csv"
+    export_dataframe(frame, output, detect_format(output))
+    loaded = pd.read_csv(output)
+    assert len(loaded) == 1
+    assert "close" in loaded.columns
+
+
+def test_normalize_symbol_rejects_empty_value() -> None:
+    """Empty symbols are rejected after trimming."""
+    with pytest.raises(ValueError, match="must not be empty"):
+        normalize_symbol("   ")
+
+
+def test_ensure_utc_handles_naive_and_aware_datetimes() -> None:
+    """UTC coercion accepts naive and timezone-aware datetimes."""
+    naive = datetime(2024, 1, 1, tzinfo=UTC).replace(tzinfo=None)
+    aware = datetime(2024, 1, 1, tzinfo=UTC)
+    assert ensure_utc(naive).tzinfo == UTC
+    assert ensure_utc(aware).tzinfo == UTC
+    assert ensure_utc("2024-01-01T00:00:00+00:00").tzinfo == UTC
+
+
+def test_recent_window_validation_errors() -> None:
+    """Recent window helpers validate mutually exclusive length arguments."""
+    with pytest.raises(ValueError, match="exactly one"):
+        recent_window()
+    with pytest.raises(ValueError, match="exactly one"):
+        recent_window(hours=1, seconds=1)
+    with pytest.raises(ValueError, match="positive"):
+        recent_window(hours=0)
+
+
+def test_recent_window_supports_seconds_argument() -> None:
+    """Recent windows can be built from a seconds-based length."""
+    end = datetime(2024, 1, 2, tzinfo=UTC)
+    start, resolved_end = recent_window(seconds=3600, date_to=end)
+    assert resolved_end == end
+    assert start < end
+
+
+def test_parse_date_range_returns_ordered_bounds() -> None:
+    """Valid date ranges return UTC-aware bounds."""
+    start, end = parse_date_range("2024-01-01", "2024-02-01")
+    assert start < end
+
+
+def test_granularity_name_falls_back_for_unknown_timeframe(
+    mocker: MockerFixture,
+) -> None:
+    """Unknown timeframe integers stringify as granularity labels."""
+    mocker.patch(
+        "mt5cli.converters._get_timeframe_name",
+        side_effect=ValueError("unknown"),
+    )
+    assert granularity_name(1) == "1"
+
+
+def test_normalize_mt5_exception_passthrough_and_generic() -> None:
+    """Normalization preserves mt5cli errors and wraps unknown exceptions."""
+    original = Mt5CliError("known")
+    assert normalize_mt5_exception(original) is original
+    assert isinstance(normalize_mt5_exception(ValueError("x")), Mt5CliError)
+
+
+def test_schema_columns_and_extra_required_validation() -> None:
+    """Schema helpers expose contracts and honor extra required columns."""
+    assert schema_columns(DataKind.rates) == REQUIRED_COLUMNS[DataKind.rates]
+    validate_schema(pd.DataFrame(), DataKind.rates)
+    frame = _sample_frame(DataKind.rates)
+    with pytest.raises(Mt5SchemaError, match="storage_symbol"):
+        validate_schema(frame, DataKind.rates, extra_required=["storage_symbol"])
+
+
+def test_normalize_dataframe_empty_and_tick_sort_paths() -> None:
+    """Normalization handles empty frames and tick time_msc sorting."""
+    empty = pd.DataFrame()
+    assert normalize_dataframe(empty, DataKind.rates).empty
+
+    ticks = _sample_frame(DataKind.ticks)
+    ticks = pd.concat([ticks, ticks], ignore_index=True)
+    sorted_ticks = normalize_dataframe(ticks, DataKind.ticks, sort=True)
+    assert len(sorted_ticks) == 2
+    unsorted_ticks = normalize_dataframe(ticks, DataKind.ticks, sort=False)
+    assert len(unsorted_ticks) == 2
+
+
+def test_normalize_dataframe_rate_timeframe_without_symbol() -> None:
+    """Rate normalization can inject timeframe without symbol metadata."""
+    frame = _sample_frame(DataKind.rates)
+    normalized = normalize_dataframe(frame, DataKind.rates, timeframe="M1")
+    assert "timeframe" in normalized.columns
+
+
+def test_normalize_dataframe_keeps_existing_symbol_and_timeframe() -> None:
+    """Normalization does not duplicate existing storage metadata columns."""
+    frame = normalize_dataframe(
+        _sample_frame(DataKind.rates),
+        DataKind.rates,
+        symbol="EURUSD",
+        timeframe="M1",
+    )
+    normalized = normalize_dataframe(
+        frame,
+        DataKind.rates,
+        symbol="GBPUSD",
+        timeframe="H1",
+    )
+    assert normalized.loc[0, "symbol"] == "EURUSD"
+    assert normalized.loc[0, "timeframe"] == 1
+
+
+def test_normalize_time_columns_skips_absent_time_fields() -> None:
+    """Time normalization ignores absent optional time columns."""
+    frame = pd.DataFrame({"open": [1.0]})
+    result = normalize_time_columns(frame, DataKind.rates)
+    assert list(result.columns) == ["open"]
+
+
+def test_normalize_time_columns_converts_unix_seconds() -> None:
+    """Numeric MT5 ``time`` values are interpreted as Unix seconds."""
+    frame = pd.DataFrame({"time": [1704067200]})
+    result = normalize_time_columns(frame, DataKind.rates)
+    assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+
+
+def test_normalize_time_columns_converts_unix_milliseconds() -> None:
+    """Numeric MT5 ``time_msc`` values are interpreted as Unix milliseconds."""
+    frame = pd.DataFrame({"time_msc": [1704067200000]})
+    result = normalize_time_columns(frame, DataKind.ticks)
+    assert result.loc[0, "time_msc"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+
+
+def test_normalize_time_columns_preserves_utc_datetimes() -> None:
+    """Already-converted datetime values remain UTC-normalized."""
+    aware = datetime(2024, 1, 1, tzinfo=UTC)
+    frame = pd.DataFrame({"time": [aware]})
+    result = normalize_time_columns(frame, DataKind.rates)
+    assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+
+
+def test_normalize_time_columns_handles_optional_order_times() -> None:
+    """Optional order/history time columns are normalized when present."""
+    frame = pd.DataFrame({
+        "time_setup": [1704067200],
+        "time_setup_msc": [1704067200000],
+        "time_done": [1704153600],
+        "time_done_msc": [1704153600000],
+    })
+    result = normalize_time_columns(frame, DataKind.orders)
+    assert result.loc[0, "time_setup"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+    assert result.loc[0, "time_setup_msc"] == pd.Timestamp(
+        "2024-01-01T00:00:00+00:00",
+    )
+    assert result.loc[0, "time_done"] == pd.Timestamp("2024-01-02T00:00:00+00:00")
+    assert result.loc[0, "time_done_msc"] == pd.Timestamp(
+        "2024-01-02T00:00:00+00:00",
+    )
+
+
+def test_time_columns_include_optional_order_fields() -> None:
+    """Schema contracts document optional MT5 time columns per dataset kind."""
+    assert "time_done" in TIME_COLUMNS[DataKind.orders]
+    assert "time_setup_msc" in TIME_COLUMNS[DataKind.history_orders]
+
+
+def test_normalize_dataframe_sorts_ticks_by_time_msc(
+    mocker: MockerFixture,
+) -> None:
+    """Tick frames without ``time`` can still sort on ``time_msc``."""
+    mocker.patch("mt5cli.schemas.validate_schema")
+    ticks = pd.concat([_sample_frame(DataKind.ticks)] * 2, ignore_index=True).drop(
+        columns=["time"],
+    )
+    ticks.loc[0, "time_msc"] = datetime(2024, 1, 1, tzinfo=UTC)
+    ticks.loc[1, "time_msc"] = datetime(2024, 1, 2, tzinfo=UTC)
+    ticks = pd.concat([ticks.iloc[[1]], ticks.iloc[[0]]], ignore_index=True)
+    normalized = normalize_dataframe(ticks, DataKind.ticks, sort=True)
+    assert normalized.iloc[0]["time_msc"] <= normalized.iloc[1]["time_msc"]
+
+
+def test_ensure_utc_columns_skips_missing_columns() -> None:
+    """UTC column coercion ignores absent columns."""
+    frame = _sample_frame(DataKind.rates)
+    result = ensure_utc_columns(frame, ["time", "missing"])
+    assert "time" in result.columns
+
+
+def test_normalize_time_columns_coerces_string_timestamps() -> None:
+    """String timestamps are parsed with timezone-aware datetime coercion."""
+    frame = pd.DataFrame({"time": ["2024-01-01T00:00:00+00:00"]})
+    result = normalize_time_columns(frame, DataKind.rates)
+    assert result.loc[0, "time"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+
+
+def test_ensure_utc_columns_coerces_non_mt5_columns() -> None:
+    """Non-MT5 columns still coerce to UTC datetimes."""
+    frame = pd.DataFrame({"created_at": ["2024-01-01T00:00:00+00:00"]})
+    result = ensure_utc_columns(frame, ["created_at"])
+    assert result.loc[0, "created_at"] == pd.Timestamp("2024-01-01T00:00:00+00:00")
+
+
+def test_mt5_session_yields_connected_client(mocker: MockerFixture) -> None:
+    """Public mt5_session yields an MT5Client bound to a connected session."""
+    connected = mocker.MagicMock()
+    context = mocker.MagicMock()
+    context.__enter__.return_value = connected
+    context.__exit__.return_value = False
+    mocker.patch("mt5cli.client.connected_client", return_value=context)
+    with mt5_session(build_config()) as client:
+        assert isinstance(client, MT5Client)
+
+
+def test_retry_with_backoff_reraises_non_recoverable_errors() -> None:
+    """Non-MT5 errors are not retried."""
+
+    def _raise() -> None:
+        message = "fatal"
+        raise ValueError(message)
+
+    with pytest.raises(ValueError, match="fatal"):
+        retry_with_backoff(_raise, retry_count=2)
+
+
+def test_storage_export_round_trip_sqlite(tmp_path: Path) -> None:
+    """Storage helpers append deduplicated frames to SQLite."""
+    frame = normalize_dataframe(
+        _sample_frame(DataKind.rates),
+        DataKind.rates,
+        symbol="EURUSD",
+        timeframe="M1",
+    )
+    output = tmp_path / "rates.db"
+    export_dataframe_to_sqlite(
+        frame,
+        output,
+        "rates",
+        deduplicate_on=DEDUP_KEYS[DataKind.rates][0],
+    )
+    with __import__("sqlite3").connect(output) as conn:
+        count = conn.execute("SELECT COUNT(*) FROM rates").fetchone()[0]
+    assert count == 1

tests/test_trading.py

@@ -0,0 +1,356 @@
+"""Tests for trading session helpers and operational utilities."""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pandas as pd
+import pytest
+from pdmt5 import Mt5RuntimeError
+from pytest_mock import MockerFixture  # noqa: TC002
+
+from mt5cli.sdk import build_config
+from mt5cli.trading import (
+    calculate_margin_and_volume,
+    detect_position_side,
+    determine_order_limits,
+    mt5_trading_session,
+)
+
+
+class TestDetectPositionSide:
+    """Tests for detect_position_side."""
+
+    def test_returns_none_when_no_positions(self) -> None:
+        """Test None is returned when no open positions exist."""
+        client = MagicMock()
+        client.positions_get_as_df.return_value = pd.DataFrame()
+
+        assert detect_position_side(client, "EURUSD") is None
+
+    def test_returns_long_for_net_buy_volume(self) -> None:
+        """Test long is returned when buy volume exceeds sell volume."""
+        client = MagicMock()
+        client.mt5.POSITION_TYPE_BUY = 0
+        client.mt5.POSITION_TYPE_SELL = 1
+        client.positions_get_as_df.return_value = pd.DataFrame(
+            {
+                "type": [0, 0, 1],
+                "volume": [0.2, 0.1, 0.05],
+            },
+        )
+
+        assert detect_position_side(client, "EURUSD") == "long"
+
+    def test_returns_short_for_net_sell_volume(self) -> None:
+        """Test short is returned when sell volume exceeds buy volume."""
+        client = MagicMock()
+        client.mt5.POSITION_TYPE_BUY = 0
+        client.mt5.POSITION_TYPE_SELL = 1
+        client.positions_get_as_df.return_value = pd.DataFrame(
+            {
+                "type": [1, 1],
+                "volume": [0.3, 0.1],
+            },
+        )
+
+        assert detect_position_side(client, "EURUSD") == "short"
+
+    def test_returns_none_for_balanced_hedged_positions(self) -> None:
+        """Test None is returned when buy and sell volumes net to zero."""
+        client = MagicMock()
+        client.mt5.POSITION_TYPE_BUY = 0
+        client.mt5.POSITION_TYPE_SELL = 1
+        client.positions_get_as_df.return_value = pd.DataFrame(
+            {
+                "type": [0, 1],
+                "volume": [0.2, 0.2],
+            },
+        )
+
+        assert detect_position_side(client, "EURUSD") is None
+
+
+class TestCalculateMarginAndVolume:
+    """Tests for calculate_margin_and_volume."""
+
+    def test_calculates_margin_budget_and_volumes(self) -> None:
+        """Test margin budget and buy/sell volumes are derived from ratios."""
+        client = MagicMock()
+        client.account_info_as_dict.return_value = {"margin_free": 1000.0}
+        client.calculate_volume_by_margin.side_effect = [0.3, 0.2]
+
+        result = calculate_margin_and_volume(
+            client,
+            "EURUSD",
+            unit_margin_ratio=0.5,
+            preserved_margin_ratio=0.2,
+        )
+
+        assert result == {
+            "margin_free": 1000.0,
+            "available_margin": 800.0,
+            "trade_margin": 400.0,
+            "buy_volume": 0.3,
+            "sell_volume": 0.2,
+        }
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "BUY")
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 400.0, "SELL")
+
+    @pytest.mark.parametrize(
+        ("account_dict", "expected_margin_free"),
+        [
+            ({"margin_free": 0.0}, 0.0),
+            ({}, 0.0),
+            ({"margin_free": None}, 0.0),
+        ],
+    )
+    def test_zero_or_missing_margin_free(
+        self,
+        account_dict: dict[str, float | None],
+        expected_margin_free: float,
+    ) -> None:
+        """Test missing or zero margin_free yields zero trade margin."""
+        client = MagicMock()
+        client.account_info_as_dict.return_value = account_dict
+        client.calculate_volume_by_margin.return_value = 0.0
+
+        result = calculate_margin_and_volume(
+            client,
+            "EURUSD",
+            unit_margin_ratio=0.5,
+            preserved_margin_ratio=0.2,
+        )
+
+        assert result["margin_free"] == expected_margin_free
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
+
+    def test_clamps_negative_margin_free_to_zero(self) -> None:
+        """Test negative margin_free is clamped to zero before sizing."""
+        client = MagicMock()
+        client.account_info_as_dict.return_value = {"margin_free": -500.0}
+        client.calculate_volume_by_margin.return_value = 0.0
+
+        result = calculate_margin_and_volume(
+            client,
+            "EURUSD",
+            unit_margin_ratio=0.5,
+            preserved_margin_ratio=0.2,
+        )
+
+        expected_margin_free = 0.0
+        assert result["margin_free"] == expected_margin_free
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "BUY")
+        client.calculate_volume_by_margin.assert_any_call("EURUSD", 0.0, "SELL")
+
+    @pytest.mark.parametrize(
+        ("unit_ratio", "preserved_ratio"),
+        [
+            (-0.1, 0.0),
+            (1.1, 0.0),
+            (0.5, -0.1),
+            (0.5, 1.1),
+        ],
+    )
+    def test_rejects_invalid_ratios(
+        self,
+        unit_ratio: float,
+        preserved_ratio: float,
+    ) -> None:
+        """Test invalid ratio values raise ValueError."""
+        with pytest.raises(ValueError, match="must be between 0 and 1"):
+            calculate_margin_and_volume(
+                MagicMock(),
+                "EURUSD",
+                unit_margin_ratio=unit_ratio,
+                preserved_margin_ratio=preserved_ratio,
+            )
+
+
+class TestDetermineOrderLimits:
+    """Tests for determine_order_limits."""
+
+    @pytest.mark.parametrize(
+        ("side", "expected_entry_key"),
+        [
+            ("long", "ask"),
+            ("short", "bid"),
+            ("buy", "ask"),
+            ("sell", "bid"),
+        ],
+    )
+    def test_uses_expected_quote_for_entry(
+        self,
+        side: str,
+        expected_entry_key: str,
+    ) -> None:
+        """Test entry price is taken from ask for long/buy and bid for short/sell."""
+        client = MagicMock()
+        client.symbol_info_tick_as_dict.return_value = {"ask": 1.1010, "bid": 1.1000}
+
+        result = determine_order_limits(
+            client,
+            "EURUSD",
+            side,
+            stop_loss_limit_ratio=0.0,
+            take_profit_limit_ratio=0.0,
+        )
+
+        assert (
+            result["entry"]
+            == client.symbol_info_tick_as_dict.return_value[expected_entry_key]
+        )
+        assert result["stop_loss"] is None
+        assert result["take_profit"] is None
+
+    def test_calculates_long_protective_levels(self) -> None:
+        """Test long stop loss and take profit are placed below/above entry."""
+        client = MagicMock()
+        client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
+
+        result = determine_order_limits(
+            client,
+            "EURUSD",
+            "long",
+            stop_loss_limit_ratio=0.02,
+            take_profit_limit_ratio=0.03,
+        )
+
+        assert result == {
+            "entry": 100.0,
+            "stop_loss": 98.0,
+            "take_profit": 103.0,
+        }
+
+    def test_calculates_short_protective_levels(self) -> None:
+        """Test short stop loss and take profit are placed above/below entry."""
+        client = MagicMock()
+        client.symbol_info_tick_as_dict.return_value = {"ask": 100.0, "bid": 99.0}
+
+        result = determine_order_limits(
+            client,
+            "EURUSD",
+            "short",
+            stop_loss_limit_ratio=0.02,
+            take_profit_limit_ratio=0.03,
+        )
+
+        assert result == {
+            "entry": 99.0,
+            "stop_loss": 100.98,
+            "take_profit": 96.03,
+        }
+
+    def test_rejects_unknown_side(self) -> None:
+        """Test unsupported side values raise ValueError."""
+        with pytest.raises(ValueError, match="Unsupported order side"):
+            determine_order_limits(
+                MagicMock(),
+                "EURUSD",
+                "flat",
+                stop_loss_limit_ratio=0.01,
+                take_profit_limit_ratio=0.01,
+            )
+
+    @pytest.mark.parametrize(
+        ("stop_loss_ratio", "take_profit_ratio"),
+        [
+            (-0.05, 0.01),
+            (0.01, 2.0),
+        ],
+    )
+    def test_rejects_invalid_protective_ratios(
+        self,
+        stop_loss_ratio: float,
+        take_profit_ratio: float,
+    ) -> None:
+        """Test out-of-range protective ratios raise ValueError."""
+        with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
+            determine_order_limits(
+                MagicMock(),
+                "EURUSD",
+                "long",
+                stop_loss_limit_ratio=stop_loss_ratio,
+                take_profit_limit_ratio=take_profit_ratio,
+            )
+
+    @pytest.mark.parametrize(
+        ("field", "ratio"),
+        [
+            ("stop_loss_limit_ratio", 1.0),
+            ("take_profit_limit_ratio", 1.0),
+        ],
+    )
+    def test_rejects_unit_boundary_protective_ratios(
+        self,
+        field: str,
+        ratio: float,
+    ) -> None:
+        """Test protective ratios of exactly 1.0 are rejected."""
+        kwargs = {
+            "stop_loss_limit_ratio": 0.01,
+            "take_profit_limit_ratio": 0.01,
+            field: ratio,
+        }
+        with pytest.raises(ValueError, match="must be at least 0 and less than 1"):
+            determine_order_limits(
+                MagicMock(),
+                "EURUSD",
+                "long",
+                **kwargs,
+            )
+
+
+class TestMt5TradingSession:
+    """Tests for the mt5_trading_session context manager."""
+
+    def test_yields_connected_client_and_shuts_down(
+        self,
+        mocker: MockerFixture,
+    ) -> None:
+        """Test mt5_trading_session connects, yields a client, and shuts down."""
+        mock_client = MagicMock()
+        trading_client = mocker.patch(
+            "mt5cli.trading.Mt5TradingClient",
+            return_value=mock_client,
+        )
+
+        with mt5_trading_session(
+            build_config(path="/opt/mt5/terminal64.exe"),
+            retry_count=2,
+        ) as client:
+            mock_client.initialize_and_login_mt5.assert_called_once()
+            assert client is mock_client
+
+        trading_client.assert_called_once()
+        assert trading_client.call_args.kwargs["retry_count"] == 2
+        assert (
+            trading_client.call_args.kwargs["config"].path == "/opt/mt5/terminal64.exe"
+        )
+        mock_client.shutdown.assert_called_once()
+
+    def test_shuts_down_when_initialize_raises(
+        self,
+        mocker: MockerFixture,
+    ) -> None:
+        """Test shutdown is called when initialization fails."""
+        mock_client = MagicMock()
+        mock_client.initialize_and_login_mt5.side_effect = Mt5RuntimeError("boom")
+        mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
+
+        with pytest.raises(Mt5RuntimeError, match="boom"), mt5_trading_session():
+            pass
+
+        mock_client.shutdown.assert_called_once()
+
+    def test_shuts_down_when_body_raises(self, mocker: MockerFixture) -> None:
+        """Test shutdown is called when the context body raises."""
+        mock_client = MagicMock()
+        mocker.patch("mt5cli.trading.Mt5TradingClient", return_value=mock_client)
+
+        body_error = "body error"
+        with pytest.raises(RuntimeError, match=body_error), mt5_trading_session():
+            raise RuntimeError(body_error)
+
+        mock_client.shutdown.assert_called_once()

tests/test_utils.py

@@ -0,0 +1,480 @@
+"""Tests for mt5cli.utils module."""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from datetime import UTC, datetime
+from typing import TYPE_CHECKING
+
+import pandas as pd
+import pytest
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+from mt5cli.utils import (
+    DATETIME_TYPE,
+    REQUEST_TYPE,
+    TICK_FLAG_MAP,
+    TICK_FLAGS_TYPE,
+    TIMEFRAME_MAP,
+    TIMEFRAME_TYPE,
+    Dataset,
+    IfExists,
+    detect_format,
+    export_dataframe,
+    export_dataframe_to_sqlite,
+    parse_datetime,
+    parse_request,
+    parse_tick_flags,
+    parse_timeframe,
+)
+
+# ---------------------------------------------------------------------------
+# detect_format
+# ---------------------------------------------------------------------------
+
+
+class TestDetectFormat:
+    """Tests for detect_format."""
+
+    def test_explicit_format_returned(self, tmp_path: Path) -> None:
+        """Test that explicit format overrides extension."""
+        result = detect_format(tmp_path / "data.txt", explicit_format="csv")
+        assert result == "csv"
+
+    @pytest.mark.parametrize(
+        ("filename", "expected"),
+        [
+            ("data.csv", "csv"),
+            ("data.json", "json"),
+            ("data.parquet", "parquet"),
+            ("data.pq", "parquet"),
+            ("data.db", "sqlite3"),
+            ("data.sqlite", "sqlite3"),
+            ("data.sqlite3", "sqlite3"),
+            ("DATA.CSV", "csv"),
+            ("DATA.JSON", "json"),
+            ("DATA.PARQUET", "parquet"),
+        ],
+    )
+    def test_auto_detect_from_extension(
+        self,
+        tmp_path: Path,
+        filename: str,
+        expected: str,
+    ) -> None:
+        """Test format auto-detection from file extension."""
+        result = detect_format(tmp_path / filename)
+        assert result == expected
+
+    def test_unknown_extension_raises(self, tmp_path: Path) -> None:
+        """Test that unknown extension raises ValueError."""
+        with pytest.raises(ValueError, match="Cannot detect format"):
+            detect_format(tmp_path / "data.xyz")
+
+
+# ---------------------------------------------------------------------------
+# export_dataframe
+# ---------------------------------------------------------------------------
+
+
+class TestExportDataframe:
+    """Tests for export_dataframe."""
+
+    @pytest.fixture
+    def sample_df(self) -> pd.DataFrame:
+        """Create a sample DataFrame for testing."""
+        return pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
+
+    def test_export_csv(self, tmp_path: Path, sample_df: pd.DataFrame) -> None:
+        """Test CSV export."""
+        output = tmp_path / "out.csv"
+        export_dataframe(sample_df, output, "csv")
+        result = pd.read_csv(output)
+        pd.testing.assert_frame_equal(result, sample_df)
+
+    def test_export_json(self, tmp_path: Path, sample_df: pd.DataFrame) -> None:
+        """Test JSON export."""
+        output = tmp_path / "out.json"
+        export_dataframe(sample_df, output, "json")
+        with output.open() as f:
+            records = json.load(f)
+        assert len(records) == 3
+        assert records[0]["a"] == 1
+
+    def test_export_parquet(self, tmp_path: Path, sample_df: pd.DataFrame) -> None:
+        """Test Parquet export."""
+        output = tmp_path / "out.parquet"
+        export_dataframe(sample_df, output, "parquet")
+        result = pd.read_parquet(output)
+        pd.testing.assert_frame_equal(result, sample_df)
+
+    def test_export_sqlite3(self, tmp_path: Path, sample_df: pd.DataFrame) -> None:
+        """Test SQLite3 export."""
+        output = tmp_path / "out.db"
+        export_dataframe(sample_df, output, "sqlite3", table_name="test_table")
+        with sqlite3.connect(output) as conn:
+            result = pd.read_sql(  # type: ignore[reportUnknownMemberType]
+                "SELECT * FROM test_table",
+                conn,
+            )
+        pd.testing.assert_frame_equal(result, sample_df)
+
+    def test_unsupported_format_raises(
+        self,
+        tmp_path: Path,
+        sample_df: pd.DataFrame,
+    ) -> None:
+        """Test that unsupported format raises ValueError."""
+        with pytest.raises(ValueError, match="Unsupported output format"):
+            export_dataframe(sample_df, tmp_path / "out.txt", "xml")
+
+
+class TestExportDataframeToSqlite:
+    """Tests for export_dataframe_to_sqlite."""
+
+    def test_append_preserves_existing_rows(self, tmp_path: Path) -> None:
+        """Test append mode keeps prior rows in the SQLite table."""
+        output = tmp_path / "append.db"
+        first = pd.DataFrame({"id": [1], "value": ["a"]})
+        second = pd.DataFrame({"id": [2], "value": ["b"]})
+        export_dataframe_to_sqlite(first, output, "items", if_exists=IfExists.REPLACE)
+        export_dataframe_to_sqlite(second, output, "items", if_exists=IfExists.APPEND)
+        with sqlite3.connect(output) as conn:
+            result = pd.read_sql(  # type: ignore[reportUnknownMemberType]
+                "SELECT id, value FROM items ORDER BY id",
+                conn,
+            )
+        pd.testing.assert_frame_equal(
+            result,
+            pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
+        )
+
+    def test_deduplicate_keeps_latest_row(self, tmp_path: Path) -> None:
+        """Test deduplication keeps the latest ROWID for key columns."""
+        output = tmp_path / "dedup.db"
+        first = pd.DataFrame({
+            "symbol": ["EURUSD", "EURUSD"],
+            "time": ["2024-01-01", "2024-01-01"],
+            "bid": [1.0, 1.1],
+        })
+        second = pd.DataFrame({
+            "symbol": ["EURUSD"],
+            "time": ["2024-01-01"],
+            "bid": [1.2],
+        })
+        export_dataframe_to_sqlite(
+            first,
+            output,
+            "ticks",
+            if_exists=IfExists.REPLACE,
+            deduplicate_on=("symbol", "time"),
+        )
+        export_dataframe_to_sqlite(
+            second,
+            output,
+            "ticks",
+            if_exists=IfExists.APPEND,
+            deduplicate_on=("symbol", "time"),
+        )
+        with sqlite3.connect(output) as conn:
+            result = pd.read_sql(  # type: ignore[reportUnknownMemberType]
+                "SELECT symbol, time, bid FROM ticks",
+                conn,
+            )
+        pd.testing.assert_frame_equal(
+            result.reset_index(drop=True),
+            pd.DataFrame({
+                "symbol": ["EURUSD"],
+                "time": ["2024-01-01"],
+                "bid": [1.2],
+            }),
+        )
+
+    def test_default_if_exists_appends_without_dropping_rows(
+        self,
+        tmp_path: Path,
+    ) -> None:
+        """Test the default append mode keeps prior rows."""
+        output = tmp_path / "default-append.db"
+        first = pd.DataFrame({"id": [1], "value": ["a"]})
+        second = pd.DataFrame({"id": [2], "value": ["b"]})
+        export_dataframe_to_sqlite(first, output, "items")
+        export_dataframe_to_sqlite(second, output, "items")
+        with sqlite3.connect(output) as conn:
+            result = pd.read_sql(  # type: ignore[reportUnknownMemberType]
+                "SELECT id, value FROM items ORDER BY id",
+                conn,
+            )
+        pd.testing.assert_frame_equal(
+            result,
+            pd.DataFrame({"id": [1, 2], "value": ["a", "b"]}),
+        )
+
+    def test_writes_index_with_label(self, tmp_path: Path) -> None:
+        """Test optional index export with a custom label."""
+        output = tmp_path / "index.db"
+        frame = pd.DataFrame(
+            {"value": [1.0]}, index=pd.Index(["EURUSD"], name="symbol")
+        )
+        export_dataframe_to_sqlite(
+            frame,
+            output,
+            "margins",
+            if_exists=IfExists.REPLACE,
+            index=True,
+            index_label="symbol",
+        )
+        with sqlite3.connect(output) as conn:
+            result = pd.read_sql(  # type: ignore[reportUnknownMemberType]
+                "SELECT symbol, value FROM margins",
+                conn,
+            )
+        pd.testing.assert_frame_equal(
+            result,
+            pd.DataFrame({"symbol": ["EURUSD"], "value": [1.0]}),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Parse helpers
+# ---------------------------------------------------------------------------
+
+
+class TestParseDatetime:
+    """Tests for parse_datetime."""
+
+    def test_valid_date(self) -> None:
+        """Test parsing a date string."""
+        result = parse_datetime("2024-01-15")
+        assert result == datetime(2024, 1, 15, tzinfo=UTC)
+
+    def test_valid_datetime_with_tz(self) -> None:
+        """Test parsing a datetime with timezone."""
+        result = parse_datetime("2024-01-15T12:00:00+00:00")
+        assert result == datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC)
+
+    def test_invalid_format_raises(self) -> None:
+        """Test that invalid format raises ValueError."""
+        with pytest.raises(ValueError, match="Invalid datetime"):
+            parse_datetime("not-a-date")
+
+
+class TestParseTimeframe:
+    """Tests for parse_timeframe."""
+
+    @pytest.mark.parametrize(
+        ("value", "expected"),
+        [("M1", 1), ("h1", 16385), ("D1", 16408), ("MN1", 49153)],
+    )
+    def test_named_timeframe(self, value: str, expected: int) -> None:
+        """Test parsing named timeframes."""
+        assert parse_timeframe(value) == expected
+
+    def test_integer_timeframe(self) -> None:
+        """Test parsing supported integer timeframes."""
+        assert parse_timeframe("1") == 1
+        assert parse_timeframe(16385) == 16385
+
+    def test_unsupported_integer_timeframe_raises(self) -> None:
+        """Test that unsupported integer timeframes raise ValueError."""
+        with pytest.raises(ValueError, match="Invalid timeframe"):
+            parse_timeframe("42")
+
+    def test_invalid_timeframe_raises(self) -> None:
+        """Test that invalid timeframe raises ValueError."""
+        with pytest.raises(ValueError, match="Invalid timeframe"):
+            parse_timeframe("INVALID")
+
+
+class TestParseTickFlags:
+    """Tests for parse_tick_flags."""
+
+    @pytest.mark.parametrize(
+        ("value", "expected"),
+        [("ALL", -1), ("info", 1), ("TRADE", 2), ("COPY_TICKS_ALL", -1)],
+    )
+    def test_named_flag(self, value: str, expected: int) -> None:
+        """Test parsing named tick flags."""
+        assert parse_tick_flags(value) == expected
+
+    def test_integer_flag(self) -> None:
+        """Test parsing supported integer tick flags."""
+        assert parse_tick_flags("-1") == -1
+        assert parse_tick_flags(2) == 2
+
+    def test_unsupported_integer_flag_raises(self) -> None:
+        """Test that unsupported integer tick flags raise ValueError."""
+        with pytest.raises(ValueError, match="Invalid tick flags"):
+            parse_tick_flags("7")
+
+    def test_invalid_flag_raises(self) -> None:
+        """Test that invalid flag raises ValueError."""
+        with pytest.raises(ValueError, match="Invalid tick flags"):
+            parse_tick_flags("INVALID")
+
+
+# ---------------------------------------------------------------------------
+# parse_request
+# ---------------------------------------------------------------------------
+
+
+class TestParseRequest:
+    """Tests for parse_request."""
+
+    def test_inline_json(self) -> None:
+        """Test parsing an inline JSON object string."""
+        result = parse_request('{"action": 1, "symbol": "EURUSD"}')
+        assert result == {"action": 1, "symbol": "EURUSD"}
+
+    def test_file_reference(self, tmp_path: Path) -> None:
+        """Test parsing JSON from a file via the @path syntax."""
+        path = tmp_path / "req.json"
+        path.write_text('{"action": 2}', encoding="utf-8")
+        result = parse_request(f"@{path}")
+        assert result == {"action": 2}
+
+    def test_invalid_json_raises(self) -> None:
+        """Test that invalid JSON raises ValueError."""
+        with pytest.raises(ValueError, match="Invalid JSON request"):
+            parse_request("not json")
+
+    def test_non_object_raises(self) -> None:
+        """Test that a non-object JSON raises ValueError."""
+        with pytest.raises(ValueError, match="must be a JSON object"):
+            parse_request("[1, 2, 3]")
+
+    def test_missing_file_raises(self, tmp_path: Path) -> None:
+        """Test that a missing request file raises ValueError."""
+        path = tmp_path / "missing.json"
+        with pytest.raises(ValueError, match="Failed to read JSON request file"):
+            parse_request(f"@{path}")
+
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+
+class TestConstants:
+    """Tests for module constants."""
+
+    def test_timeframe_map_has_expected_keys(self) -> None:
+        """Test that TIMEFRAME_MAP contains standard timeframes."""
+        for key in ("M1", "M5", "M15", "M30", "H1", "H4", "D1", "W1", "MN1"):
+            assert key in TIMEFRAME_MAP
+
+    def test_tick_flag_map_has_expected_keys(self) -> None:
+        """Test that TICK_FLAG_MAP contains standard flags with MT5 values."""
+        assert {"ALL", "INFO", "TRADE"} <= set(TICK_FLAG_MAP)
+        assert TICK_FLAG_MAP["ALL"] == -1
+        assert TICK_FLAG_MAP["INFO"] == 1
+        assert TICK_FLAG_MAP["TRADE"] == 2
+
+    @pytest.mark.parametrize(
+        ("dataset", "expected"),
+        [
+            (Dataset.rates, "rates"),
+            (Dataset.ticks, "ticks"),
+            (Dataset.history_orders, "history_orders"),
+            (Dataset.history_deals, "history_deals"),
+        ],
+    )
+    def test_dataset_table_name(self, dataset: Dataset, expected: str) -> None:
+        """Test dataset SQLite table names."""
+        assert dataset.table_name == expected
+
+
+# ---------------------------------------------------------------------------
+# Click ParamTypes
+# ---------------------------------------------------------------------------
+
+
+class TestDateTimeType:
+    """Tests for _DateTimeType."""
+
+    def test_convert_string(self) -> None:
+        """Test converting a string to datetime."""
+        result = DATETIME_TYPE.convert("2024-06-15", None, None)
+        assert result == datetime(2024, 6, 15, tzinfo=UTC)
+
+    def test_convert_datetime_passthrough(self) -> None:
+        """Test that datetime values pass through unchanged."""
+        dt = datetime(2024, 1, 1, tzinfo=UTC)
+        assert DATETIME_TYPE.convert(dt, None, None) is dt
+
+    def test_convert_invalid(self) -> None:
+        """Test that invalid values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid datetime"):
+            DATETIME_TYPE.convert("bad", None, None)
+
+
+class TestTimeframeType:
+    """Tests for _TimeframeType."""
+
+    def test_convert_string(self) -> None:
+        """Test converting a string to timeframe integer."""
+        assert TIMEFRAME_TYPE.convert("H1", None, None) == 16385
+
+    def test_convert_int(self) -> None:
+        """Test converting supported integer timeframe values."""
+        assert TIMEFRAME_TYPE.convert(16385, None, None) == 16385
+
+    def test_convert_unsupported_int(self) -> None:
+        """Test that unsupported integer values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid timeframe"):
+            TIMEFRAME_TYPE.convert(42, None, None)
+
+    def test_convert_invalid(self) -> None:
+        """Test that invalid values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid timeframe"):
+            TIMEFRAME_TYPE.convert("bad", None, None)
+
+    @pytest.mark.parametrize("value", [True, False, None, 1.5])
+    def test_convert_invalid_types(self, value: object) -> None:
+        """Test that bool, float, and None values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid timeframe"):
+            TIMEFRAME_TYPE.convert(value, None, None)
+
+
+class TestTickFlagsType:
+    """Tests for _TickFlagsType."""
+
+    def test_convert_string(self) -> None:
+        """Test converting a string to tick flags integer."""
+        assert TICK_FLAGS_TYPE.convert("ALL", None, None) == -1
+
+    def test_convert_int(self) -> None:
+        """Test converting supported integer tick flag values."""
+        assert TICK_FLAGS_TYPE.convert(2, None, None) == 2
+
+    def test_convert_unsupported_int(self) -> None:
+        """Test that unsupported integer values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid tick flags"):
+            TICK_FLAGS_TYPE.convert(7, None, None)
+
+    @pytest.mark.parametrize("value", [True, False, None, 1.5])
+    def test_convert_invalid_types(self, value: object) -> None:
+        """Test that bool, float, and None values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid tick flags"):
+            TICK_FLAGS_TYPE.convert(value, None, None)
+
+    def test_convert_invalid(self) -> None:
+        """Test that invalid values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid tick flags"):
+            TICK_FLAGS_TYPE.convert("bad", None, None)
+
+
+class TestRequestType:
+    """Tests for _RequestType."""
+
+    def test_convert_string(self) -> None:
+        """Test converting a JSON string to a request dictionary."""
+        assert REQUEST_TYPE.convert('{"action": 1}', None, None) == {"action": 1}
+
+    def test_convert_invalid(self) -> None:
+        """Test that invalid values raise BadParameter."""
+        with pytest.raises(Exception, match="Invalid JSON request"):
+            REQUEST_TYPE.convert("bad", None, None)

uv.lock

@@ -234,11 +234,11 @@ wheels = [
 
 [[package]]
 name = "idna"
-version = "3.13"
+version = "3.15"
 source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ce/cc/762dfb036166873f0059f3b7de4565e1b5bc3d6f28a414c13da27e442f99/idna-3.13.tar.gz", hash = "sha256:585ea8fe5d69b9181ec1afba340451fba6ba764af97026f92a91d4eef164a242", size = 194210, upload-time = "2026-04-22T16:42:42.314Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/82/77/7b3966d0b9d1d31a36ddf1746926a11dface89a83409bf1483f0237aa758/idna-3.15.tar.gz", hash = "sha256:ca962446ea538f7092a95e057da437618e886f4d349216d2b1e294abfdb65fdc", size = 199245, upload-time = "2026-05-12T22:45:57.011Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/5d/13/ad7d7ca3808a898b4612b6fe93cde56b53f3034dcde235acb1f0e1df24c6/idna-3.13-py3-none-any.whl", hash = "sha256:892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3", size = 68629, upload-time = "2026-04-22T16:42:40.909Z" },
+    { url = "https://files.pythonhosted.org/packages/d2/23/408243171aa9aaba178d3e2559159c24c1171a641aa83b67bdd3394ead8e/idna-3.15-py3-none-any.whl", hash = "sha256:048adeaf8c2d788c40fee287673ccaa74c24ffd8dcf09ffa555a2fbb59f10ac8", size = 72340, upload-time = "2026-05-12T22:45:55.733Z" },
 ]
 
 [[package]]
@@ -487,7 +487,7 @@ wheels = [
 
 [[package]]
 name = "mt5cli"
-version = "0.2.0"
+version = "0.7.2"
 source = { editable = "." }
 dependencies = [
     { name = "click" },
@@ -513,7 +513,7 @@ dev = [
 [package.metadata]
 requires-dist = [
     { name = "click", specifier = ">=8.1.0" },
-    { name = "pdmt5", specifier = ">=0.2.3" },
+    { name = "pdmt5", specifier = ">=0.3.0" },
     { name = "pyarrow", specifier = ">=19.0.0" },
     { name = "typer", specifier = ">=0.15.0" },
 ]
@@ -684,16 +684,16 @@ wheels = [
 
 [[package]]
 name = "pdmt5"
-version = "0.2.3"
+version = "0.3.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "metatrader5", marker = "sys_platform == 'win32'" },
     { name = "pandas" },
     { name = "pydantic" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/02/25/52d9d954504ccdd0fe91f715ab74c424d61234b237cc4160d3ebe20070f1/pdmt5-0.2.3.tar.gz", hash = "sha256:21384f5826fb0125fee3f93c90b108340f55ab53b1c819d229ceac162289d2ec", size = 226665, upload-time = "2026-02-05T13:28:21.071Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/bf/cc/c8fa3a01e0e34178fec8527992f7bb8eda5881477ce23aaacaa9b2ef7bec/pdmt5-0.3.0.tar.gz", hash = "sha256:bb612d5c2695eafac9b2a7b74756e13bd383d7e5517bd90c9a2efa92492c484c", size = 215100, upload-time = "2026-06-11T13:26:46.976Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/c1/75/c5e52a9cf459b85b2dd52f83e70857571b1b45805c9fe610b3959a26ac15/pdmt5-0.2.3-py3-none-any.whl", hash = "sha256:f92246a05cfc3b7feb3ab0cc5b48768a4d84aad6b02e7a68060948f5828718a1", size = 22967, upload-time = "2026-02-05T13:28:19.523Z" },
+    { url = "https://files.pythonhosted.org/packages/f2/03/b12cc4c9db983d971c9172b3765161b6d91136d0624e6718a04dd815e7a1/pdmt5-0.3.0-py3-none-any.whl", hash = "sha256:5388b406cc583202600cfe22c9d781679b1d931b1ed5a2b5dcf37c566149b49f", size = 26250, upload-time = "2026-06-11T13:26:45.689Z" },
 ]
 
 [[package]]
@@ -836,24 +836,24 @@ wheels = [
 
 [[package]]
 name = "pygments"
-version = "2.19.2"
+version = "2.20.0"
 source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
+    { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
 ]
 
 [[package]]
 name = "pymdown-extensions"
-version = "10.21.2"
+version = "10.21.3"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "markdown" },
     { name = "pyyaml" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/df/08/f1c908c581fd11913da4711ea7ba32c0eee40b0190000996bb863b0c9349/pymdown_extensions-10.21.2.tar.gz", hash = "sha256:c3f55a5b8a1d0edf6699e35dcbea71d978d34ff3fa79f3d807b8a5b3fa90fbdc", size = 853922, upload-time = "2026-03-29T15:01:55.233Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/f7/27/a2fc51a4a122dfd1015e921ae9d22fee3d20b0b8080d9a704578bf9deece/pymdown_extensions-10.21.2-py3-none-any.whl", hash = "sha256:5c0fd2a2bea14eb39af8ff284f1066d898ab2187d81b889b75d46d4348c01638", size = 268901, upload-time = "2026-03-29T15:01:53.244Z" },
+    { url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
 ]
 
 [[package]]
@@ -1126,11 +1126,11 @@ wheels = [
 
 [[package]]
 name = "urllib3"
-version = "2.6.3"
+version = "2.7.0"
 source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556, upload-time = "2026-01-07T16:24:43.925Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" },
+    { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" },
 ]
 
 [[package]]