Add incremental SQLite history SDK (#16)

* Add incremental SQLite history SDK for automated pipelines.

Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental history deals and stale rate view cleanup.

Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental deal filtering edge cases

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for incremental SQLite history.

Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document collect-history SQLite schema with ER diagram.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix account-event filtering and drop legacy rates resume.

Account events must follow only account_event_start, not per-symbol trade
cursors. Require normalized rates schema and fail fast when timeframe is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Validate normalized rates schema before incremental resume.

Require symbol, timeframe, and time on existing rates tables with clear
ValueError messages, and add regression tests for malformed schemas.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Daichi Narushima
2026-06-09 01:28:22 +09:00
committed by GitHub
parent 5b44318d55
commit c4232bf44d
12 changed files with 3427 additions and 335 deletions
+2
View File
@@ -152,6 +152,8 @@ mt5cli -o history.db collect-history \
History orders and deals are fetched per symbol and concatenated, so the symbol filter is applied consistently across all datasets. The `cash_events` view is derived from symbol-filtered `history_deals`, so account-level cash events with empty or non-matching symbols may be excluded. The `positions_reconstructed` view excludes positions with no closing deal, uses volume-weighted open/close prices, and reports reversal deals (`DEAL_ENTRY_INOUT`) via `volume_reversal` / `reversal_count`.
See the [SQLite History schema diagram](api/sqlite_history.md#entity-relationship-diagram) for a sample ER layout of the resulting database.
## Global Options
| Option | Description |