Add incremental SQLite history SDK (#16)

* Add incremental SQLite history SDK for automated pipelines.

Extract sqlite history helpers into a dedicated module and expose update_history APIs that resume from existing MAX(time) values instead of re-fetching fixed date ranges.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental history deals and stale rate view cleanup.

Fetch account events once during incremental updates, drop stale rate_* views when timeframes change, and avoid SQLite variable limits on wide frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix incremental deal filtering edge cases

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback for incremental SQLite history.

Make rate views collision-free, batch incremental resume queries, scope deduplication to appended boundaries, validate before opening MT5, use atomic SQLite transactions, and expand docs/tests for the new helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Document collect-history SQLite schema with ER diagram.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix account-event filtering and drop legacy rates resume.

Account events must follow only account_event_start, not per-symbol trade
cursors. Require normalized rates schema and fail fast when timeframe is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Validate normalized rates schema before incremental resume.

Require symbol, timeframe, and time on existing rates tables with clear
ValueError messages, and add regression tests for malformed schemas.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Daichi Narushima
2026-06-09 01:28:22 +09:00
committed by GitHub
parent 5b44318d55
commit c4232bf44d
12 changed files with 3427 additions and 335 deletions
+3 -3
View File
@@ -1089,7 +1089,7 @@ class TestCollectHistory:
assert all(row[0] not in {0, 1} for row in cash)
# Position 100 (BUY 1@1.10 + BUY 3@1.20 then SELL 4@1.50) is closed.
# Position 200 (BUY 2@2.00 then SELL 2@2.20) is closed.
# Position 300 (open-only) and 400 (reversal-only) are excluded.
# Position 400 (reversal-only with non-trade deal type) stays excluded.
assert set(positions) == {100, 200, 500, 600}
pos_100 = positions[100]
tol = 1e-9
@@ -1106,10 +1106,10 @@ class TestCollectHistory:
assert abs(pos_500[5] - 1.05) < tol
pos_600 = positions[600]
assert abs(pos_600[1] - 3.0) < tol
assert abs(pos_600[2] - 3.0) < tol
assert abs(pos_600[2] - 4.0) < tol # reversal + close volumes
assert abs(pos_600[3] - 1.0) < tol
assert abs(pos_600[4] - 1.10) < tol
assert abs(pos_600[5] - 1.40) < tol
assert abs(pos_600[5] - 3.5475) < tol
assert pos_600[6] == 1
def test_collect_history_filters_history_symbols_exactly(