History Collection (SQLite)¶
mt5cli.history ¶
SQLite storage helpers for the collect-history incremental data pipeline.
DEFAULT_HISTORY_TIMEFRAMES
module-attribute
¶
DEFAULT_HISTORY_TIMEFRAMES: tuple[str, ...] = tuple(
TIMEFRAME_MAP
)
RateTarget
dataclass
¶
A single rate series identified by symbol and timeframe.
Attributes:
| Name | Type | Description |
|---|---|---|
symbol |
str | None
|
MT5 symbol name, or None when the rate series is addressed only by an explicit table (for example a custom SQLite view). |
timeframe |
int | str
|
MT5 timeframe as an integer or name (for example |
__post_init__ ¶
Normalize accepted timeframe aliases to the stored integer value.
append_dataframe ¶
append_dataframe(
conn: Connection,
frame: DataFrame,
table_name: str,
if_exists: IfExists,
) -> bool
Append a DataFrame to SQLite when it has a schema.
Returns:
| Type | Description |
|---|---|
bool
|
True if a table was written, False if the frame had no columns. |
Source code in mt5cli/history.py
augment_written_columns_from_sqlite ¶
augment_written_columns_from_sqlite(
conn: Connection,
datasets: set[Dataset],
written_columns: dict[Dataset, set[str]],
) -> None
Add existing table columns to the written column map.
Source code in mt5cli/history.py
build_rate_targets ¶
build_rate_targets(
symbols: Sequence[str],
timeframes: Sequence[int | str],
*,
allow_missing_symbol: bool = False,
) -> list[RateTarget]
Build rate targets for every symbol and timeframe combination.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
symbols
|
Sequence[str]
|
MT5 symbol names. May be empty when |
required |
timeframes
|
Sequence[int | str]
|
MT5 timeframes as integers or names (for example |
required |
allow_missing_symbol
|
bool
|
When True and |
False
|
Returns:
| Type | Description |
|---|---|
list[RateTarget]
|
Targets in row-major order: every timeframe for the first symbol, then |
list[RateTarget]
|
every timeframe for the next symbol, and so on. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in mt5cli/history.py
build_rate_view_name ¶
build_rate_view_name(
*,
symbol: str,
granularity: str,
granularity_count: int,
timeframe: int,
) -> str
Return a collision-free offline optimize view name.
View names always include the timeframe integer after a __ separator so
a symbol such as EURUSD_M1 cannot collide with EURUSD at timeframe
M1.
Source code in mt5cli/history.py
create_cash_events_view ¶
Create the cash_events SQLite view derived from history_deals.
Returns:
| Type | Description |
|---|---|
bool
|
True if the view was created, False if required columns are missing. |
Source code in mt5cli/history.py
create_history_indexes ¶
create_history_indexes(
conn: Connection,
written_columns: dict[Dataset, set[str]],
) -> None
Create useful indexes for collected history tables when present.
Source code in mt5cli/history.py
create_positions_reconstructed_view ¶
Create the positions_reconstructed SQLite view derived from history_deals.
Returns:
| Type | Description |
|---|---|
bool
|
True if the view was created, False if required columns are missing. |
Source code in mt5cli/history.py
create_rate_compatibility_views ¶
Create rate compatibility views from the normalized rates table.
Source code in mt5cli/history.py
deduplicate_history_tables ¶
deduplicate_history_tables(
conn: Connection,
written_columns: dict[Dataset, set[str]],
written_tables: set[Dataset],
dedup_scopes: dict[Dataset, list[DedupScope]]
| None = None,
) -> None
Deduplicate appended history tables by stable identifiers.
Source code in mt5cli/history.py
drop_duplicates_in_table ¶
drop_duplicates_in_table(
cursor: Cursor,
table: str,
ids: list[str],
*,
keep: Literal["first", "last"] = "last",
scope_where: str | None = None,
scope_params: tuple[object, ...] = (),
) -> None
Remove duplicate rows, keeping the first or last ROWID per key group.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the table or column names are invalid. |
Source code in mt5cli/history.py
drop_rate_compatibility_views ¶
Drop all mt5cli-managed rate_* compatibility views.
Source code in mt5cli/history.py
filter_incremental_history_deals_frame ¶
filter_incremental_history_deals_frame(
frame: DataFrame,
symbols: Sequence[str],
start_by_symbol: dict[str, datetime],
account_event_start: datetime,
) -> DataFrame
Filter incrementally fetched history_deals by symbol and event start times.
Returns:
| Type | Description |
|---|---|
DataFrame
|
Rows for selected symbols at or after each symbol start, plus account |
DataFrame
|
events at or after |
Source code in mt5cli/history.py
filter_trade_history_frame ¶
filter_trade_history_frame(
frame: DataFrame,
symbols: Sequence[str],
*,
include_account_events: bool,
) -> DataFrame
Filter trade history rows to selected symbols and account events.
Returns:
| Type | Description |
|---|---|
DataFrame
|
Filtered history rows. |
Source code in mt5cli/history.py
get_history_deals_account_event_start_datetime ¶
get_history_deals_account_event_start_datetime(
conn: Connection, *, fallback_start: datetime
) -> datetime
Return the next update start for account-level history_deals rows.
Source code in mt5cli/history.py
get_incremental_start_datetime ¶
get_incremental_start_datetime(
conn: Connection,
dataset: Dataset,
*,
symbol: str,
timeframe: int | None,
fallback_start: datetime,
) -> datetime
Return the next update start datetime from existing MAX(time).
Source code in mt5cli/history.py
get_table_columns ¶
Return existing SQLite columns for a table.
Source code in mt5cli/history.py
load_incremental_start_datetimes ¶
load_incremental_start_datetimes(
conn: Connection,
dataset: Dataset,
*,
symbols: Sequence[str],
timeframes: Sequence[int] | None = None,
fallback_start: datetime,
) -> dict[tuple[str, int | None], datetime]
Return next update start datetimes keyed by symbol and optional timeframe.
Source code in mt5cli/history.py
load_rate_data ¶
load_rate_data(
conn_or_path: SqliteConnOrPath,
table: str,
count: int | None = None,
) -> DataFrame
Load rate-like data from a SQLite database path or connection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn_or_path
|
SqliteConnOrPath
|
SQLite database path or open connection. |
required |
table
|
str
|
Source table or view name. |
required |
count
|
int | None
|
Optional number of most recent rows to load. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame indexed by ascending |
Source code in mt5cli/history.py
load_rate_data_from_connection ¶
load_rate_data_from_connection(
connection: Connection,
table: str,
count: int | None = None,
) -> DataFrame
Load rate-like data from a SQLite table or view.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
Open SQLite connection. |
required |
table
|
str
|
Source table or view name. |
required |
count
|
int | None
|
Optional number of most recent rows to load. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame indexed by ascending |
Raises:
| Type | Description |
|---|---|
ValueError
|
If inputs, schema, timestamps are invalid, or the table or view contains no rows. |
Source code in mt5cli/history.py
load_rate_series_from_sqlite ¶
load_rate_series_from_sqlite(
conn_or_path: SqliteConnOrPath,
targets: Sequence[RateTarget],
count: int,
explicit_tables: Sequence[str] | None = None,
) -> dict[tuple[str | None, int], DataFrame]
Load multiple rate series from a SQLite database.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn_or_path
|
SqliteConnOrPath
|
SQLite database path or open connection. |
required |
targets
|
Sequence[RateTarget]
|
Rate targets to load. Each |
required |
count
|
int
|
Number of most recent rows to load per series. |
required |
explicit_tables
|
Sequence[str] | None
|
Optional explicit table or view names matching targets.
When omitted, managed |
None
|
Returns:
| Type | Description |
|---|---|
dict[tuple[str | None, int], DataFrame]
|
Mapping keyed by |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in mt5cli/history.py
parse_sqlite_timestamp ¶
Parse a SQLite history timestamp value.
Returns:
| Type | Description |
|---|---|
datetime | None
|
Parsed timezone-aware datetime, or None when parsing fails. |
Source code in mt5cli/history.py
quote_sqlite_identifier ¶
record_written_columns ¶
record_written_columns(
written_columns: dict[Dataset, set[str]],
dataset: Dataset,
frame: DataFrame,
) -> None
Remember columns for datasets written during collection.
Source code in mt5cli/history.py
resolve_granularity_name ¶
Return a granularity name for a timeframe integer when known.
resolve_history_datasets ¶
resolve_history_tick_flags ¶
Resolve tick copy flags from an integer or name.
Returns:
| Type | Description |
|---|---|
int
|
Integer tick flag value. |
resolve_history_timeframes ¶
Resolve rate timeframes, deduplicating aliases for the same integer.
Returns:
| Type | Description |
|---|---|
list[int]
|
Ordered list of unique timeframe integers. |
Source code in mt5cli/history.py
resolve_rate_tables ¶
resolve_rate_tables(
conn_or_path: SqliteConnOrPath | None,
targets: Sequence[RateTarget],
explicit_tables: Sequence[str] | None = None,
*,
require_existing: bool = False,
) -> list[str]
Resolve SQLite table or view names for rate targets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn_or_path
|
SqliteConnOrPath | None
|
SQLite database path or open connection. May be None when
|
required |
targets
|
Sequence[RateTarget]
|
Rate targets to resolve. |
required |
explicit_tables
|
Sequence[str] | None
|
Optional explicit table or view names. When provided, they are used as-is and must match the number of targets. |
None
|
require_existing
|
bool
|
When True, require the database and managed views to
exist for each symbol target. Ignored when |
False
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Table or view names aligned with |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in mt5cli/history.py
554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 | |
resolve_rate_view_name ¶
resolve_rate_view_name(
conn_or_path: SqliteConnOrPath | None,
symbol: str,
granularity: str,
*,
require_existing: bool = False,
) -> str
Resolve the mt5cli-managed rate compatibility view name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn_or_path
|
SqliteConnOrPath | None
|
SQLite database path or open connection. When None or a
non-existing path and |
required |
symbol
|
str
|
Symbol stored in the normalized |
required |
granularity
|
str
|
Timeframe name (for example |
required |
require_existing
|
bool
|
When True, require the database and a managed view to exist. |
False
|
Returns:
| Type | Description |
|---|---|
str
|
View name such as |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in mt5cli/history.py
resolve_rate_view_names ¶
resolve_rate_view_names(
conn_or_path: SqliteConnOrPath | None,
symbols: Sequence[str],
granularities: Sequence[str],
*,
require_existing: bool = False,
) -> list[str]
Resolve rate compatibility view names for symbol and granularity pairs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn_or_path
|
SqliteConnOrPath | None
|
SQLite database path or open connection. When None or a
non-existing path and |
required |
symbols
|
Sequence[str]
|
Symbols stored in the normalized |
required |
granularities
|
Sequence[str]
|
Timeframe names (for example |
required |
require_existing
|
bool
|
When True, require the database and managed views to exist. |
False
|
Returns:
| Type | Description |
|---|---|
list[str]
|
View names in row-major order: every |
list[str]
|
symbol, then every granularity for the next symbol, and so on. |
Source code in mt5cli/history.py
write_collected_datasets ¶
write_collected_datasets(
conn: Connection,
client: Mt5DataClient,
symbols: Sequence[str],
datasets: set[Dataset],
timeframe: int,
flags: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]
Collect selected datasets and stream each symbol frame into SQLite.
Returns:
| Type | Description |
|---|---|
tuple[set[Dataset], dict[Dataset, set[str]]]
|
Written datasets and their columns. |
Source code in mt5cli/history.py
1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 | |
write_history_dataset ¶
write_history_dataset(
conn: Connection,
fetch: Callable[..., DataFrame],
dataset: Dataset,
symbols: Sequence[str],
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
*,
include_account_events: bool = False,
) -> bool
Stream a history dataset into SQLite.
Returns:
| Type | Description |
|---|---|
bool
|
True if the target table was written. |
Source code in mt5cli/history.py
write_incremental_datasets ¶
write_incremental_datasets(
conn: Connection,
client: Mt5DataClient,
symbols: Sequence[str],
selected_datasets: set[Dataset],
resolved_timeframes: list[int],
resolved_tick_flags: int,
fallback_start: datetime,
end_date: datetime,
*,
deduplicate: bool,
create_rate_views: bool,
with_views: bool,
include_account_events: bool,
) -> tuple[set[Dataset], dict[Dataset, set[str]]]
Append selected datasets incrementally and refresh indexes and views.
Returns:
| Type | Description |
|---|---|
tuple[set[Dataset], dict[Dataset, set[str]]]
|
Written datasets and their columns. |
Source code in mt5cli/history.py
1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 | |
write_rates_dataset ¶
write_rates_dataset(
conn: Connection,
client: Mt5DataClient,
symbols: Sequence[str],
timeframe: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool
Stream rates frames into SQLite.
Returns:
| Type | Description |
|---|---|
bool
|
True if the rates table was written. |
Source code in mt5cli/history.py
write_streamed_frame ¶
write_streamed_frame(
conn: Connection,
frame: DataFrame,
dataset: Dataset,
table_exists: bool,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool
Write one streamed dataset frame and track table state.
Returns:
| Type | Description |
|---|---|
bool
|
True if the dataset table exists after this write attempt. |
Source code in mt5cli/history.py
write_ticks_dataset ¶
write_ticks_dataset(
conn: Connection,
client: Mt5DataClient,
symbols: Sequence[str],
flags: int,
date_from: datetime,
date_to: datetime,
if_exists: IfExists,
written_columns: dict[Dataset, set[str]],
) -> bool
Stream ticks frames into SQLite.
Returns:
| Type | Description |
|---|---|
bool
|
True if the ticks table was written. |
Source code in mt5cli/history.py
collect-history schema¶
The collect-history command (and the matching collect_history SDK function) writes
selected MT5 datasets into one SQLite database. Each dataset becomes a table; column
names and types mirror the pdmt5 DataFrame schema for that export, with two additions:
symbolis prepended on every table.timeframeis prepended onratesso appended runs at different bar sizes stay distinguishable.
SQLite does not declare foreign keys. Rows are linked logically by symbol, time
windows, and (for deals) position_id / order. Duplicate rows are removed on
append using dataset-specific keys (for example ticket on history tables, or
(symbol, timeframe, time) on rates).
Optional views are created when --with-views is set and the history-deals dataset
was written.
Entity-relationship diagram¶
Sample layout for a full collection with --with-views:
erDiagram
rates {
TEXT symbol "dedup key"
INTEGER timeframe "dedup key"
TEXT time "dedup key"
REAL open
REAL high
REAL low
REAL close
INTEGER tick_volume
INTEGER spread
INTEGER real_volume
}
ticks {
TEXT symbol "dedup key"
TEXT time "dedup key"
INTEGER time_msc "dedup key (preferred)"
REAL bid
REAL ask
REAL last
INTEGER volume
INTEGER flags
REAL volume_real
}
history_orders {
INTEGER ticket "dedup key"
TEXT symbol
TEXT time
INTEGER type
INTEGER state
REAL volume_initial
REAL price_open
REAL price_current
INTEGER magic
}
history_deals {
INTEGER ticket "dedup key"
INTEGER order
INTEGER position_id "groups position view"
TEXT symbol
TEXT time
INTEGER type "0/1 trade, else cash event"
INTEGER entry "0 IN, 1 OUT, 2 INOUT, 3 OUT_BY"
REAL volume
REAL price
REAL profit
REAL commission
REAL swap
REAL fee
}
cash_events {
INTEGER ticket
TEXT symbol
TEXT time
INTEGER type
REAL profit
}
positions_reconstructed {
INTEGER position_id
TEXT symbol
TEXT open_time
TEXT close_time
INTEGER direction
REAL volume_open
REAL volume_close
REAL volume_reversal
REAL open_price
REAL close_price
REAL total_profit
INTEGER reversal_count
INTEGER deals_count
}
rates ||--o{ history_deals : "symbol (logical)"
ticks ||--o{ history_deals : "symbol (logical)"
history_orders ||--o{ history_deals : "order ~ ticket (logical)"
history_deals ||--|| cash_events : "VIEW: type NOT IN (0,1)"
history_deals ||--o{ positions_reconstructed : "VIEW: GROUP BY position_id"
Tables and views¶
| Object | Kind | Source | Notes |
|---|---|---|---|
rates |
table | copy_rates_range |
Indexed on (symbol, timeframe, time) when columns exist. |
ticks |
table | copy_ticks_range |
Indexed on (symbol, time) when columns exist. |
history_orders |
table | history_orders_get |
Fetched per --symbol, then concatenated. |
history_deals |
table | history_deals_get |
Fetched per --symbol, then concatenated. Indexed on (position_id, symbol) when present. |
cash_events |
view | history_deals |
Non-trade deal types (deposits, balance ops, etc.). Requires type column. |
positions_reconstructed |
view | history_deals |
One row per closed position_id; volume-weighted prices and reversal stats. |
Column sets can vary with terminal and pdmt5 version. Views are skipped with a warning when required columns are missing.
Incremental collection¶
The update_history SDK path uses the same base tables and optional
cash_events / positions_reconstructed views. It additionally maintains
rate_<symbol>__<timeframe> compatibility views when create_rate_views=True.
Rate view resolution¶
Downstream tools can resolve mt5cli-managed compatibility view names from an existing SQLite history database without creating files or guessing legacy naming schemes:
from pathlib import Path
from mt5cli.history import resolve_rate_view_name, resolve_rate_view_names
# Single symbol and granularity
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1")
# Batch resolution in row-major order
views = resolve_rate_view_names(
Path("history.db"),
["EURUSD", "GBPUSD"],
["M1", "H1"],
)
Resolution rules:
- Returns
rate_<symbol>__<timeframe>when a symbol stores one timeframe. - Returns
rate_<symbol>__<granularity>_<timeframe>when multiple timeframes are stored for the same symbol. - When multiple naming candidates apply, prefers an existing managed
rate_*__*view from the candidate list. - Falls back to single-timeframe naming when the database path is missing or
ratesmetadata is unavailable. - Pass
require_existing=Trueto raiseValueErrorinstead of returning a best-guess name when the database or view is missing. - Accepts either a SQLite path or an open
sqlite3.Connection.
Rate data loading¶
Use load_rate_data() to load a table or view from a SQLite path, or
load_rate_data_from_connection() when you already have a connection:
from pathlib import Path
from mt5cli import load_rate_data
from mt5cli.history import resolve_rate_view_name
view = resolve_rate_view_name(Path("history.db"), "EURUSD", "M1", require_existing=True)
rates = load_rate_data(Path("history.db"), view, count=1000)
The loader accepts close-based OHLC rate data or tick-like bid/ask data. It
validates that time exists, parses timestamps with pandas, and returns a
DataFrame indexed by ascending DatetimeIndex named time.
Multi-series rate loading¶
For loading many rate series at once, build neutral RateTarget pairs and load
them from SQLite in one call. View names are resolved via the same
compatibility-view rules, or you can pass explicit_tables to bypass resolution:
from pathlib import Path
from mt5cli import build_rate_targets, load_rate_series_from_sqlite
targets = build_rate_targets(["EURUSD", "GBPUSD"], ["M1", "H1"])
series = load_rate_series_from_sqlite(Path("history.db"), targets, count=1000)
frame = series["EURUSD", 1] # keyed by (symbol, integer timeframe)
build_rate_targets()returnsRateTarget(symbol, timeframe)pairs in row-major order, normalizing timeframe names such as"M1"to their integer values; setallow_missing_symbol=Trueto address series solely byexplicit_tables(targets carrysymbol=None).resolve_rate_tables()maps targets to table or view names and validates that anyexplicit_tablescount matches the target count. Passrequire_existing=Trueto raiseValueErrorinstead of returning a best-guess name when the database or managed view is missing. Whenexplicit_tablesis provided, names are returned as-is andrequire_existingis ignored.load_rate_series_from_sqlite()returns a mapping keyed by(symbol, integer timeframe). Unlessexplicit_tablesis supplied, it requires existing managedrate_*compatibility views and raisesValueErrorwhen they are missing. Duplicate(symbol, timeframe)targets are rejected.