feat: per-TF USDCHF consolidation + H1 optimizer recovery + MT5 data guide

- data_consolidator: add consolidate_usdchf_m15/m5 producing USDCHF_master_M15.csv
  and USDCHF_master_M5.csv; H1 legacy fallback now excludes M5/M15 filenames
- processor: _USDCHF_PATH_BY_TF lookup so each TF loads its matching-frequency
  USDCHF master (H1→master.csv, M15→master_M15.csv, M5→master_M5.csv)
- main: cmd_consolidate runs all three TF consolidations in one pass with per-TF summary
- optimizer: H1 n_states reverted [3,4], obs_cov floor raised 0.1→0.5,
  MIN_OOS_TRADES_BY_TF["H1"] lowered 200→75
- risk_manager: H1 headway_cent small-account daily cap raised 1→2 trades
- README: per-TF USDCHF filename/master table; dedicated "Getting deep history from MT5"
  section (any symbol, any TF) with MetaQuotes-Demo Pro Tip
This commit is contained in:
Luke Mosoti
2026-04-12 21:40:54 +03:00
parent 29d42b44fd
commit b6b058927e
6 changed files with 242 additions and 90 deletions
+41 -6
View File
@@ -175,6 +175,19 @@ The system requires historical OHLCV CSV files exported from MetaTrader5. In MT5
> The system works with H1, M15 and M5 timeframes. More data = better optimization. Aim for at least 2 years of history; the included H1 dataset covers 2004–2025.
### Getting deep history from MT5 (any symbol, any timeframe)
Since you are already using MT5, you can download historical data directly — provided you adjust your terminal settings to allow for 10 years of bars.
**Action Required:**
- Go to **Tools → Options → Charts**. Change **"Max bars in chart"** to a very high number (e.g., `99,999,999` or just type `unlimited`).
- Go to **View → Symbols** (or `Ctrl+U`). Search for the symbol you want (e.g., **XAUUSD** or **USDCHF**).
- Select the **Bars** tab. Choose your timeframe (M5, M15, H1, etc.) and click **Request**.
- Keep clicking **"Request"** or scroll back the chart until the desired start year is reached.
- Once loaded, click **Export Bars** to save as a `.csv`.
> **Pro Tip:** If your broker (Headway) doesn't have 10 years of history, open a free **MetaQuotes-Demo** account inside MT5. MetaQuotes usually provides much deeper history than individual brokers.
### Keep your data fresh
The processor filters to the **last 10 years anchored at the end of your CSV** (not today's date). If your CSV ends in December 2025, the 20% OOS window closes at December 2025 — but `sync_validate` pulls **live MT5 data up to today**. Any months after your CSV ends will be out-of-distribution for the model.
@@ -183,15 +196,29 @@ The processor filters to the **last 10 years anchored at the end of your CSV** (
### USDCHF — cross-asset USD strength feature (optional, recommended)
USDCHF is used as an intraday DXY proxy (high correlation, natively available on Headway). To enable the `usdchf_log_return` feature:
USDCHF is used as an intraday DXY proxy (high correlation, natively available on Headway). Each trading timeframe uses a **matching-frequency USDCHF master file** so bars are aligned during feature merging.
#### File naming convention
| Trading TF | Source file in `data/raw/` | Master produced |
|------------|---------------------------|-----------------|
| H1 | `USDCHF_H1.csv` | `data/processed/USDCHF_master.csv` |
| M15 | `USDCHF_M15_<dates>.csv` | `data/processed/USDCHF_master_M15.csv` |
| M5 | `USDCHF_M5_<dates>.csv` | `data/processed/USDCHF_master_M5.csv` |
The date range in the filename is optional — anything matching `USDCHF_M15_*.csv` or `USDCHF_M5_*.csv` is picked up automatically.
#### How to get USDCHF data free from MT5
Follow the same deep-history export process described in [Getting deep history from MT5](#getting-deep-history-from-mt5-any-symbol-any-timeframe) above — just search for **USDCHF** instead of XAUUSD and select the matching timeframe (M5, M15, or H1). Save each file to `data/raw/` using the naming convention in the table above.
#### Build the master files
1. In MT5, export **USDCHF** CSV files (any timeframe, same semicolon format as XAUUSD)
2. Save to `data/raw/` — any filename containing `USDCHF` works (e.g. `USDCHF_2024.csv`)
3. Run the consolidator:
```bash
python main.py --mode consolidate
```
This produces `data/processed/USDCHF_master.csv`. The pipeline adds it automatically as a 5th feature. If absent, all timeframes degrade gracefully to the 4-feature model.
This runs all three consolidations in one pass and prints a summary. The pipeline adds USDCHF automatically as a 5th feature per timeframe. If a master is absent for a given TF, that TF degrades gracefully to the 4-feature model.
---
@@ -234,7 +261,15 @@ LIVE_BALANCE=15
python main.py --mode consolidate
```
Merges all `*USDCHF*.csv` files in `data/raw/` into `data/processed/USDCHF_master.csv`. Run once before processing. Skip if you have no USDCHF data.
Builds three per-TF master files from the matching source files in `data/raw/`:
| TF | Source | Output |
|----|--------|--------|
| H1 | `USDCHF_H1.csv` | `data/processed/USDCHF_master.csv` |
| M15 | `USDCHF_M15_*.csv` | `data/processed/USDCHF_master_M15.csv` |
| M5 | `USDCHF_M5_*.csv` | `data/processed/USDCHF_master_M5.csv` |
Run once before processing. TFs without a source file are skipped with a warning and fall back to the 4-feature model.
### Step 1 — Process raw data
+22 -10
View File
@@ -702,16 +702,28 @@ def cmd_guardian(args):
def cmd_consolidate(args):
"""Consolidate all USDCHF CSV exports in data/raw/ into USDCHF_master.csv."""
from src.data_consolidator import consolidate_usdchf
result = consolidate_usdchf()
if result.empty:
logger.error(
"Consolidation produced no data. Place USDCHF CSV files in data/raw/ "
"with 'USDCHF' in the filename (e.g. USDCHF_5m_data.csv)."
)
else:
print(f"\nUSDCHF master built: {len(result)} rows → data/processed/USDCHF_master.csv\n")
"""Consolidate USDCHF CSV exports in data/raw/ into per-TF master files.
Produces:
data/processed/USDCHF_master.csv (H1 — source: USDCHF_H1.csv)
data/processed/USDCHF_master_M15.csv (M15 — source: USDCHF_M15_*.csv)
data/processed/USDCHF_master_M5.csv (M5 — source: USDCHF_M5_*.csv)
"""
from src.data_consolidator import (
consolidate_usdchf,
consolidate_usdchf_m15,
consolidate_usdchf_m5,
)
for fn, label, out in [
(consolidate_usdchf, "H1", "data/processed/USDCHF_master.csv"),
(consolidate_usdchf_m15, "M15", "data/processed/USDCHF_master_M15.csv"),
(consolidate_usdchf_m5, "M5", "data/processed/USDCHF_master_M5.csv"),
]:
result = fn()
if not result.empty:
print(f" USDCHF {label}: {len(result)} rows → {out}")
else:
logger.warning("USDCHF %s consolidation produced no data — skipping.", label)
def cmd_listen(args):
+145 -54
View File
@@ -1,10 +1,17 @@
"""USDCHF data consolidator.
Merges all USDCHF raw CSV exports from ``data/raw/`` into a single sorted
master file at ``data/processed/USDCHF_master.csv``. The master file is used
by the training pipeline as an intraday DXY proxy (USDCHF correlates ~0.85
with DXY for XAUUSD signals and is always available on Headway as a standard
Forex pair).
Merges USDCHF raw CSV exports from ``data/raw/`` into sorted master files at
``data/processed/``. One master file is produced per trading timeframe so
each training pipeline uses a USDCHF series whose bar frequency matches the
XAUUSD bars it is merged onto:
USDCHF_master.csv — H1 (source: USDCHF_H1.csv)
USDCHF_master_M15.csv — M15 (source: USDCHF_M15_*.csv)
USDCHF_master_M5.csv — M5 (source: USDCHF_M5_*.csv)
The master files are used by the training pipeline as an intraday DXY proxy
(USDCHF correlates ~0.85 with DXY for XAUUSD signals and is always available
on Headway as a standard Forex pair).
Usage:
python main.py --mode consolidate
@@ -18,18 +25,41 @@ from src.logger import setup_logger
logger = setup_logger(__name__)
RAW_DIR = Path("data/raw")
MASTER_PATH = Path("data/processed/USDCHF_master.csv")
RAW_DIR = Path("data/raw")
MASTER_PATH = Path("data/processed/USDCHF_master.csv") # H1
MASTER_PATH_M15 = Path("data/processed/USDCHF_master_M15.csv")
MASTER_PATH_M5 = Path("data/processed/USDCHF_master_M5.csv")
def _read_usdchf_csv(f: Path) -> pd.DataFrame | None:
"""Read a single USDCHF CSV, auto-detecting delimiter and date format."""
try:
# Detect separator from the first line
header = f.read_text(encoding="utf-8", errors="replace").splitlines()[0]
sep = ";" if ";" in header else ","
"""Read a single USDCHF CSV, auto-detecting delimiter and header presence.
# Try parsing as-is first; fall back to infer_datetime_format
Handles two formats:
- Named-column exports (``Date;Open;High;Low;Close;Volume`` header) from MT5
History Center or the annual XTUP files downloaded with a ``MM/DD/YYYY`` date
- Headerless MT5 bar exports (``USDCHF_H1.csv`` etc.) where the first column
is a raw datetime string with no header row at all
"""
try:
first_line = f.read_text(encoding="utf-8", errors="replace").splitlines()[0]
sep = ";" if ";" in first_line else ","
# If the first character is a digit the file has no header row —
# MT5 "Save as CSV" exports often omit the header entirely.
if first_line[0].isdigit():
df = pd.read_csv(
f, sep=sep, header=None,
names=["Date", "Open", "High", "Low", "Close", "Volume"],
parse_dates=["Date"],
)
df.set_index("Date", inplace=True)
if pd.api.types.is_datetime64_any_dtype(df.index):
logger.info("Loaded %d rows (headerless) from %s", len(df), f)
return df
logger.warning("Could not parse dates in headerless %s — skipping.", f)
return None
# Normal file with a header row
for fmt in ["%m/%d/%Y", "%Y.%m.%d %H:%M", "%Y-%m-%d", None]:
try:
kwargs = dict(sep=sep, parse_dates=["Date"])
@@ -49,44 +79,8 @@ def _read_usdchf_csv(f: Path) -> pd.DataFrame | None:
return None
def consolidate_usdchf(
raw_dir: Path = RAW_DIR,
out_path: Path = MASTER_PATH,
) -> pd.DataFrame:
"""Consolidate all USDCHF CSV files from *raw_dir* into a single master.
Scans for files matching ``*USDCHF*.csv`` (case-insensitive) in *raw_dir*.
Supports both comma-delimited annual exports (``MM/DD/YYYY`` dates) and
semicolon-delimited MT5 exports (``YYYY.MM.DD HH:MM`` dates). The format
is auto-detected per file so mixed sources work together.
Duplicate timestamps are de-duplicated and the result is sorted
chronologically before saving.
Args:
raw_dir: Directory to scan for USDCHF CSV exports.
out_path: Destination for the consolidated master CSV.
Returns:
The consolidated DataFrame (also persisted to *out_path*).
"""
# Case-insensitive glob for Windows and Linux
files = sorted(
f for f in raw_dir.glob("*.csv")
if "usdchf" in f.name.lower()
)
if not files:
logger.warning(
"No USDCHF CSV files found in %s. "
"Export USDCHF data from MT5 History Center (any timeframe that "
"matches your trading TF) and place in data/raw/ with 'USDCHF' "
"in the filename (e.g. USDCHF_5m_data.csv).",
raw_dir,
)
return pd.DataFrame()
def _consolidate_files(files: list[Path], out_path: Path, label: str) -> pd.DataFrame:
"""Merge a list of USDCHF CSV files into a single sorted master CSV."""
frames = []
for f in files:
df = _read_usdchf_csv(f)
@@ -94,7 +88,7 @@ def consolidate_usdchf(
frames.append(df)
if not frames:
logger.error("All USDCHF source files failed to load — no master produced.")
logger.error("All %s source files failed to load — no master produced.", label)
return pd.DataFrame()
merged = pd.concat(frames)
@@ -104,7 +98,104 @@ def consolidate_usdchf(
out_path.parent.mkdir(parents=True, exist_ok=True)
merged.to_csv(out_path)
logger.info(
"USDCHF master saved: %d rows -> %s (from %d source file(s))",
len(merged), out_path, len(files),
"USDCHF %s master saved: %d rows -> %s (source: %s)",
label, len(merged), out_path, ", ".join(f.name for f in files),
)
return merged
def consolidate_usdchf(
raw_dir: Path = RAW_DIR,
out_path: Path = MASTER_PATH,
) -> pd.DataFrame:
"""Build the H1 USDCHF master CSV from ``USDCHF_H1.csv``.
Priority:
1. ``USDCHF_H1.csv`` in *raw_dir* — the preferred source. It is the
hourly MT5 export that matches the H1 trading timeframe exactly and
covers 2010-present. If this file exists, it is used exclusively and
all annual XTUP files are ignored.
2. Falls back to scanning for any ``*USDCHF*.csv`` file in *raw_dir* so
that legacy setups without ``USDCHF_H1.csv`` continue to work.
"""
h1_file = raw_dir / "USDCHF_H1.csv"
if h1_file.exists():
files = [h1_file]
logger.info("Using USDCHF_H1.csv as the sole USDCHF H1 source.")
else:
files = sorted(
f for f in raw_dir.glob("*.csv")
if "usdchf" in f.name.lower()
and "m5" not in f.name.lower()
and "m15" not in f.name.lower()
)
if not files:
logger.warning(
"No USDCHF H1 CSV found in %s. "
"Export USDCHF H1 data from MT5 History Center and save as "
"data/raw/USDCHF_H1.csv, then re-run --mode consolidate.",
raw_dir,
)
return pd.DataFrame()
return _consolidate_files(files, out_path, "H1")
def consolidate_usdchf_m15(
raw_dir: Path = RAW_DIR,
out_path: Path = MASTER_PATH_M15,
) -> pd.DataFrame:
"""Build the M15 USDCHF master CSV from ``USDCHF_M15_*.csv``.
Scans *raw_dir* for any file matching ``USDCHF_M15_*.csv`` (e.g.
``USDCHF_M15_201601040000_202603310000.csv`` exported from MT5 History
Center). Multiple matching files are merged and de-duplicated.
"""
files = sorted(raw_dir.glob("USDCHF_M15_*.csv"))
if not files:
# Broader fallback: any csv with M15 in the name
files = sorted(
f for f in raw_dir.glob("*.csv")
if "usdchf" in f.name.lower() and "m15" in f.name.lower()
)
if not files:
logger.warning(
"No USDCHF M15 CSV found in %s. "
"Export USDCHF M15 data from MT5 and save as "
"data/raw/USDCHF_M15_<dates>.csv, then re-run --mode consolidate.",
raw_dir,
)
return pd.DataFrame()
return _consolidate_files(files, out_path, "M15")
def consolidate_usdchf_m5(
raw_dir: Path = RAW_DIR,
out_path: Path = MASTER_PATH_M5,
) -> pd.DataFrame:
"""Build the M5 USDCHF master CSV from ``USDCHF_M5_*.csv``.
Scans *raw_dir* for any file matching ``USDCHF_M5_*.csv`` (e.g.
``USDCHF_M5_201601040000_202603310000.csv`` exported from MT5 History
Center). Multiple matching files are merged and de-duplicated.
"""
files = sorted(raw_dir.glob("USDCHF_M5_*.csv"))
if not files:
files = sorted(
f for f in raw_dir.glob("*.csv")
if "usdchf" in f.name.lower() and "m5" in f.name.lower()
)
if not files:
logger.warning(
"No USDCHF M5 CSV found in %s. "
"Export USDCHF M5 data from MT5 and save as "
"data/raw/USDCHF_M5_<dates>.csv, then re-run --mode consolidate.",
raw_dir,
)
return pd.DataFrame()
return _consolidate_files(files, out_path, "M5")
+7 -3
View File
@@ -44,9 +44,13 @@ DD_HARD_LIMIT = 0.15
# M5 has 288 bars/day — the OOS window (~2yr) allows up to ~870 trades at
# 2/day cap. Setting 300 forces at least 0.69 trades/day on average,
# preventing the optimizer from rewarding ultra-infrequent cherry-picked wins.
# Both M15 and H1 have ~4.1 years of OOS data (125K H1 / 493K M15 bars, 20% split),
# so 200 trades = ~49/year = ~12/quarter — active enough to validate on 3m live data.
MIN_OOS_TRADES_BY_TF: dict[str, int] = {"M5": 300, "M15": 200, "H1": 200}
# M15: ~4.1yr OOS → 200 trades = ~49/year = ~12/quarter — active enough to
# validate on 3m live data.
# H1: capped at 2 trades/day (small account) × 4.1yr OOS ≈ 2066 potential
# slots. At conservative prob thresholds (0.50–0.58) only 5–10% of days fire,
# giving 50–150 OOS trades. 75 is a realistic floor that excludes near-zero
# signal studies without discarding genuinely selective strategies.
MIN_OOS_TRADES_BY_TF: dict[str, int] = {"M5": 300, "M15": 200, "H1": 75}
MIN_OOS_TRADES = 50 # fallback for unknown TFs
RAM_HIGH_PCT = 90 # pause new trials when used RAM exceeds this %
RAM_PAUSE_SEC = 30 # seconds to sleep when RAM is low
+21 -10
View File
@@ -10,10 +10,18 @@ logger = setup_logger(__name__)
# in raw observations). M15 is ~4× noisier than H1; M5 uses a low obs_cov
# (0.05) to keep the filter very responsive for fast scalping regime detection.
# n_states_default: M5 uses 4 states to capture micro-noise; M15/H1 use 3.
# Single shared USDCHF master file — intraday proxy for DXY (correlates ~0.85
# with USD Index for XAUUSD signals and is always available on Headway).
# Run python main.py --mode consolidate to build this from MT5 exports.
USDCHF_MASTER_PATH = Path("data/processed/USDCHF_master.csv")
# Per-TF USDCHF master files — each uses bars matching the trading timeframe
# so the cross-asset feature is aligned during merge.
# Run python main.py --mode consolidate to build all three from MT5 exports.
USDCHF_MASTER_PATH = Path("data/processed/USDCHF_master.csv") # H1
USDCHF_MASTER_PATH_M15 = Path("data/processed/USDCHF_master_M15.csv") # M15
USDCHF_MASTER_PATH_M5 = Path("data/processed/USDCHF_master_M5.csv") # M5
_USDCHF_PATH_BY_TF: dict[str, Path] = {
"H1": USDCHF_MASTER_PATH,
"M15": USDCHF_MASTER_PATH_M15,
"M5": USDCHF_MASTER_PATH_M5,
}
# Legacy alias kept for any callers that still reference the old name directly.
DXY_RAW_PATH = USDCHF_MASTER_PATH
@@ -213,19 +221,22 @@ def process_pipeline(
df["atr_normalized"] = compute_atr(df)
# ── Optional USDCHF cross-asset feature ─────────────────────────────────
# USDCHF is an intraday DXY proxy — correlates ~0.85 with USD Index and
# is always available on Headway as a standard Forex pair.
# Build the master with: python main.py --mode consolidate
usdchf_df = load_usdchf_data(USDCHF_MASTER_PATH)
# Each TF uses its own USDCHF master so bar frequencies stay in sync:
# H1 → data/processed/USDCHF_master.csv
# M15 → data/processed/USDCHF_master_M15.csv
# M5 → data/processed/USDCHF_master_M5.csv
# Build all three with: python main.py --mode consolidate
usdchf_path = _USDCHF_PATH_BY_TF.get(tf, USDCHF_MASTER_PATH)
usdchf_df = load_usdchf_data(usdchf_path)
if usdchf_df is not None:
df["usdchf_log_return"] = map_usdchf_to_bars(df.index, usdchf_df)
n_usdchf = df["usdchf_log_return"].notna().sum()
logger.info("USDCHF merged: %d non-null usdchf_log_return values.", n_usdchf)
logger.info("USDCHF [%s] merged: %d non-null usdchf_log_return values.", tf, n_usdchf)
else:
logger.info(
"USDCHF master not found at %s — pipeline running with 4 base features. "
"Run python main.py --mode consolidate to enable the 5th feature.",
USDCHF_MASTER_PATH,
usdchf_path,
)
df.dropna(inplace=True)
+6 -7
View File
@@ -83,8 +83,8 @@ class AdaptiveRiskManager:
Account tiers
─────────────
≤ $50 USD — small account:
• max_daily_trades : 2 (4 for M5/M15)
• pos_per_trade : 1 (single position per signal)
• max_daily_trades : 2 (4 for M5/M15; 2 for H1 on headway_cent)
• pos_per_trade : 1 (single position per signal; 2 for M5/M15)
• total daily pos : same as max_daily_trades
> $50 USD — growth account:
@@ -135,12 +135,11 @@ class AdaptiveRiskManager:
"""
_tf = (tf or self.tf).upper()
if self.is_small_account:
# H1 on a small cent account: cap at 1 trade + 1 position per day.
# A single H1 swing can fill most of the 15% daily DD limit on a $15
# account — forcing 1 clean setup prevents compounding losses from
# multiple marginal entries.
# H1 on a small cent account: cap at 2 trades per day.
# Allows recovery if the first setup is a minor stop-out while still
# preventing over-trading the slow hourly timeframe on a $15 account.
if _tf == "H1" and self.broker == "headway_cent":
return {"max_daily_trades": 1, "pos_per_trade": 1, "total_daily_pos": 1}
return {"max_daily_trades": 2, "pos_per_trade": 1, "total_daily_pos": 2}
# M5/M15 small accounts: 4 daily position slots (2 signals × 2 positions).
# Both standard and cent use pos_per_trade=2; lot floor (0.01) in mt5_trader
# ensures notional stays minimal per position.