Gold Regime X

A hybrid machine learning trading system for XAUUSD (Gold) that combines Hidden Markov Models for regime detection, XGBoost for signal classification, and a stateful Regime-Confirmation Signal Engine that gates live trades on regime persistence, XGBoost probability, synth-VIX Z-score, and dynamic ATR band position. Designed for live execution through MetaTrader 5 on both Headway Cent (micro) and Standard accounts, with full Telegram remote control and health monitoring.

---

Table of Contents

1. How It Works
2. Architecture
3. Account Types — Cent vs Standard
4. Prerequisites
5. Installation
6. Data Setup
7. Configuration
8. Complete Workflow — Start to Live Trading
9. Command Reference
10. Signal Logic — Regime-Confirmation
11. Automatic Data Updates
12. Sensitivity Analysis
13. Interactive Research Notebook
14. Risk Management
15. Timeframe Configurations
16. Performance Metrics & Scoring
17. Optimizer Anti-Overfitting Rules
18. Telegram Remote Control
19. Multi-TF Live Trading
20. MQL5 EA (Alternative Execution)
21. Combinatorial Purged CV (CPCV)
22. Walk-Forward Analysis & Staleness Gate
23. Troubleshooting
24. Security Notes

---

How It Works

Raw OHLCV CSV
      │
      ▼
Kalman Filter  →  Smoothed log returns
      │
      ▼
GaussianHMM (k-means prior init, Baum-Welch EM)
      │  Observation features: [kalman_return, volatility, rsi_slope]
      │  All StandardScaler-normalised before fit
      ▼
Regime Labels  Bull=0  Bear=1  Chop=2  [Chop_High=3]  (n_states 3–4, optimised per TF)
      │  Median filter smoothing applied post-prediction
      │    kernel: H1=3 bars, M15=5 bars, M5=5 bars
      │    prevents rapid state oscillation between consecutive bars
      │
      ▼
Session Filter  →  London/NY overlap only  08:00–17:59 UTC
      │  Applied after all rolling indicators are computed
      │  Off-session bars excluded from training and inference
      │
      ▼
GMM Volatility Cluster  →  quiet / normal / volatile label
      │
      ▼
XGBoost Three-Model Volatility Ensemble
      │  Base features (always):
      │    [hmm_0, hmm_1, hmm_2 [, hmm_3]  ← OHE, one column per HMM state
      │     gmm_vol_cluster, rsi_slope,
      │     atr_normalized, prev_log_return]
      │  External features (optional, added when masters exist):
      │    usdchf_log_return, xagusd_log_return,
      │    xtiusd_log_return, synth_vix_zscore*, atr_band_position*
      │  Low / Med / High ATR bucket → separate XGBoost model
      │  * computed from XAUUSD price — no external CSV needed
      │
      ▼
      ▼
SignalEngine — Regime Confirmation
      │  update_regime() tracks consecutive bars in current state
      │  should_enter(): bars ≥ 2, XGBoost prob ≥ TF threshold, P(stay) ≥ TF floor
      │  MR entry: ATR band < 0.10 (buy) or ATR band > 0.90 (sell) in Chop state
      │
      ▼
TCN Confidence Scorer — **removed**
      │    (XGBoost probability gate is the direct signal quality filter)
      │
      ▼
IS / OOS Backtest
      │  Bar-by-bar SignalEngine loop — floating drawdown, Sharpe, Recovery Factor,
      │  Profit Factor, MR attribution, MT5-style equity curve
      │
      ▼
Complex Criterion Score  =  RF_c×0.35 + Sharpe_c×0.35 + (PF−1)×0.20 + Edge_c×0.10
      │  (+return_consistency×0.30 bonus cap for M5/M15)
      │  All terms symmetrically clamped — see Scoring section for details
      │
      ▼
Live Bridge  →  MT5 Market Orders
      │  IOC fill, ATR-based SL, staged TPs, ATR trail
      │  Hybrid Scalp Protection on M5 (between-bar, every 5 s)
      │  TF-specific magic numbers: H1=123456, M15=123457, M5=123458

The Optuna optimizer searches Kalman parameters, HMM state count, and XGBoost hyperparameters (including L1 and L2 regularisation), scoring every trial on OOS Complex Criterion only using Combinatorial Purged Cross-Validation (CPCV) with C(4,2) = 6 combinatorial train/test paths. The dataset is split into 4 chronological blocks; each trial evaluates all 6 path combinations with a TF-specific purge gap (H1: 24 bars, M15: 96 bars, M5: 288 bars) to prevent leakage between adjacent train and test blocks. Stage 1 (--stage xgb) uses a fast single IS/OOS hold-out (~5× faster, no CPCV); Stage 2 (--stage trading) runs full CPCV warm-started from Stage 1 params. Before each optimization run, ensure_data_updated() auto-fetches any missing bars from MT5 so the model always trains on the most recent data.

---

Architecture

File	Purpose
src/processor.py	Kalman filter, log returns, RSI, ATR, GMM vol cluster, per-TF config; London/NY session gating (08:00–17:59 UTC, applied after all rolling indicators)
src/engine_hmm.py	GaussianHMM with k-means prior init; TF persistence boost; median-filter state smoothing (kernel H1=3, M15/M5=5) to prevent rapid state oscillation
src/engine_xgb.py	XGBoost volatility ensemble; three vol-bucket models (Low/Med/High ATR); hmm_state OHE (hmm_0/hmm_1/hmm_2[/hmm_3]) in prepare_features; atr_band_position feature; compute_regime_stats() for metadata; ONNX export
src/signal_engine.py	Stateful signal engine — SignalEngine; regime-confirmation entry (persistence + XGBoost prob + synth-VIX Z-score + ATR band); exit on regime reversal, persistence collapse, profit erosion, or max hold
src/sensitivity.py	Z-Score sensitivity sweep — Bull/Bear cutoffs 1.5–3.0σ; outputs ranked table + reports/sensitivity_<TF>_<broker>.csv/json
src/backtester.py	Bar-by-bar backtest via SignalEngine — IS/OOS split, broker costs, floating drawdown, MT5 equity curve, MR attribution
src/optimizer.py	CPCV optimizer — C(4,2)=6 combinatorial purged paths; Calmar-dominant composite scoring (Calmar×0.45 + Sharpe×0.35 + PF×0.15 + Edge×0.05); CPCV_N_BLOCKS=4, CPCV_K_TEST=2, CPCV_PURGE_BARS per TF; two-stage: --stage xgb fast hold-out → --stage trading full CPCV; WFO_PARAMS/WFO_PARAMS_FAST retained for backward-compat (notebook); per-broker SQLite resume; RAM guard; Telegram heartbeat
notebooks/GoldRegimeX_Explorer.ipynb	Interactive research notebook — equity explorer, WFO window analysis (standard/fast mode), feature/regime explorer, parameter sensitivity with WFO comparison, CV Path Inspector (Section 6)
src/risk_manager.py	AdaptiveRiskManager, CentConverter, DailyEquityGate, broker cost configs
src/visualizer.py	6-chart visual report: regime overlay, equity curve, features, transition matrix, dashboard, MT5 balance/equity
src/mt5_sync.py	MT5 data downloader; ensure_data_updated() — auto-fetches missing bars before each optimization run
src/validator.py	Pre-live validation gate — SignalEngine inference + Sharpe threshold + spread-payoff erosion warning
src/mt5_trader.py	Live execution loop: bar detection, XGBoost inference, SignalEngine regime-confirmation entry/exit, order placement, ATR trailing exits, M5 scalp recycling
src/notifier.py	Telegram message sender
src/auditor.py	MT5 deal history report
src/data_updater.py	Weekly MT5 data pull — WeeklyDataUpdater appends fresh XAUUSD bars to raw CSVs every Sunday
src/guardian.py	Multi-TF rolling health monitor — fires Telegram alert when rolling Sharpe drops below threshold
src/remote_control.py	Telegram long-polling bot for remote commands
src/data_consolidator.py	USDCHF master file builder
src/monte_carlo.py	Monte Carlo simulation on trained models — distribution of expected return/drawdown outcomes
main.py	CLI entry point for all modes
mql5/GoldRegimeX.mq5	MT5 Expert Advisor with ONNX inference (alternative to Python bridge)

---

Account Types — Cent vs Standard

Choose your account type before running any command. The --broker flag controls lot sizing, P&L conversion, spread guards, and signal calibration throughout the entire pipeline.

Headway Cent Account (--broker headway_cent)

Item	Cent Account	Example
Real deposit	$15 USD	Wired to Headway
MT5 balance display	1 500.00 USC	real USD × 100
Minimum lot	0.01	= 0.01 oz gold
P&L per $1 gold move at 0.01 lot	0.01 USC	= $0.0001 real USD
MT5 history shows +15.00	÷ 100	= $0.15 real USD
Bridge balance handling	Divides raw balance by 100	Pass --balance 15 (real USD)
Spread viability guard (M5)	TP1 ≥ 1.5× spread

Best for: Learning the system and verifying signals with minimal capital at risk.

Headway Standard Account (--broker standard)

Item	Standard Account	Example
Minimum recommended	$500+ USD
Minimum lot	0.01	= 1 oz gold
P&L per $1 gold move at 0.01 lot	$1.00 real USD	100× more than cent
Bridge balance handling	Uses raw balance directly	Pass --balance 500
Spread viability guard	TP1 ≥ 3.0× spread	All timeframes

Key difference: On cent, +$2.40 in MT5 history = $0.024 real USD. On standard it is exactly $2.40. The bridge always logs P&L in real USD regardless of account type.

---

Prerequisites

• Windows — MetaTrader5 Python package is Windows-only
• Python 3.11
• MetaTrader5 terminal open and logged into your broker account
• XAUUSD visible in Market Watch
• Algorithmic trading enabled in MT5: Tools → Options → Expert Advisors → Allow Algorithmic Trading
• A Telegram bot (optional but strongly recommended for remote control and health alerts)

---

Installation

# 1. Clone the repo
git clone https://github.com/lucasmos/GoldRegime_X.git
cd GoldRegime_X

# 2. Create a virtual environment
python -m venv venv
venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy the environment template
copy .env.example .env

---

Data Setup

Export historical OHLCV CSVs from MetaTrader5 (History Center — F2 → XAUUSD → your TF → Export):

Timeframe	Expected filename
H1	data/raw/XAU_1h_data.csv
M15	data/raw/XAU_15m_data.csv
M5	data/raw/XAU_5m_data.csv

Format: semicolon-delimited, columns: Date;Open;High;Low;Close;Volume

The system filters to the last 10 years anchored at the end of your CSV. Export fresh data before each full pipeline re-run.

Getting deep history from MT5

1. Tools → Options → Charts → Max bars: 99,999,999
2. View → Symbols (Ctrl+U) → XAUUSD → Bars tab → Request (repeat until start year is reached)
3. Export Bars → save as .csv

External Asset Features (optional, recommended)

The processor enriches each XAUUSD bar with cross-asset log returns. Export each asset from MT5 and run --mode consolidate to build the master files.

Asset	Role	Source files (data/raw/)	Master produced
USDCHF	Intraday DXY proxy (~0.85 correlation)	USDCHF_H1.csv / USDCHF_M15_*.csv / USDCHF_M5_*.csv	USDCHF_master[_M15/_M5].csv
XAGUSD	Silver — cross-commodity regime signal	XAGUSD_H1.csv / XAGUSD_M15_*.csv / XAGUSD_M5_*.csv	XAGUSD_master[_M15/_M5].csv
XTIUSD	WTI crude — macro risk-on/off signal	XTIUSD_H1.csv / XTIUSD_M15_*.csv / XTIUSD_M5_*.csv	XTIUSD_master[_M15/_M5].csv
US500	S&P 500 — equity/gold correlation	US500_H1.csv / US500_M15_*.csv / US500_M5_*.csv	US500_master[_M15/_M5].csv
USDJPY	JPY safe-haven proxy	USDJPY_H1.csv / USDJPY_M15_*.csv / USDJPY_M5_*.csv	USDJPY_master[_M15/_M5].csv

All masters live in data/processed/. The synth_vix_zscore feature (Williams VIX Fix) and atr_band_position feature (dynamic ATR band position, 0 = lower band, 1 = upper band) are both computed directly from XAUUSD price — no external file is needed for either.

python main.py --mode consolidate

The pipeline degrades gracefully: any asset whose master file is absent is simply omitted from the feature set. XGBoost's get_feature_cols() only includes a column when it is >50% non-null.

---

Configuration

1. Create a Telegram Bot (optional but recommended)

1. Open Telegram → search @BotFather → /newbot
2. Copy the Bot Token
3. Find your Chat ID via https://api.telegram.org/bot<TOKEN>/getUpdates
4. Find your User ID via @userinfobot

2. Edit your .env file

TELEGRAM_BOT_TOKEN=your_bot_token_here
TELEGRAM_CHAT_ID=your_chat_id_here
ALLOWED_USER_ID=your_telegram_user_id_here

# Defaults used by the Telegram START TRADING button
LIVE_TF=H1
LIVE_BROKER=headway_cent
LIVE_BALANCE=15

---

Complete Workflow — Start to Live Trading

This is the full sequence for a first-time setup on a Headway Cent account, H1 timeframe. Adapt TF and broker flags as needed.

Phase 0 — One-time setup

# Consolidate USDCHF CSV exports into per-TF master files
python main.py --mode consolidate

Phase 1 — Build and optimise the model

Model optimisation uses a unified single-stage workflow. A single exploration (∼5× faster, no CPCV) that saves the best params to models/stage1_{tf}_{broker}.json. Stage 2 runs the full CPCV optimization warm-started from those params. Both stages are safe to interrupt and resume. Run --mode train after both stages to commit the final model to disk.

# --- Preparation ---
python main.py --mode process --tf H1

# NOTE: ensure_data_updated() runs automatically at the start of every optimize run
# to fetch any bars missing since your last CSV export — MT5 must be connected.

# === STAGE 1: Fast XGB Exploration (no CPCV, ~5x faster) ===
# Explores the hyperparameter landscape with a single IS/OOS split.
# Saves best params to models/stage1_h1_headway_cent.json.
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --trials 100

# === STAGE 2: Full CPCV Optimization (warm-started from Stage 1) ===
# Runs C(4,2)=6 CPCV path evaluation, seeded from Stage 1 params.
# Saves all trials to models/study_headway_cent.db.
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --stage trading --trials 130

# === COMMIT: Train Final Model ===
# Reads best params from study DB, trains GaussianHMM + XGBoost 3-vol-bucket ensemble.
# Saves: hmm_model_H1_headway_cent.pkl + xgb_ensemble_H1_headway_cent.pkl
python main.py --mode train --tf H1 --broker headway_cent --balance 15

# --- Optional post-training checks ---
python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15
python main.py --mode montecarlo  --tf H1 --broker headway_cent --balance 15
python main.py --mode report      --tf H1 --broker headway_cent --balance 15
python main.py --mode compare     --tf H1,M15,M5 --broker headway_cent --balance 15
python main.py --mode export      --tf H1 --broker headway_cent

Phase 2 — Validate before going live

# Download live MT5 data and check rolling Sharpe against trained model.
# Use --period 6m for H1 and M15; --period 3m for M5.
# --period 12m available for a deeper H1 look-back.
python main.py --mode sync_validate --tf H1 --broker headway_cent --balance 15 --period 6m

Status	H1 Sharpe	M15 Sharpe	M5 Sharpe	Action
PASS	≥ 0.25	≥ 0.50	≥ 0.70	Proceed
WARN	0.05–0.25	0.20–0.50	0.40–0.70	Proceed with caution or re-optimise
FAIL	< 0.05	< 0.20	< 0.40	Re-optimise + retrain before going live

FAIL exits with code 1 and blocks the live script.

Phase 3 — Walk-Forward Analysis (optional but recommended)

python main.py --mode wfa --tf H1 --broker headway_cent --balance 15

Checks that the model generalises across time periods. WFE ≥ 60% = robust.

Phase 4 — Demo test

Connect MT5 to a demo account and run:

python main.py --mode demo --tf H1 --broker headway_cent --balance 15

This sends real orders to MT5 (no simulation). Use it to verify lot sizes, TP/SL placement, and session limits before risking real money.

Phase 5 — Go live

Connect MT5 to your live account and run:

python main.py --mode live --tf H1 --broker headway_cent --balance 15

Type YES when prompted. The live loop:

1. Detects each newly completed bar (polls every 5 s)
2. Fetches 200 bars for Kalman / HMM warm-up
3. Runs Kalman → GMM → HMM → XGBoost inference
4. SignalEngine.update_regime() + should_enter() — checks persistence, XGBoost prob, synth-VIX Z-score, ATR band position
5. Applies session limits (08:00–17:59 UTC), margin check, spread viability guard
6. Places IOC market orders with ATR-based SL and staged TPs (ATR multiples)
7. Logs closed P&L in real USD after every trade
8. Every Sunday: appends fresh bars to raw CSVs (WeeklyDataUpdater)

Important: Remove the GoldRegimeX.mq5 EA from any XAUUSD chart before starting the same-TF Python bridge — they share the same magic number.

Phase 6 — Ongoing maintenance

# Weekly (M5) / monthly (M15, H1):
python main.py --mode sync_validate --tf H1 --broker headway_cent --balance 15 --period 6m
python main.py --mode wfa           --tf H1 --broker headway_cent --balance 15

# If sync_validate fails or WFE < 50% — re-run the two-stage optimize + retrain:
# Stage 1: re-run fast XGB exploration (resumes existing study, adds trials on top)
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --trials 100
# Stage 2: re-run full CPCV (warm-started from updated Stage 1 params)
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --stage trading --trials 130
# Commit: retrain final model with new best params
python main.py --mode train    --tf H1 --broker headway_cent --balance 15

The Guardian runs a continuous rolling Sharpe health check and fires Telegram alerts if performance degrades.

---

Command Reference

python main.py --mode <MODE> [OPTIONS]

Modes

Mode	Description
consolidate	Merge *USDCHF*.csv files in data/raw/ into per-TF USDCHF masters
process	Process raw CSV → features (Kalman, log returns, RSI, ATR, GMM cluster)
optimize	Run / resume Optuna hyperparameter search (OOS Complex Criterion scoring). Supports --stage xgb (fast XGB exploration) → --stage trading (full CPCV, warm-started) two-stage workflow
train	Train HMM + XGBoost; show IS/OOS breakdown
sensitivity	Z-Score sensitivity sweep (Bull/Bear cutoffs 1.5–3.0) on trained models
compare	Side-by-side OOS comparison across TFs ranked by Complex Criterion
export	Export XGBoost ensemble → ONNX for MQL5 EA
report	Generate 6-chart visual report → reports/<TF>_<broker>/
wfa	Walk-Forward Analysis — per-fold Complex Criterion scores and WFE ratio
sync_validate	Download live MT5 data + validate model health + cost audit
demo	Run live execution loop on MT5 demo account (no YES prompt)
live	Run live execution loop on MT5 live account (requires YES confirmation)
audit	Generate and Telegram-send daily MT5 deal report
guardian	Continuous rolling Sharpe health monitor + Telegram alerts
listen	Start Telegram remote control bot (with nightly 23:55 summary)
montecarlo	Monte Carlo simulation on trained models — return/drawdown outcome distribution
extract_consensus	Extract consensus best parameters across Optuna trials

All Options

Option	Type	Default	Description
--tf	str	H1	Timeframe: H1, M15, M5 (or comma-separated for compare/guardian)
--broker	str	standard	headway_cent or standard
--balance	float	15	Account size in real USD
--trials	int	250	Total Optuna trial target. Recommended: H1=400, M15=600, M5=1000.
--stage	str	None	Two-stage optimize: xgb = Stage 1 fast XGB exploration (no CPCV, ∼5× faster, saves stage1_{tf}_{broker}.json); trading = Stage 2 full CPCV warm-started from Stage 1. Omit to run standard joint optimization
--period	str	3m	Lookback for MT5 sync: 3m, 6m, 12m
--interval	int	3600	Guardian check interval in seconds
--profit_target	float	4.0 on M5	Quick-profit close in USD per position. Pass 0 to disable on M5
--skip_stale_check	flag	off	Bypass model-staleness gate on live/demo
--train_days	int	TF default	WFA IS window in calendar days (H1=365, M15=180, M5=90)
--test_days	int	TF default	WFA OOS step in calendar days (H1=90, M15=60, M5=30)
--n_jobs	int	1	Parallel Optuna workers (advanced)

Note: --balance is always in real USD. For a Headway Cent account with $15, pass --balance 15 — the bridge handles the ×100 display conversion internally.

Per-Timeframe Recommended Workflows

H1 — Headway Cent (first time):

# One-time setup
python main.py --mode consolidate
python main.py --mode process        --tf H1

# Stage 1: Fast XGB exploration (no CPCV) — saves stage1_h1_headway_cent.json
python main.py --mode optimize       --tf H1 --broker headway_cent --balance 15 --stage xgb --trials 60

# Stage 2: Full CPCV optimization (warm-started from Stage 1)
python main.py --mode optimize       --tf H1 --broker headway_cent --balance 15 --stage trading --trials 130

# Commit: Train final model with best params
python main.py --mode train          --tf H1 --broker headway_cent --balance 15

# Optional checks
python main.py --mode sensitivity    --tf H1 --broker headway_cent --balance 15
python main.py --mode montecarlo     --tf H1 --broker headway_cent --balance 15
python main.py --mode report         --tf H1 --broker headway_cent --balance 15

# Validate + go live
python main.py --mode sync_validate  --tf H1 --broker headway_cent --balance 15 --period 6m
python main.py --mode wfa            --tf H1 --broker headway_cent --balance 15
python main.py --mode demo           --tf H1 --broker headway_cent --balance 15
python main.py --mode live           --tf H1 --broker headway_cent --balance 15

M15 — Headway Cent:

python main.py --mode process        --tf M15

# Stage 1: Fast XGB exploration — saves stage1_m15_headway_cent.json
python main.py --mode optimize       --tf M15 --broker headway_cent --balance 15 --stage xgb --trials 100

# Stage 2: Full CPCV optimization (warm-started from Stage 1)
python main.py --mode optimize       --tf M15 --broker headway_cent --balance 15 --stage trading --trials 200

# Commit: Train final model with best params
python main.py --mode train          --tf M15 --broker headway_cent --balance 15

# Optional checks
python main.py --mode sensitivity    --tf M15 --broker headway_cent --balance 15
python main.py --mode montecarlo     --tf M15 --broker headway_cent --balance 15
python main.py --mode report         --tf M15 --broker headway_cent --balance 15

# Validate + go live
python main.py --mode sync_validate  --tf M15 --broker headway_cent --balance 15 --period 6m
python main.py --mode wfa            --tf M15 --broker headway_cent --balance 15
python main.py --mode demo           --tf M15 --broker headway_cent --balance 15
python main.py --mode live           --tf M15 --broker headway_cent --balance 15

M5 — Headway Cent:

python main.py --mode process        --tf M5

# Stage 1: Fast XGB exploration — saves stage1_m5_headway_cent.json
python main.py --mode optimize       --tf M5 --broker headway_cent --balance 15 --stage xgb --trials 150

# Stage 2: Full CPCV optimization (warm-started from Stage 1)
python main.py --mode optimize       --tf M5 --broker headway_cent --balance 15 --stage trading --trials 300

# Commit: Train final model with best params
python main.py --mode train          --tf M5 --broker headway_cent --balance 15

# Optional checks
python main.py --mode sensitivity    --tf M5 --broker headway_cent --balance 15
python main.py --mode montecarlo     --tf M5 --broker headway_cent --balance 15
python main.py --mode report         --tf M5 --broker headway_cent --balance 15

# Validate + go live
python main.py --mode sync_validate  --tf M5 --broker headway_cent --balance 15 --period 3m
python main.py --mode wfa            --tf M5 --broker headway_cent --balance 15
python main.py --mode demo           --tf M5 --broker headway_cent --balance 15
python main.py --mode live           --tf M5 --broker headway_cent --balance 15

---

Signal Logic — Regime-Confirmation

Stateful Regime-Confirmation Engine

The live signal gate is the SignalEngine class (src/signal_engine.py). It is stateful — it tracks how many consecutive bars the current HMM regime has persisted, whether a trade is open, and what regime the trade was entered in.

Note on HMM state encoding: XGBoost receives the HMM regime as one-hot encoded columns (hmm_0, hmm_1, hmm_2 for n_states=3; hmm_0–hmm_3 for n_states=4) rather than a single integer. This prevents the model from treating states as ordinal (0 < 1 < 2) and allows each regime to have an independent feature weight.

Trend entry conditions (all must pass):

1. Regime has persisted — at least 2 consecutive bars in the same regime (Bull or Bear)
2. synth_vix_zscore ≥ MIN_TREND_ZSCORE for this TF (volatility expansion confirmation)
3. XGBoost probability ≥ TF threshold (H1 only; M15/M5 bypass the XGB gate)
4. ATR band position within valid range (trend: < 0.80 for BUY, > 0.20 for SELL)
5. HMM self-transition probability ≥ TF persistence floor

TF	XGBoost threshold	Persistence floor	MIN_TREND_ZSCORE	MIN_MR_ZSCORE
H1	0.575	0.57	0.5	1.0
M15	bypass (0.0)	0.55	1.0	1.5
M5	bypass (0.0)	0.45	1.5	2.0

Mean Reversion entries (Chop state ≥ 2)

MR entries require ATR band position at an extreme: ATR band < 0.10 → MR_BUY; ATR band > 0.90 → MR_SELL. MR positions use 75% of the standard lot size. synth_vix_zscore ≥ MIN_MR_ZSCORE is also required.

Exit conditions

The engine exits an open trade when any of the following fires:

Exit reason	Condition
regime_reversal	HMM state changes from the entry regime
persistence_collapse	P(stay in regime) drops below the TF persistence floor
profit_erosion	Current P&L < 40% of peak P&L for this trade
max_hold	Trade open for H1: 24 bars / M15: 32 bars / M5: 48 bars

Trade gate requirements

A trade fires when all of the following pass:

Gate	Requirement
Regime confirmed	≥ 2 consecutive bars in regime
synth_vix_zscore	≥ MIN_TREND_ZSCORE for this TF
XGBoost prob	≥ TF threshold (H1 only; M15/M5 bypass)
ATR band	trend: position < 0.80 (BUY) or > 0.20 (SELL)
Persistence	P(stay in state) ≥ TF floor
Direction	BUY only in Bull (0); SELL only in Bear (1); Chop → MR only
ATR band (MR)	MR_BUY: ATR band < 0.10; MR_SELL: ATR band > 0.90
Session limit	Under daily trade cap for this TF
Margin check	Sufficient free margin for the lot size
Spread viability	TP1 ≥ spread × ratio (1.5× cent / 3.0× standard)
DailyEquityGate	Floating loss < 5% AND day gain < profit-lock threshold
Global Guard	Fewer than 4 GRX positions open across all TFs

Mean Reversion in Chop — Live Safety Gates

MR signals are gated by the engine's persistence check (P(stay in Chop state) ≥ TF floor), ATR band extremity (< 0.10 / > 0.90), and synth_vix_zscore ≥ MIN_MR_ZSCORE, matching the backtester exactly.

Logic Audit

Every bar that does not fire a trade, the bridge logs a structured reason:

[LOGIC AUDIT] BULL | state=0 prob=0.510 bars=1 P(stay)=0.82 vix_z=0.87 atr_band=0.44

Spread Viability Guard

Broker	Applied on	Minimum ratio
headway_cent	M5 only	TP1 ≥ 1.5× spread
standard	All timeframes	TP1 ≥ 3.0× spread

Two-Sided DailyEquityGate

Loss gate: Floating equity drops ≥ 5% → gate locks, all GRX positions closed. No new signals for the rest of the day.

Profit lock: Day gain reaches TF-specific threshold → no new entries.

TF	Profit Lock	Loss Gate
M5	20% day gain	5% loss
M15	10% day gain	5% loss
H1	10% day gain	5% loss

Both gates reset at UTC midnight.

ATR Trailing Exits

Phase	Trigger	Action
Phase 1 (break-even)	Floating P&L ≥ activation amount	SL → entry + 2×spread
Phase 2 (trail)	Activated	Trailing SL at ATR × trail multiplier

Activation amounts and trail multipliers by TF:

TF	Activation P&L	Trail Multiplier
H1	$1.50	2.5× ATR
M15	$1.50	1.5× ATR
M5	$1.00	1.5× ATR

Staged Take-Profits

Regime	TF	TP1	TP2 (Runner)	SL ATR mult
Bull / Bear	M5	0.8× ATR	1.5× ATR	1.5×
Bull / Bear	M15	1.2× ATR	2.5× ATR	2.0×
Bull / Bear	H1	1.5× ATR	3.0× ATR	2.0×
Chop (MR)	M5	0.5× ATR	—	1.05×
Chop (MR)	M15	0.8× ATR	—	1.4×
Chop (MR)	H1	1.0× ATR	—	1.4×

Full profit protection chain (all TFs):

1. Profit guard — SL → entry + 2×spread when price reaches 70% of TP1 distance
2. Break-even — runner SL → entry when TP1 fills
3. ATR trail — activates when floating P&L ≥ TF activation amount

M5-only Hybrid Scalp Protection (runs every 5 s between bars):

4. Fixed scalp target — position closed when floating P&L ≥ $4.00
5. Trailing guard — if peak P&L ≥ $2.00, close when P&L falls to ≤ 50% of peak
6. Chop-exit — all positions closed at market if HMM shifts to Chop mid-trade
7. M5 recycle — after scalp target close, re-entry is allowed on the same bar if regime unchanged and daily cap not hit

Automatic Data Updates

Pre-Optimisation Sync (ensure_data_updated)

src/mt5_sync.py exposes ensure_data_updated(tf, symbol), which is called automatically at the start of every --mode optimize run for each requested timeframe. It:

1. Reads the last timestamp from the corresponding raw CSV file (data/raw/XAU_*_data.csv)
2. Calculates how many bars are missing up to the current time
3. Fetches only the missing bars from MT5 using copy_rates_range
4. Appends new rows in semicolon-delimited format, matching the existing CSV structure
5. Skips silently if MT5 is unavailable — optimization continues with existing data

This ensures the model always trains on the most recent bars without requiring a manual data export before each optimization cycle.

# Sync happens automatically — just run:
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --trials 400

Weekly Data Updater (WeeklyDataUpdater)

src/data_updater.py runs automatically inside the live trading loop every Sunday. It:

1. Fetches the last ~2 weeks of XAUUSD bars from MT5 for each active TF (H1: 336 bars, M15: 1344, M5: 4032)
2. Deduplicates by timestamp and appends only new rows to the raw CSV training files
3. Writes a data/raw/.last_auto_update marker

No manual action is required. The updater is a no-op on non-Sunday days and won't re-run within the same week.

Guardian daily check

When --mode guardian\ is running it performs a daily model health check. If rolling Sharpe degrades or models appear stale, the guardian fires an alert via Telegram.

---

Sensitivity Analysis

The sensitivity analysis sweeps Bull/Bear Z-Score cutoffs across a range on your already-trained model and shows how trades, Sharpe, drawdown, and profit factor change at each level. Note: the live signal path now uses SignalEngine — sensitivity results are informational only.

python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15

What it tests: Bull (Z_CUTOFF_BULL) and Bear (Z_CUTOFF_BEAR) cutoffs from 1.5σ to 3.0σ in steps of 0.25σ. MR (Chop) cutoffs are held constant throughout.

Output — printed table:

Z     OOS Trades  Win%    Sharpe  MaxDD%  PF      RF      Return%  Status
1.50  312         52.3    0.81    14.2    1.43    1.92    18.4
1.75  241         55.1    0.97    11.8    1.61    2.34    22.1     ← current
2.00  188         58.2    1.12    10.1    1.79    2.87    26.3     ← BEST
2.25  134         60.4    1.04    9.8     1.74    2.61    24.1
...

Output files (written to reports/):

reports/sensitivity_H1_headway_cent.csv
reports/sensitivity_H1_headway_cent.json

The JSON includes current_z, best_z, and best_sharpe for automated comparison.

When to run sensitivity analysis:

• After training a new model — confirm the hardcoded TF Z cutoff is near-optimal
• When validation Sharpe is borderline — determine whether a lower cutoff would pass

---

Interactive Research Notebook

notebooks/GoldRegimeX_Explorer.ipynb is a fully interactive Jupyter notebook for exploring, tweaking, and diagnosing the system without touching any source files. Run it from the repo root:

jupyter notebook notebooks/GoldRegimeX_Explorer.ipynb

Sections

Section	Feature	Typical runtime
1 — Data & Model Loader	Select TF, broker, balance, and WFO mode (standard / fast). Loads processed data, best Optuna params, and trained HMM + XGB models.	~5 s
2 — Equity Curve Explorer	Interactive equity curve with IS/OOS split line, drawdown panel, and full metric table. Re-runs instantly on balance/broker change.	instant
3 — WFO Window Analysis	Runs _run_wfo using the loaded best params. Plots per-window OOS score bars. Drill into any window's equity curve with the Window # slider. Shows Walk-Forward Efficiency (WFE) interpretation label.	~2–3 min
4 — Feature & Regime Explorer	Price + HMM regime overlay, regime distribution pie, GMM volatility cluster histogram, feature distributions per regime, XGB feature importance.	instant
5 — Parameter Sensitivity	Sliders for n_states, max_depth, learning_rate, reg_alpha, reg_lambda, and min_child_weight. Run Comparison re-trains and shows a side-by-side metric bar chart. Run WFO Score Comparison runs proper rolling WFO (not just a full-dataset backtest) on both baseline and modified configs.	~30–60 s / ~4–8 min WFO
6 — CV Path Inspector	Runs either WFO IS CV (per-window inner fold scores) or CPCV (C(6,2)=15 paths). Displays a per-path boxplot, per-path score bar chart, and a consistency score = n_profitable_paths / total × 100%.	~2–5 min

WFO Mode (standard vs fast)

Selected via the WFO Mode dropdown in Section 1. Stored in _CACHE and used by Sections 3, 5, and 6.

Mode	IS window	OOS window	Use case
standard	H1: 1 yr / M15: 1 yr / M5: 1 yr	H1: 90d / M15: 90d / M5: 90d	Default; thorough evaluation
fast	H1: 6 mo / M15: 6 mo / M5: 6 mo	H1: 45d / M15: 45d / M5: 30d	Quicker feedback; suits intraday regime shifts

Walk-Forward Efficiency (WFE) Labels

Section 3 automatically labels the WFE ratio after each run:

WFE	Label
≥ 0.80	✅ Excellent (OOS ≥ 80% of IS CV performance)
≥ 0.50	🟡 Acceptable (OOS ≥ 50% of IS CV performance)
≥ 0.20	🟠 Marginal — consider broader regularization
< 0.20	🔴 Poor — strong overfitting signal

---

Risk Management

Position Sizing — 1% Risk Rule

lot_size = (1% × account_balance_USD) / (ATR(14) × SL_multiplier)

Minimum lot is always 0.01. All lots rounded to 2 decimal places.

Daily Exposure Limits

Account Balance	TF	Positions per Signal	Max Trades/Day
≤ $50	H1 / M15	2 (cent) / 1 (standard)	2
≤ $50	M5	2 (cent) / 1 (standard)	4
> $50	Any	3	2–3 depending on regime

Global Guard: skip any signal if ≥ 4 GRX positions are open across all TFs.

Broker Cost Configs

Broker	Spread fraction	Commission fraction	Total round-trip
headway_cent	0.02%	0.02%	0.04%
standard	0.02%	0.01%	0.03%

---

Timeframe Configurations

Parameter	M5	M15	H1
Kalman obs_cov default	0.05	4.0	1.0
Bars/day (annualisation)	288	96	24
HMM n_states search space	3–4	3–4	3–4
HMM persistence gate (training)	≥ 0.65	≥ 0.65	≥ 0.65
IS/OOS split	65% / 35%	65% / 35%	70% / 30%
Min OOS trades (hard floor)	120	60	20
Min OOS trades (penalty threshold)	350	140	25
SL ATR multiplier	1.5×	2.0×	2.0×
TP1 multiplier (trend)	0.8× SL	1.0× SL	1.5× SL
TP2 multiplier (runner)	1.5× SL	2.0× SL	3.0× SL
Hybrid Scalp Protection	On (5 s)	Off	Off
Fixed scalp target	$4 USD	Off	Off
Trailing guard activation	$2 peak	Off	Off
DailyEquityGate loss	5%	5%	5%
DailyEquityGate profit lock	20%	10%	10%
Model staleness gate	14 days	30 days	30 days
M5 optimisation freshness gate	120 h	—	—
Recommended --trials	1000	600	400

n_states restriction (all TFs): All timeframes search {3, 4}. n_states=2 is banned on all TFs — with only Bull/Bear states every bar is signal-eligible, creating excessive counter-trend noise. n_states=3 gives Bull/Bear/Chop; n_states=4 adds a Chop_High state for volatile chop regimes. The optimal count is selected by Optuna per study.

---

Performance Metrics & Scoring

Complex Criterion Score

Score = clamp(RF, -5, 5) × 0.35
      + clamp(Sharpe, -3, 3) × 0.35
      + clamp(PF − 1, -2, 2) × 0.20     ← normalised: breakeven = 0
      + clamp(avg_payoff / $0.035, 0, 2) × 0.10
      + [M5/M15 only] return_consistency × 0.30  (cap +0.30)

Score range: approximately −5 to +5. A score > 0.5 is considered a worthwhile configuration.

Component	Weight	Clamp	Measures
Recovery Factor (RF)	0.35	[−5, 5]	Capital preservation — net profit / floating max DD
Sharpe Ratio	0.35	[−3, 3]	Return smoothness — prevents outlier-driven strategies
Profit Factor − 1 (normalised)	0.20	[−2, 2]	Trade quality — breakeven = 0, loss-only = −1
Edge / Spread ratio	0.10	[0, 2]	avg payoff vs $0.035 spread proxy
Return Consistency	+0.30 bonus cap	—	Weekly P&L stability (M5/M15 only)

WFO Variance Penalty

The final WFO score aggregates per-window scores with an increased variance penalty:

wfo_score = median(window_scores) − 0.20 × std(window_scores)

The 0.20 multiplier (up from 0.15) aligns with the wider score range (−5 to +5) to meaningfully penalise inconsistent strategies.

All Reported Metrics

Metric	Description
Score	Complex Criterion (see above)
Sharpe Ratio	Annualised return / annualised volatility
Recovery Factor	Net profit / floating max drawdown (capped 20× for display)
Profit Factor	Gross wins / gross losses (capped 10× for display)
Expected Payoff	Mean per-trade return × account_size
Max Drawdown	Peak-to-trough closed-bar drawdown
Floating Max DD	Intra-bar adverse excursion using bar High/Low
Win Rate	Fraction of trades that closed profitable
Trade Count	Total closed trades in the window
Avg Efficiency	Mean ATR / spread on active-trade bars
Cost Efficiency	1 - (total_costs / gross_profit)
Total Payout	total_return × account_size in broker currency
Return Consistency	Weekly P&L stability: `1 - std / (std +
mr_trades	Count of MR trades (Chop-state signals)
mr_win_rate	Win rate of MR trades
mr_pnl	Cumulative log-return from MR trades only

IS/OOS Split Ratios

TF	IS	OOS
H1	70%	30%
M15	65%	35%
M5	65%	35%

---

Optimizer Anti-Overfitting Rules

Rule	Detail
OOS-only scoring	All scoring uses OOS data only — IS is never evaluated in the objective
Complex Criterion	RF_c×0.35 + Sharpe_c×0.35 + (PF−1)×0.20 + Edge_c×0.10 — all terms clamped symmetrically
Hard trade floor	OOS trades < hard minimum → score −50.0 (trial discarded)
Progressive trade penalty	OOS trades below soft threshold → score × 0.1
Payoff floor	OOS average edge < $0.035 → score × 0.1
Max drawdown gate	OOS floating DD > 20% → score −50.0
H1 DD guard	OOS floating DD > 5% → score × 0.5
IS/OOS generalisation	If IS Sharpe > 0.1 and OOS/IS Sharpe < 0.35 → score −50.0
HMM persistence gate	Any self-transition < 0.65 → score −100.0
M5 activity bonus	OOS trades > 300 → score × 1.2; OOS trades < 150 → score × 0.5
n_states restriction	All TFs: {3, 4}
No threshold search	SignalEngine thresholds are hardcoded TF constants — not Optuna parameters
Per-broker study isolation	All TF studies for a broker are colocated in study_{broker}.db (e.g. study_headway_cent.db). Use optuna-dashboard sqlite:///models/study_headway_cent.db to view all active studies.

---

Canonical Regime Contract

All timeframes (H1, M15, M5) use exactly 3 HMM states. This is a hard architectural invariant, not a tunable parameter.

ID	Label	Trading Allowed
0	TREND	Yes
1	MEAN_REVERSION	No (strict no-trade)
2	VOLATILITY_SHOCK	Yes

Enforcement layers

1. optimizer.py — n_states is never suggested by Optuna. It is pinned to 3 unconditionally for H1, M15, and M5. SEARCH_SPACES[tf]["n_states"] records (3, 3, "int"). _enforce_three_states(params, context) raises ValueError at every fit_hmm call site.
2. main.py / optimizer.py resolve_n_states — any call with n_states != 3 raises ValueError immediately with a descriptive message.
3. engine_hmm.py — _assert_canonical_states(states, context) is called after fit_hmm and predict_states. If any state id outside {0,1,2} is found a RuntimeError is raised. _log_transition_matrix always renders a 3×3 matrix with semantic labels (never numeric fallback columns).
4. engine_xgb.py — _validate_state_feature_schema(df, tf) is called inside prepare_features. For H1 it requires hmm_state and forbids state_3. For M15/M5 it requires state_0, state_1, state_2 and forbids state_3.

Rejected inputs

• n_states=4 or n_states=5: ValueError at resolve_n_states / _enforce_three_states
• Any HMM producing state id 3+: RuntimeError at _assert_canonical_states
• Feature matrix with state_3 column: ValueError at _validate_state_feature_schema

Legacy note

Legacy 4/5-state trials may exist in a resumed Optuna study from before this patch. They do not affect new trials (all new fits use n_states=3). The _assert_canonical_states guard ensures that if a legacy model file is loaded and somehow predicts state 3, it is caught immediately at inference time.

---

H1 Profitability Safeguards

H1 is scored with additional hard-floor penalties and a tunable entry-gate to reduce persistent negative Sharpe trials while preserving CPCV robustness.

Hard-Floor Penalties

Three floors are checked per CPCV path aggregation. A breach applies penalty = 10.0 + (floor − metric) × 10.0:

Param	Default	Meaning
h1_min_median_sharpe	0.10	Median CPCV Sharpe must exceed this
h1_min_median_expectancy	0.0	Median per-trade expectancy ($) must exceed this
h1_min_median_pf	1.02	Median profit factor must exceed this

Reason codes such as floor_sharpe:0.032<0.100 appear in the structured trial log:

trial=7 tf=H1 base=0.1812 p_var=0.0000 p_inst=0.0000 p_dd=0.0000 p_mr=0.0000 p_floor=12.3200 p_turn=0.0000 final=-12.1388 reasons=floor_sharpe:0.032<0.100|floor_pf:0.98<1.02

Turnover Penalty

Excessive H1 trade churn (measured as trades per 100 bars across CPCV test windows) is penalised:

Param	Default	Meaning
h1_max_trades_per_100	4.0	Trades per 100 bars before penalty fires

Penalty scales linearly: (trades_per_100 − max) × 2.0. Not applied on M15/M5.

Tunable Entry Gate

H1 entry uses a tunable probability threshold for the XGBoost regime classifier. During optimization, h1_entry_prob is an Optuna parameter (range [0.57, 0.66]). In live/demo mode it can be overridden at the CLI:

# Run optimization with custom floor overrides
python main.py --mode optimize --tf H1 --broker headway_cent --trials 200 \
    --h1_min_median_sharpe 0.12 \
    --h1_min_median_pf 1.05 \
    --h1_max_trades_per_100 3.5

# Run demo with fixed entry gate override
python main.py --mode demo --tf H1 --broker headway_cent \
    --h1_entry_prob 0.62

CLI Reference

Flag	Type	Default	Description
--h1_entry_prob	float	0.575	XGBoost entry probability gate for H1. Range [0.50, 0.90].
--h1_min_median_sharpe	float	0.10	H1 Sharpe floor. Range [-1.0, 2.0].
--h1_min_median_pf	float	1.02	H1 profit-factor floor. Range [0.5, 3.0].
--h1_max_trades_per_100	float	4.0	H1 max trades/100 bars. Range [0.5, 20.0].

All flags are validated at startup — out-of-range values exit immediately with a descriptive message.

Asymmetric Buy/Sell Thresholds (advanced)

For programmatic use (not CLI), pass h1_entry_prob_buy and h1_entry_prob_sell separately via evaluator_config in vectorized_backtest. Both are clamped to [0.50, 0.90].

---

| CPCV per trial | Every Optuna trial evaluates all C(4,2)=6 combinatorial purged paths — not a single IS/OOS split; CPCV_MAX_FLOAT_DD=0.20 prunes paths with > 20% floating DD | | OOS scaler consistency | OOS features are always scaled with the IS-fitted scaler (no data leakage) | | State alignment | states_oos and states_is_cv are re-aligned to df_aligned.index after NaN drops | | L2 regularisation | reg_lambda is searched for all TFs — XGBoost default of 1.0 is no longer forced | | Pruner per TF | M5: HyperbandPruner (early-stops expensive trials); H1/M15: MedianPruner(startup=10, warmup=5) |

# Hard trade floors (trials below these return -50.0 immediately):
MIN_OOS_TRADES_HARD = {"M5": 120, "M15": 60, "H1": 20}

# Soft trade floors (trials below these have score × 0.1):
TF_MIN_OOS_TRADES   = {"M5": 350, "M15": 140, "H1": 25}

# CPCV trial budgets (Stage 2 / joint optimization):
CPCV_TRIALS = {"H1": 80, "M15": 120, "M5": 200}
# Stage-1 trial budgets (fast hold-out, no CPCV):
STAGE1_TRIALS = {"H1": 60, "M15": 100, "M5": 150}

Optuna Search Space per Timeframe

Parameter	H1	M15	M5
learning_rate	0.005 – 0.15	0.005 – 0.20	0.01 – 0.15
n_estimators	50 – 400 (step 50)	100 – 500 (step 50)	200 – 600 (step 50)
max_depth	3 – 7	3 – 7	2 – 4
min_child_weight	5 – 100	3 – 30	5 – 25
reg_alpha (L1)	1e-6 – 0.5	1e-6 – 0.1	1e-6 – 0.1
reg_lambda (L2)	0.01 – 2.0	1e-6 – 0.1	1e-6 – 0.1
subsample	0.5 – 0.9	0.5 – 0.9	0.55 – 0.85
colsample_bytree	0.4 – 0.9	0.5 – 1.0	0.4 – 0.8

⚠️ Delete the study DB whenever a fundamental parameter changes (e.g. after adding a new feature, changing n_states search space, altering the score function, enabling OHE for hmm_state, or adding session gating). Old trials bias the surrogate model.

del models\study_headway_cent.db
del models\study_headway_cent_h1.db   # legacy per-TF DB if it exists
del models\study_standard.db

---

Telegram Remote Control

Start the listener alongside your trading session:

python main.py --mode listen --broker headway_cent --balance 15

Button	Action
🚀 START TRADING	Launches --mode live using .env TF/broker/balance defaults
🛑 STOP TRADING	Terminates the running live process
📉 START OPTIMIZE (M5)	Starts / resumes M5 Optuna study
📊 BOT STATUS	Last 24 h P&L, win rate, floating positions

A nightly summary is sent at 23:55 UTC while the listener runs.

Guardian — Continuous health monitor

python main.py --mode guardian --tf M5,M15,H1 --period 3m \
    --interval 3600 --broker headway_cent --balance 15

Every hour: validates rolling Sharpe for each TF — fires Telegram alert if below 0.6.

Audit — On-demand deal report

python main.py --mode audit --broker headway_cent --balance 15

Parallel Optimisation

# Open multiple terminals — each one runs an independent Optuna worker:
Terminal 1:  python main.py --mode optimize --tf M5 --broker headway_cent --trials 1000
Terminal 2:  python main.py --mode optimize --tf M5 --broker headway_cent --trials 1000

Workers share the same SQLite study safely via locking.

---

Multi-TF Live Trading

Each timeframe runs as an independent Python process with its own magic number:

TF	Magic Number	Comment format
H1	123456	GRX_H1_TREND_BUY_s0_tp1
M15	123457	GRX_M15_TREND_SELL_s1_tp2
M5	123458	GRX_M5_MR_BUY_s2_tp1

Global Exposure Guard

Before any signal, the bridge counts all open GRX positions across all TFs. If ≥ 4 positions are open, the signal is skipped.

Starting multiple TFs

# Terminal 1
python main.py --mode live --tf H1 --broker headway_cent --balance 15

# Terminal 2
python main.py --mode live --tf M15 --broker headway_cent --balance 15

# Terminal 3
python main.py --mode live --tf M5 --broker headway_cent --balance 15

At UTC midnight each bridge sends a daily P&L audit to Telegram.

---

MQL5 EA (Alternative Execution)

mql5/GoldRegimeX.mq5 is a self-contained MT5 Expert Advisor that loads the ONNX model directly inside MT5 and replicates the same regime → signal → risk logic in MQL5.

Do not run the EA and the Python bridge for the same TF simultaneously.

To use the EA:

1. python main.py --mode export --tf H1 --broker headway_cent
2. Copy mql5/GoldRegimeX.mq5 and the .onnx file to MQL5/Experts/
3. Compile in MetaEditor (F7) and attach to the XAUUSD chart

---

Combinatorial Purged CV (CPCV)

Both the Optuna optimizer (--mode optimize) and the post-training diagnostic (--mode wfa) evaluate XGBoost performance using Combinatorial Purged Cross-Validation (CPCV) — not rolling Walk-Forward windows. CPCV eliminates serial-correlation leakage by inserting a purge gap at every train/test boundary and exhausts all combinatorial path orderings rather than relying on a single rolling split.

Note: _run_wfo, WFO_PARAMS, and WFO_PARAMS_FAST are retained in src/optimizer.py for backward compatibility with the research notebook only. They are not used by the optimizer or --mode wfa.

How it works

Full dataset (e.g., 10 years of H1 bars)
  │
  Split into N_BLOCKS = 4 equal chronological blocks
  │
  Generate all C(4,2) = 6 train/test path combinations
  │
  ├── Path 1: Train=[B1,B2]  purge_gap  Test=[B3,B4]
  ├── Path 2: Train=[B1,B3]  purge_gap  Test=[B2,B4]
  ├── Path 3: Train=[B1,B4]  purge_gap  Test=[B2,B3]
  ├── Path 4: Train=[B2,B3]  purge_gap  Test=[B1,B4]
  ├── Path 5: Train=[B2,B4]  purge_gap  Test=[B1,B3]
  └── Path 6: Train=[B3,B4]  purge_gap  Test=[B1,B2]
  │
  For each path:
    1. HMM fitted strictly on purged training blocks (no lookahead)
    2. HMM applied to full dataset → predict_states
    3. StandardScaler fitted on training rows → applied to full dataset
    4. XGBoost ensemble trained on training path (train_ratio=1.0)
    5. Vectorized backtest on test path blocks only
    6. Path score = Calmar-dominant composite (Calmar×0.45 + Sharpe×0.35 + PF×0.15 + Edge×0.05)
    7. Paths pruned if floating_max_drawdown > 20% OR n_trades < hard floor
  │
  cpcv_score = median(path_scores) − 0.20 × std(path_sharpes)

CPCV Parameters

Parameter	H1	M15	M5
CPCV_N_BLOCKS	4	4	4
CPCV_K_TEST	2	2	2
Paths per trial C(4,2)	6	6	6
CPCV_PURGE_BARS (embargo)	24 bars	96 bars	288 bars
CPCV_MAX_FLOAT_DD	20%	20%	20%
MIN_TRADES_PER_PATH	15	60	100
Stage-2 / joint trials	80	120	200
Stage-1 trials (hold-out)	60	100	150

Two-stage optimization flow

Stage	CLI flag	Method	Speed	Output
Stage 1	--stage xgb	Fast single IS/OOS hold-out — no CPCV	~5× faster	models/stage1_{tf}_{broker}.json
Stage 2	--stage trading	Full CPCV — C(4,2)=6 paths, warm-started from Stage 1	Full	models/study_{broker}.db
Joint	(omit --stage)	Full CPCV from scratch	Full	models/study_{broker}.db

CPCV score interpretation

cpcv_score	Interpretation
≥ +1.0	✅ Excellent — strong consistent performance across all 6 paths
≥ +0.3	🟡 Acceptable — robust enough for live deployment
≥ 0.0	🟠 Marginal — consider additional regularisation
< 0.0	🔴 Poor — inconsistent paths; re-optimise with more trials

---

Walk-Forward Analysis & Staleness Gate

Walk-Forward Analysis

Full dataset
  ├── Window 1:  Train [Y1–Y2]       → Test [Y2 Q3]
  ├── Window 2:  Train [Y1 Q3–Y2 Q3] → Test [Y3 Q1]
  └── Aggregate: WFE = mean(OOS Sharpe) / mean(IS Sharpe)

WFE	Interpretation
≥ 60%	Robust — safe to go live
50–60%	Acceptable — monitor closely
< 50%	Fragile — model curve-fits specific years

python main.py --mode wfa --tf H1 --broker headway_cent --balance 15

Model Staleness Gate

TF	Max HMM/XGB model age
M5	14 days
M15	30 days
H1	30 days

M5 has an additional 5-day optimisation freshness gate: m5_meta_<broker>.json must exist and be < 120 hours old before going live.

Bypass the gate for demo testing:

python main.py --mode demo --tf M5 --broker headway_cent --balance 15 --skip_stale_check

Recommended Maintenance Schedule

# Weekly (M5):
python main.py --mode sync_validate --tf M5 --period 3m --broker headway_cent --balance 15
python main.py --mode wfa           --tf M5 --broker headway_cent --balance 15
# If needed:
python main.py --mode optimize      --tf M5 --broker headway_cent --balance 15 --trials 1000
python main.py --mode train         --tf M5 --broker headway_cent --balance 15

# Monthly (H1 / M15):
python main.py --mode wfa       --tf H1 --broker headway_cent --balance 15
python main.py --mode optimize  --tf H1 --broker headway_cent --balance 15 --trials 400
python main.py --mode train     --tf H1 --broker headway_cent --balance 15

---

Troubleshooting

Problem	Cause	Fix
ConnectionError: Could not connect to MT5	Terminal not running	Open MT5 and log in first
FileNotFoundError: hmm_model_H1_headway_cent.pkl	Models not trained	--mode train --tf H1 --broker headway_cent
FileNotFoundError: m5_meta_headway_cent.json	M5 not optimised	--mode optimize --tf M5 --broker headway_cent
WARNING: No Optuna study found	Wrong broker or deleted DB	--mode optimize with matching --broker
ERROR: Degenerate HMM during train	Stale params	Delete study DB, re-optimise, then train
Validation FAIL every day	Model stale or data ends before sync window	Export fresh CSV from MT5, re-run full pipeline
No signals firing	XGBoost prob or persistence below thresholds	Check [LOGIC AUDIT] logs for prob= and P(stay)= values; consider re-training
No entry after regime change	bars_in_regime < 2	Engine waits for confirmation — normal on first bar of a new regime
WFA shows many ❌ folds	Model curve-fits specific years	Loosen regularisation, increase trials, add more training data
Order failed: retcode=10006	No broker connection	Check MT5 connection indicator
Order failed: retcode=10015	Price moved past deviation	Will retry next bar; elevated deviation auto-applies on high-vol
[CONFLICT] warning at startup	GRX positions from EA already open	Stop EA / other process before starting bridge
Double positions	MQL5 EA running alongside Python bridge	Remove GoldRegimeX.mq5 EA from chart
Telegram errors in log	Wrong token or unconfigured	Check .env; regenerate via @BotFather

Emergency stop: Press Ctrl+C. Open positions remain open — close them manually from the MT5 Trade tab.

---

Security Notes

• Never commit .env — it is in .gitignore
• .env.example contains only placeholders
• ALLOWED_USER_ID is the single security gate for all Telegram commands
• If credentials are accidentally committed, immediately revoke the bot token via @BotFather /revoke
• The listener uses Telegram long-polling — no public webhook or open port required

---

State Labels

Hardcoded across all Python modules and the MQL5 EA:

Label	Integer	Signal eligibility
Bull	0	BUY (trend)
Bear	1	SELL (trend)
Chop	2	MR_BUY / MR_SELL only (n_states = 3)
Chop_Low	2	MR_BUY / MR_SELL only (n_states = 4)
Chop_High	3	MR_SELL only with higher cutoff (n_states = 4)

Model Files Reference

models/
├── hmm_model_H1_headway_cent.pkl
├── hmm_model_H1_standard.pkl
├── hmm_model_M15_headway_cent.pkl
├── hmm_model_M5_headway_cent.pkl
├── xgb_ensemble_H1_headway_cent.pkl
├── xgb_ensemble_M15_headway_cent.pkl
├── xgb_ensemble_M5_headway_cent.pkl
├── xgb_model_H1_headway_cent.onnx               ← MQL5 EA uses this
├── study_headway_cent.db               ← Optuna trials (per-broker, never shared)
├── study_standard.db
└── m5_meta_headway_cent.json           ← M5 optimisation freshness gate

Without this fix, the notebook's loaded probabilities came from a single global XGBoost model instead of the three vol-bucket models used by the optimizer and live trader, making Section 2 equity curves and Section 3 WFO scores inconsistent with production results.