Gold Regime X

A hybrid machine learning trading system for XAUUSD (Gold) that combines Hidden Markov Models for regime detection, XGBoost for signal classification, a TCN confidence scorer for dynamic threshold adjustment, and a stateful Regime-Confirmation Signal Engine that gates live trades on regime persistence, XGBoost probability, and Bollinger Band position. Designed for live execution through MetaTrader 5 on both Headway Cent (micro) and Standard accounts, with full Telegram remote control, health monitoring, and automatic TCN maintenance.

---

Table of Contents

1. How It Works
2. Architecture
3. Account Types — Cent vs Standard
4. Prerequisites
5. Installation
6. Data Setup
7. Configuration
8. Complete Workflow — Start to Live Trading
9. Command Reference
10. Signal Logic — Regime-Confirmation
11. TCN Confidence Scorer
12. Automatic Data Updates & TCN Maintenance
13. Sensitivity Analysis
14. Interactive Research Notebook
15. Risk Management
16. Timeframe Configurations
17. Performance Metrics & Scoring
18. Optimizer Anti-Overfitting Rules
19. Telegram Remote Control
20. Multi-TF Live Trading
21. MQL5 EA (Alternative Execution)
22. Walk-Forward Optimization (WFO)
23. Walk-Forward Analysis & Staleness Gate
24. Troubleshooting
25. Security Notes
26. Changelog

---

How It Works

Raw OHLCV CSV
      │
      ▼
Kalman Filter  →  Smoothed log returns
      │
      ▼
GaussianHMM (k-means prior init, Baum-Welch EM)
      │  Observation features: [kalman_return, volatility, rsi_slope]
      │  All StandardScaler-normalised before fit
      ▼
Regime Labels  Bull=0  Bear=1  Chop=2  Chop_High=3 (n_states=4)
      │
      ▼
GMM Volatility Cluster  →  quiet / normal / volatile label
      │
      ▼
XGBoost Three-Model Volatility Ensemble
      │  Features (V4, up to 6):
      │    [hmm_state, gmm_vol_cluster, rsi_slope,
      │     atr_normalized, prev_log_return, usdchf_log_return*]
      │  Low / Med / High ATR bucket → separate XGBoost model
      │  * optional: requires data/processed/USDCHF_master.csv
      │
      ▼
      ▼
SignalEngine — Regime Confirmation
      │  update_regime() tracks consecutive bars in current state
      │  should_enter(): bars ≥ 2, XGBoost prob ≥ TF threshold, P(stay) ≥ TF floor
      │  MR entry: BB ≤ 0.30 (buy) or BB ≥ 0.70 (sell) in Chop state
      │
      ▼
TCN Confidence Scorer (optional, loads automatically)
      │  4× dilated causal Conv1D → GlobalAveragePooling → Dense(32) → sigmoid
      │  Scores the current signal bar from 100-bar context sequences
      │  Outputs a confidence multiplier [0.7, 1.3]:
      │    multiplier < 0.85 → effective XGBoost prob ×1.10 (boosts entry)
      │    multiplier ≥ 0.85 → no boost
      │    multiplier = 1.0  → no adjustment (TCN not loaded)
      │
      ▼
IS / OOS Backtest
      │  Bar-by-bar SignalEngine loop — floating drawdown, Sharpe, Recovery Factor,
      │  Profit Factor, MR attribution, MT5-style equity curve
      │
      ▼
Complex Criterion Score  =  RF_c×0.35 + Sharpe_c×0.35 + (PF−1)×0.20 + Edge_c×0.10
      │  (+return_consistency×0.30 bonus cap for M5/M15)
      │  All terms symmetrically clamped — see Scoring section for details
      │
      ▼
Live Bridge  →  MT5 Market Orders
      │  IOC fill, ATR-based SL, staged TPs, ATR trail
      │  Hybrid Scalp Protection on M5 (between-bar, every 5 s)
      │  TF-specific magic numbers: H1=123456, M15=123457, M5=123458

The Optuna optimizer searches Kalman parameters, HMM state count, and XGBoost hyperparameters (including L1 and L2 regularisation), scoring every trial on OOS Complex Criterion only using the Rolling Walk-Forward Optimization (_run_wfo) with per-window IS cross-validation. Supports purged 3-fold time-series cross-validation via --split_method purged_ts. A "fast" WFO mode is available for M5/M15 with a shorter 6-month IS window for quicker intraday regime adaptation.

---

Architecture

File	Purpose
src/processor.py	Kalman filter, log returns, RSI, ATR, GMM vol cluster, per-TF config
src/engine_hmm.py	GaussianHMM with k-means prior init; TF persistence boost
src/engine_xgb.py	XGBoost volatility ensemble; compute_regime_stats() for metadata; ONNX export
src/engine_tcn.py	TCN confidence scorer — SignalConfidenceTCN, dilated causal Conv1D, load_tcn_classifier(), get_tcn_dir(); outputs multiplier used by SignalEngine
src/signal_engine.py	Stateful signal engine — SignalEngine; regime-confirmation entry (persistence + XGBoost prob); exit on regime reversal, persistence collapse, profit erosion, or max hold
src/sensitivity.py	Z-Score sensitivity analysis — sweeps Bull/Bear cutoffs 1.5–3.0, outputs comparison table + CSV/JSON
src/backtester.py	Bar-by-bar backtest via SignalEngine — IS/OOS split, broker costs, floating drawdown, MT5 equity curve, MR attribution
src/optimizer.py	Rolling WFO optimizer — per-window IS cross-validation; Revised Complex Criterion scoring; WFO_PARAMS, WFO_PARAMS_FAST, CV_FOLDS; per-broker SQLite resume; RAM guard; Telegram heartbeat
notebooks/GoldRegimeX_Explorer.ipynb	Interactive research notebook — equity explorer, WFO window analysis (standard/fast mode), feature/regime explorer, parameter sensitivity with WFO comparison, CV Path Inspector (Section 6)
src/risk_manager.py	AdaptiveRiskManager, CentConverter, DailyEquityGate, broker cost configs
src/visualizer.py	6-chart visual report: regime overlay, equity curve, features, transition matrix, dashboard, MT5 balance/equity
src/mt5_sync.py	MT5 data downloader
src/validator.py	Pre-live validation gate — SignalEngine inference + Sharpe threshold + spread-payoff erosion warning
src/mt5_trader.py	Live execution loop: bar detection, XGBoost inference, SignalEngine regime-confirmation entry/exit, order placement, ATR trailing exits, M5 scalp recycling
src/notifier.py	Telegram message sender
src/auditor.py	MT5 deal history report
src/data_updater.py	Weekly MT5 data pull — WeeklyDataUpdater appends fresh XAUUSD bars to raw CSVs every Sunday
src/tcn_maintenance.py	Hourly TCN staleness monitor — TCNMaintenanceScheduler; auto-retrains stale models via background subprocess
src/guardian.py	Multi-TF rolling health monitor + daily TCN staleness check + auto-retrain
src/remote_control.py	Telegram long-polling bot for remote commands
src/data_consolidator.py	USDCHF master file builder
main.py	CLI entry point for all modes
mql5/GoldRegimeX.mq5	MT5 Expert Advisor with ONNX inference (alternative to Python bridge)

---

Account Types — Cent vs Standard

Choose your account type before running any command. The --broker flag controls lot sizing, P&L conversion, spread guards, and signal calibration throughout the entire pipeline.

Headway Cent Account (--broker headway_cent)

Item	Cent Account	Example
Real deposit	$15 USD	Wired to Headway
MT5 balance display	1 500.00 USC	real USD × 100
Minimum lot	0.01	= 0.01 oz gold
P&L per $1 gold move at 0.01 lot	0.01 USC	= $0.0001 real USD
MT5 history shows +15.00	÷ 100	= $0.15 real USD
Bridge balance handling	Divides raw balance by 100	Pass --balance 15 (real USD)
Spread viability guard (M5)	TP1 ≥ 1.5× spread

Best for: Learning the system and verifying signals with minimal capital at risk.

Headway Standard Account (--broker standard)

Item	Standard Account	Example
Minimum recommended	$500+ USD
Minimum lot	0.01	= 1 oz gold
P&L per $1 gold move at 0.01 lot	$1.00 real USD	100× more than cent
Bridge balance handling	Uses raw balance directly	Pass --balance 500
Spread viability guard	TP1 ≥ 3.0× spread	All timeframes

Key difference: On cent, +$2.40 in MT5 history = $0.024 real USD. On standard it is exactly $2.40. The bridge always logs P&L in real USD regardless of account type.

---

Prerequisites

• Windows — MetaTrader5 Python package is Windows-only
• Python 3.11
• MetaTrader5 terminal open and logged into your broker account
• XAUUSD visible in Market Watch
• Algorithmic trading enabled in MT5: Tools → Options → Expert Advisors → Allow Algorithmic Trading
• A Telegram bot (optional but strongly recommended for remote control and health alerts)

---

Installation

# 1. Clone the repo
git clone https://github.com/lucasmos/GoldRegime_X.git
cd GoldRegime_X

# 2. Create a virtual environment
python -m venv venv
venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy the environment template
copy .env.example .env

---

Data Setup

Export historical OHLCV CSVs from MetaTrader5 (History Center — F2 → XAUUSD → your TF → Export):

Timeframe	Expected filename
H1	data/raw/XAU_1h_data.csv
M15	data/raw/XAU_15m_data.csv
M5	data/raw/XAU_5m_data.csv

Format: semicolon-delimited, columns: Date;Open;High;Low;Close;Volume

The system filters to the last 10 years anchored at the end of your CSV. Export fresh data before each full pipeline re-run.

Getting deep history from MT5

1. Tools → Options → Charts → Max bars: 99,999,999
2. View → Symbols (Ctrl+U) → XAUUSD → Bars tab → Request (repeat until start year is reached)
3. Export Bars → save as .csv

USDCHF — USD strength feature (optional, recommended)

USDCHF is an intraday DXY proxy. Export from MT5 and consolidate:

Trading TF	Source file	Master produced
H1	data/raw/USDCHF_H1.csv	data/processed/USDCHF_master.csv
M15	data/raw/USDCHF_M15_*.csv	data/processed/USDCHF_master_M15.csv
M5	data/raw/USDCHF_M5_*.csv	data/processed/USDCHF_master_M5.csv

python main.py --mode consolidate

TFs without a source file degrade gracefully to the 5-feature model.

---

Configuration

1. Create a Telegram Bot (optional but recommended)

1. Open Telegram → search @BotFather → /newbot
2. Copy the Bot Token
3. Find your Chat ID via https://api.telegram.org/bot<TOKEN>/getUpdates
4. Find your User ID via @userinfobot

2. Edit your .env file

TELEGRAM_BOT_TOKEN=your_bot_token_here
TELEGRAM_CHAT_ID=your_chat_id_here
ALLOWED_USER_ID=your_telegram_user_id_here

# Defaults used by the Telegram START TRADING button
LIVE_TF=H1
LIVE_BROKER=headway_cent
LIVE_BALANCE=15

---

Complete Workflow — Start to Live Trading

This is the full sequence for a first-time setup on a Headway Cent account, H1 timeframe. Adapt TF and broker flags as needed.

Phase 0 — One-time setup

# Consolidate USDCHF CSV exports into per-TF master files
python main.py --mode consolidate

Phase 1 — Build and optimise the model

# 1. Process raw CSV into features (Kalman, log returns, RSI, ATR, GMM cluster)
python main.py --mode process --tf H1

# 2. Optimise hyperparameters (Kalman params, HMM states, XGBoost regularisation)
#    Runs Optuna and saves every trial to models/study_headway_cent.db
#    Safe to interrupt and resume — --trials is a TOTAL target, not incremental
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --trials 400

# 3. Train the final model with the best Optuna parameters
#    Trains HMM + XGBoost; saves model files
python main.py --mode train --tf H1 --broker headway_cent --balance 15

# 4. (Optional) Run Z-Score sensitivity analysis to confirm the TF cutoff is optimal
python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15

# 5. (Optional) Generate 6-chart visual report to review IS/OOS performance
python main.py --mode report --tf H1 --broker headway_cent --balance 15

# 6. (Optional) Export model to ONNX for the MQL5 EA
python main.py --mode export --tf H1 --broker headway_cent

Phase 2 — Train the TCN confidence scorer

The TCN must be trained after the HMM+XGBoost model exists — it scores signal quality using the same processed features plus derived columns.

# Full training (first time — ~10–20 min):
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 100 --temperature 1.5

The model saves to models/tcn/H1_headway_cent/. Subsequent --mode live runs load it automatically. If no TCN model exists, the live bridge runs with no TCN boost (all XGBoost probs are used as-is).

Phase 3 — Validate before going live

# Download last 6 months of live MT5 data and check rolling Sharpe
# Use --period 6m or 8m for H1 — 3m produces very few trades
python main.py --mode sync_validate --tf H1 --broker headway_cent --balance 15 --period 6m

Status	Sharpe	Action
PASS	≥ 0.8	Proceed
WARN	0.5–0.8	Proceed with caution or re-optimise
FAIL	< 0.5	Re-optimise + retrain before going live

FAIL exits with code 1 and blocks the live script.

Phase 4 — Walk-Forward Analysis (optional but recommended)

python main.py --mode wfa --tf H1 --broker headway_cent --balance 15

Checks that the model generalises across time periods. WFE ≥ 60% = robust.

Phase 5 — Demo test

Connect MT5 to a demo account and run:

python main.py --mode demo --tf H1 --broker headway_cent --balance 15

This sends real orders to MT5 (no simulation). Use it to verify lot sizes, TP/SL placement, and session limits before risking real money.

Phase 6 — Go live

Connect MT5 to your live account and run:

python main.py --mode live --tf H1 --broker headway_cent --balance 15

Type YES when prompted. The live loop:

1. Detects each newly completed bar (polls every 5 s)
2. Fetches 200 bars for Kalman / HMM warm-up
3. Runs Kalman → GMM → HMM → XGBoost inference
4. Loads TCN; computes confidence multiplier (boosts effective XGBoost prob if multiplier < 0.85)
5. SignalEngine.update_regime() + should_enter() — checks persistence, XGBoost prob, BB position
6. Applies session limits, margin check, spread viability guard
7. Places IOC market orders with ATR-based SL and staged TPs (ATR multiples)
8. Logs closed P&L in real USD after every trade
9. Every hour: runs TCN staleness check; every Sunday: appends fresh bars to raw CSVs

Important: Remove the GoldRegimeX.mq5 EA from any XAUUSD chart before starting the same-TF Python bridge — they share the same magic number.

Phase 7 — Ongoing maintenance

# Weekly (M5) / monthly (M15, H1):
python main.py --mode sync_validate --tf H1 --broker headway_cent --balance 15 --period 6m
python main.py --mode wfa           --tf H1 --broker headway_cent --balance 15

# Re-optimise + retrain if sync_validate fails or WFE < 50%:
python main.py --mode optimize --tf H1 --broker headway_cent --balance 15 --trials 400
python main.py --mode train    --tf H1 --broker headway_cent --balance 15

# Fine-tune TCN on last 2 years (faster than full retrain):
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 20 --fine_tune --recent_years 2

# Or full TCN retrain if fine-tune isn't improving results:
python main.py --mode train_tcn --tf H1 --broker headway_cent --epochs 100

The Guardian auto-retrains the TCN every 7 days while running. The live bridge also checks staleness hourly without interrupting trading.

---

Command Reference

python main.py --mode <MODE> [OPTIONS]

Modes

Mode	Description
consolidate	Merge *USDCHF*.csv files in data/raw/ into per-TF USDCHF masters
process	Process raw CSV → features (Kalman, log returns, RSI, ATR, GMM cluster)
optimize	Run / resume Optuna hyperparameter search (OOS Complex Criterion scoring)
train	Train HMM + XGBoost; show IS/OOS breakdown
train_tcn	Train TCN confidence scorer; supports full training, fine-tune, temperature
sensitivity	Z-Score sensitivity sweep (Bull/Bear cutoffs 1.5–3.0) on trained models
compare	Side-by-side OOS comparison across TFs ranked by Complex Criterion
export	Export XGBoost ensemble → ONNX for MQL5 EA
report	Generate 6-chart visual report → reports/<TF>_<broker>/
wfa	Walk-Forward Analysis — per-fold Complex Criterion scores and WFE ratio
sync_validate	Download live MT5 data + validate model health + cost audit
demo	Run live execution loop on MT5 demo account (no YES prompt)
live	Run live execution loop on MT5 live account (requires YES confirmation)
audit	Generate and Telegram-send daily MT5 deal report
guardian	Continuous rolling Sharpe health monitor + daily TCN auto-retrain
listen	Start Telegram remote control bot (with nightly 23:55 summary)

All Options

Option	Type	Default	Description
--tf	str	H1	Timeframe: H1, M15, M5 (or comma-separated for compare/guardian)
--broker	str	standard	headway_cent or standard
--balance	float	15	Account size in real USD
--trials	int	250	Total Optuna trial target. Recommended: M5=1000, M15=600, H1=400
--period	str	3m	Lookback for MT5 sync: 3m, 6m, 12m
--interval	int	3600	Guardian check interval in seconds
--profit_target	float	4.0 on M5	Quick-profit close in USD per position. Pass 0 to disable on M5
--skip_stale_check	flag	off	Bypass model-staleness gate on live/demo
--train_days	int	TF default	WFA IS window in calendar days (H1=365, M15=180, M5=90)
--test_days	int	TF default	WFA OOS step in calendar days (H1=90, M15=60, M5=30)
--epochs	int	100	TCN training epochs for train_tcn
--seq_len	int	100	TCN input sequence length in bars
--temperature	float	1.5	TCN sigmoid temperature. >1.0 softens confidence; <1.0 sharpens
--fine_tune	flag	off	Adapt existing TCN to recent data instead of full retrain
--recent_years	int	2	Years of recent data to use when --fine_tune is set
--n_jobs	int	1	Parallel Optuna workers (advanced)

Note: --balance is always in real USD. For a Headway Cent account with $15, pass --balance 15 — the bridge handles the ×100 display conversion internally.

Per-Timeframe Recommended Workflows

H1 — Headway Cent (first time):

python main.py --mode consolidate
python main.py --mode process        --tf H1
python main.py --mode optimize       --tf H1 --broker headway_cent --balance 15 --trials 400
python main.py --mode train          --tf H1 --broker headway_cent --balance 15
python main.py --mode sensitivity    --tf H1 --broker headway_cent --balance 15
python main.py --mode train_tcn      --tf H1 --broker headway_cent --epochs 100
python main.py --mode report         --tf H1 --broker headway_cent --balance 15
python main.py --mode sync_validate  --tf H1 --broker headway_cent --balance 15 --period 6m
python main.py --mode wfa            --tf H1 --broker headway_cent --balance 15
python main.py --mode demo           --tf H1 --broker headway_cent --balance 15
python main.py --mode live           --tf H1 --broker headway_cent --balance 15

M15 — Headway Cent:

python main.py --mode process        --tf M15
python main.py --mode optimize       --tf M15 --broker headway_cent --balance 15 --trials 600
python main.py --mode train          --tf M15 --broker headway_cent --balance 15
python main.py --mode sensitivity    --tf M15 --broker headway_cent --balance 15
python main.py --mode train_tcn      --tf M15 --broker headway_cent --epochs 100
python main.py --mode report         --tf M15 --broker headway_cent --balance 15
python main.py --mode sync_validate  --tf M15 --broker headway_cent --balance 15 --period 3m
python main.py --mode demo           --tf M15 --broker headway_cent --balance 15
python main.py --mode live           --tf M15 --broker headway_cent --balance 15

M5 — Headway Cent:

python main.py --mode process        --tf M5
python main.py --mode optimize       --tf M5 --broker headway_cent --balance 15 --trials 1000
python main.py --mode train          --tf M5 --broker headway_cent --balance 15
python main.py --mode sensitivity    --tf M5 --broker headway_cent --balance 15
python main.py --mode train_tcn      --tf M5 --broker headway_cent --epochs 100
python main.py --mode report         --tf M5 --broker headway_cent --balance 15
python main.py --mode sync_validate  --tf M5 --broker headway_cent --balance 15 --period 3m
python main.py --mode demo           --tf M5 --broker headway_cent --balance 15
python main.py --mode live           --tf M5 --broker headway_cent --balance 15

---

Signal Logic — Regime-Confirmation

Stateful Regime-Confirmation Engine

The live signal gate is the SignalEngine class (src/signal_engine.py). It is stateful — it tracks how many consecutive bars the current HMM regime has persisted, whether a trade is open, and what regime the trade was entered in.

Entry requires all three conditions to pass simultaneously:

1. Regime has persisted — at least 2 consecutive bars in the same regime (Bull, Bear, or Chop)
2. XGBoost probability ≥ TF-specific threshold
3. HMM self-transition probability ≥ TF-specific persistence floor

TF	XGBoost entry threshold	Persistence floor	MR entry prob
H1	0.58	0.65	0.55
M15	0.55	0.55	0.52
M5	0.52	0.45	0.50

TCN boost: if the TCN multiplier is < 0.85, the effective XGBoost probability is multiplied by 1.10 before checking against the threshold.

Mean Reversion entries (Chop state ≥ 2)

MR entries require BB position at an extreme: BB ≤ 0.30 → MR_BUY; BB ≥ 0.70 → MR_SELL. MR positions use 75% of the standard lot size.

Exit conditions

The engine exits an open trade when any of the following fires:

Exit reason	Condition
regime_reversal	HMM state changes from the entry regime
persistence_collapse	P(stay in regime) drops below the TF persistence floor
profit_erosion	Current P&L < 40% of peak P&L for this trade
max_hold	Trade open for H1: 24 bars / M15: 32 bars / M5: 48 bars

Trade gate requirements

A trade fires when all of the following pass:

Gate	Requirement
Regime confirmed	≥ 2 consecutive bars in regime
XGBoost prob	≥ TF threshold (boosted by TCN if multiplier < 0.85)
Persistence	P(stay in state) ≥ TF floor
Direction	BUY only in Bull (0); SELL only in Bear (1); Chop → MR only
BB confluence (MR)	MR_BUY: BB ≤ 0.30; MR_SELL: BB ≥ 0.70
Session limit	Under daily trade cap for this TF
Margin check	Sufficient free margin for the lot size
Spread viability	TP1 ≥ spread × ratio (1.5× cent / 3.0× standard)
DailyEquityGate	Floating loss < 5% AND day gain < profit-lock threshold
Global Guard	Fewer than 4 GRX positions open across all TFs

Mean Reversion in Chop — Live Safety Gates

MR signals are gated by the engine's persistence check (P(stay in Chop state) ≥ TF floor) and BB extremity, matching the backtester exactly.

Logic Audit

Every bar that does not fire a trade, the bridge logs a structured reason:

[LOGIC AUDIT] BULL | state=0 prob=0.510 bars=1 P(stay)=0.82 BB=0.44 tcn=0.97

Spread Viability Guard

Broker	Applied on	Minimum ratio
headway_cent	M5 only	TP1 ≥ 1.5× spread
standard	All timeframes	TP1 ≥ 3.0× spread

Two-Sided DailyEquityGate

Loss gate: Floating equity drops ≥ 5% → gate locks, all GRX positions closed. No new signals for the rest of the day.

Profit lock: Day gain reaches TF-specific threshold → no new entries.

TF	Profit Lock	Loss Gate
M5	20% day gain	5% loss
M15	10% day gain	5% loss
H1	10% day gain	5% loss

Both gates reset at UTC midnight.

ATR Trailing Exits

Phase	Trigger	Action
Phase 1 (break-even)	Floating P&L ≥ activation amount	SL → entry + 2×spread
Phase 2 (trail)	Activated	Trailing SL at ATR × trail multiplier

Activation amounts and trail multipliers by TF:

TF	Activation P&L	Trail Multiplier
H1	$1.50	2.5× ATR
M15	$1.50	1.5× ATR
M5	$1.00	1.5× ATR

Staged Take-Profits

Regime	TF	TP1	TP2 (Runner)	SL ATR mult
Bull / Bear	M5	0.8× ATR	1.5× ATR	1.5×
Bull / Bear	M15	1.2× ATR	2.5× ATR	2.0×
Bull / Bear	H1	1.5× ATR	3.0× ATR	2.0×
Chop (MR)	M5	0.5× ATR	—	1.05×
Chop (MR)	M15	0.8× ATR	—	1.4×
Chop (MR)	H1	1.0× ATR	—	1.4×

Full profit protection chain (all TFs):

1. Profit guard — SL → entry + 2×spread when price reaches 70% of TP1 distance
2. Break-even — runner SL → entry when TP1 fills
3. ATR trail — activates when floating P&L ≥ TF activation amount

M5-only Hybrid Scalp Protection (runs every 5 s between bars):

4. Fixed scalp target — position closed when floating P&L ≥ $4.00
5. Trailing guard — if peak P&L ≥ $2.00, close when P&L falls to ≤ 50% of peak
6. Chop-exit — all positions closed at market if HMM shifts to Chop mid-trade
7. M5 recycle — after scalp target close, re-entry is allowed on the same bar if regime unchanged and daily cap not hit

---

TCN Confidence Scorer

The TCN watches the last 100 bars of market context and outputs a confidence multiplier used by the SignalEngine. Unlike the old LSTM ensemble, the TCN never blocks trades outright — it only adjusts entry difficulty by boosting the effective XGBoost probability when the multiplier is below 0.85.

Architecture

Input (100 bars, 8 features)
  → Conv1D(64, kernel=3, dilation=1, padding=causal, relu)  + Dropout(0.3)
  → Conv1D(64, kernel=3, dilation=2, padding=causal, relu)  + Dropout(0.3)
  → Conv1D(64, kernel=3, dilation=4, padding=causal, relu)  + Dropout(0.3)
  → Conv1D(64, kernel=3, dilation=8, padding=causal, relu)
  → GlobalAveragePooling1D
  → Dense(32, relu) + Dropout(0.2)
  → Dense(1, sigmoid)   ← raw confidence [0, 1]
  → temperature calibration
  → multiplier = 1.3 − (confidence × 0.6)   ← maps to [0.7, 1.3]

Causal padding ensures no future bar data leaks into the prediction. Each dilation doubles the receptive field without extra parameters.

Input features: log_return, volatility, rsi_normalized, atr_normalized, volume_ratio, bb_position, gmm_vol_cluster, dist_from_sma50.

Confidence multiplier mapping

Raw confidence	Multiplier	Effect on SignalEngine
1.0 (very confident)	0.70	Multiplier < 0.85 → XGBoost prob boosted ×1.10
0.67	1.00	No boost applied
0.0 (very uncertain)	1.30	No boost (multiplier ≥ 0.85)

The TCN multiplier is passed directly to SignalEngine.should_enter(). When it is below 0.85, the effective XGBoost probability is multiplied by 1.10 before comparing against the entry threshold — making entry easier in high-confidence conditions.

Temperature scaling

Raw sigmoid outputs are calibrated via temperature scaling before the multiplier mapping:

logit    = log(raw_conf / (1 − raw_conf + ε))
calibrated = sigmoid(logit / T)

Temperature T	Effect
> 1.0 (default 1.5)	Softens distribution — model is more honest about uncertainty
1.0	No change from raw output
< 1.0	Sharpens — pushes confidence toward 0 or 1

Training targets

The TCN is trained on a profit-based binary target, not regime state labels:

Current HMM state	Next bar condition	Target
Bull (0)	next log-return > 0	1
Bull (0)	next log-return ≤ 0	0
Bear (1)	next log-return < 0	1
Bear (1)	next log-return ≥ 0	0
Chop (2+)	`	next return
Chop (2+)	`	next return

Training commands

# Full training (first time — ~10–20 min):
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 100 --temperature 1.5

# Fine-tune on last 2 years (recommended for weekly maintenance — ~5 min):
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 20 --fine_tune --recent_years 2

# Custom sequence length:
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 100 --seq_len 50

# Custom temperature (sharper multiplier distribution):
python main.py --mode train_tcn --tf H1 --broker headway_cent \
    --epochs 100 --temperature 1.2

TCN model files

models/tcn/
├── H1_headway_cent/
│   ├── tcn_confidence_model.keras
│   ├── tcn_feature_scaler.pkl
│   └── tcn_metadata.json        ← trained_at, seq_len, n_features, temperature
├── M15_headway_cent/  …
└── M5_headway_cent/   …

The load_tcn_classifier() helper returns None silently if no model has been trained yet — the live bridge runs with no TCN boost applied.

Important: train TCN after HMM, not before

# CORRECT order:
python main.py --mode train     --tf H1 --broker headway_cent --balance 15   # HMM first
python main.py --mode train_tcn --tf H1 --broker headway_cent                # TCN second

The TCN learns from HMM regime labels. If the HMM is re-optimised on new params, state assignments may shift — retrain the TCN afterward.

---

Automatic Data Updates & TCN Maintenance

Weekly Data Updater (WeeklyDataUpdater)

src/data_updater.py runs automatically inside the live trading loop every Sunday. It:

1. Fetches the last ~2 weeks of XAUUSD bars from MT5 for each active TF (H1: 336 bars, M15: 1344, M5: 4032)
2. Deduplicates by timestamp and appends only new rows to the raw CSV training files
3. Writes a data/raw/.last_auto_update marker so TCNMaintenanceScheduler can detect the refresh

No manual action is required. The updater is a no-op on non-Sunday days and won't re-run within the same week.

Hourly TCN Maintenance (TCNMaintenanceScheduler)

src/tcn_maintenance.py runs automatically inside the live loop once per hour. It:

1. Reads tcn_metadata.json for each active TF — checks model age (no weights loaded; fast)
2. If any model is ≥ 7 days old or was never trained, launches a fine-tune subprocess:

   python main.py --mode train_tcn --tf <TF> --broker <broker> \
       --epochs 20 --fine_tune --recent_years 2 --temperature 1.5
   

3. Also triggers a full retrain cycle when WeeklyDataUpdater marks a fresh data pull
4. Uses a lock file (models/tcn/.retrain_in_progress) to prevent concurrent retrains

A stale lock older than 2 hours is cleaned up automatically.

Guardian daily check

When --mode guardian is running it also performs a daily TCN staleness check (independent of the live loop's hourly check). If any TF's model is stale, the guardian fires the same background retrain subprocess and sends a Telegram notification.

---

Sensitivity Analysis

The sensitivity analysis sweeps Bull/Bear Z-Score cutoffs across a range on your already-trained model and shows how trades, Sharpe, drawdown, and profit factor change at each level. Note: the live signal path now uses SignalEngine — sensitivity results are informational only.

python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15

What it tests: Bull (Z_CUTOFF_BULL) and Bear (Z_CUTOFF_BEAR) cutoffs from 1.5σ to 3.0σ in steps of 0.25σ. MR (Chop) cutoffs are held constant throughout.

Output — printed table:

Z     OOS Trades  Win%    Sharpe  MaxDD%  PF      RF      Return%  Status
1.50  312         52.3    0.81    14.2    1.43    1.92    18.4
1.75  241         55.1    0.97    11.8    1.61    2.34    22.1     ← current
2.00  188         58.2    1.12    10.1    1.79    2.87    26.3     ← BEST
2.25  134         60.4    1.04    9.8     1.74    2.61    24.1
...

Output files (written to reports/):

reports/sensitivity_H1_headway_cent.csv
reports/sensitivity_H1_headway_cent.json

The JSON includes current_z, best_z, and best_sharpe for automated comparison.

When to run sensitivity analysis:

• After training a new model — confirm the hardcoded TF Z cutoff is near-optimal
• When validation Sharpe is borderline — determine whether a lower cutoff would pass

---

Interactive Research Notebook

notebooks/GoldRegimeX_Explorer.ipynb is a fully interactive Jupyter notebook for exploring, tweaking, and diagnosing the system without touching any source files. Run it from the repo root:

jupyter notebook notebooks/GoldRegimeX_Explorer.ipynb

Sections

Section	Feature	Typical runtime
1 — Data & Model Loader	Select TF, broker, balance, and WFO mode (standard / fast). Loads processed data, best Optuna params, and trained HMM + XGB models.	~5 s
2 — Equity Curve Explorer	Interactive equity curve with IS/OOS split line, drawdown panel, and full metric table. Re-runs instantly on balance/broker change.	instant
3 — WFO Window Analysis	Runs _run_wfo using the loaded best params. Plots per-window OOS score bars. Drill into any window's equity curve with the Window # slider. Shows Walk-Forward Efficiency (WFE) interpretation label.	~2–3 min
4 — Feature & Regime Explorer	Price + HMM regime overlay, regime distribution pie, GMM volatility cluster histogram, feature distributions per regime, XGB feature importance.	instant
5 — Parameter Sensitivity	Sliders for n_states, max_depth, learning_rate, reg_alpha, reg_lambda, and min_child_weight. Run Comparison re-trains and shows a side-by-side metric bar chart. Run WFO Score Comparison runs proper rolling WFO (not just a full-dataset backtest) on both baseline and modified configs.	~30–60 s / ~4–8 min WFO
6 — CV Path Inspector	Runs either WFO IS CV (per-window inner fold scores) or CPCV (C(6,2)=15 paths). Displays a per-path boxplot, per-path score bar chart, and a consistency score = n_profitable_paths / total × 100%.	~2–5 min

WFO Mode (standard vs fast)

Selected via the WFO Mode dropdown in Section 1. Stored in _CACHE and used by Sections 3, 5, and 6.

Mode	IS window	OOS window	Use case
standard	H1: 1 yr / M15: 1 yr / M5: 1 yr	H1: 90d / M15: 90d / M5: 90d	Default; thorough evaluation
fast	H1: 6 mo / M15: 6 mo / M5: 6 mo	H1: 45d / M15: 45d / M5: 30d	Quicker feedback; suits intraday regime shifts

Walk-Forward Efficiency (WFE) Labels

Section 3 automatically labels the WFE ratio after each run:

WFE	Label
≥ 0.80	✅ Excellent (OOS ≥ 80% of IS CV performance)
≥ 0.50	🟡 Acceptable (OOS ≥ 50% of IS CV performance)
≥ 0.20	🟠 Marginal — consider broader regularization
< 0.20	🔴 Poor — strong overfitting signal

---

Risk Management

Position Sizing — 1% Risk Rule

lot_size = (1% × account_balance_USD) / (ATR(14) × SL_multiplier)

Minimum lot is always 0.01. All lots rounded to 2 decimal places.

Daily Exposure Limits

Account Balance	TF	Positions per Signal	Max Trades/Day
≤ $50	H1 / M15	2 (cent) / 1 (standard)	2
≤ $50	M5	2 (cent) / 1 (standard)	4
> $50	Any	3	2–3 depending on regime

Global Guard: skip any signal if ≥ 4 GRX positions are open across all TFs.

Broker Cost Configs

Broker	Spread fraction	Commission fraction	Total round-trip
headway_cent	0.02%	0.02%	0.04%
standard	0.02%	0.01%	0.03%

---

Timeframe Configurations

Parameter	M5	M15	H1
Kalman obs_cov default	0.05	4.0	1.0
Bars/day (annualisation)	288	96	24
HMM n_states search space	{4} only	3–4	3–4
HMM persistence gate (training)	≥ 0.65	≥ 0.65	≥ 0.65
IS/OOS split	65% / 35%	65% / 35%	70% / 30%
Min OOS trades (hard floor)	120	60	20
Min OOS trades (penalty threshold)	350	140	25
SL ATR multiplier	1.5×	2.0×	2.0×
TP1 multiplier (trend)	0.8× SL	1.0× SL	1.5× SL
TP2 multiplier (runner)	1.5× SL	2.0× SL	3.0× SL
Hybrid Scalp Protection	On (5 s)	Off	Off
Fixed scalp target	$4 USD	Off	Off
Trailing guard activation	$2 peak	Off	Off
DailyEquityGate loss	5%	5%	5%
DailyEquityGate profit lock	20%	10%	10%
Model staleness gate	14 days	30 days	30 days
TCN staleness gate	7 days	7 days	7 days
M5 optimisation freshness gate	120 h	—	—
Recommended --trials	1000	600	400

M5 n_states restriction: n_states=3 is always degenerate for M5 — Bull/Chop collapse to identical means, producing 500K+ HMM transitions. n_states=2 has no Chop state and causes counter-trend signals. Optimizer uses {4} only.

H1/M15 n_states restriction: n_states=2 is banned — with only Bull/Bear states every bar is signal-eligible, creating excessive counter-trend noise. Minimum is 3.

---

Performance Metrics & Scoring

Complex Criterion Score

Score = clamp(RF, -5, 5) × 0.35
      + clamp(Sharpe, -3, 3) × 0.35
      + clamp(PF − 1, -2, 2) × 0.20     ← normalised: breakeven = 0
      + clamp(avg_payoff / $0.035, 0, 2) × 0.10
      + [M5/M15 only] return_consistency × 0.30  (cap +0.30)

Score range: approximately −5 to +5. A score > 0.5 is considered a worthwhile configuration.

Component	Weight	Clamp	Measures
Recovery Factor (RF)	0.35	[−5, 5]	Capital preservation — net profit / floating max DD
Sharpe Ratio	0.35	[−3, 3]	Return smoothness — prevents outlier-driven strategies
Profit Factor − 1 (normalised)	0.20	[−2, 2]	Trade quality — breakeven = 0, loss-only = −1
Edge / Spread ratio	0.10	[0, 2]	avg payoff vs $0.035 spread proxy
Return Consistency	+0.30 bonus cap	—	Weekly P&L stability (M5/M15 only)

Why the formula changed:
The previous RF×0.4 + PF×0.3 + Sharpe×0.3 had three problems:

1. PF = 1.0 (breakeven) contributed +0.30 — rewarding zero edge.
2. No floor on Sharpe — a Sharpe of −10 contributed −3.0, swamping RF and PF.
3. No explicit edge/spread quality term, which matters for small-account sizing.
   The new formula normalises PF so breakeven = 0, symmetrically clamps all terms,
   and adds a spread-relative edge component.

WFO Variance Penalty

The final WFO score aggregates per-window scores with an increased variance penalty:

wfo_score = median(window_scores) − 0.20 × std(window_scores)

The 0.20 multiplier (up from 0.15) aligns with the wider score range (−5 to +5) to meaningfully penalise inconsistent strategies.

All Reported Metrics

Metric	Description
Score	Complex Criterion (see above)
Sharpe Ratio	Annualised return / annualised volatility
Recovery Factor	Net profit / floating max drawdown (capped 20× for display)
Profit Factor	Gross wins / gross losses (capped 10× for display)
Expected Payoff	Mean per-trade return × account_size
Max Drawdown	Peak-to-trough closed-bar drawdown
Floating Max DD	Intra-bar adverse excursion using bar High/Low
Win Rate	Fraction of trades that closed profitable
Trade Count	Total closed trades in the window
Avg Efficiency	Mean ATR / spread on active-trade bars
Cost Efficiency	1 - (total_costs / gross_profit)
Total Payout	total_return × account_size in broker currency
Return Consistency	Weekly P&L stability: `1 - std / (std +
mr_trades	Count of MR trades (Chop-state signals)
mr_win_rate	Win rate of MR trades
mr_pnl	Cumulative log-return from MR trades only

IS/OOS Split Ratios

TF	IS	OOS
H1	70%	30%
M15	65%	35%
M5	65%	35%

---

Optimizer Anti-Overfitting Rules

Rule	Detail
OOS-only scoring	All scoring uses OOS data only — IS is never evaluated in the objective
Complex Criterion	RF_c×0.35 + Sharpe_c×0.35 + (PF−1)×0.20 + Edge_c×0.10 — all terms clamped symmetrically
Hard trade floor	OOS trades < hard minimum → score −50.0 (trial discarded)
Progressive trade penalty	OOS trades below soft threshold → score × 0.1
Payoff floor	OOS average edge < $0.035 → score × 0.1
Max drawdown gate	OOS floating DD > 20% → score −50.0
H1 DD guard	OOS floating DD > 5% → score × 0.5
IS/OOS generalisation	If IS Sharpe > 0.1 and OOS/IS Sharpe < 0.35 → score −50.0
HMM persistence gate	Any self-transition < 0.65 → score −100.0
M5 activity bonus	OOS trades > 300 → score × 1.2; OOS trades < 150 → score × 0.5
n_states restriction	M5: {4}; H1/M15: {3, 4}
No threshold search	SignalEngine thresholds are hardcoded TF constants — not Optuna parameters
Per-broker study isolation	study_headway_cent.db and study_standard.db never interfere
Rolling WFO per trial	Every Optuna trial evaluates across all rolling IS/OOS windows — not a single split
IS inner CV	Each WFO window runs TF-dependent inner time-series CV: H1=2 folds, M15=3 folds, M5=4 folds
OOS scaler consistency	OOS features are always scaled with the IS-fitted scaler (no data leakage)
State alignment	states_oos and states_is_cv are re-aligned to df_aligned.index after NaN drops
L2 regularisation	reg_lambda is searched for all TFs — XGBoost default of 1.0 is no longer forced
Pruner per TF	M5: HyperbandPruner (early-stops expensive trials); H1/M15: MedianPruner(startup=10, warmup=5)

# Hard trade floors (trials below these return -50.0 immediately):
MIN_OOS_TRADES_HARD = {"M5": 120, "M15": 60, "H1": 20}

# Soft trade floors (trials below these have score × 0.1):
TF_MIN_OOS_TRADES   = {"M5": 350, "M15": 140, "H1": 25}

# WFO trial budgets (auto-selected when --trials 250 default is used):
WFO_TRIALS = {"H1": 60, "M15": 100, "M5": 120}

Optuna Search Space per Timeframe

Parameter	H1	M15	M5
learning_rate	0.005 – 0.15	0.005 – 0.20	0.01 – 0.15
n_estimators	50 – 400 (step 50)	100 – 500 (step 50)	200 – 600 (step 50)
max_depth	3 – 7	3 – 7	2 – 4
min_child_weight	5 – 100	3 – 30	5 – 25
reg_alpha (L1)	1e-6 – 0.3	0.05 – 5.0	1.0 – 30.0
reg_lambda (L2)	0.1 – 10.0	0.5 – 15.0	1.0 – 30.0
subsample	0.5 – 0.9	0.5 – 0.9	0.55 – 0.85
colsample_bytree	0.4 – 0.9	0.5 – 1.0	0.4 – 0.8

⚠️ Delete the study DB whenever a fundamental parameter changes (e.g. after adding a new feature, changing n_states search space, or altering the score function). Old trials bias the surrogate model.

del models\study_headway_cent.db
del models\study_standard.db

---

Telegram Remote Control

Start the listener alongside your trading session:

python main.py --mode listen --broker headway_cent --balance 15

Button	Action
🚀 START TRADING	Launches --mode live using .env TF/broker/balance defaults
🛑 STOP TRADING	Terminates the running live process
📉 START OPTIMIZE (M5)	Starts / resumes M5 Optuna study
📊 BOT STATUS	Last 24 h P&L, win rate, floating positions

A nightly summary is sent at 23:55 UTC while the listener runs.

Guardian — Continuous health monitor

python main.py --mode guardian --tf M5,M15,H1 --period 3m \
    --interval 3600 --broker headway_cent --balance 15

Every hour: validates rolling Sharpe for each TF — fires Telegram alert if below 0.6.

Daily TCN check (built into the guardian loop): reads trained_at from each TCN model's metadata JSON (no weights loaded — fast). If any model is older than 7 days, fires auto-retrain as a background subprocess and Telegrams the result.

Audit — On-demand deal report

python main.py --mode audit --broker headway_cent --balance 15

Parallel Optimisation

# Open multiple terminals — each one runs an independent Optuna worker:
Terminal 1:  python main.py --mode optimize --tf M5 --broker headway_cent --trials 1000
Terminal 2:  python main.py --mode optimize --tf M5 --broker headway_cent --trials 1000

Workers share the same SQLite study safely via locking.

---

Multi-TF Live Trading

Each timeframe runs as an independent Python process with its own magic number:

TF	Magic Number	Comment format
H1	123456	GRX_H1_TREND_BUY_s0_tp1
M15	123457	GRX_M15_TREND_SELL_s1_tp2
M5	123458	GRX_M5_MR_BUY_s2_tp1

Global Exposure Guard

Before any signal, the bridge counts all open GRX positions across all TFs. If ≥ 4 positions are open, the signal is skipped.

Starting multiple TFs

# Terminal 1
python main.py --mode live --tf H1 --broker headway_cent --balance 15

# Terminal 2
python main.py --mode live --tf M15 --broker headway_cent --balance 15

# Terminal 3
python main.py --mode live --tf M5 --broker headway_cent --balance 15

At UTC midnight each bridge sends a daily P&L audit to Telegram.

---

MQL5 EA (Alternative Execution)

mql5/GoldRegimeX.mq5 is a self-contained MT5 Expert Advisor that loads the ONNX model directly inside MT5 and replicates the same regime → signal → risk logic in MQL5.

Do not run the EA and the Python bridge for the same TF simultaneously.

To use the EA:

1. python main.py --mode export --tf H1 --broker headway_cent
2. Copy mql5/GoldRegimeX.mq5 and the .onnx file to MQL5/Experts/
3. Compile in MetaEditor (F7) and attach to the XAUUSD chart

---

Walk-Forward Optimization (WFO)

The Optuna optimizer evaluates every trial using Rolling Walk-Forward Optimization (_run_wfo in src/optimizer.py). This is distinct from the post-training --mode wfa diagnostic command.

How it works

Full dataset (e.g., 10 years of H1 bars)
  │
  ├── Window 1: IS=[Y1–Y2]  embargo  OOS=[Y2 Q3+90d]
  ├── Window 2: IS=[Y1+90d–Y2+90d]  embargo  OOS=[Y2 Q3+180d]
  ├── Window 3: ...
  └── Window N
  │
  For each window:
    1. HMM fitted on IS only (no lookahead)
    2. IS HMM applied to OOS bars via predict_states
    3. OOS features scaled with IS-fitted scaler (no data leakage)
    4. Inner IS cross-validation (TF-dependent folds) for consistency check
    5. Final XGBoost trained on full IS → backtest on OOS
    6. Window score = Complex Criterion on OOS results
  │
  wfo_score = median(window_scores) − 0.20 × std(window_scores)

WFO Parameters

Standard mode (wfo_mode="standard")

TF	IS window	OOS window	Embargo	Step	Approx. windows (10yr)
H1	8 760 bars (1 yr)	2 160 bars (90d)	24 bars	2 160	~23
M15	35 040 bars (1 yr)	8 640 bars (90d)	96 bars	8 640	~23
M5	105 120 bars (1 yr)	25 920 bars (90d)	288 bars	25 920	~7

Fast mode (wfo_mode="fast")

A shorter IS window for quicker intraday regime adaptation. Available via the notebook WFO Mode dropdown or by passing wfo_mode="fast" to _run_wfo / make_objective.

TF	IS window	OOS window	Embargo	Step
H1	4 380 bars (6 mo)	1 080 bars (45d)	24 bars	1 080
M15	17 520 bars (6 mo)	4 320 bars (45d)	96 bars	4 320
M5	52 560 bars (6 mo)	8 640 bars (30d)	288 bars	8 640

Inner cross-validation folds

Each WFO window runs an inner time-series CV on the IS data to check for intra-IS consistency before accepting the window score. The fold count is TF-dependent:

TF	Inner CV folds (CV_FOLDS)
H1	2
M15	3
M5	4

A deeply negative mean IS CV Sharpe (< −1.0) halves the window score to penalise IS noise memorisation.

Walk-Forward Efficiency (WFE)

WFE = mean(OOS Sharpe across windows) / mean(IS CV Sharpe across windows)

WFE	Interpretation
≥ 0.80	✅ Excellent — OOS captures ≥ 80% of IS performance
≥ 0.50	🟡 Acceptable — robust enough for live use
≥ 0.20	🟠 Marginal — consider broader regularization
< 0.20	🔴 Poor — strong overfitting signal; re-optimise

---

Walk-Forward Analysis & Staleness Gate

Walk-Forward Analysis

Full dataset
  ├── Window 1:  Train [Y1–Y2]       → Test [Y2 Q3]
  ├── Window 2:  Train [Y1 Q3–Y2 Q3] → Test [Y3 Q1]
  └── Aggregate: WFE = mean(OOS Sharpe) / mean(IS Sharpe)

WFE	Interpretation
≥ 60%	Robust — safe to go live
50–60%	Acceptable — monitor closely
< 50%	Fragile — model curve-fits specific years

python main.py --mode wfa --tf H1 --broker headway_cent --balance 15

Model Staleness Gate

TF	Max HMM/XGB model age	Max TCN model age
M5	14 days	7 days
M15	30 days	7 days
H1	30 days	7 days

M5 has an additional 5-day optimisation freshness gate: m5_meta_<broker>.json must exist and be < 120 hours old before going live.

Bypass the gate for demo testing:

python main.py --mode demo --tf M5 --broker headway_cent --balance 15 --skip_stale_check

Recommended Maintenance Schedule

# Weekly (M5):
python main.py --mode sync_validate --tf M5 --period 3m --broker headway_cent --balance 15
python main.py --mode wfa           --tf M5 --broker headway_cent --balance 15
# If needed:
python main.py --mode optimize      --tf M5 --broker headway_cent --balance 15 --trials 1000
python main.py --mode train         --tf M5 --broker headway_cent --balance 15
python main.py --mode train_tcn     --tf M5 --broker headway_cent --epochs 20 --fine_tune

# Monthly (H1 / M15):
python main.py --mode wfa       --tf H1 --broker headway_cent --balance 15
python main.py --mode optimize  --tf H1 --broker headway_cent --balance 15 --trials 400
python main.py --mode train     --tf H1 --broker headway_cent --balance 15
python main.py --mode train_tcn --tf H1 --broker headway_cent --epochs 20 --fine_tune

The live bridge handles TCN fine-tuning automatically (every 7 days in background). Manual retraining above is only needed after a full HMM/XGBoost re-optimise.

---

Troubleshooting

Problem	Cause	Fix
ConnectionError: Could not connect to MT5	Terminal not running	Open MT5 and log in first
FileNotFoundError: hmm_model_H1_headway_cent.pkl	Models not trained	--mode train --tf H1 --broker headway_cent
FileNotFoundError: m5_meta_headway_cent.json	M5 not optimised	--mode optimize --tf M5 --broker headway_cent
WARNING: No Optuna study found	Wrong broker or deleted DB	--mode optimize with matching --broker
ERROR: Degenerate HMM during train	Stale params	Delete study DB, re-optimise, then train
Validation FAIL every day	Model stale or data ends before sync window	Export fresh CSV from MT5, re-run full pipeline
No signals firing	XGBoost prob or persistence below thresholds	Check [LOGIC AUDIT] logs for prob= and P(stay)= values; consider re-training
No entry after regime change	bars_in_regime < 2	Engine waits for confirmation — normal on first bar of a new regime
TCN multiplier always ~1.3 (hardening)	TCN undertrained or wrong features	Retrain: --mode train then --mode train_tcn --epochs 100
[TCN HEALTH] FAILED at startup	Corrupt model file	Delete models/tcn/<TF>_<broker>/ and retrain
ValueError: Input shape (None, 100, 4)	TCN trained without deriving features	Fixed in current version — retrain TCN
TCN auto-retrain triggered in logs	Model ≥ 7 days old	Normal — background fine-tune fires automatically
WFA shows many ❌ folds	Model curve-fits specific years	Loosen regularisation, increase trials, add more training data
Order failed: retcode=10006	No broker connection	Check MT5 connection indicator
Order failed: retcode=10015	Price moved past deviation	Will retry next bar; elevated deviation auto-applies on high-vol
[CONFLICT] warning at startup	GRX positions from EA already open	Stop EA / other process before starting bridge
Double positions	MQL5 EA running alongside Python bridge	Remove GoldRegimeX.mq5 EA from chart
Telegram errors in log	Wrong token or unconfigured	Check .env; regenerate via @BotFather

Emergency stop: Press Ctrl+C. Open positions remain open — close them manually from the MT5 Trade tab.

---

Security Notes

• Never commit .env — it is in .gitignore
• .env.example contains only placeholders
• ALLOWED_USER_ID is the single security gate for all Telegram commands
• If credentials are accidentally committed, immediately revoke the bot token via @BotFather /revoke
• The listener uses Telegram long-polling — no public webhook or open port required

---

State Labels

Hardcoded across all Python modules and the MQL5 EA:

Label	Integer	Signal eligibility
Bull	0	BUY (trend)
Bear	1	SELL (trend)
Chop	2	MR_BUY / MR_SELL only (n_states = 3)
Chop_Low	2	MR_BUY / MR_SELL only (n_states = 4)
Chop_High	3	MR_SELL only with higher cutoff (n_states = 4)

Model Files Reference

models/
├── hmm_model_H1_headway_cent.pkl
├── hmm_model_H1_standard.pkl
├── hmm_model_M15_headway_cent.pkl
├── hmm_model_M5_headway_cent.pkl
├── xgb_ensemble_H1_headway_cent.pkl
├── xgb_ensemble_M15_headway_cent.pkl
├── xgb_ensemble_M5_headway_cent.pkl
├── xgb_model_H1_headway_cent.onnx               ← MQL5 EA uses this
├── study_headway_cent.db               ← Optuna trials (per-broker, never shared)
├── study_standard.db
├── m5_meta_headway_cent.json           ← M5 optimisation freshness gate
└── tcn/
    ├── H1_headway_cent/
    │   ├── tcn_confidence_model.keras
    │   ├── tcn_feature_scaler.pkl
    │   └── tcn_metadata.json           ← trained_at, seq_len, n_features, temperature
    ├── M15_headway_cent/  …
    └── M5_headway_cent/   …

---

Changelog

Latest — WFO Bugs, Scoring Formula & Optimizer Overhaul

🐛 Critical Bug Fixes in _run_wfo (src/optimizer.py)

#	Bug	Impact	Fix
1	OOS feature scaling leakage — OOS prepare_features fitted a fresh scaler on OOS-only data, placing IS and OOS features in different scaling spaces	Silent feature space mismatch; XGBoost sees inconsistent scales between IS training and OOS inference	Capture IS scaler (scaler_is) and pass feature_scaler=scaler_is to the OOS call
2	states_oos length mismatch — states_oos (same length as df_oos_slice) passed to vectorized_backtest alongside df_oos_aligned (shorter after NaN drops)	Silent misalignment: wrong regime states assigned to the wrong OOS bars	Re-align via states_oos_aligned = states_oos[df_oos_slice.index.isin(df_oos_aligned.index)]
3	IS CV wrong state indexing — _val_st = states_is[df_is_slice.index.isin(X_cv_val.index)] used the full IS slice as base instead of df_is_aligned	Wrong states fed to IS cross-validation backtest; CV Sharpes unreliable	Build states_is_aligned once per window (aligned to df_is_aligned) and index through it
4	Fixed 2-fold IS CV for all TFs — a 2-fold split on 105K-bar M5 IS windows produces folds of ~6 months each, giving only one data point for IS consistency	Coarse IS consistency check fails to detect within-IS regime shifts	Add CV_FOLDS = {"H1": 2, "M15": 3, "M5": 4} and use TimeSeriesSplit(n_splits=CV_FOLDS[tf])

📐 Revised Scoring Formula (_score_result)

	Old	New
Formula	RF×0.4 + PF×0.3 + Sharpe×0.3	RF_c×0.35 + Sharpe_c×0.35 + (PF−1)×0.20 + Edge_c×0.10
RF clamp	min(RF, 5.0) (one-sided)	clamp(RF, −5, 5) (symmetric)
PF normalisation	Raw PF (breakeven = +0.30 contribution)	PF − 1 (breakeven = 0)
Sharpe floor	None (−10 Sharpe → −3.0)	clamp(Sharpe, −3, 3)
Edge term	Absent	clamp(avg_payoff / $0.035, 0, 2) × 0.10
Consistency bonus cap	+0.50	+0.30
WFO variance penalty	0.15 × std	0.20 × std
CPCV inline formula	RF×0.4 + PF×0.3 + Sharpe×0.3, cap=2.0	RF×0.35 + Sharpe×0.35 + (PF−1)×0.20, cap=3.5

⚙️ Optuna Search Space Improvements

• reg_lambda (L2 regularisation) added for all TFs — was completely absent before, forcing XGBoost to use a hard-coded default of 1.0
• WFO_TRIALS raised ~20%: H1: 50 → 60, M15: 80 → 100, M5: 100 → 120 (compensates for wider search space)
• MIN_OOS_TRADES_HARD M15 raised: 30 → 60 (30 trades / 90-day OOS window is statistically meaningless)
• HyperbandPruner for M5 (early-stops expensive trials); MedianPruner(startup=10, warmup=5) for H1/M15
• Per-TF hyperparameter range revisions:

Parameter	Change	Rationale
H1 learning_rate	0.01–0.20 → 0.005–0.15	Lower floor allows slow-learning deep trees
H1 n_estimators	50–300 → 50–400	Slow lr needs more trees to converge
H1 max_depth	3–8 → 3–7	Depth-8 on H1 overfits leaf splits
H1 min_child_weight	1–50 → 5–100	Floor of 5 skips clearly overfit configs
M15 learning_rate	0.01–0.30 → 0.005–0.20	Wider low end; ceiling reduced
M15 max_depth	3–8 → 3–7	Same as H1 rationale
M15 reg_alpha	0.01–1.20 → 0.05–5.0	Previous ceiling was too tight
M15 min_child_weight	1–15 → 3–30	More representative of M15 bar count
M5 learning_rate	0.01–0.30 → 0.01–0.15	High lr + shallow trees memorises IS noise
M5 n_estimators	100–500 → 200–600	Slow lr needs more iterations; floor raised
M5 max_depth	2–3 → 2–4	Allows depth-4 for 4-state regime fitting
M5 min_child_weight	1–15 → 5–25	Large bar count requires larger MCW
M5 subsample	0.6–1.0 → 0.55–0.85	Cap at 0.85 prevents fully-correlated trees

🆕 New Constants (src/optimizer.py)

Constant	Value	Purpose
WFO_PARAMS_FAST	6-month IS windows	Fast WFO mode for intraday regime detection
WFO_TRIALS_FAST	{"H1": 60, "M15": 100, "M5": 120}	Companion trial budgets for fast mode
CV_FOLDS	{"H1": 2, "M15": 3, "M5": 4}	TF-dependent inner CV fold counts
_N_PATHS	_n_paths_total() precomputed at import	Replaces per-iteration math.comb calls
wfo_mode param	"standard" default	Selects WFO_PARAMS or WFO_PARAMS_FAST in _run_wfo and make_objective

📓 Notebook Additions (notebooks/GoldRegimeX_Explorer.ipynb)

Section	Change
Section 1	Added wfo_mode dropdown (standard / fast) stored in _CACHE; imported WFO_PARAMS_FAST, CV_FOLDS, compute_cpcv_score
Section 3	Mode-aware WFO config selection; WFE interpretation labels (✅/🟡/🟠/🔴); updated score label 0.15×std → 0.20×std
Section 5	depth5_w max raised to 7; added reg_lambda5_w (FloatLogSlider) and min_child_weight5_w (IntSlider); new Run WFO Score Comparison button
Section 6 (new)	CV Path Inspector — CPCV or WFO IS CV, per-path boxplot + bar chart, consistency score

---

Notebook Merge Conflict Resolution & API Fix

🐛 Notebook JSON Was Invalid (Committed Merge Conflicts)

The notebook file (notebooks/GoldRegimeX_Explorer.ipynb) was committed with 5 unresolved git merge conflict markers (<<<<<<<, =======, >>>>>>>), making it invalid JSON that could not be opened or executed in Jupyter.

Conflicts resolved:

Conflict	Cell	Resolution
1	cell01 imports	Kept HEAD ensemble API (load_xgb_ensemble, get_predictions_ensemble) + added OTHER's WFO_PARAMS_FAST, CV_FOLDS, compute_cpcv_score to optimizer import
2	cell03 Section 1 loader	Took OTHER — adds wfo_mode_w dropdown and stores wfo_mode in _CACHE
3	cell07 Section 3 WFO	Took OTHER — uses WFO_PARAMS_FAST, WFE interpretation labels, updated 0.20×std label
4	cell11 Section 5 start	Took OTHER — adds reg_lambda5_w, min_child_weight5_w, run_wfo5_btn, _run_wfo5_comparison
5	cell11 Section 5/6 body	Took OTHER — full _run_cv_inspector with CPCV and WFO IS CV support

🔧 Notebook Ensemble API Fix

Sections 1 (cell03), 5 (cell11) were updated to use the correct vol-bucketed ensemble API throughout, matching the optimizer and training pipeline:

Old (single-model)	New (ensemble)
load_xgb(path) → (model, metrics)	load_xgb_ensemble(path) → (models, thresholds, metrics)
train_xgb(X, y, **kws) → (model, metrics)	train_xgb_ensemble(X, y, **kws) → (models, thresholds, metrics)
get_predictions(model, X) → (preds, probs)	get_predictions_ensemble(models, thresholds, X) → (preds, probs)

Without this fix, the notebook's loaded probabilities came from a single global XGBoost model instead of the three vol-bucket models used by the optimizer and live trader, making Section 2 equity curves and Section 3 WFO scores inconsistent with production results.