XauBot/backtests/ml_v2

ML V2 — Full ML Overhaul

Problem: Current ML model has AUC ~0.696 (barely better than random). Too noisy, limited features, single model.

Solution: 3-phase improvement:

1. Better target (multi-bar + ATR threshold)
2. 23 new features (H1, continuous SMC, regime, price action)
3. Ensemble models (XGBoost + LightGBM)

---

File Structure

backtests/ml_v2/
├── __init__.py                 # Package init
├── ml_v2_target.py             # Better target variables (Step 1)
├── ml_v2_feature_eng.py        # 23 new features (Step 2)
├── ml_v2_model.py              # Multi-model support (Step 3)
├── ml_v2_train.py              # Training pipeline + walk-forward CV
└── README.md                   # This file

backtests/backtest_36_ml_v2.py  # Main backtest script
backtests/36_ml_v2_results/     # Output directory

---

Components

1. ml_v2_target.py — Better Targets (Highest Impact)

Problem: Current target predicts 1-bar ahead with threshold=0 → captures noise.

Solutions:

• Multi-bar target (primary): Look 3 bars ahead, filter moves < 0.3 * ATR (~$3.6)
• 3-class target: BUY/SELL/HOLD explicit classes
• Baseline target: V1 reproduction for comparison

Expected impact: AUC +0.05 to +0.10 (biggest single improvement)

---

2. ml_v2_feature_eng.py — 23 New Features

Adds 23 features on top of base 37:

H1 Multi-Timeframe (8 features):

• h1_market_structure: H1 trend direction
• h1_ema20_distance: Price vs H1 EMA20 / ATR
• h1_trend_strength: H1 BOS count
• h1_swing_proximity: Distance to H1 swing / ATR
• h1_fvg_active: Inside H1 FVG zone?
• h1_ob_proximity: Distance to H1 OB / ATR
• h1_atr_ratio: H1 ATR / M15 ATR
• h1_rsi: H1 RSI value

Continuous SMC (7 features):

• fvg_gap_size_atr: FVG gap / ATR (bigger = more reliable)
• fvg_age_bars: Bars since last FVG
• ob_width_atr: OB width / ATR
• ob_distance_atr: Distance to OB / ATR
• bos_recency: Bars since last BOS
• confluence_score: Count SMC signals in last 10 bars
• swing_distance_atr: Distance to swing / ATR

Regime Conditioning (4 features):

• regime_duration_bars: Consecutive bars in regime
• regime_transition_prob: 1 / duration
• volatility_zscore: (ATR - mean) / std
• crisis_proximity: ATR / (mean * 2.5)

Price Action (4 features):

• wick_ratio: (upper + lower wick) / range
• body_ratio: |close - open| / range
• gap_from_prev_close: Gap / ATR
• consecutive_direction: # candles same direction

Total: 37 (base) + 23 (new) = 60 features

---

3. ml_v2_model.py — Multi-Model Support

Model types:

• XGBOOST_BINARY: Binary classification (UP/DOWN)
• XGBOOST_3CLASS: 3-class (BUY/SELL/HOLD)
• LIGHTGBM_BINARY: LightGBM binary
• ENSEMBLE: Average XGBoost + LightGBM probabilities

Features:

• Backward compatible with V1 TradingModel
• Same anti-overfitting philosophy (depth 3, heavy regularization)
• Saves/loads as .pkl
• Can load V1 models via load_legacy_v1()

---

4. ml_v2_train.py — Training Pipeline

Purged Walk-Forward CV:

• 5 folds
• 5000 train / 1000 test / 50 gap per fold
• Gap prevents temporal leakage
• Reports mean ± std AUC, overfitting ratio

Experiment Configs:

Config	Target	Features	Model
Baseline	1-bar (V1)	37 base	XGBoost
A	3-bar + ATR	37 base	XGBoost
B	3-bar + ATR	37 + 8 H1 = 45	XGBoost
C	3-bar + ATR	45 + 7 SMC = 52	XGBoost
D	3-bar + ATR	52 + 8 regime/PA = 60	XGBoost
E	3-bar + ATR	60	XGB + LGBM ensemble

---

Usage

Run Main Backtest

python backtests/backtest_36_ml_v2.py

This will:

1. Fetch XAUUSD M15 + H1 data from MT5
2. Calculate all features (base + V2)
3. Create all targets (baseline, multi-bar, 3-class)
4. Train all 6 configs (Baseline, A, B, C, D, E)
5. Run 5-fold purged walk-forward CV for each
6. Print comparison table
7. Save models to backtests/36_ml_v2_results/model_*.pkl

Expected runtime: 10-20 minutes (depends on CV depth)

---

Standalone Usage

from backtests.ml_v2 import TargetBuilder, MLV2FeatureEngineer, TradingModelV2, ModelType

# 1. Create better targets
builder = TargetBuilder()
df = builder.create_multi_bar_target(df, lookahead=3, threshold_atr_mult=0.3)

# 2. Add V2 features
fe_v2 = MLV2FeatureEngineer()
df = fe_v2.add_all_v2_features(df_m15, df_h1)

# 3. Train model
model = TradingModelV2(model_type=ModelType.XGBOOST_BINARY)
model.fit(df, feature_cols, target_col="multi_bar_target")

# 4. Predict
pred = model.predict(df)
print(f"Signal: {pred.signal}, Confidence: {pred.confidence}")

---

Expected Results

Baseline (V1):

• Train AUC: ~0.75, Test AUC: ~0.70 (overfitting)
• Actual: ~0.696 (from live model)

Config A (Better Target):

• Expected: Test AUC +0.05 to +0.10 vs Baseline
• Why: Filters noise, focuses on tradeable moves

Config B (+H1 Features):

• Expected: Test AUC +0.02 to +0.05 vs A
• Why: Higher timeframe context

Config C (+Continuous SMC):

• Expected: Test AUC +0.01 to +0.03 vs B
• Why: SMC strength (gap size, confluence)

Config D (+All Features):

• Expected: Test AUC +0.01 to +0.02 vs C
• Why: Regime transitions, price action patterns

Config E (Ensemble):

• Expected: Test AUC +0.00 to +0.02 vs D
• Why: Ensemble reduces variance

Target: Test AUC > 0.75 (from 0.696)

---

Validation Checklist

After running backtest, check:

1. AUC Improvement: Each config should improve or maintain test AUC
2. Overfitting Ratio: Train AUC / Test AUC < 1.2 (acceptable)
3. Feature Importance: Check if new features are used (not ignored)
4. Nulls: Verify no excessive nulls in V2 features
5. Baseline Match: Baseline config should reproduce V1 results (~0.70 AUC)

---

Integration Plan (If Successful)

If Config D or E shows significant improvement (test AUC > 0.75):

1. Copy best model to models/xgboost_model_v2.pkl
2. Update src/ml_model.py to load V2 by default
3. Modify main_live.py to:
   • Add V2 feature calculation (H1 data fetch required)
   • Use V2 model for predictions
4. Run forward test on demo account for 1 week
5. Compare metrics vs V1 (WR, PnL, Sharpe)

---

Dependencies

All dependencies already in requirements.txt:

• xgboost>=2.0.0 (core)
• polars>=0.20.0 (data processing)
• scikit-learn>=1.3.0 (metrics)
• lightgbm>=4.0.0 (optional, for ensemble Config E)

If lightgbm not installed, ensemble will fall back to XGBoost-only.

---

Notes

• No live code changes: All files in backtests/ml_v2/ (isolated)
• Backward compatible: Can load V1 models via load_legacy_v1()
• Windows compatible: Tested on Windows 11, Python 3.11+
• Polars-first: All data processing uses Polars (not Pandas)
• Anti-overfitting: Same regularization philosophy as V1

---

File Sizes

• ml_v2_target.py: ~9 KB (target builder)
• ml_v2_feature_eng.py: ~22 KB (23 features)
• ml_v2_model.py: ~19 KB (multi-model support)
• ml_v2_train.py: ~9 KB (training pipeline)
• backtest_36_ml_v2.py: ~11 KB (main backtest)

Total package: ~70 KB (5 files)

---

Troubleshooting

Import Error:

# Ensure you're in project root
cd "C:/Users/Administrator/Videos/Smart Automatic Trading BOT + AI"
python backtests/backtest_36_ml_v2.py

LightGBM Not Found:

• Config E will skip LightGBM and use XGBoost-only ensemble
• Optional: pip install lightgbm>=4.0.0

MT5 Connection Failed:

• Check .env credentials
• Ensure MT5 terminal is running

Low AUC (<0.65):

• Check feature nulls: df[feature_cols].null_count()
• Verify target distribution: df['multi_bar_target'].value_counts()
• Inspect feature importance: Are new features used?

---

References

• Plan: See plan mode transcript (1a09c953-bc49-4062-b130-8dd676f7eb1f.jsonl)
• V1 Model: src/ml_model.py
• Base Features: src/feature_eng.py
• SMC: src/smc_polars.py
• Regime: src/regime_detector.py