GifariKemal
e8355b3f62
- Dark mode: class-based theme toggle with localStorage persistence and flash prevention - Trade History (/trades): paginated table, stats cards, equity curve chart with DB API endpoints - Backtest Viewer (/backtests): log parser for 35 backtest results, sidebar + detail + comparison tabs - Model Insights: dashboard card + dialog showing feature importance, regime distribution, training history - Alert/Signal Log (/alerts): signal stats, filterable table with execution tracking - API: 8 new endpoints with psycopg2 DB connection pool - Dark mode sweep across books page, about dialog, and all dashboard components - Architecture docs rewritten with Mermaid diagrams (23 docs) - README and FEATURES.md rewritten bilingual (Indonesian + English) - main_live.py: write model_metrics.json on startup and retrain Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
285 lines
7.9 KiB
Markdown
285 lines
7.9 KiB
Markdown
# ML V2 — Full ML Overhaul
|
|
|
|
**Problem:** Current ML model has AUC ~0.696 (barely better than random). Too noisy, limited features, single model.
|
|
|
|
**Solution:** 3-phase improvement:
|
|
1. Better target (multi-bar + ATR threshold)
|
|
2. 23 new features (H1, continuous SMC, regime, price action)
|
|
3. Ensemble models (XGBoost + LightGBM)
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
```
|
|
backtests/ml_v2/
|
|
├── __init__.py # Package init
|
|
├── ml_v2_target.py # Better target variables (Step 1)
|
|
├── ml_v2_feature_eng.py # 23 new features (Step 2)
|
|
├── ml_v2_model.py # Multi-model support (Step 3)
|
|
├── ml_v2_train.py # Training pipeline + walk-forward CV
|
|
└── README.md # This file
|
|
|
|
backtests/backtest_36_ml_v2.py # Main backtest script
|
|
backtests/36_ml_v2_results/ # Output directory
|
|
```
|
|
|
|
---
|
|
|
|
## Components
|
|
|
|
### 1. `ml_v2_target.py` — Better Targets (Highest Impact)
|
|
|
|
**Problem:** Current target predicts 1-bar ahead with threshold=0 → captures noise.
|
|
|
|
**Solutions:**
|
|
- **Multi-bar target** (primary): Look 3 bars ahead, filter moves < 0.3 * ATR (~$3.6)
|
|
- **3-class target**: BUY/SELL/HOLD explicit classes
|
|
- **Baseline target**: V1 reproduction for comparison
|
|
|
|
**Expected impact:** AUC +0.05 to +0.10 (biggest single improvement)
|
|
|
|
---
|
|
|
|
### 2. `ml_v2_feature_eng.py` — 23 New Features
|
|
|
|
Adds 23 features on top of base 37:
|
|
|
|
**H1 Multi-Timeframe (8 features):**
|
|
- `h1_market_structure`: H1 trend direction
|
|
- `h1_ema20_distance`: Price vs H1 EMA20 / ATR
|
|
- `h1_trend_strength`: H1 BOS count
|
|
- `h1_swing_proximity`: Distance to H1 swing / ATR
|
|
- `h1_fvg_active`: Inside H1 FVG zone?
|
|
- `h1_ob_proximity`: Distance to H1 OB / ATR
|
|
- `h1_atr_ratio`: H1 ATR / M15 ATR
|
|
- `h1_rsi`: H1 RSI value
|
|
|
|
**Continuous SMC (7 features):**
|
|
- `fvg_gap_size_atr`: FVG gap / ATR (bigger = more reliable)
|
|
- `fvg_age_bars`: Bars since last FVG
|
|
- `ob_width_atr`: OB width / ATR
|
|
- `ob_distance_atr`: Distance to OB / ATR
|
|
- `bos_recency`: Bars since last BOS
|
|
- `confluence_score`: Count SMC signals in last 10 bars
|
|
- `swing_distance_atr`: Distance to swing / ATR
|
|
|
|
**Regime Conditioning (4 features):**
|
|
- `regime_duration_bars`: Consecutive bars in regime
|
|
- `regime_transition_prob`: 1 / duration
|
|
- `volatility_zscore`: (ATR - mean) / std
|
|
- `crisis_proximity`: ATR / (mean * 2.5)
|
|
|
|
**Price Action (4 features):**
|
|
- `wick_ratio`: (upper + lower wick) / range
|
|
- `body_ratio`: |close - open| / range
|
|
- `gap_from_prev_close`: Gap / ATR
|
|
- `consecutive_direction`: # candles same direction
|
|
|
|
**Total:** 37 (base) + 23 (new) = **60 features**
|
|
|
|
---
|
|
|
|
### 3. `ml_v2_model.py` — Multi-Model Support
|
|
|
|
**Model types:**
|
|
- `XGBOOST_BINARY`: Binary classification (UP/DOWN)
|
|
- `XGBOOST_3CLASS`: 3-class (BUY/SELL/HOLD)
|
|
- `LIGHTGBM_BINARY`: LightGBM binary
|
|
- `ENSEMBLE`: Average XGBoost + LightGBM probabilities
|
|
|
|
**Features:**
|
|
- Backward compatible with V1 TradingModel
|
|
- Same anti-overfitting philosophy (depth 3, heavy regularization)
|
|
- Saves/loads as `.pkl`
|
|
- Can load V1 models via `load_legacy_v1()`
|
|
|
|
---
|
|
|
|
### 4. `ml_v2_train.py` — Training Pipeline
|
|
|
|
**Purged Walk-Forward CV:**
|
|
- 5 folds
|
|
- 5000 train / 1000 test / 50 gap per fold
|
|
- Gap prevents temporal leakage
|
|
- Reports mean ± std AUC, overfitting ratio
|
|
|
|
**Experiment Configs:**
|
|
| Config | Target | Features | Model |
|
|
|--------|--------|----------|-------|
|
|
| Baseline | 1-bar (V1) | 37 base | XGBoost |
|
|
| **A** | 3-bar + ATR | 37 base | XGBoost |
|
|
| **B** | 3-bar + ATR | 37 + 8 H1 = 45 | XGBoost |
|
|
| **C** | 3-bar + ATR | 45 + 7 SMC = 52 | XGBoost |
|
|
| **D** | 3-bar + ATR | 52 + 8 regime/PA = 60 | XGBoost |
|
|
| **E** | 3-bar + ATR | 60 | XGB + LGBM ensemble |
|
|
|
|
---
|
|
|
|
## Usage
|
|
|
|
### Run Main Backtest
|
|
|
|
```bash
|
|
python backtests/backtest_36_ml_v2.py
|
|
```
|
|
|
|
This will:
|
|
1. Fetch XAUUSD M15 + H1 data from MT5
|
|
2. Calculate all features (base + V2)
|
|
3. Create all targets (baseline, multi-bar, 3-class)
|
|
4. Train all 6 configs (Baseline, A, B, C, D, E)
|
|
5. Run 5-fold purged walk-forward CV for each
|
|
6. Print comparison table
|
|
7. Save models to `backtests/36_ml_v2_results/model_*.pkl`
|
|
|
|
**Expected runtime:** 10-20 minutes (depends on CV depth)
|
|
|
|
---
|
|
|
|
### Standalone Usage
|
|
|
|
```python
|
|
from backtests.ml_v2 import TargetBuilder, MLV2FeatureEngineer, TradingModelV2, ModelType
|
|
|
|
# 1. Create better targets
|
|
builder = TargetBuilder()
|
|
df = builder.create_multi_bar_target(df, lookahead=3, threshold_atr_mult=0.3)
|
|
|
|
# 2. Add V2 features
|
|
fe_v2 = MLV2FeatureEngineer()
|
|
df = fe_v2.add_all_v2_features(df_m15, df_h1)
|
|
|
|
# 3. Train model
|
|
model = TradingModelV2(model_type=ModelType.XGBOOST_BINARY)
|
|
model.fit(df, feature_cols, target_col="multi_bar_target")
|
|
|
|
# 4. Predict
|
|
pred = model.predict(df)
|
|
print(f"Signal: {pred.signal}, Confidence: {pred.confidence}")
|
|
```
|
|
|
|
---
|
|
|
|
## Expected Results
|
|
|
|
**Baseline (V1):**
|
|
- Train AUC: ~0.75, Test AUC: ~0.70 (overfitting)
|
|
- Actual: ~0.696 (from live model)
|
|
|
|
**Config A (Better Target):**
|
|
- Expected: Test AUC +0.05 to +0.10 vs Baseline
|
|
- Why: Filters noise, focuses on tradeable moves
|
|
|
|
**Config B (+H1 Features):**
|
|
- Expected: Test AUC +0.02 to +0.05 vs A
|
|
- Why: Higher timeframe context
|
|
|
|
**Config C (+Continuous SMC):**
|
|
- Expected: Test AUC +0.01 to +0.03 vs B
|
|
- Why: SMC strength (gap size, confluence)
|
|
|
|
**Config D (+All Features):**
|
|
- Expected: Test AUC +0.01 to +0.02 vs C
|
|
- Why: Regime transitions, price action patterns
|
|
|
|
**Config E (Ensemble):**
|
|
- Expected: Test AUC +0.00 to +0.02 vs D
|
|
- Why: Ensemble reduces variance
|
|
|
|
**Target:** Test AUC > 0.75 (from 0.696)
|
|
|
|
---
|
|
|
|
## Validation Checklist
|
|
|
|
After running backtest, check:
|
|
|
|
1. **AUC Improvement**: Each config should improve or maintain test AUC
|
|
2. **Overfitting Ratio**: Train AUC / Test AUC < 1.2 (acceptable)
|
|
3. **Feature Importance**: Check if new features are used (not ignored)
|
|
4. **Nulls**: Verify no excessive nulls in V2 features
|
|
5. **Baseline Match**: Baseline config should reproduce V1 results (~0.70 AUC)
|
|
|
|
---
|
|
|
|
## Integration Plan (If Successful)
|
|
|
|
If Config D or E shows significant improvement (test AUC > 0.75):
|
|
|
|
1. **Copy best model** to `models/xgboost_model_v2.pkl`
|
|
2. **Update `src/ml_model.py`** to load V2 by default
|
|
3. **Modify `main_live.py`** to:
|
|
- Add V2 feature calculation (H1 data fetch required)
|
|
- Use V2 model for predictions
|
|
4. **Run forward test** on demo account for 1 week
|
|
5. **Compare metrics** vs V1 (WR, PnL, Sharpe)
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
All dependencies already in `requirements.txt`:
|
|
- `xgboost>=2.0.0` (core)
|
|
- `polars>=0.20.0` (data processing)
|
|
- `scikit-learn>=1.3.0` (metrics)
|
|
- `lightgbm>=4.0.0` (optional, for ensemble Config E)
|
|
|
|
If `lightgbm` not installed, ensemble will fall back to XGBoost-only.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **No live code changes**: All files in `backtests/ml_v2/` (isolated)
|
|
- **Backward compatible**: Can load V1 models via `load_legacy_v1()`
|
|
- **Windows compatible**: Tested on Windows 11, Python 3.11+
|
|
- **Polars-first**: All data processing uses Polars (not Pandas)
|
|
- **Anti-overfitting**: Same regularization philosophy as V1
|
|
|
|
---
|
|
|
|
## File Sizes
|
|
|
|
- `ml_v2_target.py`: ~9 KB (target builder)
|
|
- `ml_v2_feature_eng.py`: ~22 KB (23 features)
|
|
- `ml_v2_model.py`: ~19 KB (multi-model support)
|
|
- `ml_v2_train.py`: ~9 KB (training pipeline)
|
|
- `backtest_36_ml_v2.py`: ~11 KB (main backtest)
|
|
|
|
**Total package**: ~70 KB (5 files)
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**Import Error:**
|
|
```bash
|
|
# Ensure you're in project root
|
|
cd "C:/Users/Administrator/Videos/Smart Automatic Trading BOT + AI"
|
|
python backtests/backtest_36_ml_v2.py
|
|
```
|
|
|
|
**LightGBM Not Found:**
|
|
- Config E will skip LightGBM and use XGBoost-only ensemble
|
|
- Optional: `pip install lightgbm>=4.0.0`
|
|
|
|
**MT5 Connection Failed:**
|
|
- Check `.env` credentials
|
|
- Ensure MT5 terminal is running
|
|
|
|
**Low AUC (<0.65):**
|
|
- Check feature nulls: `df[feature_cols].null_count()`
|
|
- Verify target distribution: `df['multi_bar_target'].value_counts()`
|
|
- Inspect feature importance: Are new features used?
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- **Plan**: See plan mode transcript (`1a09c953-bc49-4062-b130-8dd676f7eb1f.jsonl`)
|
|
- **V1 Model**: `src/ml_model.py`
|
|
- **Base Features**: `src/feature_eng.py`
|
|
- **SMC**: `src/smc_polars.py`
|
|
- **Regime**: `src/regime_detector.py`
|