buckybonez
f736f1f75b
Added research artifacts from HMM investigation: - backtest_39_h1_hmm.py — H1 vs M15 HMM comparison attempt - 39_h1_hmm_results/ — Partial backtest results - analyze_*.py — ML model and H1 feature analysis scripts - *_output.txt — Analysis outputs showing HMM degeneracy Updated: - data/risk_state.txt — Latest risk state (daily_loss: 18.77, daily_profit: 22.49) Note: Backtest #39 had import compatibility issues but led to critical discovery of alternating HMM pattern bug (fixed in c02c2e9). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
267 lines
7.7 KiB
Plaintext
267 lines
7.7 KiB
Plaintext
|
|
================================================================================
|
|
ML MODEL DEEP DIVE ANALYSIS
|
|
================================================================================
|
|
|
|
================================================================================
|
|
1. MODEL INSPECTION: xgboost_model_v2d.pkl
|
|
================================================================================
|
|
|
|
Model pickle structure:
|
|
model_type: ModelType
|
|
xgb_model: Booster
|
|
lgb_model: NoneType
|
|
feature_names: list (length=76)
|
|
confidence_threshold: float
|
|
xgb_params: dict (length=13)
|
|
lgb_params: dict (length=12)
|
|
feature_importance: dict (length=76)
|
|
train_metrics: dict (length=4)
|
|
fitted: bool
|
|
|
|
XGBoost model: <class 'xgboost.core.Booster'>
|
|
LightGBM model: <class 'NoneType'>
|
|
Total features: 76
|
|
|
|
--- TRAINING METRICS ---
|
|
xgb_train_score: 0.7385315785647933
|
|
xgb_test_score: 0.7338658466928052
|
|
train_samples: 36407
|
|
test_samples: 9052
|
|
|
|
--- XGBOOST PARAMETERS ---
|
|
objective: binary:logistic
|
|
eval_metric: auc
|
|
max_depth: 3
|
|
learning_rate: 0.05
|
|
tree_method: hist
|
|
device: cpu
|
|
min_child_weight: 10
|
|
subsample: 0.7
|
|
colsample_bytree: 0.6
|
|
reg_alpha: 1.0
|
|
reg_lambda: 5.0
|
|
gamma: 1.0
|
|
max_delta_step: 1
|
|
|
|
================================================================================
|
|
2. FEATURE IMPORTANCE RANKING
|
|
================================================================================
|
|
|
|
Total features with importance: 76
|
|
|
|
--- TOP 30 FEATURES ---
|
|
1. ob 417.354340 [M15]
|
|
2. log_returns 186.054825 [M15]
|
|
3. returns_1 167.918945 [M15]
|
|
4. ob_mitigated 127.265808 [SMC]
|
|
5. ob_distance_atr 119.370117 [SMC]
|
|
6. h1_rsi 112.250816 [M15]
|
|
7. h1_ema20_distance 95.886513 [M15]
|
|
8. macd_signal 75.965248 [M15]
|
|
9. macd 65.567200 [M15]
|
|
10. h1_market_structure 61.831249 [M15]
|
|
11. consecutive_direction 57.430664 [M15]
|
|
12. h1_trend_strength 45.872288 [M15]
|
|
13. ob_width_atr 40.723549 [SMC]
|
|
14. price_position 39.133850 [M15]
|
|
15. rsi 36.250340 [M15]
|
|
16. h1_ob_proximity 33.167027 [SMC]
|
|
17. h1_swing_proximity 30.450262 [M15]
|
|
18. volume_ratio 25.749680 [M15]
|
|
19. close_lag_5 24.403849 [M15]
|
|
20. is_fvg_bull 23.105017 [SMC]
|
|
21. ema_21 21.690825 [M15]
|
|
22. bb_percent_b 21.044502 [M15]
|
|
23. volatility_zscore 19.736000 [M15]
|
|
24. close_lag_1 18.166227 [M15]
|
|
25. close_lag_2 18.122101 [M15]
|
|
26. crisis_proximity 15.921524 [M15]
|
|
27. fvg_gap_size_atr 14.461624 [SMC]
|
|
28. hour 14.362236 [M15]
|
|
29. returns_20 11.553368 [M15]
|
|
30. dist_from_sma_20 11.102405 [M15]
|
|
|
|
--- H1 FEATURES IN TOP 10 ---
|
|
Count: 0
|
|
|
|
--- FEATURE CATEGORY SUMMARY ---
|
|
H1 features: 0
|
|
M15 technical features: 22
|
|
SMC features: 13
|
|
Average M15 importance: 19.988993
|
|
Average SMC importance: 27.545626
|
|
|
|
================================================================================
|
|
3. TARGET VARIABLE STATISTICS
|
|
================================================================================
|
|
|
|
Loading: data\training_data.parquet
|
|
Dataset shape: (8000, 72)
|
|
|
|
--- TARGET DISTRIBUTION ---
|
|
None: 1 ( 0.01%)
|
|
SELL: 3749 (46.86%)
|
|
HOLD: 4250 (53.12%)
|
|
|
|
--- RETURN ANALYSIS (M15 bars) ---
|
|
|
|
Bars with 3-bar returns > X*ATR:
|
|
> 0.1*ATR: 0 ( 0.00%)
|
|
> 0.2*ATR: 0 ( 0.00%)
|
|
> 0.3*ATR: 0 ( 0.00%)
|
|
> 0.5*ATR: 0 ( 0.00%)
|
|
> 0.7*ATR: 0 ( 0.00%)
|
|
> 1.0*ATR: 0 ( 0.00%)
|
|
|
|
Mean normalized return: 0.0000
|
|
Median normalized return: 0.0000
|
|
|
|
Positive returns: 4250 (53.12%)
|
|
Negative returns: 3744 (46.80%)
|
|
|
|
--- H1 FEATURES IN DATASET ---
|
|
H1 columns found: 0
|
|
|
|
================================================================================
|
|
4. PREDICTION CONSISTENCY ANALYSIS
|
|
================================================================================
|
|
|
|
Analyzing: logs\trading_bot_2026-02-09.log
|
|
|
|
================================================================================
|
|
5. CURRENT MODEL METRICS
|
|
================================================================================
|
|
|
|
--- MODEL METRICS (from data/model_metrics.json) ---
|
|
{
|
|
"featureImportance": [
|
|
{
|
|
"name": "ob",
|
|
"importance": 0.2126
|
|
},
|
|
{
|
|
"name": "log_returns",
|
|
"importance": 0.0948
|
|
},
|
|
{
|
|
"name": "returns_1",
|
|
"importance": 0.0856
|
|
},
|
|
{
|
|
"name": "ob_mitigated",
|
|
"importance": 0.0648
|
|
},
|
|
{
|
|
"name": "ob_distance_atr",
|
|
"importance": 0.0608
|
|
},
|
|
{
|
|
"name": "h1_rsi",
|
|
"importance": 0.0572
|
|
},
|
|
{
|
|
"name": "h1_ema20_distance",
|
|
"importance": 0.0489
|
|
},
|
|
{
|
|
"name": "macd_signal",
|
|
"importance": 0.0387
|
|
},
|
|
{
|
|
"name": "macd",
|
|
"importance": 0.0334
|
|
},
|
|
{
|
|
"name": "h1_market_structure",
|
|
"importance": 0.0315
|
|
},
|
|
{
|
|
"name": "consecutive_direction",
|
|
"importance": 0.0293
|
|
},
|
|
{
|
|
"name": "h1_trend_strength",
|
|
"importance": 0.0234
|
|
},
|
|
{
|
|
"name": "ob_width_atr",
|
|
"importance": 0.0207
|
|
},
|
|
{
|
|
"name": "price_position",
|
|
"importance": 0.0199
|
|
},
|
|
{
|
|
"name": "rsi",
|
|
"importance": 0.0185
|
|
},
|
|
{
|
|
"name": "h1_ob_proximity",
|
|
"importance": 0.0169
|
|
},
|
|
{
|
|
"name": "h1_swing_proximity",
|
|
"importance": 0.0155
|
|
},
|
|
{
|
|
"name": "volume_ratio",
|
|
"importance": 0.0131
|
|
},
|
|
{
|
|
"name": "close_lag_5",
|
|
"importance": 0.0124
|
|
},
|
|
{
|
|
"name": "is_fvg_bull",
|
|
"importance": 0.0118
|
|
}
|
|
],
|
|
"trainAuc": 0.7385315785647933,
|
|
"testAuc": 0.7338658466928052,
|
|
"sampleCount": 45459,
|
|
"updatedAt": "2026-02-09T09:01:15.440605+07:00"
|
|
}
|
|
|
|
================================================================================
|
|
6. OVERFITTING ANALYSIS
|
|
================================================================================
|
|
|
|
From training_2026-02-04.log:
|
|
Initial training: Train AUC=0.8106, Test AUC=0.6553
|
|
Overfitting gap: 0.1553 (HIGH)
|
|
|
|
Walk-forward average: Train AUC=0.8107, Test AUC=0.5722
|
|
Overfitting gap: 0.2385 (VERY HIGH)
|
|
|
|
Conclusion:
|
|
- Model shows significant overfitting
|
|
- Test AUC of 0.57-0.66 is barely better than random (0.50)
|
|
- High train AUC (0.81) but poor generalization
|
|
|
|
================================================================================
|
|
CRITICAL FINDINGS
|
|
================================================================================
|
|
|
|
1. MODEL PERFORMANCE:
|
|
- Test AUC: 0.5722 (walk-forward) - POOR
|
|
- Overfitting gap: 0.2385 - VERY HIGH
|
|
- Model barely better than random guessing
|
|
|
|
2. FEATURE IMPORTANCE:
|
|
- H1 features NOT in top 10 - Low predictive value
|
|
|
|
3. DATA QUALITY:
|
|
- Training samples: 8000
|
|
- Target imbalance likely causing issues
|
|
- Most returns < 0.3*ATR (target too weak)
|
|
|
|
4. RECOMMENDATIONS:
|
|
a. Current V2D model has POOR performance - needs replacement
|
|
b. H1 features show low importance - may not help
|
|
c. Consider new target variable (stronger signal)
|
|
d. Address class imbalance in training
|
|
e. Reduce model complexity to prevent overfitting
|
|
|
|
================================================================================
|