feat: Smart AI Trading Bot for XAUUSD with ML and SMC

- XGBoost ML model with 37 features for market direction prediction - Smart Money Concepts (SMC): Order Blocks, FVG, BOS, CHoCH - HMM market regime detection (trending/ranging/volatile) - ATR-based stop loss with 1.5 ATR minimum distance - Broker-level SL protection with fallback - Time-based exit (max 6 hours per trade) - Session-aware trading optimized for London/NY overlap - Auto-retraining based on market conditions - Telegram notifications and web dashboard - Backtest results: 63.9% win rate, 2.64 profit factor, 4.83 Sharpe Backtest period: Jan 2025 - Feb 2026, 654 trades, $4,189 net P/L Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 09:01:35 +07:00
commit 7af9183af3
121 changed files with 43387 additions and 0 deletions
@@ -0,0 +1,317 @@
+# Feature Engineering
+
+> **File:** `src/feature_eng.py`
+> **Class:** `FeatureEngineer`
+> **Framework:** Pure Polars (vectorized, tanpa loop, tanpa TA-Lib)
+
+---
+
+## Apa Itu Feature Engineering?
+
+Feature Engineering adalah proses **mengubah data harga mentah (OHLCV) menjadi 40+ fitur numerik** yang bisa dibaca oleh model machine learning. Ini adalah "mata" dari AI — tanpa fitur yang baik, model tidak bisa belajar apapun.
+
+**Analogi:** Feature Engineering adalah **alat ukur** — thermometer, barometer, kompas — yang mengubah data mentah menjadi informasi bermakna.
+
+---
+
+## Flow Utama: `calculate_all()`
+
+```
+Input: DataFrame OHLCV (open, high, low, close, volume)
+    |
+    |-- calculate_rsi()               -> rsi
+    |-- calculate_atr()               -> atr, atr_percent
+    |-- calculate_macd()              -> macd, macd_signal, macd_histogram
+    |-- calculate_bollinger_bands()   -> bb_upper, bb_lower, bb_width, bb_percent_b
+    |-- calculate_ema_crossover()     -> ema_9, ema_21, ema_cross_bull/bear
+    |-- calculate_volume_features()   -> volume_ratio, high_volume
+    |
+    |-- [jika include_ml_features=True]
+    |   calculate_ml_features()       -> returns, volatility, lags, trends, time
+    |
+    v
+Output: DataFrame dengan 40+ kolom fitur
+```
+
+**Data minimum:** 26 bar (kebutuhan MACD slow EMA) agar semua fitur stabil.
+
+---
+
+## Kategori 1: Indikator Teknikal
+
+### RSI (Relative Strength Index) — Period 14
+
+```
+Formula: RSI = 100 - (100 / (1 + RS))
+         RS  = Average Gain / Average Loss
+Smoothing: Wilder's EMA (alpha = 1/14)
+```
+
+| Nilai | Interpretasi |
+|-------|-------------|
+| RSI > 70 | Overbought (potensi turun) |
+| RSI < 30 | Oversold (potensi naik) |
+| RSI ~ 50 | Netral |
+
+**Output:** `rsi`
+
+---
+
+### ATR (Average True Range) — Period 14
+
+```
+True Range = max(High-Low, |High-PrevClose|, |Low-PrevClose|)
+ATR = Wilder's EMA dari True Range
+ATR% = (ATR / Close) * 100
+```
+
+| Kondisi | Interpretasi |
+|---------|-------------|
+| ATR tinggi | Pasar volatile (pergerakan besar) |
+| ATR rendah | Pasar tenang (pergerakan kecil) |
+
+**Output:** `atr`, `atr_percent`
+
+---
+
+### MACD (Moving Average Convergence Divergence) — 12/26/9
+
+```
+MACD Line  = EMA(12) - EMA(26)
+Signal     = EMA(MACD Line, 9)
+Histogram  = MACD Line - Signal
+```
+
+| Kondisi | Interpretasi |
+|---------|-------------|
+| Histogram > 0 & naik | Bullish momentum menguat |
+| Histogram < 0 & turun | Bearish momentum menguat |
+| MACD cross Signal ke atas | Potensi reversal naik |
+| MACD cross Signal ke bawah | Potensi reversal turun |
+
+**Output:** `macd`, `macd_signal`, `macd_histogram`
+
+---
+
+### Bollinger Bands — Period 20, StdDev 2.0
+
+```
+Middle = SMA(20)
+Upper  = Middle + 2 * StdDev
+Lower  = Middle - 2 * StdDev
+Width  = (Upper - Lower) / Middle
+%B     = (Close - Lower) / (Upper - Lower)
+```
+
+| Kondisi | Interpretasi |
+|---------|-------------|
+| %B > 1 | Harga di atas upper band (extreme bullish) |
+| %B < 0 | Harga di bawah lower band (extreme bearish) |
+| %B ~ 0.5 | Harga di tengah |
+| Width melebar | Volatilitas meningkat |
+| Width menyempit | Volatilitas menurun (squeeze) |
+
+**Output:** `bb_middle`, `bb_upper`, `bb_lower`, `bb_width`, `bb_percent_b`
+
+---
+
+### EMA Crossover — 9/21
+
+```
+EMA9  = Exponential Moving Average (cepat)
+EMA21 = Exponential Moving Average (lambat)
+```
+
+| Kondisi | Interpretasi |
+|---------|-------------|
+| EMA9 > EMA21 | Tren naik |
+| EMA9 < EMA21 | Tren turun |
+| EMA9 cross atas EMA21 | Sinyal beli |
+| EMA9 cross bawah EMA21 | Sinyal jual |
+
+**Output:** `ema_9`, `ema_21`, `ema_cross_bull`, `ema_cross_bear`
+
+---
+
+## Kategori 2: Volume Features — Period 20
+
+```
+volume_sma        = Rolling Mean(volume, 20)
+volume_ratio      = volume / volume_sma
+volume_increasing = 1 jika volume > volume sebelumnya
+high_volume       = 1 jika volume_ratio > 1.5
+```
+
+**Fungsi:** Konfirmasi breakout — pergerakan besar harus didukung volume tinggi.
+
+**Catatan:** Jika kolom volume tidak ada di data, fitur ini di-skip (graceful degradation).
+
+---
+
+## Kategori 3: ML-Specific Features
+
+### Returns & Momentum
+
+```
+returns_1   = (Close[t] / Close[t-1]) - 1     # Return 1 bar
+returns_5   = (Close[t] / Close[t-5]) - 1     # Return 5 bar
+returns_20  = (Close[t] / Close[t-20]) - 1    # Return 20 bar
+log_returns = ln(Close[t] / Close[t-1])        # Log return
+```
+
+**Fungsi:** Mengukur kecepatan dan arah pergerakan harga dalam berbagai timeframe.
+
+---
+
+### Price Position
+
+```
+price_position   = (Close - Low) / (High - Low)   # Posisi 0-1 dalam range candle
+dist_from_sma_20 = (Close / SMA20) - 1            # Jarak (%) dari rata-rata
+```
+
+**Fungsi:** Mengukur dimana harga relatif terhadap range dan rata-rata.
+
+---
+
+### Volatility
+
+```
+volatility_20       = StdDev(log_returns, 20)       # Realized volatility
+normalized_range    = (High - Low) / Close           # Range sebagai % harga
+avg_normalized_range = SMA(normalized_range, 14)     # Rata-rata range 14 bar
+```
+
+**Fungsi:** Input penting untuk HMM regime detection dan risk sizing.
+
+---
+
+### Lag Features
+
+```
+close_lag_1 = Close[t-1]
+close_lag_2 = Close[t-2]
+close_lag_3 = Close[t-3]
+close_lag_5 = Close[t-5]
+```
+
+**Fungsi:** Auto-regressive features — menangkap pola harga berulang.
+
+---
+
+### Trend Features
+
+```
+higher_high = 1 jika High[t] > High[t-1], else 0
+lower_low   = 1 jika Low[t] < Low[t-1], else 0
+hh_count_5  = Sum(higher_high, 5 bar)   # Berapa kali HH dalam 5 bar
+ll_count_5  = Sum(lower_low, 5 bar)     # Berapa kali LL dalam 5 bar
+```
+
+**Fungsi:** Mengukur konsistensi tren — banyak HH = strong uptrend.
+
+---
+
+### Time Features
+
+```
+hour           = Jam (0-23)
+weekday        = Hari (0=Senin, 6=Minggu)
+london_session = 1 jika jam 08:00-16:00 UTC
+ny_session     = 1 jika jam 13:00-21:00 UTC
+```
+
+**Fungsi:** Pasar berperilaku berbeda tiap sesi — London volatile, Asian tenang.
+
+**Catatan:** Hanya dihitung jika kolom `time` bertipe Datetime.
+
+---
+
+## Kategori 4: SMC sebagai Fitur Numerik
+
+Dari SMC Analyzer, dikonversi jadi angka untuk XGBoost:
+
+```
+swing_high       = 1 / 0
+swing_low        = -1 / 0
+fvg_signal       = 1 (bull) / -1 (bear) / 0
+ob               = 1 (bull) / -1 (bear) / 0
+bos              = 1 (bull) / -1 (bear) / 0
+choch            = 1 (bull) / -1 (bear) / 0
+market_structure = 1 (bull) / -1 (bear) / 0
+regime           = 0 / 1 / 2 (dari HMM)
+```
+
+---
+
+## Target Variable (Label Training)
+
+```python
+create_target(df, lookahead=1, threshold=0.0):
+    target = 1 jika close[t+1] > close[t]   # Harga naik
+    target = 0 jika close[t+1] <= close[t]   # Harga turun/tetap
+    target_return = close[t+1] / close[t] - 1  # Return kontinu
+```
+
+**Catatan:** Target dibuat saat training saja, tidak saat live trading.
+
+---
+
+## Preprocessing untuk ML
+
+### Penanganan Null
+```python
+# Bar awal memiliki NaN karena lookback period
+# Saat training: baris dengan NaN di-drop
+df_clean = df.select(features + [target]).drop_nulls()
+```
+
+### Penanganan Infinity
+```python
+# Saat prediksi: NaN & infinity diganti 0
+X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)
+```
+
+### Normalisasi
+**Tidak dilakukan** — XGBoost berbasis tree, scale-invariant (tidak perlu scaling).
+
+### Cleanup Kolom Temporary
+Setiap method membersihkan kolom sementara yang diawali `_` (misal `_delta`, `_avg_gain`, dll).
+
+---
+
+## Fitur yang Digunakan vs Tidak
+
+### Digunakan oleh XGBoost (24+ fitur)
+Semua indikator teknikal, returns, volatility, trend, time, SMC numerik, regime.
+
+### Tidak Digunakan (Excluded)
+- Kolom OHLCV asli: `time`, `open`, `high`, `low`, `close`, `volume`
+- Kolom meta: `spread`, `real_volume`, `target`, `target_return`
+- Kolom SMC level: `fvg_top`, `fvg_bottom`, `ob_top`, `ob_bottom`, dll
+- Kolom temporary: apapun yang diawali `_`
+
+---
+
+## Parameter Konfigurasi
+
+| Indikator | Parameter | Default | Configurable |
+|-----------|-----------|---------|-------------|
+| RSI | period | 14 | Ya |
+| ATR | period | 14 | Ya |
+| MACD | fast/slow/signal | 12/26/9 | Ya |
+| Bollinger Bands | period, std_dev | 20, 2.0 | Ya |
+| EMA Crossover | fast/slow | 9/21 | Ya |
+| Volume | period | 20 | Ya |
+| Returns | lookback | [1, 5, 20] | Hardcoded |
+| Volatility | window | 20 | Hardcoded |
+| Session | London hours | 08-16 UTC | Hardcoded |
+| Session | NY hours | 13-21 UTC | Hardcoded |
+
+---
+
+## Performa
+
+- **5000 bar features:** < 100ms (sangat cepat)
+- **Framework:** Polars vectorized (10-100x lebih cepat dari Pandas loop)
+- **Memory:** ~1.6MB untuk 40 fitur x 5000 bar