XauBot/docs/arsitektur-ai/04-Feature-Engineering.md

# *Feature Engineering*

> **File:** `src/feature_eng.py`
> **Class:** `FeatureEngineer`
> **Framework:** Pure Polars (vectorized, tanpa loop, tanpa TA-Lib — bukan Pandas)

---

## Pipeline *Feature Engineering*

```mermaid
flowchart LR
    A["OHLCV Data\n(open, high, low,\nclose, volume, time)"] --> B["calculate_all()"]

    subgraph B["calculate_all()"]
        direction TB
        B1["calculate_rsi()"]
        B2["calculate_atr()"]
        B3["calculate_macd()"]
        B4["calculate_bollinger_bands()"]
        B5["calculate_ema_crossover()"]
        B6["calculate_volume_features()"]
        B7["calculate_ml_features()\n(returns, volatility,\nlags, trend, time)"]
        B1 --> B2 --> B3 --> B4 --> B5 --> B6 --> B7
    end

    B --> C["40+ Fitur Numerik"]
    C --> D["ML Ready\n(XGBoost Input)"]

    style A fill:#2d3748,stroke:#63b3ed,color:#fff
    style C fill:#2d3748,stroke:#48bb78,color:#fff
    style D fill:#2d3748,stroke:#f6ad55,color:#fff
```

> **Performa:** Seluruh pipeline dijalankan dalam **< 100ms** untuk 5000 bar menggunakan Pure Polars (vectorized, 10-100x lebih cepat dari Pandas loop). Menghasilkan **40+ fitur** yang siap digunakan model ML.

---

## Apa Itu *Feature Engineering*?

*Feature Engineering* adalah proses **mengubah data harga mentah (OHLCV) menjadi 40+ fitur numerik** yang bisa dibaca oleh model machine learning. Ini adalah "mata" dari AI -- tanpa fitur yang baik, model tidak bisa belajar apapun.

**Analogi:** *Feature Engineering* adalah **alat ukur** -- thermometer, barometer, kompas -- yang mengubah data mentah menjadi informasi bermakna.

---

## Flow Utama: `calculate_all()`

```
Input: DataFrame OHLCV (open, high, low, close, volume)
    |
    |-- calculate_rsi()               -> rsi
    |-- calculate_atr()               -> atr, atr_percent
    |-- calculate_macd()              -> macd, macd_signal, macd_histogram
    |-- calculate_bollinger_bands()   -> bb_upper, bb_lower, bb_width, bb_percent_b
    |-- calculate_ema_crossover()     -> ema_9, ema_21, ema_cross_bull/bear
    |-- calculate_volume_features()   -> volume_ratio, high_volume
    |
    |-- [jika include_ml_features=True]
    |   calculate_ml_features()       -> returns, volatility, lags, trends, time
    |
    v
Output: DataFrame dengan 40+ kolom fitur
```

**Data minimum:** 26 bar (kebutuhan MACD slow EMA) agar semua fitur stabil.

---

## Kategori 1: Indikator Teknikal

### RSI (*Relative Strength Index*) -- Period 14

```
Formula: RSI = 100 - (100 / (1 + RS))
         RS  = Average Gain / Average Loss
Smoothing: Wilder's EMA (alpha = 1/14)
```

| Nilai | Interpretasi |
|-------|-------------|
| RSI > 70 | *Overbought* (potensi turun) |
| RSI < 30 | *Oversold* (potensi naik) |
| RSI ~ 50 | Netral |

**Output:** `rsi`

---

### ATR (*Average True Range*) -- Period 14

```
True Range = max(High-Low, |High-PrevClose|, |Low-PrevClose|)
ATR = Wilder's EMA dari True Range
ATR% = (ATR / Close) * 100
```

| Kondisi | Interpretasi |
|---------|-------------|
| ATR tinggi | Pasar *volatile* (pergerakan besar) |
| ATR rendah | Pasar tenang (pergerakan kecil) |

**Output:** `atr`, `atr_percent`

---

### MACD (*Moving Average Convergence Divergence*) -- 12/26/9

```
MACD Line  = EMA(12) - EMA(26)
Signal     = EMA(MACD Line, 9)
Histogram  = MACD Line - Signal
```

| Kondisi | Interpretasi |
|---------|-------------|
| Histogram > 0 & naik | Bullish *momentum* menguat |
| Histogram < 0 & turun | Bearish *momentum* menguat |
| MACD cross Signal ke atas | Potensi reversal naik |
| MACD cross Signal ke bawah | Potensi reversal turun |

**Output:** `macd`, `macd_signal`, `macd_histogram`

---

### *Bollinger Bands* -- Period 20, StdDev 2.0

```
Middle = SMA(20)
Upper  = Middle + 2 * StdDev
Lower  = Middle - 2 * StdDev
Width  = (Upper - Lower) / Middle
%B     = (Close - Lower) / (Upper - Lower)
```

| Kondisi | Interpretasi |
|---------|-------------|
| %B > 1 | Harga di atas upper band (extreme bullish) |
| %B < 0 | Harga di bawah lower band (extreme bearish) |
| %B ~ 0.5 | Harga di tengah |
| Width melebar | *Volatility* meningkat |
| Width menyempit | *Volatility* menurun (squeeze) |

**Output:** `bb_middle`, `bb_upper`, `bb_lower`, `bb_width`, `bb_percent_b`

---

### EMA *Crossover* -- 9/21

```
EMA9  = Exponential Moving Average (cepat)
EMA21 = Exponential Moving Average (lambat)
```

*EMA* (*Exponential Moving Average*) memberikan bobot lebih besar pada data terbaru, sehingga lebih responsif terhadap perubahan harga dibanding SMA.

| Kondisi | Interpretasi |
|---------|-------------|
| EMA9 > EMA21 | *Trend* naik |
| EMA9 < EMA21 | *Trend* turun |
| EMA9 cross atas EMA21 | Sinyal beli (*bullish crossover*) |
| EMA9 cross bawah EMA21 | Sinyal jual (*bearish crossover*) |

**Output:** `ema_9`, `ema_21`, `ema_cross_bull`, `ema_cross_bear`

---

## Kategori 2: Volume Features -- Period 20

```
volume_sma        = Rolling Mean(volume, 20)
volume_ratio      = volume / volume_sma
volume_increasing = 1 jika volume > volume sebelumnya
high_volume       = 1 jika volume_ratio > 1.5
```

**Fungsi:** Konfirmasi breakout -- pergerakan besar harus didukung volume tinggi.

**Catatan:** Jika kolom volume tidak ada di data, fitur ini di-skip (graceful degradation).

---

## Kategori 3: ML-Specific Features

### *Returns* & *Momentum*

```
returns_1   = (Close[t] / Close[t-1]) - 1     # Return 1 bar
returns_5   = (Close[t] / Close[t-5]) - 1     # Return 5 bar
returns_20  = (Close[t] / Close[t-20]) - 1    # Return 20 bar
log_returns = ln(Close[t] / Close[t-1])        # Log return
```

**Fungsi:** Mengukur kecepatan dan arah pergerakan harga (*momentum*) dalam berbagai timeframe.

---

### Price Position

```
price_position   = (Close - Low) / (High - Low)   # Posisi 0-1 dalam range candle
dist_from_sma_20 = (Close / SMA20) - 1            # Jarak (%) dari rata-rata
```

**Fungsi:** Mengukur dimana harga relatif terhadap range dan rata-rata.

---

### *Volatility*

```
volatility_20       = StdDev(log_returns, 20)       # Realized volatility
normalized_range    = (High - Low) / Close           # Range sebagai % harga
avg_normalized_range = SMA(normalized_range, 14)     # Rata-rata range 14 bar
```

**Fungsi:** Input penting untuk HMM regime detection dan risk sizing. *Volatility* yang tinggi menandakan pasar bergejolak dan mempengaruhi ukuran posisi.

---

### *Lag Features*

```
close_lag_1 = Close[t-1]
close_lag_2 = Close[t-2]
close_lag_3 = Close[t-3]
close_lag_5 = Close[t-5]
```

**Fungsi:** Auto-regressive features -- menangkap pola harga berulang. *Lag features* memberikan konteks historis langsung kepada model.

---

### *Trend* Features

```
higher_high = 1 jika High[t] > High[t-1], else 0
lower_low   = 1 jika Low[t] < Low[t-1], else 0
hh_count_5  = Sum(higher_high, 5 bar)   # Berapa kali HH dalam 5 bar
ll_count_5  = Sum(lower_low, 5 bar)     # Berapa kali LL dalam 5 bar
```

**Fungsi:** Mengukur konsistensi *trend* -- banyak HH = strong uptrend.

---

### *Time Features*

```
hour           = Jam (0-23)
weekday        = Hari (0=Senin, 6=Minggu)
london_session = 1 jika jam 08:00-16:00 UTC
ny_session     = 1 jika jam 13:00-21:00 UTC
```

**Fungsi:** Pasar berperilaku berbeda tiap sesi -- London *volatile*, Asian tenang. *Time features* membantu model mengenali pola berbasis waktu.

**Catatan:** Hanya dihitung jika kolom `time` bertipe Datetime.

---

## Kategori 4: SMC sebagai Fitur Numerik

Dari SMC Analyzer, dikonversi jadi angka untuk XGBoost:

```
swing_high       = 1 / 0
swing_low        = -1 / 0
fvg_signal       = 1 (bull) / -1 (bear) / 0
ob               = 1 (bull) / -1 (bear) / 0
bos              = 1 (bull) / -1 (bear) / 0
choch            = 1 (bull) / -1 (bear) / 0
market_structure = 1 (bull) / -1 (bear) / 0
regime           = 0 / 1 / 2 (dari HMM)
```

---

## Target Variable (Label Training)

```python
create_target(df, lookahead=1, threshold=0.0):
    target = 1 jika close[t+1] > close[t]   # Harga naik
    target = 0 jika close[t+1] <= close[t]   # Harga turun/tetap
    target_return = close[t+1] / close[t] - 1  # Return kontinu
```

**Catatan:** Target dibuat saat training saja, tidak saat live trading.

---

## Preprocessing untuk ML

### Penanganan Null
```python
# Bar awal memiliki NaN karena lookback period
# Saat training: baris dengan NaN di-drop
df_clean = df.select(features + [target]).drop_nulls()
```

### Penanganan Infinity
```python
# Saat prediksi: NaN & infinity diganti 0
X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)
```

### Normalisasi
**Tidak dilakukan** -- XGBoost berbasis tree, scale-invariant (tidak perlu scaling).

### Cleanup Kolom Temporary
Setiap method membersihkan kolom sementara yang diawali `_` (misal `_delta`, `_avg_gain`, dll).

---

## Fitur yang Digunakan vs Tidak

### Digunakan oleh XGBoost (24+ fitur)
Semua indikator teknikal, *returns*, *volatility*, *trend*, *time features*, SMC numerik, regime.

### Tidak Digunakan (Excluded)
- Kolom OHLCV asli: `time`, `open`, `high`, `low`, `close`, `volume`
- Kolom meta: `spread`, `real_volume`, `target`, `target_return`
- Kolom SMC level: `fvg_top`, `fvg_bottom`, `ob_top`, `ob_bottom`, dll
- Kolom temporary: apapun yang diawali `_`

---

## Parameter Konfigurasi

| Indikator | Parameter | Default | Configurable |
|-----------|-----------|---------|-------------|
| RSI (*Relative Strength Index*) | period | 14 | Ya |
| ATR (*Average True Range*) | period | 14 | Ya |
| MACD (*Moving Average Convergence Divergence*) | fast/slow/signal | 12/26/9 | Ya |
| *Bollinger Bands* | period, std_dev | 20, 2.0 | Ya |
| EMA *Crossover* | fast/slow | 9/21 | Ya |
| Volume | period | 20 | Ya |
| *Returns* | lookback | [1, 5, 20] | Hardcoded |
| *Volatility* | window | 20 | Hardcoded |
| Session | London hours | 08-16 UTC | Hardcoded |
| Session | NY hours | 13-21 UTC | Hardcoded |

---

## Performa

- **5000 bar features:** < 100ms (sangat cepat)
- **Framework:** Pure Polars vectorized (10-100x lebih cepat dari Pandas loop) -- **bukan Pandas**
- **Memory:** ~1.6MB untuk 40+ fitur x 5000 bar
- **Total fitur:** 40+ kolom numerik siap ML