318 lines
8.2 KiB
Markdown
318 lines
8.2 KiB
Markdown
|
# Feature Engineering
|
||
|
|
|
||
|
|
> **File:** `src/feature_eng.py`
|
||
|
|
> **Class:** `FeatureEngineer`
|
||
|
|
> **Framework:** Pure Polars (vectorized, tanpa loop, tanpa TA-Lib)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Apa Itu Feature Engineering?
|
||
|
|
|
||
|
|
Feature Engineering adalah proses **mengubah data harga mentah (OHLCV) menjadi 40+ fitur numerik** yang bisa dibaca oleh model machine learning. Ini adalah "mata" dari AI — tanpa fitur yang baik, model tidak bisa belajar apapun.
|
||
|
|
|
||
|
|
**Analogi:** Feature Engineering adalah **alat ukur** — thermometer, barometer, kompas — yang mengubah data mentah menjadi informasi bermakna.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Flow Utama: `calculate_all()`
|
||
|
|
|
||
|
|
```
|
||
|
|
Input: DataFrame OHLCV (open, high, low, close, volume)
|
||
|
|
|
|
||
|
|
|-- calculate_rsi() -> rsi
|
||
|
|
|-- calculate_atr() -> atr, atr_percent
|
||
|
|
|-- calculate_macd() -> macd, macd_signal, macd_histogram
|
||
|
|
|-- calculate_bollinger_bands() -> bb_upper, bb_lower, bb_width, bb_percent_b
|
||
|
|
|-- calculate_ema_crossover() -> ema_9, ema_21, ema_cross_bull/bear
|
||
|
|
|-- calculate_volume_features() -> volume_ratio, high_volume
|
||
|
|
|
|
||
|
|
|-- [jika include_ml_features=True]
|
||
|
|
| calculate_ml_features() -> returns, volatility, lags, trends, time
|
||
|
|
|
|
||
|
|
v
|
||
|
|
Output: DataFrame dengan 40+ kolom fitur
|
||
|
|
```
|
||
|
|
|
||
|
|
**Data minimum:** 26 bar (kebutuhan MACD slow EMA) agar semua fitur stabil.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Kategori 1: Indikator Teknikal
|
||
|
|
|
||
|
|
### RSI (Relative Strength Index) — Period 14
|
||
|
|
|
||
|
|
```
|
||
|
|
Formula: RSI = 100 - (100 / (1 + RS))
|
||
|
|
RS = Average Gain / Average Loss
|
||
|
|
Smoothing: Wilder's EMA (alpha = 1/14)
|
||
|
|
```
|
||
|
|
|
||
|
|
| Nilai | Interpretasi |
|
||
|
|
|-------|-------------|
|
||
|
|
| RSI > 70 | Overbought (potensi turun) |
|
||
|
|
| RSI < 30 | Oversold (potensi naik) |
|
||
|
|
| RSI ~ 50 | Netral |
|
||
|
|
|
||
|
|
**Output:** `rsi`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### ATR (Average True Range) — Period 14
|
||
|
|
|
||
|
|
```
|
||
|
|
True Range = max(High-Low, |High-PrevClose|, |Low-PrevClose|)
|
||
|
|
ATR = Wilder's EMA dari True Range
|
||
|
|
ATR% = (ATR / Close) * 100
|
||
|
|
```
|
||
|
|
|
||
|
|
| Kondisi | Interpretasi |
|
||
|
|
|---------|-------------|
|
||
|
|
| ATR tinggi | Pasar volatile (pergerakan besar) |
|
||
|
|
| ATR rendah | Pasar tenang (pergerakan kecil) |
|
||
|
|
|
||
|
|
**Output:** `atr`, `atr_percent`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### MACD (Moving Average Convergence Divergence) — 12/26/9
|
||
|
|
|
||
|
|
```
|
||
|
|
MACD Line = EMA(12) - EMA(26)
|
||
|
|
Signal = EMA(MACD Line, 9)
|
||
|
|
Histogram = MACD Line - Signal
|
||
|
|
```
|
||
|
|
|
||
|
|
| Kondisi | Interpretasi |
|
||
|
|
|---------|-------------|
|
||
|
|
| Histogram > 0 & naik | Bullish momentum menguat |
|
||
|
|
| Histogram < 0 & turun | Bearish momentum menguat |
|
||
|
|
| MACD cross Signal ke atas | Potensi reversal naik |
|
||
|
|
| MACD cross Signal ke bawah | Potensi reversal turun |
|
||
|
|
|
||
|
|
**Output:** `macd`, `macd_signal`, `macd_histogram`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Bollinger Bands — Period 20, StdDev 2.0
|
||
|
|
|
||
|
|
```
|
||
|
|
Middle = SMA(20)
|
||
|
|
Upper = Middle + 2 * StdDev
|
||
|
|
Lower = Middle - 2 * StdDev
|
||
|
|
Width = (Upper - Lower) / Middle
|
||
|
|
%B = (Close - Lower) / (Upper - Lower)
|
||
|
|
```
|
||
|
|
|
||
|
|
| Kondisi | Interpretasi |
|
||
|
|
|---------|-------------|
|
||
|
|
| %B > 1 | Harga di atas upper band (extreme bullish) |
|
||
|
|
| %B < 0 | Harga di bawah lower band (extreme bearish) |
|
||
|
|
| %B ~ 0.5 | Harga di tengah |
|
||
|
|
| Width melebar | Volatilitas meningkat |
|
||
|
|
| Width menyempit | Volatilitas menurun (squeeze) |
|
||
|
|
|
||
|
|
**Output:** `bb_middle`, `bb_upper`, `bb_lower`, `bb_width`, `bb_percent_b`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### EMA Crossover — 9/21
|
||
|
|
|
||
|
|
```
|
||
|
|
EMA9 = Exponential Moving Average (cepat)
|
||
|
|
EMA21 = Exponential Moving Average (lambat)
|
||
|
|
```
|
||
|
|
|
||
|
|
| Kondisi | Interpretasi |
|
||
|
|
|---------|-------------|
|
||
|
|
| EMA9 > EMA21 | Tren naik |
|
||
|
|
| EMA9 < EMA21 | Tren turun |
|
||
|
|
| EMA9 cross atas EMA21 | Sinyal beli |
|
||
|
|
| EMA9 cross bawah EMA21 | Sinyal jual |
|
||
|
|
|
||
|
|
**Output:** `ema_9`, `ema_21`, `ema_cross_bull`, `ema_cross_bear`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Kategori 2: Volume Features — Period 20
|
||
|
|
|
||
|
|
```
|
||
|
|
volume_sma = Rolling Mean(volume, 20)
|
||
|
|
volume_ratio = volume / volume_sma
|
||
|
|
volume_increasing = 1 jika volume > volume sebelumnya
|
||
|
|
high_volume = 1 jika volume_ratio > 1.5
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Konfirmasi breakout — pergerakan besar harus didukung volume tinggi.
|
||
|
|
|
||
|
|
**Catatan:** Jika kolom volume tidak ada di data, fitur ini di-skip (graceful degradation).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Kategori 3: ML-Specific Features
|
||
|
|
|
||
|
|
### Returns & Momentum
|
||
|
|
|
||
|
|
```
|
||
|
|
returns_1 = (Close[t] / Close[t-1]) - 1 # Return 1 bar
|
||
|
|
returns_5 = (Close[t] / Close[t-5]) - 1 # Return 5 bar
|
||
|
|
returns_20 = (Close[t] / Close[t-20]) - 1 # Return 20 bar
|
||
|
|
log_returns = ln(Close[t] / Close[t-1]) # Log return
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Mengukur kecepatan dan arah pergerakan harga dalam berbagai timeframe.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Price Position
|
||
|
|
|
||
|
|
```
|
||
|
|
price_position = (Close - Low) / (High - Low) # Posisi 0-1 dalam range candle
|
||
|
|
dist_from_sma_20 = (Close / SMA20) - 1 # Jarak (%) dari rata-rata
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Mengukur dimana harga relatif terhadap range dan rata-rata.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Volatility
|
||
|
|
|
||
|
|
```
|
||
|
|
volatility_20 = StdDev(log_returns, 20) # Realized volatility
|
||
|
|
normalized_range = (High - Low) / Close # Range sebagai % harga
|
||
|
|
avg_normalized_range = SMA(normalized_range, 14) # Rata-rata range 14 bar
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Input penting untuk HMM regime detection dan risk sizing.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Lag Features
|
||
|
|
|
||
|
|
```
|
||
|
|
close_lag_1 = Close[t-1]
|
||
|
|
close_lag_2 = Close[t-2]
|
||
|
|
close_lag_3 = Close[t-3]
|
||
|
|
close_lag_5 = Close[t-5]
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Auto-regressive features — menangkap pola harga berulang.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Trend Features
|
||
|
|
|
||
|
|
```
|
||
|
|
higher_high = 1 jika High[t] > High[t-1], else 0
|
||
|
|
lower_low = 1 jika Low[t] < Low[t-1], else 0
|
||
|
|
hh_count_5 = Sum(higher_high, 5 bar) # Berapa kali HH dalam 5 bar
|
||
|
|
ll_count_5 = Sum(lower_low, 5 bar) # Berapa kali LL dalam 5 bar
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Mengukur konsistensi tren — banyak HH = strong uptrend.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Time Features
|
||
|
|
|
||
|
|
```
|
||
|
|
hour = Jam (0-23)
|
||
|
|
weekday = Hari (0=Senin, 6=Minggu)
|
||
|
|
london_session = 1 jika jam 08:00-16:00 UTC
|
||
|
|
ny_session = 1 jika jam 13:00-21:00 UTC
|
||
|
|
```
|
||
|
|
|
||
|
|
**Fungsi:** Pasar berperilaku berbeda tiap sesi — London volatile, Asian tenang.
|
||
|
|
|
||
|
|
**Catatan:** Hanya dihitung jika kolom `time` bertipe Datetime.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Kategori 4: SMC sebagai Fitur Numerik
|
||
|
|
|
||
|
|
Dari SMC Analyzer, dikonversi jadi angka untuk XGBoost:
|
||
|
|
|
||
|
|
```
|
||
|
|
swing_high = 1 / 0
|
||
|
|
swing_low = -1 / 0
|
||
|
|
fvg_signal = 1 (bull) / -1 (bear) / 0
|
||
|
|
ob = 1 (bull) / -1 (bear) / 0
|
||
|
|
bos = 1 (bull) / -1 (bear) / 0
|
||
|
|
choch = 1 (bull) / -1 (bear) / 0
|
||
|
|
market_structure = 1 (bull) / -1 (bear) / 0
|
||
|
|
regime = 0 / 1 / 2 (dari HMM)
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Target Variable (Label Training)
|
||
|
|
|
||
|
|
```python
|
||
|
|
create_target(df, lookahead=1, threshold=0.0):
|
||
|
|
target = 1 jika close[t+1] > close[t] # Harga naik
|
||
|
|
target = 0 jika close[t+1] <= close[t] # Harga turun/tetap
|
||
|
|
target_return = close[t+1] / close[t] - 1 # Return kontinu
|
||
|
|
```
|
||
|
|
|
||
|
|
**Catatan:** Target dibuat saat training saja, tidak saat live trading.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Preprocessing untuk ML
|
||
|
|
|
||
|
|
### Penanganan Null
|
||
|
|
```python
|
||
|
|
# Bar awal memiliki NaN karena lookback period
|
||
|
|
# Saat training: baris dengan NaN di-drop
|
||
|
|
df_clean = df.select(features + [target]).drop_nulls()
|
||
|
|
```
|
||
|
|
|
||
|
|
### Penanganan Infinity
|
||
|
|
```python
|
||
|
|
# Saat prediksi: NaN & infinity diganti 0
|
||
|
|
X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Normalisasi
|
||
|
|
**Tidak dilakukan** — XGBoost berbasis tree, scale-invariant (tidak perlu scaling).
|
||
|
|
|
||
|
|
### Cleanup Kolom Temporary
|
||
|
|
Setiap method membersihkan kolom sementara yang diawali `_` (misal `_delta`, `_avg_gain`, dll).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Fitur yang Digunakan vs Tidak
|
||
|
|
|
||
|
|
### Digunakan oleh XGBoost (24+ fitur)
|
||
|
|
Semua indikator teknikal, returns, volatility, trend, time, SMC numerik, regime.
|
||
|
|
|
||
|
|
### Tidak Digunakan (Excluded)
|
||
|
|
- Kolom OHLCV asli: `time`, `open`, `high`, `low`, `close`, `volume`
|
||
|
|
- Kolom meta: `spread`, `real_volume`, `target`, `target_return`
|
||
|
|
- Kolom SMC level: `fvg_top`, `fvg_bottom`, `ob_top`, `ob_bottom`, dll
|
||
|
|
- Kolom temporary: apapun yang diawali `_`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Parameter Konfigurasi
|
||
|
|
|
||
|
|
| Indikator | Parameter | Default | Configurable |
|
||
|
|
|-----------|-----------|---------|-------------|
|
||
|
|
| RSI | period | 14 | Ya |
|
||
|
|
| ATR | period | 14 | Ya |
|
||
|
|
| MACD | fast/slow/signal | 12/26/9 | Ya |
|
||
|
|
| Bollinger Bands | period, std_dev | 20, 2.0 | Ya |
|
||
|
|
| EMA Crossover | fast/slow | 9/21 | Ya |
|
||
|
|
| Volume | period | 20 | Ya |
|
||
|
|
| Returns | lookback | [1, 5, 20] | Hardcoded |
|
||
|
|
| Volatility | window | 20 | Hardcoded |
|
||
|
|
| Session | London hours | 08-16 UTC | Hardcoded |
|
||
|
|
| Session | NY hours | 13-21 UTC | Hardcoded |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Performa
|
||
|
|
|
||
|
|
- **5000 bar features:** < 100ms (sangat cepat)
|
||
|
|
- **Framework:** Polars vectorized (10-100x lebih cepat dari Pandas loop)
|
||
|
|
- **Memory:** ~1.6MB untuk 40 fitur x 5000 bar
|