XauBot/docs/arsitektur-ai/04-Feature-Engineering.md

Feature Engineering

File: src/feature_eng.py Class: FeatureEngineer Framework: Pure Polars (vectorized, tanpa loop, tanpa TA-Lib — bukan Pandas)

---

Pipeline Feature Engineering

flowchart LR
    A["OHLCV Data\n(open, high, low,\nclose, volume, time)"] --> B["calculate_all()"]

    subgraph B["calculate_all()"]
        direction TB
        B1["calculate_rsi()"]
        B2["calculate_atr()"]
        B3["calculate_macd()"]
        B4["calculate_bollinger_bands()"]
        B5["calculate_ema_crossover()"]
        B6["calculate_volume_features()"]
        B7["calculate_ml_features()\n(returns, volatility,\nlags, trend, time)"]
        B1 --> B2 --> B3 --> B4 --> B5 --> B6 --> B7
    end

    B --> C["40+ Fitur Numerik"]
    C --> D["ML Ready\n(XGBoost Input)"]

    style A fill:#2d3748,stroke:#63b3ed,color:#fff
    style C fill:#2d3748,stroke:#48bb78,color:#fff
    style D fill:#2d3748,stroke:#f6ad55,color:#fff

Performa: Seluruh pipeline dijalankan dalam < 100ms untuk 5000 bar menggunakan Pure Polars (vectorized, 10-100x lebih cepat dari Pandas loop). Menghasilkan 40+ fitur yang siap digunakan model ML.

---

Apa Itu Feature Engineering?

Feature Engineering adalah proses mengubah data harga mentah (OHLCV) menjadi 40+ fitur numerik yang bisa dibaca oleh model machine learning. Ini adalah "mata" dari AI -- tanpa fitur yang baik, model tidak bisa belajar apapun.

Analogi: Feature Engineering adalah alat ukur -- thermometer, barometer, kompas -- yang mengubah data mentah menjadi informasi bermakna.

---

Flow Utama: calculate_all()

Input: DataFrame OHLCV (open, high, low, close, volume)
    |
    |-- calculate_rsi()               -> rsi
    |-- calculate_atr()               -> atr, atr_percent
    |-- calculate_macd()              -> macd, macd_signal, macd_histogram
    |-- calculate_bollinger_bands()   -> bb_upper, bb_lower, bb_width, bb_percent_b
    |-- calculate_ema_crossover()     -> ema_9, ema_21, ema_cross_bull/bear
    |-- calculate_volume_features()   -> volume_ratio, high_volume
    |
    |-- [jika include_ml_features=True]
    |   calculate_ml_features()       -> returns, volatility, lags, trends, time
    |
    v
Output: DataFrame dengan 40+ kolom fitur

Data minimum: 26 bar (kebutuhan MACD slow EMA) agar semua fitur stabil.

---

Kategori 1: Indikator Teknikal

RSI (Relative Strength Index) -- Period 14

Formula: RSI = 100 - (100 / (1 + RS))
         RS  = Average Gain / Average Loss
Smoothing: Wilder's EMA (alpha = 1/14)

Nilai	Interpretasi
RSI > 70	Overbought (potensi turun)
RSI < 30	Oversold (potensi naik)
RSI ~ 50	Netral

Output: rsi

---

ATR (Average True Range) -- Period 14

True Range = max(High-Low, |High-PrevClose|, |Low-PrevClose|)
ATR = Wilder's EMA dari True Range
ATR% = (ATR / Close) * 100

Kondisi	Interpretasi
ATR tinggi	Pasar volatile (pergerakan besar)
ATR rendah	Pasar tenang (pergerakan kecil)

Output: atr, atr_percent

---

MACD (Moving Average Convergence Divergence) -- 12/26/9

MACD Line  = EMA(12) - EMA(26)
Signal     = EMA(MACD Line, 9)
Histogram  = MACD Line - Signal

Kondisi	Interpretasi
Histogram > 0 & naik	Bullish momentum menguat
Histogram < 0 & turun	Bearish momentum menguat
MACD cross Signal ke atas	Potensi reversal naik
MACD cross Signal ke bawah	Potensi reversal turun

Output: macd, macd_signal, macd_histogram

---

Bollinger Bands -- Period 20, StdDev 2.0

Middle = SMA(20)
Upper  = Middle + 2 * StdDev
Lower  = Middle - 2 * StdDev
Width  = (Upper - Lower) / Middle
%B     = (Close - Lower) / (Upper - Lower)

Kondisi	Interpretasi
%B > 1	Harga di atas upper band (extreme bullish)
%B < 0	Harga di bawah lower band (extreme bearish)
%B ~ 0.5	Harga di tengah
Width melebar	Volatility meningkat
Width menyempit	Volatility menurun (squeeze)

Output: bb_middle, bb_upper, bb_lower, bb_width, bb_percent_b

---

EMA Crossover -- 9/21

EMA9  = Exponential Moving Average (cepat)
EMA21 = Exponential Moving Average (lambat)

EMA (Exponential Moving Average) memberikan bobot lebih besar pada data terbaru, sehingga lebih responsif terhadap perubahan harga dibanding SMA.

Kondisi	Interpretasi
EMA9 > EMA21	Trend naik
EMA9 < EMA21	Trend turun
EMA9 cross atas EMA21	Sinyal beli (bullish crossover)
EMA9 cross bawah EMA21	Sinyal jual (bearish crossover)

Output: ema_9, ema_21, ema_cross_bull, ema_cross_bear

---

Kategori 2: Volume Features -- Period 20

volume_sma        = Rolling Mean(volume, 20)
volume_ratio      = volume / volume_sma
volume_increasing = 1 jika volume > volume sebelumnya
high_volume       = 1 jika volume_ratio > 1.5

Fungsi: Konfirmasi breakout -- pergerakan besar harus didukung volume tinggi.

Catatan: Jika kolom volume tidak ada di data, fitur ini di-skip (graceful degradation).

---

Kategori 3: ML-Specific Features

Returns & Momentum

returns_1   = (Close[t] / Close[t-1]) - 1     # Return 1 bar
returns_5   = (Close[t] / Close[t-5]) - 1     # Return 5 bar
returns_20  = (Close[t] / Close[t-20]) - 1    # Return 20 bar
log_returns = ln(Close[t] / Close[t-1])        # Log return

Fungsi: Mengukur kecepatan dan arah pergerakan harga (momentum) dalam berbagai timeframe.

---

Price Position

price_position   = (Close - Low) / (High - Low)   # Posisi 0-1 dalam range candle
dist_from_sma_20 = (Close / SMA20) - 1            # Jarak (%) dari rata-rata

Fungsi: Mengukur dimana harga relatif terhadap range dan rata-rata.

---

Volatility

volatility_20       = StdDev(log_returns, 20)       # Realized volatility
normalized_range    = (High - Low) / Close           # Range sebagai % harga
avg_normalized_range = SMA(normalized_range, 14)     # Rata-rata range 14 bar

Fungsi: Input penting untuk HMM regime detection dan risk sizing. Volatility yang tinggi menandakan pasar bergejolak dan mempengaruhi ukuran posisi.

---

Lag Features

close_lag_1 = Close[t-1]
close_lag_2 = Close[t-2]
close_lag_3 = Close[t-3]
close_lag_5 = Close[t-5]

Fungsi: Auto-regressive features -- menangkap pola harga berulang. Lag features memberikan konteks historis langsung kepada model.

---

Trend Features

higher_high = 1 jika High[t] > High[t-1], else 0
lower_low   = 1 jika Low[t] < Low[t-1], else 0
hh_count_5  = Sum(higher_high, 5 bar)   # Berapa kali HH dalam 5 bar
ll_count_5  = Sum(lower_low, 5 bar)     # Berapa kali LL dalam 5 bar

Fungsi: Mengukur konsistensi trend -- banyak HH = strong uptrend.

---

Time Features

hour           = Jam (0-23)
weekday        = Hari (0=Senin, 6=Minggu)
london_session = 1 jika jam 08:00-16:00 UTC
ny_session     = 1 jika jam 13:00-21:00 UTC

Fungsi: Pasar berperilaku berbeda tiap sesi -- London volatile, Asian tenang. Time features membantu model mengenali pola berbasis waktu.

Catatan: Hanya dihitung jika kolom time bertipe Datetime.

---

Kategori 4: SMC sebagai Fitur Numerik

Dari SMC Analyzer, dikonversi jadi angka untuk XGBoost:

swing_high       = 1 / 0
swing_low        = -1 / 0
fvg_signal       = 1 (bull) / -1 (bear) / 0
ob               = 1 (bull) / -1 (bear) / 0
bos              = 1 (bull) / -1 (bear) / 0
choch            = 1 (bull) / -1 (bear) / 0
market_structure = 1 (bull) / -1 (bear) / 0
regime           = 0 / 1 / 2 (dari HMM)

---

Target Variable (Label Training)

create_target(df, lookahead=1, threshold=0.0):
    target = 1 jika close[t+1] > close[t]   # Harga naik
    target = 0 jika close[t+1] <= close[t]   # Harga turun/tetap
    target_return = close[t+1] / close[t] - 1  # Return kontinu

Catatan: Target dibuat saat training saja, tidak saat live trading.

---

Preprocessing untuk ML

Penanganan Null

# Bar awal memiliki NaN karena lookback period
# Saat training: baris dengan NaN di-drop
df_clean = df.select(features + [target]).drop_nulls()

Penanganan Infinity

# Saat prediksi: NaN & infinity diganti 0
X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)

Normalisasi

Tidak dilakukan -- XGBoost berbasis tree, scale-invariant (tidak perlu scaling).

Cleanup Kolom Temporary

Setiap method membersihkan kolom sementara yang diawali _ (misal _delta, _avg_gain, dll).

---

Fitur yang Digunakan vs Tidak

Digunakan oleh XGBoost (24+ fitur)

Semua indikator teknikal, returns, volatility, trend, time features, SMC numerik, regime.

Tidak Digunakan (Excluded)

• Kolom OHLCV asli: time, open, high, low, close, volume
• Kolom meta: spread, real_volume, target, target_return
• Kolom SMC level: fvg_top, fvg_bottom, ob_top, ob_bottom, dll
• Kolom temporary: apapun yang diawali _

---

Parameter Konfigurasi

Indikator	Parameter	Default	Configurable
RSI (Relative Strength Index)	period	14	Ya
ATR (Average True Range)	period	14	Ya
MACD (Moving Average Convergence Divergence)	fast/slow/signal	12/26/9	Ya
Bollinger Bands	period, std_dev	20, 2.0	Ya
EMA Crossover	fast/slow	9/21	Ya
Volume	period	20	Ya
Returns	lookback	[1, 5, 20]	Hardcoded
Volatility	window	20	Hardcoded
Session	London hours	08-16 UTC	Hardcoded
Session	NY hours	13-21 UTC	Hardcoded

---

Performa

• 5000 bar features: < 100ms (sangat cepat)
• Framework: Pure Polars vectorized (10-100x lebih cepat dari Pandas loop) -- bukan Pandas
• Memory: ~1.6MB untuk 40+ fitur x 5000 bar
• Total fitur: 40+ kolom numerik siap ML