XauBot/docs/arsitektur-ai/13-Auto-Trainer.md

Auto Trainer --- Sistem Retraining Otomatis

File: src/auto_trainer.py Class: AutoTrainer Database: PostgreSQL (opsional, fallback ke file)

---

Apa Itu Auto Trainer?

Auto Trainer adalah sistem yang melatih ulang model AI secara otomatis agar tetap up-to-date dengan kondisi pasar terbaru. Retraining dilakukan saat market tutup (05:00 WIB) untuk menghindari gangguan saat trading aktif.

Analogi: Auto Trainer seperti pelatih yang membuat atlet berlatih setiap malam --- setelah pertandingan selesai, atlet (model AI) dilatih dengan data terbaru agar siap menghadapi tantangan esok hari.

---

Jadwal Retraining

Tipe	Waktu	Data	Boost Rounds	Kondisi
Daily	05:00 WIB (market close)	8.000 bar	50	Senin--Jumat
Weekend	05:00 WIB Sabtu/Minggu	15.000 bar	80	Deep training
Emergency	Kapan saja	8.000 bar	50	AUC < 0.65
Initial	Pertama kali	8.000 bar	50	Belum pernah training

Visualisasi Jadwal (WIB):

Sen  Sel  Rab  Kam  Jum  Sab  Min
 |    |    |    |    |    |    |
05:00 05:00 05:00 05:00 05:00 05:00 05:00
Daily Daily Daily Daily Daily DEEP  DEEP
8K    8K    8K    8K    8K    15K   15K

---

Konfigurasi

AutoTrainer(
    models_dir="models",                  # Folder simpan model
    data_dir="data",                      # Folder data training
    daily_retrain_hour_wib=5,             # Jam retrain: 05:00 WIB
    weekend_retrain=True,                 # Deep training weekend
    min_hours_between_retrain=20,         # Min 20 jam antar retrain
    backup_models=True,                   # Backup model lama
    use_db=True,                          # Simpan history ke PostgreSQL
    min_auc_threshold=0.65,               # Alert jika AUC < 0.65
    auto_retrain_on_low_auc=True,         # Auto retrain saat AUC rendah
)

---

Proses Retraining (Step-by-Step)

Flowchart Keputusan Retraining

flowchart TD
    A[Cek should_retrain] --> B{Sudah >= 20 jam\nsejak retrain terakhir?}
    B -- Tidak --> Z[Skip retrain]
    B -- Ya --> C{Jam 05:00 WIB\natau AUC < 0.65?}
    C -- Tidak --> Z
    C -- Ya --> D{Weekend?}
    D -- Ya --> E[Deep training:\n15K bar, 80 rounds]
    D -- Tidak --> F[Daily training:\n8K bar, 50 rounds]
    E --> G[Backup model lama]
    F --> G
    G --> H[Fetch data dari MT5]
    H --> I[Feature Engineering\n+ SMC Analysis]
    I --> J[Train HMM + XGBoost]
    J --> K[Validasi AUC]
    K --> L{Test AUC >= 0.60?}
    L -- Ya --> M[Simpan model baru]
    L -- Tidak --> N[Rollback ke model lama]
    M --> O[Record hasil ke DB]
    N --> O
    O --> P[Selesai]

Detail Langkah-Langkah

1. SHOULD RETRAIN CHECK
   +-- Sudah >= 20 jam sejak retrain terakhir?
   +-- Sekarang jam 05:00 WIB (+-30 menit)?
   +-- Weekend? -> Deep training (15K bar)
   +-- AUC < 0.65? -> Emergency retrain

2. BACKUP MODEL LAMA
   +-- Copy xgboost_model.pkl -> backups/YYYYMMDD_HHMMSS/
   +-- Copy hmm_regime.pkl -> backups/YYYYMMDD_HHMMSS/
   +-- Bersihkan backup lama (simpan 5 terakhir)

3. FETCH DATA TERBARU
   +-- Ambil 8K bar (daily) atau 15K bar (weekend) dari MT5
   +-- Symbol: XAUUSD, Timeframe: M15
   +-- Validasi: minimal 1000 bar

4. FEATURE ENGINEERING
   +-- FeatureEngineer.calculate_all() -> 40+ fitur
   +-- SMCAnalyzer.calculate_all() -> struktur pasar
   +-- create_target(lookahead=1) -> label UP/DOWN

5. TRAINING HMM
   +-- MarketRegimeDetector(n_regimes=3, lookback=500)
   +-- hmm.fit(df)
   +-- Save -> models/hmm_regime.pkl

6. TRAINING XGBOOST
   +-- TradingModel(confidence_threshold=0.60)
   +-- xgb.fit(train_ratio=0.7, num_boost_round=50/80)
   +-- Early stopping: 5 rounds
   +-- Save -> models/xgboost_model.pkl

7. VALIDASI
   +-- Cek Train AUC & Test AUC
   +-- Test AUC < 0.60? -> ROLLBACK ke model lama (v4: dinaikkan dari 0.52)
   +-- Test AUC < 0.65? -> WARNING (alert)
   +-- Test AUC >= 0.65? -> SUCCESS

8. RECORD HASIL
   +-- Simpan ke PostgreSQL (training_runs table)
   +-- Backup ke file (retrain_history.txt)
   +-- Log: durasi, AUC, accuracy, status

---

Backup & Rollback

Sistem Backup

models/
+-- xgboost_model.pkl          # Model aktif
+-- hmm_regime.pkl             # Model aktif
+-- backups/
    +-- 20250206_050015/       # Backup terbaru
    |   +-- xgboost_model.pkl
    |   +-- hmm_regime.pkl
    +-- 20250205_050012/       # Backup kemarin
    |   +-- xgboost_model.pkl
    |   +-- hmm_regime.pkl
    +-- ... (max 5 backup)

Kapan Rollback?

Diagram Validasi AUC

flowchart TD
    A[Model baru selesai di-training] --> B[Hitung Test AUC]
    B --> C{Test AUC >= 0.65?}
    C -- Ya --> D[KEEP model baru]
    D --> D1[Status: SUCCESS]
    C -- Tidak --> E{Test AUC >= 0.60?}
    E -- Ya --> F[KEEP model baru\ndengan WARNING]
    F --> F1[Status: WARNING]
    F1 --> F2[Akan trigger\nemergency retrain nanti]
    E -- Tidak --> G[ROLLBACK ke model lama]
    G --> G1[Status: ROLLBACK]
    G1 --> G2[v4: threshold dinaikkan\ndari 0.52 ke 0.60]

    style D fill:#22c55e,color:#fff
    style D1 fill:#22c55e,color:#fff
    style F fill:#eab308,color:#000
    style F1 fill:#eab308,color:#000
    style F2 fill:#eab308,color:#000
    style G fill:#ef4444,color:#fff
    style G1 fill:#ef4444,color:#fff
    style G2 fill:#ef4444,color:#fff

Ringkasan Keputusan

Kondisi	Aksi	Status
AUC >= 0.65	KEEP model baru	SUCCESS
AUC 0.60--0.65	KEEP tapi WARNING (akan trigger emergency retrain nanti)	WARNING
AUC < 0.60	ROLLBACK ke model lama (v4: dinaikkan dari 0.52, karena 0.52 hampir = acak)	ROLLBACK

Method Rollback

def rollback_models(reason="Manual rollback"):
    """
    1. Ambil backup terbaru dari models/backups/
    2. Copy xgboost_model.pkl kembali ke models/
    3. Copy hmm_regime.pkl kembali ke models/
    4. Record rollback di database
    """

---

AUC Monitoring

Apa Itu AUC?

AUC (Area Under Curve) mengukur seberapa baik model membedakan sinyal BUY vs SELL:

AUC	Arti	Aksi
0.80+	Sangat bagus	Model dalam kondisi prima
0.65-0.80	Bagus	Normal, lanjut trading
0.60-0.65	Minimum	Warning, pertimbangkan retraining
< 0.60	Buruk	ROLLBACK + retraining segera (v4 threshold)
0.50	Sama dengan tebak koin	Model tidak berguna

Auto-Retrain on Low AUC

def should_retrain_due_to_low_auc():
    """
    Cek AUC saat ini:
      AUC < 0.65? -> Perlu retrain
      Tapi: sudah retrain < 4 jam lalu? -> Tunggu
      (mencegah retrain loop)
    """

---

Database Storage

PostgreSQL (Primary)

Table: training_runs
+-- id                 # Auto-increment
+-- training_type      # "daily" / "weekend"
+-- bars_used          # 8000 / 15000
+-- num_boost_rounds   # 50 / 80
+-- started_at         # Timestamp mulai
+-- completed_at       # Timestamp selesai
+-- duration_seconds   # Durasi training
+-- hmm_trained        # Boolean
+-- xgb_trained        # Boolean
+-- train_auc          # AUC di data training
+-- test_auc           # AUC di data test
+-- train_accuracy     # Akurasi training
+-- test_accuracy      # Akurasi test
+-- model_path         # Path model disimpan
+-- backup_path        # Path backup model lama
+-- success            # Boolean
+-- error_message      # Pesan error (jika gagal)

File Fallback

Jika PostgreSQL tidak tersedia:

data/retrain_history.txt
+-- 2025-02-06T05:00:15+07:00
+-- 2025-02-05T05:00:12+07:00
+-- ... (append per retrain)

---

Integrasi di Main Loop

# main_live.py — dicek setiap 20 candle M15 (~5 jam)
# v4: candle-based, bukan time-based (sebelumnya: loop_count % 300)

if candle_count % 20 == 0:  # Setiap 20 candle baru
    should_train, reason = auto_trainer.should_retrain()

    if should_train:
        logger.info(f"Auto-retraining: {reason}")

        # Retrain (blocking — tapi hanya di jam 05:00 saat market tutup)
        results = auto_trainer.retrain(
            connector=mt5,
            symbol="XAUUSD",
            timeframe="M15",
            is_weekend=(now.weekday() >= 5),
        )

        if results["success"]:
            # Reload model di memory
            ml_model.load()
            regime_detector.load()
            logger.info("Models reloaded after retraining")
        else:
            logger.error(f"Retraining failed: {results['error']}")

---

Parameter Training

Daily Training (Senin-Jumat)

Parameter	Nilai
Data	8.000 bar M15 (~83 hari)
Train/Test Split	70% / 30%
XGBoost Rounds	50
Early Stopping	5 rounds
HMM Regimes	3
HMM Lookback	500 bar

Weekend Deep Training (Sabtu-Minggu)

Parameter	Nilai
Data	15.000 bar M15 (~156 hari)
Train/Test Split	70% / 30%
XGBoost Rounds	80
Early Stopping	5 rounds
HMM Regimes	3
HMM Lookback	500 bar

---

Safety Guards

1. MIN 20 JAM ANTAR RETRAIN
   -> Mencegah retrain terlalu sering
   -> Exception: emergency retrain (min 4 jam)

2. VALIDASI DATA MINIMUM
   -> Butuh minimal 1000 bar
   -> Kurang dari itu? Skip retrain

3. BACKUP SEBELUM RETRAIN
   -> Model lama selalu di-backup
   -> Bisa rollback kapan saja

4. AUTO-ROLLBACK
   -> AUC < 0.60? Otomatis rollback (v4: dinaikkan dari 0.52)
   -> Model buruk tidak akan dipakai

5. CLEANUP BACKUP
   -> Hanya simpan 5 backup terakhir
   -> Mencegah disk penuh

6. GRACEFUL DEGRADATION
   -> DB tidak tersedia? Pakai file
   -> Retrain gagal? Model lama tetap aktif

---

Contoh Output Log

[05:00] ==================================================
[05:00] AUTO-RETRAINING STARTED
[05:00] Type: daily, Bars: 8000, Boost Rounds: 50
[05:00] ==================================================
[05:00] Models backed up to models/backups/20250206_050015
[05:00] Fetching 8000 bars of XAUUSD M15 data...
[05:01] Received 8000 bars
[05:01] Date range: 2024-11-15 to 2025-02-06
[05:01] Applying feature engineering...
[05:01] Training HMM Regime Model...
[05:01] HMM model trained and saved
[05:02] Training XGBoost Model...
[05:02] XGBoost trained: Train AUC=0.7234, Test AUC=0.6891
[05:02] Training data saved to data/training_data.parquet
[05:02] ==================================================
[05:02] AUTO-RETRAINING COMPLETED SUCCESSFULLY
[05:02] Duration: 125s
[05:02] ==================================================