- Delete all old XGBoost/HMM models trained with the order-block look-ahead
leak (models/backups/* + root models). They reproduced a fake 63.9% WR /
2.64 PF that collapses to ~35% WR / 0.95 PF once the leak is fixed.
- scripts/collect_data.py: dedicated raw M1+M15 collector (paginated)
- scripts/fast_backtest.py: vectorized GPU backtest for honest validation
- backtest_live_sync.py: read SYMBOL from env (XM uses GOLD, not XAUUSD)
- stop tracking generated data/training_data.parquet
See upstream report: GifariKemal/xaubot-ai#4
Add M1+M15 multi-timeframe SMC scalping training pipeline (GPU XGBoost),
then fix data-leakage and non-stationarity issues found in a skeptical audit.
Pipeline:
- src/triple_barrier.py: TP/SL/time labeling (ATR-scaled, asymmetric RR)
- src/multi_tf_dataset.py: M1 base + M15 HTF context, point-in-time join_asof
(only CLOSED M15 candles visible to each M1 bar - proven no leakage)
- src/economic_calendar.py: point-in-time forecast/actual/surprise provider
- src/smc_polars.py: add premium/discount + displacement SMC features
- scripts/train_multitf_scalper.py: GPU (device=cuda) training + walk-forward
- scripts/download_training_data.py: 1y data downloader
Leakage / robustness fixes (audit):
- CRITICAL: order block signal was written to the ORIGIN bar (future info);
now assigned at the CONFIRMATION bar -> matches live conditions
- replace non-stationary absolute features (ema_9/21, macd*) with scale-free
forms (ema*_dist_atr, ema_spread_atr, macd_*_bps) -> valid at any price level
- drop constant-zero calendar features from defaults (recurring provider has
no real values); re-add when a real calendar CSV is configured
- walk-forward + train/test now embargo the max_holding label horizon and drop
warmup rows (NaN->0 artifacts)
- news calendar features remain point-in-time (actual only at/after release)
Honest result: after fixes the spurious +2.35% edge collapses to ~random
(AUC 0.49). The prior edge was caused by the order-block look-ahead. Pipeline
is now leakage-free; a real edge still needs more M1 history / better features.
Also: test infra (pytest.ini asyncio, hmmlearn), TRAIN_BARS, cleanup of dead
modules. 14 tests pass.
Make the bot connect to MetaTrader5 on Linux without manual setup or
launching extra programs.
- Add mt5linux rpyc bridge: MT5 terminal + Windows Python run under Wine,
Linux client talks to it over a socket
- scripts/mt5_bridge.sh: robust, idempotent launcher (up/down/status/restart/
login-gui/install-service) with .env loading, port health-checks and a
systemd --user unit for auto-start on login
- src/mt5_connector.py: auto-detect bridge backend (lazy, non-blocking import);
auto-start the bridge via mt5_bridge.sh when the port is down so the bot
self-connects; attach-mode initialize() first, then explicit login
- Use configured SYMBOL instead of hardcoded XAUUSD in connector + news_agent
(XM names spot gold 'GOLD')
- scripts/test_mt5_bridge.py: connectivity smoke test
- docs/MT5-ARCH-LINUX-SETUP.md: full setup, root-cause notes, troubleshooting
Verified live: account 345454551 @ XMGlobal-MT5 10, balance read, GOLD M15
data + ticks streaming, trade_allowed=True.
session_name variable not in scope at line 1716.
Get current session from session_filter.get_status_report() instead.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Night Safety blocked trades at 57 pips spread during Golden Session.
Golden Session has extreme volatility, spread 50-80 pips is normal.
Change: Golden Session night spread limit 50 → 80 pips
Normal night hours remain 50 pips limit.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Trade #162626070 lost -$6.07 despite 78% conf prediction of +$3.81 recovery.
Actual market showed +$5.05 profit would have been achieved 31 min later.
Changes:
- Golden Emergency: 45s → 60s threshold (align with grace floor)
- Trajectory Override: If pred>0, conf>75%, accel>0 → delay emergency exit
- Hybrid Hold: Enable trajectory hold for never-profitable IF Golden + strong signal
- Recovery time: 47s max → up to 15 min (if strong recovery detected)
Safety nets maintained: $15 NO_RECOVERY, $20 EMERGENCY_MAX_LOSS
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Fuzzy/Kelly grace threshold: 200 ($200) → 2.0 ($2) — was suppressing ALL loss exits
- Fuzzy/Kelly grace period: unified with dynamic grace_minutes (respects ever_profitable, Golden)
- NO_RECOVERY: 1500 ($1500) → 15.0 ($15) — safety net now actually triggers
- EMERGENCY_MAX_LOSS: 2000 ($2000) → 20.0 ($20) — safety net now actually triggers
- Golden emergency exit: never-profitable + loss >$5 + 45s → immediate cut
- Golden grace floor: 1.0 min (never-prof) / 1.5 min (ever-prof), was 2.0 min
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Trajectory HOLD now requires ever_profitable=True
- Never-profitable trades: trajectory recovery prediction ignored
- Trajectory OVERRIDE for fuzzy exit also requires ever_profitable
- Saves ~$2.50 per trade (close at -$3.97 instead of -$6.47)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix#3: Grace period capped at 2min for trades that NEVER saw profit
- Fix#4: effective_max_loss and max_atr_loss can only tighten (monotonic)
- Golden Session: loss_mult*0.70, profit_mult*0.85, grace*0.60
- market_context now includes is_golden, session_name, session_volatility
- Enhanced dynamic log with [GOLDEN] tag, ratchet values, ever_profitable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL FIX: v0.2.3 was still blocking SMC signals!
Problem:
- v0.2.3 had 3-tier logic that BLOCKS SMC 60-75% if ML disagrees
- Signal BUY 63% + ML HOLD 50% was BLOCKED (wrong!)
- Original v4 intention: SMC-only, ML for boost only
Root Cause:
- v0.2.3 logic still required ML agreement for medium tier
- This contradicts "SMC patokan utama, ML pendukung"
Solution v0.2.4:
- Single threshold: SMC >= 55% executes ALWAYS
- ML role: OPTIONAL boost (average) or ignored
- SELL filter: Only exception (requires ML >= 75%)
- No more tiers, no more ML blocking
Impact:
- SMC BUY 63% + ML HOLD → Now EXECUTES (was blocked)
- True SMC-only mode restored
- ML is reference/boost only
Files:
- main_live.py: Logic v6 (SMC-only)
- VERSION: 0.2.3 → 0.2.4
- CHANGELOG.md: Full documentation
User feedback: "Perasaan tadi sebelum perbaikan, kita
mengabaikan ML dan fokus SMC saja" - NOW CORRECT!
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
PHILOSOPHY: SMC = PRIMARY, ML = SECONDARY support (not blocker)
Changes:
1. London Filter: Penalty (10%) instead of block
- Before: Block trade if ML < 70% confidence
- After: Reduce confidence by 10%, still execute
2. Signal Logic v5: 3-tier SMC-primary hierarchy
- SMC >= 75%: Execute always (ML optional boost)
- SMC 60-75%: Require ML agreement
- SMC < 60%: Skip (low conviction)
3. Removed SELL confidence filter
- SMC confidence now determines execution
- No more blanket blocking of SELL signals
Impact:
- High SMC confidence trades (75-85%) execute
- No blocking from ML HOLD predictions
- ML still boosts when agrees
- Addresses user feedback: "SMC patokan utama, ML pendukung"
Files:
- main_live.py: Signal aggregation logic rewritten
- VERSION: 0.2.2 -> 0.2.3
- CHANGELOG.md: Full documentation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
NameError: name 'df' is not defined
Line 1920: if 'atr' in df.columns:
Fixed to: cached_df = getattr(self, '_cached_df', None)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
TypeError: unsupported operand type(s) for -: 'float' and 'datetime.datetime'
Line 1462: time.time() - guard.entry_time
Fixed to: time.time() - guard.entry_time.timestamp()
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implement industrial-standard semantic versioning (SemVer 2.0.0) with
automated feature detection and comprehensive changelog management.
New Features:
- VERSION file: Single source of truth for base version (0.0.0)
- src/version.py: Centralized version manager with auto-detection
- CHANGELOG.md: Keep a Changelog format for all changes
- Auto-versioning: Features increment MINOR version automatically
- Version display: Shows in startup banner and logs
Predictive Intelligence (v6.3) Complete:
- src/trajectory_predictor.py: Forecast profit 1-5 minutes ahead
- src/momentum_persistence.py: Detect momentum continuation (0-1 score)
- src/recovery_detector.py: Analyze recovery strength from losses
- src/fuzzy_exit_logic.py: Fuzzy logic exit confidence (0-1)
- src/kalman_filter.py: Kalman filter for velocity smoothing
- src/kelly_position_scaler.py: Kelly criterion position scaling
Version Calculation:
Base 0.0.0 + Kalman(0.1) + Fuzzy(0.1) + Kelly(0.1) +
Trajectory(0.1) + Momentum(0.1) + Recovery(0.1) = v0.6.0
Modified:
- CLAUDE.md: Added comprehensive versioning documentation
- main_live.py: Display version in startup banner
- src/smart_risk_manager.py: Use centralized versioning
Documentation:
- CLAUDE.md: Full versioning guidelines (SemVer, workflows, examples)
- CHANGELOG.md: Initial release documentation with feature tracking
- VERSION: Base version 0.0.0
Benefits:
- Professional version management (industry standard)
- Automatic feature tracking and version updates
- Complete change history with Keep a Changelog format
- Clear upgrade paths (MAJOR.MINOR.PATCH)
Version: v0.6.0 (Kalman + Fuzzy + Kelly + Predictive)
Exit Strategy: v6.3 Predictive Intelligence
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When regime detection fails silently on first iteration, df lacks
regime/regime_confidence columns. XGBoost then gets 74 features
instead of 76 and throws feature_names mismatch error.
Fix: add default regime columns (regime=1, regime_confidence=1.0)
after regime detection block and in the fallback position-check path.
Also upgrade regime error logging from DEBUG to WARNING for visibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Added profit momentum feature research files from previous analysis:
- docs/research/PROFIT_MOMENTUM_CODE_SNIPPET.py — Implementation code
- docs/research/PROFIT_MOMENTUM_INTEGRATION.md — Integration guide
- tests/test_profit_momentum.py — Test script
These files document profit momentum feature exploration (unrelated to
current HMM fix but kept for reference).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enhance PositionGuard in SmartRiskManager with real-time profit velocity
($/s) and acceleration ($/s²) tracking for smarter exit decisions.
Changes:
- Add 7 velocity/acceleration fields to PositionGuard dataclass
- Add _calculate_velocity_acceleration(), _update_stagnation(), get_velocity_summary()
- Add 4 new exit checks: [VEL-EXIT], [DECEL], [VEL-WARN], [STAGNANT]
- Enhance early cut with velocity trigger alternative (vel < -0.4)
- Stricter profit_growing: requires momentum > 0 AND velocity > 0
- Reduce position check interval 10s → 5s for more data points
- Add per-ticket [MOMENTUM] log every 30s in main loop
- Revert unused momentum_tracker integration from position_manager
- Add deprecation note to profit_momentum_tracker.py
All velocity checks respect the 15-minute grace period.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
V2 model stores AUC as xgb_train_score/xgb_test_score instead of V1's
train_auc/test_auc. Dashboard now shows correct AUC: 73.4%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 15-min grace period before early cut (wait 1 M15 candle to develop)
- Replace equity chart with AI Assistant card (real-time insights in Indonesian)
- Merge filter toggles into EntryFilterCard (remove separate FiltersConfigCard)
- Fix filter config API URL typo (8001 → 8000)
- Fix tooltip blocking switch clicks (move Switch outside TooltipTrigger)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
**H1 Bias Fix:**
- Fixed cache check to ensure loop_count=1 always calculates H1 bias
- Changed exception log from DEBUG to WARNING for visibility
- Added log on first calculation (loop==1) in addition to every 4 loops
- Result: H1 bias now correctly calculated from first candle
**Filter Config Infrastructure (WIP):**
- Added FilterConfigManager (src/filter_config.py) for dynamic filter control
- Added data/filter_config.json with 11 entry filters (flash_crash, regime, risk, session, spread, h1_bias, ml_confidence, signal_combination, cooldown, time_filter, market_close)
- Added API endpoints: GET/POST /api/filters/config
- Note: Bot integration pending — requires wrapper around all filter checks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Untrack .claude/settings.local.json (IDE-specific)
- Gitignore: data/bot_status.json, signal_persistence.json, model_metrics.json
- Gitignore: logs/ (all content, not just *.log)
- Gitignore: backtests/.claude/
- Add scripts/check_trade_detail.py utility
- Update data/risk_state.txt to current state
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dark mode: class-based theme toggle with localStorage persistence and flash prevention
- Trade History (/trades): paginated table, stats cards, equity curve chart with DB API endpoints
- Backtest Viewer (/backtests): log parser for 35 backtest results, sidebar + detail + comparison tabs
- Model Insights: dashboard card + dialog showing feature importance, regime distribution, training history
- Alert/Signal Log (/alerts): signal stats, filterable table with execution tracking
- API: 8 new endpoints with psycopg2 DB connection pool
- Dark mode sweep across books page, about dialog, and all dashboard components
- Architecture docs rewritten with Mermaid diagrams (23 docs)
- README and FEATURES.md rewritten bilingual (Indonesian + English)
- main_live.py: write model_metrics.json on startup and retrain
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete temp files: _tmp_analysis.py, nul, dashboard_screenshot.png
- Move ea/ to archive/ea/ (deprecated)
- Move 12 Docker helper scripts (.bat/.sh) to docker/scripts/
- Move 5 Docker docs to docker/docs/
- Move .env.docker.example, requirements-docker.txt to docker/
- Update all scripts with cd to project root for correct path resolution
- Update all doc references to new paths
- Update .gitignore with bot.pid, bot_output.log, *.png patterns
- Update CLAUDE.md, README.md directory trees
Root reduced from ~40 files to 12 essential files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Activate SmartPositionManager (was dead code) - trailing SL, breakeven,
market close handler, drawdown-from-peak protection now wired into live loop
2. Add flash crash detection between candles - _position_check_only() now
checks for flash crashes every 5s instead of only on 15min candle close
3. Fix signal confirmation persistence - use direction-based key instead of
exact price (which changed every candle), persist to file to survive restarts
4. Cache ML/features between candles - stop recalculating 37 features + XGBoost
every 5 seconds when candle hasn't changed, reuse cached values
5. Add H1 multi-timeframe SMC bias filter - fetch H1 data, analyze BOS/CHoCH/OB,
block M15 signals that contradict H1 bias, boost confidence when aligned
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix 'dict' has no attribute 'spread' by using .get("spread", 0)
- Add 3-retry loop to close_position() with fresh price each attempt
- Match retry pattern from send_order() for consistency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move utility scripts to scripts/ (check_market, check_positions, etc.)
- Move test files to tests/ (test_modules, test_mt5_connection, etc.)
- Move deprecated dashboards to archive/
- Move research files to docs/research/
- Add sys.path fix to all moved Python files
- Rewrite README.md with architecture diagram and badges
- Add CLAUDE.md project guide
- Add MIT LICENSE
- Update .gitignore with archive/ pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major Issue #1: Confidence Calculation Calibration
- Added calculate_confidence() method with weighted scoring
- Base 40% + Structure 15% + BOS/CHoCH 12% + FVG 8% + OB 10% + Trend 10%
- Capped at 85% (never 100% certain)
Major Issue #2: Pullback Filter ATR-based
- Replaced hardcoded $2, $1.5 thresholds
- Now uses bounce_threshold = 0.15 * ATR
- consolidation_threshold = 0.10 * ATR
Major Issue #3: Smarter Time-based Exit
- Don't cut winners short if profit growing
- Check ML agreement before timeout
- Extend time to 8h if profit > $10 and growing
Major Issue #4: Slippage Validation
- Check actual vs expected price after execution
- Log warning if slippage > 0.15% of price
- Use actual price for position tracking
Major Issue #5: Partial Fill Handling
- Check if filled volume < requested volume
- Log warning with fill ratio
- Use actual volume for position tracking
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Loop: ~1 detik → candle-based (M15) + position check ~10 detik
- Exit Kondisi 3: Golden Time Hold → Early Cut (Smart Hold dihapus)
- AUC rollback threshold: 0.52 → 0.60
- Train/test: tambah 50-bar gap info
- Timer periodik: candle-based intervals
- Performance: split full analysis vs position-check-only
- Golden Time: hapus referensi hold losers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add model backups from training sessions
- Add training data parquet file
- Add risk state persistence file
- Add research documents (Gemini analysis)
- Update architecture docs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
main_live.py:
- Switch main loop from time-based (1s) to candle-based (M15)
- Add position-only checks between candles (every 10s)
- Fix memory leak in signal persistence dict (cleanup stale entries)
- Raise auto-retrain rollback AUC threshold from 0.52 to 0.60
src/ml_model.py:
- Add 50-bar gap between train/test split to prevent temporal leakage
src/smart_risk_manager.py:
- Remove dangerous "Smart Hold" behavior (holding losers waiting for golden time)
- Replace with proper early cut logic (loss >30% + negative momentum)
src/smc_polars.py:
- Fix lookahead bias in FVG detection (remove shift(-1), use confirmed bars only)
- Fix lookahead bias in Swing Points (use center=False rolling window)
- Fix lookahead bias in Order Blocks (validate with current bar, not future)
- Enforce minimum 1:2 Risk:Reward ratio on all signals
- Always use current_close as entry price (no stale FVG/OB zone prices)
- Add ATR sanity check with realistic XAUUSD default ($12)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vanszs