The previous astype() cast did not resolve the pandas 2.0 all-NaN reindex
issue. Using DatetimeIndex.as_unit('ns') (the pandas-2.0 canonical API)
on both the USDCHF series index and the target idx before calling reindex
ensures exact nanosecond-level timestamp matching regardless of whether
the source was read from CSV (datetime64[s]) or stripped from a UTC-aware
MT5 sync index via tz_localize(None) (datetime64[ns]).
- map_usdchf_to_bars: intraday path now casts idx to match series.index dtype
before reindex (datetime64[ns] from tz_localize vs datetime64[s] from CSV read
caused reindex to return all-NaN, wiping all rows after dropna)
- _apply_features (validator): use _USDCHF_PATH_BY_TF for TF-specific USDCHF
master path; add WARNING log listing any entirely-NaN columns before dropna
to aid future diagnosis
- data_consolidator: add consolidate_usdchf_m15/m5 producing USDCHF_master_M15.csv
and USDCHF_master_M5.csv; H1 legacy fallback now excludes M5/M15 filenames
- processor: _USDCHF_PATH_BY_TF lookup so each TF loads its matching-frequency
USDCHF master (H1→master.csv, M15→master_M15.csv, M5→master_M5.csv)
- main: cmd_consolidate runs all three TF consolidations in one pass with per-TF summary
- optimizer: H1 n_states reverted [3,4], obs_cov floor raised 0.1→0.5,
MIN_OOS_TRADES_BY_TF["H1"] lowered 200→75
- risk_manager: H1 headway_cent small-account daily cap raised 1→2 trades
- README: per-TF USDCHF filename/master table; dedicated "Getting deep history from MT5"
section (any symbol, any TF) with MetaQuotes-Demo Pro Tip
Three changes to reduce Risk of Ruin on H1 with a $15 balance:
1. _score_result: doubled DD penalty for H1+small tier (5.0 → 10.0)
— forces optimizer to prefer low-drawdown entries over raw Sharpe on
hourly bars where a single swing can consume the 15% daily DD limit.
2. H1 search space tightened in make_objective:
- n_states locked to 3 (Bull/Bear/Chop) — n_states=4 adds micro-regime
noise on hourly bars without OOS benefit
- max_depth reduced 3–5 → 2–4 (less IS memorisation on limited H1 data)
- min_child_weight raised 1–15 → 15–30 (requires statistically
significant split nodes, discourages noise-driven branches)
M15 search space unchanged.
3. risk_manager: H1 small cent accounts capped at 1 trade/day, 1
position/signal — forces the optimizer and live loop to select the
single highest-conviction daily setup rather than averaging into losers.
USDCHF feature: already conditionally included in prepare_features when
USDCHF_master.csv is present — no change needed, graceful 4-feature
fallback preserved.
Action required: delete models/study_headway_cent.db before re-running
optimize --tf H1 so Optuna builds a fresh surrogate without stale
n_states=4 / min_child_weight<15 trials biasing the model.
plot_equity_curve was calling compute_signals with the hardcoded default
threshold (0.65). For M5 where Optuna tunes thresholds to 0.50–0.60, almost
no signals fired → near-zero strategy returns → drawdown annotation showed
"Max DD: 0.0%".
• plot_equity_curve now accepts prob_threshold / short_threshold params
• generate_full_report threads them through
• cmd_report passes params.get("prob_threshold") and params.get("short_threshold")
so the equity curve uses the same Optuna-tuned thresholds as vectorized_backtest
- visualizer.py: _tf_dir now uses reports/<TF>_<broker>/ so M5_headway_cent
and M5_standard charts save to separate folders and never overwrite each other
- All 5 plot functions + generate_full_report accept broker= param and pass it through
- main.py cmd_report: passes broker= to generate_full_report; print updated to
show reports/<TF>_<broker>/ path
- README.md: --mode report added to all four per-TF workflows (H1, M15, M5 cent,
M5 standard); Step 6 updated with correct broker-specific folder tree; command
reference table updated with broker-specific path description
- State Labels section rewritten: shows Chop (n_states=3), Chop_Low=2
and Chop_High=3 (n_states=4) with mean-return rank explanation
- Added explanation of why M5 uses two Chop sub-states and what
distinguishes them (log-return rank, not volatility)
- Signal logic table updated: "Chop state (2 or 3)" blocked
- Regime-aligned filter note updated to reference both state 2 and 3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- mt5_trader: replace generic "No signal (need >X or <Y)" with an
explicit reason that names the actual block — regime mismatch vs
probability not cleared (e.g. "Bear state but prob=0.521 not <0.480")
so users can see at a glance whether the HMM or XGBoost gated the bar
- main: replace U+2192 arrow in migration log message with ASCII '->'
to avoid UnicodeEncodeError on Windows cp1252 terminals
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If models/m5_meta_<broker>.json is not found, fall back to the legacy
models/m5_meta.json and copy it to the broker-specific path automatically.
Same migration pattern as the study.db fallback added earlier.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Cent vs Standard account section with P&L conversion tables
(explains that +$2.40 cent = $0.024 real USD, standard is 1:1)
- Remove all paper trading / --account demo|live references;
replace with --mode demo (real orders, no prompt) and --mode live (real orders, YES prompt)
- Update model file references to broker+TF naming convention
(hmm_model_H1_headway_cent.pkl etc.)
- Update study DB references to per-broker files (study_headway_cent.db / study_standard.db)
- Add M5 standard account workflow with prob_threshold 0.55-0.60 rationale
- Signal logic table updated: regime-aligned filter (Bull=BUY only, Bear=SELL only, Chop=blocked)
- Probability threshold table per TF per broker with search ranges
- Spread viability guard: cent M5-only 1.5x vs standard all-TFs 3.0x
- Standard account guard documented in risk management table
- Data freshness section explaining IS/OOS split dates and sync_validate scope
- Degenerate HMM training abort guard documented
- Troubleshooting table updated; removed dry_run/--account demo row
- State labels table updated with signal eligibility notes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
README_LIVE.md:
- Cent vs Standard account explanation with real-USD P&L conversion table
(shows that +$2.40 on cent = $0.024 real, standard is 1:1)
- Removed all paper trading / --account demo|live references
- --mode demo and --mode live documented (both send real orders, demo skips prompt)
- Added broker-specific model file tree (hmm_model_H1_headway_cent.pkl etc.)
- Probability threshold table per TF per broker with rationale
- M5 standard account workflow (prob_threshold 0.55-0.60 vs cent 0.50-0.53)
- Updated risk management table with standard account pos_per_trade guard
- Updated troubleshooting entries; removed dry_run reference
- Added Data Requirement section explaining IS/OOS split and sync_validate scope
README_REMOTE.md:
- Study DB references updated to per-broker files (study_headway_cent.db etc.)
- Explain that --trials N is a total target, not additional
- Instructions to delete only the target broker DB, not the shared study.db
- START TRADING button note (uses --mode live + --yes via subprocess)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- optimizer.get_best_params: if study_<broker>.db not found, fall back to
legacy models/study.db so optimization runs before the per-broker split
are not silently discarded (prevents 'using defaults' fallback)
- main.cmd_train: abort and sys.exit(1) before saving if HMM min state
persistence < 0.70 — prevents deploying a garbage degenerate model when
Optuna params were not loaded; prints clear remediation steps
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cent and standard account models now live in separate files so
optimizing/training one account never overwrites the other:
- engine_hmm.py: get_model_path(tf, broker) → hmm_model_H1_headway_cent.pkl
- engine_xgb.py: get_ensemble_path(tf, broker) → xgb_ensemble_H1_standard.pkl
- optimizer.py: STUDY_DB replaced by _study_db(broker) →
models/study_headway_cent.db + models/study_standard.db (fully isolated)
- main.py: _M5_META_PATH → _m5_meta_path(broker); thread broker through
all model path calls, cmd_export, cmd_report, _check_m5_readiness
- validator.py: pass broker to hmm_model_path + get_ensemble_path
- mt5_trader.py: pass broker to hmm_model_path + get_ensemble_path
Backward-compat fallback to generic MODEL_PATH / ENSEMBLE_PKL_PATH
preserved if broker-specific file absent (e.g. models not yet retrained).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AdaptiveRiskManager: add broker param; standard guard forces pos_per_trade=1
for growth accounts with balance<$50 to prevent unsafe dual-lot margin
- backtester: thread broker through apply_session_limits + ARM instantiations
- mt5_trader: broker-aware MIN_SPREAD_RATIO dict (cent=1.5, standard=3.0);
spread viability guard now applies to all TFs on standard accounts;
pass broker= to both ARM instantiations in live loop
- optimizer: broker-aware M5 prob thresholds (cent: 0.50-0.53, standard: 0.55-0.60)
to account for higher per-trade costs on standard accounts
- main.py: pass broker= to ARM display calls in cmd_optimize/train/report
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two guards against the Bull/Chop state-collapse pattern (49K transitions,
identical state means, sub-0.01 persistence):
1. HMM quality gate: any state with self-transition prob < 0.80 → trial
scored -100.0 immediately, before wasting time on XGBoost training.
2. trans_cov ceiling for H1/M15 lowered from 0.1 to 0.03. Values above
0.03 make Kalman noisy and typically cause state collapse with n_states=3.
M5 keeps 0.001-0.1 (different bar structure and obs_cov range).
These fix the case where an optimizer run with n_states=3 returned
OOS Sharpe=2.117 but sync_validate showed Sharpe=-2.67 across all periods
due to degenerate HMM state assignments at inference time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Demo vs Live:
- New --mode demo: sends real orders to MT5 demo, no confirmation prompt
- --mode live: sends real orders, keeps WARNING + YES confirmation
- Removed --account argument (demo/live distinction now done via --mode)
- Removed dry_run feature entirely — all execution modes send real orders
- remote_control.py START commands use --mode live --yes (no --account)
P&L fixes:
- Cent account detection: raw MT5 balance > 10 000 → divide by CENT_MULTIPLIER
to convert cUSD → real USD in all P&L logs and Telegram messages
- Pips added: price movement shown as "+X.XX pts" / "-X.XX pts" in log
and Telegram after every closed position
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
n_states=2 combined with the regime-aligned signal filter has no Chop
state, forcing every bar into Bull or Bear. In a strongly trending
market (e.g. Q1-2026 gold bull run) the HMM mislabels many bullish bars
as Bear, firing SELL signals counter to the trend (WR dropped to 36.6%).
Requiring n_states>=3 for H1/M15 ensures there is always a Chop/neutral
state so ambiguous bars generate no signal instead of a wrong-direction
trade. M5 keeps its [2,4] restriction since n_states=3 is degenerate
on 5-min data.
Added WARNING in train_xgb when hmm_state feature importance=0 — this
indicates XGBoost ignores the HMM while the regime filter still gates on
it, a contradictory setup that should be caught at training time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Regime-aligned signal filter (all TFs, backtester + live trader):
- BUY fires only when hmm_state == Bull (0)
- SELL fires only when hmm_state == Bear (1)
- Chop (2) never generates a signal
Mirrors compute_signals() in backtester exactly so live behaviour
matches what Optuna optimised against.
P&L history fix:
- Retry loop extended from 5s to 20s
- Loop now requires DEAL_ENTRY_OUT (entry==1) to be present before
accepting results — fixes 0.00 P&L caused by only the opening deal
appearing within the original 5-second window
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When profit guard closes a position exactly at entry the P&L rounds to
0.00 but pnl==0.0 triggered the LOSS tag. Now shows BREAK-EVEN / ➡️
emoji so the log is no longer contradictory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
M15 generates ~96 bars/day vs M5's 288 — both now get a 4-trade/day
cap for small accounts so the optimizer can clear MIN_OOS_TRADES=200
thresholds during the coming re-optimization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trade open/close Telegram notifications + P&L history retry fix:
- _log_closed_pnl: retry history_deals_get up to 5× with 1s gap — fixes
'P&L unavailable — not yet in history' (MT5 history lags ~5s post-close)
- _log_closed_pnl: include commission in P&L, extract entry/exit prices
and direction from deal history; log entry -> exit price + lot size
- After order loop: send one Telegram message per filled signal with
direction, regime, prob, lot×N, SL, TP levels and ticket numbers
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The override params were added to run_live_loop() but not threaded
into _run_loop_inner() where the threshold resolution logic lives.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- prob_threshold: 0.52-0.65 -> 0.50-0.58 (ceiling cut stops optimizer settling
at 0.64 where only 40 trades fire per OOS window)
- short_threshold: 0.35-0.48 -> 0.42-0.50 (raises floor for symmetric buy/sell
sensitivity; was asymmetrically biased against shorts)
- README: update probability threshold table for M15 and H1
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- risk_manager: store tf in AdaptiveRiskManager.__init__; get_trade_limits
and can_trade use self.tf by default — M5 small accounts correctly get
4 trades/day without threading tf= through every call site
- mt5_trader: pass tf=tf to both AdaptiveRiskManager instantiations so
the live loop displays and enforces trades=0/4 for M5 (was 0/2)
- README: DXY -> USDCHF throughout; TF-specific model file names; M5/M15/H1
workflow sections; consolidate mode in command reference; correct
short_threshold table; M5 n_states restriction note; updated trade limit
table showing M5 4/day cap; period 6m recommendation for H1/M15 validate
- models/m5_meta.json: updated with latest optimization result (score 4.49)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
remote_control.py:
- Rename button/handler: 'START OPTIMIZE (M5)' → 'OPTIMIZE M5 (0.50-0.55)'
to match the current prob_threshold search space and make the range visible
from the keyboard itself
- Add /start and /help handler: shows a welcome message with a description of
each button; accessible to anyone (auth gate only applies to action commands)
- Enhance BOT STATUS: now shows process health block (Trading Loop ✅/❌,
Optimizer 🔄/💤) above the P&L report; passes session_limit (2 for ≤$50
accounts) so the auditor can show "1/2 trades used today"
- Default LIVE_TF env fallback changed from H1 → M5 to match active timeframe
- Optimize subprocess message updated to confirm range + trial target
auditor.py:
- get_daily_report() accepts optional session_limit param; when provided the
trade count line renders as "1/2 used today" instead of a plain count,
giving the user immediate capacity awareness from the Telegram status button
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
M5 regime overlay and equity curve were rendering 626K bars causing two
visual artefacts:
1. Straight horizontal lines — matplotlib connects Friday 23:55 directly to
Sunday/Monday bar, drawing a line across the weekend gap for every week
in 10 years of data.
2. White fill sections — fill_between() with alpha=0.15 on densely packed
5-min bars produced near-invisible colour overlap that appeared white.
Fix: _downsample() reduces display to max 15K bars (step = n // 15000),
equivalent to ~H1 resolution on M5, before any chart is rendered.
Models and backtest metrics are completely unaffected — downsampling is
cosmetic only. Alpha raised 0.15 → 0.25 and scatter dot size raised for
better visibility at the lower data density.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- optimizer: M5 prob_threshold range narrowed to 0.50-0.55 (live probs cluster
below 0.56; wider range let Optuna miss the workable window entirely)
- optimizer: H1/M15 prob_threshold uses 0.52-0.65 (wider buffer appropriate
for higher timeframes with more information per bar)
- optimizer: --trials N now means total study target, not additional trials;
remaining = max(0, N - already_done) is passed to study.optimize so
resuming with --trials 500 after 400 done runs exactly 100 more
- optimizer: progress bar shows [total_done/total_target] instead of
[session_done/session_target] so resumed runs display true context
- optimizer: Telegram heartbeat pct now based on total_done/total_target
- optimizer: early-exit with clear message when target already reached
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cmd_report was calling vectorized_backtest without prob_threshold, so it
fell back to the hardcoded default (0.65) instead of the Optuna-tuned value
(0.578). This caused the report to show far fewer trades than the optimizer
actually found (38 IS / 9 OOS vs the 50+ OOS the study required).
MIN_OOS_TRADES is now TF-aware: M5=300, M15=100, H1=50. With ~434 OOS
trading days and a 2/day cap, M5=50 allowed ultra-infrequent cherry-picked
wins (0.11 trades/day); 300 forces at least 0.69 trades/day so Optuna
cannot reward near-zero-trade setups with inflated Sharpe ratios.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- README: add dedicated Volatility Ensemble section documenting three-model
bucketing (low/med/high ATR), feature table, threshold explanation, and
ONNX file list; update pipeline diagram, train/export descriptions, and
prob_threshold range (0.505-0.60)
- optimizer.py: lower prob_threshold search space lower bound 0.52 → 0.505
so Optuna can find thresholds matching the model's real output range
- processor.py: add map_dxy_to_bars() helper that forward-fills DXY beyond
file coverage (handles sync data newer than last DXY export); fix
load_dxy_data() date format to match YYYY-MM-DD saved format
- validator.py: use load_xgb_ensemble + get_predictions_ensemble; import
and apply map_dxy_to_bars in _apply_features() so DXY is included in
validation inference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DXY is a daily index exposed by MT5 only at the day level — there is no
per-timeframe variant. Remove the three separate dxy_raw_path entries
from TF_CONFIG and replace them with a single module-level DXY_RAW_PATH
constant (data/raw/DXY_data.csv) that all timeframes share.
process_pipeline() now references DXY_RAW_PATH directly; the pipeline
degrades gracefully (4-feature model) if the file is absent.
README Data Setup section updated with DXY export instructions and the
single expected filename.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
### DXY Integration (mt5_sync.py, processor.py)
- Added DXY_SYMBOL_ALIASES + _find_dxy_symbol() for broker-agnostic USD Index lookup
- Added fetch_cross_asset_bars() — fetches XAUUSD + DXY and inner-joins on timestamps
- Added dxy_raw_path to TF_CONFIG for all timeframes (H1/M15/M5)
- Added load_dxy_data() — loads DXY CSV and computes dxy_log_return
- process_pipeline() now optionally merges dxy_log_return when DXY CSV exists;
falls back gracefully with a warning when no DXY file is present
### Dynamic Feature Detection (engine_xgb.py)
- FEATURE_COLS remains the 4-feature base; get_feature_cols(df) appends
dxy_log_return when it is present and has >50% coverage
- prepare_features() now calls get_feature_cols() so feature count adapts
automatically to data availability (4 or 5 features)
- train_xgb() now uses X.columns for feature_importance (was hardcoded FEATURE_COLS,
which broke with 5-feature data)
### Three-Model Volatility Ensemble (engine_xgb.py)
- compute_vol_thresholds() / assign_vol_bucket() — ATR tertile partitioning
- train_xgb_ensemble() — trains Low/Med/High vol bucket models on IS data only
(thresholds computed from IS to prevent look-ahead bias)
- get_predictions_ensemble() — routes each bar to its bucket model at inference
- save_xgb_ensemble() / load_xgb_ensemble() — persists all 3 models + thresholds
+ feature_cols in models/xgb_ensemble.pkl
- export_onnx_ensemble() — exports xgb_model_vol_{low,med,high}.onnx
### Pipeline wired end-to-end
- optimizer.py objective now trains and scores with train_xgb_ensemble +
get_predictions_ensemble (same OOS-only scoring, same prob_threshold search)
- main.py _train_for_tf(), cmd_train, cmd_export all updated to ensemble
- mt5_trader.py loads xgb_ensemble.pkl; compute_live_features() returns a named
DataFrame; DXY log return is fetched live if model was trained with it (falls
back to 0.0 if broker has no DXY symbol)
### Not changed (already implemented)
- Break-even SL, session limits, cent normalisation — already in place
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers installation, data setup, Telegram config, full step-by-step workflow
for H1/M15/M5 timeframes, signal logic, risk management, MQL5 EA, and
troubleshooting. Replaces the auto-generated GitHub placeholder.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bot token and chat IDs were accidentally committed with real values.
Replaced with placeholder strings. Revoke and regenerate the bot token
via @BotFather /revoke immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Luke Mosoti