- Introduced RCEVScorer class for calculating expected dollar profit based on historical trade outcomes and market conditions.
- Replaced Z-Score signal gating with RCEV scoring in SignalEvaluator for live evaluations.
- Added new configuration for RCEV thresholds and tiered floors based on timeframes.
- Enhanced MT5 trader logic to utilize RCEV scoring for trade decision-making, including regime-aware take-profit and stop-loss calculations.
- Updated ATR trailing stop logic to incorporate new RCEV parameters and configurations.
- Maintained backward compatibility with existing Z-Score logic for scenarios without RCEV calibration.
- Introduced `engine_tcn.py` for Temporal Convolutional Network (TCN) model to score confidence based on recent market data.
- Added `data_updater.py` for automatic weekly data updates from MT5, ensuring fresh data for TCN.
- Created `tcn_maintenance.py` for automatic TCN model maintenance, including staleness checks and background retraining.
- Updated `mt5_trader.py` to integrate TCN confidence scoring, replacing the previous LSTM ensemble logic.
- Enhanced logging and error handling throughout the new modules for better traceability.
- Removed LSTM context features from feature column selection in `get_feature_cols`.
- Updated `compute_live_features` to eliminate LSTM context model dependency and return the full live DataFrame.
- Enhanced `_run_loop_inner` to load LSTM regime classifier and adjust signal evaluation based on LSTM predictions.
- Introduced Z-Score cutoff parameters in `make_objective` for better signal selection.
- Added `load_data_with_hmm_labels` function to attach HMM state labels to processed data.
- Updated validation process to include Z-Score cutoff configuration.
New opt-in use_tiered mode reduces the Bull/Bear Z-Score requirement when
XGBoost outputs a high-conviction probability, down to a floor of 1.0.
MR (Chop) cutoffs are never affected. The feature is additive — all paths
with use_tiered=False (default) behave identically to before.
Changes:
- signal_evaluator.py: _calculate_tiered_override() returns 0/0.25/0.5/1.0
based on |prob - 0.50| extremity; evaluate_signal_fast() applies reduction
to bull_cut/bear_cut when use_tiered=True; evaluate_signal() applies the
same reduction to req['min_z_score'] and clears Z-based blocks post-reduction
(bar-count blocks are never overridden)
- backtester.py: use_tiered threaded through vectorized_backtest ->
compute_signals_zscore -> evaluate_signal_fast
- sensitivity.py: use_tiered param on run_sensitivity; tables/files suffixed
with _tiered for side-by-side comparison; mode label shown in header
- mt5_trader.py: use_tiered threaded through run_live_loop ->
_run_loop_inner -> evaluate_signal
- main.py: --tiered CLI flag added; threaded to cmd_sensitivity, cmd_live,
cmd_demo
Usage:
python main.py --mode sensitivity --tf M15 --broker headway_cent --balance 15 --tiered
python main.py --mode live --tf M15 --broker headway_cent --balance 15 --tiered
New --mode sensitivity command tests Z cutoffs 1.5..3.0 (step 0.25) on the
saved HMM + XGBoost ensemble without retraining. Loads data and models the
same way cmd_report does, then calls vectorized_backtest once per cutoff with
an evaluator_config override injected into SignalEvaluator.
Changes:
- src/sensitivity.py: run_sensitivity() loops over z_range, builds evaluator_config
override, calls vectorized_backtest, prints comparison table, saves CSV+JSON
- src/backtester.py: evaluator_config param added to compute_signals_zscore and
vectorized_backtest; threaded through to SignalEvaluator(config=)
- main.py: cmd_sensitivity loads models identical to cmd_report; MR cutoffs held
constant; sensitivity mode registered in choices + dispatch dict
Usage: python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15
All src.* loggers redirect to logs/goldregimex_{TF}.log on each TF-specific
command (optimize, train, compare, wfa, sync_validate, report, live).
main + notifier loggers keep writing to goldregimex.log for summaries and
Telegram. Switching TFs in a multi-TF loop (e.g. optimize M5+H1) cleanly
swaps handlers. M15 log deferred until current M15 study completes.
- Added `SignalEvaluator` class for shared adaptive signal evaluation logic across backtesting, optimization, and live trading.
- Introduced `compute_regime_stats` function to calculate per-HMM-state probability statistics for Z-Score calibration.
- Enhanced `vectorized_backtest` and `run_walk_forward` functions to support regime statistics and Z-Score signal generation.
- Updated `backtester.py` to include new signal computation method `compute_signals_zscore`.
- Modified `mt5_trader.py` to utilize the new signal evaluation logic and added regime stability tracking.
- Adjusted `optimizer.py` to compute regime statistics during model training and store them for later use.
- Updated validation logic in `validator.py` to incorporate regime statistics for improved signal evaluation.
- Refactored existing signal routing logic to accommodate new adaptive Z-Score evaluation framework.
Add weekly return consistency metric + M5/M15 scoring bonus
BACKTESTER (_compute_metrics):
- Buckets strategy_returns into ~1-trading-week windows (H1:40, M15:160, M5:480 bars)
- Computes weekly P&L std and mean; consistency = 1 - std/(std+|mean|) ∈ [0,1]
* 1.0 = identical income every week * 0.0 = erratic / zero mean
- Requires ≥4 complete windows and ≥5 trades; 0.0 otherwise
- Added as "return_consistency" to _compute_metrics return dict
- Surfaces as "oos_return_consistency" in vectorized_backtest split result
(and propagates automatically into WFA window dicts via **result_w)
OPTIMIZER (_score_result + objective):
- Consistency bonus: score += return_consistency * 0.5 for M5 and M15 only
* H1 excluded: swing TF with 1-2 trades/week; weekly bucketing is noise
* Max bonus: +0.5 on a 0-25+ scale — a gentle nudge, not a dominant term
- "oos_return_consistency" threaded into oos_result dict so _score_result sees it
- Trial log updated: Consist=%.2f column visible for all TFs
MAIN (cmd_wfa):
- fold_r includes "return_consistency" from oos_return_consistency window key
- _calc_score(fold_r, tf=tf) now passes TF so consistency bonus applies to
M5/M15 WFA fold reports (was defaulting to tf="H1", bonus was silently 0)
- main.py cmd_wfa: lower degenerate HMM guard from 0.72 to 0.65 (matches
optimizer persistence gate set in previous session)
- main.py cmd_wfa per-fold: append ⚠️PF<1.2 to fold flag when OOS profit
factor falls below the 1.2 robustness threshold (PF is already scored via
Complex Criterion but now also surfaced as an explicit standalone warning)
- backtester: add format_payout() helper — Cents (×100) for headway_cent,
USD for standard; add _compute_audit_metrics() computing avg_efficiency
(ATR/spread on active bars), cost_efficiency (1 - costs/gross_profit),
and total_net_payout (return × account_size); wire audit metrics into
vectorized_backtest for full period, IS slice, and OOS slice; update IS/OOS
log lines to show Eff and CostEff
- optimizer: import format_payout; trial logger now shows RF, PF, Payoff$,
Eff (ATR/spread), FloatDD%, trades, and multi-currency payout string
- validator: extract avg_efficiency, cost_efficiency, total_return from result;
expose all three in returned dict and logger
- main: import format_payout; cmd_train _fmt_block adds Efficiency, CostEff,
and Total Payout lines with ⚠️ warning when CostEff < 50%; cmd_sync_validate
shows efficiency + payout + warns on Eff < 1.2 or CostEff < 50%;
cmd_wfa per-fold row now includes Eff column
- backtester: lower spread-aware ER filter threshold 1.8 → 1.25 to allow
standard market conditions while blocking dead sessions (>80% spread erosion)
- engine_xgb: add TF_TRAIN_RATIO constant (H1=70%, M15/M5=65%); prepare_features
gains tf param so scaler is fitted on the correct IS fraction per timeframe
- optimizer: relax HMM persistence gate 0.80 → 0.72 for more active regime
detection; H1 prob_threshold range 0.58-0.78 → 0.55-0.72; short_threshold
0.22-0.42 → 0.28-0.45; reg_alpha H1/M15 capped at 1.2 (was 10.0); gamma
capped at 0.5 (was 5.0); max_depth H1/M15 raised to 3-6; payoff floor
$0.035 applied after trade-count penalty; oos_expected_payoff added to
oos_result dict; trial logger shows OOS_Payoff in dollars
- validator: prepare_features call passes tf param; spread-payoff ratio warning
fires when spread > 50% of edge
- main: import TF_TRAIN_RATIO; all prepare_features calls pass tf=tf; all
train_xgb_ensemble calls pass train_ratio from TF_TRAIN_RATIO
- models: delete stale study_headway_cent.db (incompatible with new scoring formula)
- Removed AdaptiveRiskManager trade limits from optimization and training outputs.
- Introduced DailyEquityGate to monitor daily equity and prevent excessive losses.
- Updated backtester to utilize new risk management features and removed session limits.
- Enhanced metrics computation to include recovery factor for better performance evaluation.
- Implemented Gaussian Mixture Model (GMM) for volatility clustering in processor and validator.
- Adjusted optimizer scoring to reward lower drawdown and incorporate recovery factor.
- Cleaned up unused model metadata files and updated database for consistency.
- Improved logging and error handling across various modules for better traceability.
- data_consolidator: add consolidate_usdchf_m15/m5 producing USDCHF_master_M15.csv
and USDCHF_master_M5.csv; H1 legacy fallback now excludes M5/M15 filenames
- processor: _USDCHF_PATH_BY_TF lookup so each TF loads its matching-frequency
USDCHF master (H1→master.csv, M15→master_M15.csv, M5→master_M5.csv)
- main: cmd_consolidate runs all three TF consolidations in one pass with per-TF summary
- optimizer: H1 n_states reverted [3,4], obs_cov floor raised 0.1→0.5,
MIN_OOS_TRADES_BY_TF["H1"] lowered 200→75
- risk_manager: H1 headway_cent small-account daily cap raised 1→2 trades
- README: per-TF USDCHF filename/master table; dedicated "Getting deep history from MT5"
section (any symbol, any TF) with MetaQuotes-Demo Pro Tip
plot_equity_curve was calling compute_signals with the hardcoded default
threshold (0.65). For M5 where Optuna tunes thresholds to 0.50–0.60, almost
no signals fired → near-zero strategy returns → drawdown annotation showed
"Max DD: 0.0%".
• plot_equity_curve now accepts prob_threshold / short_threshold params
• generate_full_report threads them through
• cmd_report passes params.get("prob_threshold") and params.get("short_threshold")
so the equity curve uses the same Optuna-tuned thresholds as vectorized_backtest
- visualizer.py: _tf_dir now uses reports/<TF>_<broker>/ so M5_headway_cent
and M5_standard charts save to separate folders and never overwrite each other
- All 5 plot functions + generate_full_report accept broker= param and pass it through
- main.py cmd_report: passes broker= to generate_full_report; print updated to
show reports/<TF>_<broker>/ path
- README.md: --mode report added to all four per-TF workflows (H1, M15, M5 cent,
M5 standard); Step 6 updated with correct broker-specific folder tree; command
reference table updated with broker-specific path description
- mt5_trader: replace generic "No signal (need >X or <Y)" with an
explicit reason that names the actual block — regime mismatch vs
probability not cleared (e.g. "Bear state but prob=0.521 not <0.480")
so users can see at a glance whether the HMM or XGBoost gated the bar
- main: replace U+2192 arrow in migration log message with ASCII '->'
to avoid UnicodeEncodeError on Windows cp1252 terminals
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If models/m5_meta_<broker>.json is not found, fall back to the legacy
models/m5_meta.json and copy it to the broker-specific path automatically.
Same migration pattern as the study.db fallback added earlier.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- optimizer.get_best_params: if study_<broker>.db not found, fall back to
legacy models/study.db so optimization runs before the per-broker split
are not silently discarded (prevents 'using defaults' fallback)
- main.cmd_train: abort and sys.exit(1) before saving if HMM min state
persistence < 0.70 — prevents deploying a garbage degenerate model when
Optuna params were not loaded; prints clear remediation steps
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cent and standard account models now live in separate files so
optimizing/training one account never overwrites the other:
- engine_hmm.py: get_model_path(tf, broker) → hmm_model_H1_headway_cent.pkl
- engine_xgb.py: get_ensemble_path(tf, broker) → xgb_ensemble_H1_standard.pkl
- optimizer.py: STUDY_DB replaced by _study_db(broker) →
models/study_headway_cent.db + models/study_standard.db (fully isolated)
- main.py: _M5_META_PATH → _m5_meta_path(broker); thread broker through
all model path calls, cmd_export, cmd_report, _check_m5_readiness
- validator.py: pass broker to hmm_model_path + get_ensemble_path
- mt5_trader.py: pass broker to hmm_model_path + get_ensemble_path
Backward-compat fallback to generic MODEL_PATH / ENSEMBLE_PKL_PATH
preserved if broker-specific file absent (e.g. models not yet retrained).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AdaptiveRiskManager: add broker param; standard guard forces pos_per_trade=1
for growth accounts with balance<$50 to prevent unsafe dual-lot margin
- backtester: thread broker through apply_session_limits + ARM instantiations
- mt5_trader: broker-aware MIN_SPREAD_RATIO dict (cent=1.5, standard=3.0);
spread viability guard now applies to all TFs on standard accounts;
pass broker= to both ARM instantiations in live loop
- optimizer: broker-aware M5 prob thresholds (cent: 0.50-0.53, standard: 0.55-0.60)
to account for higher per-trade costs on standard accounts
- main.py: pass broker= to ARM display calls in cmd_optimize/train/report
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Demo vs Live:
- New --mode demo: sends real orders to MT5 demo, no confirmation prompt
- --mode live: sends real orders, keeps WARNING + YES confirmation
- Removed --account argument (demo/live distinction now done via --mode)
- Removed dry_run feature entirely — all execution modes send real orders
- remote_control.py START commands use --mode live --yes (no --account)
P&L fixes:
- Cent account detection: raw MT5 balance > 10 000 → divide by CENT_MULTIPLIER
to convert cUSD → real USD in all P&L logs and Telegram messages
- Pips added: price movement shown as "+X.XX pts" / "-X.XX pts" in log
and Telegram after every closed position
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cmd_report was calling vectorized_backtest without prob_threshold, so it
fell back to the hardcoded default (0.65) instead of the Optuna-tuned value
(0.578). This caused the report to show far fewer trades than the optimizer
actually found (38 IS / 9 OOS vs the 50+ OOS the study required).
MIN_OOS_TRADES is now TF-aware: M5=300, M15=100, H1=50. With ~434 OOS
trading days and a 2/day cap, M5=50 allowed ultra-infrequent cherry-picked
wins (0.11 trades/day); 300 forces at least 0.69 trades/day so Optuna
cannot reward near-zero-trade setups with inflated Sharpe ratios.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
### DXY Integration (mt5_sync.py, processor.py)
- Added DXY_SYMBOL_ALIASES + _find_dxy_symbol() for broker-agnostic USD Index lookup
- Added fetch_cross_asset_bars() — fetches XAUUSD + DXY and inner-joins on timestamps
- Added dxy_raw_path to TF_CONFIG for all timeframes (H1/M15/M5)
- Added load_dxy_data() — loads DXY CSV and computes dxy_log_return
- process_pipeline() now optionally merges dxy_log_return when DXY CSV exists;
falls back gracefully with a warning when no DXY file is present
### Dynamic Feature Detection (engine_xgb.py)
- FEATURE_COLS remains the 4-feature base; get_feature_cols(df) appends
dxy_log_return when it is present and has >50% coverage
- prepare_features() now calls get_feature_cols() so feature count adapts
automatically to data availability (4 or 5 features)
- train_xgb() now uses X.columns for feature_importance (was hardcoded FEATURE_COLS,
which broke with 5-feature data)
### Three-Model Volatility Ensemble (engine_xgb.py)
- compute_vol_thresholds() / assign_vol_bucket() — ATR tertile partitioning
- train_xgb_ensemble() — trains Low/Med/High vol bucket models on IS data only
(thresholds computed from IS to prevent look-ahead bias)
- get_predictions_ensemble() — routes each bar to its bucket model at inference
- save_xgb_ensemble() / load_xgb_ensemble() — persists all 3 models + thresholds
+ feature_cols in models/xgb_ensemble.pkl
- export_onnx_ensemble() — exports xgb_model_vol_{low,med,high}.onnx
### Pipeline wired end-to-end
- optimizer.py objective now trains and scores with train_xgb_ensemble +
get_predictions_ensemble (same OOS-only scoring, same prob_threshold search)
- main.py _train_for_tf(), cmd_train, cmd_export all updated to ensemble
- mt5_trader.py loads xgb_ensemble.pkl; compute_live_features() returns a named
DataFrame; DXY log return is fetched live if model was trained with it (falls
back to 0.0 if broker has no DXY symbol)
### Not changed (already implemented)
- Break-even SL, session limits, cent normalisation — already in place
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Full pipeline: process → optimize → train → export → live
- Optuna optimizer with OOS-only Sharpe scoring and prob_threshold in search space
- MT5 live bridge with adaptive risk management and staged TPs
- M5/M15/H1 timeframe support with per-TF Kalman, HMM, and XGBoost configs
- Telegram remote control, guardian health monitor, and daily audit reports
- MQL5 EA (v2.00) with ONNX inference and cent account support
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Luke Mosoti