- Introduced `strategy_backtest.py` as a single source of truth for scoring and backtesting.
- Implemented functions to load live model bundles, build feature frames, and execute backtests.
- Ensured consistency between optimized models and their validation/trading counterparts.
- Added functionality for generating trading signals and retrieving the latest trading decision.
- Updated signal engine logic to improve trade decision-making with new probability thresholds and dynamic gates.
- Introduced new `cpcv_reporting.py` module for generating comprehensive CPCV reports, including optimization summaries and performance metrics.
- Added `optuna_param_guard.py` to enforce parameter validation for Optuna optimization, ensuring only approved parameters are used.
- Created `regime_diagnostics.py` for summarizing regime statistics and trade distributions, facilitating better analysis of trading strategies.
- Introduced `test_lifecycle_parity.py` to verify event timelines between live and backtest evaluations, ensuring consistent activation, partial close, scalp exit, and regime shift behaviors across different timeframes.
- Added `test_lifecycle_units.py` for unit testing of lifecycle configurations, including cent normalization, activation thresholds, and the absence of R-multiple fields.
- Created `test_three_state_contract.py` to enforce and validate the three-state regime (TREND, MEAN_REVERSION, VOLATILITY_SHOCK) across various components, ensuring that any attempt to use more than three states raises appropriate errors.
- Integrated regime model predictions into the validation process, allowing for fallback to ensemble models.
- Updated validation criteria to include checks for maximum drawdown, minimum trades, and MR leaks.
- Enhanced test suite to cover new validation scenarios and ensure robustness of strategy evaluation.
- Adjusted signal pipeline audit tests to reflect changes in regime handling and validation logic.
- Implemented `add_specialized_features` to compute ATR moving averages, expansion ratios, realized volatility ratios, and session proximity features.
- Integrated the new feature set into the processing pipeline.
- Updated `run_sensitivity` to utilize new Z-score thresholds from `SignalEngine`.
- Refactored `SignalEngine` to include a unified `SignalDecision` class for consistent entry/exit logic.
- Enhanced `evaluate_bar` to incorporate dynamic probability thresholds and M15 entry gates.
- Added comprehensive tests for signal generation, decision parity, and dynamic thresholds.
- Implemented auto-sync for raw CSV data from MT5 in cmd_optimize, ensuring missing bars are appended if the terminal is running.
- Added ensure_data_updated function in mt5_sync.py to check and fetch missing bars for specified timeframes, logging warnings if MT5 is unavailable.
- Introduced median filtering in engine_hmm.py to smooth HMM state predictions, reducing noise from regime flips.
- Updated predict_states to apply median filtering after state remapping for consistency.
- Added time-of-day gating in processor.py to filter out Asian-session bars, retaining only London/NY high-liquidity hours for analysis.
- Removed TCN staleness checks and auto-retraining from guardian.py.
- Eliminated TCNMaintenanceScheduler and related functions from tcn_maintenance.py.
- Updated mt5_trader.py to remove TCN maintenance checks.
- Adjusted signal_engine.py to remove TCN multiplier logic.
- Cleaned up processor.py by removing optional LSTM context features.
- Modified validation warnings to suggest appropriate periods based on trade counts.
- Updated visualizer.py to streamline signal computation and improve clarity in equity curve plotting.
- Added `--stage` argument to `cmd_optimize` for selecting between Stage-1 (XGB exploration) and Stage-2 (CPCV optimization).
- Introduced `run_optimization_stage1` for fast single hold-out XGB trials, significantly improving optimization speed.
- Updated `run_optimization` to support warm-starting from Stage-1 best parameters.
- Enhanced logging with dedicated score logging for trial monitoring.
- Adjusted hyperparameter ranges for XGBoost to allow deeper trees and finer splits.
- Added synthetic VIX feature computation to the feature application process.
- Persisted best parameters from Stage-1 to JSON for future use in Stage-2.
- Updated documentation for new optimization stages and usage instructions.
- Added WFO parameters and trial configurations for H1, M15, and M5 timeframes.
- Introduced _run_wfo function to evaluate hyperparameters across non-overlapping OOS windows.
- Updated make_objective to utilize WFO instead of CPCV for scoring.
- Enhanced asset feature processing in processor.py to include additional external assets (XAGUSD, XTIUSD, US500, USDJPY) and their log-return calculations.
- Implemented a generic load_asset_data function to streamline loading and processing of asset data.
- Added compute_synth_vix function to calculate a synthetic VIX proxy from OHLC data.
- Improved handling of missing data and logging for asset features.
- Introduced CPCV for train/test splitting, allowing for more robust evaluation across multiple market regimes.
- Updated backtester to support test masks, ensuring trades are only executed during test periods.
- Refactored optimizer to compute scores using CPCV, improving trial efficiency and accuracy.
- Adjusted signal engine logic to incorporate directional probabilities for trade entries based on XGBoost predictions.
- Improved overall code structure and readability, ensuring better maintainability and performance.
- Deleted `signal_evaluator.py`, which contained the previous Z-Score and RCEV evaluation logic.
- Added `signal_engine.py`, implementing a new stateful signal engine for regime confirmation and trade management.
- The new engine evaluates entry and exit conditions based on regime stability, XGBoost confidence, and Bollinger Band positions.
- Introduced constants for minimum confirmation bars, entry probabilities, maximum hold bars, and profit erosion thresholds.
- Introduced RCEVScorer class for calculating expected dollar profit based on historical trade outcomes and market conditions.
- Replaced Z-Score signal gating with RCEV scoring in SignalEvaluator for live evaluations.
- Added new configuration for RCEV thresholds and tiered floors based on timeframes.
- Enhanced MT5 trader logic to utilize RCEV scoring for trade decision-making, including regime-aware take-profit and stop-loss calculations.
- Updated ATR trailing stop logic to incorporate new RCEV parameters and configurations.
- Maintained backward compatibility with existing Z-Score logic for scenarios without RCEV calibration.
- Introduced `engine_tcn.py` for Temporal Convolutional Network (TCN) model to score confidence based on recent market data.
- Added `data_updater.py` for automatic weekly data updates from MT5, ensuring fresh data for TCN.
- Created `tcn_maintenance.py` for automatic TCN model maintenance, including staleness checks and background retraining.
- Updated `mt5_trader.py` to integrate TCN confidence scoring, replacing the previous LSTM ensemble logic.
- Enhanced logging and error handling throughout the new modules for better traceability.
- Removed LSTM context features from feature column selection in `get_feature_cols`.
- Updated `compute_live_features` to eliminate LSTM context model dependency and return the full live DataFrame.
- Enhanced `_run_loop_inner` to load LSTM regime classifier and adjust signal evaluation based on LSTM predictions.
- Introduced Z-Score cutoff parameters in `make_objective` for better signal selection.
- Added `load_data_with_hmm_labels` function to attach HMM state labels to processed data.
- Updated validation process to include Z-Score cutoff configuration.
New opt-in use_tiered mode reduces the Bull/Bear Z-Score requirement when
XGBoost outputs a high-conviction probability, down to a floor of 1.0.
MR (Chop) cutoffs are never affected. The feature is additive — all paths
with use_tiered=False (default) behave identically to before.
Changes:
- signal_evaluator.py: _calculate_tiered_override() returns 0/0.25/0.5/1.0
based on |prob - 0.50| extremity; evaluate_signal_fast() applies reduction
to bull_cut/bear_cut when use_tiered=True; evaluate_signal() applies the
same reduction to req['min_z_score'] and clears Z-based blocks post-reduction
(bar-count blocks are never overridden)
- backtester.py: use_tiered threaded through vectorized_backtest ->
compute_signals_zscore -> evaluate_signal_fast
- sensitivity.py: use_tiered param on run_sensitivity; tables/files suffixed
with _tiered for side-by-side comparison; mode label shown in header
- mt5_trader.py: use_tiered threaded through run_live_loop ->
_run_loop_inner -> evaluate_signal
- main.py: --tiered CLI flag added; threaded to cmd_sensitivity, cmd_live,
cmd_demo
Usage:
python main.py --mode sensitivity --tf M15 --broker headway_cent --balance 15 --tiered
python main.py --mode live --tf M15 --broker headway_cent --balance 15 --tiered
New --mode sensitivity command tests Z cutoffs 1.5..3.0 (step 0.25) on the
saved HMM + XGBoost ensemble without retraining. Loads data and models the
same way cmd_report does, then calls vectorized_backtest once per cutoff with
an evaluator_config override injected into SignalEvaluator.
Changes:
- src/sensitivity.py: run_sensitivity() loops over z_range, builds evaluator_config
override, calls vectorized_backtest, prints comparison table, saves CSV+JSON
- src/backtester.py: evaluator_config param added to compute_signals_zscore and
vectorized_backtest; threaded through to SignalEvaluator(config=)
- main.py: cmd_sensitivity loads models identical to cmd_report; MR cutoffs held
constant; sensitivity mode registered in choices + dispatch dict
Usage: python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15
All src.* loggers redirect to logs/goldregimex_{TF}.log on each TF-specific
command (optimize, train, compare, wfa, sync_validate, report, live).
main + notifier loggers keep writing to goldregimex.log for summaries and
Telegram. Switching TFs in a multi-TF loop (e.g. optimize M5+H1) cleanly
swaps handlers. M15 log deferred until current M15 study completes.
- Added `SignalEvaluator` class for shared adaptive signal evaluation logic across backtesting, optimization, and live trading.
- Introduced `compute_regime_stats` function to calculate per-HMM-state probability statistics for Z-Score calibration.
- Enhanced `vectorized_backtest` and `run_walk_forward` functions to support regime statistics and Z-Score signal generation.
- Updated `backtester.py` to include new signal computation method `compute_signals_zscore`.
- Modified `mt5_trader.py` to utilize the new signal evaluation logic and added regime stability tracking.
- Adjusted `optimizer.py` to compute regime statistics during model training and store them for later use.
- Updated validation logic in `validator.py` to incorporate regime statistics for improved signal evaluation.
- Refactored existing signal routing logic to accommodate new adaptive Z-Score evaluation framework.
Add weekly return consistency metric + M5/M15 scoring bonus
BACKTESTER (_compute_metrics):
- Buckets strategy_returns into ~1-trading-week windows (H1:40, M15:160, M5:480 bars)
- Computes weekly P&L std and mean; consistency = 1 - std/(std+|mean|) ∈ [0,1]
* 1.0 = identical income every week * 0.0 = erratic / zero mean
- Requires ≥4 complete windows and ≥5 trades; 0.0 otherwise
- Added as "return_consistency" to _compute_metrics return dict
- Surfaces as "oos_return_consistency" in vectorized_backtest split result
(and propagates automatically into WFA window dicts via **result_w)
OPTIMIZER (_score_result + objective):
- Consistency bonus: score += return_consistency * 0.5 for M5 and M15 only
* H1 excluded: swing TF with 1-2 trades/week; weekly bucketing is noise
* Max bonus: +0.5 on a 0-25+ scale — a gentle nudge, not a dominant term
- "oos_return_consistency" threaded into oos_result dict so _score_result sees it
- Trial log updated: Consist=%.2f column visible for all TFs
MAIN (cmd_wfa):
- fold_r includes "return_consistency" from oos_return_consistency window key
- _calc_score(fold_r, tf=tf) now passes TF so consistency bonus applies to
M5/M15 WFA fold reports (was defaulting to tf="H1", bonus was silently 0)
- main.py cmd_wfa: lower degenerate HMM guard from 0.72 to 0.65 (matches
optimizer persistence gate set in previous session)
- main.py cmd_wfa per-fold: append ⚠️PF<1.2 to fold flag when OOS profit
factor falls below the 1.2 robustness threshold (PF is already scored via
Complex Criterion but now also surfaced as an explicit standalone warning)
- backtester: add format_payout() helper — Cents (×100) for headway_cent,
USD for standard; add _compute_audit_metrics() computing avg_efficiency
(ATR/spread on active bars), cost_efficiency (1 - costs/gross_profit),
and total_net_payout (return × account_size); wire audit metrics into
vectorized_backtest for full period, IS slice, and OOS slice; update IS/OOS
log lines to show Eff and CostEff
- optimizer: import format_payout; trial logger now shows RF, PF, Payoff$,
Eff (ATR/spread), FloatDD%, trades, and multi-currency payout string
- validator: extract avg_efficiency, cost_efficiency, total_return from result;
expose all three in returned dict and logger
- main: import format_payout; cmd_train _fmt_block adds Efficiency, CostEff,
and Total Payout lines with ⚠️ warning when CostEff < 50%; cmd_sync_validate
shows efficiency + payout + warns on Eff < 1.2 or CostEff < 50%;
cmd_wfa per-fold row now includes Eff column
- backtester: lower spread-aware ER filter threshold 1.8 → 1.25 to allow
standard market conditions while blocking dead sessions (>80% spread erosion)
- engine_xgb: add TF_TRAIN_RATIO constant (H1=70%, M15/M5=65%); prepare_features
gains tf param so scaler is fitted on the correct IS fraction per timeframe
- optimizer: relax HMM persistence gate 0.80 → 0.72 for more active regime
detection; H1 prob_threshold range 0.58-0.78 → 0.55-0.72; short_threshold
0.22-0.42 → 0.28-0.45; reg_alpha H1/M15 capped at 1.2 (was 10.0); gamma
capped at 0.5 (was 5.0); max_depth H1/M15 raised to 3-6; payoff floor
$0.035 applied after trade-count penalty; oos_expected_payoff added to
oos_result dict; trial logger shows OOS_Payoff in dollars
- validator: prepare_features call passes tf param; spread-payoff ratio warning
fires when spread > 50% of edge
- main: import TF_TRAIN_RATIO; all prepare_features calls pass tf=tf; all
train_xgb_ensemble calls pass train_ratio from TF_TRAIN_RATIO
- models: delete stale study_headway_cent.db (incompatible with new scoring formula)
- Removed AdaptiveRiskManager trade limits from optimization and training outputs.
- Introduced DailyEquityGate to monitor daily equity and prevent excessive losses.
- Updated backtester to utilize new risk management features and removed session limits.
- Enhanced metrics computation to include recovery factor for better performance evaluation.
- Implemented Gaussian Mixture Model (GMM) for volatility clustering in processor and validator.
- Adjusted optimizer scoring to reward lower drawdown and incorporate recovery factor.
- Cleaned up unused model metadata files and updated database for consistency.
- Improved logging and error handling across various modules for better traceability.
- data_consolidator: add consolidate_usdchf_m15/m5 producing USDCHF_master_M15.csv
and USDCHF_master_M5.csv; H1 legacy fallback now excludes M5/M15 filenames
- processor: _USDCHF_PATH_BY_TF lookup so each TF loads its matching-frequency
USDCHF master (H1→master.csv, M15→master_M15.csv, M5→master_M5.csv)
- main: cmd_consolidate runs all three TF consolidations in one pass with per-TF summary
- optimizer: H1 n_states reverted [3,4], obs_cov floor raised 0.1→0.5,
MIN_OOS_TRADES_BY_TF["H1"] lowered 200→75
- risk_manager: H1 headway_cent small-account daily cap raised 1→2 trades
- README: per-TF USDCHF filename/master table; dedicated "Getting deep history from MT5"
section (any symbol, any TF) with MetaQuotes-Demo Pro Tip
plot_equity_curve was calling compute_signals with the hardcoded default
threshold (0.65). For M5 where Optuna tunes thresholds to 0.50–0.60, almost
no signals fired → near-zero strategy returns → drawdown annotation showed
"Max DD: 0.0%".
• plot_equity_curve now accepts prob_threshold / short_threshold params
• generate_full_report threads them through
• cmd_report passes params.get("prob_threshold") and params.get("short_threshold")
so the equity curve uses the same Optuna-tuned thresholds as vectorized_backtest
- visualizer.py: _tf_dir now uses reports/<TF>_<broker>/ so M5_headway_cent
and M5_standard charts save to separate folders and never overwrite each other
- All 5 plot functions + generate_full_report accept broker= param and pass it through
- main.py cmd_report: passes broker= to generate_full_report; print updated to
show reports/<TF>_<broker>/ path
- README.md: --mode report added to all four per-TF workflows (H1, M15, M5 cent,
M5 standard); Step 6 updated with correct broker-specific folder tree; command
reference table updated with broker-specific path description
- mt5_trader: replace generic "No signal (need >X or <Y)" with an
explicit reason that names the actual block — regime mismatch vs
probability not cleared (e.g. "Bear state but prob=0.521 not <0.480")
so users can see at a glance whether the HMM or XGBoost gated the bar
- main: replace U+2192 arrow in migration log message with ASCII '->'
to avoid UnicodeEncodeError on Windows cp1252 terminals
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If models/m5_meta_<broker>.json is not found, fall back to the legacy
models/m5_meta.json and copy it to the broker-specific path automatically.
Same migration pattern as the study.db fallback added earlier.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- optimizer.get_best_params: if study_<broker>.db not found, fall back to
legacy models/study.db so optimization runs before the per-broker split
are not silently discarded (prevents 'using defaults' fallback)
- main.cmd_train: abort and sys.exit(1) before saving if HMM min state
persistence < 0.70 — prevents deploying a garbage degenerate model when
Optuna params were not loaded; prints clear remediation steps
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cent and standard account models now live in separate files so
optimizing/training one account never overwrites the other:
- engine_hmm.py: get_model_path(tf, broker) → hmm_model_H1_headway_cent.pkl
- engine_xgb.py: get_ensemble_path(tf, broker) → xgb_ensemble_H1_standard.pkl
- optimizer.py: STUDY_DB replaced by _study_db(broker) →
models/study_headway_cent.db + models/study_standard.db (fully isolated)
- main.py: _M5_META_PATH → _m5_meta_path(broker); thread broker through
all model path calls, cmd_export, cmd_report, _check_m5_readiness
- validator.py: pass broker to hmm_model_path + get_ensemble_path
- mt5_trader.py: pass broker to hmm_model_path + get_ensemble_path
Backward-compat fallback to generic MODEL_PATH / ENSEMBLE_PKL_PATH
preserved if broker-specific file absent (e.g. models not yet retrained).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AdaptiveRiskManager: add broker param; standard guard forces pos_per_trade=1
for growth accounts with balance<$50 to prevent unsafe dual-lot margin
- backtester: thread broker through apply_session_limits + ARM instantiations
- mt5_trader: broker-aware MIN_SPREAD_RATIO dict (cent=1.5, standard=3.0);
spread viability guard now applies to all TFs on standard accounts;
pass broker= to both ARM instantiations in live loop
- optimizer: broker-aware M5 prob thresholds (cent: 0.50-0.53, standard: 0.55-0.60)
to account for higher per-trade costs on standard accounts
- main.py: pass broker= to ARM display calls in cmd_optimize/train/report
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Demo vs Live:
- New --mode demo: sends real orders to MT5 demo, no confirmation prompt
- --mode live: sends real orders, keeps WARNING + YES confirmation
- Removed --account argument (demo/live distinction now done via --mode)
- Removed dry_run feature entirely — all execution modes send real orders
- remote_control.py START commands use --mode live --yes (no --account)
P&L fixes:
- Cent account detection: raw MT5 balance > 10 000 → divide by CENT_MULTIPLIER
to convert cUSD → real USD in all P&L logs and Telegram messages
- Pips added: price movement shown as "+X.XX pts" / "-X.XX pts" in log
and Telegram after every closed position
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Luke Mosoti