This commit introduces a new smoke test script that exercises every phase of the pipeline observability process. The test generates synthetic data for two timeframes (M15 and M5), simulating various stages of the pipeline and verifying that all artifacts are produced and non-empty. The script includes assertions to ensure the integrity of the generated outputs and checks for expected conditions in the decision log, survival matrix, and audit JSON.
- Updated GoldRegime_X.code-workspace to remove deprecated server configuration.
- Added package.json and package-lock.json to manage dependencies, specifically @notionhq/client.
- Introduced patch_explorer_cell46.py to resolve ModuleNotFoundError in the PIPELINE VERIFICATION import cell of GoldRegimeX_Explorer.ipynb.
- Created patch_explorer_cell48.py to fix NameError for TIMEFRAMES in the Dataset Integrity verification cell, ensuring it can derive TIMEFRAMES from the pipeline if not already defined.
- Introduced `pipeline_verification.py` containing multiple verifier classes for dataset integrity, feature leakage, evaluation integrity, prediction alignment, and more.
- Added `SessionFilter` class in `session_filter.py` for unified session/time-of-day filtering.
- Implemented `VerificationResult` data class for structured verification results.
- Created utility functions for hashing and manifest building.
- Established a test suite for the session filter to ensure correct functionality.
- Updated `__init__.py` to include the new shared module.
- Deleted `test_lifecycle_parity.py`, which contained tests for lifecycle event timelines in live and backtest scenarios.
- Deleted `test_lifecycle_units.py`, which included unit tests for lifecycle configurations and floating PnL calculations.
- Deleted `test_three_state_contract.py`, which verified the enforcement of three semantic states across various components.
- Updated parallel processing output times to reflect new execution durations.
- Modified strategy results table with new entries and adjusted values for fixed_tp and fixed_tp_plus_mr strategies.
- Increased maximum drawdown percentage from 30.0 to 35.0.
- Enhanced filtering logic to ensure proper retrieval of results from all_results or valid_results.
- Implemented `generate_bridge_csv.py` to generate reports for strategy winners from the Strategy Tester notebook.
- Added `smoke_test_explorer.py` to perform smoke tests on the Explorer notebook integration, validating core logic and data processing.
- Updated signal engine logic to improve trade decision-making with new probability thresholds and dynamic gates.
- Introduced new `cpcv_reporting.py` module for generating comprehensive CPCV reports, including optimization summaries and performance metrics.
- Added `optuna_param_guard.py` to enforce parameter validation for Optuna optimization, ensuring only approved parameters are used.
- Created `regime_diagnostics.py` for summarizing regime statistics and trade distributions, facilitating better analysis of trading strategies.
- Introduced `test_lifecycle_parity.py` to verify event timelines between live and backtest evaluations, ensuring consistent activation, partial close, scalp exit, and regime shift behaviors across different timeframes.
- Added `test_lifecycle_units.py` for unit testing of lifecycle configurations, including cent normalization, activation thresholds, and the absence of R-multiple fields.
- Created `test_three_state_contract.py` to enforce and validate the three-state regime (TREND, MEAN_REVERSION, VOLATILITY_SHOCK) across various components, ensuring that any attempt to use more than three states raises appropriate errors.
- Integrated regime model predictions into the validation process, allowing for fallback to ensemble models.
- Updated validation criteria to include checks for maximum drawdown, minimum trades, and MR leaks.
- Enhanced test suite to cover new validation scenarios and ensure robustness of strategy evaluation.
- Adjusted signal pipeline audit tests to reflect changes in regime handling and validation logic.
- Implemented `add_specialized_features` to compute ATR moving averages, expansion ratios, realized volatility ratios, and session proximity features.
- Integrated the new feature set into the processing pipeline.
- Updated `run_sensitivity` to utilize new Z-score thresholds from `SignalEngine`.
- Refactored `SignalEngine` to include a unified `SignalDecision` class for consistent entry/exit logic.
- Enhanced `evaluate_bar` to incorporate dynamic probability thresholds and M15 entry gates.
- Added comprehensive tests for signal generation, decision parity, and dynamic thresholds.
- Updated `get_feature_cols` to handle one-hot encoding for HMM states in M5/M15.
- Modified `prepare_features` to include a rolling median for HMM states and drop the original integer state.
- Enhanced `compute_live_features` to generate one-hot encoded features for M5/M15 and retain the integer state for H1.
- Implemented runtime checks to ensure live feature keys match training columns, raising errors for mismatches.
- Adjusted scoring logic in `optimizer.py` to apply log-loss for M5/M15 and backtest for H1, improving robustness against over-regularization.
- Updated search space for `n_states` in models M15 and M5 from categorical to integer range (3 to 4).
- Lowered `reg_alpha` and `reg_lambda` ranges to allow HMM OHE signal through for models M15 and M5.
- Adjusted the `make_objective` function to reflect changes in parameter suggestions for M5 and M15.
- Introduced new model configuration file for stage1 M15 with optimized parameters and score.
- Added binary changes to the study_headway_cent database.
- Implemented auto-sync for raw CSV data from MT5 in cmd_optimize, ensuring missing bars are appended if the terminal is running.
- Added ensure_data_updated function in mt5_sync.py to check and fetch missing bars for specified timeframes, logging warnings if MT5 is unavailable.
- Introduced median filtering in engine_hmm.py to smooth HMM state predictions, reducing noise from regime flips.
- Updated predict_states to apply median filtering after state remapping for consistency.
- Added time-of-day gating in processor.py to filter out Asian-session bars, retaining only London/NY high-liquidity hours for analysis.
- Removed obsolete model configuration file `stage1_m15_headway_cent.json`.
- Updated database file `study_headway_cent.db` to reflect changes in model parameters.
- Enhanced backtester to utilize dynamic ATR bands and synth VIX z-score for entry signals.
- Modified engine to include new features: `atr_band_position` and adjusted feature importance.
- Implemented dynamic ATR band calculations in the processor module.
- Adjusted signal engine logic to incorporate new thresholds for trend and mean-reversion entries based on ATR bands and z-scores.
- Updated optimizer parameters for improved model performance.
- Removed redundant `gmm_vol_cluster` feature from XGBoost training to improve efficiency.
- Adjusted feature selection for bucket models in `train_xgb_ensemble`.
- Updated probability thresholds for trading signals to enhance decision-making.
- Standardized exit confirmation bars for different timeframes in the signal engine.
- Removed TCN staleness checks and auto-retraining from guardian.py.
- Eliminated TCNMaintenanceScheduler and related functions from tcn_maintenance.py.
- Updated mt5_trader.py to remove TCN maintenance checks.
- Adjusted signal_engine.py to remove TCN multiplier logic.
- Cleaned up processor.py by removing optional LSTM context features.
- Modified validation warnings to suggest appropriate periods based on trade counts.
- Updated visualizer.py to streamline signal computation and improve clarity in equity curve plotting.
- Added `--stage` argument to `cmd_optimize` for selecting between Stage-1 (XGB exploration) and Stage-2 (CPCV optimization).
- Introduced `run_optimization_stage1` for fast single hold-out XGB trials, significantly improving optimization speed.
- Updated `run_optimization` to support warm-starting from Stage-1 best parameters.
- Enhanced logging with dedicated score logging for trial monitoring.
- Adjusted hyperparameter ranges for XGBoost to allow deeper trees and finer splits.
- Added synthetic VIX feature computation to the feature application process.
- Persisted best parameters from Stage-1 to JSON for future use in Stage-2.
- Updated documentation for new optimization stages and usage instructions.
- Added a cooldown period to prevent immediate re-entries after a trade, reducing churn in rapidly oscillating regimes.
- Simplified exit conditions by removing ATR trailing, partial close, and profit guard during optimization.
- Adjusted minimum confirmation bars for H1 to enhance stability before entries and exits.
- Lowered the floor for reg_lambda in optimizer to allow for better HMM state signals.
- Implemented a regime oscillation guard in the CPCV execution to skip paths with excessive transitions, mitigating potential drawdowns.
- Bump optimizer version from 3.6 to 3.7.
- Refactor make_objective function to use Combinatorial Purged Cross-Validation (CPCV) instead of WFO.
- Implement execute_cpcv function for CPCV analysis with strict HMM and scaler fitting.
- Update run_wfa to utilize CPCV and maintain backward compatibility with WFO key aliases.
- Remove deprecated WFO-related code and metrics, replacing them with CPCV equivalents.
- Enhance logging for CPCV trials and results.
- Removed the early stopping guard fold in the XGBoost training process to enhance model generalization.
- Updated optimizer version to 3.6 and adjusted search space for regularization parameters (reg_alpha and reg_lambda) to prevent overfitting.
- Implemented a purged inner cross-validation split to avoid leakage between training and validation sets.
- Adjusted scoring penalties for WFE to better reflect model performance discrepancies.
- Introduced adaptive min_child_weight cap in `engine_xgb.py` to prevent zero-split models for small training sets.
- Implemented early stopping with a guard fold in `train_xgb` function for better model validation.
- Updated optimizer version to 3.5 and expanded search space parameters in `optimizer.py`, including adjustments to `reg_lambda`, `min_child_weight`, and added `subsample`, `colsample_bytree`, and `gamma` parameters.
- Improved WFO score calculation to only consider valid windows, addressing overfitting detection.
- Bump optimizer version from 3.1 to 3.3.
- Adjusted H1 step size to 4320 bars for better independent test periods.
- Introduced MAX_WFO_WINDOWS to limit the number of evaluated windows per trial.
- Refactored window selection logic to pre-compute valid start positions and skip oldest windows if exceeding the maximum.
- Modified scoring parameters: reduced reg_lambda max from 10.0 to 5.0 and min_child_weight from 100 to 40.
- Enhanced WFE calculation to avoid misinterpretation of consistently negative models.
- Updated the state remapping logic in `engine_hmm.py` to ensure consistent labeling of states based on volatility and returns, addressing issues with out-of-sample regime inconsistency.
- Modified `train_xgb` in `engine_xgb.py` to handle cases where the dataset may not have a holdout set, ensuring robust model training and evaluation.
- Introduced annualization in the scoring function within `optimizer.py` to improve the Calmar ratio calculation, making it more comparable across different timeframes.
- Added penalties for high cross-window Sharpe variance and negative WFE ratios in the objective function to discourage overfitting and promote model consistency.
- Enhanced logging to provide clearer insights during the WFO process, including warnings for skipped windows and potential issues with model training.
- Resolve 5 committed git merge conflict markers in GoldRegimeX_Explorer.ipynb
that rendered it invalid JSON and unexecutable in Jupyter
- Conflict resolution: keep wfo_mode_w dropdown, reg_lambda5_w / min_child_weight5_w
sliders, Run WFO Score Comparison button, and Section 6 CV Path Inspector from
copilot branch; keep ensemble API imports (load_xgb_ensemble, get_predictions_ensemble)
from HEAD
- Fix notebook Sections 1 and 5 to use load_xgb_ensemble / train_xgb_ensemble /
get_predictions_ensemble instead of old single-model API; ensures equity curves
and WFO scores match production vol-bucketed ensemble
- Update README changelog with merge conflict resolution and ensemble API fix entry
- Updated MIN_OOS_TRADES_HARD for M15 from 30 to 60.
- Introduced WFO_PARAMS_FAST for quicker intraday regime detection.
- Adjusted WFO_TRIALS and added WFO_TRIALS_FAST to accommodate wider search space.
- Revised scoring formula in compute_cpcv_score to improve accuracy and range.
- Enhanced _run_wfo to support different WFO modes and adjusted CV folds based on TF.
- Aligned feature preparation and state alignment for consistency in backtesting.
- Updated make_objective to include new parameters and improved tree depth and learning rate settings.
- Changed pruner strategy based on TF in run_optimization for better trial management.
Updated the out-of-sample (OOS) backtesting process to ensure that the states used are aligned with the OOS data frame index. This change addresses potential mismatches by applying a mask to the states based on the aligned index, improving the accuracy of the backtest results.
- Added WFO parameters and trial configurations for H1, M15, and M5 timeframes.
- Introduced _run_wfo function to evaluate hyperparameters across non-overlapping OOS windows.
- Updated make_objective to utilize WFO instead of CPCV for scoring.
- Enhanced asset feature processing in processor.py to include additional external assets (XAGUSD, XTIUSD, US500, USDJPY) and their log-return calculations.
- Implemented a generic load_asset_data function to streamline loading and processing of asset data.
- Added compute_synth_vix function to calculate a synthetic VIX proxy from OHLC data.
- Improved handling of missing data and logging for asset features.
- Introduced CPCV for train/test splitting, allowing for more robust evaluation across multiple market regimes.
- Updated backtester to support test masks, ensuring trades are only executed during test periods.
- Refactored optimizer to compute scores using CPCV, improving trial efficiency and accuracy.
- Adjusted signal engine logic to incorporate directional probabilities for trade entries based on XGBoost predictions.
- Improved overall code structure and readability, ensuring better maintainability and performance.
- Deleted `signal_evaluator.py`, which contained the previous Z-Score and RCEV evaluation logic.
- Added `signal_engine.py`, implementing a new stateful signal engine for regime confirmation and trade management.
- The new engine evaluates entry and exit conditions based on regime stability, XGBoost confidence, and Bollinger Band positions.
- Introduced constants for minimum confirmation bars, entry probabilities, maximum hold bars, and profit erosion thresholds.
- Introduced RCEVScorer class for calculating expected dollar profit based on historical trade outcomes and market conditions.
- Replaced Z-Score signal gating with RCEV scoring in SignalEvaluator for live evaluations.
- Added new configuration for RCEV thresholds and tiered floors based on timeframes.
- Enhanced MT5 trader logic to utilize RCEV scoring for trade decision-making, including regime-aware take-profit and stop-loss calculations.
- Updated ATR trailing stop logic to incorporate new RCEV parameters and configurations.
- Maintained backward compatibility with existing Z-Score logic for scenarios without RCEV calibration.
- Created a new StandardScaler object for feature scaling, saved as tcn_feature_scaler.pkl.
- Added metadata in JSON format for the M15_headway_cent model, including sequence length, number of features, number of states, temperature, and training timestamp.
- Deleted ensemble metadata, feature scalers, and model metadata for H1 headway cent models.
- Removed LSTM regime classifier files for H1 headway cent models.
- Deleted corresponding files for model_0, model_1, and model_2 in both H1 and M15 headway cent directories.
- Cleaned up unnecessary files to streamline the model directory structure.
- Introduced `engine_tcn.py` for Temporal Convolutional Network (TCN) model to score confidence based on recent market data.
- Added `data_updater.py` for automatic weekly data updates from MT5, ensuring fresh data for TCN.
- Created `tcn_maintenance.py` for automatic TCN model maintenance, including staleness checks and background retraining.
- Updated `mt5_trader.py` to integrate TCN confidence scoring, replacing the previous LSTM ensemble logic.
- Enhanced logging and error handling throughout the new modules for better traceability.
- Full workflow from raw data to live trading (Phase 0–7)
- LSTM ensemble, temperature calibration, fine-tuning, staleness/auto-retrain
- Sensitivity analysis and tiered Z-Score trading modes
- Updated command reference (all 16 modes + 22 CLI options)
- Signal logic with exact TF Z-Score cutoffs
- Guardian LSTM auto-retrain documentation
- Model files layout and troubleshooting guide
- Removed LSTM context features from feature column selection in `get_feature_cols`.
- Updated `compute_live_features` to eliminate LSTM context model dependency and return the full live DataFrame.
- Enhanced `_run_loop_inner` to load LSTM regime classifier and adjust signal evaluation based on LSTM predictions.
- Introduced Z-Score cutoff parameters in `make_objective` for better signal selection.
- Added `load_data_with_hmm_labels` function to attach HMM state labels to processed data.
- Updated validation process to include Z-Score cutoff configuration.
Signal pipeline audit + diagnostic instrumentation:
signal_evaluator.py:
- calculate_z_score: adds logger.debug per-bar Z trace + logger.warning on
fallback stats (helps confirm per-regime stats are being used in live logs)
- evaluate_signal_fast: refactored to collect sig before returning; adds
three ERROR-level audit assertions (BUY in non-Bull, SELL in non-Bear,
MR in non-Chop) — non-crashing, safe in live trading
- evaluate_signal: same three assertions before final return
tests/audit_signal_pipeline.py (new):
- 13 deterministic unit tests covering:
T01-T03: Bull only fires BUY, never SELL
T04-T06: Bear only fires SELL, never BUY
T07-T09: Chop only fires MR_BUY/MR_SELL, never trend signals
T10: per-state Z-Scores are distinct (regime_stats used correctly)
T11: tiered promotes a sub-threshold trade into BUY
T12: tiered floor of 1.0 respected regardless of config cutoff
T13: backtester collapses MR_BUY→+1, MR_SELL→-1 (unified encoding)
All 13 pass: exit 0
Result: 13/13 passed — pipeline is correct
New opt-in use_tiered mode reduces the Bull/Bear Z-Score requirement when
XGBoost outputs a high-conviction probability, down to a floor of 1.0.
MR (Chop) cutoffs are never affected. The feature is additive — all paths
with use_tiered=False (default) behave identically to before.
Changes:
- signal_evaluator.py: _calculate_tiered_override() returns 0/0.25/0.5/1.0
based on |prob - 0.50| extremity; evaluate_signal_fast() applies reduction
to bull_cut/bear_cut when use_tiered=True; evaluate_signal() applies the
same reduction to req['min_z_score'] and clears Z-based blocks post-reduction
(bar-count blocks are never overridden)
- backtester.py: use_tiered threaded through vectorized_backtest ->
compute_signals_zscore -> evaluate_signal_fast
- sensitivity.py: use_tiered param on run_sensitivity; tables/files suffixed
with _tiered for side-by-side comparison; mode label shown in header
- mt5_trader.py: use_tiered threaded through run_live_loop ->
_run_loop_inner -> evaluate_signal
- main.py: --tiered CLI flag added; threaded to cmd_sensitivity, cmd_live,
cmd_demo
Usage:
python main.py --mode sensitivity --tf M15 --broker headway_cent --balance 15 --tiered
python main.py --mode live --tf M15 --broker headway_cent --balance 15 --tiered
Sensitivity analysis showed Z=2.0 produces 64.2% total return vs 51.4% at
Z=2.25, with 9.3% max drawdown and Sharpe 1.693. H1 (2.5) and M5 (2.8)
are unchanged.
New --mode sensitivity command tests Z cutoffs 1.5..3.0 (step 0.25) on the
saved HMM + XGBoost ensemble without retraining. Loads data and models the
same way cmd_report does, then calls vectorized_backtest once per cutoff with
an evaluator_config override injected into SignalEvaluator.
Changes:
- src/sensitivity.py: run_sensitivity() loops over z_range, builds evaluator_config
override, calls vectorized_backtest, prints comparison table, saves CSV+JSON
- src/backtester.py: evaluator_config param added to compute_signals_zscore and
vectorized_backtest; threaded through to SignalEvaluator(config=)
- main.py: cmd_sensitivity loads models identical to cmd_report; MR cutoffs held
constant; sensitivity mode registered in choices + dispatch dict
Usage: python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15
All src.* loggers redirect to logs/goldregimex_{TF}.log on each TF-specific
command (optimize, train, compare, wfa, sync_validate, report, live).
main + notifier loggers keep writing to goldregimex.log for summaries and
Telegram. Switching TFs in a multi-TF loop (e.g. optimize M5+H1) cleanly
swaps handlers. M15 log deferred until current M15 study completes.
engine_xgb: std floor 0.02 → 0.003 in compute_regime_stats()
The 0.02 floor was dominating all states (all showing std=0.0200 in logs)
because XGB probability variance for weak-signal H1 is ~0.003-0.006.
Flooring at 0.02 inflated Z-Score thresholds by ~0.02 in prob space,
reducing H1 OOS trades to ~16 across 3 years.
optimizer: TF_MIN_OOS_TRADES H1: 60 → 15, M15: 150 → 100, M5: 400 → 300
With Z-Score filtering being more selective than fixed thresholds,
trade counts are lower. The old 60-trade penalty threshold caused a 0.1x
suppression on every H1 trial (true score ~3.24 was reported as 0.3235),
not breaking optimization but masking real quality in the output banner.
15 trades is appropriate for Z-Score filtered H1 OOS evaluation.
- Added `SignalEvaluator` class for shared adaptive signal evaluation logic across backtesting, optimization, and live trading.
- Introduced `compute_regime_stats` function to calculate per-HMM-state probability statistics for Z-Score calibration.
- Enhanced `vectorized_backtest` and `run_walk_forward` functions to support regime statistics and Z-Score signal generation.
- Updated `backtester.py` to include new signal computation method `compute_signals_zscore`.
- Modified `mt5_trader.py` to utilize the new signal evaluation logic and added regime stability tracking.
- Adjusted `optimizer.py` to compute regime statistics during model training and store them for later use.
- Updated validation logic in `validator.py` to incorporate regime statistics for improved signal evaluation.
- Refactored existing signal routing logic to accommodate new adaptive Z-Score evaluation framework.
Activity Bonus (optimizer.py):
- M5 OOS n_trades < 150: score *= 0.5 (extra penalty on top of progressive 0.1)
- M5 OOS n_trades > 300: score *= 1.2 (bonus rewards high-freq robustness)
- Forces Optuna to find the M5 'heartbeat' (150-400 OOS trades) rather than
low-activity solutions that hide drawdown by barely trading
Consecutive Loss Guard (risk_manager.py):
- AdaptiveRiskManager.MAX_STREAK = 3, LOCKOUT_HOURS = 4.0
- record_trade_result(won) increments streak on loss, resets on win
- can_trade() returns False during lockout window; auto-resets on expiry
- Prevents Death by a Thousand Cuts on M5 MR trades during trending breakouts
- __init__ accepts consecutive_losses + streak_locked_until for state handoff
between per-bar ARM re-instantiations in the live loop
M5 Efficiency Threshold (backtester.py):
- TF_MIN_EFFICIENCY dict: M5=7.0x, all other TFs=1.25x (default)
- vectorized_backtest applies tf-specific ER threshold via TF_MIN_EFFICIENCY.get(tf, 1.25)
- Blocks M5 signals where ATR < 7x spread (~$2.80 ATR on $2000 gold)
filtering out quiet night-session noise where MR edge is lowest
Max Monetary Risk label (visualizer.py):
- OOS section now shows "Max DD" % + dedicated "Max Monetary Risk: $X.XX USD" row
- Calculated as oos_fdd * account_size for clear monetary context on $15 accounts
TF_MAGIC_MAP + dynamic magic:
- Replace static MAGIC_NUMBER=123456 with TF_MAGIC_MAP {H1:123456, M15:123457, M5:123458}
- ALL_GRX_MAGICS frozenset for cross-TF queries
- MAGIC_NUMBER kept as H1 alias (MQL5 EA backwards-compat)
- _run_loop_inner resolves magic=TF_MAGIC_MAP[tf] at startup
- All loop-body references (position queries, close orders, has_open_position,
order sends, _execute_partial_close, _close_position) now use dynamic magic
- _execute_partial_close and _close_position gain magic= param (default=MAGIC_NUMBER)
TF-tagged order comments:
- New format: GRX_{tf}_{direction}_s{hmm_state}_tp{p+1}
e.g. GRX_H1_BUY_s0_tp1, GRX_M5_SELL_s1_tp2
- Visible in MT5 Trade tab so each timeframe's trades are instantly identifiable
Global multi-TF exposure guard (step 8):
- Counts open positions across ALL_GRX_MAGICS before any new signal
- If >= 4 total positions: logs [GLOBAL GUARD] and skips the signal
- Prevents over-leverage when two or more TFs fire on the same $15 balance
Daily P&L audit report:
- send_daily_audit_report(mt5, broker): queries history_deals_get for prior 24h,
groups DEAL_ENTRY_OUT fills by TF magic, formats per-TF and total P&L
- Called automatically at UTC midnight reset (try/except so crashes don't kill loop)
- Telegram output: "H1: +$0.45 (1 trade)\nM15: +$1.20 (4 trades)\n..."
_log_closed_pnl:
- Added tf= parameter; Telegram message now shows "[H1] Trade closed" tag
- DEAL_ENTRY_OUT wait was already implemented; added tf tag to surface TF in notification
Rewrite setup_logger to use io.TextIOWrapper(sys.stderr.buffer, encoding=utf-8)
instead of reconfigure(). reconfigure() silently fails under PowerShell on
Windows (the console stream has no .buffer equiv in some configurations),
leaving the handler on cp1252 which crashes on any Unicode > U+00FF.
New approach:
- Wraps sys.stderr.buffer directly in a fresh TextIOWrapper(utf-8, errors=replace)
- Falls back to sys.stderr as-is if .buffer is absent (IDLE / test runners)
- FileHandler explicitly opens goldregimex.log with encoding=utf-8
- Eliminates all future cp1252 crashes from emoji, box-drawing, or math symbols