- Implemented `generate_bridge_csv.py` to generate reports for strategy winners from the Strategy Tester notebook.
- Added `smoke_test_explorer.py` to perform smoke tests on the Explorer notebook integration, validating core logic and data processing.
- Updated signal engine logic to improve trade decision-making with new probability thresholds and dynamic gates.
- Introduced new `cpcv_reporting.py` module for generating comprehensive CPCV reports, including optimization summaries and performance metrics.
- Added `optuna_param_guard.py` to enforce parameter validation for Optuna optimization, ensuring only approved parameters are used.
- Created `regime_diagnostics.py` for summarizing regime statistics and trade distributions, facilitating better analysis of trading strategies.
- Introduced `test_lifecycle_parity.py` to verify event timelines between live and backtest evaluations, ensuring consistent activation, partial close, scalp exit, and regime shift behaviors across different timeframes.
- Added `test_lifecycle_units.py` for unit testing of lifecycle configurations, including cent normalization, activation thresholds, and the absence of R-multiple fields.
- Created `test_three_state_contract.py` to enforce and validate the three-state regime (TREND, MEAN_REVERSION, VOLATILITY_SHOCK) across various components, ensuring that any attempt to use more than three states raises appropriate errors.
- Integrated regime model predictions into the validation process, allowing for fallback to ensemble models.
- Updated validation criteria to include checks for maximum drawdown, minimum trades, and MR leaks.
- Enhanced test suite to cover new validation scenarios and ensure robustness of strategy evaluation.
- Adjusted signal pipeline audit tests to reflect changes in regime handling and validation logic.
- Implemented `add_specialized_features` to compute ATR moving averages, expansion ratios, realized volatility ratios, and session proximity features.
- Integrated the new feature set into the processing pipeline.
- Updated `run_sensitivity` to utilize new Z-score thresholds from `SignalEngine`.
- Refactored `SignalEngine` to include a unified `SignalDecision` class for consistent entry/exit logic.
- Enhanced `evaluate_bar` to incorporate dynamic probability thresholds and M15 entry gates.
- Added comprehensive tests for signal generation, decision parity, and dynamic thresholds.
- Updated `get_feature_cols` to handle one-hot encoding for HMM states in M5/M15.
- Modified `prepare_features` to include a rolling median for HMM states and drop the original integer state.
- Enhanced `compute_live_features` to generate one-hot encoded features for M5/M15 and retain the integer state for H1.
- Implemented runtime checks to ensure live feature keys match training columns, raising errors for mismatches.
- Adjusted scoring logic in `optimizer.py` to apply log-loss for M5/M15 and backtest for H1, improving robustness against over-regularization.
- Updated search space for `n_states` in models M15 and M5 from categorical to integer range (3 to 4).
- Lowered `reg_alpha` and `reg_lambda` ranges to allow HMM OHE signal through for models M15 and M5.
- Adjusted the `make_objective` function to reflect changes in parameter suggestions for M5 and M15.
- Introduced new model configuration file for stage1 M15 with optimized parameters and score.
- Added binary changes to the study_headway_cent database.
- Implemented auto-sync for raw CSV data from MT5 in cmd_optimize, ensuring missing bars are appended if the terminal is running.
- Added ensure_data_updated function in mt5_sync.py to check and fetch missing bars for specified timeframes, logging warnings if MT5 is unavailable.
- Introduced median filtering in engine_hmm.py to smooth HMM state predictions, reducing noise from regime flips.
- Updated predict_states to apply median filtering after state remapping for consistency.
- Added time-of-day gating in processor.py to filter out Asian-session bars, retaining only London/NY high-liquidity hours for analysis.
- Removed obsolete model configuration file `stage1_m15_headway_cent.json`.
- Updated database file `study_headway_cent.db` to reflect changes in model parameters.
- Enhanced backtester to utilize dynamic ATR bands and synth VIX z-score for entry signals.
- Modified engine to include new features: `atr_band_position` and adjusted feature importance.
- Implemented dynamic ATR band calculations in the processor module.
- Adjusted signal engine logic to incorporate new thresholds for trend and mean-reversion entries based on ATR bands and z-scores.
- Updated optimizer parameters for improved model performance.
- Removed redundant `gmm_vol_cluster` feature from XGBoost training to improve efficiency.
- Adjusted feature selection for bucket models in `train_xgb_ensemble`.
- Updated probability thresholds for trading signals to enhance decision-making.
- Standardized exit confirmation bars for different timeframes in the signal engine.
- Removed TCN staleness checks and auto-retraining from guardian.py.
- Eliminated TCNMaintenanceScheduler and related functions from tcn_maintenance.py.
- Updated mt5_trader.py to remove TCN maintenance checks.
- Adjusted signal_engine.py to remove TCN multiplier logic.
- Cleaned up processor.py by removing optional LSTM context features.
- Modified validation warnings to suggest appropriate periods based on trade counts.
- Updated visualizer.py to streamline signal computation and improve clarity in equity curve plotting.
- Added `--stage` argument to `cmd_optimize` for selecting between Stage-1 (XGB exploration) and Stage-2 (CPCV optimization).
- Introduced `run_optimization_stage1` for fast single hold-out XGB trials, significantly improving optimization speed.
- Updated `run_optimization` to support warm-starting from Stage-1 best parameters.
- Enhanced logging with dedicated score logging for trial monitoring.
- Adjusted hyperparameter ranges for XGBoost to allow deeper trees and finer splits.
- Added synthetic VIX feature computation to the feature application process.
- Persisted best parameters from Stage-1 to JSON for future use in Stage-2.
- Updated documentation for new optimization stages and usage instructions.
- Added a cooldown period to prevent immediate re-entries after a trade, reducing churn in rapidly oscillating regimes.
- Simplified exit conditions by removing ATR trailing, partial close, and profit guard during optimization.
- Adjusted minimum confirmation bars for H1 to enhance stability before entries and exits.
- Lowered the floor for reg_lambda in optimizer to allow for better HMM state signals.
- Implemented a regime oscillation guard in the CPCV execution to skip paths with excessive transitions, mitigating potential drawdowns.
- Bump optimizer version from 3.6 to 3.7.
- Refactor make_objective function to use Combinatorial Purged Cross-Validation (CPCV) instead of WFO.
- Implement execute_cpcv function for CPCV analysis with strict HMM and scaler fitting.
- Update run_wfa to utilize CPCV and maintain backward compatibility with WFO key aliases.
- Remove deprecated WFO-related code and metrics, replacing them with CPCV equivalents.
- Enhance logging for CPCV trials and results.
- Removed the early stopping guard fold in the XGBoost training process to enhance model generalization.
- Updated optimizer version to 3.6 and adjusted search space for regularization parameters (reg_alpha and reg_lambda) to prevent overfitting.
- Implemented a purged inner cross-validation split to avoid leakage between training and validation sets.
- Adjusted scoring penalties for WFE to better reflect model performance discrepancies.
- Introduced adaptive min_child_weight cap in `engine_xgb.py` to prevent zero-split models for small training sets.
- Implemented early stopping with a guard fold in `train_xgb` function for better model validation.
- Updated optimizer version to 3.5 and expanded search space parameters in `optimizer.py`, including adjustments to `reg_lambda`, `min_child_weight`, and added `subsample`, `colsample_bytree`, and `gamma` parameters.
- Improved WFO score calculation to only consider valid windows, addressing overfitting detection.
- Bump optimizer version from 3.1 to 3.3.
- Adjusted H1 step size to 4320 bars for better independent test periods.
- Introduced MAX_WFO_WINDOWS to limit the number of evaluated windows per trial.
- Refactored window selection logic to pre-compute valid start positions and skip oldest windows if exceeding the maximum.
- Modified scoring parameters: reduced reg_lambda max from 10.0 to 5.0 and min_child_weight from 100 to 40.
- Enhanced WFE calculation to avoid misinterpretation of consistently negative models.
- Updated the state remapping logic in `engine_hmm.py` to ensure consistent labeling of states based on volatility and returns, addressing issues with out-of-sample regime inconsistency.
- Modified `train_xgb` in `engine_xgb.py` to handle cases where the dataset may not have a holdout set, ensuring robust model training and evaluation.
- Introduced annualization in the scoring function within `optimizer.py` to improve the Calmar ratio calculation, making it more comparable across different timeframes.
- Added penalties for high cross-window Sharpe variance and negative WFE ratios in the objective function to discourage overfitting and promote model consistency.
- Enhanced logging to provide clearer insights during the WFO process, including warnings for skipped windows and potential issues with model training.
- Resolve 5 committed git merge conflict markers in GoldRegimeX_Explorer.ipynb
that rendered it invalid JSON and unexecutable in Jupyter
- Conflict resolution: keep wfo_mode_w dropdown, reg_lambda5_w / min_child_weight5_w
sliders, Run WFO Score Comparison button, and Section 6 CV Path Inspector from
copilot branch; keep ensemble API imports (load_xgb_ensemble, get_predictions_ensemble)
from HEAD
- Fix notebook Sections 1 and 5 to use load_xgb_ensemble / train_xgb_ensemble /
get_predictions_ensemble instead of old single-model API; ensures equity curves
and WFO scores match production vol-bucketed ensemble
- Update README changelog with merge conflict resolution and ensemble API fix entry
- Updated MIN_OOS_TRADES_HARD for M15 from 30 to 60.
- Introduced WFO_PARAMS_FAST for quicker intraday regime detection.
- Adjusted WFO_TRIALS and added WFO_TRIALS_FAST to accommodate wider search space.
- Revised scoring formula in compute_cpcv_score to improve accuracy and range.
- Enhanced _run_wfo to support different WFO modes and adjusted CV folds based on TF.
- Aligned feature preparation and state alignment for consistency in backtesting.
- Updated make_objective to include new parameters and improved tree depth and learning rate settings.
- Changed pruner strategy based on TF in run_optimization for better trial management.
Updated the out-of-sample (OOS) backtesting process to ensure that the states used are aligned with the OOS data frame index. This change addresses potential mismatches by applying a mask to the states based on the aligned index, improving the accuracy of the backtest results.
- Added WFO parameters and trial configurations for H1, M15, and M5 timeframes.
- Introduced _run_wfo function to evaluate hyperparameters across non-overlapping OOS windows.
- Updated make_objective to utilize WFO instead of CPCV for scoring.
- Enhanced asset feature processing in processor.py to include additional external assets (XAGUSD, XTIUSD, US500, USDJPY) and their log-return calculations.
- Implemented a generic load_asset_data function to streamline loading and processing of asset data.
- Added compute_synth_vix function to calculate a synthetic VIX proxy from OHLC data.
- Improved handling of missing data and logging for asset features.
Luke Mosoti