This commit introduces a new smoke test script that exercises every phase of the pipeline observability process. The test generates synthetic data for two timeframes (M15 and M5), simulating various stages of the pipeline and verifying that all artifacts are produced and non-empty. The script includes assertions to ensure the integrity of the generated outputs and checks for expected conditions in the decision log, survival matrix, and audit JSON.
- Updated GoldRegime_X.code-workspace to remove deprecated server configuration.
- Added package.json and package-lock.json to manage dependencies, specifically @notionhq/client.
- Introduced patch_explorer_cell46.py to resolve ModuleNotFoundError in the PIPELINE VERIFICATION import cell of GoldRegimeX_Explorer.ipynb.
- Created patch_explorer_cell48.py to fix NameError for TIMEFRAMES in the Dataset Integrity verification cell, ensuring it can derive TIMEFRAMES from the pipeline if not already defined.
- Introduced `pipeline_verification.py` containing multiple verifier classes for dataset integrity, feature leakage, evaluation integrity, prediction alignment, and more.
- Added `SessionFilter` class in `session_filter.py` for unified session/time-of-day filtering.
- Implemented `VerificationResult` data class for structured verification results.
- Created utility functions for hashing and manifest building.
- Established a test suite for the session filter to ensure correct functionality.
- Updated `__init__.py` to include the new shared module.
- Deleted `test_lifecycle_parity.py`, which contained tests for lifecycle event timelines in live and backtest scenarios.
- Deleted `test_lifecycle_units.py`, which included unit tests for lifecycle configurations and floating PnL calculations.
- Deleted `test_three_state_contract.py`, which verified the enforcement of three semantic states across various components.
- Updated parallel processing output times to reflect new execution durations.
- Modified strategy results table with new entries and adjusted values for fixed_tp and fixed_tp_plus_mr strategies.
- Increased maximum drawdown percentage from 30.0 to 35.0.
- Enhanced filtering logic to ensure proper retrieval of results from all_results or valid_results.
- Implemented `generate_bridge_csv.py` to generate reports for strategy winners from the Strategy Tester notebook.
- Added `smoke_test_explorer.py` to perform smoke tests on the Explorer notebook integration, validating core logic and data processing.
- Updated signal engine logic to improve trade decision-making with new probability thresholds and dynamic gates.
- Introduced new `cpcv_reporting.py` module for generating comprehensive CPCV reports, including optimization summaries and performance metrics.
- Added `optuna_param_guard.py` to enforce parameter validation for Optuna optimization, ensuring only approved parameters are used.
- Created `regime_diagnostics.py` for summarizing regime statistics and trade distributions, facilitating better analysis of trading strategies.
- Introduced `test_lifecycle_parity.py` to verify event timelines between live and backtest evaluations, ensuring consistent activation, partial close, scalp exit, and regime shift behaviors across different timeframes.
- Added `test_lifecycle_units.py` for unit testing of lifecycle configurations, including cent normalization, activation thresholds, and the absence of R-multiple fields.
- Created `test_three_state_contract.py` to enforce and validate the three-state regime (TREND, MEAN_REVERSION, VOLATILITY_SHOCK) across various components, ensuring that any attempt to use more than three states raises appropriate errors.
- Integrated regime model predictions into the validation process, allowing for fallback to ensemble models.
- Updated validation criteria to include checks for maximum drawdown, minimum trades, and MR leaks.
- Enhanced test suite to cover new validation scenarios and ensure robustness of strategy evaluation.
- Adjusted signal pipeline audit tests to reflect changes in regime handling and validation logic.
- Implemented `add_specialized_features` to compute ATR moving averages, expansion ratios, realized volatility ratios, and session proximity features.
- Integrated the new feature set into the processing pipeline.
- Updated `run_sensitivity` to utilize new Z-score thresholds from `SignalEngine`.
- Refactored `SignalEngine` to include a unified `SignalDecision` class for consistent entry/exit logic.
- Enhanced `evaluate_bar` to incorporate dynamic probability thresholds and M15 entry gates.
- Added comprehensive tests for signal generation, decision parity, and dynamic thresholds.
- Updated `get_feature_cols` to handle one-hot encoding for HMM states in M5/M15.
- Modified `prepare_features` to include a rolling median for HMM states and drop the original integer state.
- Enhanced `compute_live_features` to generate one-hot encoded features for M5/M15 and retain the integer state for H1.
- Implemented runtime checks to ensure live feature keys match training columns, raising errors for mismatches.
- Adjusted scoring logic in `optimizer.py` to apply log-loss for M5/M15 and backtest for H1, improving robustness against over-regularization.
- Updated search space for `n_states` in models M15 and M5 from categorical to integer range (3 to 4).
- Lowered `reg_alpha` and `reg_lambda` ranges to allow HMM OHE signal through for models M15 and M5.
- Adjusted the `make_objective` function to reflect changes in parameter suggestions for M5 and M15.
- Introduced new model configuration file for stage1 M15 with optimized parameters and score.
- Added binary changes to the study_headway_cent database.
- Implemented auto-sync for raw CSV data from MT5 in cmd_optimize, ensuring missing bars are appended if the terminal is running.
- Added ensure_data_updated function in mt5_sync.py to check and fetch missing bars for specified timeframes, logging warnings if MT5 is unavailable.
- Introduced median filtering in engine_hmm.py to smooth HMM state predictions, reducing noise from regime flips.
- Updated predict_states to apply median filtering after state remapping for consistency.
- Added time-of-day gating in processor.py to filter out Asian-session bars, retaining only London/NY high-liquidity hours for analysis.
- Removed obsolete model configuration file `stage1_m15_headway_cent.json`.
- Updated database file `study_headway_cent.db` to reflect changes in model parameters.
- Enhanced backtester to utilize dynamic ATR bands and synth VIX z-score for entry signals.
- Modified engine to include new features: `atr_band_position` and adjusted feature importance.
- Implemented dynamic ATR band calculations in the processor module.
- Adjusted signal engine logic to incorporate new thresholds for trend and mean-reversion entries based on ATR bands and z-scores.
- Updated optimizer parameters for improved model performance.
- Removed redundant `gmm_vol_cluster` feature from XGBoost training to improve efficiency.
- Adjusted feature selection for bucket models in `train_xgb_ensemble`.
- Updated probability thresholds for trading signals to enhance decision-making.
- Standardized exit confirmation bars for different timeframes in the signal engine.
- Removed TCN staleness checks and auto-retraining from guardian.py.
- Eliminated TCNMaintenanceScheduler and related functions from tcn_maintenance.py.
- Updated mt5_trader.py to remove TCN maintenance checks.
- Adjusted signal_engine.py to remove TCN multiplier logic.
- Cleaned up processor.py by removing optional LSTM context features.
- Modified validation warnings to suggest appropriate periods based on trade counts.
- Updated visualizer.py to streamline signal computation and improve clarity in equity curve plotting.
- Added `--stage` argument to `cmd_optimize` for selecting between Stage-1 (XGB exploration) and Stage-2 (CPCV optimization).
- Introduced `run_optimization_stage1` for fast single hold-out XGB trials, significantly improving optimization speed.
- Updated `run_optimization` to support warm-starting from Stage-1 best parameters.
- Enhanced logging with dedicated score logging for trial monitoring.
- Adjusted hyperparameter ranges for XGBoost to allow deeper trees and finer splits.
- Added synthetic VIX feature computation to the feature application process.
- Persisted best parameters from Stage-1 to JSON for future use in Stage-2.
- Updated documentation for new optimization stages and usage instructions.
- Added a cooldown period to prevent immediate re-entries after a trade, reducing churn in rapidly oscillating regimes.
- Simplified exit conditions by removing ATR trailing, partial close, and profit guard during optimization.
- Adjusted minimum confirmation bars for H1 to enhance stability before entries and exits.
- Lowered the floor for reg_lambda in optimizer to allow for better HMM state signals.
- Implemented a regime oscillation guard in the CPCV execution to skip paths with excessive transitions, mitigating potential drawdowns.
- Bump optimizer version from 3.6 to 3.7.
- Refactor make_objective function to use Combinatorial Purged Cross-Validation (CPCV) instead of WFO.
- Implement execute_cpcv function for CPCV analysis with strict HMM and scaler fitting.
- Update run_wfa to utilize CPCV and maintain backward compatibility with WFO key aliases.
- Remove deprecated WFO-related code and metrics, replacing them with CPCV equivalents.
- Enhance logging for CPCV trials and results.
- Removed the early stopping guard fold in the XGBoost training process to enhance model generalization.
- Updated optimizer version to 3.6 and adjusted search space for regularization parameters (reg_alpha and reg_lambda) to prevent overfitting.
- Implemented a purged inner cross-validation split to avoid leakage between training and validation sets.
- Adjusted scoring penalties for WFE to better reflect model performance discrepancies.
- Introduced adaptive min_child_weight cap in `engine_xgb.py` to prevent zero-split models for small training sets.
- Implemented early stopping with a guard fold in `train_xgb` function for better model validation.
- Updated optimizer version to 3.5 and expanded search space parameters in `optimizer.py`, including adjustments to `reg_lambda`, `min_child_weight`, and added `subsample`, `colsample_bytree`, and `gamma` parameters.
- Improved WFO score calculation to only consider valid windows, addressing overfitting detection.
- Bump optimizer version from 3.1 to 3.3.
- Adjusted H1 step size to 4320 bars for better independent test periods.
- Introduced MAX_WFO_WINDOWS to limit the number of evaluated windows per trial.
- Refactored window selection logic to pre-compute valid start positions and skip oldest windows if exceeding the maximum.
- Modified scoring parameters: reduced reg_lambda max from 10.0 to 5.0 and min_child_weight from 100 to 40.
- Enhanced WFE calculation to avoid misinterpretation of consistently negative models.
- Updated the state remapping logic in `engine_hmm.py` to ensure consistent labeling of states based on volatility and returns, addressing issues with out-of-sample regime inconsistency.
- Modified `train_xgb` in `engine_xgb.py` to handle cases where the dataset may not have a holdout set, ensuring robust model training and evaluation.
- Introduced annualization in the scoring function within `optimizer.py` to improve the Calmar ratio calculation, making it more comparable across different timeframes.
- Added penalties for high cross-window Sharpe variance and negative WFE ratios in the objective function to discourage overfitting and promote model consistency.
- Enhanced logging to provide clearer insights during the WFO process, including warnings for skipped windows and potential issues with model training.
- Resolve 5 committed git merge conflict markers in GoldRegimeX_Explorer.ipynb
that rendered it invalid JSON and unexecutable in Jupyter
- Conflict resolution: keep wfo_mode_w dropdown, reg_lambda5_w / min_child_weight5_w
sliders, Run WFO Score Comparison button, and Section 6 CV Path Inspector from
copilot branch; keep ensemble API imports (load_xgb_ensemble, get_predictions_ensemble)
from HEAD
- Fix notebook Sections 1 and 5 to use load_xgb_ensemble / train_xgb_ensemble /
get_predictions_ensemble instead of old single-model API; ensures equity curves
and WFO scores match production vol-bucketed ensemble
- Update README changelog with merge conflict resolution and ensemble API fix entry
- Updated MIN_OOS_TRADES_HARD for M15 from 30 to 60.
- Introduced WFO_PARAMS_FAST for quicker intraday regime detection.
- Adjusted WFO_TRIALS and added WFO_TRIALS_FAST to accommodate wider search space.
- Revised scoring formula in compute_cpcv_score to improve accuracy and range.
- Enhanced _run_wfo to support different WFO modes and adjusted CV folds based on TF.
- Aligned feature preparation and state alignment for consistency in backtesting.
- Updated make_objective to include new parameters and improved tree depth and learning rate settings.
- Changed pruner strategy based on TF in run_optimization for better trial management.
Luke Mosoti