- Removed TCN staleness checks and auto-retraining from guardian.py.
- Eliminated TCNMaintenanceScheduler and related functions from tcn_maintenance.py.
- Updated mt5_trader.py to remove TCN maintenance checks.
- Adjusted signal_engine.py to remove TCN multiplier logic.
- Cleaned up processor.py by removing optional LSTM context features.
- Modified validation warnings to suggest appropriate periods based on trade counts.
- Updated visualizer.py to streamline signal computation and improve clarity in equity curve plotting.
- Added `--stage` argument to `cmd_optimize` for selecting between Stage-1 (XGB exploration) and Stage-2 (CPCV optimization).
- Introduced `run_optimization_stage1` for fast single hold-out XGB trials, significantly improving optimization speed.
- Updated `run_optimization` to support warm-starting from Stage-1 best parameters.
- Enhanced logging with dedicated score logging for trial monitoring.
- Adjusted hyperparameter ranges for XGBoost to allow deeper trees and finer splits.
- Added synthetic VIX feature computation to the feature application process.
- Persisted best parameters from Stage-1 to JSON for future use in Stage-2.
- Updated documentation for new optimization stages and usage instructions.
- Added a cooldown period to prevent immediate re-entries after a trade, reducing churn in rapidly oscillating regimes.
- Simplified exit conditions by removing ATR trailing, partial close, and profit guard during optimization.
- Adjusted minimum confirmation bars for H1 to enhance stability before entries and exits.
- Lowered the floor for reg_lambda in optimizer to allow for better HMM state signals.
- Implemented a regime oscillation guard in the CPCV execution to skip paths with excessive transitions, mitigating potential drawdowns.
- Bump optimizer version from 3.6 to 3.7.
- Refactor make_objective function to use Combinatorial Purged Cross-Validation (CPCV) instead of WFO.
- Implement execute_cpcv function for CPCV analysis with strict HMM and scaler fitting.
- Update run_wfa to utilize CPCV and maintain backward compatibility with WFO key aliases.
- Remove deprecated WFO-related code and metrics, replacing them with CPCV equivalents.
- Enhance logging for CPCV trials and results.
- Removed the early stopping guard fold in the XGBoost training process to enhance model generalization.
- Updated optimizer version to 3.6 and adjusted search space for regularization parameters (reg_alpha and reg_lambda) to prevent overfitting.
- Implemented a purged inner cross-validation split to avoid leakage between training and validation sets.
- Adjusted scoring penalties for WFE to better reflect model performance discrepancies.
- Introduced adaptive min_child_weight cap in `engine_xgb.py` to prevent zero-split models for small training sets.
- Implemented early stopping with a guard fold in `train_xgb` function for better model validation.
- Updated optimizer version to 3.5 and expanded search space parameters in `optimizer.py`, including adjustments to `reg_lambda`, `min_child_weight`, and added `subsample`, `colsample_bytree`, and `gamma` parameters.
- Improved WFO score calculation to only consider valid windows, addressing overfitting detection.
- Bump optimizer version from 3.1 to 3.3.
- Adjusted H1 step size to 4320 bars for better independent test periods.
- Introduced MAX_WFO_WINDOWS to limit the number of evaluated windows per trial.
- Refactored window selection logic to pre-compute valid start positions and skip oldest windows if exceeding the maximum.
- Modified scoring parameters: reduced reg_lambda max from 10.0 to 5.0 and min_child_weight from 100 to 40.
- Enhanced WFE calculation to avoid misinterpretation of consistently negative models.
- Updated the state remapping logic in `engine_hmm.py` to ensure consistent labeling of states based on volatility and returns, addressing issues with out-of-sample regime inconsistency.
- Modified `train_xgb` in `engine_xgb.py` to handle cases where the dataset may not have a holdout set, ensuring robust model training and evaluation.
- Introduced annualization in the scoring function within `optimizer.py` to improve the Calmar ratio calculation, making it more comparable across different timeframes.
- Added penalties for high cross-window Sharpe variance and negative WFE ratios in the objective function to discourage overfitting and promote model consistency.
- Enhanced logging to provide clearer insights during the WFO process, including warnings for skipped windows and potential issues with model training.
- Resolve 5 committed git merge conflict markers in GoldRegimeX_Explorer.ipynb
that rendered it invalid JSON and unexecutable in Jupyter
- Conflict resolution: keep wfo_mode_w dropdown, reg_lambda5_w / min_child_weight5_w
sliders, Run WFO Score Comparison button, and Section 6 CV Path Inspector from
copilot branch; keep ensemble API imports (load_xgb_ensemble, get_predictions_ensemble)
from HEAD
- Fix notebook Sections 1 and 5 to use load_xgb_ensemble / train_xgb_ensemble /
get_predictions_ensemble instead of old single-model API; ensures equity curves
and WFO scores match production vol-bucketed ensemble
- Update README changelog with merge conflict resolution and ensemble API fix entry
- Updated MIN_OOS_TRADES_HARD for M15 from 30 to 60.
- Introduced WFO_PARAMS_FAST for quicker intraday regime detection.
- Adjusted WFO_TRIALS and added WFO_TRIALS_FAST to accommodate wider search space.
- Revised scoring formula in compute_cpcv_score to improve accuracy and range.
- Enhanced _run_wfo to support different WFO modes and adjusted CV folds based on TF.
- Aligned feature preparation and state alignment for consistency in backtesting.
- Updated make_objective to include new parameters and improved tree depth and learning rate settings.
- Changed pruner strategy based on TF in run_optimization for better trial management.
Updated the out-of-sample (OOS) backtesting process to ensure that the states used are aligned with the OOS data frame index. This change addresses potential mismatches by applying a mask to the states based on the aligned index, improving the accuracy of the backtest results.
- Added WFO parameters and trial configurations for H1, M15, and M5 timeframes.
- Introduced _run_wfo function to evaluate hyperparameters across non-overlapping OOS windows.
- Updated make_objective to utilize WFO instead of CPCV for scoring.
- Enhanced asset feature processing in processor.py to include additional external assets (XAGUSD, XTIUSD, US500, USDJPY) and their log-return calculations.
- Implemented a generic load_asset_data function to streamline loading and processing of asset data.
- Added compute_synth_vix function to calculate a synthetic VIX proxy from OHLC data.
- Improved handling of missing data and logging for asset features.
- Introduced CPCV for train/test splitting, allowing for more robust evaluation across multiple market regimes.
- Updated backtester to support test masks, ensuring trades are only executed during test periods.
- Refactored optimizer to compute scores using CPCV, improving trial efficiency and accuracy.
- Adjusted signal engine logic to incorporate directional probabilities for trade entries based on XGBoost predictions.
- Improved overall code structure and readability, ensuring better maintainability and performance.
- Deleted `signal_evaluator.py`, which contained the previous Z-Score and RCEV evaluation logic.
- Added `signal_engine.py`, implementing a new stateful signal engine for regime confirmation and trade management.
- The new engine evaluates entry and exit conditions based on regime stability, XGBoost confidence, and Bollinger Band positions.
- Introduced constants for minimum confirmation bars, entry probabilities, maximum hold bars, and profit erosion thresholds.
- Introduced RCEVScorer class for calculating expected dollar profit based on historical trade outcomes and market conditions.
- Replaced Z-Score signal gating with RCEV scoring in SignalEvaluator for live evaluations.
- Added new configuration for RCEV thresholds and tiered floors based on timeframes.
- Enhanced MT5 trader logic to utilize RCEV scoring for trade decision-making, including regime-aware take-profit and stop-loss calculations.
- Updated ATR trailing stop logic to incorporate new RCEV parameters and configurations.
- Maintained backward compatibility with existing Z-Score logic for scenarios without RCEV calibration.
- Created a new StandardScaler object for feature scaling, saved as tcn_feature_scaler.pkl.
- Added metadata in JSON format for the M15_headway_cent model, including sequence length, number of features, number of states, temperature, and training timestamp.
- Deleted ensemble metadata, feature scalers, and model metadata for H1 headway cent models.
- Removed LSTM regime classifier files for H1 headway cent models.
- Deleted corresponding files for model_0, model_1, and model_2 in both H1 and M15 headway cent directories.
- Cleaned up unnecessary files to streamline the model directory structure.
- Introduced `engine_tcn.py` for Temporal Convolutional Network (TCN) model to score confidence based on recent market data.
- Added `data_updater.py` for automatic weekly data updates from MT5, ensuring fresh data for TCN.
- Created `tcn_maintenance.py` for automatic TCN model maintenance, including staleness checks and background retraining.
- Updated `mt5_trader.py` to integrate TCN confidence scoring, replacing the previous LSTM ensemble logic.
- Enhanced logging and error handling throughout the new modules for better traceability.
- Full workflow from raw data to live trading (Phase 0–7)
- LSTM ensemble, temperature calibration, fine-tuning, staleness/auto-retrain
- Sensitivity analysis and tiered Z-Score trading modes
- Updated command reference (all 16 modes + 22 CLI options)
- Signal logic with exact TF Z-Score cutoffs
- Guardian LSTM auto-retrain documentation
- Model files layout and troubleshooting guide
Luke Mosoti