New --mode sensitivity command tests Z cutoffs 1.5..3.0 (step 0.25) on the
saved HMM + XGBoost ensemble without retraining. Loads data and models the
same way cmd_report does, then calls vectorized_backtest once per cutoff with
an evaluator_config override injected into SignalEvaluator.
Changes:
- src/sensitivity.py: run_sensitivity() loops over z_range, builds evaluator_config
override, calls vectorized_backtest, prints comparison table, saves CSV+JSON
- src/backtester.py: evaluator_config param added to compute_signals_zscore and
vectorized_backtest; threaded through to SignalEvaluator(config=)
- main.py: cmd_sensitivity loads models identical to cmd_report; MR cutoffs held
constant; sensitivity mode registered in choices + dispatch dict
Usage: python main.py --mode sensitivity --tf H1 --broker headway_cent --balance 15
All src.* loggers redirect to logs/goldregimex_{TF}.log on each TF-specific
command (optimize, train, compare, wfa, sync_validate, report, live).
main + notifier loggers keep writing to goldregimex.log for summaries and
Telegram. Switching TFs in a multi-TF loop (e.g. optimize M5+H1) cleanly
swaps handlers. M15 log deferred until current M15 study completes.
engine_xgb: std floor 0.02 → 0.003 in compute_regime_stats()
The 0.02 floor was dominating all states (all showing std=0.0200 in logs)
because XGB probability variance for weak-signal H1 is ~0.003-0.006.
Flooring at 0.02 inflated Z-Score thresholds by ~0.02 in prob space,
reducing H1 OOS trades to ~16 across 3 years.
optimizer: TF_MIN_OOS_TRADES H1: 60 → 15, M15: 150 → 100, M5: 400 → 300
With Z-Score filtering being more selective than fixed thresholds,
trade counts are lower. The old 60-trade penalty threshold caused a 0.1x
suppression on every H1 trial (true score ~3.24 was reported as 0.3235),
not breaking optimization but masking real quality in the output banner.
15 trades is appropriate for Z-Score filtered H1 OOS evaluation.
- Added `SignalEvaluator` class for shared adaptive signal evaluation logic across backtesting, optimization, and live trading.
- Introduced `compute_regime_stats` function to calculate per-HMM-state probability statistics for Z-Score calibration.
- Enhanced `vectorized_backtest` and `run_walk_forward` functions to support regime statistics and Z-Score signal generation.
- Updated `backtester.py` to include new signal computation method `compute_signals_zscore`.
- Modified `mt5_trader.py` to utilize the new signal evaluation logic and added regime stability tracking.
- Adjusted `optimizer.py` to compute regime statistics during model training and store them for later use.
- Updated validation logic in `validator.py` to incorporate regime statistics for improved signal evaluation.
- Refactored existing signal routing logic to accommodate new adaptive Z-Score evaluation framework.
Activity Bonus (optimizer.py):
- M5 OOS n_trades < 150: score *= 0.5 (extra penalty on top of progressive 0.1)
- M5 OOS n_trades > 300: score *= 1.2 (bonus rewards high-freq robustness)
- Forces Optuna to find the M5 'heartbeat' (150-400 OOS trades) rather than
low-activity solutions that hide drawdown by barely trading
Consecutive Loss Guard (risk_manager.py):
- AdaptiveRiskManager.MAX_STREAK = 3, LOCKOUT_HOURS = 4.0
- record_trade_result(won) increments streak on loss, resets on win
- can_trade() returns False during lockout window; auto-resets on expiry
- Prevents Death by a Thousand Cuts on M5 MR trades during trending breakouts
- __init__ accepts consecutive_losses + streak_locked_until for state handoff
between per-bar ARM re-instantiations in the live loop
M5 Efficiency Threshold (backtester.py):
- TF_MIN_EFFICIENCY dict: M5=7.0x, all other TFs=1.25x (default)
- vectorized_backtest applies tf-specific ER threshold via TF_MIN_EFFICIENCY.get(tf, 1.25)
- Blocks M5 signals where ATR < 7x spread (~$2.80 ATR on $2000 gold)
filtering out quiet night-session noise where MR edge is lowest
Max Monetary Risk label (visualizer.py):
- OOS section now shows "Max DD" % + dedicated "Max Monetary Risk: $X.XX USD" row
- Calculated as oos_fdd * account_size for clear monetary context on $15 accounts
TF_MAGIC_MAP + dynamic magic:
- Replace static MAGIC_NUMBER=123456 with TF_MAGIC_MAP {H1:123456, M15:123457, M5:123458}
- ALL_GRX_MAGICS frozenset for cross-TF queries
- MAGIC_NUMBER kept as H1 alias (MQL5 EA backwards-compat)
- _run_loop_inner resolves magic=TF_MAGIC_MAP[tf] at startup
- All loop-body references (position queries, close orders, has_open_position,
order sends, _execute_partial_close, _close_position) now use dynamic magic
- _execute_partial_close and _close_position gain magic= param (default=MAGIC_NUMBER)
TF-tagged order comments:
- New format: GRX_{tf}_{direction}_s{hmm_state}_tp{p+1}
e.g. GRX_H1_BUY_s0_tp1, GRX_M5_SELL_s1_tp2
- Visible in MT5 Trade tab so each timeframe's trades are instantly identifiable
Global multi-TF exposure guard (step 8):
- Counts open positions across ALL_GRX_MAGICS before any new signal
- If >= 4 total positions: logs [GLOBAL GUARD] and skips the signal
- Prevents over-leverage when two or more TFs fire on the same $15 balance
Daily P&L audit report:
- send_daily_audit_report(mt5, broker): queries history_deals_get for prior 24h,
groups DEAL_ENTRY_OUT fills by TF magic, formats per-TF and total P&L
- Called automatically at UTC midnight reset (try/except so crashes don't kill loop)
- Telegram output: "H1: +$0.45 (1 trade)\nM15: +$1.20 (4 trades)\n..."
_log_closed_pnl:
- Added tf= parameter; Telegram message now shows "[H1] Trade closed" tag
- DEAL_ENTRY_OUT wait was already implemented; added tf tag to surface TF in notification
Rewrite setup_logger to use io.TextIOWrapper(sys.stderr.buffer, encoding=utf-8)
instead of reconfigure(). reconfigure() silently fails under PowerShell on
Windows (the console stream has no .buffer equiv in some configurations),
leaving the handler on cp1252 which crashes on any Unicode > U+00FF.
New approach:
- Wraps sys.stderr.buffer directly in a fresh TextIOWrapper(utf-8, errors=replace)
- Falls back to sys.stderr as-is if .buffer is absent (IDLE / test runners)
- FileHandler explicitly opens goldregimex.log with encoding=utf-8
- Eliminates all future cp1252 crashes from emoji, box-drawing, or math symbols
Add weekly return consistency metric + M5/M15 scoring bonus
BACKTESTER (_compute_metrics):
- Buckets strategy_returns into ~1-trading-week windows (H1:40, M15:160, M5:480 bars)
- Computes weekly P&L std and mean; consistency = 1 - std/(std+|mean|) ∈ [0,1]
* 1.0 = identical income every week * 0.0 = erratic / zero mean
- Requires ≥4 complete windows and ≥5 trades; 0.0 otherwise
- Added as "return_consistency" to _compute_metrics return dict
- Surfaces as "oos_return_consistency" in vectorized_backtest split result
(and propagates automatically into WFA window dicts via **result_w)
OPTIMIZER (_score_result + objective):
- Consistency bonus: score += return_consistency * 0.5 for M5 and M15 only
* H1 excluded: swing TF with 1-2 trades/week; weekly bucketing is noise
* Max bonus: +0.5 on a 0-25+ scale — a gentle nudge, not a dominant term
- "oos_return_consistency" threaded into oos_result dict so _score_result sees it
- Trial log updated: Consist=%.2f column visible for all TFs
MAIN (cmd_wfa):
- fold_r includes "return_consistency" from oos_return_consistency window key
- _calc_score(fold_r, tf=tf) now passes TF so consistency bonus applies to
M5/M15 WFA fold reports (was defaulting to tf="H1", bonus was silently 0)
LOGIC AUDIT (replaces generic "No signal" log):
- Extracts gmm_cluster from features_df each bar for vol context
- Classifies no-signal bars as DIVERGENCE or ALIGNMENT:
* DIVERGENCE: HMM and ML lean in opposite directions — names both
* ALIGNMENT Bull/Bear: same direction, shows exact gap to threshold
* CHOP REGIME: shows MR thresholds and current prob distance
- Two-line output per idle bar:
[LOGIC AUDIT] BULL Regime | ML DOWN (0.482) | DIVERGENCE — ...
Distance → BUY: -0.168 | SELL: -0.134
Positive distance = would have fired; negative = still needs to move
GMM HIGH-VOL MR WARNING:
- When a Mean Reversion signal fires (Chop state, extreme prob), adds a
WARNING log if gmm_cluster == 2 (High Volatility environment)
- Does not block the trade — surfaces the risk so the user can decide
- Reason: MR edge degrades in breakout-vol conditions; low/med vol (0,1)
is the natural habitat for snap-back mean reversion on H1
- Import STATE_NAMES_2/3/4 from engine_hmm for state label lookup
- Bar log + Telegram: state label now uses model_hmm.n_components to select
the correct name dict (3-state: "Chop"; 4-state: "Chop_Low"/"Chop_High")
- Trend chop-exit: hmm_state == CHOP_STATE → hmm_state >= CHOP_STATE
(was silently missing Chop_High/state-3 in 4-state models — trend runners
would stay open through the entire Chop_High phase)
- MR breakout-exit: hmm_state != CHOP_STATE → hmm_state < CHOP_STATE
(was incorrectly closing MR positions when entering Chop_High, treating
an intra-chop state transition as a directional breakout)
- engine_hmm and backtester already used >= / < idioms; no changes there
ENGINE_HMM:
- fit_hmm: StandardScaler on [kalman_return, volatility, rsi_slope] before
GaussianHMM.fit() — prevents rsi_slope (range ±5) from dominating the
regime classification over kalman_return (±0.003) and producing extreme
Bear states that only fire after moves are exhausted
- model._obs_scaler stored as joblib-persistent attribute; no API changes
- predict_states: reuses stored scaler transparently; old models without
the attribute degrade gracefully (no scaling applied)
- State means log header updated: "scaled: ..." to reflect z-score space
OPTIMIZER (H1 only):
- prob_threshold range: 0.55–0.72 → 0.65–0.80 (forces higher certainty)
- short_threshold range: 0.28–0.45 → 0.20–0.35 (symmetric tightening)
- min_child_weight: 15–30 → 20–50 (deeper leaf nodes reduce IS memorisation)
- H1 DD guard: if oos_floating_dd > 5%, score *= 0.5 — strongly biases
Optuna away from solutions that survive IS but lose 10%+ OOS on H1
- main.py cmd_wfa: lower degenerate HMM guard from 0.72 to 0.65 (matches
optimizer persistence gate set in previous session)
- main.py cmd_wfa per-fold: append ⚠️PF<1.2 to fold flag when OOS profit
factor falls below the 1.2 robustness threshold (PF is already scored via
Complex Criterion but now also surfaced as an explicit standalone warning)
- backtester: add format_payout() helper — Cents (×100) for headway_cent,
USD for standard; add _compute_audit_metrics() computing avg_efficiency
(ATR/spread on active bars), cost_efficiency (1 - costs/gross_profit),
and total_net_payout (return × account_size); wire audit metrics into
vectorized_backtest for full period, IS slice, and OOS slice; update IS/OOS
log lines to show Eff and CostEff
- optimizer: import format_payout; trial logger now shows RF, PF, Payoff$,
Eff (ATR/spread), FloatDD%, trades, and multi-currency payout string
- validator: extract avg_efficiency, cost_efficiency, total_return from result;
expose all three in returned dict and logger
- main: import format_payout; cmd_train _fmt_block adds Efficiency, CostEff,
and Total Payout lines with ⚠️ warning when CostEff < 50%; cmd_sync_validate
shows efficiency + payout + warns on Eff < 1.2 or CostEff < 50%;
cmd_wfa per-fold row now includes Eff column
- backtester: lower spread-aware ER filter threshold 1.8 → 1.25 to allow
standard market conditions while blocking dead sessions (>80% spread erosion)
- engine_xgb: add TF_TRAIN_RATIO constant (H1=70%, M15/M5=65%); prepare_features
gains tf param so scaler is fitted on the correct IS fraction per timeframe
- optimizer: relax HMM persistence gate 0.80 → 0.72 for more active regime
detection; H1 prob_threshold range 0.58-0.78 → 0.55-0.72; short_threshold
0.22-0.42 → 0.28-0.45; reg_alpha H1/M15 capped at 1.2 (was 10.0); gamma
capped at 0.5 (was 5.0); max_depth H1/M15 raised to 3-6; payoff floor
$0.035 applied after trade-count penalty; oos_expected_payoff added to
oos_result dict; trial logger shows OOS_Payoff in dollars
- validator: prepare_features call passes tf param; spread-payoff ratio warning
fires when spread > 50% of edge
- main: import TF_TRAIN_RATIO; all prepare_features calls pass tf=tf; all
train_xgb_ensemble calls pass train_ratio from TF_TRAIN_RATIO
- models: delete stale study_headway_cent.db (incompatible with new scoring formula)
- Removed AdaptiveRiskManager trade limits from optimization and training outputs.
- Introduced DailyEquityGate to monitor daily equity and prevent excessive losses.
- Updated backtester to utilize new risk management features and removed session limits.
- Enhanced metrics computation to include recovery factor for better performance evaluation.
- Implemented Gaussian Mixture Model (GMM) for volatility clustering in processor and validator.
- Adjusted optimizer scoring to reward lower drawdown and incorporate recovery factor.
- Cleaned up unused model metadata files and updated database for consistency.
- Improved logging and error handling across various modules for better traceability.
The function was tz-stripping df_index to create idx for reindex matching,
then returning a Series indexed by tz-naive idx. When the caller assigns
df['usdchf_log_return'] = map_usdchf_to_bars(df.index, ...)
pandas 2.0 realigns by index — tz-aware df.index vs tz-naive result index
never match, producing an all-NaN column that wipes all rows after dropna.
Fix: wrap both return paths in pd.Series(..., index=df_index) so the
returned Series always carries the original tz-aware index, giving pandas
exact index alignment when assigned back to the sync DataFrame.
Luke Mosoti