fix(backtest): replace broken MC permutation test with binomial win-rate test

The previous monte_carlo_trade_pvalue() used sum(permuted_trades) as test
statistic, which is permutation-invariant (sum is commutative), so beat/n
was always 1.0 and MC_p was always 1.00 for every strategy.

Replace with a one-sided binomial test on trade win rate vs 50% baseline.
Tests whether the observed win rate could occur by chance under H0: p=0.5.

Also add _shift_daily_constant_factor_if_needed() to predix_full_eval.py
so re-evaluations apply the look-ahead bias correction for daily factors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
TPTBusiness
2026-04-24 09:55:45 +02:00
parent 78fb607dbe
commit dfbbffd7ea
2 changed files with 63 additions and 12 deletions
+13 -12
View File
@@ -356,11 +356,12 @@ def monte_carlo_trade_pvalue(
"""
Monte Carlo permutation test on trade-level P&L.
Shuffles the order of trade returns ``n_permutations`` times and computes
the fraction of runs whose total return is >= the real total return.
Runs a one-sided binomial test on trade-level win rate.
p < 0.05 → strategy has a statistically significant edge (real return
beats 95% of random sequences with the same set of trades).
Tests H0: win_rate = 0.5 (random trading) against H1: win_rate > 0.5.
The ``n_permutations`` parameter is kept for API compatibility but is unused.
p < 0.05 → win rate is significantly above 50%, indicating a genuine per-trade edge.
Parameters
----------
@@ -379,14 +380,14 @@ def monte_carlo_trade_pvalue(
if len(trade_pnl) < 2:
return 1.0
trades = trade_pnl.values.copy()
real_total = float(trades.sum())
rng = np.random.default_rng(seed)
beat = 0
for _ in range(n_permutations):
perm = rng.permutation(trades)
if perm.sum() >= real_total:
beat += 1
return beat / n_permutations
# Binomial test: is the win rate significantly above 50%?
# p = probability of observing >= n_wins out of n_trades under null (win_rate=0.5).
# Low p → strategy has a significant positive edge per trade.
from scipy.stats import binomtest
n_wins = int((trades > 0).sum())
n_total = len(trades)
result = binomtest(n_wins, n_total, p=0.5, alternative="greater")
return float(result.pvalue)
def walk_forward_rolling(