bench: fix the red runs, trim the matrix, add a grid job (#7)

Five consecutive red runs, two unrelated causes.

Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.

The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.

Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.

Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.
This commit is contained in:
Exocet92
2026-08-20 17:00:41 +02:00
committed by GitHub
parent 52cbe1ba54
commit a4040375e2
4 changed files with 204 additions and 16 deletions
+16 -4
View File
@@ -59,6 +59,10 @@ class Workload:
title: str
why: str
params: Dict[str, Any] = field(default_factory=dict)
# Longest series this workload is valid on, or None for no ceiling. A
# workload can stop being a comparison before it stops running: see the fee
# workload, whose account this exists to keep alive.
max_bars: int | None = None
# Engine name -> Note. An engine with no entry here is expected to agree
# with the reference down to float-reordering noise, and a disagreement is
# a failure that withholds the timing.
@@ -70,11 +74,11 @@ WORKLOADS: Dict[str, Workload] = {
for w in (
Workload(
key="sma_cross",
title="SMA 10/50 crossover, long-only, no cost",
title="SMA 30/150 crossover, long-only, no cost",
why="The canonical baseline. Unambiguous indicator, no fee policy, "
"no stop semantics: if the engines disagree here, nothing else "
"in the suite is worth reading.",
params=dict(fast=10, slow=50, alloc=1.0),
params=dict(fast=30, slow=150, alloc=1.0),
),
Workload(
key="ema_rsi_fees",
@@ -87,6 +91,14 @@ WORKLOADS: Dict[str, Workload] = {
"engines on a wiped-out account compares rounding noise.",
params=dict(fast=12, slow=26, rsi_period=14, rsi_lo=30.0, rsi_hi=70.0,
units=5.0, fee_bps=5.0),
# The account has to survive, or the engines are being compared on
# rounding noise around zero. Measured: -15% of capital at 1M bars,
# -74% at 5M, and exactly -100% at 10M, where fees reach 99,611 of
# the 100,000 started with. Past that the two disagree by thousands
# of round-trips while both sit at zero equity, which is a fact
# about a bankrupt strategy and not about either engine. The ceiling
# is set where the comparison still means something.
max_bars=1_000_000,
notes={
"raptorbt": Note(
"unsupported",
@@ -110,7 +122,7 @@ WORKLOADS: Dict[str, Workload] = {
),
Workload(
key="sma_cross_metrics",
title="SMA 10/50 crossover, with a performance summary",
title="SMA 30/150 crossover, with a performance summary",
why="The same simulation as `sma_cross`, but both engines are asked "
"for what a user actually reads: max drawdown, Sharpe, Sortino "
"and volatility alongside the return. manifoldbt computes them "
@@ -120,7 +132,7 @@ WORKLOADS: Dict[str, Workload] = {
"`sma_cross` above is the same work without the summary, and the "
"two are reported side by side so the reader can see what the "
"summary costs each engine.",
params=dict(fast=10, slow=50, alloc=1.0, metrics=True),
params=dict(fast=30, slow=150, alloc=1.0, metrics=True),
),
Workload(
key="bracket_sl_tp",