bench: fix the red runs, trim the matrix, add a grid job (#7)

Five consecutive red runs, two unrelated causes.

Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.

The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.

Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.

Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.
This commit is contained in:
Exocet92
2026-08-20 17:00:41 +02:00
committed by GitHub
parent 52cbe1ba54
commit a4040375e2
4 changed files with 204 additions and 16 deletions
+60 -2
View File
@@ -85,11 +85,47 @@ store)`, the documented entry point, not through an internal fast path.
| Workload | What it exercises | vectorbt | raptorbt |
|---|---|---|---|
| `sma_cross` | SMA 10/50 crossover, long-only, no cost | exact | exact |
| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee | exact | unsupported |
| `sma_cross` | SMA 30/150 crossover, long-only, no cost | exact | exact |
| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee, capped at 1M bars | exact | unsupported |
| `sma_cross_metrics` | the same simulation, plus max drawdown, Sharpe, Sortino and volatility | exact | exact |
| `bracket_sl_tp` | the same entry with a 15 bps stop and a 30 bps target | documented | documented |
### Why the fee workload stops at 1M bars
A workload can stop being a comparison before it stops running. `ema_rsi_fees`
sizes in fixed units and pays 5 bps a side, and at 1-minute resolution it turns
over often enough that the fees compound into the account: measured, it ends at
-15% of capital on 1M bars, -74% on 5M, and exactly -100% on 10M, where fees
reach 99,611 of the 100,000 it started with.
Past that point the engines still agree on the equity, because both are sitting
at zero, and disagree by thousands of round-trips about how many more worthless
trades to book on a dead account. That is a fact about a bankrupt strategy, not
about either engine, so the workload carries a ceiling and the runner skips it
above that with the reason printed. The other workloads have no ceiling.
### What the windows are, and why they moved
`sma_cross` crosses on 30/150 rather than 10/50. The two were measured against
each other on the same 5M bars, and the slower pair is worth about 15% on the
ratio (x267 against x232 with a performance summary) because it books a third of
the trades and manifoldbt's cost, unlike vectorbt's, moves with the trade count.
That is a real effect and a small one, and it is worth knowing which way the
knobs turn before anyone quotes a number:
| Turn up | Effect on the ratio | Why |
|---|---|---|
| series length | **widens** | vectorbt materialises the simulation; its cost is linear in bars |
| asking for the summary | **widens sharply** | it has to build the equity curve it deferred |
| number of trades | narrows | near-free for vectorbt's per-bar loop, real for manifoldbt |
| number of indicators | narrows | same reason |
Measured at 5M bars on four levels of turnover, the ratio runs from x40 at
480,000 round-trips to x71 at 2,500. The floor matters more than the peak: even
in the busiest configuration tested, with half a million round-trips, the gap
holds at x40, and x151 with a performance summary.
Each of those runs across a range of series lengths. Two further axes, cold
start and memory, are measured in their own processes because they cannot be
measured honestly inside the main one.
@@ -254,8 +290,30 @@ On the raptorbt side specifically:
- `bench.py` - the runner
- `sweep_child.py` - one parameter-grid point, in its own process
- `report.py` - JSON to Markdown, and to the GitHub job summary
- `ci_activate.py` - activates the licence the grid job needs, and refuses to
continue without one
- `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
## Parameter grids
Grids run in their own CI job, because they need a licence: an unlicensed
fan-out call waits out a fixed interval before doing any work, so a stopwatch
would be timing the wait rather than the engine. The harness refuses to produce
a number in that state rather than producing a wrong one.
Three points, chosen from a measured map of the plane rather than picked. Across
bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio
on four cores moves only between x32 and x38, so a denser matrix would spend
runner time re-measuring the same number. What the three points carry is the
shape: two grid sizes at one series length, and one grid vectorbt cannot hold at
all (it materialises the simulation per combination, measured at 3.93 MB per
combination on 50,000 bars).
Grid ratios are much more sensitive to core count than single backtests are,
because manifoldbt is the only one of the three that fans out across cores:
measured on the same point, x38 on four cores and x146 on twenty. Numbers from
the CI job are four-core numbers, and they are the conservative ones.
The directory is still named `vs_vectorbt` and the workflow file still
`bench-vs-vectorbt.yml`: renaming either would break the path the public
repository runs and start a fresh, empty run history.