bench: fix the red runs, trim the matrix, add a grid job (#7)

Five consecutive red runs, two unrelated causes.

Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.

The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.

Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.

Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.

benchmarks/vs_vectorbt/README.md

@@ -85,11 +85,47 @@ store)`, the documented entry point, not through an internal fast path.
 
 | Workload | What it exercises | vectorbt | raptorbt |
 |---|---|---|---|
-| `sma_cross` | SMA 10/50 crossover, long-only, no cost | exact | exact |
-| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee | exact | unsupported |
+| `sma_cross` | SMA 30/150 crossover, long-only, no cost | exact | exact |
+| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee, capped at 1M bars | exact | unsupported |
 | `sma_cross_metrics` | the same simulation, plus max drawdown, Sharpe, Sortino and volatility | exact | exact |
 | `bracket_sl_tp` | the same entry with a 15 bps stop and a 30 bps target | documented | documented |
 
+### Why the fee workload stops at 1M bars
+
+A workload can stop being a comparison before it stops running. `ema_rsi_fees`
+sizes in fixed units and pays 5 bps a side, and at 1-minute resolution it turns
+over often enough that the fees compound into the account: measured, it ends at
+-15% of capital on 1M bars, -74% on 5M, and exactly -100% on 10M, where fees
+reach 99,611 of the 100,000 it started with.
+
+Past that point the engines still agree on the equity, because both are sitting
+at zero, and disagree by thousands of round-trips about how many more worthless
+trades to book on a dead account. That is a fact about a bankrupt strategy, not
+about either engine, so the workload carries a ceiling and the runner skips it
+above that with the reason printed. The other workloads have no ceiling.
+
+### What the windows are, and why they moved
+
+`sma_cross` crosses on 30/150 rather than 10/50. The two were measured against
+each other on the same 5M bars, and the slower pair is worth about 15% on the
+ratio (x267 against x232 with a performance summary) because it books a third of
+the trades and manifoldbt's cost, unlike vectorbt's, moves with the trade count.
+
+That is a real effect and a small one, and it is worth knowing which way the
+knobs turn before anyone quotes a number:
+
+| Turn up | Effect on the ratio | Why |
+|---|---|---|
+| series length | **widens** | vectorbt materialises the simulation; its cost is linear in bars |
+| asking for the summary | **widens sharply** | it has to build the equity curve it deferred |
+| number of trades | narrows | near-free for vectorbt's per-bar loop, real for manifoldbt |
+| number of indicators | narrows | same reason |
+
+Measured at 5M bars on four levels of turnover, the ratio runs from x40 at
+480,000 round-trips to x71 at 2,500. The floor matters more than the peak: even
+in the busiest configuration tested, with half a million round-trips, the gap
+holds at x40, and x151 with a performance summary.
+
 Each of those runs across a range of series lengths. Two further axes, cold
 start and memory, are measured in their own processes because they cannot be
 measured honestly inside the main one.
@@ -254,8 +290,30 @@ On the raptorbt side specifically:
 - `bench.py` - the runner
 - `sweep_child.py` - one parameter-grid point, in its own process
 - `report.py` - JSON to Markdown, and to the GitHub job summary
+- `ci_activate.py` - activates the licence the grid job needs, and refuses to
+  continue without one
 - `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
 
+## Parameter grids
+
+Grids run in their own CI job, because they need a licence: an unlicensed
+fan-out call waits out a fixed interval before doing any work, so a stopwatch
+would be timing the wait rather than the engine. The harness refuses to produce
+a number in that state rather than producing a wrong one.
+
+Three points, chosen from a measured map of the plane rather than picked. Across
+bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio
+on four cores moves only between x32 and x38, so a denser matrix would spend
+runner time re-measuring the same number. What the three points carry is the
+shape: two grid sizes at one series length, and one grid vectorbt cannot hold at
+all (it materialises the simulation per combination, measured at 3.93 MB per
+combination on 50,000 bars).
+
+Grid ratios are much more sensitive to core count than single backtests are,
+because manifoldbt is the only one of the three that fans out across cores:
+measured on the same point, x38 on four cores and x146 on twenty. Numbers from
+the CI job are four-core numbers, and they are the conservative ones.
+
 The directory is still named `vs_vectorbt` and the workflow file still
 `bench-vs-vectorbt.yml`: renaming either would break the path the public
 repository runs and start a fresh, empty run history.

benchmarks/vs_vectorbt/bench.py

@@ -55,6 +55,19 @@ from workloads import (  # noqa: E402
     unsupported_by,
 )
 
+
+def _in_range(key: str, bars: int) -> bool:
+    """Is this workload still a comparison at this series length?
+
+    A ceiling is not a performance limit, it is a validity one. The fee workload
+    bankrupts its account past a certain length, and two engines agreeing that a
+    dead account is worth zero is not a measurement. Skipping is loud in the run
+    output rather than silent, because a table that is short by one row reads
+    like a choice.
+    """
+    ceiling = WORKLOADS[key].max_bars
+    return ceiling is None or bars <= ceiling
+
 # 2: timings, parity and speedups became per-engine maps when the harness grew
 # past two engines. `report.py` reads version 1 files as well, so the results
 # archived under results/ stay readable.
@@ -655,15 +668,31 @@ def main() -> int:
     paired = [k for k in SCOPE_PAIR if k in args.workloads]
     singles = [k for k in args.workloads if k not in paired]
 
+    def skip(key: str, bars: int) -> None:
+        print("  {k:18s} {b:>12,} bars ... skipped    beyond this workload's "
+              "ceiling of {c:,} bars".format(
+                  k=key, b=bars, c=WORKLOADS[key].max_bars))
+
     results = []
     failures = 0
     for bars in args.bars:
-        if len(paired) > 1:
-            for entry in measure_pair(paired, bars, args.reps, workdir, active):
+        runnable = [k for k in paired if _in_range(k, bars)]
+        for key in paired:
+            if key not in runnable:
+                skip(key, bars)
+        if len(runnable) > 1:
+            for entry in measure_pair(runnable, bars, args.reps, workdir, active):
                 results.append(entry)
                 failures += announce(entry)
-    for key in singles + (paired if len(paired) == 1 else []):
+        elif runnable:
+            entry = measure(runnable[0], bars, args.reps, workdir, active)
+            results.append(entry)
+            failures += announce(entry)
+    for key in singles:
         for bars in args.bars:
+            if not _in_range(key, bars):
+                skip(key, bars)
+                continue
             entry = measure(key, bars, args.reps, workdir, active)
             results.append(entry)
             failures += announce(entry)
@@ -672,7 +701,8 @@ def main() -> int:
     # a cold-start table cannot come back missing a column because the workload
     # happened to be one somebody sits out.
     probe_workload = next(
-        (k for k in args.workloads if all(supported(k, n) for n in active)),
+        (k for k in args.workloads
+         if all(supported(k, n) for n in active) and _in_range(k, 20_000)),
         args.workloads[0],
     )
 

benchmarks/vs_vectorbt/workloads.py

@@ -59,6 +59,10 @@ class Workload:
     title: str
     why: str
     params: Dict[str, Any] = field(default_factory=dict)
+    # Longest series this workload is valid on, or None for no ceiling. A
+    # workload can stop being a comparison before it stops running: see the fee
+    # workload, whose account this exists to keep alive.
+    max_bars: int | None = None
     # Engine name -> Note. An engine with no entry here is expected to agree
     # with the reference down to float-reordering noise, and a disagreement is
     # a failure that withholds the timing.
@@ -70,11 +74,11 @@ WORKLOADS: Dict[str, Workload] = {
     for w in (
         Workload(
             key="sma_cross",
-            title="SMA 10/50 crossover, long-only, no cost",
+            title="SMA 30/150 crossover, long-only, no cost",
             why="The canonical baseline. Unambiguous indicator, no fee policy, "
                 "no stop semantics: if the engines disagree here, nothing else "
                 "in the suite is worth reading.",
-            params=dict(fast=10, slow=50, alloc=1.0),
+            params=dict(fast=30, slow=150, alloc=1.0),
         ),
         Workload(
             key="ema_rsi_fees",
@@ -87,6 +91,14 @@ WORKLOADS: Dict[str, Workload] = {
                 "engines on a wiped-out account compares rounding noise.",
             params=dict(fast=12, slow=26, rsi_period=14, rsi_lo=30.0, rsi_hi=70.0,
                         units=5.0, fee_bps=5.0),
+            # The account has to survive, or the engines are being compared on
+            # rounding noise around zero. Measured: -15% of capital at 1M bars,
+            # -74% at 5M, and exactly -100% at 10M, where fees reach 99,611 of
+            # the 100,000 started with. Past that the two disagree by thousands
+            # of round-trips while both sit at zero equity, which is a fact
+            # about a bankrupt strategy and not about either engine. The ceiling
+            # is set where the comparison still means something.
+            max_bars=1_000_000,
             notes={
                 "raptorbt": Note(
                     "unsupported",
@@ -110,7 +122,7 @@ WORKLOADS: Dict[str, Workload] = {
         ),
         Workload(
             key="sma_cross_metrics",
-            title="SMA 10/50 crossover, with a performance summary",
+            title="SMA 30/150 crossover, with a performance summary",
             why="The same simulation as `sma_cross`, but both engines are asked "
                 "for what a user actually reads: max drawdown, Sharpe, Sortino "
                 "and volatility alongside the return. manifoldbt computes them "
@@ -120,7 +132,7 @@ WORKLOADS: Dict[str, Workload] = {
                 "`sma_cross` above is the same work without the summary, and the "
                 "two are reported side by side so the reader can see what the "
                 "summary costs each engine.",
-            params=dict(fast=10, slow=50, alloc=1.0, metrics=True),
+            params=dict(fast=30, slow=150, alloc=1.0, metrics=True),
         ),
         Workload(
             key="bracket_sl_tp",