bench: bigger sweep points, and call them sweeps (#9)

The three points were sized before this runner had ever run one. It has now,
so they are sized from what it measured: 87.5 us per combination for manifoldbt
at 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for
raptorbt at 200,000 bars.

20,000 x 5,000 stays, because it is the only one of the three vectorbt can hold:
it materialises 1.57 MB per combination at that length, so 5,000 already costs
it 2.5 GB. The other two grow to 20,000 and 10,000 combinations and put it out
of scope, which is where a sweep stops being a speed comparison and becomes a
capability one.

raptorbt sets the budget, not manifoldbt. With no fan-out API its sweep is a
Python loop costing a full backtest per cell, so the large point goes deep in
combinations on a short series rather than the reverse: 20,000 combinations on
20,000 bars costs it 27 s a call, where 5,000 combinations on a million bars
would cost it 25 minutes.

Also: sweeps, not grids. `run_sweep`, `run_sweep_lite` and `--sweep` are what
the product calls this, and a second word for the same thing is a second thing
to learn. `grid` is kept only where it means the parameter space itself.

.github/workflows/bench-vs-vectorbt.yml

@@ -149,10 +149,10 @@ jobs:
           if-no-files-found: warn
 
   # ------------------------------------------------------------------------ #
-  # Parameter grids, which need a licence and therefore a job of their own
+  # Parameter sweeps, which need a licence and therefore a job of their own
   # ------------------------------------------------------------------------ #
   sweeps:
-    name: grids (ubuntu-latest)
+    name: sweeps (ubuntu-latest)
     runs-on: ubuntu-latest
     if: github.event_name != 'release' || github.event.release.prerelease == false
     timeout-minutes: 45
@@ -175,7 +175,7 @@ jobs:
           if [ -n "$VERSION" ]; then pip install "manifoldbt==${VERSION}"; else pip install manifoldbt; fi
           pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
 
-      # A grid benchmark without a licence does not fail, it produces a wrong
+      # A sweep benchmark without a licence does not fail, it produces a wrong
       # number: every unlicensed fan-out call waits a fixed interval before any
       # work starts, so the stopwatch would time the wait. This step exits
       # non-zero rather than let that happen, and the harness refuses again on
@@ -187,16 +187,27 @@ jobs:
         working-directory: benchmarks/vs_vectorbt
         run: python ci_activate.py
 
-      - name: Run the grids
+      - name: Run the sweeps
         shell: bash
         working-directory: benchmarks/vs_vectorbt
-        # Three points, chosen from a measured map of the bars-by-combinations
+        # Three points, sized from what this runner actually did rather than
-        # plane rather than picked: across it the ratio moves between x32 and
+        # guessed. Measured here: 87.5 us per combination for manifoldbt at
-        # x38, so a denser matrix would spend runner time re-measuring the same
+        # 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms
-        # number. What the three do carry is the shape of the thing: two grid
+        # for raptorbt at 200,000 bars.
-        # sizes at one series length, and one grid vectorbt cannot hold at all.
+        #
+        # The first point is the only one vectorbt can hold: at 20,000 bars it
+        # materialises 1.57 MB per combination, so 5,000 of them already cost it
+        # 2.5 GB and 20,000 would need 31 GB. The other two are out of its
+        # scope, and they are where a sweep stops being a speed comparison and
+        # becomes a capability one.
+        #
+        # raptorbt sets the budget, not manifoldbt: with no fan-out API its
+        # sweep is a Python loop, so it costs a full backtest per cell. That is
+        # why the large point goes deep in combinations on short series rather
+        # than the reverse -- 20,000 combinations on 20,000 bars costs it 27 s a
+        # call, where 5,000 combinations on a million bars would cost it 25 min.
         run: |
-          python bench.py             --workloads sma_cross             --bars 100000             --reps 1             --cold-start-reps 0             --sweep 20000:2500 20000:5000 200000:2500:oos             --sweep-reps "${{ inputs.reps || '2' }}"             --out "results-sweeps.json"
+          python bench.py             --workloads sma_cross             --bars 100000             --reps 1             --cold-start-reps 0             --sweep 20000:5000 20000:20000:oos 200000:10000:oos             --sweep-reps "${{ inputs.reps || '2' }}"             --out "results-sweeps.json"
 
       - name: Render the report
         if: always()

benchmarks/vs_vectorbt/README.md

@@ -290,30 +290,36 @@ On the raptorbt side specifically:
 - `bench.py` - the runner
 - `sweep_child.py` - one parameter-grid point, in its own process
 - `report.py` - JSON to Markdown, and to the GitHub job summary
-- `ci_activate.py` - activates the licence the grid job needs, and refuses to
+- `ci_activate.py` - activates the licence the sweep job needs, and refuses to
   continue without one
 - `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
 
-## Parameter grids
+## Parameter sweeps
 
-Grids run in their own CI job, because they need a licence: an unlicensed
+Sweeps run in their own CI job, because they need a licence: an unlicensed
 fan-out call waits out a fixed interval before doing any work, so a stopwatch
-would be timing the wait rather than the engine. The harness refuses to produce
+would be timing the wait. The harness refuses to produce a number in that state
-a number in that state rather than producing a wrong one.
+rather than producing a wrong one.
 
-Three points, chosen from a measured map of the plane rather than picked. Across
+Three points, sized from what the runner actually did rather than guessed.
-bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio
+Measured there: 87.5 us per combination for manifoldbt at 20,000 bars, 1.16 ms
-on four cores moves only between x32 and x38, so a denser matrix would spend
+for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for raptorbt at 200,000 bars.
-runner time re-measuring the same number. What the three points carry is the
-shape: two grid sizes at one series length, and one grid vectorbt cannot hold at
-all (it materialises the simulation per combination, measured at 3.93 MB per
-combination on 50,000 bars).
 
-Grid ratios are much more sensitive to core count than single backtests are,
+Only the first point is one vectorbt can hold. It materialises the simulation
-because manifoldbt is the only one of the three that fans out across cores:
+per combination -- 1.57 MB of it at 20,000 bars -- so 5,000 combinations already
-measured on the same point, x38 on four cores and x146 on twenty. Numbers from
+cost it 2.5 GB and 20,000 would need 31 GB. The other two are out of its scope,
-the CI job are four-core numbers, and they are the conservative ones.
+and that is where a sweep stops being a speed comparison and becomes a
+capability one: the question is no longer how long it takes but whether the
+machine can hold it at all.
 
-The directory is still named `vs_vectorbt` and the workflow file still
+raptorbt sets the time budget rather than manifoldbt. With no fan-out API for a
-`bench-vs-vectorbt.yml`: renaming either would break the path the public
+parameter grid on one instrument, its sweep is a Python loop costing a full
-repository runs and start a fresh, empty run history.
+backtest per cell, which is why the large point goes deep in combinations on a
+short series instead of the reverse: 20,000 combinations on 20,000 bars costs it
+27 seconds a call, where 5,000 combinations on a million bars would cost it 25
+minutes.
+
+Sweep ratios depend on the machine far more than single-backtest ratios do,
+because manifoldbt is the only one of the three that fans out across cores. The
+same point measured x13 on a four-vCPU cloud runner and x32 on two fast desktop
+cores; the CI numbers are the conservative ones, and they are the ones published.