bench: fix the red runs, trim the matrix, add a grid job (#7)

Five consecutive red runs, two unrelated causes.

Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.

The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.

Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.

Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.

.github/workflows/bench-vs-vectorbt.yml

@@ -44,15 +44,26 @@ jobs:
   bench:
     name: ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 90
+    # A published release is not always a released version. The preview tags
+    # 0.18.0rc1 and rc2 went to the public repository without ever reaching
+    # PyPI, so `pip install manifoldbt==0.18.0rc2` failed and this workflow went
+    # red four times in two days for a reason that had nothing to do with any
+    # engine. Pre-releases are skipped; a manual dispatch can still benchmark
+    # one by naming the version, if it ever exists on PyPI.
+    if: github.event_name != 'release' || github.event.release.prerelease == false
+    timeout-minutes: 60
     strategy:
       fail-fast: false
       matrix:
         include:
+          # Three lengths, not five. 10k was sub-millisecond on the engine
+          # side, which measures the clock rather than the work, and 5M sat
+          # between two points that already bracket it. What is left is a
+          # decade apart each step, which is what makes the trend readable.
           - os: ubuntu-latest
-            bars: "10000 100000 1000000 5000000 10000000"
+            bars: "100000 1000000 10000000"
           - os: windows-latest
-            bars: "10000 100000 1000000 5000000 10000000"
+            bars: "100000 1000000 10000000"
           # macOS runners ship 7 GB of RAM against 16 GB elsewhere, and vectorbt
           # materialises the simulation in memory (roughly 150 MB per million
           # bars, measured). The top size is trimmed so a point is never lost to
@@ -60,7 +71,7 @@ jobs:
           # arithmetic is why 10M bars is added on the other two and not here:
           # measured, that point adds 1.55 GB on vectorbt's side alone.
           - os: macos-latest
-            bars: "10000 100000 1000000"
+            bars: "100000 1000000"
 
     env:
       PYTHONUNBUFFERED: "1"
@@ -115,11 +126,19 @@ jobs:
       - name: Render the report
         # Runs even when the benchmark exits non-zero: a parity failure is the
         # most interesting thing that can happen here, and it must be readable
-        # in the job summary rather than buried in a red step.
+        # in the job summary rather than buried in a red step. But only if there
+        # is something to render: when the install step failed, this used to die
+        # on a missing file and put a FileNotFoundError on top of the real
+        # error, which is how a run reports the wrong cause twice.
         if: always()
         shell: bash
         working-directory: benchmarks/vs_vectorbt
-        run: python report.py "results-${{ matrix.os }}.json"
+        run: |
+          if [ -f "results-${{ matrix.os }}.json" ]; then
+            python report.py "results-${{ matrix.os }}.json"
+          else
+            echo "no result file: the benchmark did not get far enough to write one"
+          fi
 
       - name: Upload the raw result
         if: always()
@@ -128,3 +147,72 @@ jobs:
           name: bench-${{ matrix.os }}
           path: benchmarks/vs_vectorbt/results-*.json
           if-no-files-found: warn
+
+  # ------------------------------------------------------------------------ #
+  # Parameter grids, which need a licence and therefore a job of their own
+  # ------------------------------------------------------------------------ #
+  sweeps:
+    name: sweeps (ubuntu)
+    runs-on: ubuntu-latest
+    if: github.event_name != 'release' || github.event.release.prerelease == false
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install engines from PyPI
+        shell: bash
+        run: |
+          set -e
+          VERSION="${{ inputs.manifoldbt_version }}"
+          if [ -z "$VERSION" ] && [ "${{ github.event_name }}" = "release" ]; then
+            VERSION="$(echo '${{ github.event.release.tag_name }}' | sed 's/^v//')"
+          fi
+          python -m pip install --upgrade pip
+          if [ -n "$VERSION" ]; then pip install "manifoldbt==${VERSION}"; else pip install manifoldbt; fi
+          pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
+
+      # A grid benchmark without a licence does not fail, it produces a wrong
+      # number: every unlicensed fan-out call waits a fixed interval before any
+      # work starts, so the stopwatch would time the wait. This step exits
+      # non-zero rather than let that happen, and the harness refuses again on
+      # its own if the tier is not what it expects.
+      - name: Activate the benchmark licence
+        env:
+          MANIFOLDBT_CI_LICENSE: ${{ secrets.MANIFOLDBT_CI_LICENSE }}
+        shell: bash
+        working-directory: benchmarks/vs_vectorbt
+        run: python ci_activate.py
+
+      - name: Run the grids
+        shell: bash
+        working-directory: benchmarks/vs_vectorbt
+        # Three points, chosen from a measured map of the bars-by-combinations
+        # plane rather than picked: across it the ratio moves between x32 and
+        # x38, so a denser matrix would spend runner time re-measuring the same
+        # number. What the three do carry is the shape of the thing: two grid
+        # sizes at one series length, and one grid vectorbt cannot hold at all.
+        run: |
+          python bench.py             --workloads sma_cross             --bars 100000             --reps 1             --cold-start-reps 0             --sweep 20000:2500 20000:5000 200000:2500:oos             --sweep-reps "${{ inputs.reps || '2' }}"             --out "results-sweeps.json"
+
+      - name: Render the report
+        if: always()
+        shell: bash
+        working-directory: benchmarks/vs_vectorbt
+        run: |
+          if [ -f results-sweeps.json ]; then
+            python report.py results-sweeps.json
+          else
+            echo "no result file: the grids did not get far enough to write one"
+          fi
+
+      - name: Upload the raw result
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: bench-sweeps
+          path: benchmarks/vs_vectorbt/results-sweeps.json
+          if-no-files-found: warn