diff --git a/.github/workflows/bench-vs-vectorbt.yml b/.github/workflows/bench-vs-vectorbt.yml index 719c965..bdc9d83 100644 --- a/.github/workflows/bench-vs-vectorbt.yml +++ b/.github/workflows/bench-vs-vectorbt.yml @@ -149,10 +149,10 @@ jobs: if-no-files-found: warn # ------------------------------------------------------------------------ # - # Parameter grids, which need a licence and therefore a job of their own + # Parameter sweeps, which need a licence and therefore a job of their own # ------------------------------------------------------------------------ # sweeps: - name: grids (ubuntu-latest) + name: sweeps (ubuntu-latest) runs-on: ubuntu-latest if: github.event_name != 'release' || github.event.release.prerelease == false timeout-minutes: 45 @@ -175,7 +175,7 @@ jobs: if [ -n "$VERSION" ]; then pip install "manifoldbt==${VERSION}"; else pip install manifoldbt; fi pip install -r benchmarks/vs_vectorbt/requirements-lock.txt - # A grid benchmark without a licence does not fail, it produces a wrong + # A sweep benchmark without a licence does not fail, it produces a wrong # number: every unlicensed fan-out call waits a fixed interval before any # work starts, so the stopwatch would time the wait. This step exits # non-zero rather than let that happen, and the harness refuses again on @@ -187,16 +187,27 @@ jobs: working-directory: benchmarks/vs_vectorbt run: python ci_activate.py - - name: Run the grids + - name: Run the sweeps shell: bash working-directory: benchmarks/vs_vectorbt - # Three points, chosen from a measured map of the bars-by-combinations - # plane rather than picked: across it the ratio moves between x32 and - # x38, so a denser matrix would spend runner time re-measuring the same - # number. What the three do carry is the shape of the thing: two grid - # sizes at one series length, and one grid vectorbt cannot hold at all. + # Three points, sized from what this runner actually did rather than + # guessed. Measured here: 87.5 us per combination for manifoldbt at + # 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms + # for raptorbt at 200,000 bars. + # + # The first point is the only one vectorbt can hold: at 20,000 bars it + # materialises 1.57 MB per combination, so 5,000 of them already cost it + # 2.5 GB and 20,000 would need 31 GB. The other two are out of its + # scope, and they are where a sweep stops being a speed comparison and + # becomes a capability one. + # + # raptorbt sets the budget, not manifoldbt: with no fan-out API its + # sweep is a Python loop, so it costs a full backtest per cell. That is + # why the large point goes deep in combinations on short series rather + # than the reverse -- 20,000 combinations on 20,000 bars costs it 27 s a + # call, where 5,000 combinations on a million bars would cost it 25 min. run: | - python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:2500 20000:5000 200000:2500:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json" + python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:5000 20000:20000:oos 200000:10000:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json" - name: Render the report if: always() diff --git a/benchmarks/vs_vectorbt/README.md b/benchmarks/vs_vectorbt/README.md index c40c7e7..5a8ddda 100644 --- a/benchmarks/vs_vectorbt/README.md +++ b/benchmarks/vs_vectorbt/README.md @@ -290,30 +290,36 @@ On the raptorbt side specifically: - `bench.py` - the runner - `sweep_child.py` - one parameter-grid point, in its own process - `report.py` - JSON to Markdown, and to the GitHub job summary -- `ci_activate.py` - activates the licence the grid job needs, and refuses to +- `ci_activate.py` - activates the licence the sweep job needs, and refuses to continue without one - `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository -## Parameter grids +## Parameter sweeps -Grids run in their own CI job, because they need a licence: an unlicensed +Sweeps run in their own CI job, because they need a licence: an unlicensed fan-out call waits out a fixed interval before doing any work, so a stopwatch -would be timing the wait rather than the engine. The harness refuses to produce -a number in that state rather than producing a wrong one. +would be timing the wait. The harness refuses to produce a number in that state +rather than producing a wrong one. -Three points, chosen from a measured map of the plane rather than picked. Across -bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio -on four cores moves only between x32 and x38, so a denser matrix would spend -runner time re-measuring the same number. What the three points carry is the -shape: two grid sizes at one series length, and one grid vectorbt cannot hold at -all (it materialises the simulation per combination, measured at 3.93 MB per -combination on 50,000 bars). +Three points, sized from what the runner actually did rather than guessed. +Measured there: 87.5 us per combination for manifoldbt at 20,000 bars, 1.16 ms +for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for raptorbt at 200,000 bars. -Grid ratios are much more sensitive to core count than single backtests are, -because manifoldbt is the only one of the three that fans out across cores: -measured on the same point, x38 on four cores and x146 on twenty. Numbers from -the CI job are four-core numbers, and they are the conservative ones. +Only the first point is one vectorbt can hold. It materialises the simulation +per combination -- 1.57 MB of it at 20,000 bars -- so 5,000 combinations already +cost it 2.5 GB and 20,000 would need 31 GB. The other two are out of its scope, +and that is where a sweep stops being a speed comparison and becomes a +capability one: the question is no longer how long it takes but whether the +machine can hold it at all. -The directory is still named `vs_vectorbt` and the workflow file still -`bench-vs-vectorbt.yml`: renaming either would break the path the public -repository runs and start a fresh, empty run history. +raptorbt sets the time budget rather than manifoldbt. With no fan-out API for a +parameter grid on one instrument, its sweep is a Python loop costing a full +backtest per cell, which is why the large point goes deep in combinations on a +short series instead of the reverse: 20,000 combinations on 20,000 bars costs it +27 seconds a call, where 5,000 combinations on a million bars would cost it 25 +minutes. + +Sweep ratios depend on the machine far more than single-backtest ratios do, +because manifoldbt is the only one of the three that fans out across cores. The +same point measured x13 on a four-vCPU cloud runner and x32 on two fast desktop +cores; the CI numbers are the conservative ones, and they are the ones published.