gavindiaz/manifoldbt
1
0
Fork 0
mirror of https://github.com/manifoldbt/manifoldbt.git synced 2026-08-24 14:38:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench: bigger sweep points, and call them sweeps (#9)

Browse Source 
The three points were sized before this runner had ever run one. It has now,
so they are sized from what it measured: 87.5 us per combination for manifoldbt
at 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for
raptorbt at 200,000 bars.

20,000 x 5,000 stays, because it is the only one of the three vectorbt can hold:
it materialises 1.57 MB per combination at that length, so 5,000 already costs
it 2.5 GB. The other two grow to 20,000 and 10,000 combinations and put it out
of scope, which is where a sweep stops being a speed comparison and becomes a
capability one.

raptorbt sets the budget, not manifoldbt. With no fan-out API its sweep is a
Python loop costing a full backtest per cell, so the large point goes deep in
combinations on a short series rather than the reverse: 20,000 combinations on
20,000 bars costs it 27 s a call, where 5,000 combinations on a million bars
would cost it 25 minutes.

Also: sweeps, not grids. `run_sweep`, `run_sweep_lite` and `--sweep` are what
the product calls this, and a second word for the same thing is a second thing
to learn. `grid` is kept only where it means the parameter space itself.
This commit is contained in:
Exocet92
2026-08-20 17:14:13 +02:00
committed by GitHub
parent df13224efc
commit 9ddefc64df
2 changed files with 46 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/bench-vs-vectorbt.yml
+21 -10
View File
@@ -149,10 +149,10 @@ jobs:
if-no-files-found: warn
# ------------------------------------------------------------------------ #
# Parameter grids, which need a licence and therefore a job of their own
# Parameter sweeps, which need a licence and therefore a job of their own
# ------------------------------------------------------------------------ #
sweeps:
name: grids (ubuntu-latest)
name: sweeps (ubuntu-latest)
runs-on: ubuntu-latest
if: github.event_name != 'release' || github.event.release.prerelease == false
timeout-minutes: 45
@@ -175,7 +175,7 @@ jobs:
if [ -n "$VERSION" ]; then pip install "manifoldbt==${VERSION}"; else pip install manifoldbt; fi
pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
# A grid benchmark without a licence does not fail, it produces a wrong
# A sweep benchmark without a licence does not fail, it produces a wrong
# number: every unlicensed fan-out call waits a fixed interval before any
# work starts, so the stopwatch would time the wait. This step exits
# non-zero rather than let that happen, and the harness refuses again on
@@ -187,16 +187,27 @@ jobs:
working-directory: benchmarks/vs_vectorbt
run: python ci_activate.py
- name: Run the grids
- name: Run the sweeps
shell: bash
working-directory: benchmarks/vs_vectorbt
# Three points, chosen from a measured map of the bars-by-combinations
# plane rather than picked: across it the ratio moves between x32 and
# x38, so a denser matrix would spend runner time re-measuring the same
# number. What the three do carry is the shape of the thing: two grid
# sizes at one series length, and one grid vectorbt cannot hold at all.
# Three points, sized from what this runner actually did rather than
# guessed. Measured here: 87.5 us per combination for manifoldbt at
# 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms
# for raptorbt at 200,000 bars.
#
# The first point is the only one vectorbt can hold: at 20,000 bars it
# materialises 1.57 MB per combination, so 5,000 of them already cost it
# 2.5 GB and 20,000 would need 31 GB. The other two are out of its
# scope, and they are where a sweep stops being a speed comparison and
# becomes a capability one.
#
# raptorbt sets the budget, not manifoldbt: with no fan-out API its
# sweep is a Python loop, so it costs a full backtest per cell. That is
# why the large point goes deep in combinations on short series rather
# than the reverse -- 20,000 combinations on 20,000 bars costs it 27 s a
# call, where 5,000 combinations on a million bars would cost it 25 min.
run: |
python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:2500 20000:5000 200000:2500:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json"
python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:5000 20000:20000:oos 200000:10000:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json"
- name: Render the report
if: always()
benchmarks/vs_vectorbt/README.md
+25 -19
View File
@@ -290,30 +290,36 @@ On the raptorbt side specifically:
- `bench.py` - the runner
- `sweep_child.py` - one parameter-grid point, in its own process
- `report.py` - JSON to Markdown, and to the GitHub job summary
- `ci_activate.py` - activates the licence the grid job needs, and refuses to
- `ci_activate.py` - activates the licence the sweep job needs, and refuses to
continue without one
- `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
## Parameter grids
## Parameter sweeps
Grids run in their own CI job, because they need a licence: an unlicensed
Sweeps run in their own CI job, because they need a licence: an unlicensed
fan-out call waits out a fixed interval before doing any work, so a stopwatch
would be timing the wait rather than the engine. The harness refuses to produce
a number in that state rather than producing a wrong one.
would be timing the wait. The harness refuses to produce a number in that state
rather than producing a wrong one.
Three points, chosen from a measured map of the plane rather than picked. Across
bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio
on four cores moves only between x32 and x38, so a denser matrix would spend
runner time re-measuring the same number. What the three points carry is the
shape: two grid sizes at one series length, and one grid vectorbt cannot hold at
all (it materialises the simulation per combination, measured at 3.93 MB per
combination on 50,000 bars).
Three points, sized from what the runner actually did rather than guessed.
Measured there: 87.5 us per combination for manifoldbt at 20,000 bars, 1.16 ms
for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for raptorbt at 200,000 bars.
Grid ratios are much more sensitive to core count than single backtests are,
because manifoldbt is the only one of the three that fans out across cores:
measured on the same point, x38 on four cores and x146 on twenty. Numbers from
the CI job are four-core numbers, and they are the conservative ones.
Only the first point is one vectorbt can hold. It materialises the simulation
per combination -- 1.57 MB of it at 20,000 bars -- so 5,000 combinations already
cost it 2.5 GB and 20,000 would need 31 GB. The other two are out of its scope,
and that is where a sweep stops being a speed comparison and becomes a
capability one: the question is no longer how long it takes but whether the
machine can hold it at all.
The directory is still named `vs_vectorbt` and the workflow file still
`bench-vs-vectorbt.yml`: renaming either would break the path the public
repository runs and start a fresh, empty run history.
raptorbt sets the time budget rather than manifoldbt. With no fan-out API for a
parameter grid on one instrument, its sweep is a Python loop costing a full
backtest per cell, which is why the large point goes deep in combinations on a
short series instead of the reverse: 20,000 combinations on 20,000 bars costs it
27 seconds a call, where 5,000 combinations on a million bars would cost it 25
minutes.
Sweep ratios depend on the machine far more than single-backtest ratios do,
because manifoldbt is the only one of the three that fans out across cores. The
same point measured x13 on a four-vCPU cloud runner and x32 on two fast desktop
cores; the CI numbers are the conservative ones, and they are the ones published.