mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-08-25 15:08:03 +00:00
Exocet92andGitHub
9ddefc64df
The three points were sized before this runner had ever run one. It has now, so they are sized from what it measured: 87.5 us per combination for manifoldbt at 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for raptorbt at 200,000 bars. 20,000 x 5,000 stays, because it is the only one of the three vectorbt can hold: it materialises 1.57 MB per combination at that length, so 5,000 already costs it 2.5 GB. The other two grow to 20,000 and 10,000 combinations and put it out of scope, which is where a sweep stops being a speed comparison and becomes a capability one. raptorbt sets the budget, not manifoldbt. With no fan-out API its sweep is a Python loop costing a full backtest per cell, so the large point goes deep in combinations on a short series rather than the reverse: 20,000 combinations on 20,000 bars costs it 27 s a call, where 5,000 combinations on a million bars would cost it 25 minutes. Also: sweeps, not grids. `run_sweep`, `run_sweep_lite` and `--sweep` are what the product calls this, and a second word for the same thing is a second thing to learn. `grid` is kept only where it means the parameter space itself.
230 lines
10 KiB
YAML
230 lines
10 KiB
YAML
# The benchmark runs here, in the open, on standard GitHub-hosted runners.
|
|
#
|
|
# Nothing in this workflow has access to the engine source: it installs the
|
|
# published wheel from PyPI, exactly like any user would. That is what makes the
|
|
# result independent rather than self-reported, and it is why anyone can fork
|
|
# this repository and press "Run workflow" to reproduce the numbers on their own
|
|
# runner. Same for the engines it is compared against: they come from PyPI at
|
|
# pinned versions, and the run prints the versions it resolved.
|
|
|
|
name: Benchmark vs vectorbt and raptorbt
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
manifoldbt_version:
|
|
description: "manifoldbt version to install from PyPI (blank = latest)"
|
|
required: false
|
|
default: ""
|
|
reps:
|
|
description: "Interleaved repetitions per point"
|
|
required: false
|
|
# 2, not 7. The budget went into a longer series instead: at 10M bars
|
|
# one vectorbt call costs 99 s, so seven of them would put the job past
|
|
# its timeout on Windows. Two repetitions still bracket the number (min,
|
|
# median and max are all published, and the median of two is their mean)
|
|
# but the noise flag gets cruder, since an interquartile range wants at
|
|
# least four samples to mean anything. A point that looks surprising is
|
|
# worth re-running at a higher `reps` before it is quoted anywhere.
|
|
default: "2"
|
|
release:
|
|
types: [published]
|
|
schedule:
|
|
# Weekly, to catch a slowdown introduced by a dependency rather than by us.
|
|
- cron: "17 5 * * 1"
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: bench-vs-vectorbt-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
bench:
|
|
# Named for what the job measures, not for the runner it landed on. The
|
|
# runner is already on the row; what a reader needs from the job list is
|
|
# which half of the benchmark it is.
|
|
name: backtests (${{ matrix.os }})
|
|
runs-on: ${{ matrix.os }}
|
|
# A published release is not always a released version. The preview tags
|
|
# 0.18.0rc1 and rc2 went to the public repository without ever reaching
|
|
# PyPI, so `pip install manifoldbt==0.18.0rc2` failed and this workflow went
|
|
# red four times in two days for a reason that had nothing to do with any
|
|
# engine. Pre-releases are skipped; a manual dispatch can still benchmark
|
|
# one by naming the version, if it ever exists on PyPI.
|
|
if: github.event_name != 'release' || github.event.release.prerelease == false
|
|
timeout-minutes: 60
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
# Linux only, on purpose. The other two runners were carried for a
|
|
# reason that does not survive examination: they were the only place
|
|
# anything installed the published wheel on Windows or macOS, which
|
|
# made this benchmark a smoke test by accident. That check is worth
|
|
# having and worth 40 seconds next to the build in release.yml, not 13
|
|
# minutes inside a performance measurement, and nobody reads a
|
|
# benchmark to find out whether a package imports.
|
|
#
|
|
# What is lost is a per-platform timing, which was never quoted: the
|
|
# numbers that get published are the Linux ones. Adding a platform
|
|
# back is one entry here.
|
|
- os: ubuntu-latest
|
|
bars: "100000 1000000 10000000"
|
|
|
|
env:
|
|
PYTHONUNBUFFERED: "1"
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
# 3.12, not 3.13, and it is raptorbt that pins it: it is built against
|
|
# pyo3 0.20.3, whose maximum supported CPython is 3.12. No release up
|
|
# to 0.9.0 publishes a cp313 wheel and a source build refuses outright
|
|
# ("the configured Python interpreter version (3.13) is newer than
|
|
# PyO3's maximum supported version (3.12)"). Comparing engines means
|
|
# running them in one environment, and the environment has to be one
|
|
# they all support. Runs before 2026-08-20 used 3.13 with two engines,
|
|
# so their absolute timings are not directly comparable with these.
|
|
python-version: "3.12"
|
|
|
|
- name: Install engines from PyPI
|
|
shell: bash
|
|
run: |
|
|
set -e
|
|
VERSION="${{ inputs.manifoldbt_version }}"
|
|
# A release run benchmarks the version that was just published.
|
|
if [ -z "$VERSION" ] && [ "${{ github.event_name }}" = "release" ]; then
|
|
VERSION="$(echo '${{ github.event.release.tag_name }}' | sed 's/^v//')"
|
|
fi
|
|
python -m pip install --upgrade pip
|
|
if [ -n "$VERSION" ]; then
|
|
pip install "manifoldbt==${VERSION}"
|
|
else
|
|
pip install manifoldbt
|
|
fi
|
|
pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
|
|
|
|
- name: Record the resolved environment
|
|
shell: bash
|
|
run: pip freeze | grep -iE '^(manifoldbt|vectorbt|raptorbt|numpy|numba|pandas|psutil)=' || true
|
|
|
|
- name: Run the benchmark
|
|
shell: bash
|
|
working-directory: benchmarks/vs_vectorbt
|
|
run: |
|
|
python bench.py \
|
|
--bars ${{ matrix.bars }} \
|
|
--reps "${{ inputs.reps || '7' }}" \
|
|
--cold-start-reps 3 \
|
|
--memory-bars 2000000 \
|
|
--out "results-${{ matrix.os }}.json"
|
|
|
|
- name: Render the report
|
|
# Runs even when the benchmark exits non-zero: a parity failure is the
|
|
# most interesting thing that can happen here, and it must be readable
|
|
# in the job summary rather than buried in a red step. But only if there
|
|
# is something to render: when the install step failed, this used to die
|
|
# on a missing file and put a FileNotFoundError on top of the real
|
|
# error, which is how a run reports the wrong cause twice.
|
|
if: always()
|
|
shell: bash
|
|
working-directory: benchmarks/vs_vectorbt
|
|
run: |
|
|
if [ -f "results-${{ matrix.os }}.json" ]; then
|
|
python report.py "results-${{ matrix.os }}.json"
|
|
else
|
|
echo "no result file: the benchmark did not get far enough to write one"
|
|
fi
|
|
|
|
- name: Upload the raw result
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: bench-${{ matrix.os }}
|
|
path: benchmarks/vs_vectorbt/results-*.json
|
|
if-no-files-found: warn
|
|
|
|
# ------------------------------------------------------------------------ #
|
|
# Parameter sweeps, which need a licence and therefore a job of their own
|
|
# ------------------------------------------------------------------------ #
|
|
sweeps:
|
|
name: sweeps (ubuntu-latest)
|
|
runs-on: ubuntu-latest
|
|
if: github.event_name != 'release' || github.event.release.prerelease == false
|
|
timeout-minutes: 45
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Install engines from PyPI
|
|
shell: bash
|
|
run: |
|
|
set -e
|
|
VERSION="${{ inputs.manifoldbt_version }}"
|
|
if [ -z "$VERSION" ] && [ "${{ github.event_name }}" = "release" ]; then
|
|
VERSION="$(echo '${{ github.event.release.tag_name }}' | sed 's/^v//')"
|
|
fi
|
|
python -m pip install --upgrade pip
|
|
if [ -n "$VERSION" ]; then pip install "manifoldbt==${VERSION}"; else pip install manifoldbt; fi
|
|
pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
|
|
|
|
# A sweep benchmark without a licence does not fail, it produces a wrong
|
|
# number: every unlicensed fan-out call waits a fixed interval before any
|
|
# work starts, so the stopwatch would time the wait. This step exits
|
|
# non-zero rather than let that happen, and the harness refuses again on
|
|
# its own if the tier is not what it expects.
|
|
- name: Activate the benchmark licence
|
|
env:
|
|
MANIFOLDBT_CI_LICENSE: ${{ secrets.MANIFOLDBT_CI_LICENSE }}
|
|
shell: bash
|
|
working-directory: benchmarks/vs_vectorbt
|
|
run: python ci_activate.py
|
|
|
|
- name: Run the sweeps
|
|
shell: bash
|
|
working-directory: benchmarks/vs_vectorbt
|
|
# Three points, sized from what this runner actually did rather than
|
|
# guessed. Measured here: 87.5 us per combination for manifoldbt at
|
|
# 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms
|
|
# for raptorbt at 200,000 bars.
|
|
#
|
|
# The first point is the only one vectorbt can hold: at 20,000 bars it
|
|
# materialises 1.57 MB per combination, so 5,000 of them already cost it
|
|
# 2.5 GB and 20,000 would need 31 GB. The other two are out of its
|
|
# scope, and they are where a sweep stops being a speed comparison and
|
|
# becomes a capability one.
|
|
#
|
|
# raptorbt sets the budget, not manifoldbt: with no fan-out API its
|
|
# sweep is a Python loop, so it costs a full backtest per cell. That is
|
|
# why the large point goes deep in combinations on short series rather
|
|
# than the reverse -- 20,000 combinations on 20,000 bars costs it 27 s a
|
|
# call, where 5,000 combinations on a million bars would cost it 25 min.
|
|
run: |
|
|
python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:5000 20000:20000:oos 200000:10000:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json"
|
|
|
|
- name: Render the report
|
|
if: always()
|
|
shell: bash
|
|
working-directory: benchmarks/vs_vectorbt
|
|
run: |
|
|
if [ -f results-sweeps.json ]; then
|
|
python report.py results-sweeps.json
|
|
else
|
|
echo "no result file: the grids did not get far enough to write one"
|
|
fi
|
|
|
|
- name: Upload the raw result
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: bench-sweeps
|
|
path: benchmarks/vs_vectorbt/results-sweeps.json
|
|
if-no-files-found: warn
|