mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-11-02 07:56:11 +00:00
bench: rename the workflow, give the backtests job more time, note the cost
The workflow now carries NautilusTrader, so its name says so. The backtests job goes from a 60 to a 75 minute timeout: it ran 30 minutes with three engines, and NautilusTrader adds about 7 minutes on a fast workstation (6 workloads at 100,000 bars, 8 calls each), which is 11 to 15 on a hosted runner, going by how much slower the runner was than the workstation on the engines both have. The sweeps job keeps its 45 minutes: Nautilus is out of scope on all three points by its own sweep ceiling and costs only its 16-cell anchors there, about 2.5 minutes on the workstation. The sweep command is unchanged. Install adds a 189 MB wheel that unpacks to about 610 MB (measured on Linux with Python 3.12, from a fresh environment built the way the job builds it), and a few seconds to install. The resolved-environment step now lists nautilus_trader.
This commit is contained in:
@@ -7,7 +7,7 @@
|
||||
# runner. Same for the engines it is compared against: they come from PyPI at
|
||||
# pinned versions, and the run prints the versions it resolved.
|
||||
|
||||
name: Benchmark vs vectorbt and raptorbt
|
||||
name: Benchmark vs vectorbt, raptorbt and NautilusTrader
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
@@ -63,7 +63,14 @@ jobs:
|
||||
# engine. Pre-releases are skipped; a manual dispatch can still benchmark
|
||||
# one by naming the version, if it ever exists on PyPI.
|
||||
if: github.event_name != 'release' || github.event.release.prerelease == false
|
||||
timeout-minutes: 60
|
||||
# 75, not 60. The job ran 30 minutes before NautilusTrader joined it. An
|
||||
# event-driven engine replays every bar through a simulated exchange, so it
|
||||
# costs about 7 minutes of this job on a fast workstation (6 workloads at
|
||||
# 100,000 bars, 8 calls each) and, scaled by how much slower this runner is
|
||||
# than that workstation on the engines both have, 11 to 15 on a hosted
|
||||
# runner. The ceiling that keeps it off the longer series is in
|
||||
# engines.py; this is the margin on top of it.
|
||||
timeout-minutes: 75
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -142,11 +149,13 @@ jobs:
|
||||
else
|
||||
pip install manifoldbt
|
||||
fi
|
||||
# The lock adds NautilusTrader: a 189 MB wheel (manylinux_2_35) that
|
||||
# unpacks to about 610 MB, and 6 s to install on a fast connection.
|
||||
pip install -r benchmarks/vs_vectorbt/requirements-lock.txt
|
||||
|
||||
- name: Record the resolved environment
|
||||
shell: bash
|
||||
run: pip freeze | grep -iE '^(manifoldbt|vectorbt|raptorbt|numpy|numba|pandas|psutil|plotly)=' || true
|
||||
run: pip freeze | grep -iE '^(manifoldbt|vectorbt|raptorbt|nautilus[-_]trader|numpy|numba|pandas|psutil|plotly)==' || true
|
||||
|
||||
- name: Run the benchmark
|
||||
shell: bash
|
||||
@@ -271,6 +280,18 @@ jobs:
|
||||
# why the large point goes deep in combinations on short series rather
|
||||
# than the reverse -- 20,000 combinations on 20,000 bars costs it 27 s a
|
||||
# call, where 5,000 combinations on a million bars would cost it 25 min.
|
||||
#
|
||||
# NautilusTrader takes part without being named below. It has no sweep
|
||||
# API either, and its loop is the expensive one: a full event-driven
|
||||
# replay per cell, 0.75 s at 20,000 bars and 7.8 s at 200,000 on a fast
|
||||
# workstation, which makes these grids 1 hour, 5 hours and 22 hours.
|
||||
# Its sweep ceiling (64 combinations, engines.py) puts it out of scope
|
||||
# on all three points by itself, so the recipe cannot send it into them.
|
||||
# What it does cost here is its anchor: 16 cells spread across each grid,
|
||||
# run on both engines and compared cell by cell, which is where the
|
||||
# per-backtest time in its reason comes from. About 2.5 minutes on that
|
||||
# workstation, most of it the 200,000-bar point; budget 4 to 5 on this
|
||||
# runner.
|
||||
run: |
|
||||
python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 --sweep 20000:5000 20000:20000:oos 200000:10000:oos --sweep-reps "${{ inputs.reps || '2' }}" --out "results-sweeps.json"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user