Both sweep benchmarks gain the point a GPU sweep is for. The CPU job runs
it on manifoldbt alone: vectorbt is out of scope as on the other large
points, and raptorbt is declared out of scope too, its loop measured at
14.8 ms a backtest, so about 37 minutes a call. The sweeps job gets 60
minutes instead of 45 for the ten or so this point adds. The reason
printed for raptorbt now names raptorbt: the sentence for an engine with
no grid API was written for NautilusTrader and would have said so.
The CPU job now asks for the CPU path by name. With the engine's default
device ("auto"), a workstation with a GPU and a licence that covers it
ran this job's sweeps on the card: 2.0 s for this point where the CPU
takes tens of seconds. The GitHub runner has no GPU, so no published
number was affected.
Measured: 2.79 s on a Modal L4 and 1.75 s on an RTX 3090, against 42 s on
four CPU threads of the same workstation, every cell bit-identical. The
Modal run grows from about 30 s to about 2.5 minutes, most of it the
untimed CPU parity run of this point.
Correction to 923559d: the local run whose data digests did not match the
CPU job's differed by platform, not by numpy version. Windows produces
other digests than Linux with the same numpy 2.4.3; the report keeps
refusing to pair them.
The CPU reference for a sweep is the CPU job's, published from its
GitHub runner. Timing the CPU again in the Modal container added a second
CPU number from another machine, most of the run's billed time, and a
ratio that measured the container's CPU as much as the card.
The CPU path now runs once per point, untimed, only so that every GPU
cell is still checked against it. The report reads the CPU column from
vs_vectorbt/results/latest.json, matched on bars, combinations and data
digest, and gives a ratio only when both runs measured the same version.
Validated on a Modal L4 with manifoldbt 0.28.1: 32 ms, 107 ms and 223 ms
against 228 ms, 902 ms and 3.53 s in the CPU job's run of the same
version (7.1x, 8.4x, 15.9x), every cell bit-identical, 32 s from start to
finish instead of 120 s. A local run on a different numpy produced other
data digests and was, as intended, not matched with the CPU job.
Modal checks the card a function is declared with when the app starts,
before with_options(gpu=...) applies. A function declared on an L4 and
called on a T4 therefore still failed wherever an L4 could not be had,
which made the workflow's card choice useless. The card is now read from
BENCH_GPU when modal_app.py declares the function, and the workflow sets
that variable instead of passing --gpu.
First runs on a Modal L4 (manifoldbt 0.28.1, CPU side on its 4 reserved
cores): 9.1x, 10.3x and 18.8x on the CPU job's three grids, every cell
bit-identical to the CPU, same data digests as the published CPU run,
two minutes from start to finish. The README now quotes them.
With five repetitions a single slow one (48 ms among 32 ms) flagged the
first point as noisy; seven, as in the CPU benchmark, did not, so seven
is the default. The container saw exactly the 4 reserved cores. Its CPU
model is not readable inside Modal's sandbox: cpu_model now also tries
lscpu and the vendor, family and model fields before recording unknown.
GitHub's hosted runners have no GPU, so the sweeps job only ever measures
the CPU path. A new workflow, bench-gpu.yml, runs the same three grids, on
the same data and the same strategy imported from vs_vectorbt, on a Modal
L4, with the CPU of that machine beside it on 4 reserved cores.
The harness fails a point when the GPU sweep falls back to the CPU (the
engine only warns), when any cell differs from the CPU grid beyond the
gate the CPU job uses, when a repetition does not reproduce the first
call, or when the licence tier changes. It calibrates the card and runs
each point once untimed before timing, and reports that first call apart.
The result is published to benchmarks/gpu/results/latest.json, separate
from the CPU file, only for a green run of the latest version on the L4.
Without a Modal token the job warns and measures nothing.
Checked on a workstation (RTX 3090, PyPI wheel 0.28.1): every cell
bit-identical to the CPU on all three points; with the card hidden, a run
stops at calibration, and a sweep forced onto the CPU raises the warning
the harness catches (the verdict it then gives is covered by the tests);
19 offline tests pass and actionlint is clean. Not yet run on Modal: that
needs the token.
github-actions[bot]