The site joins the GPU run with the CPU run on the digest of their data, and
that digest depended on the processor. The walk was exp(cumsum(log returns)),
and numpy's exp rounds differently on an AVX-512 CPU, without AVX-512 and on
Windows: one seed, three datasets. The CPU run of 2026-10-25 landed on an
EPYC 9V45 (Zen 5) and the GPU run of 2026-10-26 on a Modal host without
AVX-512, so their points no longer matched and the ratios dropped out.
The walk is now 100 * cumprod(1 + r): additions and multiplications only,
rounded exactly whatever the CPU. Checked on numpy 2.4.3: the normal and
uniform draws and the new close are the same bits on Linux and Windows, where
exp was not. test_data.py pins it (the walk equals a plain Python product of
the draws, and the digest of the default dataset is fixed), and the
bench-report CI job now runs it.
The GPU report also names the runner's processor beside the ratio, since
GitHub hands out different generations and the ratio moves with them: the
same L4 times read x17.7 against an EPYC 9V45 and x36.3 against an EPYC 7763.
The dataset changes, so digests recorded before this commit will not match
runs made after it.
Both sweep benchmarks gain the point a GPU sweep is for. The CPU job runs
it on manifoldbt alone: vectorbt is out of scope as on the other large
points, and raptorbt is declared out of scope too, its loop measured at
14.8 ms a backtest, so about 37 minutes a call. The sweeps job gets 60
minutes instead of 45 for the ten or so this point adds. The reason
printed for raptorbt now names raptorbt: the sentence for an engine with
no grid API was written for NautilusTrader and would have said so.
The CPU job now asks for the CPU path by name. With the engine's default
device ("auto"), a workstation with a GPU and a licence that covers it
ran this job's sweeps on the card: 2.0 s for this point where the CPU
takes tens of seconds. The GitHub runner has no GPU, so no published
number was affected.
Measured: 2.79 s on a Modal L4 and 1.75 s on an RTX 3090, against 42 s on
four CPU threads of the same workstation, every cell bit-identical. The
Modal run grows from about 30 s to about 2.5 minutes, most of it the
untimed CPU parity run of this point.
Correction to 923559d: the local run whose data digests did not match the
CPU job's differed by platform, not by numpy version. Windows produces
other digests than Linux with the same numpy 2.4.3; the report keeps
refusing to pair them.
Modal checks the card a function is declared with when the app starts,
before with_options(gpu=...) applies. A function declared on an L4 and
called on a T4 therefore still failed wherever an L4 could not be had,
which made the workflow's card choice useless. The card is now read from
BENCH_GPU when modal_app.py declares the function, and the workflow sets
that variable instead of passing --gpu.
First runs on a Modal L4 (manifoldbt 0.28.1, CPU side on its 4 reserved
cores): 9.1x, 10.3x and 18.8x on the CPU job's three grids, every cell
bit-identical to the CPU, same data digests as the published CPU run,
two minutes from start to finish. The README now quotes them.
With five repetitions a single slow one (48 ms among 32 ms) flagged the
first point as noisy; seven, as in the CPU benchmark, did not, so seven
is the default. The container saw exactly the 4 reserved cores. Its CPU
model is not readable inside Modal's sandbox: cpu_model now also tries
lscpu and the vendor, family and model fields before recording unknown.
GitHub's hosted runners have no GPU, so the sweeps job only ever measures
the CPU path. A new workflow, bench-gpu.yml, runs the same three grids, on
the same data and the same strategy imported from vs_vectorbt, on a Modal
L4, with the CPU of that machine beside it on 4 reserved cores.
The harness fails a point when the GPU sweep falls back to the CPU (the
engine only warns), when any cell differs from the CPU grid beyond the
gate the CPU job uses, when a repetition does not reproduce the first
call, or when the licence tier changes. It calibrates the card and runs
each point once untimed before timing, and reports that first call apart.
The result is published to benchmarks/gpu/results/latest.json, separate
from the CPU file, only for a green run of the latest version on the L4.
Without a Modal token the job warns and measures nothing.
Checked on a workstation (RTX 3090, PyPI wheel 0.28.1): every cell
bit-identical to the CPU on all three points; with the card hidden, a run
stops at calibration, and a sweep forced onto the CPU raises the warning
the harness catches (the verdict it then gives is covered by the tests);
19 offline tests pass and actionlint is clean. Not yet run on Modal: that
needs the token.
NautilusTrader 1.231.0, the latest stable release, asks for exactly
fsspec 2026.2.0, inside the range of GHSA-27vj-qcqg-25rc (fixed in
2026.6.0). Every 1.229-1.231 release has the same exact pin, so no
stable Nautilus accepts a patched fsspec.
Both benchmarks now pin fsspec 2026.9.0 in requirements-lock.txt and
install Nautilus from its own requirements-nautilus.txt with --no-deps,
after the lock, which pins each of its runtime dependencies (uvloop was
missing from the HFT lock and is now pinned).
Checked on Linux, Python 3.12, fresh environments: pip check reports
only the fsspec override, every engine imports, and a 10,000-bar run of
the bar benchmark matches the published comparisons (exact on five
workloads, documented on the bracket one).
One pull request for both benchmark changes: NautilusTrader as a fourth
engine in the bar benchmark (was #18), and the market-making and HFT
benchmark.
The first run with NautilusTrader on a GitHub-hosted runner (4 vCPU, dispatched
with manifoldbt 0.28.0 pinned so that it measures without publishing; run
39945358568) took 23 min 57 s end to end, against 16 min 8 s for the last run
without it.
The benchmark step ran 21 min 58 s against 15 min 4 s: 6 min 54 s for
NautilusTrader at the 3 repetitions of a dispatch. The medians of that run put
the 8 calls of a release or weekly run at about 12 to 13 minutes, which is the
11 to 15 the comment already gave. The sweeps step ran 23 min 1 s against
14 min 23 s: 8 min 38 s, where 4 to 5 had been budgeted, because a runner takes
about twice the time per backtest that the workstation did (1.45 to 1.63 s at
20,000 bars, 15.3 s at 200,000). Both stay far inside their timeouts.
Parity on the runner: five workloads exact against manifoldbt, the bracket
workload documented, and the 16-cell cross-engine check of each sweep point exact.
The workflow comments and the README carry the measurements instead of the
estimates.
The latency family gave `execution_grid` two axes, `latency.order` and `latency.cancel`, with the four delays on each. The engine sweeps execution axes as a product, so it ran 16 latency pairs instead of 4: 64 results for the 16 combinations counted, and a combinations-per-second figure that divided the announced count by the time of four times the work. hftbacktest and NautilusTrader sweep the delay tied (the cancel takes the order delay).
The latency family now passes one axis, `latency`, whose value is the whole latency model with the order and cancel delays equal and the response and feed delays unchanged. The durations go in as nanoseconds: the wheel converts an `Interval` only when it is the whole value of an axis, not inside a dictionary.
sweep_mbt.py and sweep.py now refuse to report a call whose number of results differs from the number of combinations announced, for every family. A full sweep must also carry the announced combinations one for one, read from the parameters and the latency model of each result. The identity sample, which only ran for the maker family, now runs for the latency family too (and the workflow asks for 8), and finds each combination in the sweep by those labels instead of by position.
Checked on 2025-08-13, 10:00-12:00 UTC, 4 workers pinned to 4 cpus, manifoldbt 0.28.0: 16 results for 16 combinations, each result carries the pair it was asked for, the 16 journals are bit for bit the order == cancel results of the former 64, and each combination equals the run alone. The maker (4 to 256) and taker (12) families already returned as many results as combinations.
The workflow now carries NautilusTrader, so its name says so. The
backtests job goes from a 60 to a 75 minute timeout: it ran 30 minutes
with three engines, and NautilusTrader adds about 7 minutes on a fast
workstation (6 workloads at 100,000 bars, 8 calls each), which is 11 to
15 on a hosted runner, going by how much slower the runner was than the
workstation on the engines both have. The sweeps job keeps its 45 minutes:
Nautilus is out of scope on all three points by its own sweep ceiling and
costs only its 16-cell anchors there, about 2.5 minutes on the
workstation. The sweep command is unchanged.
Install adds a 189 MB wheel that unpacks to about 610 MB (measured on
Linux with Python 3.12, from a fresh environment built the way the job
builds it), and a few seconds to install. The resolved-environment step
now lists nautilus_trader.
A workflow of its own, not a job of the vectorbt one. This benchmark downloads
gigabytes from two third parties and runs for most of an hour, and an outage at
one of those archives must not turn the other benchmark red, nor delay the
results file the website reads from it.
Three measuring jobs run in parallel on three runners, so that no measurement is
disturbed by another: the books (25 levels and best bid/ask, plus the probe and
the taker), the order-by-order day (which also runs the exact-replay judgment
on the journals of the run), and the sweeps. The prepared data and the ITCH
extract are cached by the hash of the code that prepares them, so a run only
downloads what changed. Each job installs manifoldbt from PyPI, waiting for the
index to serve a version that was just released, and activates the licence from
the MANIFOLDBT_CI_LICENSE secret that the vectorbt benchmark already uses.
A fourth job publishes benchmarks/vs_hft/results/latest.json, with the same
rules as the other one: only when all three measuring jobs are green, never for a
dispatch that pins a manifoldbt version, committed with [skip ci], with the
run's start time resolved through the API.
It runs on a published release (not a pre-release), weekly on Wednesday so that
it never competes with the vectorbt benchmark for runners, and on demand.
publish.py records GITHUB_RUN_STARTED_AT, a variable Actions never sets,
so every published result had an empty run_started_at. The publish job
now asks `gh run view` for the run's startedAt, with actions: read to be
allowed to, and passes it through `date` so that only a date can reach
$GITHUB_ENV. A failed lookup leaves the field empty, as before.
The sweep harness and ci_activate.py read the tier with the public
mbt.license_info(), and the parity-only check uses the published
Community limit of 256 combinations per sweep. Checked locally: a
licensed run times the grid, an unlicensed one is skipped, and
parity-only still runs below the limit and is skipped above it.
plotly 7.0.0 landed on PyPI and removed the mapbox trace types that
vectorbt 0.28.4's template still references, so 'import vectorbt' died
before a single bar ran. plotly was the one transitive dependency the
lock did not cover; it does now, on 6.9.0, the last 6.x line. The
version printout includes it so the results stay comparable.
The root testpaths points at python/tests, so nothing collected
benchmarks/vs_vectorbt/test_report.py. A light job (pytest alone, no
wheel) runs them on every push and pull request.
Authored upstream; the release sync will keep carrying this copy.
PyPI accepte un televersement bien avant que son index le serve. Mesure sur la
publication de 0.18.0: le televersement s'est termine a 01:48:18 et ce workflow
a demande la version a 01:48:50, trente-deux secondes plus tard, pour se faire
repondre "from versions: ..., 0.17.3".
Ce n'est pas un hasard de timing. Le workflow est declenche par
`release: published`, et cette release est creee juste apres le televersement:
la course est donc GARANTIE a chaque publication. Elle n'avait jamais pu se voir
parce que ce chemin n'avait jamais tourne sur une vraie release, seulement sur
des declenchements manuels ou la version existait depuis longtemps.
Les deux jobs attendent desormais que l'index serve la version, jusqu'a dix
minutes, contre soixante de budget. `--no-cache-dir` parce que pip met en cache
la reponse de l'index, y compris celle qui ne connait pas encore la version: sans
cela les tentatives suivantes reliraient la meme reponse perimee.
Logique verifiee sur quatre cas avant de pousser (disponible tout de suite,
disponible apres trois echecs, jamais disponible donc echec franc, et pas de
version imposee), plutot que de la decouvrir a la prochaine release.
An artifact is not a publication. It needs a token to download, expires
after ninety days, and nothing outside GitHub can link to it, so a number
that only lives in an artifact is a number nobody can check.
A third job merges a green run's two payloads into
benchmarks/vs_vectorbt/results/latest.json and commits it. The two are
stored side by side rather than folded into one table: they run on two
runners, and timings from two machines are not rows of the same table.
Only a run where both measuring jobs came back green is published, and a
dispatch that pins an old version measures and reports without becoming
the published number.
`cancel-in-progress: true` is right for a CI triggered by pushes, where a newer
commit makes an older run pointless. It is wrong here. Three times in one
afternoon a second dispatch killed a run that was mid-measurement, once eight
minutes in, and each left a cancelled entry in a run list whose whole job is to
be readable by someone checking the numbers.
The fix people reach for then is deleting runs, which breaks the run_url the
published data links to, and with it the only reason to believe the numbers.
A manual run is never superseded: nobody dispatches a benchmark to invalidate
the one already running. A release or the weekly cron still supersedes, because
there an older run really is measuring a version nobody asks about any more.
Two gaps a reader could name without running anything: costs appeared on one
workload out of four, and nothing in the suite was a portfolio.
Costs could not simply be switched on across the board, and the reason is
measured. On FractionOfEquity sizing a 5 bps fee puts the engines 1.3e-4 of
capital apart and 2 bps of slippage 2.1e-5, against a 1e-9 tolerance, while the
round-trip counts stay identical: the trading agrees, the cost arithmetic does
not, because one charges the fee on top of the notional and the other reserves
it out of cash first. In fixed units both land exactly, to 1e-15. So
`sma_cross_costs` carries a fee and slippage on the headline signal, sized in
units, and the price of that is visible rather than hidden: x48.0 against x50.6
at 100k bars.
`multi_asset` runs five independent series in one shared book. It is the
workload manifoldbt does worst on, and it is here for that reason: going from
one asset to five costs it 6.1x and vectorbt 1.4x, so the ratio falls from x36.7
to x8.8 at a million bars. Broadcasting a column per asset is close to free;
walking five books is not. A portfolio is also what people actually run, and a
suite that only measures where it wins is not evidence.
Both are capped where a materialised five-column simulation would stop measuring
the engine and start measuring the swap file, and `ema_rsi_fees` keeps the
ceiling it got for going bankrupt.
Repetitions go from two to three: the floor at which a median is a median rather
than the mean of two.
The three points were sized before this runner had ever run one. It has now,
so they are sized from what it measured: 87.5 us per combination for manifoldbt
at 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for
raptorbt at 200,000 bars.
20,000 x 5,000 stays, because it is the only one of the three vectorbt can hold:
it materialises 1.57 MB per combination at that length, so 5,000 already costs
it 2.5 GB. The other two grow to 20,000 and 10,000 combinations and put it out
of scope, which is where a sweep stops being a speed comparison and becomes a
capability one.
raptorbt sets the budget, not manifoldbt. With no fan-out API its sweep is a
Python loop costing a full backtest per cell, so the large point goes deep in
combinations on a short series rather than the reverse: 20,000 combinations on
20,000 bars costs it 27 s a call, where 5,000 combinations on a million bars
would cost it 25 minutes.
Also: sweeps, not grids. `run_sweep`, `run_sweep_lite` and `--sweep` are what
the product calls this, and a second word for the same thing is a second thing
to learn. `grid` is kept only where it means the parameter space itself.
Windows and macOS were carried on an argument that does not survive
examination. They were the only place in the whole chain that installed the
published wheel and ran it, which made this benchmark an install smoke test by
accident. That check is worth having, and worth forty seconds next to the build
in release.yml rather than thirteen minutes inside a performance measurement:
nobody reads a benchmark to find out whether a package imports. release.yml
already builds on all four targets, it just never executes what it built.
What is lost is a per-platform timing that was never quoted; the numbers that
get published are the Linux ones. Adding a platform back is one matrix entry.
The jobs are also named for what they measure rather than for the runner they
landed on, which the row already says: `backtests (ubuntu-latest)` and `grids
(ubuntu-latest)` instead of a raw label next to a hand-written one.
Five consecutive red runs, two unrelated causes.
Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.
The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.
Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.
Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.
The harness compared two engines everywhere; it now compares N against a
reference. manifoldbt is the reference: every parity check and every ratio is
a challenger against it, never two challengers against each other.
raptorbt 0.9.0 joins on three of the four workloads. Its sma_cross comes back
bit-identical to the reference's final equity, and its rsi matches to the last
bit; its ema seeds on a different warmup and it has no fixed-quantity sizing,
so the fee workload records it as unsupported with the reason rather than
leaving a blank cell. On the bracket it diverges in its own documented way: it
never re-arms while the entry level holds, so it books exactly the reference's
round-trips minus the ones that re-enter on the exit bar.
Python moves to 3.12, which raptorbt pins rather than we do: it is built
against pyo3 0.20.3, whose maximum supported CPython is 3.12. Timings from runs
before this change are therefore not directly comparable.
The bar matrix gains 10M and the repetition default drops from 7 to 2. Measured,
those two almost cancel: the job stays around 16 minutes. macOS keeps its old
ceiling, since 10M bars adds 1.55 GB on vectorbt's side alone and that runner
has 7 GB.
A speed claim a reader cannot reproduce is a screenshot. This harness
installs manifoldbt from PyPI like any user would, generates its own data,
and gates every timing behind a parity check: a workload where the two
engines disagree publishes nothing and fails the run.
It lives here rather than in the engine repository because it benchmarks the
published wheel, not the source. Anyone can fork this repository and press
"Run workflow" to get the same table on their own runner.
The workflow runs on demand, weekly, and on every published release, so a
version that gets slower says so in public.
github-actions[bot]