Since 0.29.0, license_info() reports "Researcher" for a Researcher licence
instead of "Pro". The sweep jobs only accepted "Pro", so on the 0.29.0 release
both the CPU sweeps job and the GPU job stopped at the activation step and
nothing was published.
ci_activate.py and sweep_child.tier() now accept any paid tier. The "pro" key
in the published results keeps its name and shape.
Checked with manifoldbt 0.29.0 and a Researcher licence: license_info() returns
"Researcher", ci_activate.py reports a licensed tier, tier() returns
{"pro": true}; benchmarks/gpu tests (21) and vs_vectorbt tests (36) pass.
Both sweep benchmarks gain the point a GPU sweep is for. The CPU job runs
it on manifoldbt alone: vectorbt is out of scope as on the other large
points, and raptorbt is declared out of scope too, its loop measured at
14.8 ms a backtest, so about 37 minutes a call. The sweeps job gets 60
minutes instead of 45 for the ten or so this point adds. The reason
printed for raptorbt now names raptorbt: the sentence for an engine with
no grid API was written for NautilusTrader and would have said so.
The CPU job now asks for the CPU path by name. With the engine's default
device ("auto"), a workstation with a GPU and a licence that covers it
ran this job's sweeps on the card: 2.0 s for this point where the CPU
takes tens of seconds. The GitHub runner has no GPU, so no published
number was affected.
Measured: 2.79 s on a Modal L4 and 1.75 s on an RTX 3090, against 42 s on
four CPU threads of the same workstation, every cell bit-identical. The
Modal run grows from about 30 s to about 2.5 minutes, most of it the
untimed CPU parity run of this point.
Correction to 923559d: the local run whose data digests did not match the
CPU job's differed by platform, not by numpy version. Windows produces
other digests than Linux with the same numpy 2.4.3; the report keeps
refusing to pair them.
The CPU reference for a sweep is the CPU job's, published from its
GitHub runner. Timing the CPU again in the Modal container added a second
CPU number from another machine, most of the run's billed time, and a
ratio that measured the container's CPU as much as the card.
The CPU path now runs once per point, untimed, only so that every GPU
cell is still checked against it. The report reads the CPU column from
vs_vectorbt/results/latest.json, matched on bars, combinations and data
digest, and gives a ratio only when both runs measured the same version.
Validated on a Modal L4 with manifoldbt 0.28.1: 32 ms, 107 ms and 223 ms
against 228 ms, 902 ms and 3.53 s in the CPU job's run of the same
version (7.1x, 8.4x, 15.9x), every cell bit-identical, 32 s from start to
finish instead of 120 s. A local run on a different numpy produced other
data digests and was, as intended, not matched with the CPU job.
Modal checks the card a function is declared with when the app starts,
before with_options(gpu=...) applies. A function declared on an L4 and
called on a T4 therefore still failed wherever an L4 could not be had,
which made the workflow's card choice useless. The card is now read from
BENCH_GPU when modal_app.py declares the function, and the workflow sets
that variable instead of passing --gpu.
First runs on a Modal L4 (manifoldbt 0.28.1, CPU side on its 4 reserved
cores): 9.1x, 10.3x and 18.8x on the CPU job's three grids, every cell
bit-identical to the CPU, same data digests as the published CPU run,
two minutes from start to finish. The README now quotes them.
With five repetitions a single slow one (48 ms among 32 ms) flagged the
first point as noisy; seven, as in the CPU benchmark, did not, so seven
is the default. The container saw exactly the 4 reserved cores. Its CPU
model is not readable inside Modal's sandbox: cpu_model now also tries
lscpu and the vendor, family and model fields before recording unknown.
GitHub's hosted runners have no GPU, so the sweeps job only ever measures
the CPU path. A new workflow, bench-gpu.yml, runs the same three grids, on
the same data and the same strategy imported from vs_vectorbt, on a Modal
L4, with the CPU of that machine beside it on 4 reserved cores.
The harness fails a point when the GPU sweep falls back to the CPU (the
engine only warns), when any cell differs from the CPU grid beyond the
gate the CPU job uses, when a repetition does not reproduce the first
call, or when the licence tier changes. It calibrates the card and runs
each point once untimed before timing, and reports that first call apart.
The result is published to benchmarks/gpu/results/latest.json, separate
from the CPU file, only for a green run of the latest version on the L4.
Without a Modal token the job warns and measures nothing.
Checked on a workstation (RTX 3090, PyPI wheel 0.28.1): every cell
bit-identical to the CPU on all three points; with the card hidden, a run
stops at calibration, and a sweep forced onto the CPU raises the warning
the harness catches (the verdict it then gives is covered by the tests);
19 offline tests pass and actionlint is clean. Not yet run on Modal: that
needs the token.
run_mbt now passes device and max_parallelism through to run_sweep_lite,
so the GPU benchmark can time this exact sweep on a card and on a fixed
number of CPU threads. Both default to the engine's own defaults, so the
call the sweeps job times is unchanged.
NautilusTrader 1.231.0, the latest stable release, asks for exactly
fsspec 2026.2.0, inside the range of GHSA-27vj-qcqg-25rc (fixed in
2026.6.0). Every 1.229-1.231 release has the same exact pin, so no
stable Nautilus accepts a patched fsspec.
Both benchmarks now pin fsspec 2026.9.0 in requirements-lock.txt and
install Nautilus from its own requirements-nautilus.txt with --no-deps,
after the lock, which pins each of its runtime dependencies (uvloop was
missing from the HFT lock and is now pinned).
Checked on Linux, Python 3.12, fresh environments: pip check reports
only the fsspec override, every engine imports, and a 10,000-bar run of
the bar benchmark matches the published comparisons (exact on five
workloads, documented on the bracket one).
Reads back the quotes, trades and book deltas the adapter builds and checks
they hold the prices and sizes they were built from, on whichever build of
Nautilus is installed. On the Linux wheel the previous adapter reads a price of
120101.6 as 0.01201016.
The adapter scaled raw prices and quantities by 10 ** (9 - precision). That is
the scale of the standard wheel (Windows). The Linux wheel is built in high
precision and its raw integers carry 16 decimals, so every price and size
reached the engine 10 ** 7 times too small. Nothing failed while the data was
read: the books looked empty, the maker quoted twice and never traded, and the
flow taker's market orders filled for a quantity that rounds to nothing, which
opens a flat position and trips the assertion in PositionOpened (the failure of
the first run on a runner).
The scale is now read from nautilus_trader (FIXED_PRECISION) and checked before
a run. The raw values are built as Python ints, because a 16-decimal raw price
does not fit in an int64. The taker's fill price is rounded to its tick, since
Price.as_double() is one unit in the last place off on some prices in high
precision.
Checked on the pinned day, with the same lock on Windows and on Ubuntu 24.04
(WSL): the journals of the six workloads are identical bit for bit between the
unchanged adapter on Windows, the new one on Windows and the new one on Linux,
and so are the parity counts against manifoldbt and the 14 combinations of the
maker, latency and taker sweeps run through the adapter.
One pull request for both benchmark changes: NautilusTrader as a fourth
engine in the bar benchmark (was #18), and the market-making and HFT
benchmark.
The README now says that the latency family sweeps one axis carrying the whole latency model, so that the order and cancel delays are the same value in every combination in the three engines, and that two separate axes would be swept as a product. It also says what sweep_mbt.py refuses: a call whose number of results differs from the number of combinations announced, and a full sweep whose results are not the announced combinations.
The first run with NautilusTrader on a GitHub-hosted runner (4 vCPU, dispatched
with manifoldbt 0.28.0 pinned so that it measures without publishing; run
39945358568) took 23 min 57 s end to end, against 16 min 8 s for the last run
without it.
The benchmark step ran 21 min 58 s against 15 min 4 s: 6 min 54 s for
NautilusTrader at the 3 repetitions of a dispatch. The medians of that run put
the 8 calls of a release or weekly run at about 12 to 13 minutes, which is the
11 to 15 the comment already gave. The sweeps step ran 23 min 1 s against
14 min 23 s: 8 min 38 s, where 4 to 5 had been budgeted, because a runner takes
about twice the time per backtest that the workstation did (1.45 to 1.63 s at
20,000 bars, 15.3 s at 200,000). Both stay far inside their timeouts.
Parity on the runner: five workloads exact against manifoldbt, the bracket
workload documented, and the 16-cell cross-engine check of each sweep point exact.
The workflow comments and the README carry the measurements instead of the
estimates.
The latency family gave `execution_grid` two axes, `latency.order` and `latency.cancel`, with the four delays on each. The engine sweeps execution axes as a product, so it ran 16 latency pairs instead of 4: 64 results for the 16 combinations counted, and a combinations-per-second figure that divided the announced count by the time of four times the work. hftbacktest and NautilusTrader sweep the delay tied (the cancel takes the order delay).
The latency family now passes one axis, `latency`, whose value is the whole latency model with the order and cancel delays equal and the response and feed delays unchanged. The durations go in as nanoseconds: the wheel converts an `Interval` only when it is the whole value of an axis, not inside a dictionary.
sweep_mbt.py and sweep.py now refuse to report a call whose number of results differs from the number of combinations announced, for every family. A full sweep must also carry the announced combinations one for one, read from the parameters and the latency model of each result. The identity sample, which only ran for the maker family, now runs for the latency family too (and the workflow asks for 8), and finds each combination in the sweep by those labels instead of by position.
Checked on 2025-08-13, 10:00-12:00 UTC, 4 workers pinned to 4 cpus, manifoldbt 0.28.0: 16 results for 16 combinations, each result carries the pair it was asked for, the 16 journals are bit for bit the order == cancel results of the former 64, and each combination equals the run alone. The maker (4 to 256) and taker (12) families already returned as many results as combinations.
Of the 16 quotes only manifoldbt serves on the probe, 15 are posted less than 100 ms before the print that crosses their level and are cancelled before the next book snapshot shows the crossing. The README now says so, instead of only that one engine reads the tape and the other the book.
Adds the engine to the workload table and a section on what is different
about it: what is timed, why it stops at 100,000 bars, how each workload
is expressed and why (fill at the close, no volume-limited fills, slippage
by shifting the book, an 8-decimal quote currency), the fixed-point
rounding, the RSI seed, the documented bracket_sl_tp divergence, the
missing drawdown, and cold start and memory. The sweeps section says why
a loop of full replays is out of scope on the three published points and
what it costs on a grid it can hold.
Every figure in it is a measurement from a local run pinned to four
logical cores, or from a check run for the purpose: the USD-quote
rounding, the BestPriceFillModel result on the bracket workload, the RSI
seed gap, the wheel size. The module docstring of engine_nt.py carries the
same two measurements.
The size of the book ahead of an order is not capped by the displayed size in NautilusTrader, where the other two engines bring it down when the displayed size shrinks. That is a statement about the queue, not about the displayed size serving the order.
The workflow now carries NautilusTrader, so its name says so. The
backtests job goes from a 60 to a 75 minute timeout: it ran 30 minutes
with three engines, and NautilusTrader adds about 7 minutes on a fast
workstation (6 workloads at 100,000 bars, 8 calls each), which is 11 to
15 on a hosted runner, going by how much slower the runner was than the
workstation on the engines both have. The sweeps job keeps its 45 minutes:
Nautilus is out of scope on all three points by its own sweep ceiling and
costs only its 16-cell anchors there, about 2.5 minutes on the
workstation. The sweep command is unchanged.
Install adds a 189 MB wheel that unpacks to about 610 MB (measured on
Linux with Python 3.12, from a fresh environment built the way the job
builds it), and a few seconds to install. The resolved-environment step
now lists nautilus_trader.
The introduction and the docstring of bench.py said that the engines are checked for doing the same work and only then timed. The harness reports how far they did the same work next to every timing and does not hold a timing back, as the previous commit says in the README. The wording now matches.
The report assumed every engine ran on every row. With a ceiling that is
no longer true, so:
- "Not run, and why" prints one reason once for all the workloads it
covers, instead of once per workload.
- The CPU-over-wall row reads each engine at the longest series it ran,
where it used to need every engine at one size and would have vanished.
- A memory figure taken at an engine's ceiling carries its bar count.
- An engine with no maximum drawdown is said to have none, rather than
"it was not compared", and its summary workload is described as gated on
total return and round-trips.
- A documented divergence where an engine takes the same trades and sizes
one differently reports how many re-entries and how far apart.
- Probes that died for a challenger are listed under the table.
test_harness.py adds guards for the gate (an engine without a drawdown,
the documented bracket divergence), the ceilings, the sweep scope rules
(including that a recipe saying only `:oos` cannot start Nautilus on a
big grid), the out-of-scope reasons and these report changes. Like
test_report.py it needs no engine installed.
The README described exact / documented / failed statuses that withhold a timing, as the vectorbt benchmark does. This benchmark does not have that gate: it prints the counts of the three parity layers beside every timing, and the differences between the engines are the conventions listed in "Why the engines differ". What the run enforces is that every engine finished every workload.
Also: the taker's identical decisions are a measurement on the pinned day, not a check, and the sweep samples are recorded in the result for comparison, not compared by the harness.
The workflow runs probe_l2_nominal beside probe_l2_zero: the same open-loop probe with the venue's 5 / 5 / 5 / 2 ms latencies. The README only named the first, and the sentence about instruments spoke of one.
A workflow of its own, not a job of the vectorbt one. This benchmark downloads
gigabytes from two third parties and runs for most of an hour, and an outage at
one of those archives must not turn the other benchmark red, nor delay the
results file the website reads from it.
Three measuring jobs run in parallel on three runners, so that no measurement is
disturbed by another: the books (25 levels and best bid/ask, plus the probe and
the taker), the order-by-order day (which also runs the exact-replay judgment
on the journals of the run), and the sweeps. The prepared data and the ITCH
extract are cached by the hash of the code that prepares them, so a run only
downloads what changed. Each job installs manifoldbt from PyPI, waiting for the
index to serve a version that was just released, and activates the licence from
the MANIFOLDBT_CI_LICENSE secret that the vectorbt benchmark already uses.
A fourth job publishes benchmarks/vs_hft/results/latest.json, with the same
rules as the other one: only when all three measuring jobs are green, never for a
dispatch that pins a manifoldbt version, committed with [skip ci], with the
run's start time resolved through the API.
It runs on a published release (not a pre-release), weekly on Wednesday so that
it never competes with the vectorbt benchmark for runners, and on demand.
New directory benchmarks/vs_hft. The rules are written once (spec.py) and each
engine implements them in its own idiom: a two-sided maker on 25 book levels and
on best bid/ask, a flow taker on the trade clock, a maker that joins the best on
an order-by-order feed, and an open-loop probe whose quotes depend on the market
only, so that each one gets a verdict that does not depend on what the engine did
before.
The data is public and pinned. One Bybit spot BTCUSDT day (book and trades) and
the Nasdaq TotalView-ITCH half session of 28 November 2025 (one symbol), each
archive checked against a SHA-256 before anything runs.
Parity comes before timing, and it is not read from fill counts: an engine that
serves an order in pieces shows more fills for the same quantity. It is read as
actions (which quotes were sent), verdicts (served or not, for how much, how
late) and totals, with manifoldbt as the reference and the same exact /
documented / failed statuses as the vectorbt benchmark. On the order-by-order day
every engine is also judged against an exact replay of the raw messages
(fifo_truth.py), so none of them has to be trusted as the reference. The README
has a section, "Why the engines differ", that names each convention behind a
difference in the results and gives its figure for each engine.
Parameter sweeps are included: manifoldbt through run_sweep and run_sweep_lite,
hftbacktest and NautilusTrader as the loop and the pool of processes a user would
write. The result of a pool is checked against the loop, and a sample of sweep
combinations against the same combination run alone.
journal_of keeps the instant of a first fill as an integer. A nanosecond epoch in a
float64 is only exact to 256 ns, and the exact-replay judgment compares those
instants with the order of the raw messages, so the column never goes through a
float. test_journal.py checks it to the nanosecond.
The lock file pins the engines and their dependencies; every pinned version has a
manylinux wheel for CPython 3.12, and manifoldbt resolves together with it.
An engine that replays every bar through an exchange simulator cannot be
repeated on the lengths the others run at. bench.py now takes a per-engine
ceiling from the registry: an engine past it is not run on that row, and
the row says why in the field the website already prints under "Not run,
and why" (`unsupported`), plus a structured `beyond_ceiling`.
--engine-max-bars lifts it for a local run, and the ceilings used are
recorded in the environment block of the result.
Sweeps follow the same idea. An engine with no sweep API is looped over
full backtests, so it has a ceiling on grid size too (--engine-max-combos).
A point can name the engines it leaves out (`:oos=vectorbt,nautilus_trader`,
bare `:oos` still means vectorbt), and an engine over its sweep ceiling is
left out on its own, so the recipe the website rebuilds from a result
(which only knows `:oos`) cannot send it into hours of work. The reason
carries the per-backtest time measured on 16 cells spread across that
point's own grid, and the same 16 cells are compared with the reference,
so a point still carries a cross-engine check of the same code path.
Memory is measured at the engine's ceiling and the record keeps the bar
count it was measured at. A cold-start or memory probe that dies for a
challenger now costs that engine's cell and is recorded, instead of
raising out of the run after everything else was measured.
sweep_child.py gains the Nautilus loop (one engine, data loaded once,
reset() and run() per cell, a fresh pair of moving averages each time)
and replaces --vectorbt oos by --out-of-scope ENGINE...
publish.py records GITHUB_RUN_STARTED_AT, a variable Actions never sets,
so every published result had an empty run_started_at. The publish job
now asks `gh run view` for the run's startedAt, with actions: read to be
allowed to, and passes it through `date` so that only a date can reach
$GITHUB_ENV. A failed lookup leaves the field empty, as before.
NautilusTrader is an event-driven engine: it replays bars through a
simulated exchange and a Python strategy, where the three engines already
in the benchmark are whole-series loops. engine_nt.py expresses the six
workloads the way a Nautilus user writes them (low-level BacktestEngine,
Strategy subclass, the engine's own SMA/EMA/RSI, market orders, native
bracket orders) and the parity gate decides what gets published.
Five workloads come back exact against manifoldbt: sma_cross,
sma_cross_metrics, sma_cross_costs, multi_asset and ema_rsi_fees, with
final equity within 6e-11 of capital (1e-9 is the gate), identical
round-trips and fees. The same on Windows (9-decimal fixed point) and on
Linux (16-decimal wheel), because the adapter uses 9 on both.
bracket_sl_tp is documented for this engine, with its cause: entries,
exits and fill prices match round-trip for round-trip, including the
re-entry on the exit bar, but the re-entry is sized from the cash after
the exit rather than from the equity at the close. diagnose() measures
the affected re-entries and their quantity gaps.
Choices that keep the work equal, each explained in the module docstring:
fills at the close of the signal bar, no volume-limited fills (a fill
model with deep liquidity, because Nautilus caps a market order at a
quarter of the bar's volume), slippage in bps by shifting the book, an
8-decimal quote currency so realised PnL is not rounded to cents.
Nautilus has no drawdown statistic, so sma_cross_metrics is gated on
money and round-trips for it, and the gate says so (max_drawdown_absent)
instead of rebuilding one from an equity curve. Replays after the first
are checked against it and stop the run if they drift.
nautilus_trader 1.231.0 is pinned in the lock with its runtime
dependencies, and the registry carries a per-engine ceiling (100,000
bars, 64 combinations for a grid) because the engine costs seconds per
100,000 bars where the others cost milliseconds.
github-actions[bot]