plotly 7.0.0 landed on PyPI and removed the mapbox trace types that
vectorbt 0.28.4's template still references, so 'import vectorbt' died
before a single bar ran. plotly was the one transitive dependency the
lock did not cover; it does now, on 6.9.0, the last 6.x line. The
version printout includes it so the results stay comparable.
The root testpaths points at python/tests, so nothing collected
benchmarks/vs_vectorbt/test_report.py. A light job (pytest alone, no
wheel) runs them on every push and pull request.
Authored upstream; the release sync will keep carrying this copy.
A documented point whose timing was never published appears nowhere in
the report, so a measured size for it would be an explanation with
nothing to explain. Fails against the parent of the previous commit.
render() built the notes from `documented`, so a documented row without
a published timing would have got a measured-size line under a table it
does not appear in. The notes explain the timings published above them,
so they come from `timed`. Follow-up to the previous commit, which gave
every point its own line and left the population as it found it.
render() wrote the size of the first documented point and stopped, so a
run with three bracket_sl_tp timings published one measured line for
three rows. A divergence can grow with the number of bars, so each point
now gets its own line, as a list item, and the cause follows once.
test_report.py renders hand-built payloads and imports no engine.
Names which paths are maintained on this mirror and which are
overwritten by the release sync, and states that contributions are
accepted under Apache 2.0 without the Commons Clause.
The other two tests in the file run plain backtests on daily data and
still measure the leak on a Community wheel; only the detect_lookahead
call is Pro-gated, so the guard is per-test.
This CI runs the mirrored suite against the PyPI wheel without a licence.
Twelve tests exercised paths Community gates: walk-forward optimization
raises LicenseError before any geometry is read, and the look-ahead
detector both needs Pro and counts sub-daily bars that the Community
daily floor reduces to four. Same module-level guard the sweep tests
already carry, with the reason named.
Seven badges on one wrapped line buried the only one meant to be clicked. Discord
now sits alone above the rest, in the large style, and the second line keeps four
informational ones: PyPI, Python, Rust, benchmarks. Dropped the licence badge,
which repeated the LICENSE link already in the footer.
The pyversions badge was rendering "missing" in red, and it was right to. Every
published wheel carries NO `Programming Language :: Python :: 3.x` classifier:
crates/bt-python/pyproject.toml, the file maturin actually builds from, has no
`classifiers` field at all. The public pyproject.toml has one, but it does not
build anything, so its list never reached PyPI. `requires-python = ">=3.9"` is a
separate field and does not populate classifiers.
Replaced with a static "python 3.9+" badge, since PyPI cannot answer the question
for 0.18.0 whatever we ask it. The engine-side fix travels separately; once a
wheel ships with classifiers, PyPI's sidebar will list the versions too and this
can go back to being dynamic.
The performance table was still quoting run #11 (manifoldbt 0.17.3), while the
site publishes run #13 (0.18.0) from the same workflow. Same shape, different
numbers, and the headline claim disagreed with the site by a few percent:
10M bars with the metric suite is 329 ms against 102.38 s, a factor of 311,
not 317 ms / 97.46 s / 308. Every row now comes from run 32469701489, the one
data/benchmark.json is synced from.
Added the stop-loss/take-profit bracket row, where raptorbt is level with us
(x1.0). It belongs next to the five-asset row for the same reason that one is
there: deciding which of the stop and the target triggers first is a sequential
intra-bar walk in both engines, so there is nothing left to vectorise. Publishing
only the workloads we win looks better and reads worse.
Qualified the backtrader figure. It is measured on a developer machine with a
different script, not on the CI runner, so quoting it beside a CI table implied a
comparability it does not have. It also no longer appears in the headline bullet:
the site makes no numeric backtrader claim, and the two engines are not doing the
same work.
Documented the two additions that were reachable but undocumented:
- tf("1h").apply(expr) evaluates on that timeframe's own grid, so periods count
in ITS bars. The note warns about sma(tf("1h").close, 20), which looks
equivalent and is not: it counts simulation bars over a step-held series, so
on a 1m run it is a 20-minute smoothing of an hourly staircase.
- choice(name, {branch: expr}) makes a selector a grid axis.
Every snippet in these two sections was executed verbatim against the 0.19.0rc1
wheel before committing, including the warning, which was checked to actually
differ rather than asserted.
Header badges: PyPI version and Python versions (both live), Rust core, CUDA,
benchmarks-in-CI linking the workflow, and the licence.
PyPI accepte un televersement bien avant que son index le serve. Mesure sur la
publication de 0.18.0: le televersement s'est termine a 01:48:18 et ce workflow
a demande la version a 01:48:50, trente-deux secondes plus tard, pour se faire
repondre "from versions: ..., 0.17.3".
Ce n'est pas un hasard de timing. Le workflow est declenche par
`release: published`, et cette release est creee juste apres le televersement:
la course est donc GARANTIE a chaque publication. Elle n'avait jamais pu se voir
parce que ce chemin n'avait jamais tourne sur une vraie release, seulement sur
des declenchements manuels ou la version existait depuis longtemps.
Les deux jobs attendent desormais que l'index serve la version, jusqu'a dix
minutes, contre soixante de budget. `--no-cache-dir` parce que pip met en cache
la reponse de l'index, y compris celle qui ne connait pas encore la version: sans
cela les tentatives suivantes reliraient la meme reponse perimee.
Logique verifiee sur quatre cas avant de pousser (disponible tout de suite,
disponible apres trois echecs, jamais disponible donc echec franc, et pas de
version imposee), plutot que de la decouvrir a la prochaine release.
An artifact is not a publication. It needs a token to download, expires
after ninety days, and nothing outside GitHub can link to it, so a number
that only lives in an artifact is a number nobody can check.
A third job merges a green run's two payloads into
benchmarks/vs_vectorbt/results/latest.json and commits it. The two are
stored side by side rather than folded into one table: they run on two
runners, and timings from two machines are not rows of the same table.
Only a run where both measuring jobs came back green is published, and a
dispatch that pins an old version measures and reports without becoming
the published number.
`cancel-in-progress: true` is right for a CI triggered by pushes, where a newer
commit makes an older run pointless. It is wrong here. Three times in one
afternoon a second dispatch killed a run that was mid-measurement, once eight
minutes in, and each left a cancelled entry in a run list whose whole job is to
be readable by someone checking the numbers.
The fix people reach for then is deleting runs, which breaks the run_url the
published data links to, and with it the only reason to believe the numbers.
A manual run is never superseded: nobody dispatches a benchmark to invalidate
the one already running. A release or the weekly cron still supersedes, because
there an older run really is measuring a version nobody asks about any more.
Two gaps a reader could name without running anything: costs appeared on one
workload out of four, and nothing in the suite was a portfolio.
Costs could not simply be switched on across the board, and the reason is
measured. On FractionOfEquity sizing a 5 bps fee puts the engines 1.3e-4 of
capital apart and 2 bps of slippage 2.1e-5, against a 1e-9 tolerance, while the
round-trip counts stay identical: the trading agrees, the cost arithmetic does
not, because one charges the fee on top of the notional and the other reserves
it out of cash first. In fixed units both land exactly, to 1e-15. So
`sma_cross_costs` carries a fee and slippage on the headline signal, sized in
units, and the price of that is visible rather than hidden: x48.0 against x50.6
at 100k bars.
`multi_asset` runs five independent series in one shared book. It is the
workload manifoldbt does worst on, and it is here for that reason: going from
one asset to five costs it 6.1x and vectorbt 1.4x, so the ratio falls from x36.7
to x8.8 at a million bars. Broadcasting a column per asset is close to free;
walking five books is not. A portfolio is also what people actually run, and a
suite that only measures where it wins is not evidence.
Both are capped where a materialised five-column simulation would stop measuring
the engine and start measuring the swap file, and `ema_rsi_fees` keeps the
ceiling it got for going bankrupt.
Repetitions go from two to three: the floor at which a median is a median rather
than the mean of two.
The three points were sized before this runner had ever run one. It has now,
so they are sized from what it measured: 87.5 us per combination for manifoldbt
at 20,000 bars, 1.16 ms for vectorbt, 1.34 ms for raptorbt, and 15.0 ms for
raptorbt at 200,000 bars.
20,000 x 5,000 stays, because it is the only one of the three vectorbt can hold:
it materialises 1.57 MB per combination at that length, so 5,000 already costs
it 2.5 GB. The other two grow to 20,000 and 10,000 combinations and put it out
of scope, which is where a sweep stops being a speed comparison and becomes a
capability one.
raptorbt sets the budget, not manifoldbt. With no fan-out API its sweep is a
Python loop costing a full backtest per cell, so the large point goes deep in
combinations on a short series rather than the reverse: 20,000 combinations on
20,000 bars costs it 27 s a call, where 5,000 combinations on a million bars
would cost it 25 minutes.
Also: sweeps, not grids. `run_sweep`, `run_sweep_lite` and `--sweep` are what
the product calls this, and a second word for the same thing is a second thing
to learn. `grid` is kept only where it means the parameter space itself.
Windows and macOS were carried on an argument that does not survive
examination. They were the only place in the whole chain that installed the
published wheel and ran it, which made this benchmark an install smoke test by
accident. That check is worth having, and worth forty seconds next to the build
in release.yml rather than thirteen minutes inside a performance measurement:
nobody reads a benchmark to find out whether a package imports. release.yml
already builds on all four targets, it just never executes what it built.
What is lost is a per-platform timing that was never quoted; the numbers that
get published are the Linux ones. Adding a platform back is one matrix entry.
The jobs are also named for what they measure rather than for the runner they
landed on, which the row already says: `backtests (ubuntu-latest)` and `grids
(ubuntu-latest)` instead of a raw label next to a hand-written one.
Five consecutive red runs, two unrelated causes.
Four of them never reached an engine: the workflow installs the tag it is
handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI.
Pre-releases are now skipped, and the report step checks for its input file
instead of dying on a missing one and reporting the wrong cause twice.
The fifth came from adding 10M bars, which broke a workload whose validity
depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and
pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole
account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees
reach 99,611 of the 100,000 it started with. Both engines then sit at zero and
disagree by 9,085 round-trips about how many worthless trades to book on a dead
account. Workloads can now declare a ceiling, and the runner skips past it out
loud.
Fewer points per axis: three series lengths instead of five, a decade apart
each step. 10k measured the clock rather than the work, and 5M sat between two
points that already bracketed it. sma_cross crosses on 30/150 rather than
10/50, worth about 15% on the ratio because it books a third of the trades.
Grids get their own job, licensed through ci_activate.py, which refuses to run
unlicensed rather than time a wait. Three points, not a matrix: across the
plane the four-core ratio moves only between x32 and x38.
The harness compared two engines everywhere; it now compares N against a
reference. manifoldbt is the reference: every parity check and every ratio is
a challenger against it, never two challengers against each other.
raptorbt 0.9.0 joins on three of the four workloads. Its sma_cross comes back
bit-identical to the reference's final equity, and its rsi matches to the last
bit; its ema seeds on a different warmup and it has no fixed-quantity sizing,
so the fee workload records it as unsupported with the reason rather than
leaving a blank cell. On the bracket it diverges in its own documented way: it
never re-arms while the entry level holds, so it books exactly the reference's
round-trips minus the ones that re-enter on the exit bar.
Python moves to 3.12, which raptorbt pins rather than we do: it is built
against pyo3 0.20.3, whose maximum supported CPython is 3.12. Timings from runs
before this change are therefore not directly comparable.
The bar matrix gains 10M and the repetition default drops from 7 to 2. Measured,
those two almost cancel: the job stays around 16 minutes. macOS keeps its old
ceiling, since 10M bars adds 1.55 GB on vectorbt's side alone and that runner
has 7 GB.
