bench: add a cost workload and a multi-asset one, and go to three repetitions (#10)

Two gaps a reader could name without running anything: costs appeared on one
workload out of four, and nothing in the suite was a portfolio.

Costs could not simply be switched on across the board, and the reason is
measured. On FractionOfEquity sizing a 5 bps fee puts the engines 1.3e-4 of
capital apart and 2 bps of slippage 2.1e-5, against a 1e-9 tolerance, while the
round-trip counts stay identical: the trading agrees, the cost arithmetic does
not, because one charges the fee on top of the notional and the other reserves
it out of cash first. In fixed units both land exactly, to 1e-15. So
`sma_cross_costs` carries a fee and slippage on the headline signal, sized in
units, and the price of that is visible rather than hidden: x48.0 against x50.6
at 100k bars.

`multi_asset` runs five independent series in one shared book. It is the
workload manifoldbt does worst on, and it is here for that reason: going from
one asset to five costs it 6.1x and vectorbt 1.4x, so the ratio falls from x36.7
to x8.8 at a million bars. Broadcasting a column per asset is close to free;
walking five books is not. A portfolio is also what people actually run, and a
suite that only measures where it wins is not evidence.

Both are capped where a materialised five-column simulation would stop measuring
the engine and start measuring the swap file, and `ema_rsi_fees` keeps the
ceiling it got for going bankrupt.

Repetitions go from two to three: the floor at which a median is a median rather
than the mean of two.
This commit is contained in:
Exocet92
2026-08-20 18:45:05 +02:00
committed by GitHub
parent 9ddefc64df
commit 6cae686283
6 changed files with 203 additions and 25 deletions
+53
View File
@@ -134,6 +134,59 @@ WORKLOADS: Dict[str, Workload] = {
"summary costs each engine.",
params=dict(fast=30, slow=150, alloc=1.0, metrics=True),
),
Workload(
key="sma_cross_costs",
title="SMA 30/150 with a 5 bps fee and 2 bps of slippage",
why="The same signal as the headline workload, run against a cost "
"model. Costs were the easiest objection to make of a benchmark "
"that had them on one workload out of four, and the answer is "
"not to sprinkle them everywhere: measured, a fee or a slippage "
"on FractionOfEquity sizing makes the engines disagree by 1e-4 "
"of capital against a 1e-9 tolerance, because they resolve the "
"same policy differently. In fixed units the arithmetic is "
"comparable and both costs land exactly, so this is where they "
"belong.",
params=dict(fast=30, slow=150, units=5.0, fee_bps=5.0, slippage_bps=2.0),
notes={
"raptorbt": Note(
"unsupported",
"Same blocker as the other fixed-quantity workload: raptorbt "
"has no units sizing, and a cost model on top of a fraction "
"of equity compares policy rather than arithmetic. See "
"`ema_rsi_fees` for the measurements behind that.",
),
},
),
Workload(
key="multi_asset",
title="Five assets in one book, SMA 30/150, 5 bps fee",
why="A single-asset benchmark measures a loop; a portfolio measures "
"the thing people actually run. It is also where the two designs "
"differ in kind rather than in speed: manifoldbt walks a universe "
"against one shared book, while vectorbt broadcasts a column per "
"asset and has to be told to share cash at all. The five series "
"are independent rather than correlated, so a sizing bug cannot "
"cancel itself out across the book.",
params=dict(fast=30, slow=150, units=2.0, fee_bps=5.0, assets=5),
# Five columns of a materialised simulation, not one. vectorbt adds
# 1.55 GB on a 10M-bar single asset, so five of them would ask a
# 16 GB runner for around 8 GB on top of 2.4 GB of generated frames.
# The ceiling is set where the point is certain to be measuring the
# engine rather than the swap file.
max_bars=1_000_000,
notes={
"raptorbt": Note(
"unsupported",
"No multi-instrument portfolio on the entry point this "
"harness drives: `run_single_backtest` is one instrument, and "
"`run_multi_backtest` broadcasts over strategies rather than "
"over assets. 0.9.0 does add `run_portfolio_backtest`, but "
"its allocation model is a different one and would need its "
"own parity work before any timing from it could be "
"published. The units blocker applies here too.",
),
},
),
Workload(
key="bracket_sl_tp",
title="SMA 10/50 entry with a 15 bps stop / 30 bps target bracket",