mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-08-24 14:38:04 +00:00
bench: fix the red runs, trim the matrix, add a grid job (#7)
Five consecutive red runs, two unrelated causes. Four of them never reached an engine: the workflow installs the tag it is handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI. Pre-releases are now skipped, and the report step checks for its input file instead of dying on a missing one and reporting the wrong cause twice. The fifth came from adding 10M bars, which broke a workload whose validity depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees reach 99,611 of the 100,000 it started with. Both engines then sit at zero and disagree by 9,085 round-trips about how many worthless trades to book on a dead account. Workloads can now declare a ceiling, and the runner skips past it out loud. Fewer points per axis: three series lengths instead of five, a decade apart each step. 10k measured the clock rather than the work, and 5M sat between two points that already bracketed it. sma_cross crosses on 30/150 rather than 10/50, worth about 15% on the ratio because it books a third of the trades. Grids get their own job, licensed through ci_activate.py, which refuses to run unlicensed rather than time a wait. Three points, not a matrix: across the plane the four-core ratio moves only between x32 and x38.
This commit is contained in:
@@ -85,11 +85,47 @@ store)`, the documented entry point, not through an internal fast path.
|
||||
|
||||
| Workload | What it exercises | vectorbt | raptorbt |
|
||||
|---|---|---|---|
|
||||
| `sma_cross` | SMA 10/50 crossover, long-only, no cost | exact | exact |
|
||||
| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee | exact | unsupported |
|
||||
| `sma_cross` | SMA 30/150 crossover, long-only, no cost | exact | exact |
|
||||
| `ema_rsi_fees` | EMA 12/26 crossover with an RSI(14) filter and a 5 bps taker fee, capped at 1M bars | exact | unsupported |
|
||||
| `sma_cross_metrics` | the same simulation, plus max drawdown, Sharpe, Sortino and volatility | exact | exact |
|
||||
| `bracket_sl_tp` | the same entry with a 15 bps stop and a 30 bps target | documented | documented |
|
||||
|
||||
### Why the fee workload stops at 1M bars
|
||||
|
||||
A workload can stop being a comparison before it stops running. `ema_rsi_fees`
|
||||
sizes in fixed units and pays 5 bps a side, and at 1-minute resolution it turns
|
||||
over often enough that the fees compound into the account: measured, it ends at
|
||||
-15% of capital on 1M bars, -74% on 5M, and exactly -100% on 10M, where fees
|
||||
reach 99,611 of the 100,000 it started with.
|
||||
|
||||
Past that point the engines still agree on the equity, because both are sitting
|
||||
at zero, and disagree by thousands of round-trips about how many more worthless
|
||||
trades to book on a dead account. That is a fact about a bankrupt strategy, not
|
||||
about either engine, so the workload carries a ceiling and the runner skips it
|
||||
above that with the reason printed. The other workloads have no ceiling.
|
||||
|
||||
### What the windows are, and why they moved
|
||||
|
||||
`sma_cross` crosses on 30/150 rather than 10/50. The two were measured against
|
||||
each other on the same 5M bars, and the slower pair is worth about 15% on the
|
||||
ratio (x267 against x232 with a performance summary) because it books a third of
|
||||
the trades and manifoldbt's cost, unlike vectorbt's, moves with the trade count.
|
||||
|
||||
That is a real effect and a small one, and it is worth knowing which way the
|
||||
knobs turn before anyone quotes a number:
|
||||
|
||||
| Turn up | Effect on the ratio | Why |
|
||||
|---|---|---|
|
||||
| series length | **widens** | vectorbt materialises the simulation; its cost is linear in bars |
|
||||
| asking for the summary | **widens sharply** | it has to build the equity curve it deferred |
|
||||
| number of trades | narrows | near-free for vectorbt's per-bar loop, real for manifoldbt |
|
||||
| number of indicators | narrows | same reason |
|
||||
|
||||
Measured at 5M bars on four levels of turnover, the ratio runs from x40 at
|
||||
480,000 round-trips to x71 at 2,500. The floor matters more than the peak: even
|
||||
in the busiest configuration tested, with half a million round-trips, the gap
|
||||
holds at x40, and x151 with a performance summary.
|
||||
|
||||
Each of those runs across a range of series lengths. Two further axes, cold
|
||||
start and memory, are measured in their own processes because they cannot be
|
||||
measured honestly inside the main one.
|
||||
@@ -254,8 +290,30 @@ On the raptorbt side specifically:
|
||||
- `bench.py` - the runner
|
||||
- `sweep_child.py` - one parameter-grid point, in its own process
|
||||
- `report.py` - JSON to Markdown, and to the GitHub job summary
|
||||
- `ci_activate.py` - activates the licence the grid job needs, and refuses to
|
||||
continue without one
|
||||
- `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
|
||||
|
||||
## Parameter grids
|
||||
|
||||
Grids run in their own CI job, because they need a licence: an unlicensed
|
||||
fan-out call waits out a fixed interval before doing any work, so a stopwatch
|
||||
would be timing the wait rather than the engine. The harness refuses to produce
|
||||
a number in that state rather than producing a wrong one.
|
||||
|
||||
Three points, chosen from a measured map of the plane rather than picked. Across
|
||||
bars from 5,000 to 200,000 and grids from 500 to 10,000 combinations, the ratio
|
||||
on four cores moves only between x32 and x38, so a denser matrix would spend
|
||||
runner time re-measuring the same number. What the three points carry is the
|
||||
shape: two grid sizes at one series length, and one grid vectorbt cannot hold at
|
||||
all (it materialises the simulation per combination, measured at 3.93 MB per
|
||||
combination on 50,000 bars).
|
||||
|
||||
Grid ratios are much more sensitive to core count than single backtests are,
|
||||
because manifoldbt is the only one of the three that fans out across cores:
|
||||
measured on the same point, x38 on four cores and x146 on twenty. Numbers from
|
||||
the CI job are four-core numbers, and they are the conservative ones.
|
||||
|
||||
The directory is still named `vs_vectorbt` and the workflow file still
|
||||
`bench-vs-vectorbt.yml`: renaming either would break the path the public
|
||||
repository runs and start a fresh, empty run history.
|
||||
|
||||
Reference in New Issue
Block a user