mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-08-24 14:38:04 +00:00
docs: normalise the speed figures on the published benchmark, document tf().apply and choice
The performance table was still quoting run #11 (manifoldbt 0.17.3), while the site publishes run #13 (0.18.0) from the same workflow. Same shape, different numbers, and the headline claim disagreed with the site by a few percent: 10M bars with the metric suite is 329 ms against 102.38 s, a factor of 311, not 317 ms / 97.46 s / 308. Every row now comes from run 32469701489, the one data/benchmark.json is synced from. Added the stop-loss/take-profit bracket row, where raptorbt is level with us (x1.0). It belongs next to the five-asset row for the same reason that one is there: deciding which of the stop and the target triggers first is a sequential intra-bar walk in both engines, so there is nothing left to vectorise. Publishing only the workloads we win looks better and reads worse. Qualified the backtrader figure. It is measured on a developer machine with a different script, not on the CI runner, so quoting it beside a CI table implied a comparability it does not have. It also no longer appears in the headline bullet: the site makes no numeric backtrader claim, and the two engines are not doing the same work. Documented the two additions that were reachable but undocumented: - tf("1h").apply(expr) evaluates on that timeframe's own grid, so periods count in ITS bars. The note warns about sma(tf("1h").close, 20), which looks equivalent and is not: it counts simulation bars over a step-held series, so on a 1m run it is a 20-minute smoothing of an hourly staircase. - choice(name, {branch: expr}) makes a selector a grid axis. Every snippet in these two sections was executed verbatim against the 0.19.0rc1 wheel before committing, including the warning, which was checked to actually differ rather than asserted. Header badges: PyPI version and Python versions (both live), Rust core, CUDA, benchmarks-in-CI linking the workflow, and the licence.
This commit is contained in:
@@ -8,6 +8,12 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
|
<a href="https://pypi.org/project/manifoldbt/"><img src="https://img.shields.io/pypi/v/manifoldbt?logo=pypi&logoColor=white&color=2f6fed" alt="PyPI"></a>
|
||||||
|
<a href="https://pypi.org/project/manifoldbt/"><img src="https://img.shields.io/pypi/pyversions/manifoldbt?logo=python&logoColor=white" alt="Python versions"></a>
|
||||||
|
<img src="https://img.shields.io/badge/core-Rust-dea584?logo=rust&logoColor=white" alt="Rust core">
|
||||||
|
<img src="https://img.shields.io/badge/GPU-CUDA%20(Pro)-76b900?logo=nvidia&logoColor=white" alt="CUDA">
|
||||||
|
<a href="https://github.com/manifoldbt/manifoldbt/actions/workflows/bench-vs-vectorbt.yml"><img src="https://img.shields.io/badge/benchmarks-public%20CI-2ea44f?logo=githubactions&logoColor=white" alt="Benchmarks in public CI"></a>
|
||||||
|
<a href="https://github.com/manifoldbt/manifoldbt/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Commons%20Clause-lightgrey" alt="License"></a>
|
||||||
<a href="https://discord.gg/bvU6Wjc72d"><img src="https://img.shields.io/badge/Discord-join%20the%20community-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
|
<a href="https://discord.gg/bvU6Wjc72d"><img src="https://img.shields.io/badge/Discord-join%20the%20community-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
@@ -26,7 +32,7 @@ sequential fill simulation with realistic fees, slippage, funding and look-ahead
|
|||||||
|
|
||||||
## Why ManifoldBT
|
## Why ManifoldBT
|
||||||
|
|
||||||
- **Fast**: 10M bars in 317 ms. 78x faster than vectorbt, 308x once you also want drawdown and Sharpe, ~3,500x faster than backtrader. [Measured in public CI](#performance), every run linked.
|
- **Fast**: 10M bars in 329 ms. 79x faster than vectorbt, and 311x once you also want drawdown and Sharpe. [Measured in public CI](#performance), every run linked.
|
||||||
- **Expressive**: fluent DSL with 30+ indicators, conditional logic, cross-asset references
|
- **Expressive**: fluent DSL with 30+ indicators, conditional logic, cross-asset references
|
||||||
- **Rigorous**: Monte Carlo, walk-forward, parameter sweeps, lookahead detection, exposure diagnostics
|
- **Rigorous**: Monte Carlo, walk-forward, parameter sweeps, lookahead detection, exposure diagnostics
|
||||||
- **Portable**: `pip install`, no Rust toolchain needed. Works on Python 3.9+.
|
- **Portable**: `pip install`, no Rust toolchain needed. Works on Python 3.9+.
|
||||||
@@ -114,6 +120,59 @@ manifoldbt import-csv data.csv --symbol EURUSD --symbol-id 1 --interval 1m
|
|||||||
manifoldbt ingest --provider binance --symbol BTCUSDT --symbol-id 1 --start ... --end ...
|
manifoldbt ingest --provider binance --symbol BTCUSDT --symbol-id 1 --start ... --end ...
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Higher timeframes
|
||||||
|
|
||||||
|
Declare the timeframes you want alongside the simulation one, then read them
|
||||||
|
with `mbt.tf(...)`. Columns are forward-filled onto the simulation grid, and a
|
||||||
|
bar's value only becomes readable once that bar has closed, so there is no
|
||||||
|
look-ahead.
|
||||||
|
|
||||||
|
```python
|
||||||
|
config = mbt.BacktestConfig(
|
||||||
|
...,
|
||||||
|
bar_interval=Interval.minutes(1), # simulate on 1m
|
||||||
|
extra_timeframes={"1h": Interval.hours(1)}, # also resample to 1h
|
||||||
|
)
|
||||||
|
|
||||||
|
h1 = mbt.tf("1h")
|
||||||
|
h1.close # the last closed hourly close, held across the minute bars
|
||||||
|
```
|
||||||
|
|
||||||
|
For an **indicator** on a higher timeframe, use `.apply(...)`. It evaluates the
|
||||||
|
expression on that timeframe's own grid, so the period counts in *its* bars:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from manifoldbt.indicators import close, sma
|
||||||
|
|
||||||
|
band = mbt.tf("1h").apply(sma(close, 20)) # mean of 20 HOURLY closes
|
||||||
|
```
|
||||||
|
|
||||||
|
> Careful: `sma(mbt.tf("1h").close, 20)` is **not** the same thing. That reads
|
||||||
|
> the step-held hourly series on the simulation grid, so the period counts in
|
||||||
|
> simulation bars: on a 1m simulation it is a 20-*minute* smoothing of an hourly
|
||||||
|
> staircase. Use `.apply(...)` whenever you want an indicator *of* the higher
|
||||||
|
> timeframe.
|
||||||
|
|
||||||
|
## Sweeping a choice, not just a number
|
||||||
|
|
||||||
|
`mbt.param(...)` sweeps numbers. `mbt.choice(...)` sweeps *expressions*: the
|
||||||
|
selector becomes a grid axis, and each combination resolves to its branch before
|
||||||
|
the simulation runs, so the branches it did not pick cost nothing.
|
||||||
|
|
||||||
|
```python
|
||||||
|
band = mbt.choice("band", {
|
||||||
|
"30m": mbt.tf("30m").apply(sma(close, mbt.param("len"))),
|
||||||
|
"1h": mbt.tf("1h").apply(sma(close, mbt.param("len"))),
|
||||||
|
"2h": mbt.tf("2h").apply(sma(close, mbt.param("len"))),
|
||||||
|
})
|
||||||
|
|
||||||
|
sweep = mbt.run_sweep(strategy, {"band": ["30m", "1h", "2h"],
|
||||||
|
"len": range(10, 210, 10)}, config, store)
|
||||||
|
```
|
||||||
|
|
||||||
|
The branches can hold any expression, so the same mechanism sweeps which
|
||||||
|
exogenous column to use, which asset to reference, or which indicator to apply.
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
| # | Example | What it shows |
|
| # | Example | What it shows |
|
||||||
@@ -146,26 +205,30 @@ engine from PyPI the way a user would, generates its own data, checks that the
|
|||||||
engines produced the **same result**, and only then reports how long each took:
|
engines produced the **same result**, and only then reports how long each took:
|
||||||
a workload they disagree on gets no published timing at all.
|
a workload they disagree on gets no published timing at all.
|
||||||
|
|
||||||
**Latest run: [#11](https://github.com/manifoldbt/manifoldbt/actions/runs/32396472073)**
|
**Latest run: [#13](https://github.com/manifoldbt/manifoldbt/actions/runs/32469701489)**
|
||||||
ran on Linux x86_64, 4 vCPU, Python 3.12, manifoldbt 0.17.3 / vectorbt 0.28.4 /
|
ran on Linux x86_64, 4 vCPU (AMD EPYC 7763), Python 3.12, manifoldbt 0.18.0 /
|
||||||
raptorbt 0.9.0, 3 interleaved repetitions.
|
vectorbt 0.28.4 / raptorbt 0.9.0, 3 interleaved repetitions, medians reported.
|
||||||
|
|
||||||
| Workload | Bars | ManifoldBT | vectorbt | raptorbt |
|
| Workload | Bars | ManifoldBT | vectorbt | raptorbt |
|
||||||
|---|---:|---:|---:|---:|
|
|---|---:|---:|---:|---:|
|
||||||
| SMA crossover | 10M | **317 ms** | 24.75 s (x78) | 878 ms (x2.8) |
|
| SMA crossover | 10M | **327 ms** | 26.12 s (x79) | 913 ms (x2.8) |
|
||||||
| ...with drawdown, Sharpe, Sortino, volatility | 10M | **317 ms** | 97.46 s (**x308**) | 894 ms (x2.8) |
|
| ...with drawdown, Sharpe, Sortino, volatility | 10M | **329 ms** | 102.38 s (**x311**) | 909 ms (x2.8) |
|
||||||
| ...with a 5 bps fee and 2 bps slippage | 10M | **316 ms** | 24.53 s (x78) | not supported |
|
| ...with a 5 bps fee and 2 bps slippage | 10M | **337 ms** | 26.08 s (x79) | not supported |
|
||||||
| EMA + RSI filter, 5 bps fee | 1M | **52 ms** | 2.21 s (x41) | not supported |
|
| EMA + RSI filter, 5 bps fee | 1M | **57 ms** | 2.35 s (x40) | not supported |
|
||||||
| Five assets in one book | 1M | **140 ms** | 2.34 s (x17) | not supported |
|
| Five assets in one book | 1M | **148 ms** | 2.54 s (x17) | not supported |
|
||||||
|
| Stop-loss and take-profit bracket | 10M | **934 ms** | 26.24 s (x28) | 916 ms (**x1.0**) |
|
||||||
|
|
||||||
The second row is the one worth reading twice. Asking for a performance summary
|
The second row is the one worth reading twice. Asking for a performance summary
|
||||||
costs ManifoldBT nothing measurable, because it computes one during the run
|
costs ManifoldBT nothing measurable, because it computes one during the run
|
||||||
whether you read it or not, and costs vectorbt 73 seconds, because it defers the
|
whether you read it or not, and costs vectorbt 102 seconds, because it defers
|
||||||
equity curve until a risk metric needs it and then has to build one.
|
the equity curve until a risk metric needs it and then has to build one.
|
||||||
|
|
||||||
The fifth row is the one where ManifoldBT does worst, and it is published for
|
The last two rows are the ones where ManifoldBT does worst, and they are
|
||||||
that reason: broadcasting a column per asset is close to free for vectorbt,
|
published for that reason. Broadcasting a column per asset is close to free for
|
||||||
while walking five books is not free for anything.
|
vectorbt, while walking five books is not free for anything. And on a
|
||||||
|
stop-loss/take-profit bracket, raptorbt is level with us: the intra-bar check
|
||||||
|
that decides which of the two triggers first is a sequential walk in both
|
||||||
|
engines, so there is no vectorization left to win with.
|
||||||
|
|
||||||
### Parameter sweeps
|
### Parameter sweeps
|
||||||
|
|
||||||
@@ -190,9 +253,14 @@ The method, the parity gate and the known divergences are written up in
|
|||||||
|
|
||||||
backtrader runs the same EMA(12/26) + RSI(14) strategy on 500K 1-minute bars in
|
backtrader runs the same EMA(12/26) + RSI(14) strategy on 500K 1-minute bars in
|
||||||
**46,944 ms**, against **13 ms** for ManifoldBT: a factor of **3,556**. Measured
|
**46,944 ms**, against **13 ms** for ManifoldBT: a factor of **3,556**. Measured
|
||||||
with `benchmarks/bench_vs_competitors.py`, median of 3 runs. It sits outside the
|
with `benchmarks/bench_vs_competitors.py`, median of 3 runs, on a developer
|
||||||
CI suite because its event-driven fills produce a different PnL, and the parity
|
machine and not the CI runner, so it is not comparable line-for-line with the
|
||||||
gate publishes no timing for engines that did not do the same work.
|
table above.
|
||||||
|
|
||||||
|
It sits outside the CI suite because its event-driven fills produce a different
|
||||||
|
PnL, and the parity gate publishes no timing for engines that did not do the
|
||||||
|
same work. Treat it as an order of magnitude, not a benchmark: the two engines
|
||||||
|
are not doing the same thing.
|
||||||
|
|
||||||
### How it compares
|
### How it compares
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user