mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-11-02 07:56:11 +00:00
Merge bench/nautilus: NautilusTrader in the bar benchmark
One pull request for both benchmark changes: NautilusTrader as a fourth engine in the bar benchmark (was #18), and the market-making and HFT benchmark.
This commit is contained in:
Jimmy7892
@@ -1,4 +1,4 @@
|
||||
# manifoldbt vs vectorbt vs raptorbt
|
||||
# manifoldbt vs vectorbt vs raptorbt vs NautilusTrader
|
||||
|
||||
An engine-to-engine benchmark you can re-run yourself. It installs every engine
|
||||
from PyPI, generates its own data, checks that they produced the **same
|
||||
@@ -19,12 +19,13 @@ behind it.
|
||||
whose maximum supported CPython is 3.12: no release up to 0.9.0 publishes a
|
||||
cp313 wheel, and a source build refuses outright. Comparing engines means
|
||||
running them in one environment, and the environment has to be one all of them
|
||||
support. Add or drop a challenger with `--engines`; an engine that is not
|
||||
support. NautilusTrader asks for 3.12 or later, so it does not move the
|
||||
interpreter. Add or drop a challenger with `--engines`; an engine that is not
|
||||
installed is skipped with a printed line rather than crashing the run.
|
||||
|
||||
**manifoldbt is the reference.** Every parity check and every ratio is a
|
||||
challenger against it, never two challengers against each other: three engines
|
||||
make three pairs, and a table of pairs is a matrix, not a benchmark. The
|
||||
challenger against it, never two challengers against each other: four engines
|
||||
make six pairs, and a table of pairs is a matrix, not a benchmark. The
|
||||
reference gets no advantage from the position, it is simply the one every
|
||||
timing is divided by.
|
||||
|
||||
@@ -51,6 +52,13 @@ report prints the reason instead of an empty cell. A blank in a speed table
|
||||
reads as a defeat, and "this engine has no fixed-quantity sizing" is not a
|
||||
defeat, it is a different fact.
|
||||
|
||||
A fifth, `not run above N bars`, is a time budget rather than a verdict, and it
|
||||
applies to one engine only: an event-driven engine replays every bar through a
|
||||
simulated exchange and costs seconds where the others cost milliseconds, so it
|
||||
is not repeated on series it would need hours for. It is printed with its
|
||||
reason, like `unsupported`, and the ceiling is a command-line option (see
|
||||
[NautilusTrader is a different kind of engine](#nautilustrader-is-a-different-kind-of-engine)).
|
||||
|
||||
The `failed` path is not decoration. It is the reason the other numbers can be
|
||||
trusted, and it makes the benchmark fail loudly if a future release of either
|
||||
engine changes a fill rule.
|
||||
@@ -80,17 +88,24 @@ simulation plus reading the headline metrics, nothing else.
|
||||
|
||||
**Only public APIs.** manifoldbt is driven through `bt.run(strategy, config,
|
||||
store)`, the documented entry point, not through an internal fast path.
|
||||
NautilusTrader is driven through its low-level `BacktestEngine`, a `Strategy`
|
||||
subclass and its own indicators, which is what its documentation shows a user
|
||||
writing.
|
||||
|
||||
## What is compared
|
||||
|
||||
| Workload | What it exercises | vectorbt | raptorbt |
|
||||
|---|---|---|---|
|
||||
| `sma_cross` | SMA 30/150 crossover, long-only, no cost | exact | exact |
|
||||
| `sma_cross_metrics` | the same simulation, plus max drawdown, Sharpe, Sortino and volatility | exact | exact |
|
||||
| `sma_cross_costs` | the same signal with a 5 bps fee and 2 bps of slippage | exact | unsupported |
|
||||
| `multi_asset` | five assets in one shared book, capped at 1M bars | exact | unsupported |
|
||||
| `ema_rsi_fees` | EMA 12/26 with an RSI(14) filter and a 5 bps taker fee, capped at 1M bars | exact | unsupported |
|
||||
| `bracket_sl_tp` | the same entry with a 15 bps stop and a 30 bps target | documented | documented |
|
||||
| Workload | What it exercises | vectorbt | raptorbt | NautilusTrader |
|
||||
|---|---|---|---|---|
|
||||
| `sma_cross` | SMA 30/150 crossover, long-only, no cost | exact | exact | exact |
|
||||
| `sma_cross_metrics` | the same simulation, plus max drawdown, Sharpe, Sortino and volatility | exact | exact | exact, on money and round-trips |
|
||||
| `sma_cross_costs` | the same signal with a 5 bps fee and 2 bps of slippage | exact | unsupported | exact |
|
||||
| `multi_asset` | five assets in one shared book, capped at 1M bars | exact | unsupported | exact |
|
||||
| `ema_rsi_fees` | EMA 12/26 with an RSI(14) filter and a 5 bps taker fee, capped at 1M bars | exact | unsupported | exact |
|
||||
| `bracket_sl_tp` | the same entry with a 15 bps stop and a 30 bps target | documented | documented | documented |
|
||||
|
||||
NautilusTrader is run up to 100,000 bars (see below), and on the summary
|
||||
workload it is gated on total return and round-trips only, because it has no
|
||||
drawdown statistic to compare.
|
||||
|
||||
### Costs live in their own workload, and not by preference
|
||||
|
||||
@@ -262,12 +277,170 @@ of which 82 re-enter on the exit bar, and raptorbt books exactly 214 - 82 = 132.
|
||||
The report states the count and the share, so the size of the difference is
|
||||
measured instead of asserted.
|
||||
|
||||
## NautilusTrader is a different kind of engine
|
||||
|
||||
The other three engines are whole-series loops: vectorised, or compiled, or
|
||||
both. NautilusTrader is event-driven. It replays the bars one at a time through
|
||||
a simulated exchange with an order matching engine, and calls a Python strategy
|
||||
on each of them, which is what lets the same strategy run unchanged against a
|
||||
live venue. That design is the point of the engine, and it is also why it spends
|
||||
seconds where the others spend milliseconds on a workload like this one: a
|
||||
moving-average crossover is exactly the case a vectorised engine was built for.
|
||||
The comparison below is what this workload costs on each design, and says
|
||||
nothing about what each is for.
|
||||
|
||||
### What is timed
|
||||
|
||||
Everything that can be prepared once is prepared before timing, as for the other
|
||||
engines: the instrument, the `Bar` objects (`BarDataWrangler`), the venue, the
|
||||
account, the strategy, and the `add_data` call that hands the bars to the
|
||||
engine. The timed closure is what a user repeats, `engine.reset()`,
|
||||
`engine.run()` and reading the result. A replay is checked rather than assumed:
|
||||
the first call is the one the parity gate reads, and every later call must
|
||||
return exactly what it returned, or the run stops.
|
||||
|
||||
### Why it stops at 100,000 bars
|
||||
|
||||
Measured locally, pinned to four logical cores of a 13th-generation Core i5 (a
|
||||
hosted runner is slower): one `sma_cross` call costs 3.4 s at 100,000 bars,
|
||||
which is about 29,000 bars per second, and the cost is linear in the series. At
|
||||
10,000,000 bars one call would take about six minutes, and the eight calls of
|
||||
one workload (the gated call and seven repetitions) close to an hour, before the
|
||||
other five workloads. The ceiling is therefore 100,000 bars, it is a time budget
|
||||
and not a verdict, and each row beyond it says so instead of leaving a blank.
|
||||
`bench.py --engine-max-bars nautilus_trader=1000000` lifts it for a local run.
|
||||
|
||||
The cost is not only the bars. Every fill is an order that goes through the risk
|
||||
engine, the matching engine and the account, and costs on the order of a
|
||||
millisecond (from the table: about 0.8 ms per extra fill between `sma_cross` and
|
||||
`ema_rsi_fees`): `ema_rsi_fees`, which trades about 6,400 times per 100,000
|
||||
bars, takes 7.6 s where `sma_cross` takes 3.4 s, and the five-asset
|
||||
`multi_asset` takes 27 s because it replays five series.
|
||||
|
||||
### How each workload is expressed
|
||||
|
||||
- **Fill at the close of the signal bar.** The strategy submits a market order
|
||||
from `on_bar`. The simulated exchange has already moved the market to that
|
||||
bar's close, and the order is matched before the next bar arrives, so the fill
|
||||
lands on the same bar at the close. That is `signal_delay=0` with
|
||||
`execution_price="AtClose"`, and it needs no setting.
|
||||
- **No volume-limited fills.** Nautilus builds its order book from a bar by
|
||||
splitting the bar's volume over four synthetic updates (open, high, low,
|
||||
close) and treats that quantity as the liquidity available. A market order
|
||||
larger than a quarter of the bar's volume is partially filled, and the
|
||||
remainder is booked one tick worse. The generated volumes are random between
|
||||
100 and 10,000 units, and full-equity sizing is about 1,000 units, so that
|
||||
rule fires on about four full-equity orders in ten. None of the other engines
|
||||
has a volume constraint, so the venue is given a fill model that answers
|
||||
market orders from a book of effectively unlimited depth at the close. It is
|
||||
the mechanism Nautilus ships for this (`BestPriceFillModel`), written out only
|
||||
because the bracket workload needs stop and limit orders left on the standard
|
||||
logic. `BestPriceFillModel` prices every order from the best bid and ask,
|
||||
stops and limits included, and with it `bracket_sl_tp` ends 6.2e-3 of capital
|
||||
away from the run that fills them at their own levels.
|
||||
- **Slippage in basis points** is not a Nautilus setting: `prob_slippage` moves
|
||||
a fill by one *tick*, at random. The same fill model shifts the book instead,
|
||||
asks 2 bps above the close and bids 2 bps below, which is the documented
|
||||
extension point for custom slippage and gives what `Slippage.fixed_bps` gives.
|
||||
- **Fees** are the instrument's taker fee, charged on notional by the venue's
|
||||
default fee model. A market order always takes.
|
||||
- **Account and sizing.** A margin account with leverage 1, in a quote currency
|
||||
with eight decimals (USDT). Not USD, which has two: Nautilus rounds each
|
||||
realised PnL to the currency's precision, and measured with a USD quote,
|
||||
`sma_cross` ends 2.8e-7 of capital from the reference at 20,000 bars and
|
||||
2.1e-7 at 100,000, over two hundred times the gate. `FractionOfEquity` is the
|
||||
cash balance at the signal bar divided by the close, rounded *down* to the
|
||||
quantity precision; units sizing is the fixed quantity.
|
||||
- **Indicators** are Nautilus's own `SimpleMovingAverage`,
|
||||
`ExponentialMovingAverage` and `RelativeStrengthIndex` (Wilder), registered on
|
||||
the bar type so the engine updates them before `on_bar` runs.
|
||||
- **Brackets** are Nautilus's native contingent orders (`OrderFactory.bracket`):
|
||||
a stop-market and a limit, one-cancels-other, armed when the market entry
|
||||
fills, at levels computed from the signal bar's close. The venue's standard
|
||||
logic decides the fills: a stop the bar trades through fills at its trigger
|
||||
price, a limit the bar reaches fills at its limit price.
|
||||
|
||||
### Fixed-point rounding
|
||||
|
||||
Prices and quantities in Nautilus are fixed-point integers. The Windows wheel
|
||||
carries 9 decimals and the Linux wheel 16, and the adapter uses 9 on both so
|
||||
that the platform changes almost nothing: the parity results on the two agree to
|
||||
at most 2e-16 of capital. Rounding the generated float64 prices to 9 decimals
|
||||
moves the final equity by 3e-13 to 6e-11 of capital across the five workloads
|
||||
that agree, at 100,000 bars, and up to 1.5e-10 on a 20,000-bar grid whose short windows trade
|
||||
every few bars. The error grows with the square root of the number of fills. The
|
||||
gate is 1e-9, and the number is printed in every run, not hidden.
|
||||
|
||||
### The RSI seed
|
||||
|
||||
manifoldbt seeds Wilder's RSI with the simple average of the first `period`
|
||||
deltas and emits from bar `period`. Nautilus seeds it on the first observation
|
||||
and reports it as initialised after `period` updates. Same recursion, different
|
||||
seed, so the two series differ at the start and converge geometrically: on the
|
||||
benchmark data the largest gap is 4.5e-2 (on a 0-1 scale) in the first 50 bars,
|
||||
1.2e-4 between bars 100 and 200, and 2e-16 from bar 800 on. Whether that moves a
|
||||
decision is what the gate answers, and on this data it does not: the entry
|
||||
signals are identical, `ema_rsi_fees` comes back with the reference's
|
||||
round-trips and fees. That is a property of this dataset, which is why the
|
||||
difference is written down here rather than left for the gate to hide. The
|
||||
adapter uses the engine's own indicator and its own `initialized` flag; the EMA
|
||||
is identical to the reference to the last bit and the SMA to about 1e-13.
|
||||
|
||||
### The documented divergence on `bracket_sl_tp`
|
||||
|
||||
Entries, exits and fill prices are the same trades as the reference, round-trip
|
||||
for round-trip (448 against 448 at 20,000 bars, 2,309 against 2,309 at 100,000).
|
||||
Because the simulated exchange runs before the strategy sees the bar, a position
|
||||
stopped out intrabar is entered again at that bar's close when the signal still
|
||||
holds, exactly as the reference does, and not on the next bar as in vectorbt or
|
||||
never as in raptorbt.
|
||||
|
||||
What differs is the quantity of that re-entry. The reference sizes it from the
|
||||
equity at the close, counting the position it has just closed at the close
|
||||
price, so it buys back the quantity it held. Nautilus sizes it from the cash
|
||||
balance after the stop or target has been booked at its own price. Neither is
|
||||
wrong, the two quantities differ by the gap between the exit price and the close
|
||||
(a median of 2.7e-04 in relative terms, at most 1.1e-03), and the difference is
|
||||
carried into every later trade: final equity ends 7.1e-06 of capital apart at
|
||||
100,000 bars. The harness counts the affected re-entries and reports their
|
||||
quantity gaps instead of asserting any of this.
|
||||
|
||||
Nautilus also books a stop in two pieces when a quarter of the bar's volume is
|
||||
smaller than the position (a partial fill, then the remainder one tick worse, by
|
||||
the engine's documented rule), so its fill count is higher than its round-trip
|
||||
count implies. Fills are not gated, and the extra tick on the remainder is 1e-9.
|
||||
|
||||
### No drawdown, and ratios on their own basis
|
||||
|
||||
Nautilus's portfolio analyser reports volatility, Sharpe and Sortino and no
|
||||
maximum drawdown. The harness does not rebuild one from an equity curve for it,
|
||||
as it does not rebuild raptorbt's missing volatility: that column shows what the
|
||||
engine hands a user. `sma_cross_metrics` is therefore gated on total return and
|
||||
round-trips for Nautilus, which is said in the report. Its ratios are on their
|
||||
own basis (labelled 252 days, built from the returns of closed positions), not
|
||||
over 365 days from a daily equity curve, so they are recorded and not compared.
|
||||
|
||||
### Cold start and memory
|
||||
|
||||
Importing NautilusTrader and building an engine costs 1.2 s over the bare
|
||||
interpreter (against 0.15 s for manifoldbt in the same run). Memory is measured
|
||||
the same way as for the others, as what the run adds on top of data the process
|
||||
already holds, and for Nautilus that is a few megabytes at any length: 0.4 MB at
|
||||
100,000 bars and 2.6 MB at 2,000,000 (a one-off measurement, not part of the
|
||||
run, since 2,000,000 bars take minutes). The probe measures it at its ceiling,
|
||||
and the report prints the length beside the figure. What the exclusion hides is
|
||||
the cost of holding the data: the `Bar` objects an engine keeps in memory take
|
||||
230 to 280 MB per million bars (measured on the loaded engine), against 48 MB
|
||||
for the same bars as a float64 frame. Building each engine's data representation
|
||||
is a different question and is excluded on every side, but a reader sizing a job
|
||||
for a long series should know the number.
|
||||
|
||||
## Alignment choices, and why each one exists
|
||||
|
||||
Engines only produce identical numbers if they are told to do the same thing.
|
||||
These are the conventions the harness sets, all of them visible in
|
||||
[engine_mbt.py](engine_mbt.py), [engine_vbt.py](engine_vbt.py) and
|
||||
[engine_rbt.py](engine_rbt.py):
|
||||
[engine_mbt.py](engine_mbt.py), [engine_vbt.py](engine_vbt.py),
|
||||
[engine_rbt.py](engine_rbt.py) and [engine_nt.py](engine_nt.py):
|
||||
|
||||
- `signal_delay=0` with `execution_price="AtClose"`, matching what
|
||||
`Portfolio.from_signals` does by default: a signal fills at the close of the
|
||||
@@ -316,6 +489,19 @@ On the raptorbt side specifically:
|
||||
handicap the harness imposed, it is the only spelling available, and the
|
||||
moving averages are hoisted out of the loop so it gets the same courtesy
|
||||
vectorbt gets on its own grid path.
|
||||
- NautilusTrader has no fan-out API either, and its loop is the expensive kind:
|
||||
each cell is a full event-driven replay. It is looped the way the engine
|
||||
documents repeat runs (one engine, data loaded once, `reset()` and `run()` per
|
||||
cell). The moving averages cannot be hoisted out of the loop for it, because a
|
||||
Nautilus indicator lives inside the event loop; each cell computes its own,
|
||||
which is what a Nautilus strategy does.
|
||||
- A simple signal strategy on bars is the cheapest thing an event-driven engine
|
||||
can be asked to do, and it is also the case a vectorised engine was designed
|
||||
around. What NautilusTrader buys, and these workloads do not use, is the
|
||||
matching engine, order types and contingency, order book and tick data, and the
|
||||
same strategy code in live trading. This benchmark measures bar-replay
|
||||
throughput of one simple strategy on each design; it does not measure what
|
||||
either is for.
|
||||
|
||||
## Files
|
||||
|
||||
@@ -323,14 +509,17 @@ On the raptorbt side specifically:
|
||||
- `probe_child.py` - the cold-start and memory probes, each in a fresh process
|
||||
- `workloads.py` - the parameters every engine reads, and the per-engine notes
|
||||
- `engines.py` - the registry: who is in the comparison, and who is the reference
|
||||
- `engine_mbt.py` / `engine_vbt.py` / `engine_rbt.py` - one adapter per engine
|
||||
- `engine_mbt.py` / `engine_vbt.py` / `engine_rbt.py` / `engine_nt.py` - one
|
||||
adapter per engine
|
||||
- `parity.py` - the gate
|
||||
- `bench.py` - the runner
|
||||
- `sweep_child.py` - one parameter-grid point, in its own process
|
||||
- `report.py` - JSON to Markdown, and to the GitHub job summary
|
||||
- `ci_activate.py` - activates the licence the sweep job needs, and refuses to
|
||||
continue without one
|
||||
- `ci/bench-vs-vectorbt.yml` - the workflow, deployed to the public repository
|
||||
- `test_report.py` / `test_harness.py` - guards for the report, the gate, the
|
||||
ceilings and the sweep plumbing; they need no engine installed
|
||||
- `.github/workflows/bench-vs-vectorbt.yml` - the workflow
|
||||
|
||||
## Parameter sweeps
|
||||
|
||||
@@ -357,6 +546,39 @@ short series instead of the reverse: 20,000 combinations on 20,000 bars costs it
|
||||
minutes.
|
||||
|
||||
Sweep ratios depend on the machine far more than single-backtest ratios do,
|
||||
because manifoldbt is the only one of the three that fans out across cores. The
|
||||
because manifoldbt is the only one of the four that fans out across cores. The
|
||||
same point measured x13 on a four-vCPU cloud runner and x32 on two fast desktop
|
||||
cores; the CI numbers are the conservative ones, and they are the ones published.
|
||||
|
||||
### NautilusTrader and the grid
|
||||
|
||||
A grid on NautilusTrader is a loop of full replays, so its cost is the number of
|
||||
combinations times one backtest, and a backtest is seconds: 0.75 s on average
|
||||
across the 20,000-bar grid above, 7.8 s at 200,000 bars, and more where the
|
||||
windows are short enough to trade every few bars. Locally, the three points
|
||||
above would cost it 63 minutes for 5,000 combinations at 20,000 bars, 4.7 hours
|
||||
for 20,000, and 21.6 hours for 10,000 combinations at 200,000 bars, against
|
||||
under three seconds for manifoldbt in each case. On a four-vCPU hosted runner a
|
||||
backtest costs about twice as much (1.45 to 1.63 s at 20,000 bars and 15.3 s at
|
||||
200,000, run 39945358568), and the same three grids come to 2.0, 9.1 and 42.5
|
||||
hours. That is not a comparison a job can repeat, so it is out of scope on all
|
||||
three, and the reason printed beside each point carries the per-backtest time
|
||||
measured on 16 cells spread across that point's own grid, and what that makes of
|
||||
the whole grid. Those 16 cells are also run on manifoldbt and compared cell by
|
||||
cell, so the point still carries a cross-engine check of the same code path.
|
||||
|
||||
The engine is held back by a sweep ceiling of 64 combinations, applied on its
|
||||
own so that a point never needs to say so, and `--engine-max-combos
|
||||
nautilus_trader=N` lifts it. Below the ceiling it runs, and the grid is checked
|
||||
cell by cell like the others. A grid it can hold, run locally on the pinned
|
||||
workstation:
|
||||
|
||||
```bash
|
||||
python bench.py --workloads sma_cross --bars 100000 --reps 1 --cold-start-reps 0 \
|
||||
--sweep 20000:36 --sweep-reps 2
|
||||
```
|
||||
|
||||
On 36 combinations at 20,000 bars that loop takes 41 s, against 11 ms for
|
||||
manifoldbt's single fan-out call, 79 ms for vectorbt and 45 ms for raptorbt, and
|
||||
the 36 cells agree with the reference to 1.2e-10 of capital. It adds 6 MB of
|
||||
memory.
|
||||
|
||||
+180
-32
@@ -46,7 +46,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import data as data_mod # noqa: E402
|
||||
import engines as engines_mod # noqa: E402
|
||||
import parity as parity_mod # noqa: E402
|
||||
from engines import CHALLENGERS, ENGINES, REFERENCE # noqa: E402
|
||||
from engines import CHALLENGERS, ENGINES, REFERENCE, ceiling, sweep_ceiling # noqa: E402
|
||||
from workloads import ( # noqa: E402
|
||||
DEFAULT_KEYS,
|
||||
SCOPE_PAIR,
|
||||
@@ -74,6 +74,24 @@ def _in_range(key: str, bars: int) -> bool:
|
||||
SCHEMA_VERSION = 2
|
||||
NOISE_THRESHOLD = 0.15
|
||||
|
||||
# `--engine-max-bars` overrides, engine name -> bars (0 lifts the ceiling). Module
|
||||
# level because every measuring function below needs it and none of them should
|
||||
# have to be told twice.
|
||||
CEILING_OVERRIDES: Dict[str, int] = {}
|
||||
SWEEP_CEILING_OVERRIDES: Dict[str, int] = {}
|
||||
|
||||
|
||||
def _within_ceiling(name: str, bars: int) -> bool:
|
||||
"""Is this engine run on a series this long?
|
||||
|
||||
The ceiling is a time budget (see ``engines.py``), not a validity limit like
|
||||
a workload's ``max_bars``, and the two are kept apart on purpose: a workload
|
||||
skipped everywhere says something about the workload, an engine skipped at
|
||||
one size says something about what the job can afford.
|
||||
"""
|
||||
limit = ceiling(name, CEILING_OVERRIDES)
|
||||
return limit is None or bars <= limit
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Environment: what a reader needs to know before trusting a number
|
||||
@@ -246,11 +264,11 @@ def _roll_up(verdicts: Dict[str, Dict[str, Any]]) -> str:
|
||||
|
||||
|
||||
def _prepare_all(key: str, frame, workdir: str, active: List[str]):
|
||||
"""Build one timed closure per engine that can run this workload."""
|
||||
"""Build one timed closure per engine that can run this workload, here."""
|
||||
return {
|
||||
name: engines_mod.adapter(name).prepare(key, frame, workdir)
|
||||
for name in active
|
||||
if supported(key, name)
|
||||
if supported(key, name) and _within_ceiling(name, len(frame))
|
||||
}
|
||||
|
||||
|
||||
@@ -271,7 +289,8 @@ def _warm_and_gate(key: str, runners: Dict[str, Callable]) -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def _entry(key: str, bars: int, frame, verdicts: Dict[str, Any], engines_run: List[str]):
|
||||
def _entry(key: str, bars: int, frame, verdicts: Dict[str, Any], engines_run: List[str],
|
||||
active: List[str]):
|
||||
entry: Dict[str, Any] = {
|
||||
"workload": key,
|
||||
"title": WORKLOADS[key].title,
|
||||
@@ -281,9 +300,23 @@ def _entry(key: str, bars: int, frame, verdicts: Dict[str, Any], engines_run: Li
|
||||
"parity": verdicts,
|
||||
"status": _roll_up(verdicts),
|
||||
}
|
||||
skipped = unsupported_by(key)
|
||||
skipped = dict(unsupported_by(key))
|
||||
# An engine held back by its ceiling is not run on this row, and its reason
|
||||
# goes where the other "not run" reasons go, which is the one place a reader
|
||||
# (and the website) already looks for one. It is also kept apart, in
|
||||
# `beyond_ceiling`, because "cannot express this workload" and "too slow to
|
||||
# repeat at this length" are different facts.
|
||||
held_back = {
|
||||
name: ceiling(name, CEILING_OVERRIDES)
|
||||
for name in active
|
||||
if supported(key, name) and not _within_ceiling(name, bars)
|
||||
}
|
||||
for name, limit in held_back.items():
|
||||
skipped.setdefault(name, ENGINES[name].ceiling_why.format(limit=limit))
|
||||
if skipped:
|
||||
entry["unsupported"] = skipped
|
||||
if held_back:
|
||||
entry["beyond_ceiling"] = held_back
|
||||
return entry
|
||||
|
||||
|
||||
@@ -344,7 +377,7 @@ def measure_pair(keys: List[str], bars: int, reps: int, workdir: str,
|
||||
for key in keys:
|
||||
runners[key] = _prepare_all(key, frame, workdir, active)
|
||||
verdicts = _warm_and_gate(key, runners[key])
|
||||
entries[key] = _entry(key, bars, frame, verdicts, list(runners[key]))
|
||||
entries[key] = _entry(key, bars, frame, verdicts, list(runners[key]), active)
|
||||
entries[key]["paired_with"] = [k for k in keys if k != key]
|
||||
|
||||
samples = {key: {name: [] for name in runners[key]} for key in keys}
|
||||
@@ -364,7 +397,7 @@ def measure(key: str, bars: int, reps: int, workdir: str, active: List[str]) ->
|
||||
frame = data_mod.make_ohlcv(bars)
|
||||
runners = _prepare_all(key, frame, workdir, active)
|
||||
verdicts = _warm_and_gate(key, runners)
|
||||
entry = _entry(key, bars, frame, verdicts, list(runners))
|
||||
entry = _entry(key, bars, frame, verdicts, list(runners), active)
|
||||
|
||||
scales = {}
|
||||
for name, verdict in verdicts.items():
|
||||
@@ -411,6 +444,23 @@ def _probe(mode: str, engine: str, workload: str, bars: int) -> Dict[str, Any]:
|
||||
raise RuntimeError("probe produced no result: " + out[-500:])
|
||||
|
||||
|
||||
def _probe_or_none(mode: str, name: str, workload: str, bars: int, failed: Dict[str, str]):
|
||||
"""One probe, where a challenger that dies costs its own cell and nothing else.
|
||||
|
||||
The reference is not forgiven: without it there is nothing to divide by, and
|
||||
a run that quietly lost its baseline would publish a table of absolutes.
|
||||
"""
|
||||
try:
|
||||
return _probe(mode, ENGINES[name].code, workload, bars)
|
||||
except Exception as exc: # noqa: BLE001 - recorded, not hidden
|
||||
if name == REFERENCE:
|
||||
raise
|
||||
detail = getattr(exc, "stderr", None) or str(exc)
|
||||
failed[name] = "{} probe died: {}".format(mode, str(detail)[-300:].strip())
|
||||
print(" ! {} {} probe died; its column stays empty".format(name, mode))
|
||||
return None
|
||||
|
||||
|
||||
def cold_start(workload: str, bars: int, reps: int, active: List[str]) -> Dict[str, Any]:
|
||||
"""Time to a first backtest in a process that has never seen any engine.
|
||||
|
||||
@@ -419,14 +469,24 @@ def cold_start(workload: str, bars: int, reps: int, active: List[str]) -> Dict[s
|
||||
engines alternate here too, and the interpreter/numpy/pandas baseline is
|
||||
measured alongside so each engine's own share is visible.
|
||||
"""
|
||||
order = [name for name in active if supported(workload, name)]
|
||||
order = [name for name in active
|
||||
if supported(workload, name) and _within_ceiling(name, bars)]
|
||||
samples: Dict[str, List[float]] = {name: [] for name in order}
|
||||
samples["baseline"] = []
|
||||
failed: Dict[str, str] = {}
|
||||
for _ in range(reps):
|
||||
for name in order:
|
||||
samples[name].append(
|
||||
_probe("coldstart", ENGINES[name].code, workload, bars)["seconds"])
|
||||
if name in failed:
|
||||
continue
|
||||
result = _probe_or_none("coldstart", name, workload, bars, failed)
|
||||
if result is not None:
|
||||
samples[name].append(result["seconds"])
|
||||
samples["baseline"].append(_probe("baseline", "none", workload, bars)["seconds"])
|
||||
# An engine that died on any repetition leaves the table: a median over the
|
||||
# repetitions it survived would be a different measurement from the others.
|
||||
for name in failed:
|
||||
order.remove(name)
|
||||
del samples[name]
|
||||
medians = {k: statistics.median(v) for k, v in samples.items()}
|
||||
base = medians["baseline"]
|
||||
reference = medians[REFERENCE]
|
||||
@@ -434,6 +494,7 @@ def cold_start(workload: str, bars: int, reps: int, active: List[str]) -> Dict[s
|
||||
"workload": workload,
|
||||
"bars": bars,
|
||||
"engines": order,
|
||||
**({"failed": failed} if failed else {}),
|
||||
"samples_s": samples,
|
||||
"median_s": medians,
|
||||
"engine_share_s": {name: medians[name] - base for name in order},
|
||||
@@ -455,8 +516,21 @@ def memory(workload: str, bars: int, active: List[str]) -> Dict[str, Any]:
|
||||
"""
|
||||
order = [name for name in active if supported(workload, name)]
|
||||
out: Dict[str, Any] = {"workload": workload, "bars": bars, "engines": order}
|
||||
failed: Dict[str, str] = {}
|
||||
for name in order:
|
||||
out[name] = _probe("memory", ENGINES[name].code, workload, bars)
|
||||
# An engine past its ceiling is measured AT its ceiling, and its record
|
||||
# carries the bar count it was measured at. Memory added by a run is not
|
||||
# proportional to the series for an engine that streams (Nautilus adds a
|
||||
# few MB at 100,000 and at 2,000,000 bars alike), so the figure is
|
||||
# reported as measured, and the report prints the length beside it.
|
||||
limit = ceiling(name, CEILING_OVERRIDES)
|
||||
at = bars if limit is None else min(bars, limit)
|
||||
result = _probe_or_none("memory", name, workload, at, failed)
|
||||
if result is not None:
|
||||
out[name] = result
|
||||
out["engines"] = [name for name in order if name in out]
|
||||
if failed:
|
||||
out["failed"] = failed
|
||||
return out
|
||||
|
||||
|
||||
@@ -464,23 +538,60 @@ SWEEP_CHILD = os.path.join(os.path.dirname(os.path.abspath(__file__)), "sweep_ch
|
||||
|
||||
|
||||
def parse_point(spec: str) -> tuple:
|
||||
"""`bars:combos[:oos]`, e.g. `20000:5000` or `20000:100000:oos`.
|
||||
"""`bars:combos[:oos[=engine,engine]]`, e.g. `20000:5000` or `20000:20000:oos`.
|
||||
|
||||
The `oos` suffix declares the array-materialising engines out of scope for
|
||||
that point: it is for grids whose vectorbt side needs tens of gigabytes,
|
||||
where running it anyway would time the swap file rather than the engine.
|
||||
The `oos` suffix declares engines out of scope for that point. Bare `oos`
|
||||
means vectorbt, which is what it always meant: it is for grids whose
|
||||
vectorbt side needs tens of gigabytes, where running it anyway would time the
|
||||
swap file rather than the engine. `oos=vectorbt,raptorbt` names the engines
|
||||
explicitly. An engine with no sweep API is also held back on its own, by its
|
||||
sweep ceiling (see engines.py), with no marker needed.
|
||||
|
||||
Returns ``(bars, combos, out_of_scope)`` with ``out_of_scope`` a tuple of
|
||||
engine names.
|
||||
"""
|
||||
parts = spec.split(":")
|
||||
if len(parts) not in (2, 3) or not parts[1]:
|
||||
raise argparse.ArgumentTypeError(
|
||||
f"expected bars:combos or bars:combos:oos, got {spec!r}")
|
||||
mode = "run"
|
||||
f"expected bars:combos or bars:combos:oos[=engines], got {spec!r}")
|
||||
out_of_scope: tuple = ()
|
||||
if len(parts) == 3:
|
||||
if parts[2] != "oos":
|
||||
head, _, names = parts[2].partition("=")
|
||||
if head != "oos":
|
||||
raise argparse.ArgumentTypeError(
|
||||
f"third field must be 'oos', got {parts[2]!r}")
|
||||
mode = "oos"
|
||||
return int(parts[0]), int(parts[1]), mode
|
||||
f"third field must be 'oos' or 'oos=engine,engine', got {parts[2]!r}")
|
||||
out_of_scope = tuple(n for n in names.split(",") if n) or ("vectorbt",)
|
||||
unknown = [n for n in out_of_scope if n not in ENGINES or n == REFERENCE]
|
||||
if unknown:
|
||||
raise argparse.ArgumentTypeError(
|
||||
f"cannot declare {unknown} out of scope: pick from "
|
||||
f"{sorted(CHALLENGERS)}")
|
||||
return int(parts[0]), int(parts[1]), out_of_scope
|
||||
|
||||
|
||||
def _sweep_out_of_scope(combos: int, declared: tuple, active: List[str]) -> List[str]:
|
||||
"""Engines not run on a grid this size: declared (`:oos`) or held back.
|
||||
|
||||
An engine with no sweep API is held back on its own by its sweep ceiling: a
|
||||
grid bigger than what a loop of full backtests can finish is not run whether
|
||||
or not the point says so, so a recipe that only says `:oos` cannot send an
|
||||
engine into hours of work.
|
||||
"""
|
||||
out = []
|
||||
for name in active:
|
||||
limit = sweep_ceiling(name, SWEEP_CEILING_OVERRIDES)
|
||||
if name in declared or (limit is not None and combos > limit):
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
|
||||
def _loop_limit_args(oos: List[str]) -> List[str]:
|
||||
"""Tell the child the sweep ceiling that held an engine back, to quote it."""
|
||||
for name in oos:
|
||||
limit = sweep_ceiling(name, SWEEP_CEILING_OVERRIDES)
|
||||
if limit is not None:
|
||||
return ["--loop-limit", str(limit)]
|
||||
return []
|
||||
|
||||
|
||||
def sweep(points: List[tuple], reps: int, active: List[str]) -> List[Dict[str, Any]]:
|
||||
@@ -493,12 +604,14 @@ def sweep(points: List[tuple], reps: int, active: List[str]) -> List[Dict[str, A
|
||||
it. A dead child is recorded as a crashed point and the run continues.
|
||||
"""
|
||||
out: List[Dict[str, Any]] = []
|
||||
challengers = [n for n in active if n != REFERENCE]
|
||||
for bars, combos, mode in points:
|
||||
for bars, combos, out_of_scope in points:
|
||||
oos = _sweep_out_of_scope(combos, out_of_scope, active)
|
||||
proc = subprocess.run(
|
||||
[sys.executable, SWEEP_CHILD, "--bars", str(bars),
|
||||
"--combos", str(combos), "--reps", str(reps),
|
||||
"--engines", *active, "--vectorbt", mode],
|
||||
"--engines", *active,
|
||||
*(["--out-of-scope", *oos] if oos else []),
|
||||
*_loop_limit_args(oos)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
payload = None
|
||||
@@ -511,7 +624,7 @@ def sweep(points: List[tuple], reps: int, active: List[str]) -> List[Dict[str, A
|
||||
payload = {
|
||||
"bars": bars,
|
||||
"combos": combos,
|
||||
"vectorbt_mode": mode,
|
||||
"out_of_scope_engines": oos,
|
||||
"status": "crashed",
|
||||
"reason": (proc.stderr or proc.stdout or "no output")[-400:].strip(),
|
||||
"exit_code": proc.returncode,
|
||||
@@ -523,14 +636,19 @@ def sweep(points: List[tuple], reps: int, active: List[str]) -> List[Dict[str, A
|
||||
# engine's call is the whole process, the other engines' allocations
|
||||
# included. Measured both ways, manifoldbt read 6.7 GB interleaved
|
||||
# against 94 MB alone, at 5000 combinations.
|
||||
payload["memory"] = _sweep_memory(bars, combos, mode, active)
|
||||
payload["memory"] = _sweep_memory(bars, combos, oos, active)
|
||||
out.append(payload)
|
||||
return out
|
||||
|
||||
|
||||
def _sweep_memory(bars: int, combos: int, mode: str, active: List[str]) -> Dict[str, Any]:
|
||||
"""Peak each engine adds running the grid once, one engine per process."""
|
||||
names = [REFERENCE] if mode == "oos" else list(active)
|
||||
def _sweep_memory(bars: int, combos: int, oos: List[str], active: List[str]) -> Dict[str, Any]:
|
||||
"""Peak each engine adds running the grid once, one engine per process.
|
||||
|
||||
An engine declared out of scope for the point is not run here either: the
|
||||
reason it is out of scope (memory, or hours of wall time) applies to this
|
||||
probe as much as to the timing.
|
||||
"""
|
||||
names = [n for n in active if n not in oos]
|
||||
added: Dict[str, Any] = {name: None for name in active}
|
||||
for name in names:
|
||||
proc = subprocess.run(
|
||||
@@ -591,6 +709,15 @@ def main() -> int:
|
||||
"is the only shape whose wall time is the engine's "
|
||||
"own rather than the slowest challenger's.")
|
||||
parser.add_argument("--reps", type=int, default=7)
|
||||
parser.add_argument("--engine-max-bars", nargs="*", default=[], metavar="ENGINE=BARS",
|
||||
help="override an engine's default ceiling, e.g. "
|
||||
"nautilus_trader=1000000, or nautilus_trader=0 for none. "
|
||||
"A ceiling is a time budget (see engines.py): an "
|
||||
"event-driven engine is not run on a series it would "
|
||||
"need hours to repeat, and the report says so")
|
||||
parser.add_argument("--engine-max-combos", nargs="*", default=[], metavar="ENGINE=COMBOS",
|
||||
help="override the largest grid an engine with no sweep API is "
|
||||
"looped over, e.g. nautilus_trader=500, or =0 for none")
|
||||
parser.add_argument("--out", default="results.json")
|
||||
parser.add_argument("--workdir", default=None, help="where the engine store is built")
|
||||
parser.add_argument("--cold-start-reps", type=int, default=3,
|
||||
@@ -598,9 +725,11 @@ def main() -> int:
|
||||
parser.add_argument("--memory-bars", type=int, default=0,
|
||||
help="bars for the memory probe; 0 disables it")
|
||||
parser.add_argument("--sweep", type=parse_point, nargs="*", default=[],
|
||||
metavar="BARS:COMBOS",
|
||||
help="parameter-grid points, e.g. 20000:5000. Needs a "
|
||||
"licence: sweep timings are published from a "
|
||||
metavar="BARS:COMBOS[:oos[=ENGINES]]",
|
||||
help="parameter-grid points, e.g. 20000:5000, with ':oos' "
|
||||
"to declare vectorbt out of scope or "
|
||||
"':oos=vectorbt,nautilus_trader' to name the engines. "
|
||||
"Needs a licence: sweep timings are published from a "
|
||||
"licensed run only")
|
||||
parser.add_argument("--sweep-reps", type=int, default=3,
|
||||
help="repetitions per sweep point; fewer than --reps "
|
||||
@@ -611,6 +740,17 @@ def main() -> int:
|
||||
"sees; 0 leaves the machine alone")
|
||||
args = parser.parse_args()
|
||||
|
||||
for spec in args.engine_max_bars:
|
||||
name, _, value = spec.partition("=")
|
||||
if name not in ENGINES or not value.isdigit():
|
||||
parser.error("--engine-max-bars wants ENGINE=BARS, got {!r}".format(spec))
|
||||
CEILING_OVERRIDES[name] = int(value)
|
||||
for spec in args.engine_max_combos:
|
||||
name, _, value = spec.partition("=")
|
||||
if name not in ENGINES or not value.isdigit():
|
||||
parser.error("--engine-max-combos wants ENGINE=COMBOS, got {!r}".format(spec))
|
||||
SWEEP_CEILING_OVERRIDES[name] = int(value)
|
||||
|
||||
if not engines_mod.installed(REFERENCE):
|
||||
print("cannot run: the reference engine ({}) is not installed".format(REFERENCE))
|
||||
return 2
|
||||
@@ -638,6 +778,14 @@ def main() -> int:
|
||||
env = environment()
|
||||
env["pinned_cores"] = pinned
|
||||
env["engines"] = active
|
||||
env["engine_max_bars"] = {
|
||||
name: ceiling(name, CEILING_OVERRIDES)
|
||||
for name in active if ceiling(name, CEILING_OVERRIDES) is not None
|
||||
}
|
||||
env["engine_max_sweep_combos"] = {
|
||||
name: sweep_ceiling(name, SWEEP_CEILING_OVERRIDES)
|
||||
for name in active if sweep_ceiling(name, SWEEP_CEILING_OVERRIDES) is not None
|
||||
}
|
||||
versions = env["versions"]
|
||||
print(" vs ".join(
|
||||
"{} {}".format(name, versions[ENGINES[name].distribution]) for name in active))
|
||||
|
||||
@@ -0,0 +1,492 @@
|
||||
"""NautilusTrader adapter.
|
||||
|
||||
NautilusTrader is an event-driven engine: it replays bars one at a time through a
|
||||
simulated exchange and calls a Python strategy on each of them. The other three
|
||||
engines in this harness are vectorised or compiled whole-series loops, so this is
|
||||
the one adapter where the *design* differs, not only the speed. What is compared
|
||||
is therefore the same workload, expressed the way a Nautilus user would write it:
|
||||
the low-level ``BacktestEngine``, a ``Strategy`` subclass, built-in indicators,
|
||||
market orders, and the venue's own matching engine.
|
||||
|
||||
What is timed, and what is not
|
||||
------------------------------
|
||||
Everything that can be prepared once is prepared before timing: the instrument,
|
||||
the ``Bar`` objects (``BarDataWrangler``), the venue, the account, the strategy,
|
||||
and the ``add_data`` call that hands the bars to the engine. That is the same
|
||||
courtesy the other adapters get (a prepared store, prepared Series, prepared
|
||||
arrays), and it is the data-loading exclusion the README states.
|
||||
|
||||
The timed closure is what a user repeats: ``engine.reset()``, ``engine.run()``,
|
||||
then reading the headline numbers off the engine. ``reset()`` followed by ``run()``
|
||||
is Nautilus's own way of replaying the same data again, and it is checked rather
|
||||
than assumed. The strategy clears its own indicators and state in ``on_reset``
|
||||
(without that, the second replay starts from the first one's moving averages and
|
||||
returns a different equity, which was measured), the parity gate reads the first
|
||||
call, and every later call is compared with it: a replay that drifted from the
|
||||
first would stop the run instead of timing a different computation.
|
||||
|
||||
Execution conventions, chosen to line up with the reference
|
||||
-----------------------------------------------------------
|
||||
* **Fill at the close of the signal bar.** The strategy submits a market order from
|
||||
``on_bar``; the simulated exchange has already moved the market to that bar's
|
||||
close, and the order is matched before the next bar arrives, so the fill lands on
|
||||
the same bar at the close. That is manifoldbt's ``signal_delay=0`` plus
|
||||
``execution_price="AtClose"``, with no setting needed for it.
|
||||
* **No volume-limited fills.** Nautilus builds its book from a bar by splitting the
|
||||
bar's volume over four synthetic updates (open, high, low, close) and uses that
|
||||
quantity as the liquidity available. A market order larger than a quarter of the
|
||||
bar's volume is *partially* filled and the remainder is booked one tick worse.
|
||||
The generated bars carry a random volume between 100 and 10,000 units, so with
|
||||
full-equity sizing (about 1,000 units) that rule fires on about four
|
||||
full-equity orders in ten. The other three engines have no volume constraint at all, so the venue is
|
||||
given a fill model (``_BenchFillModel``) that answers every market order from a
|
||||
book with effectively unlimited depth at the bar's close. That is the same
|
||||
mechanism Nautilus ships for it (``BestPriceFillModel``), written out here
|
||||
because the bracket workload needs stop and limit orders left on the standard
|
||||
logic. ``BestPriceFillModel`` prices every order from the best bid and ask,
|
||||
stops and limits included, and with it ``bracket_sl_tp`` ends 6.2e-3 of capital
|
||||
away from the run that fills them at their own levels (measured at 20,000 bars).
|
||||
* **Slippage** in basis points is not a Nautilus setting (``prob_slippage`` moves a
|
||||
fill by one *tick*, at random). The fill model shifts the book instead: asks sit
|
||||
``slippage_bps`` above the close and bids ``slippage_bps`` below it, which is the
|
||||
documented extension point for custom slippage and produces exactly what
|
||||
``Slippage.fixed_bps`` produces on the reference.
|
||||
* **Fees** are the instrument's taker fee (market orders always take), charged on
|
||||
notional by the venue's default ``MakerTakerFeeModel``.
|
||||
* **Account and sizing.** A margin account with leverage 1 in a high-precision
|
||||
quote currency (USDT, 8 decimals). Not USD: USD carries 2 decimals, and Nautilus
|
||||
rounds each realised PnL to the currency's precision. Measured with a USD quote,
|
||||
``sma_cross`` ends 2.8e-7 of capital from the reference at 20,000 bars and
|
||||
2.1e-7 at 100,000, against a gate of 1e-9. Prices and quantities are fixed point with 9
|
||||
decimals on both platforms (the standard-precision Windows wheel allows 9, the
|
||||
Linux wheel 16, and using the same on both keeps the results identical to the
|
||||
digit); the rounding that introduces is 1e-12 to 2e-10 of capital depending on
|
||||
how often the strategy trades, and is reported by the gate, not hidden.
|
||||
``FractionOfEquity`` is the cash balance at the signal bar divided by the close,
|
||||
rounded *down* to the quantity precision; units sizing is the fixed quantity.
|
||||
* **Indicators** are Nautilus's own ``SimpleMovingAverage``,
|
||||
``ExponentialMovingAverage`` and ``RelativeStrengthIndex`` (Wilder), registered
|
||||
on the bar type so the engine updates them before ``on_bar`` runs. The EMA
|
||||
matches the reference bit for bit and the SMA to ~1e-13. The RSI is seeded
|
||||
differently (see README.md, "The RSI seed"): it agrees to the last bit after
|
||||
~800 bars and to within 4.5e-2 before that, and the parity gate is what says
|
||||
whether that moved a decision on the benchmark data.
|
||||
|
||||
Brackets
|
||||
--------
|
||||
The stop and the target are Nautilus's native contingent orders
|
||||
(``OrderFactory.bracket``): a stop-market and a limit, linked one-cancels-other,
|
||||
armed when the market entry fills. Their levels are computed from the signal bar's
|
||||
close, which is the fill price of the entry. The venue's standard logic decides the
|
||||
fills: a stop that the bar trades through fills at its trigger price, a limit that
|
||||
the bar reaches fills at its limit price.
|
||||
|
||||
Nautilus runs the exchange *before* it hands the bar to the strategy, so a stop
|
||||
that fires intrabar leaves the account flat by the time ``on_bar`` runs, and if the
|
||||
entry condition still holds that same bar's close, the strategy enters again on
|
||||
that bar. That is the two-orders-on-the-exit-bar behaviour of the reference, not
|
||||
vectorbt's "next bar" and not raptorbt's "wait for a new crossing". The one thing
|
||||
that still differs is how the re-entry is sized, which is why the workload is
|
||||
``documented`` for this engine (see ``workloads.py`` and ``diagnose`` below).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.metadata as md
|
||||
from decimal import Decimal
|
||||
from typing import Any, Callable, Dict, List
|
||||
|
||||
import data as data_mod
|
||||
from nautilus_trader.backtest.engine import BacktestEngine
|
||||
from nautilus_trader.backtest.models import FillModel
|
||||
from nautilus_trader.config import BacktestEngineConfig, LoggingConfig
|
||||
from nautilus_trader.indicators import (
|
||||
ExponentialMovingAverage,
|
||||
RelativeStrengthIndex,
|
||||
SimpleMovingAverage,
|
||||
)
|
||||
from nautilus_trader.indicators.averages import MovingAverageType
|
||||
from nautilus_trader.model.book import OrderBook
|
||||
from nautilus_trader.model.currencies import BTC, USDT
|
||||
from nautilus_trader.model.data import BarType, BookOrder
|
||||
from nautilus_trader.model.enums import AccountType, BookType, OmsType, OrderSide, OrderType
|
||||
from nautilus_trader.model.identifiers import InstrumentId, Symbol, Venue
|
||||
from nautilus_trader.model.instruments import CurrencyPair
|
||||
from nautilus_trader.model.objects import Money, Price, Quantity
|
||||
from nautilus_trader.persistence.wranglers import BarDataWrangler
|
||||
from nautilus_trader.trading.strategy import Strategy, StrategyConfig
|
||||
|
||||
from workloads import CAPITAL, WORKLOADS
|
||||
|
||||
NAME = "nautilus_trader"
|
||||
|
||||
VENUE = Venue("BENCH")
|
||||
QUOTE = USDT
|
||||
PRICE_PRECISION = 9
|
||||
SIZE_PRECISION = 9
|
||||
|
||||
# Depth the fill model offers at the close. The largest order any workload places
|
||||
# is full equity at the lowest price a 10M-bar random walk reaches (~20,000
|
||||
# units), so a million is "unlimited" with room to spare, as in Nautilus's own
|
||||
# BestPriceFillModel.
|
||||
UNLIMITED = 1_000_000
|
||||
|
||||
|
||||
def probe() -> Dict[str, Any]:
|
||||
# From the installed metadata, as for raptorbt: the version a benchmark
|
||||
# reports should be the one pip resolved, not a constant in an __init__.
|
||||
try:
|
||||
version = md.version("nautilus_trader")
|
||||
except Exception:
|
||||
import nautilus_trader
|
||||
|
||||
version = getattr(nautilus_trader, "__version__", "unknown")
|
||||
return {"engine": NAME, "version": version}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The venue's view of the market
|
||||
# --------------------------------------------------------------------------- #
|
||||
class _BenchFillModel(FillModel):
|
||||
"""Unlimited depth at the close for market orders, standard logic otherwise.
|
||||
|
||||
``get_orderbook_for_fill_simulation`` is called by the matching engine for
|
||||
every fill. Returning an ``OrderBook`` restricts the fill to that book's
|
||||
liquidity; returning ``None`` leaves it on the venue's standard logic, which is
|
||||
what stop and limit orders get here: their fill price is the trigger or limit
|
||||
price the bar crossed, and that is the behaviour being compared.
|
||||
"""
|
||||
|
||||
def __init__(self, slippage_bps: float = 0.0) -> None:
|
||||
super().__init__()
|
||||
self._shift = float(slippage_bps) / 10_000.0
|
||||
|
||||
def get_orderbook_for_fill_simulation(self, instrument, order, best_bid, best_ask):
|
||||
if order.order_type != OrderType.MARKET or best_bid is None or best_ask is None:
|
||||
return None
|
||||
if self._shift:
|
||||
bid = instrument.make_price(best_bid.as_double() * (1.0 - self._shift))
|
||||
ask = instrument.make_price(best_ask.as_double() * (1.0 + self._shift))
|
||||
else:
|
||||
bid, ask = best_bid, best_ask
|
||||
depth = Quantity(UNLIMITED, instrument.size_precision)
|
||||
book = OrderBook(instrument.id, BookType.L2_MBP)
|
||||
book.add(BookOrder(OrderSide.BUY, bid, depth, 1), 0, 0)
|
||||
book.add(BookOrder(OrderSide.SELL, ask, depth, 2), 0, 0)
|
||||
return book
|
||||
|
||||
|
||||
def _instrument(symbol: str, fee_bps: float) -> CurrencyPair:
|
||||
fee = Decimal(repr(fee_bps)) / Decimal(10_000)
|
||||
return CurrencyPair(
|
||||
instrument_id=InstrumentId(Symbol(symbol), VENUE),
|
||||
raw_symbol=Symbol(symbol),
|
||||
base_currency=BTC,
|
||||
quote_currency=QUOTE,
|
||||
price_precision=PRICE_PRECISION,
|
||||
size_precision=SIZE_PRECISION,
|
||||
price_increment=Price(10 ** -PRICE_PRECISION, PRICE_PRECISION),
|
||||
size_increment=Quantity(10 ** -SIZE_PRECISION, SIZE_PRECISION),
|
||||
ts_event=0,
|
||||
ts_init=0,
|
||||
maker_fee=fee,
|
||||
taker_fee=fee,
|
||||
# No margin requirement: the account is leverage 1 and the strategy never
|
||||
# asks for more than its cash, so there is nothing for the risk engine to
|
||||
# veto and nothing for it to round.
|
||||
margin_init=Decimal(0),
|
||||
margin_maint=Decimal(0),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The strategy
|
||||
# --------------------------------------------------------------------------- #
|
||||
class _Leg:
|
||||
"""Indicators and state of one instrument."""
|
||||
|
||||
__slots__ = ("instrument", "bar_type", "fast", "slow", "rsi", "long")
|
||||
|
||||
def __init__(self, instrument, bar_type, p, kind):
|
||||
self.instrument = instrument
|
||||
self.bar_type = bar_type
|
||||
if kind == "ema_rsi":
|
||||
self.fast = ExponentialMovingAverage(p["fast"])
|
||||
self.slow = ExponentialMovingAverage(p["slow"])
|
||||
self.rsi = RelativeStrengthIndex(p["rsi_period"], MovingAverageType.WILDER)
|
||||
else:
|
||||
self.fast = SimpleMovingAverage(p["fast"])
|
||||
self.slow = SimpleMovingAverage(p["slow"])
|
||||
self.rsi = None
|
||||
self.long = False
|
||||
|
||||
def reset(self):
|
||||
self.fast.reset()
|
||||
self.slow.reset()
|
||||
if self.rsi is not None:
|
||||
self.rsi.reset()
|
||||
self.long = False
|
||||
|
||||
|
||||
class _StratConfig(StrategyConfig, frozen=True):
|
||||
pass
|
||||
|
||||
|
||||
class _Bench(Strategy):
|
||||
def __init__(self, legs: List[_Leg], p: Dict[str, Any], kind: str) -> None:
|
||||
super().__init__(_StratConfig())
|
||||
self._legs = {leg.bar_type: leg for leg in legs}
|
||||
self._by_instrument = {leg.instrument.id: leg for leg in legs}
|
||||
self._kind = kind
|
||||
self._units = p.get("units")
|
||||
self._alloc = p.get("alloc", 1.0)
|
||||
self._sl = p["sl_pct"] / 100.0 if "sl_pct" in p else None
|
||||
self._tp = p["tp_pct"] / 100.0 if "tp_pct" in p else None
|
||||
self._rsi_lo = p.get("rsi_lo", 30.0) / 100.0
|
||||
self._rsi_hi = p.get("rsi_hi", 70.0) / 100.0
|
||||
self.fills = 0
|
||||
self.round_trips = 0
|
||||
self.fees = 0.0
|
||||
# Only the untimed diagnosis switches this on.
|
||||
self.journal = None
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
def on_start(self):
|
||||
for leg in self._legs.values():
|
||||
self.register_indicator_for_bars(leg.bar_type, leg.fast)
|
||||
self.register_indicator_for_bars(leg.bar_type, leg.slow)
|
||||
if leg.rsi is not None:
|
||||
self.register_indicator_for_bars(leg.bar_type, leg.rsi)
|
||||
self.subscribe_bars(leg.bar_type)
|
||||
|
||||
def on_reset(self):
|
||||
for leg in self._legs.values():
|
||||
leg.reset()
|
||||
self.fills = 0
|
||||
self.round_trips = 0
|
||||
self.fees = 0.0
|
||||
|
||||
# -- the strategy ------------------------------------------------------
|
||||
def on_bar(self, bar):
|
||||
leg = self._legs[bar.bar_type]
|
||||
if self._kind == "ema_rsi":
|
||||
level = (
|
||||
leg.rsi.initialized
|
||||
and leg.fast.value > leg.slow.value
|
||||
and self._rsi_lo < leg.rsi.value < self._rsi_hi
|
||||
)
|
||||
else:
|
||||
level = leg.slow.initialized and leg.fast.value > leg.slow.value
|
||||
|
||||
if level:
|
||||
if not leg.long:
|
||||
self._enter(leg, bar)
|
||||
elif leg.long:
|
||||
self._exit(leg)
|
||||
|
||||
def _enter(self, leg, bar):
|
||||
instrument = leg.instrument
|
||||
close = bar.close.as_double()
|
||||
if self._units is not None:
|
||||
quantity = instrument.make_qty(self._units)
|
||||
else:
|
||||
cash = self.portfolio.account(VENUE).balance_total(QUOTE).as_double()
|
||||
quantity = instrument.make_qty(self._alloc * cash / close, round_down=True)
|
||||
|
||||
if self._sl is not None:
|
||||
orders = self.order_factory.bracket(
|
||||
instrument.id,
|
||||
OrderSide.BUY,
|
||||
quantity,
|
||||
sl_trigger_price=instrument.make_price(close * (1.0 - self._sl)),
|
||||
tp_price=instrument.make_price(close * (1.0 + self._tp)),
|
||||
)
|
||||
self.submit_order_list(orders)
|
||||
else:
|
||||
self.submit_order(self.order_factory.market(instrument.id, OrderSide.BUY, quantity))
|
||||
leg.long = True
|
||||
|
||||
def _exit(self, leg):
|
||||
if self._sl is not None:
|
||||
# The bracket's stop and target are still resting when the signal
|
||||
# turns off, and they have to go with the position.
|
||||
self.cancel_all_orders(leg.instrument.id)
|
||||
self.close_all_positions(leg.instrument.id)
|
||||
leg.long = False
|
||||
|
||||
# -- accounting --------------------------------------------------------
|
||||
def on_order_filled(self, event):
|
||||
self.fills += 1
|
||||
self.fees += event.commission.as_double()
|
||||
bought = event.order_side == OrderSide.BUY
|
||||
self._by_instrument[event.instrument_id].long = bought
|
||||
if self.journal is not None:
|
||||
self.journal.append((event.ts_event, bought, float(event.last_qty)))
|
||||
|
||||
def on_position_closed(self, event):
|
||||
# A round trip is a position that went flat. Counting sell fills instead
|
||||
# would count a stop that the venue filled in two pieces twice.
|
||||
self.round_trips += 1
|
||||
|
||||
|
||||
def _bar_type(symbol: str) -> BarType:
|
||||
return BarType.from_str("{}.{}-1-MINUTE-LAST-EXTERNAL".format(symbol, VENUE.value))
|
||||
|
||||
|
||||
def _frames(key: str, df) -> Dict[str, Any]:
|
||||
"""Symbol -> frame. The universe is the reference's: same generator, same seed."""
|
||||
assets = int(WORKLOADS[key].params.get("assets", 1))
|
||||
if assets > 1:
|
||||
return {"A%d" % sid: frame for sid, frame in data_mod.make_universe(len(df), assets).items()}
|
||||
return {"BENCH": df}
|
||||
|
||||
|
||||
def _build(key: str, df):
|
||||
p = WORKLOADS[key].params
|
||||
kind = "ema_rsi" if key == "ema_rsi_fees" else "sma"
|
||||
fee_bps = float(p.get("fee_bps", 0.0))
|
||||
|
||||
legs, bars = [], []
|
||||
for symbol, frame in _frames(key, df).items():
|
||||
instrument = _instrument(symbol, fee_bps)
|
||||
bar_type = _bar_type(symbol)
|
||||
bars.append((instrument, BarDataWrangler(bar_type, instrument).process(
|
||||
frame.set_index("timestamp")[["open", "high", "low", "close", "volume"]])))
|
||||
legs.append(_Leg(instrument, bar_type, p, kind))
|
||||
|
||||
engine = BacktestEngine(config=BacktestEngineConfig(
|
||||
# Errors only: at INFO every order event is formatted and written, which
|
||||
# is a cost of the log and not of the engine, and no benchmark of a
|
||||
# simulator should be timing its own console.
|
||||
logging=LoggingConfig(log_level="ERROR"),
|
||||
))
|
||||
engine.add_venue(
|
||||
venue=VENUE,
|
||||
oms_type=OmsType.NETTING,
|
||||
account_type=AccountType.MARGIN,
|
||||
base_currency=QUOTE,
|
||||
default_leverage=Decimal(1),
|
||||
starting_balances=[Money(CAPITAL, QUOTE)],
|
||||
fill_model=_BenchFillModel(float(p.get("slippage_bps", 0.0))),
|
||||
)
|
||||
for instrument, leg_bars in bars:
|
||||
engine.add_instrument(instrument)
|
||||
engine.add_data(leg_bars)
|
||||
|
||||
strategy = _Bench(legs, p, kind)
|
||||
engine.add_strategy(strategy)
|
||||
return engine, strategy
|
||||
|
||||
|
||||
def _final_equity(engine) -> float:
|
||||
"""Cash plus the open position marked at the last price, in the quote currency."""
|
||||
portfolio = engine.portfolio
|
||||
cash = portfolio.account(VENUE).balance_total(QUOTE).as_double()
|
||||
unrealized = portfolio.unrealized_pnls(VENUE).get(QUOTE)
|
||||
return cash + (unrealized.as_double() if unrealized is not None else 0.0)
|
||||
|
||||
|
||||
def prepare(key: str, df, workdir: str | None = None) -> Callable[[], Dict[str, Any]]:
|
||||
"""Untimed setup; returns the closure the harness times."""
|
||||
p = WORKLOADS[key].params
|
||||
wants_metrics = bool(p.get("metrics"))
|
||||
engine, strategy = _build(key, df)
|
||||
|
||||
first: List[tuple] = []
|
||||
|
||||
def run() -> Dict[str, Any]:
|
||||
engine.reset()
|
||||
engine.run()
|
||||
final_equity = _final_equity(engine)
|
||||
# The gate reads the first call only, so every replay after it is held to
|
||||
# the first one here. Both are deterministic, and a difference means the
|
||||
# reset left state behind: fail loudly rather than time another run.
|
||||
outcome = (final_equity, strategy.round_trips, strategy.fills)
|
||||
if not first:
|
||||
first.append(outcome)
|
||||
elif outcome != first[0]:
|
||||
raise RuntimeError(
|
||||
"nautilus_trader replay drifted from the first run: {} against {}. "
|
||||
"engine.reset() left state behind, so the timed runs would not be "
|
||||
"the run the parity gate checked.".format(outcome, first[0]))
|
||||
out = {
|
||||
"total_return": final_equity / CAPITAL - 1.0,
|
||||
"final_equity": final_equity,
|
||||
"round_trips": strategy.round_trips,
|
||||
"fills": strategy.fills,
|
||||
"total_fees": strategy.fees,
|
||||
}
|
||||
if wants_metrics:
|
||||
stats = engine.get_result().stats_returns
|
||||
out.update({
|
||||
# Nautilus has no drawdown statistic. Left absent rather than
|
||||
# rebuilt from an equity curve the harness would have to sample
|
||||
# bar by bar: this column shows what the engine hands a user.
|
||||
"max_drawdown": None,
|
||||
"sharpe": stats.get("Sharpe Ratio (252 days)"),
|
||||
"sortino": stats.get("Sortino Ratio (252 days)"),
|
||||
"volatility": stats.get("Returns Volatility (252 days)"),
|
||||
})
|
||||
return out
|
||||
|
||||
return run
|
||||
|
||||
|
||||
def diagnose(key: str, df, workdir: str | None = None) -> Dict[str, Any]:
|
||||
"""Untimed measurement of how far the bracket divergence goes on this side.
|
||||
|
||||
Entries and exits are the same trades at the same prices as the reference's.
|
||||
What differs is the quantity of a re-entry on the exit bar, so that is what is
|
||||
counted: how many of them there are, and how far each one's quantity sits from
|
||||
the position that was just closed on the same bar (which is the quantity the
|
||||
reference re-enters with, because it sizes from the equity at the close).
|
||||
"""
|
||||
note = WORKLOADS[key].notes.get(NAME)
|
||||
if note is None or note.status != "documented":
|
||||
return {}
|
||||
engine, strategy = _build(key, df)
|
||||
strategy.journal = []
|
||||
engine.reset()
|
||||
engine.run()
|
||||
|
||||
sold: Dict[int, float] = {}
|
||||
bought: Dict[int, float] = {}
|
||||
for ts, is_buy, quantity in strategy.journal:
|
||||
book = bought if is_buy else sold
|
||||
book[ts] = book.get(ts, 0.0) + quantity
|
||||
gaps = sorted(abs(quantity - sold[ts]) / sold[ts] for ts, quantity in bought.items()
|
||||
if ts in sold and sold[ts] > 0.0)
|
||||
return {
|
||||
"round_trips": strategy.round_trips,
|
||||
"final_equity": _final_equity(engine),
|
||||
"reentries_on_exit_bar": len(gaps),
|
||||
"reentry_quantity_gap_median": gaps[len(gaps) // 2] if gaps else 0.0,
|
||||
"reentry_quantity_gap_max": gaps[-1] if gaps else 0.0,
|
||||
}
|
||||
|
||||
|
||||
def sweep_runner(df, fast_vals, slow_vals) -> Callable[[], Dict[tuple, float]]:
|
||||
"""A parameter grid the only way Nautilus spells one: a loop of backtests.
|
||||
|
||||
There is no fan-out entry point. The engine is built once, with the data in
|
||||
it, and each cell swaps in a new pair of moving averages, resets the engine
|
||||
and replays the same bars: the repeat-run path the engine documents, and the
|
||||
cheapest one, since it pays for the instrument, the venue and the data once
|
||||
rather than once per cell. Nothing is shared between cells beyond that: each
|
||||
cell recomputes its own averages inside the event loop, because that is where
|
||||
a Nautilus indicator lives.
|
||||
"""
|
||||
engine, strategy = _build("sma_cross", df)
|
||||
leg = next(iter(strategy._legs.values()))
|
||||
|
||||
def call() -> Dict[tuple, float]:
|
||||
out = {}
|
||||
for fast in fast_vals:
|
||||
for slow in slow_vals:
|
||||
leg.fast = SimpleMovingAverage(fast)
|
||||
leg.slow = SimpleMovingAverage(slow)
|
||||
engine.reset()
|
||||
engine.run()
|
||||
out[(fast, slow)] = _final_equity(engine) / CAPITAL - 1.0
|
||||
return out
|
||||
|
||||
return call
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
One backtester is the *reference* and the others are *challengers*. Every parity
|
||||
check joins a challenger to the reference, never two challengers to each other:
|
||||
with three engines there are three pairs, and reporting all of them turns a
|
||||
with four engines there are six pairs, and reporting all of them turns a
|
||||
speed benchmark into a matrix nobody reads. manifoldbt is the reference because
|
||||
this harness ships with it, which is a statement about the plumbing and not a
|
||||
claim about the engine: the reference gets no advantage from the position, it is
|
||||
@@ -18,10 +18,23 @@ Ratio basis
|
||||
``ratio_basis`` records how an engine annualises Sharpe and Sortino. manifoldbt
|
||||
and the vectorbt adapter both compute them on daily returns annualised by
|
||||
sqrt(365), so their ratios are directly comparable and small differences are
|
||||
worth reporting. raptorbt returns its own, on its own basis, and comparing those
|
||||
numbers to manifoldbt's would report a units mismatch as a disagreement. The
|
||||
parity gate never depends on this either way: it gates on money, round-trips and
|
||||
drawdown, all three basis-free.
|
||||
worth reporting. raptorbt and NautilusTrader return their own, on their own
|
||||
bases, and comparing those numbers to manifoldbt's would report a units mismatch
|
||||
as a disagreement. The parity gate never depends on this either way: it gates on
|
||||
money, round-trips and, where an engine offers one, drawdown, all basis-free.
|
||||
|
||||
Ceilings
|
||||
--------
|
||||
``max_bars`` is the longest series an engine is run on by default, and it exists
|
||||
for one reason: an event-driven engine replays every bar through a simulated
|
||||
exchange and a Python callback, so its cost is seconds per 100,000 bars, where
|
||||
the others spend milliseconds. An interleaved repetition loop at 10 million bars
|
||||
would run for hours inside a job that has to finish. The ceiling is a time
|
||||
budget, not a verdict, and it is printed in the report with its reason rather
|
||||
than left as an empty cell. ``bench.py --engine-max-bars`` lifts it for a local
|
||||
run. ``max_sweep_combos`` is the same idea for a parameter grid: an engine with
|
||||
no sweep API loops a full backtest per cell, and a grid of thousands of cells at
|
||||
seconds each is hours.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -40,6 +53,15 @@ class Engine:
|
||||
ratio_basis: str
|
||||
# PyPI distribution name, for the version stamp in the result envelope.
|
||||
distribution: str
|
||||
# Longest series (in bars) this engine is run on by default; None for no
|
||||
# ceiling. See the module docstring.
|
||||
max_bars: int | None = None
|
||||
# What the report says where the ceiling stopped a run.
|
||||
ceiling_why: str = ""
|
||||
# Largest parameter grid (in combinations) this engine is looped over by
|
||||
# default, or None. Only an engine with no sweep API needs one: its grid is a
|
||||
# loop of full backtests, so the cost is combinations times one backtest.
|
||||
max_sweep_combos: int | None = None
|
||||
|
||||
|
||||
REFERENCE = "manifoldbt"
|
||||
@@ -50,6 +72,20 @@ ENGINES: Dict[str, Engine] = {
|
||||
Engine("manifoldbt", "mbt", "engine_mbt", "daily_365", "manifoldbt"),
|
||||
Engine("vectorbt", "vbt", "engine_vbt", "daily_365", "vectorbt"),
|
||||
Engine("raptorbt", "rbt", "engine_rbt", "native", "raptorbt"),
|
||||
Engine(
|
||||
"nautilus_trader", "nt", "engine_nt", "native", "nautilus_trader",
|
||||
max_bars=100_000,
|
||||
max_sweep_combos=64,
|
||||
ceiling_why=(
|
||||
"Not run above {limit:,} bars. NautilusTrader is event-driven: every "
|
||||
"bar goes through a simulated exchange and a Python strategy "
|
||||
"callback, so one call costs seconds per 100,000 bars and grows "
|
||||
"linearly with the series. Repeated and interleaved, a longer series "
|
||||
"does not fit the time budget of the job. The measured time at the "
|
||||
"ceiling is in the table, and `bench.py --engine-max-bars "
|
||||
"nautilus_trader=N` lifts the ceiling for a local run."
|
||||
),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -79,3 +115,20 @@ def installed(name: str) -> bool:
|
||||
def present(names: List[str]) -> List[str]:
|
||||
"""Filter to the engines actually importable here, preserving order."""
|
||||
return [n for n in names if installed(n)]
|
||||
|
||||
|
||||
def sweep_ceiling(name: str, overrides: Dict[str, int] | None = None) -> int | None:
|
||||
"""The largest grid (combinations) this engine is looped over; 0 removes it."""
|
||||
if overrides and name in overrides:
|
||||
return overrides[name] or None
|
||||
return ENGINES[name].max_sweep_combos
|
||||
|
||||
|
||||
def ceiling(name: str, overrides: Dict[str, int] | None = None) -> int | None:
|
||||
"""The longest series this engine is run on, honouring a command-line override.
|
||||
|
||||
An override of 0 removes the ceiling.
|
||||
"""
|
||||
if overrides and name in overrides:
|
||||
return overrides[name] or None
|
||||
return ENGINES[name].max_bars
|
||||
|
||||
@@ -71,14 +71,24 @@ def compare(ref: Dict[str, Any], other: Dict[str, Any], key: str, engine: str) -
|
||||
# buckets its daily returns slightly differently from vectorbt, a difference
|
||||
# worth stating rather than hiding, and worth nothing at all as an argument
|
||||
# about speed.
|
||||
if other.get("max_drawdown") is not None and ref.get("max_drawdown") is not None:
|
||||
diffs["max_drawdown_rel"] = _rel(ref["max_drawdown"], other["max_drawdown"])
|
||||
agrees = agrees and diffs["max_drawdown_rel"] <= REL_TOL
|
||||
#
|
||||
# An engine with no drawdown statistic of its own (NautilusTrader reports
|
||||
# volatility, Sharpe and Sortino and stops there) hands over ``None``. The
|
||||
# harness does not rebuild one from an equity curve for it: the summary
|
||||
# column is meant to show what the engine gives a user, and that gate then
|
||||
# runs on money and round-trips only, which is said out loud in the report.
|
||||
if ref.get("max_drawdown") is not None and "max_drawdown" in other:
|
||||
same_basis = ENGINES[engine].ratio_basis == ENGINES[REFERENCE].ratio_basis
|
||||
if other["max_drawdown"] is not None:
|
||||
diffs["max_drawdown_rel"] = _rel(ref["max_drawdown"], other["max_drawdown"])
|
||||
agrees = agrees and diffs["max_drawdown_rel"] <= REL_TOL
|
||||
else:
|
||||
diffs["max_drawdown_absent"] = True
|
||||
# Only between engines that annualise the same way. raptorbt returns its
|
||||
# ratios on its own basis, and subtracting those from the reference's
|
||||
# would publish a units mismatch as a disagreement (measured: Sharpe
|
||||
# 0.21 against 8.14 on a run whose equity curve is bit-identical).
|
||||
if ENGINES[engine].ratio_basis == ENGINES[REFERENCE].ratio_basis:
|
||||
if same_basis:
|
||||
diffs["advisory_ratio_rel"] = {
|
||||
name: _rel(ref[name], other[name])
|
||||
for name in ("sharpe", "sortino", "volatility")
|
||||
|
||||
@@ -201,15 +201,23 @@ def _summary_cost(exact: List[Dict[str, Any]], payload: Dict[str, Any]) -> List[
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
top = summarised[shared[-1]]
|
||||
advisory = {}
|
||||
basis_differs = {}
|
||||
for engine, verdict in top["parity"].items():
|
||||
diffs = verdict.get("diffs", {})
|
||||
if diffs.get("advisory_ratio_rel"):
|
||||
advisory[engine] = max(diffs["advisory_ratio_rel"].values())
|
||||
elif diffs.get("ratio_basis"):
|
||||
basis_differs[engine] = diffs.get("max_drawdown_rel")
|
||||
no_drawdown = set()
|
||||
# Each engine is read at the longest series it ran on: one held back by its
|
||||
# ceiling has no verdict at the longest size, and reading only that row would
|
||||
# leave it out of the sentence.
|
||||
for bars in reversed(shared):
|
||||
for engine, verdict in summarised[bars]["parity"].items():
|
||||
if engine in advisory or engine in basis_differs:
|
||||
continue
|
||||
diffs = verdict.get("diffs", {})
|
||||
if diffs.get("max_drawdown_absent"):
|
||||
no_drawdown.add(engine)
|
||||
if diffs.get("advisory_ratio_rel"):
|
||||
advisory[engine] = max(diffs["advisory_ratio_rel"].values())
|
||||
elif diffs.get("ratio_basis"):
|
||||
basis_differs[engine] = diffs.get("max_drawdown_rel")
|
||||
if advisory:
|
||||
lines += [
|
||||
"Gated on total return, round-trips and max drawdown (exact). Sharpe, "
|
||||
@@ -217,6 +225,17 @@ def _summary_cost(exact: List[Dict[str, Any]], payload: Dict[str, Any]) -> List[
|
||||
"",
|
||||
]
|
||||
for engine, drawdown_rel in basis_differs.items():
|
||||
if engine in no_drawdown:
|
||||
# Not "it was not compared": it does not exist. The engine offers
|
||||
# volatility, Sharpe and Sortino and no drawdown statistic, and the
|
||||
# harness does not build one for it.
|
||||
lines += [
|
||||
"{e} annualises its ratios on its own basis and reports no maximum "
|
||||
"drawdown, so this workload is gated on total return and "
|
||||
"round-trips only (exact).".format(e=engine),
|
||||
"",
|
||||
]
|
||||
continue
|
||||
if drawdown_rel is None:
|
||||
agreement = "was not compared"
|
||||
elif drawdown_rel == 0.0:
|
||||
@@ -235,12 +254,19 @@ def _side_measures(payload: Dict[str, Any], results: List[Dict[str, Any]]) -> Li
|
||||
cold = payload.get("cold_start")
|
||||
mem = payload.get("memory")
|
||||
engines = _engines(payload)
|
||||
threading_rows = [
|
||||
r for r in results
|
||||
if r.get("cpu_over_wall")
|
||||
and all(r["cpu_over_wall"].get(e) is not None for e in engines)
|
||||
]
|
||||
if not (cold or mem or threading_rows):
|
||||
|
||||
# CPU over wall time, per engine, at the longest series that engine ran and
|
||||
# produced a reading for. An engine held back by its ceiling has no reading at
|
||||
# the longest size, and requiring every engine at one size would drop the row
|
||||
# for all of them.
|
||||
threading: Dict[str, Any] = {}
|
||||
for engine in engines:
|
||||
rows = [r for r in results
|
||||
if (r.get("cpu_over_wall") or {}).get(engine) is not None]
|
||||
if rows:
|
||||
best = max(rows, key=lambda r: r["bars"])
|
||||
threading[engine] = (best["cpu_over_wall"][engine], best["bars"])
|
||||
if not (cold or mem or threading):
|
||||
return []
|
||||
|
||||
lines = [
|
||||
@@ -257,15 +283,38 @@ def _side_measures(payload: Dict[str, Any], results: List[Dict[str, Any]]) -> Li
|
||||
+ " | ".join("{:.2f} s".format(share[e]) if e in share else "-"
|
||||
for e in engines) + " |")
|
||||
if mem:
|
||||
lines.append("| RAM added by the run, per 1M bars | " + " | ".join(
|
||||
"{:.0f} MB".format(mem[e]["added_mb_per_million_bars"]) if e in mem else "-"
|
||||
for e in engines) + " |")
|
||||
if threading_rows:
|
||||
biggest = max(threading_rows, key=lambda r: r["bars"])
|
||||
ratio = biggest["cpu_over_wall"]
|
||||
lines.append("| CPU over wall time at {:,} bars | ".format(biggest["bars"])
|
||||
+ " | ".join("{:.2f}".format(ratio[e]) for e in engines) + " |")
|
||||
def memory_cell(engine: str) -> str:
|
||||
if engine not in mem or not isinstance(mem[engine], dict):
|
||||
return "-"
|
||||
probe = mem[engine]
|
||||
text = "{:.0f} MB".format(probe["added_mb_per_million_bars"])
|
||||
# An engine held back by its ceiling is measured at the ceiling, and
|
||||
# the figure is scaled to a million bars like the others. Say so
|
||||
# beside the number rather than leave a reader to assume the same
|
||||
# series length.
|
||||
if probe.get("bars") and probe["bars"] != mem.get("bars"):
|
||||
text += " (at {:,} bars)".format(probe["bars"])
|
||||
return text
|
||||
|
||||
lines.append("| RAM added by the run, per 1M bars | "
|
||||
+ " | ".join(memory_cell(e) for e in engines) + " |")
|
||||
if threading:
|
||||
sizes = {bars for _, bars in threading.values()}
|
||||
if len(sizes) == 1:
|
||||
lines.append("| CPU over wall time at {:,} bars | ".format(sizes.pop())
|
||||
+ " | ".join("{:.2f}".format(threading[e][0]) if e in threading else "-"
|
||||
for e in engines) + " |")
|
||||
else:
|
||||
lines.append("| CPU over wall time, at the longest series each ran | " + " | ".join(
|
||||
"{:.2f} (at {:,})".format(*threading[e]) if e in threading else "-"
|
||||
for e in engines) + " |")
|
||||
lines.append("")
|
||||
if mem and mem.get("failed"):
|
||||
for engine, why in mem["failed"].items():
|
||||
lines += ["- Memory probe for {}: {}".format(engine, why), ""]
|
||||
if cold and cold.get("failed"):
|
||||
for engine, why in cold["failed"].items():
|
||||
lines += ["- Cold-start probe for {}: {}".format(engine, why), ""]
|
||||
return lines
|
||||
|
||||
|
||||
@@ -275,7 +324,9 @@ def _not_run(payload: Dict[str, Any], results: List[Dict[str, Any]]) -> List[str
|
||||
Kept as prose rather than a table because the reason is the content. An
|
||||
engine that cannot express a workload has told you something about itself,
|
||||
and compressing that into an empty cell would throw away the only part worth
|
||||
reading.
|
||||
reading. An engine held back by its ceiling gets the same treatment, and the
|
||||
same reason is printed once with the list of workloads it applies to, not
|
||||
once per workload.
|
||||
"""
|
||||
seen: Dict[str, Dict[str, str]] = {}
|
||||
for row in results:
|
||||
@@ -287,8 +338,12 @@ def _not_run(payload: Dict[str, Any], results: List[Dict[str, Any]]) -> List[str
|
||||
|
||||
lines = ["## Not run, and why", ""]
|
||||
for engine, entries in seen.items():
|
||||
by_reason: Dict[str, List[str]] = {}
|
||||
for workload, why in entries.items():
|
||||
lines += ["- **{e} on `{w}`.** {why}".format(e=engine, w=workload, why=why), ""]
|
||||
by_reason.setdefault(why, []).append(workload)
|
||||
for why, workloads in by_reason.items():
|
||||
lines += ["- **{e} on {w}.** {why}".format(
|
||||
e=engine, w=", ".join("`{}`".format(w) for w in workloads), why=why), ""]
|
||||
return lines
|
||||
|
||||
|
||||
@@ -398,6 +453,15 @@ def _divergence_note(row: Dict[str, Any], payload: Dict[str, Any]) -> List[str]:
|
||||
if engine == reference or "round_trips" not in scale:
|
||||
continue
|
||||
note += " {e} books {n}.".format(e=engine, n=scale["round_trips"])
|
||||
# An engine that takes the same trades and sizes one of them differently
|
||||
# says how far: a count and a median, not an adjective.
|
||||
if scale.get("reentries_on_exit_bar") is not None:
|
||||
note += (" Of those, {n} re-enter on the exit bar, with a quantity that "
|
||||
"sits a median of {m:.1e} (at most {x:.1e}), in relative terms, "
|
||||
"away from the position closed on that bar.".format(
|
||||
n=scale["reentries_on_exit_bar"],
|
||||
m=scale.get("reentry_quantity_gap_median", 0.0),
|
||||
x=scale.get("reentry_quantity_gap_max", 0.0)))
|
||||
return [note]
|
||||
|
||||
|
||||
|
||||
@@ -23,6 +23,25 @@ vectorbt==0.28.4
|
||||
# 6.x line and the one this harness is validated against.
|
||||
plotly==6.9.0
|
||||
raptorbt==0.9.0
|
||||
# NautilusTrader: an event-driven engine with a Rust core and a Python API. It
|
||||
# asks for Python >=3.12,<3.15 and publishes cp312 wheels (checked on Linux,
|
||||
# manylinux_2_35, and on Windows), so it sits inside the interpreter that raptorbt
|
||||
# already pins us to. 1.231.0 is the latest stable release.
|
||||
nautilus_trader==1.231.0
|
||||
# Its runtime dependencies, pinned for the same reason plotly is above: nothing
|
||||
# in the set is allowed to move because a transitive dependency shipped a
|
||||
# release the day before the run. numpy, pandas and psutil are already pinned
|
||||
# further down; uvloop is what it installs on Linux and macOS only.
|
||||
msgspec==0.22.0
|
||||
portion==2.6.3
|
||||
sortedcontainers==2.4.0
|
||||
click==8.5.0
|
||||
fsspec==2026.2.0
|
||||
pyarrow==26.0.0
|
||||
pytz==2026.5
|
||||
tzdata==2026.5
|
||||
tqdm==4.70.1
|
||||
uvloop==0.22.1 ; sys_platform != "win32"
|
||||
numpy==2.4.3
|
||||
numba==0.64.0
|
||||
pandas==2.3.3
|
||||
|
||||
@@ -36,6 +36,19 @@ imposed by the harness, it is the only spelling available, and it is the one a
|
||||
raptorbt user would write. It gets the same courtesy vectorbt gets on its own
|
||||
path: each distinct moving average is computed once and reused across every
|
||||
combination it appears in, rather than recomputed per cell.
|
||||
|
||||
NautilusTrader has no grid either, and its loop is the expensive one
|
||||
-------------------------------------------------------------------
|
||||
Each cell of a Nautilus grid is a full event-driven replay: every bar goes
|
||||
through the simulated exchange and a Python callback. So the loop is run the way
|
||||
the engine documents repeat runs (one engine, data loaded once, ``reset()`` and
|
||||
``run()`` per cell), and the cost is measured rather than guessed: on a grid
|
||||
larger than the engine's sweep ceiling (``engines.py``) the runner declares it
|
||||
out of scope, and the reason carries the per-backtest time measured on a small
|
||||
anchor grid of the same code path, and what that makes of this grid.
|
||||
Moving averages cannot be hoisted out of the loop for it: a Nautilus indicator
|
||||
lives inside the event loop, so each cell computes its own, which is what a
|
||||
Nautilus user's strategy does.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -311,6 +324,18 @@ def run_rbt(df, fast_vals, slow_vals, metrics=False):
|
||||
return call
|
||||
|
||||
|
||||
def run_nt(df, fast_vals, slow_vals, metrics=False):
|
||||
"""NautilusTrader's grid path: one replay per cell, in Python (see engine_nt)."""
|
||||
if metrics:
|
||||
raise ValueError(
|
||||
"nautilus_trader reports no drawdown, so a grid asked for a "
|
||||
"performance summary has nothing to compare there")
|
||||
|
||||
import engine_nt
|
||||
|
||||
return engine_nt.sweep_runner(df, fast_vals, slow_vals)
|
||||
|
||||
|
||||
def builder(name: str, df, fast_vals, slow_vals, workdir: str, metrics: bool = False):
|
||||
"""The grid closure for one engine, ready to be timed."""
|
||||
if name == "manifoldbt":
|
||||
@@ -319,6 +344,8 @@ def builder(name: str, df, fast_vals, slow_vals, workdir: str, metrics: bool = F
|
||||
return run_vbt(df, fast_vals, slow_vals, metrics)
|
||||
if name == "raptorbt":
|
||||
return run_rbt(df, fast_vals, slow_vals, metrics)
|
||||
if name == "nautilus_trader":
|
||||
return run_nt(df, fast_vals, slow_vals, metrics)
|
||||
raise KeyError("no sweep path for engine {!r}".format(name))
|
||||
|
||||
|
||||
@@ -366,15 +393,66 @@ def compare(reference_out: dict, other_out: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# vectorbt is the only engine here whose grid footprint grows with the number of
|
||||
# combinations, so it is the only one a point can declare out of scope. The
|
||||
# others are looped or streamed and cost the same memory at any grid size.
|
||||
OOS_ENGINE = "vectorbt"
|
||||
OOS_REASON = (
|
||||
# Why an engine can be declared out of scope at a point, and how big a grid it
|
||||
# still runs so that the point carries a cross-engine check of the same code path.
|
||||
#
|
||||
# vectorbt is the one engine whose grid footprint grows with the number of
|
||||
# combinations, so a point can be out of its reach for memory. NautilusTrader's
|
||||
# footprint is flat, but its wall time is a full replay per cell, so a point can
|
||||
# be out of its reach for time. The others are looped or streamed and cost the
|
||||
# same at any grid size.
|
||||
OOS_VBT_REASON = (
|
||||
"vectorbt materialises the simulation per combination (measured: 1.57 MB "
|
||||
"per combination at 20k bars), so this grid would need tens of gigabytes. "
|
||||
"Running it would measure the swap file."
|
||||
)
|
||||
# vectorbt's anchor is a smaller grid of its own (a near-square grid of that many
|
||||
# cells). NautilusTrader's is a subsample of THIS grid, ANCHOR_SPREAD values of
|
||||
# each parameter spread evenly from smallest to largest, because what it has to
|
||||
# measure is the cost of a replay, and that depends on the windows: short moving
|
||||
# averages trade often and a Nautilus replay pays about a millisecond per fill, so
|
||||
# the cheap corner of a grid and the expensive corner differ several-fold.
|
||||
ANCHOR_CELLS = {"vectorbt": 250}
|
||||
ANCHOR_SPREAD = 4
|
||||
|
||||
|
||||
def spread(values, count):
|
||||
"""`count` values taken evenly across `values`, both ends included."""
|
||||
if len(values) <= count:
|
||||
return list(values)
|
||||
last = len(values) - 1
|
||||
return [values[round(i * last / (count - 1))] for i in range(count)]
|
||||
|
||||
|
||||
def _duration(seconds: float) -> str:
|
||||
if seconds < 90:
|
||||
return "{:.0f} seconds".format(seconds)
|
||||
if seconds < 90 * 60:
|
||||
return "{:.0f} minutes".format(seconds / 60.0)
|
||||
return "{:.1f} hours".format(seconds / 3600.0)
|
||||
|
||||
|
||||
def oos_reason(name: str, bars: int, combos: int, anchor_cells: int, anchor_s: float,
|
||||
loop_limit: int | None = None) -> str:
|
||||
"""The sentence a reader gets in place of an empty cell."""
|
||||
if name == "vectorbt":
|
||||
return OOS_VBT_REASON
|
||||
per_cell = anchor_s / anchor_cells
|
||||
held = (
|
||||
"Looped grids are held to {:,} combinations, so that a point stays inside "
|
||||
"the time budget of the job.".format(loop_limit)
|
||||
if loop_limit else "That is past the time budget of the job."
|
||||
)
|
||||
return (
|
||||
"NautilusTrader has no parameter-grid API, so a grid is a Python loop of "
|
||||
"full backtests, each one replaying every bar through the simulated "
|
||||
"exchange. Measured: {per:.2f} s per backtest at {bars:,} bars (mean over "
|
||||
"{cells} cells spread across this grid), which puts this grid of "
|
||||
"{combos:,} combinations at about {total} per call. {held} It is not "
|
||||
"run.".format(
|
||||
per=per_cell, bars=bars, cells=anchor_cells, combos=combos,
|
||||
total=_duration(per_cell * combos), held=held)
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
@@ -436,20 +514,32 @@ def ap_parse():
|
||||
"says anything useful without a licence.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--vectorbt",
|
||||
choices=("run", "oos"),
|
||||
default="run",
|
||||
help="'oos' declares vectorbt out of scope for this point and runs the "
|
||||
"others. For grids whose vectorbt side does not fit in memory on "
|
||||
"any ordinary machine: not running it is the honest reading, since "
|
||||
"the alternative is a number produced by swapping.",
|
||||
"--out-of-scope",
|
||||
nargs="+",
|
||||
default=[],
|
||||
choices=sorted(set(ENGINES) - {REFERENCE}),
|
||||
metavar="ENGINE",
|
||||
help="engines declared out of scope for this point; the others run. For "
|
||||
"grids an engine cannot hold: vectorbt when its materialised "
|
||||
"simulation needs tens of gigabytes (not running it is the honest "
|
||||
"reading, since the alternative is a number produced by swapping), "
|
||||
"nautilus_trader when a loop of full replays would need hours.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--loop-limit",
|
||||
type=int,
|
||||
default=None,
|
||||
help="the largest grid an engine with no sweep API is looped over, as the "
|
||||
"runner applied it; only quoted in the reason an engine is out of scope",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--parity-anchor",
|
||||
type=int,
|
||||
default=250,
|
||||
help="with --vectorbt oos, the grid size vectorbt DOES run, so the point "
|
||||
"still carries a cross-engine check of the same code path.",
|
||||
default=None,
|
||||
help="with --out-of-scope, the grid size vectorbt DOES run, so the "
|
||||
"point still carries a cross-engine check of the same code path. "
|
||||
"(default {}). NautilusTrader's check is a {}x{} subsample of the "
|
||||
"grid itself.".format(ANCHOR_CELLS["vectorbt"], ANCHOR_SPREAD, ANCHOR_SPREAD),
|
||||
)
|
||||
ap.add_argument(
|
||||
"--memory-only",
|
||||
@@ -542,9 +632,24 @@ def _main(args) -> int:
|
||||
|
||||
running = [REFERENCE] + [n for n in args.engines if n != REFERENCE]
|
||||
out_of_scope = {}
|
||||
if args.vectorbt == "oos" and OOS_ENGINE in running:
|
||||
running.remove(OOS_ENGINE)
|
||||
out_of_scope[OOS_ENGINE] = {"status": "out of scope", "reason": OOS_REASON}
|
||||
for name in args.out_of_scope:
|
||||
if name in running:
|
||||
running.remove(name)
|
||||
# The reason is filled in below, once the anchor grid has been run:
|
||||
# for an engine out of reach for time, it carries what that grid
|
||||
# measured.
|
||||
out_of_scope[name] = {"status": "out of scope", "reason": ""}
|
||||
if args.metrics and "nautilus_trader" in running:
|
||||
print(json.dumps({
|
||||
"bars": args.bars,
|
||||
"combos": actual,
|
||||
"status": "skipped",
|
||||
"reason": "nautilus_trader reports no drawdown, so there is no performance "
|
||||
"summary to compare on a grid; drop it with "
|
||||
"--out-of-scope nautilus_trader or leave --metrics off",
|
||||
"tier_before": tier_before,
|
||||
}))
|
||||
return 2
|
||||
|
||||
calls = {
|
||||
name: builder(name, df, fast_vals, slow_vals,
|
||||
@@ -580,18 +685,26 @@ def _main(args) -> int:
|
||||
# same code path at a size that engine can hold: same data, same
|
||||
# strategy, same adapters, fewer cells. That is weaker than checking the
|
||||
# grid itself, and the report says so rather than implying otherwise.
|
||||
a_fast, a_slow = grid(args.parity_anchor)
|
||||
anchor = compare(
|
||||
builder(REFERENCE, df, a_fast, a_slow,
|
||||
os.path.join(workdir, "anchor"), args.metrics)(),
|
||||
builder(name, df, a_fast, a_slow, workdir, args.metrics)(),
|
||||
)
|
||||
if name in ANCHOR_CELLS:
|
||||
a_fast, a_slow = grid(args.parity_anchor or ANCHOR_CELLS[name])
|
||||
else:
|
||||
a_fast, a_slow = spread(fast_vals, ANCHOR_SPREAD), spread(slow_vals, ANCHOR_SPREAD)
|
||||
anchor_cells = len(a_fast) * len(a_slow)
|
||||
reference_anchor = builder(REFERENCE, df, a_fast, a_slow,
|
||||
os.path.join(workdir, "anchor"), args.metrics)()
|
||||
challenger = builder(name, df, a_fast, a_slow, workdir, args.metrics)
|
||||
started = time.perf_counter()
|
||||
challenger_anchor = challenger()
|
||||
anchor_s = time.perf_counter() - started
|
||||
verdicts[name] = {
|
||||
**anchor,
|
||||
"checked_at_combos": len(a_fast) * len(a_slow),
|
||||
**compare(reference_anchor, challenger_anchor),
|
||||
"checked_at_combos": anchor_cells,
|
||||
"scope": "anchor grid, not this grid",
|
||||
}
|
||||
detail["parity_scope"] = "anchor grid"
|
||||
detail["reason"] = oos_reason(name, args.bars, actual, anchor_cells, anchor_s,
|
||||
args.loop_limit if name != "vectorbt" else None)
|
||||
detail["anchor_seconds"] = round(anchor_s, 3)
|
||||
|
||||
result["parity"] = verdicts
|
||||
result["status"] = "failed" if any(
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Guards for the parts of the harness that decide what gets published.
|
||||
|
||||
These run without any engine installed: they exercise the gate, the ceilings and
|
||||
the sweep plumbing, which are plain Python. Run with
|
||||
``pytest benchmarks/vs_vectorbt/test_harness.py``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
import pytest
|
||||
|
||||
import bench
|
||||
import parity
|
||||
import report
|
||||
import sweep_child
|
||||
from engines import ENGINES, REFERENCE, ceiling
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The gate, for an engine that has no drawdown of its own
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _metrics(**over):
|
||||
base = {
|
||||
"total_return": 0.01, "final_equity": 101_000.0, "round_trips": 10,
|
||||
"fills": 20, "total_fees": 0.0,
|
||||
"max_drawdown": -0.02, "sharpe": 1.0, "sortino": 2.0, "volatility": 0.1,
|
||||
}
|
||||
base.update(over)
|
||||
return base
|
||||
|
||||
|
||||
def test_an_engine_without_a_drawdown_is_gated_on_money_and_round_trips():
|
||||
verdict = parity.compare(
|
||||
_metrics(), _metrics(max_drawdown=None, sharpe=9.0), "sma_cross_metrics",
|
||||
"nautilus_trader")
|
||||
|
||||
assert verdict["status"] == "exact"
|
||||
assert verdict["diffs"]["max_drawdown_absent"] is True
|
||||
assert "max_drawdown_rel" not in verdict["diffs"]
|
||||
# Its ratios sit on their own basis, so they are not subtracted from ours.
|
||||
assert verdict["diffs"]["ratio_basis"] == "native"
|
||||
assert "advisory_ratio_rel" not in verdict["diffs"]
|
||||
|
||||
|
||||
def test_an_engine_without_a_drawdown_still_fails_on_money():
|
||||
verdict = parity.compare(
|
||||
_metrics(), _metrics(max_drawdown=None, final_equity=101_050.0),
|
||||
"sma_cross_metrics", "nautilus_trader")
|
||||
|
||||
assert verdict["status"] == "failed"
|
||||
|
||||
|
||||
def test_a_drawdown_that_is_present_is_still_gated():
|
||||
verdict = parity.compare(
|
||||
_metrics(), _metrics(max_drawdown=-0.03), "sma_cross_metrics", "raptorbt")
|
||||
|
||||
assert verdict["status"] == "failed"
|
||||
assert verdict["diffs"]["max_drawdown_rel"] > parity.REL_TOL
|
||||
|
||||
|
||||
def test_the_documented_bracket_divergence_is_expected_for_nautilus():
|
||||
verdict = parity.compare(
|
||||
_metrics(), _metrics(final_equity=101_000.7, max_drawdown=None),
|
||||
"bracket_sl_tp", "nautilus_trader")
|
||||
|
||||
assert verdict["expected"] == "documented"
|
||||
assert verdict["status"] == "documented"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Ceilings
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_overrides():
|
||||
bench.CEILING_OVERRIDES.clear()
|
||||
bench.SWEEP_CEILING_OVERRIDES.clear()
|
||||
yield
|
||||
bench.CEILING_OVERRIDES.clear()
|
||||
bench.SWEEP_CEILING_OVERRIDES.clear()
|
||||
|
||||
|
||||
def test_only_the_event_driven_engine_has_a_ceiling_by_default():
|
||||
limited = {n for n in ENGINES if ceiling(n) is not None}
|
||||
|
||||
assert limited == {"nautilus_trader"}
|
||||
assert REFERENCE not in limited
|
||||
|
||||
|
||||
def test_an_engine_runs_up_to_its_ceiling_and_not_beyond():
|
||||
limit = ceiling("nautilus_trader")
|
||||
|
||||
assert bench._within_ceiling("nautilus_trader", limit)
|
||||
assert not bench._within_ceiling("nautilus_trader", limit + 1)
|
||||
assert bench._within_ceiling("vectorbt", 10_000_000)
|
||||
|
||||
|
||||
def test_an_override_moves_a_ceiling_and_zero_removes_it():
|
||||
bench.CEILING_OVERRIDES["nautilus_trader"] = 1_000_000
|
||||
assert bench._within_ceiling("nautilus_trader", 1_000_000)
|
||||
assert not bench._within_ceiling("nautilus_trader", 1_000_001)
|
||||
|
||||
bench.CEILING_OVERRIDES["nautilus_trader"] = 0
|
||||
assert bench._within_ceiling("nautilus_trader", 100_000_000)
|
||||
|
||||
|
||||
def test_a_held_back_engine_gets_a_reason_where_the_site_already_looks_for_one():
|
||||
"""`unsupported` is the field the website prints under "Not run, and why"."""
|
||||
active = ["manifoldbt", "vectorbt", "nautilus_trader"]
|
||||
long_run = bench._entry("sma_cross", 1_000_000, frame=_Frame(1_000_000), verdicts={},
|
||||
engines_run=["manifoldbt", "vectorbt"], active=active)
|
||||
short_run = bench._entry("sma_cross", 100_000, frame=_Frame(100_000), verdicts={},
|
||||
engines_run=active, active=active)
|
||||
|
||||
assert "nautilus_trader" in long_run["unsupported"]
|
||||
assert long_run["unsupported"]["nautilus_trader"].startswith("Not run above 100,000 bars")
|
||||
assert long_run["beyond_ceiling"] == {"nautilus_trader": 100_000}
|
||||
assert "unsupported" not in short_run
|
||||
assert "beyond_ceiling" not in short_run
|
||||
|
||||
|
||||
def test_a_ceiling_is_not_reported_for_a_workload_the_engine_cannot_run_anyway():
|
||||
"""raptorbt's blocker is the reason on those rows, not a ceiling."""
|
||||
entry = bench._entry("sma_cross_costs", 1_000_000, frame=_Frame(1_000_000), verdicts={},
|
||||
engines_run=["manifoldbt", "vectorbt"],
|
||||
active=["manifoldbt", "vectorbt", "raptorbt"])
|
||||
|
||||
assert set(entry["unsupported"]) == {"raptorbt"}
|
||||
assert "beyond_ceiling" not in entry
|
||||
|
||||
|
||||
class _Frame:
|
||||
"""Stands in for a DataFrame where only the length and a digest are read."""
|
||||
|
||||
def __init__(self, rows):
|
||||
self.rows = rows
|
||||
|
||||
def __len__(self):
|
||||
return self.rows
|
||||
|
||||
def __getitem__(self, column):
|
||||
import numpy as np
|
||||
|
||||
return _Column(np.zeros(self.rows))
|
||||
|
||||
|
||||
class _Column:
|
||||
def __init__(self, values):
|
||||
self.values = values
|
||||
|
||||
def to_numpy(self, dtype=None):
|
||||
return self.values
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Sweep points
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_plain_point_runs_everyone():
|
||||
assert bench.parse_point("20000:5000") == (20_000, 5_000, ())
|
||||
|
||||
|
||||
def test_a_bare_oos_still_means_vectorbt():
|
||||
assert bench.parse_point("20000:20000:oos") == (20_000, 20_000, ("vectorbt",))
|
||||
|
||||
|
||||
def test_oos_can_name_several_engines():
|
||||
assert bench.parse_point("20000:5000:oos=vectorbt,nautilus_trader") == (
|
||||
20_000, 5_000, ("vectorbt", "nautilus_trader"))
|
||||
|
||||
|
||||
def test_the_reference_and_strangers_cannot_be_declared_out_of_scope():
|
||||
with pytest.raises(argparse.ArgumentTypeError):
|
||||
bench.parse_point("20000:5000:oos=manifoldbt")
|
||||
with pytest.raises(argparse.ArgumentTypeError):
|
||||
bench.parse_point("20000:5000:oos=zipline")
|
||||
with pytest.raises(argparse.ArgumentTypeError):
|
||||
bench.parse_point("20000:5000:maybe")
|
||||
|
||||
|
||||
ALL = ["manifoldbt", "vectorbt", "raptorbt", "nautilus_trader"]
|
||||
|
||||
|
||||
def test_a_big_grid_holds_back_the_looped_engine_without_being_told_to():
|
||||
"""The recipe on the website only says `:oos`; it must not run Nautilus for hours."""
|
||||
assert bench._sweep_out_of_scope(5_000, (), ALL) == ["nautilus_trader"]
|
||||
assert bench._sweep_out_of_scope(5_000, ("vectorbt",), ALL) == ["vectorbt", "nautilus_trader"]
|
||||
|
||||
|
||||
def test_a_small_grid_is_run_by_everyone():
|
||||
assert bench._sweep_out_of_scope(36, (), ALL) == []
|
||||
assert bench._sweep_out_of_scope(64, (), ALL) == []
|
||||
assert bench._sweep_out_of_scope(65, (), ALL) == ["nautilus_trader"]
|
||||
|
||||
|
||||
def test_the_sweep_ceiling_can_be_moved_or_lifted():
|
||||
bench.SWEEP_CEILING_OVERRIDES["nautilus_trader"] = 500
|
||||
assert bench._sweep_out_of_scope(500, (), ALL) == []
|
||||
bench.SWEEP_CEILING_OVERRIDES["nautilus_trader"] = 0
|
||||
assert bench._sweep_out_of_scope(50_000, (), ALL) == []
|
||||
|
||||
|
||||
def test_an_engine_that_is_not_running_is_not_listed_as_out_of_scope():
|
||||
assert bench._sweep_out_of_scope(5_000, ("vectorbt",), ["manifoldbt", "raptorbt"]) == []
|
||||
|
||||
|
||||
def test_the_nautilus_anchor_spans_the_grid_it_stands_in_for():
|
||||
values = list(range(100))
|
||||
|
||||
picked = sweep_child.spread(values, 4)
|
||||
|
||||
assert picked[0] == 0 and picked[-1] == 99
|
||||
assert len(picked) == 4
|
||||
assert picked == sorted(set(picked))
|
||||
|
||||
|
||||
def test_a_small_grid_is_its_own_anchor():
|
||||
assert sweep_child.spread([5, 7], 4) == [5, 7]
|
||||
|
||||
|
||||
def test_the_reason_for_a_looped_engine_carries_what_was_measured():
|
||||
text = sweep_child.oos_reason("nautilus_trader", 20_000, 5_000,
|
||||
anchor_cells=16, anchor_s=16.0)
|
||||
|
||||
assert "1.00 s per backtest at 20,000 bars" in text
|
||||
assert "5,000 combinations" in text
|
||||
assert "about 83 minutes" in text
|
||||
|
||||
|
||||
def test_a_grid_held_back_by_the_sweep_ceiling_says_so_even_when_it_is_cheap():
|
||||
text = sweep_child.oos_reason("nautilus_trader", 5_000, 100, anchor_cells=16,
|
||||
anchor_s=3.2, loop_limit=64)
|
||||
|
||||
assert "about 20 seconds per call" in text
|
||||
assert "held to 64 combinations" in text
|
||||
assert "past the time budget" not in text
|
||||
|
||||
|
||||
def test_the_vectorbt_reason_is_the_memory_one():
|
||||
text = sweep_child.oos_reason("vectorbt", 20_000, 20_000, anchor_cells=250, anchor_s=1.0)
|
||||
|
||||
assert text == sweep_child.OOS_VBT_REASON
|
||||
|
||||
|
||||
@pytest.mark.parametrize("seconds,expected", [
|
||||
(42.0, "42 seconds"), (600.0, "10 minutes"), (9_000.0, "2.5 hours"),
|
||||
])
|
||||
def test_durations_read_in_the_unit_a_person_would_use(seconds, expected):
|
||||
assert sweep_child._duration(seconds) == expected
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The report, for an engine that was held back
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _row(workload, bars, engines, **extra):
|
||||
row = {
|
||||
"workload": workload, "title": workload, "bars": bars, "status": "exact",
|
||||
"engines": engines,
|
||||
"timings": {e: {"median_s": 0.01, "iqr_over_median": 0.01} for e in engines},
|
||||
"parity": {e: {"status": "exact", "publishable": True, "diffs": {}}
|
||||
for e in engines if e != "manifoldbt"},
|
||||
"speedup": {e: {"median_of_ratios": 2.0, "min": 1.0, "max": 3.0}
|
||||
for e in engines if e != "manifoldbt"},
|
||||
}
|
||||
row.update(extra)
|
||||
return row
|
||||
|
||||
|
||||
def _payload(rows, **extra):
|
||||
engines = ["manifoldbt", "nautilus_trader"]
|
||||
payload = {
|
||||
"schema_version": 2, "reference": "manifoldbt", "engines": engines, "reps": 3,
|
||||
"generated_at": "2026-10-22T00:00:00+00:00",
|
||||
"environment": {
|
||||
"os": "Linux", "arch": "x86_64", "cpu": "a cpu", "logical_cores": 4,
|
||||
"ram_gb": 16.0, "python": "3.12.0", "pinned_cores": None,
|
||||
"versions": {"manifoldbt": "0.28.0", "nautilus_trader": "1.231.0",
|
||||
"numpy": "2.4.3", "numba": "0.64.0", "pandas": "2.3.3"},
|
||||
},
|
||||
"results": rows,
|
||||
}
|
||||
payload.update(extra)
|
||||
return payload
|
||||
|
||||
|
||||
WHY = "Not run above 100,000 bars. Event-driven."
|
||||
|
||||
|
||||
def test_one_reason_is_printed_once_for_all_the_workloads_it_covers():
|
||||
rows = [
|
||||
_row("sma_cross", 1_000_000, ["manifoldbt"], unsupported={"nautilus_trader": WHY}),
|
||||
_row("multi_asset", 1_000_000, ["manifoldbt"], unsupported={"nautilus_trader": WHY}),
|
||||
]
|
||||
|
||||
text = report.render(_payload(rows))
|
||||
|
||||
assert text.count(WHY) == 1
|
||||
assert "`sma_cross`, `multi_asset`" in text
|
||||
|
||||
|
||||
def test_an_engine_without_a_drawdown_is_said_not_to_have_one():
|
||||
summarised = _row("sma_cross_metrics", 100_000, ["manifoldbt", "nautilus_trader"],
|
||||
paired_with=["sma_cross"])
|
||||
summarised["parity"]["nautilus_trader"]["diffs"] = {
|
||||
"max_drawdown_absent": True, "ratio_basis": "native"}
|
||||
plain = _row("sma_cross", 100_000, ["manifoldbt", "nautilus_trader"],
|
||||
paired_with=["sma_cross_metrics"])
|
||||
# The longest row is one the engine did not run on: its verdict is read from
|
||||
# the longest row it did.
|
||||
longer = _row("sma_cross_metrics", 1_000_000, ["manifoldbt"],
|
||||
paired_with=["sma_cross"], unsupported={"nautilus_trader": WHY})
|
||||
longer_plain = _row("sma_cross", 1_000_000, ["manifoldbt"],
|
||||
paired_with=["sma_cross_metrics"], unsupported={"nautilus_trader": WHY})
|
||||
|
||||
text = report.render(_payload([plain, summarised, longer_plain, longer]))
|
||||
|
||||
assert "reports no maximum drawdown" in text
|
||||
assert "was not compared" not in text
|
||||
|
||||
|
||||
def test_a_memory_figure_taken_at_the_ceiling_says_so():
|
||||
memory = {
|
||||
"workload": "sma_cross", "bars": 2_000_000, "engines": ["manifoldbt", "nautilus_trader"],
|
||||
"manifoldbt": {"added_mb_per_million_bars": 0.006, "bars": 2_000_000},
|
||||
"nautilus_trader": {"added_mb_per_million_bars": 4.8, "bars": 100_000},
|
||||
}
|
||||
|
||||
text = report.render(_payload([_row("sma_cross", 100_000, ["manifoldbt"])], memory=memory))
|
||||
|
||||
assert "5 MB (at 100,000 bars)" in text
|
||||
assert "0 MB |" in text
|
||||
|
||||
|
||||
def test_the_cpu_row_survives_an_engine_that_did_not_run_the_longest_series():
|
||||
rows = [
|
||||
_row("sma_cross", 100_000, ["manifoldbt", "nautilus_trader"],
|
||||
cpu_over_wall={"manifoldbt": 1.0, "nautilus_trader": 0.99}),
|
||||
_row("sma_cross", 10_000_000, ["manifoldbt"], cpu_over_wall={"manifoldbt": 3.5}),
|
||||
]
|
||||
|
||||
text = report.render(_payload(rows, memory=None, cold_start=None))
|
||||
|
||||
assert "CPU over wall time, at the longest series each ran" in text
|
||||
assert "3.50 (at 10,000,000)" in text
|
||||
assert "0.99 (at 100,000)" in text
|
||||
|
||||
|
||||
def test_a_reentry_that_differs_only_in_quantity_reports_how_far():
|
||||
row = _row("bracket_sl_tp", 100_000, ["manifoldbt", "nautilus_trader"], status="documented")
|
||||
row["parity"]["nautilus_trader"] = {"status": "documented", "publishable": False,
|
||||
"diffs": {"final_equity_vs_capital": 7e-6}}
|
||||
row["divergence_scale"] = {
|
||||
"manifoldbt": {"reentries_on_exit_bar": 900, "round_trips": 2300,
|
||||
"share_of_round_trips": 0.39, "sl_exits": 400, "tp_exits": 300},
|
||||
"nautilus_trader": {"round_trips": 2300, "reentries_on_exit_bar": 900,
|
||||
"reentry_quantity_gap_median": 3e-4,
|
||||
"reentry_quantity_gap_max": 1e-3},
|
||||
}
|
||||
|
||||
text = report.render(_payload([row]))
|
||||
|
||||
assert "nautilus_trader books 2300." in text
|
||||
assert "a median of 3.0e-04" in text
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Workload definitions: the numbers every engine reads, in one place.
|
||||
|
||||
Nothing here is engine-specific. Each adapter (``engine_mbt``, ``engine_vbt``,
|
||||
``engine_rbt``) reads the same constants, so a workload cannot drift between two
|
||||
``engine_rbt``, ``engine_nt``) reads the same constants, so a workload cannot drift between two
|
||||
sides by someone editing one file and forgetting another.
|
||||
|
||||
What a workload *does* carry per engine is a note: the places where an engine is
|
||||
@@ -219,6 +219,25 @@ WORKLOADS: Dict[str, Workload] = {
|
||||
"The count of affected round-trips is measured rather than "
|
||||
"asserted, and it is the same population for both challengers.",
|
||||
),
|
||||
"nautilus_trader": Note(
|
||||
"documented",
|
||||
"Re-entry sizing. Entries, exits and fill prices are the same "
|
||||
"trades as the reference's, round-trip for round-trip: the stop "
|
||||
"and the target are native contingent orders, they fill at their "
|
||||
"own levels, and because the simulated exchange runs before the "
|
||||
"strategy sees the bar, a position stopped out intrabar is "
|
||||
"re-entered at that bar's close just as the reference does. What "
|
||||
"differs is the quantity of that re-entry. The reference sizes it "
|
||||
"from the equity at the close, counting the position it has just "
|
||||
"closed at the close price, so it buys back the quantity it "
|
||||
"held; Nautilus sizes it from the cash balance after the stop or "
|
||||
"target has been booked at its own price. Neither is wrong, the "
|
||||
"two quantities differ by the gap between the exit price and the "
|
||||
"close, and that gap is carried into every later trade. The "
|
||||
"harness measures it (re-entries on the exit bar, and how far "
|
||||
"each one's quantity sits from the position closed on that bar) "
|
||||
"instead of asserting it.",
|
||||
),
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user