mirror of
https://github.com/manifoldbt/manifoldbt.git
synced 2026-08-24 22:48:05 +00:00
bench: fix the red runs, trim the matrix, add a grid job (#7)
Five consecutive red runs, two unrelated causes. Four of them never reached an engine: the workflow installs the tag it is handed, but 0.18.0rc1 and rc2 were previews that never reached PyPI. Pre-releases are now skipped, and the report step checks for its input file instead of dying on a missing one and reporting the wrong cause twice. The fifth came from adding 10M bars, which broke a workload whose validity depended on the ladder stopping at 5M. ema_rsi_fees sizes in fixed units and pays 5 bps a side, so over 10M one-minute bars the fees compound into the whole account: -15% of capital at 1M, -74% at 5M, exactly -100% at 10M, where fees reach 99,611 of the 100,000 it started with. Both engines then sit at zero and disagree by 9,085 round-trips about how many worthless trades to book on a dead account. Workloads can now declare a ceiling, and the runner skips past it out loud. Fewer points per axis: three series lengths instead of five, a decade apart each step. 10k measured the clock rather than the work, and 5M sat between two points that already bracketed it. sma_cross crosses on 30/150 rather than 10/50, worth about 15% on the ratio because it books a third of the trades. Grids get their own job, licensed through ci_activate.py, which refuses to run unlicensed rather than time a wait. Three points, not a matrix: across the plane the four-core ratio moves only between x32 and x38.
This commit is contained in:
@@ -59,6 +59,10 @@ class Workload:
|
||||
title: str
|
||||
why: str
|
||||
params: Dict[str, Any] = field(default_factory=dict)
|
||||
# Longest series this workload is valid on, or None for no ceiling. A
|
||||
# workload can stop being a comparison before it stops running: see the fee
|
||||
# workload, whose account this exists to keep alive.
|
||||
max_bars: int | None = None
|
||||
# Engine name -> Note. An engine with no entry here is expected to agree
|
||||
# with the reference down to float-reordering noise, and a disagreement is
|
||||
# a failure that withholds the timing.
|
||||
@@ -70,11 +74,11 @@ WORKLOADS: Dict[str, Workload] = {
|
||||
for w in (
|
||||
Workload(
|
||||
key="sma_cross",
|
||||
title="SMA 10/50 crossover, long-only, no cost",
|
||||
title="SMA 30/150 crossover, long-only, no cost",
|
||||
why="The canonical baseline. Unambiguous indicator, no fee policy, "
|
||||
"no stop semantics: if the engines disagree here, nothing else "
|
||||
"in the suite is worth reading.",
|
||||
params=dict(fast=10, slow=50, alloc=1.0),
|
||||
params=dict(fast=30, slow=150, alloc=1.0),
|
||||
),
|
||||
Workload(
|
||||
key="ema_rsi_fees",
|
||||
@@ -87,6 +91,14 @@ WORKLOADS: Dict[str, Workload] = {
|
||||
"engines on a wiped-out account compares rounding noise.",
|
||||
params=dict(fast=12, slow=26, rsi_period=14, rsi_lo=30.0, rsi_hi=70.0,
|
||||
units=5.0, fee_bps=5.0),
|
||||
# The account has to survive, or the engines are being compared on
|
||||
# rounding noise around zero. Measured: -15% of capital at 1M bars,
|
||||
# -74% at 5M, and exactly -100% at 10M, where fees reach 99,611 of
|
||||
# the 100,000 started with. Past that the two disagree by thousands
|
||||
# of round-trips while both sit at zero equity, which is a fact
|
||||
# about a bankrupt strategy and not about either engine. The ceiling
|
||||
# is set where the comparison still means something.
|
||||
max_bars=1_000_000,
|
||||
notes={
|
||||
"raptorbt": Note(
|
||||
"unsupported",
|
||||
@@ -110,7 +122,7 @@ WORKLOADS: Dict[str, Workload] = {
|
||||
),
|
||||
Workload(
|
||||
key="sma_cross_metrics",
|
||||
title="SMA 10/50 crossover, with a performance summary",
|
||||
title="SMA 30/150 crossover, with a performance summary",
|
||||
why="The same simulation as `sma_cross`, but both engines are asked "
|
||||
"for what a user actually reads: max drawdown, Sharpe, Sortino "
|
||||
"and volatility alongside the return. manifoldbt computes them "
|
||||
@@ -120,7 +132,7 @@ WORKLOADS: Dict[str, Workload] = {
|
||||
"`sma_cross` above is the same work without the summary, and the "
|
||||
"two are reported side by side so the reader can see what the "
|
||||
"summary costs each engine.",
|
||||
params=dict(fast=10, slow=50, alloc=1.0, metrics=True),
|
||||
params=dict(fast=30, slow=150, alloc=1.0, metrics=True),
|
||||
),
|
||||
Workload(
|
||||
key="bracket_sl_tp",
|
||||
|
||||
Reference in New Issue
Block a user