gavindiaz/manifoldbt
1
0
Fork 0
mirror of https://github.com/manifoldbt/manifoldbt.git synced 2026-08-24 14:38:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

release: v0.14.0

Browse Source
This commit is contained in:
github-actions[bot]
2026-07-19 02:07:07 +00:00
parent d36bc7ee4c
commit 8bb39852d8
32 changed files with 1410 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/perf_audit_2026-07-18.md
+100
View File
@@ -0,0 +1,100 @@
# Performance audit: path to 2x on the hot paths (2026-07-18)
Audit read-only: no engine code was modified. Machine: i5-13600KF (6P+8E, 20 threads),
RTX 3090, build `maturin develop --release --features cuda` (LTO, cgu=1, debuginfo kept
via env overrides). Every number is the median of at least 3 runs; machine load was
verified with the MC-10M-CPU sentinel (6.3-6.5s clean on both sides of each suite; one
loaded suite at 8.6-9.3s was discarded and rerun).
**Conclusion: the CPU lite sweep can reach 2x (three workstreams). The GPU sweep cannot
under the fp64 bit-parity constraint (ceiling ~1.35x). The single run tops out around
1.7x. The full `run_sweep` API still carries the unfixed twin of the JSON-getter bug:
its Python post-processing costs 3.6x the entire lite compute.**
## Method
Sampling profilers were unavailable (samply/ETW needs Administrator; py-spy cannot see
rayon threads). Attribution comes from the engine's own instrumentation, which proved
sufficient: 98% of wall*threads is attributed.
- `ProfileData` (orchestrator.rs:400): per-phase us counters on every result, including
each lite sweep combo. Summed across combos and compared to wall*threads.
- `BT_PHASE6_DEBUG=1`: sub-splits output_build (metrics / capm / trade_stats / gather).
- `MBT_GPU_PROBE=1`: GPU phases (nvrtc / hoist / h2d / sim / metrics / d2h).
- Ablation: bars scaling (26k vs 100k), buy_and_hold vs ema_cross, max_parallelism 1..20,
full vs lite API.
Harness: copy of the session scratchpad `audit_harness.py` (subcommands: sentinel,
profile_sweep, full_vs_lite, single1m, par_scale, batch3, load, access).
## Measured baselines (synthetic BTCUSDT 1h store, `benchmarks/_sweep_common.py` config)
| Path | Result | Phase split |
|---|---|---|
| Single run, 26k bars | 0.85ms | signal 43%, sim 22%, output 32% |
| Single run, 1M bars | 27ms | signal 9.9ms (37%), sim 8.4ms (31%), output 8.3ms (31%: metrics 4.5, capm 1.0, trade_stats ~2, gather/positions ~2.3) |
| CPU lite sweep, 10k combos x 26k bars | 171ms = 58k c/s | per combo: signal_eval 96us (28%), sim phase 239us (70%) |
| inside the sim phase | | per-bar loop ~200us (~8ns/bar = 36 cycles/bar), CAPM ~25us (measured 1ns/bar), O(days) metrics ~10us |
| CPU sweep, 500k combos | rsi 9.1s / ema 9.9s / trix 12.4s | 40-55k c/s |
| GPU sweep, 500k combos | rsi 1.67s / ema 0.98s / trix 2.35s | 5.4-10.2x vs CPU |
| GPU, 100k combos x 26k bars | 240ms = 418k c/s | sim kernel 176ms (73%), metrics kernel 34ms (14%), h2d 6ms, d2h ~2ms |
| Data load, 1M bars | cold 57ms, warm re-align ~0.5ms | store cache holds across calls |
| Lite packaging/access | ~1us per `.metrics` | fixed path is healthy |
| Full `run_sweep`, 900 combos | 80ms vs 16ms lite (5x) | then `.best()` 15us/combo, `.to_df()` 49us/combo in Python |
Cross-checks: these numbers predict the historical 1M x 100k CPU sweep at ~107s, inside
the observed 84-150s band. Parallel scaling is 9.9x on 20 threads, near the realistic
ceiling (~11-12x) for 6P+8E: rayon granularity is not a lever. Combo enumeration is now
lazy mixed-radix (sweep.rs:98); the old ~500B/combo materialization survives only in
walk_forward/sweep_2d. Per-combo strategy recompile (dynamic periods): ~2%, cleared.
## Ranked opportunities
Gain = on the bench that path owns. Effort: L (<1 day), M (days), H (week+).
Parity = risk to CPU==GPU==general bit-identity (anchored to bt-expr, never a mirror).
| # | Opportunity | Evidence | Where | Approach | Gain | Effort | Parity risk |
|---|---|---|---|---|---|---|---|
| 1 | Full-result `.metrics` = unfixed JSON twin | `.to_df()`+`.best()` = 58ms vs 16ms lite compute @900 combos | bt-python/src/result.rs:24-31, python sweep.py:75-93, dataframe.py:146-166 | Reuse `metrics_to_pydict` (backtest.rs:331); better: route `SweepResult.to_df/best` through `sweep_columns` | ~50x on full-sweep post-processing | L | None |
| 2 | CPU transpiled sweep: kill per-combo signal materialization | signal_eval = 28% of sweep CPU | orchestrator.rs:2402-2709; reference core orchestrator.rs:5766 | Do on CPU what the GPU does: hoist plan + in-loop target eval; `sim_fast_lite_core_single` is the transpile-ready reference | ~1.33x CPU sweep | H | Low; re-anchor goldens, deliberately break the old path to prove coverage |
| 3 | Hoist combo-invariant work: (a) CAPM benchmark pass, (b) ohl_nan/funding_nan copies | (a) measured 1ns/bar = ~7% of sweep; (b) 3 full O(bars) copies per combo on SL/TP sweeps, est. 25-40% tax (code-confirmed, not yet measured) | (a) orchestrator.rs:3395-3411 and 2082-2108; (b) orchestrator.rs:2821-2849 | Compute once in the sweep/batch drivers, pass by reference | 1.08x plain sweeps; ~1.2-1.4x SL/TP sweeps | L-M | None (same values, computed once) |
| 4 | Per-bar loop tightening | loop ~58% of sweep CPU at 36 cycles/bar; dependency floor ~12-18 cycles | simulate_fast_lite_single / core (orchestrator.rs:5307/5766) | Branch elimination, layout. No FP reordering, no FMA: op order is the parity anchor | 1.15-1.3x sweep | H | High if careless; bit-verify vs bt-expr |
| 5 | Single-run: parallelize independent signals + output_build components | @1M: signal 37% (fast/slow EMA sequential), output 31% (independent components) | orchestrator.rs:833-1204, 2059-2230 | rayon-join independent signal exprs (bit-safe); overlap output components; never parallelize the metric reductions | ~1.4-1.7x single run | M | Low if reductions stay sequential |
| 6 | SIMD via runtime dispatch (ships SSE2 baseline: no target-cpu anywhere) | elementwise ops are a large slice of signal eval | bt-expr evaluator kernels | `is_x86_feature_detected!` dispatch on elementwise kernels only; folds/scans stay scalar | ~1.1-1.15x sweep | M | None for elementwise; forbidden for folds |
| 7 | GPU metrics kernel fuse/overlap | 34ms of 240ms (14%) | gpu_sweep.rs:4395-4656 | Fuse into sim epilogue or stream overlap, same op order | <=1.16x GPU | M-H | Low |
| 8 | Full `run_sweep` materializes full traces per combo | 5x lite; ~600KB/combo retained | orchestrator.rs:3480-3547 | Optional trace retention; steer to lite + `sweep_columns` | up to 5x for full-sweep users | M | None |
| 9 | Coverage gates (funding on `run()` fast path, multi-asset+brackets on GPU, multi-asset kernel 33% occupancy) | gate list in bt-core | orchestrator.rs:1358-1364, 3998-4002 | Extend fast/GPU coverage case by case | 2-5x for affected configs | M-H each | Per-gate golden work |
| 10 | Data loading (no mmap on Arrow IPC, exact-key cache only, N+1 sqlite symbol_info) | cold 1M load 57ms | bt-data arrow_ipc_store.rs:327/461, metadata.rs:28 | mmap like mega_store, superset-slice cache, batched lookups | <1% on benches | L-M | None |
Maintenance note: the per-bar loop exists in four near-identical transcriptions
(full/lite x general/fast) plus two ~130-LOC bracket macros; every loop optimization is
written and parity-tested four times. Mechanical extraction of the shared fill/equity
blocks is a safe enabler for #4. Any unification touching WHICH metrics lite computes
would hit the lite contract (cagr/calmar/ulcer != run()), which stays as-is.
## Top 3 to reach 2x (CPU lite sweep)
1. **#3 hoists** (CAPM + bracket/funding copies): low effort, zero parity risk, ~1.08x
plain sweeps, biggest single win on SL/TP sweeps.
2. **#2 CPU transpiled sweep**: ~1.33x, structural; the architecture is proven on GPU
and the CPU core is already the kernel's reference. With #3: ~1.45x.
3. **#4 loop tightening** (+ #6 SIMD on residual signal eval): 36 -> ~24 cycles/bar
closes the gap. Composite: **~1.9-2.2x**.
## Theoretical ceilings
- CPU sweep: signal+CAPM removed, loop untouched -> max 1.44x. 2x requires the loop
work; at the ~12-18 cycle dependency floor the composite ceiling is ~3x.
- GPU sweep: sim kernel 73%, already SASS-audited to the fp64 parity ceiling; everything
else free -> 1.37x max. 2x is not reachable under parity; opt-in fp32 remains the out.
- Single run 1M bars: sim loop is serial; practical ceiling ~1.7x.
- Data loading and lite packaging: <1% at bench scale, nothing to win.
## Compatibility guarantee for every item above
None of the proposals removes or restricts any strategy feature. #2/#4/#6 follow the
same pattern as the GPU path: strategies that qualify take the faster path, everything
else falls back to today's code, and the fallback stays golden-tested (the "break the
old path on purpose" rule applies when work moves). #3/#5/#7/#8/#10 compute identical
values in fewer places. #9 strictly widens fast-path coverage. Bit-for-bit parity is
re-verified against bt-expr for every change.
docs/perf_plan_2026-07.md
+292
View File
@@ -0,0 +1,292 @@
# Perf plan: 2x on the CPU sweep (and friends) — execution order
Source: `docs/perf_audit_2026-07-18.md` (measured baselines, ranked table, ceilings).
Scope: run_sweep_lite CPU is the 2x target. Single run gets ~1.5-1.7x. GPU 2x is a
non-goal (fp64 parity ceiling ~1.35x, documented). Lite contract stays as-is.
Ground rules for every phase:
- Bit-for-bit parity CPU==GPU==general, verified against bt-expr goldens, never a mirror.
- When work moves off a path, break the old path on purpose to prove tests still bite.
- Portability: no target-cpu in shipped config; runtime feature detection only.
- Every perf claim: median of >=3 runs, MC-10M-CPU sentinel ~6s clean on both sides.
- One branch per workstream, `perf:` commits, no cross-stream stacking unless noted.
## End-to-end result so far (2026-07-18)
Measured A-B-A at the build level (HEAD, then main's engine sources, then HEAD
again) so drift between builds is visible rather than assumed. "after" is the
mean of the two HEAD runs.
| | before (main) | after | gain |
|---|---|---|---|
| CPU sweep (10k combos x 26k bars) | 51,332 c/s | ~64,567 c/s | **+26%** |
| SL/TP sweep | 27,662 c/s | ~33,885 c/s | **+22%** |
| `SweepResult.to_df()` | 63.2 us/combo | 12.0 us | **5.3x** |
| `SweepResult.best()` | 25.4 us/combo | 2.4 us | **10.5x** |
| `.metrics` per result | 27.9 us | 3.0 us | **9.4x** |
| GPU sweep (untouched, drift canary) | 164,067 c/s | ~167,436 c/s | +2% |
The GPU line is the control: nothing in this work touches that path and it does
not move, so the harness is not systematically biased.
**Caveats, so these are not over-read.** The two identical HEAD builds differed
by 6.7% (62,474 vs 66,661 c/s) and the baseline was measured once, so read the
sweep numbers as +26% with roughly +/-7%, solid in direction and magnitude but
not to the point. The post-processing ratios are far above the noise (the two
HEAD runs agree to 0.3%) and are reliable.
**One unattributed slice.** The CPU sweep gained ~26% while the paired A/B
credits the CAPM hoist with 11.9%. The likely source of the remaining ~14% is
the daily-equity unification, which was intended as a pure refactor: the merged
form does one division and one comparison per bar where the old form did
`last() == Some(ts/nanos*nanos)` (division AND multiplication) and then possibly
a second `last()/nanos != day` test. Redundant per-bar arithmetic was removed
without aiming for it. This attribution is PLAUSIBLE BUT UNVERIFIED; it needs
its own paired A/B before being claimed.
## Phase 0 — lock the ruler (DONE 2026-07-18)
- [x] Harness checked in at `benchmarks/audit_harness.py` (subcommands: sentinel,
profile_sweep, bracket_probe, full_vs_lite, single1m, par_scale, sweep_cpu/gpu, access).
- [x] 2026-07-18 baselines are the reference row (see the audit doc).
- [x] Bracket-sweep probe run to size finding #3b.
**Measured (10k combos, ema grid, sentinel-clean):**
| bars | plain | SL/TP bracket | ratio | sim us/combo (plain -> bracket) |
|---|---|---|---|---|
| 26k | 54,800 c/s | 26,440 c/s | 2.07x slower | 242 -> 547 |
| 100k | 11,863 c/s | 3,737 c/s | 3.17x slower | 1,082 -> 3,972 |
**Re-sizing that forces (finding #3b):** the bracket penalty is large but scales
SUPER-linearly with bars (2.07x -> 3.17x), so it is dominated by per-bar bracket
check work, NOT by the `ohl_nan` copy (which is linear in bars). The copy is a
minority slice of the +305us/combo (26k). The audit's "25-40% tax, ~1.2-1.4x from
hoisting" was too optimistic: expect ~1.1x on bracket sweeps from the copy hoist
alone. The real bracket win lives in Phase 3 (loop/macro work), not Phase 1.
Consequence: **the ohl_nan/funding_nan hoist is demoted out of Phase 1** and folded
into the Phase 3 bracket work; Phase 1b keeps only the CAPM hoist (universal ~7%).
## Phase 1 — quick wins, zero parity risk (~2-3 days total)
Branch `perf/full-metrics-pydict` (DONE 2026-07-18)
- [x] `PyBacktestResult.metrics` + `profile` (result.rs): build via the existing
`metrics_to_pydict` / `profile_to_pydict` (backtest.rs), no JSON round-trip.
- [x] **Nested `trade_stats` hand-mirrored too** (`trade_stats_to_pydict`). This was
the missing half: `metrics_to_pydict` still round-tripped the nested
TradeStatistics through JSON. Harmless for LITE sweeps (trade_stats = None) but
the full `run_sweep` populates it on EVERY combo, so it was the dominant residual
cost. Fixing only the outer getter gave 1.7x; adding this gave 4-6x.
- [x] Serde-parity pinned: 5 unit tests green, incl. a new
`metrics_dict_matches_serde_with_trade_stats_signal_quality` for the
nested-nested `signal_quality`. Drift guard verified to BITE (sabotaging one
field fails both trade_stats tests with "drifted from serde").
- [x] End-to-end oracle: full `run_sweep` vs dedicated `mbt.run()` per combo (both
full path, so exact) -- 9 combos x (189 scalar + 162 trade_stats) fields,
**0 mismatches**.
- [ ] (deferred, optional) Route `to_df()`/`best()` through the `sweep_columns`
buffer path. Not needed to close Phase 1: the getter fix already removed the
JSON round-trip; what remains is inherent Python dict/DataFrame building.
**Measured (900 combos):**
| op | before | after | gain |
|---|---|---|---|
| `SweepResult.to_df()` | 49 us/combo | 11.7 us/combo | **4.2x** |
| `SweepResult.best()` | 15 us/combo | 2.4 us/combo | **6.3x** |
| `.metrics` per result | ~18 us | 2.8 us | **~6.4x** |
Note: the audit's "~50x" projection was wrong -- it carried over the scale of the
ORIGINAL 1M-sweep lite bug rather than this path's measured baseline. Real: 4-6x.
Python suite: 63 passed, 2 failed. Both failures (`test_golden_buy_and_hold`,
`test_sweep::test_sweep_returns_one_result_per_combo`) are **pre-existing** --
proven by stashing the change, rebuilding, and reproducing the identical
`2 failed, 63 passed`. Both share one root cause: the golden fixture yields a flat/
truncated equity curve (`[1000.0, 1000.0]`), which makes total_return 0.0 for every
combo. Tracked separately (see the pending `fix/python-test-suite` branch).
Branch `perf/hoist-capm` (Phase 1b, DONE 2026-07-18 -- perf number provisional)
- [x] CAPM benchmark returns hoisted into `run_sweep_lite` and `run_batch_lite`
via `hoist_capm_benchmark()`; `run_lite_on_aligned` takes
`hoisted_benchmark: Option<&[f64]>` and falls back to computing per run when
None. walk_forward passes None (unchanged behaviour).
- **Guard (verified in code):** hoisting is bit-identical ONLY when the run's
`sim_bars` is `&aligned.symbol_bars`, i.e. `coarse_bars` is None
(orchestrator.rs:2300 `if signal_ns > native_ns`). The helper re-detects
`native_ns` on the post-pre-resample bars and returns None under hybrid
resampling, where closes come from coarse bars the driver has no handle on.
A blind driver-side hoist would have silently changed alpha/beta there.
- [x] **Proven equivalent, not merely untested.** A temporary probe recomputed the
per-run benchmark alongside the hoisted one and asserted `to_bits()`
equality: green. Then the probe was made to `panic!` on entry to prove the
hoisted branch is actually REACHED -- it is, by 8 tests, and exactly the
right ones: `lite_matches_full_with_{stop_loss,take_profit,trailing_stop,
gap_through_stop,stop_loss_short,full_bracket_and_costs}_at_native_resolution`,
`lite_and_full_agree_on_max_drawdown_sign`, and
`per_strategy_orders_apply_and_batch_is_heterogeneous` (batch_lite).
Probe removed; `cargo test -p bt-core` = 91 passed, 0 failed.
- Note: `golden_buy_and_hold` fails under `--release` but passes in debug with
`BT_UNLOCKED=1`. That is the known licensing artifact (the dev bypass is
`#[cfg(debug_assertions)]`, so release runs locked and hits the Pro output
floor), not this change. It also does not exercise this change at all: it runs
`run()`, i.e. the full kernel, whose CAPM block was left untouched.
**Perf: MEASURED, interleaved A/B (10k combos x 26k bars, ema_cross).**
A plain before/after was NOT usable here: the Phase 0 baseline was taken while
the paper dashboard was loading the box, so comparing it against a later quiet
run would have credited the hoist with someone else's CPU. Instead the hoist was
put behind a temporary env toggle so ONE binary could run both arms, interleaved
A,B,A,B, in one environment. Paired deltas cancel any drift. Toggle removed after.
| | hoist OFF | hoist ON | delta |
|---|---|---|---|
| sim / combo | 246.2 us | 206.1 us | **+16.5%** |
| wall | 179.9 ms | 159.0 ms | **+11.9%** |
| throughput | 55,593 c/s | 62,883 c/s | |
Four paired rounds, tightly clustered (sim +15.9/+16.6/+17.5/+16.3%), which is
how we know the pairing worked. **The plan's ~1.08x estimate was too
conservative: the real figure is 1.12x on wall.** The audit had priced CAPM at
~1ns/bar (~26us/combo); the measured removal is ~40us/combo, because the pass
also does a resample-to-daily, a step_returns and their allocations per combo,
not just a linear scan.
### The MC-10M sentinel is NOT a load detector (correction)
Recorded because the old heuristic ("~6s clean vs ~16s loaded") is misleading
and cost real time this session. With every competing process killed and the
sweeps posting their best numbers of the day (66,458 c/s), the sentinel still
read ~16s, and inside one 3-run batch it printed `[16.14, 16.36, 7.47]`. It is
bimodal for reasons unrelated to CPU contention (10M paths: allocation /
first-touch / page-cache state), so it flags "loaded" on a quiet machine.
Use instead: interleaved A/B with paired deltas, which is robust to drift by
construction and needs no external notion of "clean".
Gate to close Phase 1: parity suite green (Rust goldens + Python mirrors), benches
re-run per protocol, numbers recorded in the audit doc.
## Phase 2 — medium effort, contained risk (~1-2 weeks)
Branch `perf/single-run-parallel` -- **CANCELLED 2026-07-18, premise was false.**
- [x] ~~rayon-join independent signal expressions~~ **ALREADY IMPLEMENTED.**
orchestrator.rs:889 evaluates each dependency level with `level.par_iter()`
whenever `level.len() >= 2`. The audit's exploration agent reported "fast/slow
EMA computed sequentially"; that was a misread, and it propagated into this
plan. Measured at 1M bars, N independent EMAs in one strategy:
| signals | signal_eval | us/signal | if it were sequential |
|---|---|---|---|
| 1 | 6,843 us | 6,843 | - |
| 2 | 10,071 us | 5,036 | 13,686 |
| 4 | 10,580 us | 2,645 | 27,372 |
| 16 | 17,110 us | 1,069 | 109,488 |
16 signals cost 2.5x one signal, not 16x. The parallelism is real and
working. Nothing to win here. (Each individual EMA is a sequential scan, so
the residual per-signal cost is irreducible without changing the recurrence.)
- [x] ~~Overlap output_build components~~ **NOT WORTH IT.** `metrics` borrows
`trace_equities` while the gather MOVES it, and metrics must run on
full-resolution equity (max_drawdown depends on it, orchestrator.rs comment
at the metrics call). Overlapping them needs an 8MB clone at 1M bars, which
eats most of the ~2.8ms theoretical saving. Best case was ~7% of a 27ms run.
**Consequence for the ceiling:** the audit put the single run at ~1.7x reachable.
That number assumed a sequential signal phase that does not exist. With signals
already parallel and output_build ownership-bound, the single-run path is close to
its practical ceiling; expect well under 1.2x, and it is NOT where the 2x lives.
The 2x target remains the CPU sweep, where per-combo signal work is genuine CPU
load (the sweep saturates all threads across combos, so intra-combo signal
parallelism is degenerate there and the transpiled-sweep item still stands).
Branch `perf/simd-dispatch` (target: ~1.1-1.15x sweep, more on signal-heavy runs)
- [ ] `is_x86_feature_detected!` runtime dispatch on bt-expr elementwise kernels only
(compare, IfElse, arithmetic). Folds and scans (EWM, rolling, sums) stay scalar.
- [ ] Parity: elementwise same-op-per-lane is bit-identical; add a test asserting
dispatch on/off equality on goldens. Baseline fallback keeps portability.
Optional branch `perf/full-sweep-traces`
- [ ] Optional trace retention on full run_sweep (orchestrator.rs:3480-3547), or at
minimum docs steering sweep users to lite + sweep_columns.
## Phase 3 — structural, the 2x closers (~3-5 weeks, sequential)
Branch `refactor/loop-extraction` FIRST (enabler, no behavior change)
- [ ] Mechanically extract the shared fill/equity/daily blocks from the four loop
transcriptions (full/lite x general/fast) and the two bracket macros.
Pure extraction: does not change WHICH metrics lite computes (lite contract).
- [ ] Golden + parity suites must be bit-identical before/after.
**De-risked 2026-07-18: the 1.33x premise holds, and is probably conservative.**
Measured with the paired-A/B discipline (each variant in its own process,
interleaved, only paired deltas trusted), 2500 combos x 26k bars:
| paired delta | value | rounds | reading |
|---|---|---|---|
| +1 indicator (fixed span, declared, unreferenced) | **-0.5 us/combo** | -0.8, -0.3, +11.1, -0.7 | an indicator costs ~nothing per combo |
| +1 elementwise pass (compare+when+add) | **+180 us/combo** | 178, 157, 183, 241 | one pass ~= 1.34x the WHOLE signal phase |
base: signal_eval 134.3 us/combo, simulation 199.7 us/combo.
The ~0 indicator delta is NOT pruning: the compiler compiles every entry of
`def.signals` with no dead-signal elimination
(crates/bt-strategy/src/compiler.rs:74-82). It is AMORTIZATION. A fixed-span EMA
is computed once and every combo hits IndicatorCache; a swept EMA over a 50x50
grid has 50 distinct spans shared by 50 combos each. So indicator math is
effectively free per combo in a 2D sweep, and nearly all of signal_eval is
per-combo overhead (env build, param binding, the elementwise chain, output
allocation) -- precisely what fusing the target into the per-bar loop removes.
Ceiling: the naive arithmetic says 1.68x, but do not quote that. It leans on a
slightly negative indicator delta (so "101% removable", an artifact), the
elementwise delta has 46% spread, and a real transpiled sweep still pays
per-combo param binding and indicator lookup. **Defensible: >= 1.33x, plausibly
1.4-1.7x.** Enough to justify the work; re-measure against the real
implementation rather than trusting this number.
Branch `perf/cpu-transpiled-sweep` (target: ~1.33x, stacked on Phase 1 => ~1.45x)
- [ ] Reuse the GPU hoist plan (build_hoist_plan) on CPU: fill hoisted indicator series
once per sweep, compute the target in-loop via `sim_fast_lite_core_single`
(orchestrator.rs:5766), which is already the CUDA kernel's CPU reference.
- [ ] Same eligibility gates as the GPU transpiler; anything else falls back to the
current vectorized-signal path, unchanged.
- [ ] Deliberately break the old vectorized path (temporarily) to prove the fallback
is still covered by tests. Parity anchored to bt-expr.
Branch `perf/lite-loop-tightening` (target: 36 -> ~24 cycles/bar, ~1.15-1.3x sweep)
- [ ] Branch elimination in the single-asset core: hoist the sizing_mode match, the
no-rebalance gates, null-path dispatch. Control flow and layout ONLY.
- [ ] Forbidden: FP reordering, FMA, fast-math of any kind. Op order is the parity
anchor. Bit-verify against bt-expr after every commit.
Gate to close Phase 3: composite CPU sweep >= 1.9x vs Phase 0 baseline on the 3-strategy
500k bench (median of 3, sentinel-clean), GPU numbers unchanged, parity green.
## Phase 4 — coverage (optional, per-gate decisions)
- [ ] Funding on run()'s CPU fast path (orchestrator.rs:1358-1364): aligns run() with
lite/GPU, big win for perp single runs.
- [ ] GPU metrics kernel fuse/overlap (gpu_sweep.rs:4395-4656): <=1.16x GPU.
- [ ] Multi-asset GPU: brackets support, occupancy (33% today).
Each: own golden work, own decision. None blocks the 2x goal.
## Non-goals (explicit)
- GPU 2x under fp64 bit-parity: not reachable (sim kernel 73%, SASS-audited ceiling).
fp32 stays the documented opt-in for speed-over-bits users.
- Changing the lite contract (cagr/calmar/ulcer != run()).
- Machine-specific build flags in shipped wheels.
- Data-loading work (cold 57ms @1M bars, <1% at bench scale): revisit only if a
many-fresh-process workflow becomes a product path.
## Compatibility invariant
No strategy loses support at any phase. Fast paths widen or stay put; everything not
eligible falls back to today's code, which keeps its own test coverage (proven by the
deliberate-break rule). Example strategies remain testable and sweepable throughout;
`examples/` runs green at every phase gate.
examples/00_template.py
+1 -1
View File
@@ -72,4 +72,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {perf_counter() - t0:.2f}s")
mbt.plot.tearsheet(result, show=True)
mbt.plot.tearsheet(result)
examples/01_trend_following.py
+1 -1
View File
@@ -75,4 +75,4 @@ if __name__ == "__main__":
print(f"\nElapsed: {elapsed:.3f}s")
# Plot
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
examples/02_mean_reversion.py
+1 -1
View File
@@ -63,4 +63,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
examples/03_multi_asset_momentum.py
+1 -1
View File
@@ -70,4 +70,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
examples/04_linear_regression.py
+1 -1
View File
@@ -101,4 +101,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
examples/05_stat_arb.py
+1 -1
View File
@@ -71,4 +71,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
examples/06_full_visualization.py
+13 -14
View File
@@ -92,7 +92,7 @@ if __name__ == "__main__":
)
# -- 3. Summary 3-panel ---------------------------------------------------
mbt.plot.summary(result, show=True)
mbt.plot.summary(result)
# -- 4. Candlestick chart (first symbol in universe) --------------------
mbt.plot.chart(
@@ -101,18 +101,17 @@ if __name__ == "__main__":
smas=[50],
n_bars=120,
interactive=False,
show=True,
)
# -- 5. Individual charts -------------------------------------------------
mbt.plot.equity(result, show=True)
mbt.plot.drawdown(result, show=True)
mbt.plot.monthly_returns(result, show=True)
mbt.plot.annual_returns(result, show=True)
mbt.plot.returns_histogram(result, show=True)
mbt.plot.var_chart(result, show=True)
mbt.plot.rolling_sharpe(result, show=True)
mbt.plot.rolling_volatility(result, show=True)
mbt.plot.equity(result)
mbt.plot.drawdown(result)
mbt.plot.monthly_returns(result)
mbt.plot.annual_returns(result)
mbt.plot.returns_histogram(result)
mbt.plot.var_chart(result)
mbt.plot.rolling_sharpe(result)
mbt.plot.rolling_volatility(result)
# -- 6. Sweep heatmap 2D -------------------------------------------------
# Sweep over RSI period and oversold threshold
@@ -160,7 +159,7 @@ if __name__ == "__main__":
"metric_grid": metric_grid,
}
print(f"Sweep done in {time.perf_counter() - t0:.1f}s")
mbt.plot.heatmap_2d(sweep_result, show=True)
mbt.plot.heatmap_2d(sweep_result)
# -- 7. Walk-forward validation -------------------------------------------
print("\nRunning walk-forward (manual folds)...")
@@ -201,11 +200,11 @@ if __name__ == "__main__":
"folds": wf_folds,
}
print(f"Walk-forward done in {time.perf_counter() - t0:.1f}s")
mbt.plot.walk_forward(wf_result, show=True)
mbt.plot.walk_forward(wf_result)
# -- 8. Monte Carlo -------------------------------------------------------
print("\nRunning Monte Carlo (1000 paths)...")
mbt.plot.monte_carlo(result, n_simulations=1000, seed=42, show=True)
mbt.plot.monte_carlo(result, n_simulations=1000, seed=42)
# -- 9. Parameter stability -----------------------------------------------
print("\nRunning stability analysis (RSI period)...")
@@ -241,7 +240,7 @@ if __name__ == "__main__":
"stability_score": 1.0 - (std_m / abs(mean_m)) if mean_m != 0 else 0.0,
}
print(f"Stability done in {time.perf_counter() - t0:.1f}s")
mbt.plot.stability(stab_result, show=True)
mbt.plot.stability(stab_result)
# -- 10. Research report (composite) --------------------------------------
print("\nGenerating research report...")
examples/07_walk_forward.py
+1 -1
View File
@@ -93,4 +93,4 @@ if __name__ == "__main__":
print(f"\n{len(folds)} folds in {elapsed:.2f}s")
if folds:
mbt.plot.walk_forward({"optimize_metric": metric, "folds": folds}, show=True)
mbt.plot.walk_forward({"optimize_metric": metric, "folds": folds})
examples/08_sweep_2d_heatmap.py
+1 -1
View File
@@ -87,4 +87,4 @@ if __name__ == "__main__":
"y_values": slow_values,
"metric": "t-stat(alpha)",
"metric_grid": metric_grid,
}, show=True)
})
examples/09_surface_3d.py
+1 -1
View File
@@ -84,4 +84,4 @@ if __name__ == "__main__":
"y_values": slow_values,
"metric": "t-stat(alpha)",
"metric_grid": metric_grid,
}, show=True)
})
examples/10_monte_carlo.py
+1 -1
View File
@@ -66,4 +66,4 @@ if __name__ == "__main__":
print(f"Elapsed: {time.perf_counter() - t0:.3f}s\n")
# 2. Monte Carlo fan chart
mbt.plot.monte_carlo(result, n_simulations=10000, seed=42, show=True)
mbt.plot.monte_carlo(result, n_simulations=10000, seed=42)
examples/11_portfolio.py
+1 -1
View File
@@ -72,4 +72,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.tearsheet(result, show=True)
mbt.plot.tearsheet(result)
examples/13_stochastic_simulation.py
-1
View File
@@ -141,5 +141,4 @@ if __name__ == "__main__":
mbt.plot.stochastic_paths(
result,
title=f"Mean-reverting model (S0=80, target=100, {N_PLOT:,} paths)",
show=True,
)
examples/14_multi_timeframe.py
+1 -1
View File
@@ -83,4 +83,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
mbt.plot.equity(result, show=True)
mbt.plot.equity(result)
examples/15_cross_exchange.py
+1 -1
View File
@@ -105,4 +105,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
result.plot_equity(show=True)
result.plot_equity()
examples/16_hashrate_exogene.py
+1 -1
View File
@@ -211,4 +211,4 @@ if __name__ == "__main__":
print(result.summary())
print(f"\nElapsed: {elapsed:.3f}s")
result.plot_equity(show=True)
result.plot_equity()
pyproject.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "manifoldbt"
version = "0.13.2"
version = "0.14.0"
description = "Rust-powered backtesting engine for quantitative research"
requires-python = ">=3.9"
license = { file = "LICENSE" }
python/manifoldbt/__init__.py
+173 -3
View File
@@ -29,6 +29,7 @@ from manifoldbt._native import (
run_portfolio as _run_portfolio_native,
py_ingest as _ingest_native,
py_import_csv as _import_csv_native,
py_import_dataframe as _import_dataframe_native,
)
from manifoldbt._serde import scalar_value_to_json
from manifoldbt.config import (
@@ -183,6 +184,32 @@ def _require_pro_over_combos(n_combos: int, what: str) -> None:
)
def _validate_swept_params(strategy: "Strategy", names, what: str) -> None:
"""Reject swept parameter names the strategy never declares.
Sweeping a name the strategy does not use is a silent no-op: the value is
merged into a parameter map nothing reads, so every combo runs the same
backtest and the sweep returns N identical results with no warning. That
is worse than an error, because an "optimisation" over thousands of combos
looks like it worked and its best result is meaningless.
A parameter counts as declared whether it came from ``mbt.param()`` inside
an expression or from an explicit ``.param()`` call: ``to_json_dict()``
merges both into ``parameters`` (and is memoised, so this costs nothing).
"""
declared = set(strategy.to_json_dict().get("parameters") or {})
unknown = [n for n in names if n not in declared]
if not unknown:
return
known = ", ".join(sorted(declared)) if declared else "none"
raise StrategyError(
f"{what}: parameter(s) {unknown} are not declared by strategy "
f"'{strategy.name}' (declared: {known}). Sweeping them would run the "
f"same backtest for every combination. Use mbt.param(\"name\") where "
f"the value is consumed, e.g. ema(close, mbt.param(\"fast\"))."
)
def _classify_error(exc: Exception) -> Exception:
"""Wrap a Rust ValueError/RuntimeError in a more specific exception."""
msg = str(exc)
@@ -682,6 +709,128 @@ def import_csv(
)
_BARS_REQUIRED_COLUMNS = ("timestamp", "open", "high", "low", "close", "volume")
def _df_to_bars_batch(data):
"""Normalise a pandas/polars DataFrame (or dict) to a pyarrow RecordBatch.
Output contract (what the native import expects): columns
``timestamp`` (timestamp[ns, UTC]), ``open/high/low/close/volume`` (f64).
Naive timestamps are assumed UTC. A pandas DatetimeIndex is promoted to
the ``timestamp`` column when the column is absent.
"""
import pyarrow as pa
# --- to Arrow Table (same dispatch as register_exo) ---
if hasattr(data, "to_arrow"):
# Polars DataFrame
table = data.to_arrow()
elif hasattr(data, "columns"):
# Pandas DataFrame
import pandas as pd
if "timestamp" not in data.columns and isinstance(data.index, pd.DatetimeIndex):
data = data.reset_index(names="timestamp")
table = pa.Table.from_pandas(data, preserve_index=False)
elif isinstance(data, dict):
table = pa.table(data)
else:
raise TypeError(
f"Unsupported data type: {type(data)}. Use a pandas/polars DataFrame or dict."
)
missing = [c for c in _BARS_REQUIRED_COLUMNS if c not in table.column_names]
if missing:
raise DataError(
f"DataFrame is missing required column(s): {', '.join(missing)}. "
f"Expected: {', '.join(_BARS_REQUIRED_COLUMNS)}"
)
table = table.select(list(_BARS_REQUIRED_COLUMNS))
# --- timestamp → timestamp[ns, UTC] ---
ts_type = table.schema.field("timestamp").type
if not pa.types.is_timestamp(ts_type):
raise DataError(
f"'timestamp' column must be a datetime type, got {ts_type}. "
"For epoch integers, convert first: pd.to_datetime(ts, unit='ms', utc=True)"
)
target_ts = pa.timestamp("ns", tz="UTC")
if ts_type != target_ts:
table = table.set_column(
0, pa.field("timestamp", target_ts), table.column(0).cast(target_ts)
)
# --- value columns → float64 ---
for i, name in enumerate(_BARS_REQUIRED_COLUMNS[1:], start=1):
if table.schema.field(i).type != pa.float64():
table = table.set_column(
i, pa.field(name, pa.float64()), table.column(i).cast(pa.float64())
)
if table.num_rows == 0:
raise DataError("DataFrame contains no data rows")
# Single contiguous batch for the zero-copy FFI crossing.
return table.combine_chunks().to_batches()[0]
def import_dataframe(
data,
symbol: str,
symbol_id: int,
*,
interval: str = "1m",
data_root: str = "data",
metadata_db: str = "metadata/metadata.sqlite",
exchange: str = "DATAFRAME",
asset_class: str = "crypto_spot",
) -> DataStore:
"""Import bars from an in-memory DataFrame into the Arrow IPC store. Free on all tiers.
The in-memory twin of :func:`import_csv`: edit your data as a DataFrame,
then import it directly — no intermediate CSV. Returns a :class:`DataStore`
ready for :func:`run` (same store, metadata and versioning as ``bt.ingest``).
Accepts a pandas DataFrame, polars DataFrame, or dict of columns with
``timestamp`` (datetime; naive values are assumed UTC), ``open``, ``high``,
``low``, ``close``, ``volume``. A pandas DatetimeIndex is used as
``timestamp`` if that column is absent. Rows must be sorted by timestamp.
Example::
df = pd.read_parquet("EURUSD_1m.parquet")
df["close"] = df["close"].clip(upper=1.5) # edit in memory
store = bt.import_dataframe(df, symbol="EURUSD", symbol_id=1,
interval="1m", asset_class="forex")
result = bt.run(strategy, config, store)
Args:
data: pandas/polars DataFrame or dict of columns.
symbol: Ticker name (e.g. ``"EURUSD"``, ``"BTCUSDT"``).
symbol_id: Unique integer ID for this symbol in the store.
interval: Bar interval of the rows (``"1m"``, ``"5m"``, ``"1h"``, ``"1d"``, ...).
data_root: Store directory (default ``"data"``).
metadata_db: Metadata SQLite path.
exchange: Exchange label for metadata (default ``"DATAFRAME"``).
asset_class: ``crypto_spot``, ``crypto_perp``, ``equity``, ``future``,
``option``, ``forex``, or ``index``.
"""
batch = _df_to_bars_batch(data)
try:
return _import_dataframe_native(
batch,
symbol=symbol,
symbol_id=symbol_id,
interval=interval,
data_root=data_root,
metadata_db=metadata_db,
exchange=exchange,
asset_class=asset_class,
)
except (ValueError, RuntimeError) as exc:
raise _classify_error(exc) from exc
def _ingest_single(
*, provider, symbol, symbol_id, start, end, interval, dataset,
data_root, metadata_db, exchange, asset_class, progress,
@@ -759,6 +908,7 @@ def run_sweep(
A :class:`SweepResult` with ``.to_df()``, ``.best()``, ``.plot_metric()``.
"""
_require_pro_over_combos(_grid_combos(param_grid), "Parameter sweep")
_validate_swept_params(strategy, param_grid.keys(), "Parameter sweep")
try:
config = _cap_output_resolution(config)
store = _resolve_store(config, store)
@@ -930,6 +1080,7 @@ def run_sweep_lite(
One :class:`BatchResultLite` per combo (Cartesian product order).
"""
_require_pro_over_combos(_grid_combos(param_grid), "Parameter sweep")
_validate_swept_params(strategy, param_grid.keys(), "Parameter sweep")
_require_pro_for_gpu(device, "GPU sweep")
try:
config = _cap_output_resolution(config)
@@ -939,7 +1090,11 @@ def run_sweep_lite(
name: [scalar_value_to_json(v) for v in values]
for name, values in param_grid.items()
})
return _run_sweep_lite_native(
# Wrapped in a list subclass: echoing a sweep in a notebook cell
# printed one BatchResultLite line per combo. Indexing, iteration and
# len() are unchanged.
from manifoldbt._reprs import wrap_sweep_lite
return wrap_sweep_lite(_run_sweep_lite_native(
strategy.to_json(),
grid_json,
cfg_json,
@@ -947,7 +1102,7 @@ def run_sweep_lite(
max_parallelism,
device,
precision,
)
))
except (ValueError, RuntimeError) as exc:
raise _classify_error(exc) from exc
@@ -1029,9 +1184,15 @@ def run_walk_forward(
# in `py_run_walk_forward` (check_feature("walk_forward")), so this cannot be
# bypassed by calling the native function directly.
_require_pro("Walk-forward optimization")
_validate_swept_params(strategy, (wf_config.get("param_grid") or {}).keys(),
"Walk-forward")
config = _prepare_config(config, strategy, store)
wf_json = json.dumps(_convert_param_grid_in_config(wf_config))
return _run_walk_forward_native(strategy.to_json(), wf_json, config.to_json(), store)
raw = _run_walk_forward_native(strategy.to_json(), wf_json, config.to_json(), store)
# Wrapped in a dict subclass: the raw dict holds a full equity curve per
# fold, so echoing it in a cell printed tens of thousands of floats.
from manifoldbt._reprs import wrap_walk_forward
return wrap_walk_forward(raw)
def run_sweep_2d(
@@ -1061,6 +1222,10 @@ def run_sweep_2d(
len(sweep_config.get("x_values", [])) * len(sweep_config.get("y_values", [])),
"2D parameter sweep",
)
_validate_swept_params(
strategy,
[n for n in (sweep_config.get("x_param"), sweep_config.get("y_param")) if n],
"2D parameter sweep")
config = _prepare_config(config, strategy, store)
sweep_json = json.dumps(_convert_scalar_values_in_sweep(sweep_config))
return _run_sweep_2d_native(strategy.to_json(), sweep_json, config.to_json(), store)
@@ -1088,6 +1253,10 @@ def run_stability(
Dict with ``stability_score``, ``metric_values``, ``mean_metric``, ``std_metric``.
"""
_require_pro_over_combos(len(stability_config.get("values", [])), "Parameter stability analysis")
_validate_swept_params(
strategy,
[n for n in (stability_config.get("param_name"),) if n],
"Parameter stability analysis")
config = _prepare_config(config, strategy, store)
stab_json = json.dumps(_convert_scalar_values_in_stability(stability_config))
return _run_stability_native(strategy.to_json(), stab_json, config.to_json(), store)
@@ -1404,6 +1573,7 @@ __all__ = [
# Data ingestion
"ingest",
"import_csv",
"import_dataframe",
# Run functions
"run",
"run_sweep",
python/manifoldbt/_reprs.py
+107
View File
@@ -0,0 +1,107 @@
"""Compact reprs for the big containers returned to notebooks.
A sweep returns one object per combo and a walk-forward carries a full
equity curve per fold, so echoing either in a Jupyter cell used to print
thousands of lines. These wrappers subclass ``list``/``dict`` so every
existing access keeps working (indexing, iteration, ``.keys()``, JSON
round-trips); only the repr changes.
"""
from __future__ import annotations
from typing import Any, Dict, List
_MAX_SCAN = 100_000 # cap the repr's own cost on million-combo sweeps
def _fmt(v: float) -> str:
"""Compact number: 3 significant-ish digits, thousands as k."""
if v is None:
return "?"
a = abs(v)
if a >= 1_000_000:
return f"{v / 1_000_000:.2f}M"
if a >= 1_000:
return f"{v / 1_000:.2f}k"
if a >= 1:
return f"{v:.2f}"
return f"{v:.4g}"
def _span(values) -> str:
vals = [v for v in values if v is not None]
if not vals:
return "n/a"
lo, hi = min(vals), max(vals)
return _fmt(lo) if lo == hi else f"{_fmt(lo)}..{_fmt(hi)}"
class SweepLiteResults(list):
"""``run_sweep_lite`` output: a list, with a one-line repr.
Printing 400 combos used to emit 400 lines of ``BatchResultLite(...)``.
"""
def __repr__(self) -> str:
n = len(self)
if n == 0:
return "SweepLiteResults(empty)"
head = self[:_MAX_SCAN]
eq = _span([getattr(r, "final_equity", None) for r in head])
sharpes = []
for r in head:
m = getattr(r, "metrics", None)
if isinstance(m, dict):
sharpes.append(m.get("sharpe"))
name = getattr(self[0], "strategy_name", "?")
parts = [f"{n:,} combos", f"strategy {name!r}", f"final_equity {eq}"]
if any(s is not None for s in sharpes):
parts.append(f"sharpe {_span(sharpes)}")
if n > _MAX_SCAN:
parts.append(f"(range over first {_MAX_SCAN:,})")
return ("<SweepLiteResults: " + " | ".join(parts) +
"\n r[i] for one combo, mbt.sweep_columns(r, 'sharpe') for arrays>")
class WalkForwardResult(dict):
"""``run_walk_forward`` output: a dict, with a one-line repr.
The raw dict carries a full IS and OOS equity curve per fold, so echoing
it in a cell used to print tens of thousands of floats.
"""
def __repr__(self) -> str:
folds = self.get("folds") or []
if not folds:
return "<WalkForwardResult: no folds>"
metric = self.get("optimize_metric", "sharpe")
def _m(fold, key):
v = fold.get(key)
return v.get(metric) if isinstance(v, dict) else v
is_v = [_m(f, "is_metrics") for f in folds]
oos_v = [_m(f, "oos_metrics") for f in folds]
lines = [
f"<WalkForwardResult: {len(folds)} folds | metric {metric!r} | "
f"IS {_span(is_v)} | OOS {_span(oos_v)}"
]
for f in folds:
best = f.get("best_params") or {}
flat = {k: (list(v.values())[0] if isinstance(v, dict) else v)
for k, v in best.items()}
i, o = _m(f, "is_metrics"), _m(f, "oos_metrics")
lines.append(
f" fold {f.get('fold_index', '?')}: "
f"IS {_fmt(i) if i is not None else '?':>8} "
f"OOS {_fmt(o) if o is not None else '?':>8} {flat}"
)
lines.append(" keys: " + ", ".join(sorted(self.keys())) + ">")
return "\n".join(lines)
def wrap_sweep_lite(results: List[Any]) -> "SweepLiteResults":
return SweepLiteResults(results)
def wrap_walk_forward(result: Dict[str, Any]) -> "WalkForwardResult":
return WalkForwardResult(result) if isinstance(result, dict) else result
python/manifoldbt/plot/__init__.py
+16 -3
View File
@@ -10,11 +10,24 @@ Quick start::
result = bt.run(strategy, config, store)
bt.plot.tearsheet(result) # full-page dashboard
bt.plot.equity(result, show=True) # single chart
bt.plot.equity(result) # single chart, opens on its own
Every chart is interactive (crosshair, hover, zoom). ``show=True`` opens it
in a native window (``pip install manifoldbt[window]``; falls back to a
Every chart is interactive (crosshair, hover, zoom) and **shows itself by
default**: plotting is what you asked for, so no ``show=`` is needed. Charts
open in a native window (``pip install manifoldbt[window]``; falls back to a
browser tab, which you can also force with ``show="browser"``).
Three cases opt out of showing automatically, because showing would be
wrong: passing ``save=`` (you asked for a file, not a window), running
under pytest/CI (a window there blocks the run), and running inside a
notebook, where the cell already renders the returned Figure and showing
would print a second copy of the same chart.
Pass an explicit ``show=True`` to override any of them, or ``show=False``
to get the Figure back silently and compose it yourself. To place a chart
in the middle of a notebook cell, where there is no trailing expression for
Jupyter to display, call IPython's ``display(fig)`` on the returned figure.
``save=".html"`` writes a responsive interactive page. Static ``save=".png"``
is optional and needs ``pip install manifoldbt[png]`` (pulls a headless Chromium).
"""
python/manifoldbt/plot/_utils.py
+71 -4
View File
@@ -29,6 +29,57 @@ def new_figure(
return fig
def _in_notebook() -> bool:
"""True inside a Jupyter/IPython kernel (not a plain terminal REPL)."""
try:
from IPython import get_ipython # type: ignore
except ImportError:
return False
try:
ip = get_ipython()
except Exception:
return False
return ip is not None and hasattr(ip, "kernel")
def _in_test_or_ci() -> bool:
"""True under pytest or on a CI runner.
A test that builds a figure must not queue a window: show() runs from
the atexit hook and blocks on the window process, so one bare plot call
in a test suite hangs the whole run until a human closes it.
"""
import os
return bool(os.environ.get("PYTEST_CURRENT_TEST") or os.environ.get("CI"))
def resolve_show(show: "bool | str | None",
save: Optional[Union[str, Path]]) -> "bool | str":
"""Resolve the ``show=None`` auto default.
A chart you asked for is a chart you want to see, so the default shows
it. Three cases opt out, because showing there would be wrong:
- ``save`` was given: you asked for a file, not a window.
- pytest/CI: show() runs from atexit and blocks on the window process.
- a notebook: the cell already renders the returned Figure. Calling
show() here too would emit a SECOND copy of the same chart, so the
notebook path stays silent and lets the cell do the rendering.
Explicit ``True``/``False``/``"browser"`` always wins. There is no
"render it inline" value to pass, because that is what the notebook
already does with the returned Figure; to place a chart mid-cell, call
IPython's ``display(fig)``.
"""
if show is not None:
return show
if save is not None:
return False
if _in_test_or_ci():
return False
return False if _in_notebook() else True
def format_pct(value: float, decimals: int = 1) -> str:
"""Format a decimal fraction as a percentage string."""
return f"{value * 100:+.{decimals}f}%"
@@ -43,7 +94,7 @@ def format_currency(value: float, currency: str = "USD") -> str:
def finalize(
fig,
*,
show: "bool | str" = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
dpi: int = 150,
window_size: Optional[Tuple[int, int]] = None,
@@ -52,10 +103,20 @@ def finalize(
``save`` routes on extension: ``.html`` writes a responsive interactive
page; image extensions (.png/.svg/.pdf/...) go through kaleido.
``show``: ``True`` (or ``"window"``) opens a native window (needs pywebview,
else falls back to a browser tab); ``"browser"`` forces a browser tab.
``show``: ``None`` (default) shows the chart unless ``save`` was given or
we are in a notebook (see :func:`resolve_show`); ``True`` (or ``"window"``)
opens a native window (needs pywebview, else falls back to a browser tab);
``"browser"`` forces a browser tab; ``False`` returns the figure silently.
``dpi`` is kept for backward compatibility and maps to an export scale.
"""
show = resolve_show(show, save)
if _in_notebook():
# new_figure() sets a pixel width sized for a window (1120px by
# default). A notebook cell is narrower than that, so the chart
# overflowed its output area: the right edge and the modebar were
# pushed out of view. Drop the fixed width and let it track the cell,
# keeping the height so the cell still has a definite size.
fig.update_layout(width=None, autosize=True)
if save is not None:
path = Path(save)
ext = path.suffix.lower()
@@ -71,7 +132,13 @@ def finalize(
) from exc
else:
write_responsive_html(fig, path)
if show == "browser":
if show == "inline" and _in_notebook():
# No-op on purpose. Rendering in the cell IS the notebook default, so
# calling show() here would emit a second copy of the chart the cell
# is already going to render. To place a chart mid-cell, where there
# is no trailing expression, use IPython's display(fig).
pass
elif show in ("browser", "inline"):
fig.show()
elif show: # True or "window" -> native window (browser tab fallback)
from manifoldbt.plot._window import open_in_window
python/manifoldbt/plot/backtest.py
+10 -10
View File
@@ -71,7 +71,7 @@ def summary(
result,
*,
figsize: Tuple[float, float] = (14, 8),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""The essential chart: TWR equity + buy-and-hold benchmark, trade activity.
@@ -264,7 +264,7 @@ def equity(
color: str = ACCENT,
title: str = "Equity Curve",
figsize: Tuple[float, float] = (14, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Plot the portfolio equity curve over time.
@@ -298,7 +298,7 @@ def benchmark_equity(
labels: Tuple[str, str] = ("Strategy", "Buy & Hold"),
title: str = "Strategy vs Benchmark",
figsize: Tuple[float, float] = (14, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Overlay strategy equity and a benchmark, both normalized to 100."""
@@ -339,7 +339,7 @@ def drawdown(
color: str = RED,
title: str = "Drawdown",
figsize: Tuple[float, float] = (14, 3),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Plot the drawdown as a filled area chart."""
@@ -373,7 +373,7 @@ def monthly_returns(
annotate: bool = True,
title: str = "Monthly Returns (%)",
figsize: Tuple[float, float] = (12, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Monthly returns heatmap (year rows x month columns + annual)."""
@@ -439,7 +439,7 @@ def annual_returns(
ax=None,
title: str = "Annual Returns",
figsize: Tuple[float, float] = (10, 4),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Annual returns bar chart with green/red conditional coloring."""
@@ -479,7 +479,7 @@ def returns_histogram(
bins: int = 100,
title: str = "Returns Distribution",
figsize: Tuple[float, float] = (12, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Histogram of daily returns with green/red coloring by sign."""
@@ -541,7 +541,7 @@ def var_chart(
bins: int = 120,
title: str = "Value at Risk",
figsize: Tuple[float, float] = (12, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Returns histogram with VaR and CVaR lines at 5% and 1% levels."""
@@ -610,7 +610,7 @@ def rolling_sharpe(
title: str = "Rolling Sharpe",
trading_days_per_year: float = 365.25,
figsize: Tuple[float, float] = (14, 4),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Rolling annualized Sharpe ratio."""
@@ -652,7 +652,7 @@ def rolling_volatility(
title: str = "Rolling Volatility",
trading_days_per_year: float = 365.25,
figsize: Tuple[float, float] = (14, 4),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Rolling annualized volatility."""
python/manifoldbt/plot/chart.py
+1 -1
View File
@@ -212,7 +212,7 @@ def chart(
n_bars: int = 120,
interactive: bool = True,
figsize: Tuple[float, float] = (14, 7),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
):
"""Plot candlestick chart with indicators and trade markers.
python/manifoldbt/plot/research.py
+290 -41
View File
@@ -112,14 +112,29 @@ def heatmap_2d(
annotate: bool = True,
fmt: str = ".3f",
highlight_best: bool = True,
zones: "bool | List[float] | None" = None,
drift: int = 2,
title: Optional[str] = None,
figsize: Tuple[float, float] = (10, 8),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""2D parameter sweep heatmap from ``run_sweep_2d()`` result.
Expected keys: metric_grid, x_values, y_values, x_param, y_param, metric.
Args:
zones: Colour cells by discrete robustness zone instead of by the
metric. A zone is what the area still guarantees when the
parameters drift by ``drift`` cells, so a lucky spike is shown in
a low zone despite scoring well on its own cell. ``True`` picks
the bands (conventional 0/0.5/1.0/1.5 for risk-adjusted ratios,
an even split of the observed range otherwise); pass a list of
thresholds to set them yourself. The metric value stays on hover
and in the cell labels. Same option as ``surface_3d``.
drift: Neighbourhood radius in grid cells for the worst case. Cells,
not parameter units: with an x step of 1 and a y step of 5,
``drift=2`` means +/-2 on x but +/-10 on y.
"""
with theme_context():
grid = np.array(sweep_result["metric_grid"], dtype=np.float64)
@@ -135,22 +150,50 @@ def heatmap_2d(
text = np.vectorize(lambda v: "" if np.isnan(v) else f"{v:{fmt}}")(grid)
fig = new_figure(figsize)
fig.add_trace(go.Heatmap(
z=grid, x=x_vals, y=y_vals,
colorscale=CS_SEQUENTIAL,
text=text, texttemplate="%{text}" if text is not None else None,
textfont=dict(size=9),
hovertemplate=(
f"{x_param} %{{x}}<br>{y_param} %{{y}}<br>"
f"{metric} %{{z:{fmt}}}<extra></extra>"
),
colorbar=dict(outlinewidth=0, thickness=12),
hoverongaps=False,
))
if zones:
worst = _worst_case(grid, drift)
band, scale, labels, edges, n_bands = _zone_bands(worst, zones, metric)
# z carries the band so colour is discrete; the metric and what it
# holds ride along in customdata so the cell still reports both.
fig.add_trace(go.Heatmap(
z=band, x=x_vals, y=y_vals,
colorscale=scale, zmin=-0.5, zmax=n_bands - 0.5,
customdata=np.dstack((grid, worst)),
text=text, texttemplate="%{text}" if text is not None else None,
textfont=dict(size=9),
hovertemplate=(
f"{x_param} %{{x}}<br>{y_param} %{{y}}<br>"
f"{metric} %{{customdata[0]:{fmt}}}<br>"
f"held %{{customdata[1]:{fmt}}}<extra></extra>"
),
colorbar=dict(
title=dict(text=f"{metric}<br>held under drift", side="right"),
outlinewidth=0, thickness=12,
tickmode="array", tickvals=list(range(n_bands)),
ticktext=labels),
hoverongaps=False,
))
else:
fig.add_trace(go.Heatmap(
z=grid, x=x_vals, y=y_vals,
colorscale=CS_SEQUENTIAL,
text=text, texttemplate="%{text}" if text is not None else None,
textfont=dict(size=9),
hovertemplate=(
f"{x_param} %{{x}}<br>{y_param} %{{y}}<br>"
f"{metric} %{{z:{fmt}}}<extra></extra>"
),
colorbar=dict(outlinewidth=0, thickness=12),
hoverongaps=False,
))
best_label = None
if highlight_best:
best_idx = _plateau_best(grid)
if zones:
# Match the colouring: best = what holds up, not the spike.
best_idx = np.unravel_index(np.argmax(worst), worst.shape)
else:
best_idx = _plateau_best(grid)
best_val = grid[best_idx]
best_x = x_vals[best_idx[1]]
best_y = y_vals[best_idx[0]]
@@ -163,7 +206,12 @@ def heatmap_2d(
x0=best_x - dx, x1=best_x + dx, y0=best_y - dy, y1=best_y + dy,
line=dict(color="white", width=2.5),
)
best_label = f"best: {best_val:{fmt}} ({x_param}={best_x:.0f}, {y_param}={best_y:.0f})"
kind = "most robust" if zones else "plateau centre"
best_label = (f"{kind}: {best_val:{fmt}} "
f"({x_param}={best_x:.0f}, {y_param}={best_y:.0f})")
if zones:
best_label += (f", holds {worst[best_idx]:{fmt}} "
f"under +/-{drift} cells")
combos = nx * ny
main_title = title or f"{metric} · Parameter Sweep ({combos:,} combos)"
@@ -188,21 +236,162 @@ def heatmap_2d(
# ── 3D Surface Plot ─────────────────────────────────────────────────────────
def _worst_case(grid: np.ndarray, radius: int) -> np.ndarray:
"""Lowest value reachable within +/-``radius`` cells of each cell.
This is what a combo still returns if the parameters drift, as opposed
to what its own cell scored. Lucky spikes collapse to their surroundings;
plateaus keep their value. Edges are replicated so the border is not
flattered by having fewer neighbours.
"""
if radius < 1:
return grid
filled = np.nan_to_num(grid, nan=np.nanmin(grid))
padded = np.pad(filled, radius, mode="edge")
n, m = filled.shape
stack = np.stack([padded[i:i + n, j:j + m]
for i in range(2 * radius + 1)
for j in range(2 * radius + 1)])
return stack.min(axis=0)
# Metrics where 0 separates losing from winning, so 0 is worth keeping as a
# band edge even when the data would not have put one there.
_RATIO_METRICS = ("sharpe", "sortino", "calmar", "tstat_alpha", "information")
_ZONE_COLORS = ["#3f1d1d", "#7c3a1d", "#8a7a1e", "#2f6b3a", ACCENT]
def _nice_step(span: float, n_bands: int) -> float:
"""A 1/2/2.5/5 x 10^k step covering ``span`` in about ``n_bands`` steps.
Rounded steps keep the legend readable: "0.8 - 1.2" rather than
"0.7834 - 1.2017".
"""
if not np.isfinite(span) or span <= 0:
return 1.0
raw = span / n_bands
mag = 10.0 ** np.floor(np.log10(raw))
for m in (1.0, 2.0, 2.5, 5.0):
if raw <= m * mag:
return m * mag
return 10.0 * mag
def _auto_edges(worst: np.ndarray, metric: str, n_bands: int = 5):
"""Band edges fitted to the data, snapped to round numbers.
Fixed conventional thresholds (0/0.5/1.0/1.5 for a Sharpe) collapse to a
single flat band whenever the sweep happens to live inside one of them,
which is common: a grid whose guaranteed Sharpe runs 1.5-2.0 came out
entirely one colour.
Edges sit at -1.5 to +1.5 standard deviations around the sweep's mean, so
the zones say how exceptional a region is *within this sweep*. That is a
relative statement, not a quality certificate: a sweep where every combo
loses money still has a top zone, it is just the least bad. Read the
colourbar, which prints the real thresholds, and pass explicit
``zones=[...]`` whenever the bands must mean something absolute.
"""
finite = worst[np.isfinite(worst)]
if finite.size == 0:
return [0.0]
lo, hi = float(finite.min()), float(finite.max())
if hi <= lo: # a flat grid has nothing to band
return [lo]
# Quantiles, not an even split of the range. Taking a minimum over the
# drift window skews the distribution hard toward its low tail, so even
# edges dumped 93% of the cells into one band and the map came out flat.
# Quantiles balance the bands by construction; the snap keeps the numbers
# readable and the colourbar prints them.
# Bands in standard deviations around the mean of the sweep.
#
# Quantiles were the other candidate and they are worse here: they force
# ~20% of cells into every band, so a grid that is genuinely uniform
# after erosion still comes out looking structured. Sigma bands scale
# with the actual dispersion, so a flat sweep reads flat and a sweep with
# a real standout region shows it. They also carry a meaning a reader can
# use: "+1 sigma" is how exceptional the region is for THIS sweep.
mu, sd = float(np.mean(finite)), float(np.std(finite))
if sd <= 0:
return [lo]
sigmas = np.linspace(-1.5, 1.5, n_bands - 1) # 5 bands -> -1.5..+1.5
raw_edges = mu + sigmas * sd
step = _nice_step(float(raw_edges[-1] - raw_edges[0]),
max(len(raw_edges) - 1, 1))
edges = sorted({round(float(np.round(e / step) * step), 10)
for e in raw_edges})
edges = [e for e in edges if lo < e < hi]
if len(edges) < len(raw_edges):
# Rounding merged edges (a very tight spread): keep them unsnapped.
edges = sorted({float(f"{e:.4g}") for e in raw_edges if lo < e < hi})
# 0 is a real boundary for a ratio: above it you make money, below you
# lose it. Keep it even if the rounding would have skipped it.
if any(k in metric.lower() for k in _RATIO_METRICS) and lo < 0.0 < hi:
edges = sorted(set(edges + [0.0]))
if len(edges) > n_bands - 1: # drop the edge nearest 0, not 0 itself
nonzero = [e for e in edges if e != 0.0]
drop = min(nonzero, key=lambda e: abs(e))
edges.remove(drop)
return edges or [(lo + hi) / 2.0]
def _zone_bands(worst: np.ndarray, zones, metric: str):
"""Resolve ``zones`` into thresholds, then bucket ``worst`` into bands."""
if zones is True:
edges = _auto_edges(worst, metric)
else:
edges = sorted(float(z) for z in zones)
band = np.digitize(worst, edges).astype(float)
n_bands = len(edges) + 1
labels = [f"< {edges[0]:g}"]
labels += [f"{edges[i]:g} - {edges[i + 1]:g}" for i in range(len(edges) - 1)]
labels.append(f">= {edges[-1]:g}")
colors = _ZONE_COLORS
if n_bands != len(colors): # stretch or trim the ramp to the band count
idx = np.linspace(0, len(colors) - 1, n_bands).round().astype(int)
colors = [colors[i] for i in idx]
scale = []
for i, c in enumerate(colors): # duplicated stops = hard borders
scale.append([i / n_bands, c])
scale.append([(i + 1) / n_bands, c])
return band, scale, labels, edges, n_bands
def surface_3d(
sweep_result: Dict[str, Any],
*,
highlight_best: bool = True,
zones: "bool | List[float] | None" = None,
drift: int = 2,
title: Optional[str] = None,
figsize: Tuple[float, float] = (12, 8),
elev: float = 30,
azim: float = -45,
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""3D surface plot from a 2D parameter sweep result.
Same input format as ``heatmap_2d``. ``elev``/``azim`` are kept for
backward compatibility and mapped to the plotly camera.
Args:
zones: Colour the surface by discrete robustness zones instead of by
height. Height still shows the metric; colour shows what each
area still guarantees when the parameters drift by ``drift``
cells, so a lucky spike lands in a low zone despite standing
tall. ``True`` picks the bands (conventional 0/0.5/1.0/1.5 for
risk-adjusted ratios, an even split of the observed range
otherwise); pass a list of thresholds to set them yourself,
which is what you want whenever the bands carry meaning.
drift: Neighbourhood radius in grid cells used for the worst case.
Note this is cells, not parameter units: with an x step of 1 and
a y step of 5, ``drift=2`` means +/-2 on x but +/-10 on y.
"""
with theme_context():
grid = np.array(sweep_result["metric_grid"], dtype=np.float64)
@@ -213,34 +402,94 @@ def surface_3d(
metric = sweep_result.get("metric", "metric")
fig = new_figure(figsize)
fig.add_trace(go.Surface(
x=x_vals, y=y_vals, z=grid,
colorscale=CS_SEQUENTIAL, opacity=0.98,
colorbar=dict(title=dict(text=metric, side="right"),
outlinewidth=0, thickness=13, len=0.6),
lighting=dict(ambient=0.75, diffuse=0.5, roughness=0.9, specular=0.1),
contours=dict(z=dict(show=True, usecolormap=True, project_z=True,
width=1)),
hovertemplate=(
f"{x_param} %{{x:.2f}}<br>{y_param} %{{y:.2f}}<br>"
f"{metric} %{{z:.3f}}<extra></extra>"
),
))
lighting = dict(ambient=0.75, diffuse=0.5, roughness=0.9, specular=0.1)
if zones:
worst = _worst_case(grid, drift)
band, scale, labels, edges, n_bands = _zone_bands(worst, zones, metric)
fig.add_trace(go.Surface(
x=x_vals, y=y_vals, z=grid,
surfacecolor=band, colorscale=scale,
cmin=-0.5, cmax=n_bands - 0.5, opacity=0.98,
colorbar=dict(
title=dict(text=f"{metric}<br>held under drift", side="right"),
outlinewidth=0, thickness=13, len=0.62,
tickmode="array", tickvals=list(range(n_bands)),
ticktext=labels),
lighting=lighting,
contours=dict(z=dict(show=True, color="rgba(255,255,255,0.13)",
width=1)),
customdata=worst,
hovertemplate=(
f"{x_param} %{{x:.2f}}<br>{y_param} %{{y:.2f}}<br>"
f"{metric} %{{z:.3f}}<br>held %{{customdata:.3f}}"
f"<extra></extra>"
),
))
else:
fig.add_trace(go.Surface(
x=x_vals, y=y_vals, z=grid,
colorscale=CS_SEQUENTIAL, opacity=0.98,
colorbar=dict(title=dict(text=metric, side="right"),
outlinewidth=0, thickness=13, len=0.6),
lighting=lighting,
contours=dict(z=dict(show=True, usecolormap=True, project_z=True,
width=1)),
hovertemplate=(
f"{x_param} %{{x:.2f}}<br>{y_param} %{{y:.2f}}<br>"
f"{metric} %{{z:.3f}}<extra></extra>"
),
))
best_label = None
if highlight_best:
best_idx = _plateau_best(grid)
if zones:
# With zones on, "best" means the combo that holds up best
# under drift, not the tallest cell. Reporting the spike here
# would contradict the colouring right next to it.
best_idx = np.unravel_index(np.argmax(worst), worst.shape)
held = worst[best_idx]
else:
best_idx = _plateau_best(grid)
held = None
best_val = grid[best_idx]
bx = x_vals[best_idx[1]]
by = y_vals[best_idx[0]]
# A dot sitting exactly at best_val is half-buried in the surface
# it marks, and a stem dropped to the floor runs underneath that
# surface, hidden by it. So the marker is a pin standing ABOVE
# the peak: the stalk clears the geometry and stays readable from
# any camera angle and over any colour.
span = float(np.nanmax(grid) - np.nanmin(grid)) or 1.0
tip = best_val + span * 0.10
fig.add_trace(go.Scatter3d(
x=[bx], y=[by], z=[best_val], mode="markers",
marker=dict(color="white", size=6,
line=dict(color="black", width=2)),
name="best", showlegend=False,
hovertemplate=f"best {metric} %{{z:.3f}}<extra></extra>",
x=[bx, bx], y=[by, by], z=[best_val, tip], mode="lines",
line=dict(color=WHITE, width=5),
name="best", showlegend=False, hoverinfo="skip",
))
best_label = f"best: {best_val:.3f} ({x_param}={bx:.0f}, {y_param}={by:.0f})"
# Name the criterion. Calling this "best <metric>" was a lie
# whenever zones were on: it is not the highest cell, it is the
# one that survives drift, and the highest cell is elsewhere and
# visibly taller.
if zones:
pin_text = (f"most robust<br>{metric} {best_val:.3f}"
f"<br>holds {worst[best_idx]:.3f} "
f"under +/-{drift} cells")
else:
pin_text = f"plateau centre<br>{metric} {best_val:.3f}"
fig.add_trace(go.Scatter3d(
x=[bx], y=[by], z=[tip], mode="markers",
marker=dict(color=WHITE, size=9, symbol="diamond",
line=dict(color="black", width=3)),
name="best", showlegend=False,
hovertemplate=pin_text + "<extra></extra>",
))
kind = "most robust" if zones else "plateau centre"
best_label = (f"{kind}: {best_val:.3f} "
f"({x_param}={bx:.0f}, {y_param}={by:.0f})")
if held is not None:
best_label += f", holds {held:.3f} under +/-{drift} cells"
# Map matplotlib elev/azim to a plotly camera eye position
r = 1.9
@@ -283,7 +532,7 @@ def walk_forward(
oos_color: str = ORANGE,
title: Optional[str] = None,
figsize: Tuple[float, float] = (10, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Walk-forward analysis chart.
@@ -507,7 +756,7 @@ def stability(
band_alpha: float = 0.15,
title: Optional[str] = None,
figsize: Tuple[float, float] = (10, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Parameter stability chart with mean +/- std shaded bands.
@@ -572,7 +821,7 @@ def correlation_matrix(
annotate: bool = True,
title: str = "Correlation Matrix",
figsize: Tuple[float, float] = (8, 7),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Symbol correlation matrix heatmap."""
@@ -652,7 +901,7 @@ def monte_carlo(
title: Optional[str] = None,
figsize: Tuple[float, float] = (12, 5),
seed: Optional[int] = None,
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Monte Carlo fan chart with percentile bands, sample paths, and risk stats.
@@ -805,7 +1054,7 @@ def stochastic_paths(
band_color: str = ACCENT,
title: Optional[str] = None,
figsize: Tuple[float, float] = (12, 5),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
) -> go.Figure:
"""Fan chart for stochastic simulation paths with percentile bands.
python/manifoldbt/plot/tearsheet.py
+21 -16
View File
@@ -16,7 +16,7 @@ from manifoldbt.plot._theme import (
theme_context,
)
from manifoldbt.plot._convert import equity_with_dates
from manifoldbt.plot._utils import auto_title, chart_div, format_pct
from manifoldbt.plot._utils import auto_title, chart_div, format_pct, resolve_show
from manifoldbt.plot.backtest import (
annual_returns,
drawdown,
@@ -139,7 +139,7 @@ def tearsheet(
*,
benchmark=None,
title: Optional[str] = None,
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
dpi: int = 150,
plotlyjs: str = "cdn",
@@ -165,14 +165,16 @@ def tearsheet(
# ── Generate interactive chart divs ────────────────────────────
with theme_context():
div_summary = _div(summary(result), height=520)
div_dd = _div(drawdown(result), height=210)
div_annual = _div(annual_returns(result), height=330)
div_monthly = _div(monthly_returns(result), height=340)
div_hist = _div(returns_histogram(result), height=340)
div_sharpe = _div(rolling_sharpe(result), height=300)
div_vol = _div(rolling_volatility(result), height=300)
div_var = _div(var_chart(result), height=340)
# show=False on every panel: these are embedded as divs in the page
# below, so the auto-show default would open 8 stray windows.
div_summary = _div(summary(result, show=False), height=520)
div_dd = _div(drawdown(result, show=False), height=210)
div_annual = _div(annual_returns(result, show=False), height=330)
div_monthly = _div(monthly_returns(result, show=False), height=340)
div_hist = _div(returns_histogram(result, show=False), height=340)
div_sharpe = _div(rolling_sharpe(result, show=False), height=300)
div_vol = _div(rolling_volatility(result, show=False), height=300)
div_var = _div(var_chart(result, show=False), height=340)
# ── Metrics ───────────────────────────────────────────────────
ret = metrics.get("total_return", 0)
@@ -273,7 +275,9 @@ def tearsheet(
if save is not None:
Path(save).write_text(html, encoding="utf-8")
if show:
# A report is an HTML page, not a Figure: it always opens in a browser
# tab, so "inline" resolves to the same thing here.
if resolve_show(show, save):
if save is not None:
report_path = Path(save).resolve()
else:
@@ -295,7 +299,7 @@ def research_report(
*,
title: str = "Research Report",
figsize: tuple = (14, 6),
show: bool = False,
show: "bool | str | None" = None,
save: Optional[Union[str, Path]] = None,
dpi: int = 150,
) -> List[Any]:
@@ -309,12 +313,13 @@ def research_report(
_ = title
figs = []
with theme_context():
# show=False: this function does its own showing at the end.
if sweep_result is not None:
figs.append(heatmap_2d(sweep_result, figsize=figsize))
figs.append(heatmap_2d(sweep_result, figsize=figsize, show=False))
if wf_result is not None:
figs.append(walk_forward(wf_result, figsize=figsize))
figs.append(walk_forward(wf_result, figsize=figsize, show=False))
if stability_result is not None:
figs.append(stability(stability_result, figsize=figsize))
figs.append(stability(stability_result, figsize=figsize, show=False))
if not figs:
raise ValueError("At least one result (sweep, wf, or stability) required.")
@@ -330,7 +335,7 @@ def research_report(
else:
scale = max(1.0, dpi / 96.0)
f.write_image(str(out), scale=scale)
if show:
if resolve_show(show, save):
for f in figs:
f.show()
python/manifoldbt/sweep.py
+3 -1
View File
@@ -107,7 +107,9 @@ class SweepResult:
df = self.to_df(backend="pandas")
param_cols = [c for c in df.columns if c.startswith("param_")]
show = kwargs.pop("show", True)
# None = the auto default (show, unless save= or a notebook); both
# branches below hand it to finalize(), which resolves it.
show = kwargs.pop("show", None)
save = kwargs.pop("save", None)
if len(param_cols) == 2:
python/tests/test_golden_buy_and_hold.py
+21 -9
View File
@@ -11,13 +11,13 @@ import pytest
import manifoldbt as bt
from manifoldbt import run_with_parquet
# The golden fixtures were generated at full (Pro) resolution; the Community
# resolution cap changes the equity-point count and the comparison is
# meaningless. CI unlocks via BT_UNLOCKED=1 (debug builds); locally this needs
# an activated Pro license.
# The golden fixtures assert on 1-second output resolution, below even the Pro
# floor (60s) — exactly like the Rust golden test, which sets BT_UNLOCKED=1.
# The override is only honored by debug builds (cargo test / maturin develop),
# so this needs BOTH: a dev build and BT_UNLOCKED=1 in the environment.
pytestmark = pytest.mark.skipif(
bt.license_info()[0] != "Pro",
reason="requires Pro (fixtures generated at sub-daily resolution); activate a license or use a BT_UNLOCKED dev build",
os.environ.get("BT_UNLOCKED") != "1",
reason="requires BT_UNLOCKED=1 on a dev (debug) build: fixtures assert 1s output, below the Pro 60s floor",
)
@@ -38,9 +38,14 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir):
universe=[1],
time_range_start=0,
time_range_end=4_000_000_000,
bar_interval={"Days": 1},
# The fixture is 4 bars at 1-second spacing; the Rust golden test runs
# them at Seconds(1) with per-bar output. Days(1) would resample the
# whole range into a single bar and the comparison would be meaningless.
bar_interval={"Seconds": 1},
output_resolution={"Seconds": 1},
initial_capital=1000.0,
currency="USD",
risk_free_rate=0.025,
execution=bt.ExecutionConfig(
signal_delay=1,
execution_price="AtClose",
@@ -92,8 +97,11 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir):
with open(os.path.join(golden_buy_hold_dir, "expected_metrics.json")) as f:
expected_metrics = json.load(f)
# Mirror the Rust golden test: annualized metrics (CAGR, volatility,
# sharpe, sortino, calmar) are not compared because the fixture uses 4
# synthetic 1-second bars, making annualization numerically extreme.
metrics = result.metrics
for key in expected_metrics:
for key in ("total_return", "max_drawdown"):
assert abs(metrics[key] - expected_metrics[key]) <= 1e-12, (
f"Metric {key}: {metrics[key]} != {expected_metrics[key]}"
)
@@ -102,7 +110,11 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir):
with open(os.path.join(golden_buy_hold_dir, "expected_manifest_snapshot.json")) as f:
expected_manifest = json.load(f)
# Mirror the Rust golden test: engine_version is excluded from the snapshot
# (it tracks the crate version and would break on every release bump);
# assert only that it is populated.
manifest = result.manifest
assert manifest["strategy_name"] == expected_manifest["strategy_name"]
assert manifest["engine_version"] == expected_manifest["engine_version"]
assert manifest["engine_version"], "engine_version should be populated"
assert manifest["data_versions"].get("bars_1m", "") == expected_manifest["data_version"]
assert manifest["config"] == expected_manifest["config"]
python/tests/test_import_dataframe.py
+164
View File
@@ -0,0 +1,164 @@
"""Tests for bt.import_dataframe — in-memory DataFrame → Arrow IPC store.
The contract under test: import_dataframe is the in-memory twin of
import_csv. Same data through either path must produce an identical store
(same backtest results), and the normalisation layer must give clear errors
for bad inputs instead of a Rust panic.
"""
import os
import pytest
import manifoldbt as bt
pd = pytest.importorskip("pandas")
N_BARS = 120
START_MS = 1_577_836_800_000 # 2020-01-01T00:00:00Z
def _bars_df(n=N_BARS, tz="UTC"):
"""Synthetic 1m bars as a pandas DataFrame."""
ts = pd.date_range("2020-01-01", periods=n, freq="1min", tz=tz)
close = [100.0 + i * 0.5 for i in range(n)]
return pd.DataFrame(
{
"timestamp": ts,
"open": close,
"high": [c + 1.0 for c in close],
"low": [c - 1.0 for c in close],
"close": close,
"volume": [10.0] * n,
}
)
def _store_paths(tmp_path, name):
root = tmp_path / name
return str(root / "data"), str(root / "metadata.sqlite")
def _import_df(df, tmp_path, name="df", **kw):
data_root, metadata_db = _store_paths(tmp_path, name)
os.makedirs(os.path.dirname(metadata_db), exist_ok=True)
return bt.import_dataframe(
df, symbol="BTCUSDT", symbol_id=1,
data_root=data_root, metadata_db=metadata_db, **kw
)
def _run_buy_and_hold(store):
strategy = bt.Strategy(
name="bh",
signals={"signal": bt.lit(1.0)},
position_sizing=bt.col("signal"),
)
config = bt.BacktestConfig(
universe=[1],
time_range_start=0,
time_range_end=START_MS * 1_000_000 + N_BARS * 60_000_000_000,
bar_interval={"Minutes": 1},
initial_capital=1000.0,
currency="USD",
execution=bt.ExecutionConfig(
signal_delay=1,
execution_price="AtClose",
max_position_pct=1.0,
allow_short=False,
allow_fractional=True,
skip_gap_bars=False,
position_sizing_mode="Units",
),
fees=bt.FeeConfig(),
slippage={"FixedBps": {"bps": 0.0}},
rng_seed=7,
)
return bt.run(strategy, config, store)
def test_import_dataframe_roundtrip(tmp_path):
"""DataFrame → store → run produces a usable backtest."""
store = _import_df(_bars_df(), tmp_path)
assert store.resolve_symbol("BTCUSDT") == 1
result = _run_buy_and_hold(store)
equity = result.equity_curve.to_pylist()
assert len(equity) > 0
# Price rises monotonically → buy & hold ends above initial capital.
assert equity[-1] > 1000.0
def test_import_dataframe_matches_import_csv(tmp_path):
"""Same bars through import_csv and import_dataframe → identical results."""
df = _bars_df()
# CSV path (standard format: epoch-ms timestamp).
#
# Built from START_MS rather than derived from the datetime column:
# `.astype("int64")` returns the underlying integer in the COLUMN's
# resolution, which pandas picks for itself. Locally that was ns (so
# //1e6 gave ms), on CI it was us (so //1e6 gave seconds) and the import
# rejected the row. The bars are 1 minute apart by construction here, so
# spelling the epoch out keeps the CSV identical on every pandas.
csv_df = df.copy()
csv_df["timestamp"] = [START_MS + i * 60_000 for i in range(len(csv_df))]
csv_path = tmp_path / "bars.csv"
csv_df.to_csv(csv_path, index=False)
csv_root, csv_meta = _store_paths(tmp_path, "csv")
os.makedirs(os.path.dirname(csv_meta), exist_ok=True)
store_csv = bt.import_csv(
str(csv_path), symbol="BTCUSDT", symbol_id=1,
data_root=csv_root, metadata_db=csv_meta,
)
store_df = _import_df(df, tmp_path)
res_csv = _run_buy_and_hold(store_csv)
res_df = _run_buy_and_hold(store_df)
assert res_df.equity_curve.to_pylist() == res_csv.equity_curve.to_pylist()
assert res_df.metrics == res_csv.metrics
def test_import_dataframe_naive_timestamps_assumed_utc(tmp_path):
"""tz-naive datetimes are accepted and treated as UTC."""
naive = _bars_df(tz=None)
aware = _bars_df(tz="UTC")
store_naive = _import_df(naive, tmp_path, name="naive")
store_aware = _import_df(aware, tmp_path, name="aware")
assert _run_buy_and_hold(store_naive).equity_curve.to_pylist() == \
_run_buy_and_hold(store_aware).equity_curve.to_pylist()
def test_import_dataframe_datetime_index_promoted(tmp_path):
"""A pandas DatetimeIndex is used as the timestamp column."""
df = _bars_df().set_index("timestamp")
assert "timestamp" not in df.columns
store = _import_df(df, tmp_path)
assert store.resolve_symbol("BTCUSDT") == 1
def test_import_dataframe_polars(tmp_path):
"""Polars DataFrames go through the zero-copy to_arrow path."""
pl = pytest.importorskip("polars")
df = pl.from_pandas(_bars_df())
store = _import_df(df, tmp_path)
assert store.resolve_symbol("BTCUSDT") == 1
def test_import_dataframe_missing_column_raises(tmp_path):
df = _bars_df().drop(columns=["volume"])
with pytest.raises(bt.DataError, match="volume"):
_import_df(df, tmp_path)
def test_import_dataframe_integer_timestamp_raises(tmp_path):
"""Epoch integers are ambiguous (ms? ns?) — require datetimes."""
df = _bars_df()
df["timestamp"] = df["timestamp"].astype("int64")
with pytest.raises(bt.DataError, match="datetime"):
_import_df(df, tmp_path)
def test_import_dataframe_empty_raises(tmp_path):
with pytest.raises(bt.DataError, match="no data rows"):
_import_df(_bars_df(0), tmp_path)
python/tests/test_sweep.py
+4 -2
View File
@@ -31,7 +31,9 @@ def test_sweep_returns_one_result_per_combo(golden_buy_hold_dir):
universe=[1],
time_range_start=0,
time_range_end=4_000_000_000,
bar_interval={"Days": 1},
# Fixture bars are 1-second spaced; Days(1) collapses them into a
# single bar and signal_delay=1 then never fills → zero trades.
bar_interval={"Seconds": 1},
execution=bt.ExecutionConfig(
signal_delay=1,
execution_price="AtClose",
@@ -89,7 +91,7 @@ def test_sweep_golden_grid_deterministic_order(golden_buy_hold_dir):
universe=[1],
time_range_start=0,
time_range_end=4_000_000_000,
bar_interval={"Days": 1},
bar_interval={"Seconds": 1},
execution=bt.ExecutionConfig(
signal_delay=1,
execution_price="AtClose",
python/tests/test_sweep_validation.py
+109
View File
@@ -0,0 +1,109 @@
"""Sweeping a parameter the strategy never declares must fail loudly.
It used to be a silent no-op: the unknown name landed in a parameter map
nothing reads, so every combination ran the same backtest and the sweep
returned N identical results with no warning. An "optimisation" over
thousands of combos looked like it had worked.
These tests only exercise the Python-side guard, so they need no data store:
validation happens before any native call.
"""
import pytest
import manifoldbt as bt
from manifoldbt.exceptions import StrategyError
from manifoldbt.indicators import close, ema
def _declared():
"""Strategy whose 'fast' comes from mbt.param() inside an indicator."""
fast = ema(close, bt.param("fast"))
return (
bt.Strategy.create("declared")
.signal("fast", fast)
.size(bt.when(close > fast, 1.0, 0.0))
)
def _hardcoded():
"""The shape that caused the bug: the period is a literal, not a param."""
fast = ema(close, 12)
return (
bt.Strategy.create("hardcoded")
.signal("fast", fast)
.size(bt.when(close > fast, 1.0, 0.0))
)
def _cfg():
# Never reaches the engine: the guard raises before config is used.
return bt.BacktestConfig(universe={"binance": ["BTC-USDT:perp"]})
def test_sweep_lite_rejects_undeclared_param():
with pytest.raises(StrategyError) as exc:
bt.run_sweep_lite(_hardcoded(), {"fast": [10, 20, 30]}, _cfg(), None)
msg = str(exc.value)
assert "fast" in msg
# The message must say what to do, not just that it failed.
assert "mbt.param" in msg
def test_sweep_rejects_undeclared_param():
with pytest.raises(StrategyError):
bt.run_sweep(_hardcoded(), {"fast": [10, 20]}, _cfg(), None)
def test_walk_forward_rejects_undeclared_param():
wf = {
"method": "Rolling", "n_splits": 2, "train_ratio": 0.7,
"optimize_metric": "sharpe", "param_grid": {"fast": [10, 20]},
}
with pytest.raises((StrategyError, bt.LicenseError)) as exc:
bt.run_walk_forward(_hardcoded(), wf, _cfg(), None)
# Walk-forward is Pro-gated first; only assert our message when we got past it.
if isinstance(exc.value, StrategyError):
assert "fast" in str(exc.value)
def test_sweep_2d_rejects_undeclared_params():
sweep = {
"x_param": "fast", "y_param": "slow",
"x_values": [5, 10], "y_values": [20, 40], "metric": "sharpe",
}
with pytest.raises(StrategyError) as exc:
bt.run_sweep_2d(_hardcoded(), sweep, _cfg(), None)
assert "fast" in str(exc.value) and "slow" in str(exc.value)
def test_stability_rejects_undeclared_param():
stab = {"param_name": "fast", "values": [5, 10, 15], "metric": "sharpe"}
with pytest.raises(StrategyError) as exc:
bt.run_stability(_hardcoded(), stab, _cfg(), None)
assert "fast" in str(exc.value)
def test_declared_param_passes_validation():
"""A declared param must get past the guard (it then fails on the store)."""
with pytest.raises(Exception) as exc:
bt.run_sweep_lite(_declared(), {"fast": [10, 20]}, _cfg(), None)
# Whatever stops it next, it must not be our guard.
assert "not declared" not in str(exc.value)
def test_explicit_param_call_counts_as_declared():
""".param() declares a name even when no expression references it."""
strat = _hardcoded().param("fast", default=12)
with pytest.raises(Exception) as exc:
bt.run_sweep_lite(strat, {"fast": [10, 20]}, _cfg(), None)
assert "not declared" not in str(exc.value)
def test_message_lists_only_the_unknown_names():
"""A mixed grid must blame the unknown name, not the good one."""
with pytest.raises(StrategyError) as exc:
bt.run_sweep_lite(_declared(), {"fast": [10], "slow": [50]}, _cfg(), None)
msg = str(exc.value)
assert "slow" in msg
# 'fast' is declared, so it must appear as available, never as unknown.
assert "['slow']" in msg