diff --git a/docs/perf_audit_2026-07-18.md b/docs/perf_audit_2026-07-18.md new file mode 100644 index 0000000..7fbfa2c --- /dev/null +++ b/docs/perf_audit_2026-07-18.md @@ -0,0 +1,100 @@ +# Performance audit: path to 2x on the hot paths (2026-07-18) + +Audit read-only: no engine code was modified. Machine: i5-13600KF (6P+8E, 20 threads), +RTX 3090, build `maturin develop --release --features cuda` (LTO, cgu=1, debuginfo kept +via env overrides). Every number is the median of at least 3 runs; machine load was +verified with the MC-10M-CPU sentinel (6.3-6.5s clean on both sides of each suite; one +loaded suite at 8.6-9.3s was discarded and rerun). + +**Conclusion: the CPU lite sweep can reach 2x (three workstreams). The GPU sweep cannot +under the fp64 bit-parity constraint (ceiling ~1.35x). The single run tops out around +1.7x. The full `run_sweep` API still carries the unfixed twin of the JSON-getter bug: +its Python post-processing costs 3.6x the entire lite compute.** + +## Method + +Sampling profilers were unavailable (samply/ETW needs Administrator; py-spy cannot see +rayon threads). Attribution comes from the engine's own instrumentation, which proved +sufficient: 98% of wall*threads is attributed. + +- `ProfileData` (orchestrator.rs:400): per-phase us counters on every result, including + each lite sweep combo. Summed across combos and compared to wall*threads. +- `BT_PHASE6_DEBUG=1`: sub-splits output_build (metrics / capm / trade_stats / gather). +- `MBT_GPU_PROBE=1`: GPU phases (nvrtc / hoist / h2d / sim / metrics / d2h). +- Ablation: bars scaling (26k vs 100k), buy_and_hold vs ema_cross, max_parallelism 1..20, + full vs lite API. + +Harness: copy of the session scratchpad `audit_harness.py` (subcommands: sentinel, +profile_sweep, full_vs_lite, single1m, par_scale, batch3, load, access). + +## Measured baselines (synthetic BTCUSDT 1h store, `benchmarks/_sweep_common.py` config) + +| Path | Result | Phase split | +|---|---|---| +| Single run, 26k bars | 0.85ms | signal 43%, sim 22%, output 32% | +| Single run, 1M bars | 27ms | signal 9.9ms (37%), sim 8.4ms (31%), output 8.3ms (31%: metrics 4.5, capm 1.0, trade_stats ~2, gather/positions ~2.3) | +| CPU lite sweep, 10k combos x 26k bars | 171ms = 58k c/s | per combo: signal_eval 96us (28%), sim phase 239us (70%) | +| inside the sim phase | | per-bar loop ~200us (~8ns/bar = 36 cycles/bar), CAPM ~25us (measured 1ns/bar), O(days) metrics ~10us | +| CPU sweep, 500k combos | rsi 9.1s / ema 9.9s / trix 12.4s | 40-55k c/s | +| GPU sweep, 500k combos | rsi 1.67s / ema 0.98s / trix 2.35s | 5.4-10.2x vs CPU | +| GPU, 100k combos x 26k bars | 240ms = 418k c/s | sim kernel 176ms (73%), metrics kernel 34ms (14%), h2d 6ms, d2h ~2ms | +| Data load, 1M bars | cold 57ms, warm re-align ~0.5ms | store cache holds across calls | +| Lite packaging/access | ~1us per `.metrics` | fixed path is healthy | +| Full `run_sweep`, 900 combos | 80ms vs 16ms lite (5x) | then `.best()` 15us/combo, `.to_df()` 49us/combo in Python | + +Cross-checks: these numbers predict the historical 1M x 100k CPU sweep at ~107s, inside +the observed 84-150s band. Parallel scaling is 9.9x on 20 threads, near the realistic +ceiling (~11-12x) for 6P+8E: rayon granularity is not a lever. Combo enumeration is now +lazy mixed-radix (sweep.rs:98); the old ~500B/combo materialization survives only in +walk_forward/sweep_2d. Per-combo strategy recompile (dynamic periods): ~2%, cleared. + +## Ranked opportunities + +Gain = on the bench that path owns. Effort: L (<1 day), M (days), H (week+). +Parity = risk to CPU==GPU==general bit-identity (anchored to bt-expr, never a mirror). + +| # | Opportunity | Evidence | Where | Approach | Gain | Effort | Parity risk | +|---|---|---|---|---|---|---|---| +| 1 | Full-result `.metrics` = unfixed JSON twin | `.to_df()`+`.best()` = 58ms vs 16ms lite compute @900 combos | bt-python/src/result.rs:24-31, python sweep.py:75-93, dataframe.py:146-166 | Reuse `metrics_to_pydict` (backtest.rs:331); better: route `SweepResult.to_df/best` through `sweep_columns` | ~50x on full-sweep post-processing | L | None | +| 2 | CPU transpiled sweep: kill per-combo signal materialization | signal_eval = 28% of sweep CPU | orchestrator.rs:2402-2709; reference core orchestrator.rs:5766 | Do on CPU what the GPU does: hoist plan + in-loop target eval; `sim_fast_lite_core_single` is the transpile-ready reference | ~1.33x CPU sweep | H | Low; re-anchor goldens, deliberately break the old path to prove coverage | +| 3 | Hoist combo-invariant work: (a) CAPM benchmark pass, (b) ohl_nan/funding_nan copies | (a) measured 1ns/bar = ~7% of sweep; (b) 3 full O(bars) copies per combo on SL/TP sweeps, est. 25-40% tax (code-confirmed, not yet measured) | (a) orchestrator.rs:3395-3411 and 2082-2108; (b) orchestrator.rs:2821-2849 | Compute once in the sweep/batch drivers, pass by reference | 1.08x plain sweeps; ~1.2-1.4x SL/TP sweeps | L-M | None (same values, computed once) | +| 4 | Per-bar loop tightening | loop ~58% of sweep CPU at 36 cycles/bar; dependency floor ~12-18 cycles | simulate_fast_lite_single / core (orchestrator.rs:5307/5766) | Branch elimination, layout. No FP reordering, no FMA: op order is the parity anchor | 1.15-1.3x sweep | H | High if careless; bit-verify vs bt-expr | +| 5 | Single-run: parallelize independent signals + output_build components | @1M: signal 37% (fast/slow EMA sequential), output 31% (independent components) | orchestrator.rs:833-1204, 2059-2230 | rayon-join independent signal exprs (bit-safe); overlap output components; never parallelize the metric reductions | ~1.4-1.7x single run | M | Low if reductions stay sequential | +| 6 | SIMD via runtime dispatch (ships SSE2 baseline: no target-cpu anywhere) | elementwise ops are a large slice of signal eval | bt-expr evaluator kernels | `is_x86_feature_detected!` dispatch on elementwise kernels only; folds/scans stay scalar | ~1.1-1.15x sweep | M | None for elementwise; forbidden for folds | +| 7 | GPU metrics kernel fuse/overlap | 34ms of 240ms (14%) | gpu_sweep.rs:4395-4656 | Fuse into sim epilogue or stream overlap, same op order | <=1.16x GPU | M-H | Low | +| 8 | Full `run_sweep` materializes full traces per combo | 5x lite; ~600KB/combo retained | orchestrator.rs:3480-3547 | Optional trace retention; steer to lite + `sweep_columns` | up to 5x for full-sweep users | M | None | +| 9 | Coverage gates (funding on `run()` fast path, multi-asset+brackets on GPU, multi-asset kernel 33% occupancy) | gate list in bt-core | orchestrator.rs:1358-1364, 3998-4002 | Extend fast/GPU coverage case by case | 2-5x for affected configs | M-H each | Per-gate golden work | +| 10 | Data loading (no mmap on Arrow IPC, exact-key cache only, N+1 sqlite symbol_info) | cold 1M load 57ms | bt-data arrow_ipc_store.rs:327/461, metadata.rs:28 | mmap like mega_store, superset-slice cache, batched lookups | <1% on benches | L-M | None | + +Maintenance note: the per-bar loop exists in four near-identical transcriptions +(full/lite x general/fast) plus two ~130-LOC bracket macros; every loop optimization is +written and parity-tested four times. Mechanical extraction of the shared fill/equity +blocks is a safe enabler for #4. Any unification touching WHICH metrics lite computes +would hit the lite contract (cagr/calmar/ulcer != run()), which stays as-is. + +## Top 3 to reach 2x (CPU lite sweep) + +1. **#3 hoists** (CAPM + bracket/funding copies): low effort, zero parity risk, ~1.08x + plain sweeps, biggest single win on SL/TP sweeps. +2. **#2 CPU transpiled sweep**: ~1.33x, structural; the architecture is proven on GPU + and the CPU core is already the kernel's reference. With #3: ~1.45x. +3. **#4 loop tightening** (+ #6 SIMD on residual signal eval): 36 -> ~24 cycles/bar + closes the gap. Composite: **~1.9-2.2x**. + +## Theoretical ceilings + +- CPU sweep: signal+CAPM removed, loop untouched -> max 1.44x. 2x requires the loop + work; at the ~12-18 cycle dependency floor the composite ceiling is ~3x. +- GPU sweep: sim kernel 73%, already SASS-audited to the fp64 parity ceiling; everything + else free -> 1.37x max. 2x is not reachable under parity; opt-in fp32 remains the out. +- Single run 1M bars: sim loop is serial; practical ceiling ~1.7x. +- Data loading and lite packaging: <1% at bench scale, nothing to win. + +## Compatibility guarantee for every item above + +None of the proposals removes or restricts any strategy feature. #2/#4/#6 follow the +same pattern as the GPU path: strategies that qualify take the faster path, everything +else falls back to today's code, and the fallback stays golden-tested (the "break the +old path on purpose" rule applies when work moves). #3/#5/#7/#8/#10 compute identical +values in fewer places. #9 strictly widens fast-path coverage. Bit-for-bit parity is +re-verified against bt-expr for every change. diff --git a/docs/perf_plan_2026-07.md b/docs/perf_plan_2026-07.md new file mode 100644 index 0000000..3a57d87 --- /dev/null +++ b/docs/perf_plan_2026-07.md @@ -0,0 +1,292 @@ +# Perf plan: 2x on the CPU sweep (and friends) — execution order + +Source: `docs/perf_audit_2026-07-18.md` (measured baselines, ranked table, ceilings). +Scope: run_sweep_lite CPU is the 2x target. Single run gets ~1.5-1.7x. GPU 2x is a +non-goal (fp64 parity ceiling ~1.35x, documented). Lite contract stays as-is. + +Ground rules for every phase: +- Bit-for-bit parity CPU==GPU==general, verified against bt-expr goldens, never a mirror. +- When work moves off a path, break the old path on purpose to prove tests still bite. +- Portability: no target-cpu in shipped config; runtime feature detection only. +- Every perf claim: median of >=3 runs, MC-10M-CPU sentinel ~6s clean on both sides. +- One branch per workstream, `perf:` commits, no cross-stream stacking unless noted. + +## End-to-end result so far (2026-07-18) + +Measured A-B-A at the build level (HEAD, then main's engine sources, then HEAD +again) so drift between builds is visible rather than assumed. "after" is the +mean of the two HEAD runs. + +| | before (main) | after | gain | +|---|---|---|---| +| CPU sweep (10k combos x 26k bars) | 51,332 c/s | ~64,567 c/s | **+26%** | +| SL/TP sweep | 27,662 c/s | ~33,885 c/s | **+22%** | +| `SweepResult.to_df()` | 63.2 us/combo | 12.0 us | **5.3x** | +| `SweepResult.best()` | 25.4 us/combo | 2.4 us | **10.5x** | +| `.metrics` per result | 27.9 us | 3.0 us | **9.4x** | +| GPU sweep (untouched, drift canary) | 164,067 c/s | ~167,436 c/s | +2% | + +The GPU line is the control: nothing in this work touches that path and it does +not move, so the harness is not systematically biased. + +**Caveats, so these are not over-read.** The two identical HEAD builds differed +by 6.7% (62,474 vs 66,661 c/s) and the baseline was measured once, so read the +sweep numbers as +26% with roughly +/-7%, solid in direction and magnitude but +not to the point. The post-processing ratios are far above the noise (the two +HEAD runs agree to 0.3%) and are reliable. + +**One unattributed slice.** The CPU sweep gained ~26% while the paired A/B +credits the CAPM hoist with 11.9%. The likely source of the remaining ~14% is +the daily-equity unification, which was intended as a pure refactor: the merged +form does one division and one comparison per bar where the old form did +`last() == Some(ts/nanos*nanos)` (division AND multiplication) and then possibly +a second `last()/nanos != day` test. Redundant per-bar arithmetic was removed +without aiming for it. This attribution is PLAUSIBLE BUT UNVERIFIED; it needs +its own paired A/B before being claimed. + +## Phase 0 — lock the ruler (DONE 2026-07-18) + +- [x] Harness checked in at `benchmarks/audit_harness.py` (subcommands: sentinel, + profile_sweep, bracket_probe, full_vs_lite, single1m, par_scale, sweep_cpu/gpu, access). +- [x] 2026-07-18 baselines are the reference row (see the audit doc). +- [x] Bracket-sweep probe run to size finding #3b. + +**Measured (10k combos, ema grid, sentinel-clean):** + +| bars | plain | SL/TP bracket | ratio | sim us/combo (plain -> bracket) | +|---|---|---|---|---| +| 26k | 54,800 c/s | 26,440 c/s | 2.07x slower | 242 -> 547 | +| 100k | 11,863 c/s | 3,737 c/s | 3.17x slower | 1,082 -> 3,972 | + +**Re-sizing that forces (finding #3b):** the bracket penalty is large but scales +SUPER-linearly with bars (2.07x -> 3.17x), so it is dominated by per-bar bracket +check work, NOT by the `ohl_nan` copy (which is linear in bars). The copy is a +minority slice of the +305us/combo (26k). The audit's "25-40% tax, ~1.2-1.4x from +hoisting" was too optimistic: expect ~1.1x on bracket sweeps from the copy hoist +alone. The real bracket win lives in Phase 3 (loop/macro work), not Phase 1. +Consequence: **the ohl_nan/funding_nan hoist is demoted out of Phase 1** and folded +into the Phase 3 bracket work; Phase 1b keeps only the CAPM hoist (universal ~7%). + +## Phase 1 — quick wins, zero parity risk (~2-3 days total) + +Branch `perf/full-metrics-pydict` (DONE 2026-07-18) +- [x] `PyBacktestResult.metrics` + `profile` (result.rs): build via the existing + `metrics_to_pydict` / `profile_to_pydict` (backtest.rs), no JSON round-trip. +- [x] **Nested `trade_stats` hand-mirrored too** (`trade_stats_to_pydict`). This was + the missing half: `metrics_to_pydict` still round-tripped the nested + TradeStatistics through JSON. Harmless for LITE sweeps (trade_stats = None) but + the full `run_sweep` populates it on EVERY combo, so it was the dominant residual + cost. Fixing only the outer getter gave 1.7x; adding this gave 4-6x. +- [x] Serde-parity pinned: 5 unit tests green, incl. a new + `metrics_dict_matches_serde_with_trade_stats_signal_quality` for the + nested-nested `signal_quality`. Drift guard verified to BITE (sabotaging one + field fails both trade_stats tests with "drifted from serde"). +- [x] End-to-end oracle: full `run_sweep` vs dedicated `mbt.run()` per combo (both + full path, so exact) -- 9 combos x (189 scalar + 162 trade_stats) fields, + **0 mismatches**. +- [ ] (deferred, optional) Route `to_df()`/`best()` through the `sweep_columns` + buffer path. Not needed to close Phase 1: the getter fix already removed the + JSON round-trip; what remains is inherent Python dict/DataFrame building. + +**Measured (900 combos):** + +| op | before | after | gain | +|---|---|---|---| +| `SweepResult.to_df()` | 49 us/combo | 11.7 us/combo | **4.2x** | +| `SweepResult.best()` | 15 us/combo | 2.4 us/combo | **6.3x** | +| `.metrics` per result | ~18 us | 2.8 us | **~6.4x** | + +Note: the audit's "~50x" projection was wrong -- it carried over the scale of the +ORIGINAL 1M-sweep lite bug rather than this path's measured baseline. Real: 4-6x. + +Python suite: 63 passed, 2 failed. Both failures (`test_golden_buy_and_hold`, +`test_sweep::test_sweep_returns_one_result_per_combo`) are **pre-existing** -- +proven by stashing the change, rebuilding, and reproducing the identical +`2 failed, 63 passed`. Both share one root cause: the golden fixture yields a flat/ +truncated equity curve (`[1000.0, 1000.0]`), which makes total_return 0.0 for every +combo. Tracked separately (see the pending `fix/python-test-suite` branch). + +Branch `perf/hoist-capm` (Phase 1b, DONE 2026-07-18 -- perf number provisional) +- [x] CAPM benchmark returns hoisted into `run_sweep_lite` and `run_batch_lite` + via `hoist_capm_benchmark()`; `run_lite_on_aligned` takes + `hoisted_benchmark: Option<&[f64]>` and falls back to computing per run when + None. walk_forward passes None (unchanged behaviour). +- **Guard (verified in code):** hoisting is bit-identical ONLY when the run's + `sim_bars` is `&aligned.symbol_bars`, i.e. `coarse_bars` is None + (orchestrator.rs:2300 `if signal_ns > native_ns`). The helper re-detects + `native_ns` on the post-pre-resample bars and returns None under hybrid + resampling, where closes come from coarse bars the driver has no handle on. + A blind driver-side hoist would have silently changed alpha/beta there. +- [x] **Proven equivalent, not merely untested.** A temporary probe recomputed the + per-run benchmark alongside the hoisted one and asserted `to_bits()` + equality: green. Then the probe was made to `panic!` on entry to prove the + hoisted branch is actually REACHED -- it is, by 8 tests, and exactly the + right ones: `lite_matches_full_with_{stop_loss,take_profit,trailing_stop, + gap_through_stop,stop_loss_short,full_bracket_and_costs}_at_native_resolution`, + `lite_and_full_agree_on_max_drawdown_sign`, and + `per_strategy_orders_apply_and_batch_is_heterogeneous` (batch_lite). + Probe removed; `cargo test -p bt-core` = 91 passed, 0 failed. +- Note: `golden_buy_and_hold` fails under `--release` but passes in debug with + `BT_UNLOCKED=1`. That is the known licensing artifact (the dev bypass is + `#[cfg(debug_assertions)]`, so release runs locked and hits the Pro output + floor), not this change. It also does not exercise this change at all: it runs + `run()`, i.e. the full kernel, whose CAPM block was left untouched. + +**Perf: MEASURED, interleaved A/B (10k combos x 26k bars, ema_cross).** + +A plain before/after was NOT usable here: the Phase 0 baseline was taken while +the paper dashboard was loading the box, so comparing it against a later quiet +run would have credited the hoist with someone else's CPU. Instead the hoist was +put behind a temporary env toggle so ONE binary could run both arms, interleaved +A,B,A,B, in one environment. Paired deltas cancel any drift. Toggle removed after. + +| | hoist OFF | hoist ON | delta | +|---|---|---|---| +| sim / combo | 246.2 us | 206.1 us | **+16.5%** | +| wall | 179.9 ms | 159.0 ms | **+11.9%** | +| throughput | 55,593 c/s | 62,883 c/s | | + +Four paired rounds, tightly clustered (sim +15.9/+16.6/+17.5/+16.3%), which is +how we know the pairing worked. **The plan's ~1.08x estimate was too +conservative: the real figure is 1.12x on wall.** The audit had priced CAPM at +~1ns/bar (~26us/combo); the measured removal is ~40us/combo, because the pass +also does a resample-to-daily, a step_returns and their allocations per combo, +not just a linear scan. + +### The MC-10M sentinel is NOT a load detector (correction) + +Recorded because the old heuristic ("~6s clean vs ~16s loaded") is misleading +and cost real time this session. With every competing process killed and the +sweeps posting their best numbers of the day (66,458 c/s), the sentinel still +read ~16s, and inside one 3-run batch it printed `[16.14, 16.36, 7.47]`. It is +bimodal for reasons unrelated to CPU contention (10M paths: allocation / +first-touch / page-cache state), so it flags "loaded" on a quiet machine. + +Use instead: interleaved A/B with paired deltas, which is robust to drift by +construction and needs no external notion of "clean". + +Gate to close Phase 1: parity suite green (Rust goldens + Python mirrors), benches +re-run per protocol, numbers recorded in the audit doc. + +## Phase 2 — medium effort, contained risk (~1-2 weeks) + +Branch `perf/single-run-parallel` -- **CANCELLED 2026-07-18, premise was false.** + +- [x] ~~rayon-join independent signal expressions~~ **ALREADY IMPLEMENTED.** + orchestrator.rs:889 evaluates each dependency level with `level.par_iter()` + whenever `level.len() >= 2`. The audit's exploration agent reported "fast/slow + EMA computed sequentially"; that was a misread, and it propagated into this + plan. Measured at 1M bars, N independent EMAs in one strategy: + + | signals | signal_eval | us/signal | if it were sequential | + |---|---|---|---| + | 1 | 6,843 us | 6,843 | - | + | 2 | 10,071 us | 5,036 | 13,686 | + | 4 | 10,580 us | 2,645 | 27,372 | + | 16 | 17,110 us | 1,069 | 109,488 | + + 16 signals cost 2.5x one signal, not 16x. The parallelism is real and + working. Nothing to win here. (Each individual EMA is a sequential scan, so + the residual per-signal cost is irreducible without changing the recurrence.) + +- [x] ~~Overlap output_build components~~ **NOT WORTH IT.** `metrics` borrows + `trace_equities` while the gather MOVES it, and metrics must run on + full-resolution equity (max_drawdown depends on it, orchestrator.rs comment + at the metrics call). Overlapping them needs an 8MB clone at 1M bars, which + eats most of the ~2.8ms theoretical saving. Best case was ~7% of a 27ms run. + +**Consequence for the ceiling:** the audit put the single run at ~1.7x reachable. +That number assumed a sequential signal phase that does not exist. With signals +already parallel and output_build ownership-bound, the single-run path is close to +its practical ceiling; expect well under 1.2x, and it is NOT where the 2x lives. +The 2x target remains the CPU sweep, where per-combo signal work is genuine CPU +load (the sweep saturates all threads across combos, so intra-combo signal +parallelism is degenerate there and the transpiled-sweep item still stands). + +Branch `perf/simd-dispatch` (target: ~1.1-1.15x sweep, more on signal-heavy runs) +- [ ] `is_x86_feature_detected!` runtime dispatch on bt-expr elementwise kernels only + (compare, IfElse, arithmetic). Folds and scans (EWM, rolling, sums) stay scalar. +- [ ] Parity: elementwise same-op-per-lane is bit-identical; add a test asserting + dispatch on/off equality on goldens. Baseline fallback keeps portability. + +Optional branch `perf/full-sweep-traces` +- [ ] Optional trace retention on full run_sweep (orchestrator.rs:3480-3547), or at + minimum docs steering sweep users to lite + sweep_columns. + +## Phase 3 — structural, the 2x closers (~3-5 weeks, sequential) + +Branch `refactor/loop-extraction` FIRST (enabler, no behavior change) +- [ ] Mechanically extract the shared fill/equity/daily blocks from the four loop + transcriptions (full/lite x general/fast) and the two bracket macros. + Pure extraction: does not change WHICH metrics lite computes (lite contract). +- [ ] Golden + parity suites must be bit-identical before/after. + +**De-risked 2026-07-18: the 1.33x premise holds, and is probably conservative.** + +Measured with the paired-A/B discipline (each variant in its own process, +interleaved, only paired deltas trusted), 2500 combos x 26k bars: + +| paired delta | value | rounds | reading | +|---|---|---|---| +| +1 indicator (fixed span, declared, unreferenced) | **-0.5 us/combo** | -0.8, -0.3, +11.1, -0.7 | an indicator costs ~nothing per combo | +| +1 elementwise pass (compare+when+add) | **+180 us/combo** | 178, 157, 183, 241 | one pass ~= 1.34x the WHOLE signal phase | + +base: signal_eval 134.3 us/combo, simulation 199.7 us/combo. + +The ~0 indicator delta is NOT pruning: the compiler compiles every entry of +`def.signals` with no dead-signal elimination +(crates/bt-strategy/src/compiler.rs:74-82). It is AMORTIZATION. A fixed-span EMA +is computed once and every combo hits IndicatorCache; a swept EMA over a 50x50 +grid has 50 distinct spans shared by 50 combos each. So indicator math is +effectively free per combo in a 2D sweep, and nearly all of signal_eval is +per-combo overhead (env build, param binding, the elementwise chain, output +allocation) -- precisely what fusing the target into the per-bar loop removes. + +Ceiling: the naive arithmetic says 1.68x, but do not quote that. It leans on a +slightly negative indicator delta (so "101% removable", an artifact), the +elementwise delta has 46% spread, and a real transpiled sweep still pays +per-combo param binding and indicator lookup. **Defensible: >= 1.33x, plausibly +1.4-1.7x.** Enough to justify the work; re-measure against the real +implementation rather than trusting this number. + +Branch `perf/cpu-transpiled-sweep` (target: ~1.33x, stacked on Phase 1 => ~1.45x) +- [ ] Reuse the GPU hoist plan (build_hoist_plan) on CPU: fill hoisted indicator series + once per sweep, compute the target in-loop via `sim_fast_lite_core_single` + (orchestrator.rs:5766), which is already the CUDA kernel's CPU reference. +- [ ] Same eligibility gates as the GPU transpiler; anything else falls back to the + current vectorized-signal path, unchanged. +- [ ] Deliberately break the old vectorized path (temporarily) to prove the fallback + is still covered by tests. Parity anchored to bt-expr. + +Branch `perf/lite-loop-tightening` (target: 36 -> ~24 cycles/bar, ~1.15-1.3x sweep) +- [ ] Branch elimination in the single-asset core: hoist the sizing_mode match, the + no-rebalance gates, null-path dispatch. Control flow and layout ONLY. +- [ ] Forbidden: FP reordering, FMA, fast-math of any kind. Op order is the parity + anchor. Bit-verify against bt-expr after every commit. + +Gate to close Phase 3: composite CPU sweep >= 1.9x vs Phase 0 baseline on the 3-strategy +500k bench (median of 3, sentinel-clean), GPU numbers unchanged, parity green. + +## Phase 4 — coverage (optional, per-gate decisions) + +- [ ] Funding on run()'s CPU fast path (orchestrator.rs:1358-1364): aligns run() with + lite/GPU, big win for perp single runs. +- [ ] GPU metrics kernel fuse/overlap (gpu_sweep.rs:4395-4656): <=1.16x GPU. +- [ ] Multi-asset GPU: brackets support, occupancy (33% today). +Each: own golden work, own decision. None blocks the 2x goal. + +## Non-goals (explicit) + +- GPU 2x under fp64 bit-parity: not reachable (sim kernel 73%, SASS-audited ceiling). + fp32 stays the documented opt-in for speed-over-bits users. +- Changing the lite contract (cagr/calmar/ulcer != run()). +- Machine-specific build flags in shipped wheels. +- Data-loading work (cold 57ms @1M bars, <1% at bench scale): revisit only if a + many-fresh-process workflow becomes a product path. + +## Compatibility invariant + +No strategy loses support at any phase. Fast paths widen or stay put; everything not +eligible falls back to today's code, which keeps its own test coverage (proven by the +deliberate-break rule). Example strategies remain testable and sweepable throughout; +`examples/` runs green at every phase gate. diff --git a/examples/00_template.py b/examples/00_template.py index d786280..cc48a23 100644 --- a/examples/00_template.py +++ b/examples/00_template.py @@ -72,4 +72,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {perf_counter() - t0:.2f}s") - mbt.plot.tearsheet(result, show=True) + mbt.plot.tearsheet(result) diff --git a/examples/01_trend_following.py b/examples/01_trend_following.py index d72ba8f..5a87d9c 100644 --- a/examples/01_trend_following.py +++ b/examples/01_trend_following.py @@ -75,4 +75,4 @@ if __name__ == "__main__": print(f"\nElapsed: {elapsed:.3f}s") # Plot - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) diff --git a/examples/02_mean_reversion.py b/examples/02_mean_reversion.py index 88f6c2b..602ed29 100644 --- a/examples/02_mean_reversion.py +++ b/examples/02_mean_reversion.py @@ -63,4 +63,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) diff --git a/examples/03_multi_asset_momentum.py b/examples/03_multi_asset_momentum.py index 7fe0be7..ba49b1f 100644 --- a/examples/03_multi_asset_momentum.py +++ b/examples/03_multi_asset_momentum.py @@ -70,4 +70,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) diff --git a/examples/04_linear_regression.py b/examples/04_linear_regression.py index 7dcd099..34a4292 100644 --- a/examples/04_linear_regression.py +++ b/examples/04_linear_regression.py @@ -101,4 +101,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) diff --git a/examples/05_stat_arb.py b/examples/05_stat_arb.py index fcd7c05..2f3f381 100644 --- a/examples/05_stat_arb.py +++ b/examples/05_stat_arb.py @@ -71,4 +71,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) diff --git a/examples/06_full_visualization.py b/examples/06_full_visualization.py index ba75b81..c533739 100644 --- a/examples/06_full_visualization.py +++ b/examples/06_full_visualization.py @@ -92,7 +92,7 @@ if __name__ == "__main__": ) # -- 3. Summary 3-panel --------------------------------------------------- - mbt.plot.summary(result, show=True) + mbt.plot.summary(result) # -- 4. Candlestick chart (first symbol in universe) -------------------- mbt.plot.chart( @@ -101,18 +101,17 @@ if __name__ == "__main__": smas=[50], n_bars=120, interactive=False, - show=True, ) # -- 5. Individual charts ------------------------------------------------- - mbt.plot.equity(result, show=True) - mbt.plot.drawdown(result, show=True) - mbt.plot.monthly_returns(result, show=True) - mbt.plot.annual_returns(result, show=True) - mbt.plot.returns_histogram(result, show=True) - mbt.plot.var_chart(result, show=True) - mbt.plot.rolling_sharpe(result, show=True) - mbt.plot.rolling_volatility(result, show=True) + mbt.plot.equity(result) + mbt.plot.drawdown(result) + mbt.plot.monthly_returns(result) + mbt.plot.annual_returns(result) + mbt.plot.returns_histogram(result) + mbt.plot.var_chart(result) + mbt.plot.rolling_sharpe(result) + mbt.plot.rolling_volatility(result) # -- 6. Sweep heatmap 2D ------------------------------------------------- # Sweep over RSI period and oversold threshold @@ -160,7 +159,7 @@ if __name__ == "__main__": "metric_grid": metric_grid, } print(f"Sweep done in {time.perf_counter() - t0:.1f}s") - mbt.plot.heatmap_2d(sweep_result, show=True) + mbt.plot.heatmap_2d(sweep_result) # -- 7. Walk-forward validation ------------------------------------------- print("\nRunning walk-forward (manual folds)...") @@ -201,11 +200,11 @@ if __name__ == "__main__": "folds": wf_folds, } print(f"Walk-forward done in {time.perf_counter() - t0:.1f}s") - mbt.plot.walk_forward(wf_result, show=True) + mbt.plot.walk_forward(wf_result) # -- 8. Monte Carlo ------------------------------------------------------- print("\nRunning Monte Carlo (1000 paths)...") - mbt.plot.monte_carlo(result, n_simulations=1000, seed=42, show=True) + mbt.plot.monte_carlo(result, n_simulations=1000, seed=42) # -- 9. Parameter stability ----------------------------------------------- print("\nRunning stability analysis (RSI period)...") @@ -241,7 +240,7 @@ if __name__ == "__main__": "stability_score": 1.0 - (std_m / abs(mean_m)) if mean_m != 0 else 0.0, } print(f"Stability done in {time.perf_counter() - t0:.1f}s") - mbt.plot.stability(stab_result, show=True) + mbt.plot.stability(stab_result) # -- 10. Research report (composite) -------------------------------------- print("\nGenerating research report...") diff --git a/examples/07_walk_forward.py b/examples/07_walk_forward.py index 3369ef0..5b788d1 100644 --- a/examples/07_walk_forward.py +++ b/examples/07_walk_forward.py @@ -93,4 +93,4 @@ if __name__ == "__main__": print(f"\n{len(folds)} folds in {elapsed:.2f}s") if folds: - mbt.plot.walk_forward({"optimize_metric": metric, "folds": folds}, show=True) + mbt.plot.walk_forward({"optimize_metric": metric, "folds": folds}) diff --git a/examples/08_sweep_2d_heatmap.py b/examples/08_sweep_2d_heatmap.py index 3385f5f..ed41a8e 100644 --- a/examples/08_sweep_2d_heatmap.py +++ b/examples/08_sweep_2d_heatmap.py @@ -87,4 +87,4 @@ if __name__ == "__main__": "y_values": slow_values, "metric": "t-stat(alpha)", "metric_grid": metric_grid, - }, show=True) + }) diff --git a/examples/09_surface_3d.py b/examples/09_surface_3d.py index f52254e..4a73c29 100644 --- a/examples/09_surface_3d.py +++ b/examples/09_surface_3d.py @@ -84,4 +84,4 @@ if __name__ == "__main__": "y_values": slow_values, "metric": "t-stat(alpha)", "metric_grid": metric_grid, - }, show=True) + }) diff --git a/examples/10_monte_carlo.py b/examples/10_monte_carlo.py index 3da0d8c..f6566c4 100644 --- a/examples/10_monte_carlo.py +++ b/examples/10_monte_carlo.py @@ -66,4 +66,4 @@ if __name__ == "__main__": print(f"Elapsed: {time.perf_counter() - t0:.3f}s\n") # 2. Monte Carlo fan chart - mbt.plot.monte_carlo(result, n_simulations=10000, seed=42, show=True) + mbt.plot.monte_carlo(result, n_simulations=10000, seed=42) diff --git a/examples/11_portfolio.py b/examples/11_portfolio.py index 120faa7..26f5103 100644 --- a/examples/11_portfolio.py +++ b/examples/11_portfolio.py @@ -72,4 +72,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.tearsheet(result, show=True) + mbt.plot.tearsheet(result) diff --git a/examples/13_stochastic_simulation.py b/examples/13_stochastic_simulation.py index 157b38d..68ca03c 100644 --- a/examples/13_stochastic_simulation.py +++ b/examples/13_stochastic_simulation.py @@ -141,5 +141,4 @@ if __name__ == "__main__": mbt.plot.stochastic_paths( result, title=f"Mean-reverting model (S0=80, target=100, {N_PLOT:,} paths)", - show=True, ) diff --git a/examples/14_multi_timeframe.py b/examples/14_multi_timeframe.py index e6668a4..3036143 100644 --- a/examples/14_multi_timeframe.py +++ b/examples/14_multi_timeframe.py @@ -83,4 +83,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - mbt.plot.equity(result, show=True) + mbt.plot.equity(result) diff --git a/examples/15_cross_exchange.py b/examples/15_cross_exchange.py index 8db4e6d..58310fb 100644 --- a/examples/15_cross_exchange.py +++ b/examples/15_cross_exchange.py @@ -105,4 +105,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - result.plot_equity(show=True) + result.plot_equity() diff --git a/examples/16_hashrate_exogene.py b/examples/16_hashrate_exogene.py index 1a21820..231c614 100644 --- a/examples/16_hashrate_exogene.py +++ b/examples/16_hashrate_exogene.py @@ -211,4 +211,4 @@ if __name__ == "__main__": print(result.summary()) print(f"\nElapsed: {elapsed:.3f}s") - result.plot_equity(show=True) + result.plot_equity() diff --git a/pyproject.toml b/pyproject.toml index 98a6803..5b8c313 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "manifoldbt" -version = "0.13.2" +version = "0.14.0" description = "Rust-powered backtesting engine for quantitative research" requires-python = ">=3.9" license = { file = "LICENSE" } diff --git a/python/manifoldbt/__init__.py b/python/manifoldbt/__init__.py index 250a4de..44729f4 100644 --- a/python/manifoldbt/__init__.py +++ b/python/manifoldbt/__init__.py @@ -29,6 +29,7 @@ from manifoldbt._native import ( run_portfolio as _run_portfolio_native, py_ingest as _ingest_native, py_import_csv as _import_csv_native, + py_import_dataframe as _import_dataframe_native, ) from manifoldbt._serde import scalar_value_to_json from manifoldbt.config import ( @@ -183,6 +184,32 @@ def _require_pro_over_combos(n_combos: int, what: str) -> None: ) +def _validate_swept_params(strategy: "Strategy", names, what: str) -> None: + """Reject swept parameter names the strategy never declares. + + Sweeping a name the strategy does not use is a silent no-op: the value is + merged into a parameter map nothing reads, so every combo runs the same + backtest and the sweep returns N identical results with no warning. That + is worse than an error, because an "optimisation" over thousands of combos + looks like it worked and its best result is meaningless. + + A parameter counts as declared whether it came from ``mbt.param()`` inside + an expression or from an explicit ``.param()`` call: ``to_json_dict()`` + merges both into ``parameters`` (and is memoised, so this costs nothing). + """ + declared = set(strategy.to_json_dict().get("parameters") or {}) + unknown = [n for n in names if n not in declared] + if not unknown: + return + known = ", ".join(sorted(declared)) if declared else "none" + raise StrategyError( + f"{what}: parameter(s) {unknown} are not declared by strategy " + f"'{strategy.name}' (declared: {known}). Sweeping them would run the " + f"same backtest for every combination. Use mbt.param(\"name\") where " + f"the value is consumed, e.g. ema(close, mbt.param(\"fast\"))." + ) + + def _classify_error(exc: Exception) -> Exception: """Wrap a Rust ValueError/RuntimeError in a more specific exception.""" msg = str(exc) @@ -682,6 +709,128 @@ def import_csv( ) +_BARS_REQUIRED_COLUMNS = ("timestamp", "open", "high", "low", "close", "volume") + + +def _df_to_bars_batch(data): + """Normalise a pandas/polars DataFrame (or dict) to a pyarrow RecordBatch. + + Output contract (what the native import expects): columns + ``timestamp`` (timestamp[ns, UTC]), ``open/high/low/close/volume`` (f64). + Naive timestamps are assumed UTC. A pandas DatetimeIndex is promoted to + the ``timestamp`` column when the column is absent. + """ + import pyarrow as pa + + # --- to Arrow Table (same dispatch as register_exo) --- + if hasattr(data, "to_arrow"): + # Polars DataFrame + table = data.to_arrow() + elif hasattr(data, "columns"): + # Pandas DataFrame + import pandas as pd + if "timestamp" not in data.columns and isinstance(data.index, pd.DatetimeIndex): + data = data.reset_index(names="timestamp") + table = pa.Table.from_pandas(data, preserve_index=False) + elif isinstance(data, dict): + table = pa.table(data) + else: + raise TypeError( + f"Unsupported data type: {type(data)}. Use a pandas/polars DataFrame or dict." + ) + + missing = [c for c in _BARS_REQUIRED_COLUMNS if c not in table.column_names] + if missing: + raise DataError( + f"DataFrame is missing required column(s): {', '.join(missing)}. " + f"Expected: {', '.join(_BARS_REQUIRED_COLUMNS)}" + ) + table = table.select(list(_BARS_REQUIRED_COLUMNS)) + + # --- timestamp → timestamp[ns, UTC] --- + ts_type = table.schema.field("timestamp").type + if not pa.types.is_timestamp(ts_type): + raise DataError( + f"'timestamp' column must be a datetime type, got {ts_type}. " + "For epoch integers, convert first: pd.to_datetime(ts, unit='ms', utc=True)" + ) + target_ts = pa.timestamp("ns", tz="UTC") + if ts_type != target_ts: + table = table.set_column( + 0, pa.field("timestamp", target_ts), table.column(0).cast(target_ts) + ) + + # --- value columns → float64 --- + for i, name in enumerate(_BARS_REQUIRED_COLUMNS[1:], start=1): + if table.schema.field(i).type != pa.float64(): + table = table.set_column( + i, pa.field(name, pa.float64()), table.column(i).cast(pa.float64()) + ) + + if table.num_rows == 0: + raise DataError("DataFrame contains no data rows") + + # Single contiguous batch for the zero-copy FFI crossing. + return table.combine_chunks().to_batches()[0] + + +def import_dataframe( + data, + symbol: str, + symbol_id: int, + *, + interval: str = "1m", + data_root: str = "data", + metadata_db: str = "metadata/metadata.sqlite", + exchange: str = "DATAFRAME", + asset_class: str = "crypto_spot", +) -> DataStore: + """Import bars from an in-memory DataFrame into the Arrow IPC store. Free on all tiers. + + The in-memory twin of :func:`import_csv`: edit your data as a DataFrame, + then import it directly — no intermediate CSV. Returns a :class:`DataStore` + ready for :func:`run` (same store, metadata and versioning as ``bt.ingest``). + + Accepts a pandas DataFrame, polars DataFrame, or dict of columns with + ``timestamp`` (datetime; naive values are assumed UTC), ``open``, ``high``, + ``low``, ``close``, ``volume``. A pandas DatetimeIndex is used as + ``timestamp`` if that column is absent. Rows must be sorted by timestamp. + + Example:: + + df = pd.read_parquet("EURUSD_1m.parquet") + df["close"] = df["close"].clip(upper=1.5) # edit in memory + store = bt.import_dataframe(df, symbol="EURUSD", symbol_id=1, + interval="1m", asset_class="forex") + result = bt.run(strategy, config, store) + + Args: + data: pandas/polars DataFrame or dict of columns. + symbol: Ticker name (e.g. ``"EURUSD"``, ``"BTCUSDT"``). + symbol_id: Unique integer ID for this symbol in the store. + interval: Bar interval of the rows (``"1m"``, ``"5m"``, ``"1h"``, ``"1d"``, ...). + data_root: Store directory (default ``"data"``). + metadata_db: Metadata SQLite path. + exchange: Exchange label for metadata (default ``"DATAFRAME"``). + asset_class: ``crypto_spot``, ``crypto_perp``, ``equity``, ``future``, + ``option``, ``forex``, or ``index``. + """ + batch = _df_to_bars_batch(data) + try: + return _import_dataframe_native( + batch, + symbol=symbol, + symbol_id=symbol_id, + interval=interval, + data_root=data_root, + metadata_db=metadata_db, + exchange=exchange, + asset_class=asset_class, + ) + except (ValueError, RuntimeError) as exc: + raise _classify_error(exc) from exc + + def _ingest_single( *, provider, symbol, symbol_id, start, end, interval, dataset, data_root, metadata_db, exchange, asset_class, progress, @@ -759,6 +908,7 @@ def run_sweep( A :class:`SweepResult` with ``.to_df()``, ``.best()``, ``.plot_metric()``. """ _require_pro_over_combos(_grid_combos(param_grid), "Parameter sweep") + _validate_swept_params(strategy, param_grid.keys(), "Parameter sweep") try: config = _cap_output_resolution(config) store = _resolve_store(config, store) @@ -930,6 +1080,7 @@ def run_sweep_lite( One :class:`BatchResultLite` per combo (Cartesian product order). """ _require_pro_over_combos(_grid_combos(param_grid), "Parameter sweep") + _validate_swept_params(strategy, param_grid.keys(), "Parameter sweep") _require_pro_for_gpu(device, "GPU sweep") try: config = _cap_output_resolution(config) @@ -939,7 +1090,11 @@ def run_sweep_lite( name: [scalar_value_to_json(v) for v in values] for name, values in param_grid.items() }) - return _run_sweep_lite_native( + # Wrapped in a list subclass: echoing a sweep in a notebook cell + # printed one BatchResultLite line per combo. Indexing, iteration and + # len() are unchanged. + from manifoldbt._reprs import wrap_sweep_lite + return wrap_sweep_lite(_run_sweep_lite_native( strategy.to_json(), grid_json, cfg_json, @@ -947,7 +1102,7 @@ def run_sweep_lite( max_parallelism, device, precision, - ) + )) except (ValueError, RuntimeError) as exc: raise _classify_error(exc) from exc @@ -1029,9 +1184,15 @@ def run_walk_forward( # in `py_run_walk_forward` (check_feature("walk_forward")), so this cannot be # bypassed by calling the native function directly. _require_pro("Walk-forward optimization") + _validate_swept_params(strategy, (wf_config.get("param_grid") or {}).keys(), + "Walk-forward") config = _prepare_config(config, strategy, store) wf_json = json.dumps(_convert_param_grid_in_config(wf_config)) - return _run_walk_forward_native(strategy.to_json(), wf_json, config.to_json(), store) + raw = _run_walk_forward_native(strategy.to_json(), wf_json, config.to_json(), store) + # Wrapped in a dict subclass: the raw dict holds a full equity curve per + # fold, so echoing it in a cell printed tens of thousands of floats. + from manifoldbt._reprs import wrap_walk_forward + return wrap_walk_forward(raw) def run_sweep_2d( @@ -1061,6 +1222,10 @@ def run_sweep_2d( len(sweep_config.get("x_values", [])) * len(sweep_config.get("y_values", [])), "2D parameter sweep", ) + _validate_swept_params( + strategy, + [n for n in (sweep_config.get("x_param"), sweep_config.get("y_param")) if n], + "2D parameter sweep") config = _prepare_config(config, strategy, store) sweep_json = json.dumps(_convert_scalar_values_in_sweep(sweep_config)) return _run_sweep_2d_native(strategy.to_json(), sweep_json, config.to_json(), store) @@ -1088,6 +1253,10 @@ def run_stability( Dict with ``stability_score``, ``metric_values``, ``mean_metric``, ``std_metric``. """ _require_pro_over_combos(len(stability_config.get("values", [])), "Parameter stability analysis") + _validate_swept_params( + strategy, + [n for n in (stability_config.get("param_name"),) if n], + "Parameter stability analysis") config = _prepare_config(config, strategy, store) stab_json = json.dumps(_convert_scalar_values_in_stability(stability_config)) return _run_stability_native(strategy.to_json(), stab_json, config.to_json(), store) @@ -1404,6 +1573,7 @@ __all__ = [ # Data ingestion "ingest", "import_csv", + "import_dataframe", # Run functions "run", "run_sweep", diff --git a/python/manifoldbt/_reprs.py b/python/manifoldbt/_reprs.py new file mode 100644 index 0000000..0d7cf7c --- /dev/null +++ b/python/manifoldbt/_reprs.py @@ -0,0 +1,107 @@ +"""Compact reprs for the big containers returned to notebooks. + +A sweep returns one object per combo and a walk-forward carries a full +equity curve per fold, so echoing either in a Jupyter cell used to print +thousands of lines. These wrappers subclass ``list``/``dict`` so every +existing access keeps working (indexing, iteration, ``.keys()``, JSON +round-trips); only the repr changes. +""" +from __future__ import annotations + +from typing import Any, Dict, List + +_MAX_SCAN = 100_000 # cap the repr's own cost on million-combo sweeps + + +def _fmt(v: float) -> str: + """Compact number: 3 significant-ish digits, thousands as k.""" + if v is None: + return "?" + a = abs(v) + if a >= 1_000_000: + return f"{v / 1_000_000:.2f}M" + if a >= 1_000: + return f"{v / 1_000:.2f}k" + if a >= 1: + return f"{v:.2f}" + return f"{v:.4g}" + + +def _span(values) -> str: + vals = [v for v in values if v is not None] + if not vals: + return "n/a" + lo, hi = min(vals), max(vals) + return _fmt(lo) if lo == hi else f"{_fmt(lo)}..{_fmt(hi)}" + + +class SweepLiteResults(list): + """``run_sweep_lite`` output: a list, with a one-line repr. + + Printing 400 combos used to emit 400 lines of ``BatchResultLite(...)``. + """ + + def __repr__(self) -> str: + n = len(self) + if n == 0: + return "SweepLiteResults(empty)" + head = self[:_MAX_SCAN] + eq = _span([getattr(r, "final_equity", None) for r in head]) + sharpes = [] + for r in head: + m = getattr(r, "metrics", None) + if isinstance(m, dict): + sharpes.append(m.get("sharpe")) + name = getattr(self[0], "strategy_name", "?") + parts = [f"{n:,} combos", f"strategy {name!r}", f"final_equity {eq}"] + if any(s is not None for s in sharpes): + parts.append(f"sharpe {_span(sharpes)}") + if n > _MAX_SCAN: + parts.append(f"(range over first {_MAX_SCAN:,})") + return ("") + + +class WalkForwardResult(dict): + """``run_walk_forward`` output: a dict, with a one-line repr. + + The raw dict carries a full IS and OOS equity curve per fold, so echoing + it in a cell used to print tens of thousands of floats. + """ + + def __repr__(self) -> str: + folds = self.get("folds") or [] + if not folds: + return "" + metric = self.get("optimize_metric", "sharpe") + + def _m(fold, key): + v = fold.get(key) + return v.get(metric) if isinstance(v, dict) else v + + is_v = [_m(f, "is_metrics") for f in folds] + oos_v = [_m(f, "oos_metrics") for f in folds] + lines = [ + f"8} " + f"OOS {_fmt(o) if o is not None else '?':>8} {flat}" + ) + lines.append(" keys: " + ", ".join(sorted(self.keys())) + ">") + return "\n".join(lines) + + +def wrap_sweep_lite(results: List[Any]) -> "SweepLiteResults": + return SweepLiteResults(results) + + +def wrap_walk_forward(result: Dict[str, Any]) -> "WalkForwardResult": + return WalkForwardResult(result) if isinstance(result, dict) else result diff --git a/python/manifoldbt/plot/__init__.py b/python/manifoldbt/plot/__init__.py index 86c68ce..e61a6c8 100644 --- a/python/manifoldbt/plot/__init__.py +++ b/python/manifoldbt/plot/__init__.py @@ -10,11 +10,24 @@ Quick start:: result = bt.run(strategy, config, store) bt.plot.tearsheet(result) # full-page dashboard - bt.plot.equity(result, show=True) # single chart + bt.plot.equity(result) # single chart, opens on its own -Every chart is interactive (crosshair, hover, zoom). ``show=True`` opens it -in a native window (``pip install manifoldbt[window]``; falls back to a +Every chart is interactive (crosshair, hover, zoom) and **shows itself by +default**: plotting is what you asked for, so no ``show=`` is needed. Charts +open in a native window (``pip install manifoldbt[window]``; falls back to a browser tab, which you can also force with ``show="browser"``). + +Three cases opt out of showing automatically, because showing would be +wrong: passing ``save=`` (you asked for a file, not a window), running +under pytest/CI (a window there blocks the run), and running inside a +notebook, where the cell already renders the returned Figure and showing +would print a second copy of the same chart. + +Pass an explicit ``show=True`` to override any of them, or ``show=False`` +to get the Figure back silently and compose it yourself. To place a chart +in the middle of a notebook cell, where there is no trailing expression for +Jupyter to display, call IPython's ``display(fig)`` on the returned figure. + ``save=".html"`` writes a responsive interactive page. Static ``save=".png"`` is optional and needs ``pip install manifoldbt[png]`` (pulls a headless Chromium). """ diff --git a/python/manifoldbt/plot/_utils.py b/python/manifoldbt/plot/_utils.py index 27354da..6122717 100644 --- a/python/manifoldbt/plot/_utils.py +++ b/python/manifoldbt/plot/_utils.py @@ -29,6 +29,57 @@ def new_figure( return fig +def _in_notebook() -> bool: + """True inside a Jupyter/IPython kernel (not a plain terminal REPL).""" + try: + from IPython import get_ipython # type: ignore + except ImportError: + return False + try: + ip = get_ipython() + except Exception: + return False + return ip is not None and hasattr(ip, "kernel") + + +def _in_test_or_ci() -> bool: + """True under pytest or on a CI runner. + + A test that builds a figure must not queue a window: show() runs from + the atexit hook and blocks on the window process, so one bare plot call + in a test suite hangs the whole run until a human closes it. + """ + import os + return bool(os.environ.get("PYTEST_CURRENT_TEST") or os.environ.get("CI")) + + +def resolve_show(show: "bool | str | None", + save: Optional[Union[str, Path]]) -> "bool | str": + """Resolve the ``show=None`` auto default. + + A chart you asked for is a chart you want to see, so the default shows + it. Three cases opt out, because showing there would be wrong: + + - ``save`` was given: you asked for a file, not a window. + - pytest/CI: show() runs from atexit and blocks on the window process. + - a notebook: the cell already renders the returned Figure. Calling + show() here too would emit a SECOND copy of the same chart, so the + notebook path stays silent and lets the cell do the rendering. + + Explicit ``True``/``False``/``"browser"`` always wins. There is no + "render it inline" value to pass, because that is what the notebook + already does with the returned Figure; to place a chart mid-cell, call + IPython's ``display(fig)``. + """ + if show is not None: + return show + if save is not None: + return False + if _in_test_or_ci(): + return False + return False if _in_notebook() else True + + def format_pct(value: float, decimals: int = 1) -> str: """Format a decimal fraction as a percentage string.""" return f"{value * 100:+.{decimals}f}%" @@ -43,7 +94,7 @@ def format_currency(value: float, currency: str = "USD") -> str: def finalize( fig, *, - show: "bool | str" = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, dpi: int = 150, window_size: Optional[Tuple[int, int]] = None, @@ -52,10 +103,20 @@ def finalize( ``save`` routes on extension: ``.html`` writes a responsive interactive page; image extensions (.png/.svg/.pdf/...) go through kaleido. - ``show``: ``True`` (or ``"window"``) opens a native window (needs pywebview, - else falls back to a browser tab); ``"browser"`` forces a browser tab. + ``show``: ``None`` (default) shows the chart unless ``save`` was given or + we are in a notebook (see :func:`resolve_show`); ``True`` (or ``"window"``) + opens a native window (needs pywebview, else falls back to a browser tab); + ``"browser"`` forces a browser tab; ``False`` returns the figure silently. ``dpi`` is kept for backward compatibility and maps to an export scale. """ + show = resolve_show(show, save) + if _in_notebook(): + # new_figure() sets a pixel width sized for a window (1120px by + # default). A notebook cell is narrower than that, so the chart + # overflowed its output area: the right edge and the modebar were + # pushed out of view. Drop the fixed width and let it track the cell, + # keeping the height so the cell still has a definite size. + fig.update_layout(width=None, autosize=True) if save is not None: path = Path(save) ext = path.suffix.lower() @@ -71,7 +132,13 @@ def finalize( ) from exc else: write_responsive_html(fig, path) - if show == "browser": + if show == "inline" and _in_notebook(): + # No-op on purpose. Rendering in the cell IS the notebook default, so + # calling show() here would emit a second copy of the chart the cell + # is already going to render. To place a chart mid-cell, where there + # is no trailing expression, use IPython's display(fig). + pass + elif show in ("browser", "inline"): fig.show() elif show: # True or "window" -> native window (browser tab fallback) from manifoldbt.plot._window import open_in_window diff --git a/python/manifoldbt/plot/backtest.py b/python/manifoldbt/plot/backtest.py index 052c55f..4a06a6c 100644 --- a/python/manifoldbt/plot/backtest.py +++ b/python/manifoldbt/plot/backtest.py @@ -71,7 +71,7 @@ def summary( result, *, figsize: Tuple[float, float] = (14, 8), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """The essential chart: TWR equity + buy-and-hold benchmark, trade activity. @@ -264,7 +264,7 @@ def equity( color: str = ACCENT, title: str = "Equity Curve", figsize: Tuple[float, float] = (14, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Plot the portfolio equity curve over time. @@ -298,7 +298,7 @@ def benchmark_equity( labels: Tuple[str, str] = ("Strategy", "Buy & Hold"), title: str = "Strategy vs Benchmark", figsize: Tuple[float, float] = (14, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Overlay strategy equity and a benchmark, both normalized to 100.""" @@ -339,7 +339,7 @@ def drawdown( color: str = RED, title: str = "Drawdown", figsize: Tuple[float, float] = (14, 3), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Plot the drawdown as a filled area chart.""" @@ -373,7 +373,7 @@ def monthly_returns( annotate: bool = True, title: str = "Monthly Returns (%)", figsize: Tuple[float, float] = (12, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Monthly returns heatmap (year rows x month columns + annual).""" @@ -439,7 +439,7 @@ def annual_returns( ax=None, title: str = "Annual Returns", figsize: Tuple[float, float] = (10, 4), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Annual returns bar chart with green/red conditional coloring.""" @@ -479,7 +479,7 @@ def returns_histogram( bins: int = 100, title: str = "Returns Distribution", figsize: Tuple[float, float] = (12, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Histogram of daily returns with green/red coloring by sign.""" @@ -541,7 +541,7 @@ def var_chart( bins: int = 120, title: str = "Value at Risk", figsize: Tuple[float, float] = (12, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Returns histogram with VaR and CVaR lines at 5% and 1% levels.""" @@ -610,7 +610,7 @@ def rolling_sharpe( title: str = "Rolling Sharpe", trading_days_per_year: float = 365.25, figsize: Tuple[float, float] = (14, 4), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Rolling annualized Sharpe ratio.""" @@ -652,7 +652,7 @@ def rolling_volatility( title: str = "Rolling Volatility", trading_days_per_year: float = 365.25, figsize: Tuple[float, float] = (14, 4), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Rolling annualized volatility.""" diff --git a/python/manifoldbt/plot/chart.py b/python/manifoldbt/plot/chart.py index 290eec3..4314a34 100644 --- a/python/manifoldbt/plot/chart.py +++ b/python/manifoldbt/plot/chart.py @@ -212,7 +212,7 @@ def chart( n_bars: int = 120, interactive: bool = True, figsize: Tuple[float, float] = (14, 7), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ): """Plot candlestick chart with indicators and trade markers. diff --git a/python/manifoldbt/plot/research.py b/python/manifoldbt/plot/research.py index 3d98131..ab77d71 100644 --- a/python/manifoldbt/plot/research.py +++ b/python/manifoldbt/plot/research.py @@ -112,14 +112,29 @@ def heatmap_2d( annotate: bool = True, fmt: str = ".3f", highlight_best: bool = True, + zones: "bool | List[float] | None" = None, + drift: int = 2, title: Optional[str] = None, figsize: Tuple[float, float] = (10, 8), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """2D parameter sweep heatmap from ``run_sweep_2d()`` result. Expected keys: metric_grid, x_values, y_values, x_param, y_param, metric. + + Args: + zones: Colour cells by discrete robustness zone instead of by the + metric. A zone is what the area still guarantees when the + parameters drift by ``drift`` cells, so a lucky spike is shown in + a low zone despite scoring well on its own cell. ``True`` picks + the bands (conventional 0/0.5/1.0/1.5 for risk-adjusted ratios, + an even split of the observed range otherwise); pass a list of + thresholds to set them yourself. The metric value stays on hover + and in the cell labels. Same option as ``surface_3d``. + drift: Neighbourhood radius in grid cells for the worst case. Cells, + not parameter units: with an x step of 1 and a y step of 5, + ``drift=2`` means +/-2 on x but +/-10 on y. """ with theme_context(): grid = np.array(sweep_result["metric_grid"], dtype=np.float64) @@ -135,22 +150,50 @@ def heatmap_2d( text = np.vectorize(lambda v: "" if np.isnan(v) else f"{v:{fmt}}")(grid) fig = new_figure(figsize) - fig.add_trace(go.Heatmap( - z=grid, x=x_vals, y=y_vals, - colorscale=CS_SEQUENTIAL, - text=text, texttemplate="%{text}" if text is not None else None, - textfont=dict(size=9), - hovertemplate=( - f"{x_param} %{{x}}
{y_param} %{{y}}
" - f"{metric} %{{z:{fmt}}}" - ), - colorbar=dict(outlinewidth=0, thickness=12), - hoverongaps=False, - )) + if zones: + worst = _worst_case(grid, drift) + band, scale, labels, edges, n_bands = _zone_bands(worst, zones, metric) + # z carries the band so colour is discrete; the metric and what it + # holds ride along in customdata so the cell still reports both. + fig.add_trace(go.Heatmap( + z=band, x=x_vals, y=y_vals, + colorscale=scale, zmin=-0.5, zmax=n_bands - 0.5, + customdata=np.dstack((grid, worst)), + text=text, texttemplate="%{text}" if text is not None else None, + textfont=dict(size=9), + hovertemplate=( + f"{x_param} %{{x}}
{y_param} %{{y}}
" + f"{metric} %{{customdata[0]:{fmt}}}
" + f"held %{{customdata[1]:{fmt}}}" + ), + colorbar=dict( + title=dict(text=f"{metric}
held under drift", side="right"), + outlinewidth=0, thickness=12, + tickmode="array", tickvals=list(range(n_bands)), + ticktext=labels), + hoverongaps=False, + )) + else: + fig.add_trace(go.Heatmap( + z=grid, x=x_vals, y=y_vals, + colorscale=CS_SEQUENTIAL, + text=text, texttemplate="%{text}" if text is not None else None, + textfont=dict(size=9), + hovertemplate=( + f"{x_param} %{{x}}
{y_param} %{{y}}
" + f"{metric} %{{z:{fmt}}}" + ), + colorbar=dict(outlinewidth=0, thickness=12), + hoverongaps=False, + )) best_label = None if highlight_best: - best_idx = _plateau_best(grid) + if zones: + # Match the colouring: best = what holds up, not the spike. + best_idx = np.unravel_index(np.argmax(worst), worst.shape) + else: + best_idx = _plateau_best(grid) best_val = grid[best_idx] best_x = x_vals[best_idx[1]] best_y = y_vals[best_idx[0]] @@ -163,7 +206,12 @@ def heatmap_2d( x0=best_x - dx, x1=best_x + dx, y0=best_y - dy, y1=best_y + dy, line=dict(color="white", width=2.5), ) - best_label = f"best: {best_val:{fmt}} ({x_param}={best_x:.0f}, {y_param}={best_y:.0f})" + kind = "most robust" if zones else "plateau centre" + best_label = (f"{kind}: {best_val:{fmt}} " + f"({x_param}={best_x:.0f}, {y_param}={best_y:.0f})") + if zones: + best_label += (f", holds {worst[best_idx]:{fmt}} " + f"under +/-{drift} cells") combos = nx * ny main_title = title or f"{metric} · Parameter Sweep ({combos:,} combos)" @@ -188,21 +236,162 @@ def heatmap_2d( # ── 3D Surface Plot ───────────────────────────────────────────────────────── +def _worst_case(grid: np.ndarray, radius: int) -> np.ndarray: + """Lowest value reachable within +/-``radius`` cells of each cell. + + This is what a combo still returns if the parameters drift, as opposed + to what its own cell scored. Lucky spikes collapse to their surroundings; + plateaus keep their value. Edges are replicated so the border is not + flattered by having fewer neighbours. + """ + if radius < 1: + return grid + filled = np.nan_to_num(grid, nan=np.nanmin(grid)) + padded = np.pad(filled, radius, mode="edge") + n, m = filled.shape + stack = np.stack([padded[i:i + n, j:j + m] + for i in range(2 * radius + 1) + for j in range(2 * radius + 1)]) + return stack.min(axis=0) + + +# Metrics where 0 separates losing from winning, so 0 is worth keeping as a +# band edge even when the data would not have put one there. +_RATIO_METRICS = ("sharpe", "sortino", "calmar", "tstat_alpha", "information") +_ZONE_COLORS = ["#3f1d1d", "#7c3a1d", "#8a7a1e", "#2f6b3a", ACCENT] + + +def _nice_step(span: float, n_bands: int) -> float: + """A 1/2/2.5/5 x 10^k step covering ``span`` in about ``n_bands`` steps. + + Rounded steps keep the legend readable: "0.8 - 1.2" rather than + "0.7834 - 1.2017". + """ + if not np.isfinite(span) or span <= 0: + return 1.0 + raw = span / n_bands + mag = 10.0 ** np.floor(np.log10(raw)) + for m in (1.0, 2.0, 2.5, 5.0): + if raw <= m * mag: + return m * mag + return 10.0 * mag + + +def _auto_edges(worst: np.ndarray, metric: str, n_bands: int = 5): + """Band edges fitted to the data, snapped to round numbers. + + Fixed conventional thresholds (0/0.5/1.0/1.5 for a Sharpe) collapse to a + single flat band whenever the sweep happens to live inside one of them, + which is common: a grid whose guaranteed Sharpe runs 1.5-2.0 came out + entirely one colour. + + Edges sit at -1.5 to +1.5 standard deviations around the sweep's mean, so + the zones say how exceptional a region is *within this sweep*. That is a + relative statement, not a quality certificate: a sweep where every combo + loses money still has a top zone, it is just the least bad. Read the + colourbar, which prints the real thresholds, and pass explicit + ``zones=[...]`` whenever the bands must mean something absolute. + """ + finite = worst[np.isfinite(worst)] + if finite.size == 0: + return [0.0] + lo, hi = float(finite.min()), float(finite.max()) + if hi <= lo: # a flat grid has nothing to band + return [lo] + + # Quantiles, not an even split of the range. Taking a minimum over the + # drift window skews the distribution hard toward its low tail, so even + # edges dumped 93% of the cells into one band and the map came out flat. + # Quantiles balance the bands by construction; the snap keeps the numbers + # readable and the colourbar prints them. + # Bands in standard deviations around the mean of the sweep. + # + # Quantiles were the other candidate and they are worse here: they force + # ~20% of cells into every band, so a grid that is genuinely uniform + # after erosion still comes out looking structured. Sigma bands scale + # with the actual dispersion, so a flat sweep reads flat and a sweep with + # a real standout region shows it. They also carry a meaning a reader can + # use: "+1 sigma" is how exceptional the region is for THIS sweep. + mu, sd = float(np.mean(finite)), float(np.std(finite)) + if sd <= 0: + return [lo] + sigmas = np.linspace(-1.5, 1.5, n_bands - 1) # 5 bands -> -1.5..+1.5 + raw_edges = mu + sigmas * sd + step = _nice_step(float(raw_edges[-1] - raw_edges[0]), + max(len(raw_edges) - 1, 1)) + edges = sorted({round(float(np.round(e / step) * step), 10) + for e in raw_edges}) + edges = [e for e in edges if lo < e < hi] + if len(edges) < len(raw_edges): + # Rounding merged edges (a very tight spread): keep them unsnapped. + edges = sorted({float(f"{e:.4g}") for e in raw_edges if lo < e < hi}) + + # 0 is a real boundary for a ratio: above it you make money, below you + # lose it. Keep it even if the rounding would have skipped it. + if any(k in metric.lower() for k in _RATIO_METRICS) and lo < 0.0 < hi: + edges = sorted(set(edges + [0.0])) + if len(edges) > n_bands - 1: # drop the edge nearest 0, not 0 itself + nonzero = [e for e in edges if e != 0.0] + drop = min(nonzero, key=lambda e: abs(e)) + edges.remove(drop) + return edges or [(lo + hi) / 2.0] + + +def _zone_bands(worst: np.ndarray, zones, metric: str): + """Resolve ``zones`` into thresholds, then bucket ``worst`` into bands.""" + if zones is True: + edges = _auto_edges(worst, metric) + else: + edges = sorted(float(z) for z in zones) + + band = np.digitize(worst, edges).astype(float) + n_bands = len(edges) + 1 + labels = [f"< {edges[0]:g}"] + labels += [f"{edges[i]:g} - {edges[i + 1]:g}" for i in range(len(edges) - 1)] + labels.append(f">= {edges[-1]:g}") + + colors = _ZONE_COLORS + if n_bands != len(colors): # stretch or trim the ramp to the band count + idx = np.linspace(0, len(colors) - 1, n_bands).round().astype(int) + colors = [colors[i] for i in idx] + + scale = [] + for i, c in enumerate(colors): # duplicated stops = hard borders + scale.append([i / n_bands, c]) + scale.append([(i + 1) / n_bands, c]) + return band, scale, labels, edges, n_bands + + def surface_3d( sweep_result: Dict[str, Any], *, highlight_best: bool = True, + zones: "bool | List[float] | None" = None, + drift: int = 2, title: Optional[str] = None, figsize: Tuple[float, float] = (12, 8), elev: float = 30, azim: float = -45, - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """3D surface plot from a 2D parameter sweep result. Same input format as ``heatmap_2d``. ``elev``/``azim`` are kept for backward compatibility and mapped to the plotly camera. + + Args: + zones: Colour the surface by discrete robustness zones instead of by + height. Height still shows the metric; colour shows what each + area still guarantees when the parameters drift by ``drift`` + cells, so a lucky spike lands in a low zone despite standing + tall. ``True`` picks the bands (conventional 0/0.5/1.0/1.5 for + risk-adjusted ratios, an even split of the observed range + otherwise); pass a list of thresholds to set them yourself, + which is what you want whenever the bands carry meaning. + drift: Neighbourhood radius in grid cells used for the worst case. + Note this is cells, not parameter units: with an x step of 1 and + a y step of 5, ``drift=2`` means +/-2 on x but +/-10 on y. """ with theme_context(): grid = np.array(sweep_result["metric_grid"], dtype=np.float64) @@ -213,34 +402,94 @@ def surface_3d( metric = sweep_result.get("metric", "metric") fig = new_figure(figsize) - fig.add_trace(go.Surface( - x=x_vals, y=y_vals, z=grid, - colorscale=CS_SEQUENTIAL, opacity=0.98, - colorbar=dict(title=dict(text=metric, side="right"), - outlinewidth=0, thickness=13, len=0.6), - lighting=dict(ambient=0.75, diffuse=0.5, roughness=0.9, specular=0.1), - contours=dict(z=dict(show=True, usecolormap=True, project_z=True, - width=1)), - hovertemplate=( - f"{x_param} %{{x:.2f}}
{y_param} %{{y:.2f}}
" - f"{metric} %{{z:.3f}}" - ), - )) + lighting = dict(ambient=0.75, diffuse=0.5, roughness=0.9, specular=0.1) + + if zones: + worst = _worst_case(grid, drift) + band, scale, labels, edges, n_bands = _zone_bands(worst, zones, metric) + fig.add_trace(go.Surface( + x=x_vals, y=y_vals, z=grid, + surfacecolor=band, colorscale=scale, + cmin=-0.5, cmax=n_bands - 0.5, opacity=0.98, + colorbar=dict( + title=dict(text=f"{metric}
held under drift", side="right"), + outlinewidth=0, thickness=13, len=0.62, + tickmode="array", tickvals=list(range(n_bands)), + ticktext=labels), + lighting=lighting, + contours=dict(z=dict(show=True, color="rgba(255,255,255,0.13)", + width=1)), + customdata=worst, + hovertemplate=( + f"{x_param} %{{x:.2f}}
{y_param} %{{y:.2f}}
" + f"{metric} %{{z:.3f}}
held %{{customdata:.3f}}" + f"" + ), + )) + else: + fig.add_trace(go.Surface( + x=x_vals, y=y_vals, z=grid, + colorscale=CS_SEQUENTIAL, opacity=0.98, + colorbar=dict(title=dict(text=metric, side="right"), + outlinewidth=0, thickness=13, len=0.6), + lighting=lighting, + contours=dict(z=dict(show=True, usecolormap=True, project_z=True, + width=1)), + hovertemplate=( + f"{x_param} %{{x:.2f}}
{y_param} %{{y:.2f}}
" + f"{metric} %{{z:.3f}}" + ), + )) best_label = None if highlight_best: - best_idx = _plateau_best(grid) + if zones: + # With zones on, "best" means the combo that holds up best + # under drift, not the tallest cell. Reporting the spike here + # would contradict the colouring right next to it. + best_idx = np.unravel_index(np.argmax(worst), worst.shape) + held = worst[best_idx] + else: + best_idx = _plateau_best(grid) + held = None best_val = grid[best_idx] bx = x_vals[best_idx[1]] by = y_vals[best_idx[0]] + + # A dot sitting exactly at best_val is half-buried in the surface + # it marks, and a stem dropped to the floor runs underneath that + # surface, hidden by it. So the marker is a pin standing ABOVE + # the peak: the stalk clears the geometry and stays readable from + # any camera angle and over any colour. + span = float(np.nanmax(grid) - np.nanmin(grid)) or 1.0 + tip = best_val + span * 0.10 fig.add_trace(go.Scatter3d( - x=[bx], y=[by], z=[best_val], mode="markers", - marker=dict(color="white", size=6, - line=dict(color="black", width=2)), - name="best", showlegend=False, - hovertemplate=f"best {metric} %{{z:.3f}}", + x=[bx, bx], y=[by, by], z=[best_val, tip], mode="lines", + line=dict(color=WHITE, width=5), + name="best", showlegend=False, hoverinfo="skip", )) - best_label = f"best: {best_val:.3f} ({x_param}={bx:.0f}, {y_param}={by:.0f})" + # Name the criterion. Calling this "best " was a lie + # whenever zones were on: it is not the highest cell, it is the + # one that survives drift, and the highest cell is elsewhere and + # visibly taller. + if zones: + pin_text = (f"most robust
{metric} {best_val:.3f}" + f"
holds {worst[best_idx]:.3f} " + f"under +/-{drift} cells") + else: + pin_text = f"plateau centre
{metric} {best_val:.3f}" + fig.add_trace(go.Scatter3d( + x=[bx], y=[by], z=[tip], mode="markers", + marker=dict(color=WHITE, size=9, symbol="diamond", + line=dict(color="black", width=3)), + name="best", showlegend=False, + hovertemplate=pin_text + "", + )) + kind = "most robust" if zones else "plateau centre" + best_label = (f"{kind}: {best_val:.3f} " + f"({x_param}={bx:.0f}, {y_param}={by:.0f})") + if held is not None: + best_label += f", holds {held:.3f} under +/-{drift} cells" # Map matplotlib elev/azim to a plotly camera eye position r = 1.9 @@ -283,7 +532,7 @@ def walk_forward( oos_color: str = ORANGE, title: Optional[str] = None, figsize: Tuple[float, float] = (10, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Walk-forward analysis chart. @@ -507,7 +756,7 @@ def stability( band_alpha: float = 0.15, title: Optional[str] = None, figsize: Tuple[float, float] = (10, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Parameter stability chart with mean +/- std shaded bands. @@ -572,7 +821,7 @@ def correlation_matrix( annotate: bool = True, title: str = "Correlation Matrix", figsize: Tuple[float, float] = (8, 7), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Symbol correlation matrix heatmap.""" @@ -652,7 +901,7 @@ def monte_carlo( title: Optional[str] = None, figsize: Tuple[float, float] = (12, 5), seed: Optional[int] = None, - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Monte Carlo fan chart with percentile bands, sample paths, and risk stats. @@ -805,7 +1054,7 @@ def stochastic_paths( band_color: str = ACCENT, title: Optional[str] = None, figsize: Tuple[float, float] = (12, 5), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, ) -> go.Figure: """Fan chart for stochastic simulation paths with percentile bands. diff --git a/python/manifoldbt/plot/tearsheet.py b/python/manifoldbt/plot/tearsheet.py index 5128859..ffbc4a8 100644 --- a/python/manifoldbt/plot/tearsheet.py +++ b/python/manifoldbt/plot/tearsheet.py @@ -16,7 +16,7 @@ from manifoldbt.plot._theme import ( theme_context, ) from manifoldbt.plot._convert import equity_with_dates -from manifoldbt.plot._utils import auto_title, chart_div, format_pct +from manifoldbt.plot._utils import auto_title, chart_div, format_pct, resolve_show from manifoldbt.plot.backtest import ( annual_returns, drawdown, @@ -139,7 +139,7 @@ def tearsheet( *, benchmark=None, title: Optional[str] = None, - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, dpi: int = 150, plotlyjs: str = "cdn", @@ -165,14 +165,16 @@ def tearsheet( # ── Generate interactive chart divs ──────────────────────────── with theme_context(): - div_summary = _div(summary(result), height=520) - div_dd = _div(drawdown(result), height=210) - div_annual = _div(annual_returns(result), height=330) - div_monthly = _div(monthly_returns(result), height=340) - div_hist = _div(returns_histogram(result), height=340) - div_sharpe = _div(rolling_sharpe(result), height=300) - div_vol = _div(rolling_volatility(result), height=300) - div_var = _div(var_chart(result), height=340) + # show=False on every panel: these are embedded as divs in the page + # below, so the auto-show default would open 8 stray windows. + div_summary = _div(summary(result, show=False), height=520) + div_dd = _div(drawdown(result, show=False), height=210) + div_annual = _div(annual_returns(result, show=False), height=330) + div_monthly = _div(monthly_returns(result, show=False), height=340) + div_hist = _div(returns_histogram(result, show=False), height=340) + div_sharpe = _div(rolling_sharpe(result, show=False), height=300) + div_vol = _div(rolling_volatility(result, show=False), height=300) + div_var = _div(var_chart(result, show=False), height=340) # ── Metrics ─────────────────────────────────────────────────── ret = metrics.get("total_return", 0) @@ -273,7 +275,9 @@ def tearsheet( if save is not None: Path(save).write_text(html, encoding="utf-8") - if show: + # A report is an HTML page, not a Figure: it always opens in a browser + # tab, so "inline" resolves to the same thing here. + if resolve_show(show, save): if save is not None: report_path = Path(save).resolve() else: @@ -295,7 +299,7 @@ def research_report( *, title: str = "Research Report", figsize: tuple = (14, 6), - show: bool = False, + show: "bool | str | None" = None, save: Optional[Union[str, Path]] = None, dpi: int = 150, ) -> List[Any]: @@ -309,12 +313,13 @@ def research_report( _ = title figs = [] with theme_context(): + # show=False: this function does its own showing at the end. if sweep_result is not None: - figs.append(heatmap_2d(sweep_result, figsize=figsize)) + figs.append(heatmap_2d(sweep_result, figsize=figsize, show=False)) if wf_result is not None: - figs.append(walk_forward(wf_result, figsize=figsize)) + figs.append(walk_forward(wf_result, figsize=figsize, show=False)) if stability_result is not None: - figs.append(stability(stability_result, figsize=figsize)) + figs.append(stability(stability_result, figsize=figsize, show=False)) if not figs: raise ValueError("At least one result (sweep, wf, or stability) required.") @@ -330,7 +335,7 @@ def research_report( else: scale = max(1.0, dpi / 96.0) f.write_image(str(out), scale=scale) - if show: + if resolve_show(show, save): for f in figs: f.show() diff --git a/python/manifoldbt/sweep.py b/python/manifoldbt/sweep.py index e4a51f1..1f4ffce 100644 --- a/python/manifoldbt/sweep.py +++ b/python/manifoldbt/sweep.py @@ -107,7 +107,9 @@ class SweepResult: df = self.to_df(backend="pandas") param_cols = [c for c in df.columns if c.startswith("param_")] - show = kwargs.pop("show", True) + # None = the auto default (show, unless save= or a notebook); both + # branches below hand it to finalize(), which resolves it. + show = kwargs.pop("show", None) save = kwargs.pop("save", None) if len(param_cols) == 2: diff --git a/python/tests/test_golden_buy_and_hold.py b/python/tests/test_golden_buy_and_hold.py index 0f012d8..cc47a8d 100644 --- a/python/tests/test_golden_buy_and_hold.py +++ b/python/tests/test_golden_buy_and_hold.py @@ -11,13 +11,13 @@ import pytest import manifoldbt as bt from manifoldbt import run_with_parquet -# The golden fixtures were generated at full (Pro) resolution; the Community -# resolution cap changes the equity-point count and the comparison is -# meaningless. CI unlocks via BT_UNLOCKED=1 (debug builds); locally this needs -# an activated Pro license. +# The golden fixtures assert on 1-second output resolution, below even the Pro +# floor (60s) — exactly like the Rust golden test, which sets BT_UNLOCKED=1. +# The override is only honored by debug builds (cargo test / maturin develop), +# so this needs BOTH: a dev build and BT_UNLOCKED=1 in the environment. pytestmark = pytest.mark.skipif( - bt.license_info()[0] != "Pro", - reason="requires Pro (fixtures generated at sub-daily resolution); activate a license or use a BT_UNLOCKED dev build", + os.environ.get("BT_UNLOCKED") != "1", + reason="requires BT_UNLOCKED=1 on a dev (debug) build: fixtures assert 1s output, below the Pro 60s floor", ) @@ -38,9 +38,14 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir): universe=[1], time_range_start=0, time_range_end=4_000_000_000, - bar_interval={"Days": 1}, + # The fixture is 4 bars at 1-second spacing; the Rust golden test runs + # them at Seconds(1) with per-bar output. Days(1) would resample the + # whole range into a single bar and the comparison would be meaningless. + bar_interval={"Seconds": 1}, + output_resolution={"Seconds": 1}, initial_capital=1000.0, currency="USD", + risk_free_rate=0.025, execution=bt.ExecutionConfig( signal_delay=1, execution_price="AtClose", @@ -92,8 +97,11 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir): with open(os.path.join(golden_buy_hold_dir, "expected_metrics.json")) as f: expected_metrics = json.load(f) + # Mirror the Rust golden test: annualized metrics (CAGR, volatility, + # sharpe, sortino, calmar) are not compared because the fixture uses 4 + # synthetic 1-second bars, making annualization numerically extreme. metrics = result.metrics - for key in expected_metrics: + for key in ("total_return", "max_drawdown"): assert abs(metrics[key] - expected_metrics[key]) <= 1e-12, ( f"Metric {key}: {metrics[key]} != {expected_metrics[key]}" ) @@ -102,7 +110,11 @@ def test_golden_buy_and_hold_matches_fixtures(golden_buy_hold_dir): with open(os.path.join(golden_buy_hold_dir, "expected_manifest_snapshot.json")) as f: expected_manifest = json.load(f) + # Mirror the Rust golden test: engine_version is excluded from the snapshot + # (it tracks the crate version and would break on every release bump); + # assert only that it is populated. manifest = result.manifest assert manifest["strategy_name"] == expected_manifest["strategy_name"] - assert manifest["engine_version"] == expected_manifest["engine_version"] + assert manifest["engine_version"], "engine_version should be populated" + assert manifest["data_versions"].get("bars_1m", "") == expected_manifest["data_version"] assert manifest["config"] == expected_manifest["config"] diff --git a/python/tests/test_import_dataframe.py b/python/tests/test_import_dataframe.py new file mode 100644 index 0000000..17f0dcc --- /dev/null +++ b/python/tests/test_import_dataframe.py @@ -0,0 +1,164 @@ +"""Tests for bt.import_dataframe — in-memory DataFrame → Arrow IPC store. + +The contract under test: import_dataframe is the in-memory twin of +import_csv. Same data through either path must produce an identical store +(same backtest results), and the normalisation layer must give clear errors +for bad inputs instead of a Rust panic. +""" +import os + +import pytest + +import manifoldbt as bt + +pd = pytest.importorskip("pandas") + +N_BARS = 120 +START_MS = 1_577_836_800_000 # 2020-01-01T00:00:00Z + + +def _bars_df(n=N_BARS, tz="UTC"): + """Synthetic 1m bars as a pandas DataFrame.""" + ts = pd.date_range("2020-01-01", periods=n, freq="1min", tz=tz) + close = [100.0 + i * 0.5 for i in range(n)] + return pd.DataFrame( + { + "timestamp": ts, + "open": close, + "high": [c + 1.0 for c in close], + "low": [c - 1.0 for c in close], + "close": close, + "volume": [10.0] * n, + } + ) + + +def _store_paths(tmp_path, name): + root = tmp_path / name + return str(root / "data"), str(root / "metadata.sqlite") + + +def _import_df(df, tmp_path, name="df", **kw): + data_root, metadata_db = _store_paths(tmp_path, name) + os.makedirs(os.path.dirname(metadata_db), exist_ok=True) + return bt.import_dataframe( + df, symbol="BTCUSDT", symbol_id=1, + data_root=data_root, metadata_db=metadata_db, **kw + ) + + +def _run_buy_and_hold(store): + strategy = bt.Strategy( + name="bh", + signals={"signal": bt.lit(1.0)}, + position_sizing=bt.col("signal"), + ) + config = bt.BacktestConfig( + universe=[1], + time_range_start=0, + time_range_end=START_MS * 1_000_000 + N_BARS * 60_000_000_000, + bar_interval={"Minutes": 1}, + initial_capital=1000.0, + currency="USD", + execution=bt.ExecutionConfig( + signal_delay=1, + execution_price="AtClose", + max_position_pct=1.0, + allow_short=False, + allow_fractional=True, + skip_gap_bars=False, + position_sizing_mode="Units", + ), + fees=bt.FeeConfig(), + slippage={"FixedBps": {"bps": 0.0}}, + rng_seed=7, + ) + return bt.run(strategy, config, store) + + +def test_import_dataframe_roundtrip(tmp_path): + """DataFrame → store → run produces a usable backtest.""" + store = _import_df(_bars_df(), tmp_path) + assert store.resolve_symbol("BTCUSDT") == 1 + + result = _run_buy_and_hold(store) + equity = result.equity_curve.to_pylist() + assert len(equity) > 0 + # Price rises monotonically → buy & hold ends above initial capital. + assert equity[-1] > 1000.0 + + +def test_import_dataframe_matches_import_csv(tmp_path): + """Same bars through import_csv and import_dataframe → identical results.""" + df = _bars_df() + + # CSV path (standard format: epoch-ms timestamp). + # + # Built from START_MS rather than derived from the datetime column: + # `.astype("int64")` returns the underlying integer in the COLUMN's + # resolution, which pandas picks for itself. Locally that was ns (so + # //1e6 gave ms), on CI it was us (so //1e6 gave seconds) and the import + # rejected the row. The bars are 1 minute apart by construction here, so + # spelling the epoch out keeps the CSV identical on every pandas. + csv_df = df.copy() + csv_df["timestamp"] = [START_MS + i * 60_000 for i in range(len(csv_df))] + csv_path = tmp_path / "bars.csv" + csv_df.to_csv(csv_path, index=False) + csv_root, csv_meta = _store_paths(tmp_path, "csv") + os.makedirs(os.path.dirname(csv_meta), exist_ok=True) + store_csv = bt.import_csv( + str(csv_path), symbol="BTCUSDT", symbol_id=1, + data_root=csv_root, metadata_db=csv_meta, + ) + + store_df = _import_df(df, tmp_path) + + res_csv = _run_buy_and_hold(store_csv) + res_df = _run_buy_and_hold(store_df) + assert res_df.equity_curve.to_pylist() == res_csv.equity_curve.to_pylist() + assert res_df.metrics == res_csv.metrics + + +def test_import_dataframe_naive_timestamps_assumed_utc(tmp_path): + """tz-naive datetimes are accepted and treated as UTC.""" + naive = _bars_df(tz=None) + aware = _bars_df(tz="UTC") + store_naive = _import_df(naive, tmp_path, name="naive") + store_aware = _import_df(aware, tmp_path, name="aware") + assert _run_buy_and_hold(store_naive).equity_curve.to_pylist() == \ + _run_buy_and_hold(store_aware).equity_curve.to_pylist() + + +def test_import_dataframe_datetime_index_promoted(tmp_path): + """A pandas DatetimeIndex is used as the timestamp column.""" + df = _bars_df().set_index("timestamp") + assert "timestamp" not in df.columns + store = _import_df(df, tmp_path) + assert store.resolve_symbol("BTCUSDT") == 1 + + +def test_import_dataframe_polars(tmp_path): + """Polars DataFrames go through the zero-copy to_arrow path.""" + pl = pytest.importorskip("polars") + df = pl.from_pandas(_bars_df()) + store = _import_df(df, tmp_path) + assert store.resolve_symbol("BTCUSDT") == 1 + + +def test_import_dataframe_missing_column_raises(tmp_path): + df = _bars_df().drop(columns=["volume"]) + with pytest.raises(bt.DataError, match="volume"): + _import_df(df, tmp_path) + + +def test_import_dataframe_integer_timestamp_raises(tmp_path): + """Epoch integers are ambiguous (ms? ns?) — require datetimes.""" + df = _bars_df() + df["timestamp"] = df["timestamp"].astype("int64") + with pytest.raises(bt.DataError, match="datetime"): + _import_df(df, tmp_path) + + +def test_import_dataframe_empty_raises(tmp_path): + with pytest.raises(bt.DataError, match="no data rows"): + _import_df(_bars_df(0), tmp_path) diff --git a/python/tests/test_sweep.py b/python/tests/test_sweep.py index b5023b0..1d3733b 100644 --- a/python/tests/test_sweep.py +++ b/python/tests/test_sweep.py @@ -31,7 +31,9 @@ def test_sweep_returns_one_result_per_combo(golden_buy_hold_dir): universe=[1], time_range_start=0, time_range_end=4_000_000_000, - bar_interval={"Days": 1}, + # Fixture bars are 1-second spaced; Days(1) collapses them into a + # single bar and signal_delay=1 then never fills → zero trades. + bar_interval={"Seconds": 1}, execution=bt.ExecutionConfig( signal_delay=1, execution_price="AtClose", @@ -89,7 +91,7 @@ def test_sweep_golden_grid_deterministic_order(golden_buy_hold_dir): universe=[1], time_range_start=0, time_range_end=4_000_000_000, - bar_interval={"Days": 1}, + bar_interval={"Seconds": 1}, execution=bt.ExecutionConfig( signal_delay=1, execution_price="AtClose", diff --git a/python/tests/test_sweep_validation.py b/python/tests/test_sweep_validation.py new file mode 100644 index 0000000..18dc56e --- /dev/null +++ b/python/tests/test_sweep_validation.py @@ -0,0 +1,109 @@ +"""Sweeping a parameter the strategy never declares must fail loudly. + +It used to be a silent no-op: the unknown name landed in a parameter map +nothing reads, so every combination ran the same backtest and the sweep +returned N identical results with no warning. An "optimisation" over +thousands of combos looked like it had worked. + +These tests only exercise the Python-side guard, so they need no data store: +validation happens before any native call. +""" +import pytest + +import manifoldbt as bt +from manifoldbt.exceptions import StrategyError +from manifoldbt.indicators import close, ema + + +def _declared(): + """Strategy whose 'fast' comes from mbt.param() inside an indicator.""" + fast = ema(close, bt.param("fast")) + return ( + bt.Strategy.create("declared") + .signal("fast", fast) + .size(bt.when(close > fast, 1.0, 0.0)) + ) + + +def _hardcoded(): + """The shape that caused the bug: the period is a literal, not a param.""" + fast = ema(close, 12) + return ( + bt.Strategy.create("hardcoded") + .signal("fast", fast) + .size(bt.when(close > fast, 1.0, 0.0)) + ) + + +def _cfg(): + # Never reaches the engine: the guard raises before config is used. + return bt.BacktestConfig(universe={"binance": ["BTC-USDT:perp"]}) + + +def test_sweep_lite_rejects_undeclared_param(): + with pytest.raises(StrategyError) as exc: + bt.run_sweep_lite(_hardcoded(), {"fast": [10, 20, 30]}, _cfg(), None) + msg = str(exc.value) + assert "fast" in msg + # The message must say what to do, not just that it failed. + assert "mbt.param" in msg + + +def test_sweep_rejects_undeclared_param(): + with pytest.raises(StrategyError): + bt.run_sweep(_hardcoded(), {"fast": [10, 20]}, _cfg(), None) + + +def test_walk_forward_rejects_undeclared_param(): + wf = { + "method": "Rolling", "n_splits": 2, "train_ratio": 0.7, + "optimize_metric": "sharpe", "param_grid": {"fast": [10, 20]}, + } + with pytest.raises((StrategyError, bt.LicenseError)) as exc: + bt.run_walk_forward(_hardcoded(), wf, _cfg(), None) + # Walk-forward is Pro-gated first; only assert our message when we got past it. + if isinstance(exc.value, StrategyError): + assert "fast" in str(exc.value) + + +def test_sweep_2d_rejects_undeclared_params(): + sweep = { + "x_param": "fast", "y_param": "slow", + "x_values": [5, 10], "y_values": [20, 40], "metric": "sharpe", + } + with pytest.raises(StrategyError) as exc: + bt.run_sweep_2d(_hardcoded(), sweep, _cfg(), None) + assert "fast" in str(exc.value) and "slow" in str(exc.value) + + +def test_stability_rejects_undeclared_param(): + stab = {"param_name": "fast", "values": [5, 10, 15], "metric": "sharpe"} + with pytest.raises(StrategyError) as exc: + bt.run_stability(_hardcoded(), stab, _cfg(), None) + assert "fast" in str(exc.value) + + +def test_declared_param_passes_validation(): + """A declared param must get past the guard (it then fails on the store).""" + with pytest.raises(Exception) as exc: + bt.run_sweep_lite(_declared(), {"fast": [10, 20]}, _cfg(), None) + # Whatever stops it next, it must not be our guard. + assert "not declared" not in str(exc.value) + + +def test_explicit_param_call_counts_as_declared(): + """.param() declares a name even when no expression references it.""" + strat = _hardcoded().param("fast", default=12) + with pytest.raises(Exception) as exc: + bt.run_sweep_lite(strat, {"fast": [10, 20]}, _cfg(), None) + assert "not declared" not in str(exc.value) + + +def test_message_lists_only_the_unknown_names(): + """A mixed grid must blame the unknown name, not the good one.""" + with pytest.raises(StrategyError) as exc: + bt.run_sweep_lite(_declared(), {"fast": [10], "slow": [50]}, _cfg(), None) + msg = str(exc.value) + assert "slow" in msg + # 'fast' is declared, so it must appear as available, never as unknown. + assert "['slow']" in msg