manifoldbt/docs/perf_plan_2026-07.md

# Perf plan: 2x on the CPU sweep (and friends) — execution order

Source: `docs/perf_audit_2026-07-18.md` (measured baselines, ranked table, ceilings).
Scope: run_sweep_lite CPU is the 2x target. Single run gets ~1.5-1.7x. GPU 2x is a
non-goal (fp64 parity ceiling ~1.35x, documented). Lite contract stays as-is.

Ground rules for every phase:
- Bit-for-bit parity CPU==GPU==general, verified against bt-expr goldens, never a mirror.
- When work moves off a path, break the old path on purpose to prove tests still bite.
- Portability: no target-cpu in shipped config; runtime feature detection only.
- Every perf claim: median of >=3 runs, MC-10M-CPU sentinel ~6s clean on both sides.
- One branch per workstream, `perf:` commits, no cross-stream stacking unless noted.

## Status board (updated 2026-07-19)

| phase | state | outcome |
|---|---|---|
| 0 ruler | **done** | harness + bracket probe; re-sized finding #3b downward |
| 1a metrics pydict | **done, merged-ready** | 5.3x to_df / 10.5x best / 9.4x .metrics |
| 1b CAPM hoist | **done, measured** | +11.9% wall, +16.5% sim (paired A/B) |
| 2 single-run parallel | **cancelled** | premise false: signals already parallel |
| 2 simd-dispatch | open | ~1.1-1.15x, off the critical path |
| 3 loop-extraction | **DONE** | daily-equity + bracket-exit + fill/sizing all unified |
| 3 transpiled sweep | **de-risked, not started** | >=1.33x confirmed, plausibly 1.4-1.7x |
| 3 loop tightening | **unblocked, not started** | enabler done: write it once, not four times |
| 4 coverage | not started | optional, per-gate |

Open debts, both found by deliberately breaking things:
- The integration suite is blind to daily-equity drift (only 2 unit tests bite).
- ~14% of the measured +26% sweep gain is unattributed; likely the daily-equity
  simplification, but that needs its own paired A/B before it is claimed.

## End-to-end result so far (2026-07-18)

Measured A-B-A at the build level (HEAD, then main's engine sources, then HEAD
again) so drift between builds is visible rather than assumed. "after" is the
mean of the two HEAD runs.

| | before (main) | after | gain |
|---|---|---|---|
| CPU sweep (10k combos x 26k bars) | 51,332 c/s | ~64,567 c/s | **+26%** |
| SL/TP sweep | 27,662 c/s | ~33,885 c/s | **+22%** |
| `SweepResult.to_df()` | 63.2 us/combo | 12.0 us | **5.3x** |
| `SweepResult.best()` | 25.4 us/combo | 2.4 us | **10.5x** |
| `.metrics` per result | 27.9 us | 3.0 us | **9.4x** |
| GPU sweep (untouched, drift canary) | 164,067 c/s | ~167,436 c/s | +2% |

The GPU line is the control: nothing in this work touches that path and it does
not move, so the harness is not systematically biased.

**Caveats, so these are not over-read.** The two identical HEAD builds differed
by 6.7% (62,474 vs 66,661 c/s) and the baseline was measured once, so read the
sweep numbers as +26% with roughly +/-7%, solid in direction and magnitude but
not to the point. The post-processing ratios are far above the noise (the two
HEAD runs agree to 0.3%) and are reliable.

**One unattributed slice.** The CPU sweep gained ~26% while the paired A/B
credits the CAPM hoist with 11.9%. The likely source of the remaining ~14% is
the daily-equity unification, which was intended as a pure refactor: the merged
form does one division and one comparison per bar where the old form did
`last() == Some(ts/nanos*nanos)` (division AND multiplication) and then possibly
a second `last()/nanos != day` test. Redundant per-bar arithmetic was removed
without aiming for it. This attribution is PLAUSIBLE BUT UNVERIFIED; it needs
its own paired A/B before being claimed.

## Phase 0 — lock the ruler (DONE 2026-07-18)

- [x] Harness checked in at `benchmarks/audit_harness.py` (subcommands: sentinel,
      profile_sweep, bracket_probe, full_vs_lite, single1m, par_scale, sweep_cpu/gpu, access).
- [x] 2026-07-18 baselines are the reference row (see the audit doc).
- [x] Bracket-sweep probe run to size finding #3b.

**Measured (10k combos, ema grid, sentinel-clean):**

| bars | plain | SL/TP bracket | ratio | sim us/combo (plain -> bracket) |
|---|---|---|---|---|
| 26k | 54,800 c/s | 26,440 c/s | 2.07x slower | 242 -> 547 |
| 100k | 11,863 c/s | 3,737 c/s | 3.17x slower | 1,082 -> 3,972 |

**Re-sizing that forces (finding #3b):** the bracket penalty is large but scales
SUPER-linearly with bars (2.07x -> 3.17x), so it is dominated by per-bar bracket
check work, NOT by the `ohl_nan` copy (which is linear in bars). The copy is a
minority slice of the +305us/combo (26k). The audit's "25-40% tax, ~1.2-1.4x from
hoisting" was too optimistic: expect ~1.1x on bracket sweeps from the copy hoist
alone. The real bracket win lives in Phase 3 (loop/macro work), not Phase 1.
Consequence: **the ohl_nan/funding_nan hoist is demoted out of Phase 1** and folded
into the Phase 3 bracket work; Phase 1b keeps only the CAPM hoist (universal ~7%).

## Phase 1 — quick wins, zero parity risk (~2-3 days total)

Branch `perf/full-metrics-pydict` (DONE 2026-07-18)
- [x] `PyBacktestResult.metrics` + `profile` (result.rs): build via the existing
      `metrics_to_pydict` / `profile_to_pydict` (backtest.rs), no JSON round-trip.
- [x] **Nested `trade_stats` hand-mirrored too** (`trade_stats_to_pydict`). This was
      the missing half: `metrics_to_pydict` still round-tripped the nested
      TradeStatistics through JSON. Harmless for LITE sweeps (trade_stats = None) but
      the full `run_sweep` populates it on EVERY combo, so it was the dominant residual
      cost. Fixing only the outer getter gave 1.7x; adding this gave 4-6x.
- [x] Serde-parity pinned: 5 unit tests green, incl. a new
      `metrics_dict_matches_serde_with_trade_stats_signal_quality` for the
      nested-nested `signal_quality`. Drift guard verified to BITE (sabotaging one
      field fails both trade_stats tests with "drifted from serde").
- [x] End-to-end oracle: full `run_sweep` vs dedicated `mbt.run()` per combo (both
      full path, so exact) -- 9 combos x (189 scalar + 162 trade_stats) fields,
      **0 mismatches**.
- [ ] (deferred, optional) Route `to_df()`/`best()` through the `sweep_columns`
      buffer path. Not needed to close Phase 1: the getter fix already removed the
      JSON round-trip; what remains is inherent Python dict/DataFrame building.

**Measured (900 combos):**

| op | before | after | gain |
|---|---|---|---|
| `SweepResult.to_df()` | 49 us/combo | 11.7 us/combo | **4.2x** |
| `SweepResult.best()` | 15 us/combo | 2.4 us/combo | **6.3x** |
| `.metrics` per result | ~18 us | 2.8 us | **~6.4x** |

Note: the audit's "~50x" projection was wrong -- it carried over the scale of the
ORIGINAL 1M-sweep lite bug rather than this path's measured baseline. Real: 4-6x.

Python suite: 63 passed, 2 failed. Both failures (`test_golden_buy_and_hold`,
`test_sweep::test_sweep_returns_one_result_per_combo`) are **pre-existing** --
proven by stashing the change, rebuilding, and reproducing the identical
`2 failed, 63 passed`. Both share one root cause: the golden fixture yields a flat/
truncated equity curve (`[1000.0, 1000.0]`), which makes total_return 0.0 for every
combo. Tracked separately (see the pending `fix/python-test-suite` branch).

Branch `perf/hoist-capm` (Phase 1b, DONE 2026-07-18 -- measured: +11.9% wall)
- [x] CAPM benchmark returns hoisted into `run_sweep_lite` and `run_batch_lite`
      via `hoist_capm_benchmark()`; `run_lite_on_aligned` takes
      `hoisted_benchmark: Option<&[f64]>` and falls back to computing per run when
      None. walk_forward passes None (unchanged behaviour).
- **Guard (verified in code):** hoisting is bit-identical ONLY when the run's
  `sim_bars` is `&aligned.symbol_bars`, i.e. `coarse_bars` is None
  (orchestrator.rs:2300 `if signal_ns > native_ns`). The helper re-detects
  `native_ns` on the post-pre-resample bars and returns None under hybrid
  resampling, where closes come from coarse bars the driver has no handle on.
  A blind driver-side hoist would have silently changed alpha/beta there.
- [x] **Proven equivalent, not merely untested.** A temporary probe recomputed the
      per-run benchmark alongside the hoisted one and asserted `to_bits()`
      equality: green. Then the probe was made to `panic!` on entry to prove the
      hoisted branch is actually REACHED -- it is, by 8 tests, and exactly the
      right ones: `lite_matches_full_with_{stop_loss,take_profit,trailing_stop,
      gap_through_stop,stop_loss_short,full_bracket_and_costs}_at_native_resolution`,
      `lite_and_full_agree_on_max_drawdown_sign`, and
      `per_strategy_orders_apply_and_batch_is_heterogeneous` (batch_lite).
      Probe removed; `cargo test -p bt-core` = 91 passed, 0 failed.
- Note: `golden_buy_and_hold` fails under `--release` but passes in debug with
  `BT_UNLOCKED=1`. That is the known licensing artifact (the dev bypass is
  `#[cfg(debug_assertions)]`, so release runs locked and hits the Pro output
  floor), not this change. It also does not exercise this change at all: it runs
  `run()`, i.e. the full kernel, whose CAPM block was left untouched.

**Perf: MEASURED, interleaved A/B (10k combos x 26k bars, ema_cross).**

A plain before/after was NOT usable here: the Phase 0 baseline was taken while
the paper dashboard was loading the box, so comparing it against a later quiet
run would have credited the hoist with someone else's CPU. Instead the hoist was
put behind a temporary env toggle so ONE binary could run both arms, interleaved
A,B,A,B, in one environment. Paired deltas cancel any drift. Toggle removed after.

| | hoist OFF | hoist ON | delta |
|---|---|---|---|
| sim / combo | 246.2 us | 206.1 us | **+16.5%** |
| wall | 179.9 ms | 159.0 ms | **+11.9%** |
| throughput | 55,593 c/s | 62,883 c/s | |

Four paired rounds, tightly clustered (sim +15.9/+16.6/+17.5/+16.3%), which is
how we know the pairing worked. **The plan's ~1.08x estimate was too
conservative: the real figure is 1.12x on wall.** The audit had priced CAPM at
~1ns/bar (~26us/combo); the measured removal is ~40us/combo, because the pass
also does a resample-to-daily, a step_returns and their allocations per combo,
not just a linear scan.

### The MC-10M sentinel is NOT a load detector (correction)

Recorded because the old heuristic ("~6s clean vs ~16s loaded") is misleading
and cost real time this session. With every competing process killed and the
sweeps posting their best numbers of the day (66,458 c/s), the sentinel still
read ~16s, and inside one 3-run batch it printed `[16.14, 16.36, 7.47]`. It is
bimodal for reasons unrelated to CPU contention (10M paths: allocation /
first-touch / page-cache state), so it flags "loaded" on a quiet machine.

Use instead: interleaved A/B with paired deltas, which is robust to drift by
construction and needs no external notion of "clean".

**Gate to close Phase 1: PASSED 2026-07-18.** `cargo test -p bt-core` 91 passed /
0 failed; bt-python serde-parity 5/5. Python suite 63 passed / 2 failed, and both
failures were proven pre-existing by stashing the change, rebuilding and
reproducing the identical `2 failed, 63 passed` (they share one root cause: the
golden fixture yields a flat equity curve `[1000.0, 1000.0]`). Benches re-run per
protocol, numbers recorded above and in the audit doc.

## Phase 2 — medium effort, contained risk (~1-2 weeks)

Status: the single-run half is cancelled (below); only `perf/simd-dispatch`
remains open, and it is no longer on the critical path to 2x.

Branch `perf/single-run-parallel` -- **CANCELLED 2026-07-18, premise was false.**

- [x] ~~rayon-join independent signal expressions~~ **ALREADY IMPLEMENTED.**
      orchestrator.rs:889 evaluates each dependency level with `level.par_iter()`
      whenever `level.len() >= 2`. The audit's exploration agent reported "fast/slow
      EMA computed sequentially"; that was a misread, and it propagated into this
      plan. Measured at 1M bars, N independent EMAs in one strategy:

      | signals | signal_eval | us/signal | if it were sequential |
      |---|---|---|---|
      | 1 | 6,843 us | 6,843 | - |
      | 2 | 10,071 us | 5,036 | 13,686 |
      | 4 | 10,580 us | 2,645 | 27,372 |
      | 16 | 17,110 us | 1,069 | 109,488 |

      16 signals cost 2.5x one signal, not 16x. The parallelism is real and
      working. Nothing to win here. (Each individual EMA is a sequential scan, so
      the residual per-signal cost is irreducible without changing the recurrence.)

- [x] ~~Overlap output_build components~~ **NOT WORTH IT.** `metrics` borrows
      `trace_equities` while the gather MOVES it, and metrics must run on
      full-resolution equity (max_drawdown depends on it, orchestrator.rs comment
      at the metrics call). Overlapping them needs an 8MB clone at 1M bars, which
      eats most of the ~2.8ms theoretical saving. Best case was ~7% of a 27ms run.

**Consequence for the ceiling:** the audit put the single run at ~1.7x reachable.
That number assumed a sequential signal phase that does not exist. With signals
already parallel and output_build ownership-bound, the single-run path is close to
its practical ceiling; expect well under 1.2x, and it is NOT where the 2x lives.
The 2x target remains the CPU sweep, where per-combo signal work is genuine CPU
load (the sweep saturates all threads across combos, so intra-combo signal
parallelism is degenerate there and the transpiled-sweep item still stands).

Branch `perf/simd-dispatch` (target: ~1.1-1.15x sweep, more on signal-heavy runs)
- [ ] `is_x86_feature_detected!` runtime dispatch on bt-expr elementwise kernels only
      (compare, IfElse, arithmetic). Folds and scans (EWM, rolling, sums) stay scalar.
- [ ] Parity: elementwise same-op-per-lane is bit-identical; add a test asserting
      dispatch on/off equality on goldens. Baseline fallback keeps portability.

Optional branch `perf/full-sweep-traces`
- [ ] Optional trace retention on full run_sweep (orchestrator.rs:3480-3547), or at
      minimum docs steering sweep users to lite + sweep_columns.

## Phase 3 — structural, the 2x closers (~3-5 weeks, sequential)

Branch `refactor/loop-extraction` FIRST (enabler, no behavior change) -- PARTIAL

- [x] **daily-equity rule unified** (commit a48121a). The three lite loops
      (general, multi-asset fast, single-asset fast) each carried their own
      transcription, and they had already drifted into three different forms: two
      decided from `daily_timestamps.last()` with a midnight special case, the
      third from a `current_day` cursor. Now one `record_daily_equity`. The
      cursor is gone: dead state in two of the three sites.
- [x] Verified by sabotage, not by a green suite: perturbing the overwritten
      equity by 1.0001x fails `test_fast_lite_core_matches_simulate_fast_lite` on
      a bitwise `daily_equity[0] bits` assertion. Reverted, 91 passed / 0 failed.
- [x] **bracket-exit rule unified** (commit aabc701). `check_bracket_exit!`
      (~150 LOC) and `check_bracket_exit_lite!` (~100 LOC) were line-for-line
      identical on the decision and pricing: null high/low guard,
      check_stop/check_tp, the gap-aware fill, the slippage call, the clamp to
      the bar range, taker-vs-maker fees. They differed only in what they
      recorded. Now one `resolve_bracket_exit` returning a `BracketExit`; it
      deliberately does not touch capital or positions, so applying the fill and
      the bookkeeping stays at each call site and one function serves two kernels
      that own different state.
- [x] Sabotage-verified on BOTH arms: perturbing the stop fill fails 8 tests
      (incl. the `lite_matches_full_*` family and the batch_lite order test),
      perturbing the take-profit fails 5 (incl.
      `lite_matches_full_with_take_profit`). Reverted, 91 passed / 0 failed.
- [x] No perf regression from turning macro-inlined code into a call in the
      per-bar loop. A-B-A at the build level: bracket sweep 29,602 c/s (macros)
      vs 32,369 c/s (extracted, mean of two builds); plain sweep and GPU flat.
      The two identical extracted builds differ by 7.9%, so the apparent +9% is
      inside the noise and is NOT claimed; "no regression" is what it shows.
- [x] **fill/sizing rule unified** (commit 3a284fb). `simulate_fast`,
      `simulate_fast_lite` and `simulate_fast_lite_single` each had the same
      sequence: sanitize the target into units, round/clamp for
      fractional/short, cap at max_position_pct, delta against the position,
      bail under 1e-12, price AtClose + FixedBps with a min-fee floor. They
      differed only in scalar-vs-Vec storage and `continue` vs `break 'signal`.
      Now `resolve_fast_fill` -> `FastFill`, with the loop-invariants bundled in
      `FastSizing`/`FastCosts` built once before the bar loop. Capital and
      positions stay with the caller, which is what lets one function serve a
      kernel holding `positions[si]` and one holding a scalar `position`.
      `#[inline(always)]`, FP op order preserved exactly (it is what the CUDA
      kernel is transpiled against).
- [x] Sabotage-verified: perturbing the fee fails
      `test_fast_lite_core_matches_simulate_fast_lite`,
      `test_fast_lite_core_units_and_short` and
      `fast_path_matches_general_with_fees` (the lite-vs-core and
      fast-vs-general comparisons). Reverted, 91 passed / 0 failed.
- [x] No perf regression on the hottest code in the engine. A-B-A: plain sweep
      57,304 -> 59,394 c/s, bracket 32,232 -> 30,559 c/s, both inside their own
      build-to-build spread (3.2% and 5.4%). Nothing resolvable either way.

**The enabler is now complete for the lite/fast kernels.** The general full loop
keeps its own fill path (pending orders, limit entries, per-venue fees make it a
different shape); it is out of scope for `perf/lite-loop-tightening`, which
targets the lite kernels. Loop tightening can now be written once.

**Coverage gap found by the sabotage, not fixed here.** With the perturbation
live, ONLY the two lib unit tests failed: all 28 backtest_orders, 14
backtest_single_asset, 7 backtest_multi_asset, per_venue_fees and the goldens
passed with a visibly wrong daily equity curve. So the daily-equity rule is
bit-guarded only by `gpu_sweep_core_tests`, and the integration suite is blind to
drift in the series feeding sharpe, volatility and sortino. Same shape as the
golden that was blind to max_drawdown because dd was 0.0.

**De-risked 2026-07-18: the 1.33x premise holds, and is probably conservative.**

Measured with the paired-A/B discipline (each variant in its own process,
interleaved, only paired deltas trusted), 2500 combos x 26k bars:

| paired delta | value | rounds | reading |
|---|---|---|---|
| +1 indicator (fixed span, declared, unreferenced) | **-0.5 us/combo** | -0.8, -0.3, +11.1, -0.7 | an indicator costs ~nothing per combo |
| +1 elementwise pass (compare+when+add) | **+180 us/combo** | 178, 157, 183, 241 | one pass ~= 1.34x the WHOLE signal phase |

base: signal_eval 134.3 us/combo, simulation 199.7 us/combo.

The ~0 indicator delta is NOT pruning: the compiler compiles every entry of
`def.signals` with no dead-signal elimination
(crates/bt-strategy/src/compiler.rs:74-82). It is AMORTIZATION. A fixed-span EMA
is computed once and every combo hits IndicatorCache; a swept EMA over a 50x50
grid has 50 distinct spans shared by 50 combos each. So indicator math is
effectively free per combo in a 2D sweep, and nearly all of signal_eval is
per-combo overhead (env build, param binding, the elementwise chain, output
allocation) -- precisely what fusing the target into the per-bar loop removes.

Ceiling: the naive arithmetic says 1.68x, but do not quote that. It leans on a
slightly negative indicator delta (so "101% removable", an artifact), the
elementwise delta has 46% spread, and a real transpiled sweep still pays
per-combo param binding and indicator lookup. **Defensible: >= 1.33x, plausibly
1.4-1.7x.** Enough to justify the work; re-measure against the real
implementation rather than trusting this number.

Branch `perf/cpu-transpiled-sweep` (target: ~1.33x, stacked on Phase 1 => ~1.45x)
- [ ] Reuse the GPU hoist plan (build_hoist_plan) on CPU: fill hoisted indicator series
      once per sweep, compute the target in-loop via `sim_fast_lite_core_single`
      (orchestrator.rs:5766), which is already the CUDA kernel's CPU reference.
- [ ] Same eligibility gates as the GPU transpiler; anything else falls back to the
      current vectorized-signal path, unchanged.
- [ ] Deliberately break the old vectorized path (temporarily) to prove the fallback
      is still covered by tests. Parity anchored to bt-expr.

Branch `perf/lite-loop-tightening` (target: 36 -> ~24 cycles/bar, ~1.15-1.3x sweep)
- [ ] Branch elimination in the single-asset core: hoist the sizing_mode match, the
      no-rebalance gates, null-path dispatch. Control flow and layout ONLY.
- [ ] Forbidden: FP reordering, FMA, fast-math of any kind. Op order is the parity
      anchor. Bit-verify against bt-expr after every commit.

Gate to close Phase 3: composite CPU sweep >= 1.9x vs Phase 0 baseline on the 3-strategy
500k bench (median of 3, sentinel-clean), GPU numbers unchanged, parity green.

## Phase 4 — coverage (optional, per-gate decisions)

- [ ] Funding on run()'s CPU fast path (orchestrator.rs:1358-1364): aligns run() with
      lite/GPU, big win for perp single runs.
- [ ] GPU metrics kernel fuse/overlap (gpu_sweep.rs:4395-4656): <=1.16x GPU.
- [ ] Multi-asset GPU: brackets support, occupancy (33% today).
Each: own golden work, own decision. None blocks the 2x goal.

## Non-goals (explicit)

- GPU 2x under fp64 bit-parity: not reachable (sim kernel 73%, SASS-audited ceiling).
  fp32 stays the documented opt-in for speed-over-bits users.
- Changing the lite contract (cagr/calmar/ulcer != run()).
- Machine-specific build flags in shipped wheels.
- Data-loading work (cold 57ms @1M bars, <1% at bench scale): revisit only if a
  many-fresh-process workflow becomes a product path.

## Compatibility invariant

No strategy loses support at any phase. Fast paths widen or stay put; everything not
eligible falls back to today's code, which keeps its own test coverage (proven by the
deliberate-break rule). Example strategies remain testable and sweepable throughout;
`examples/` runs green at every phase gate.