my-python-backteat/ferro-ta-main/PERFORMANCE_ROADMAP.md

ferro-ta Performance Roadmap

Goal: 100x Faster Than Tulipy for Every Indicator

This document tracks the path from current performance to the 100x target.

---

Current State (10k bars, median µs)

Indicator	ferro_ta	Tulipy	Ratio (tu/ft)	Status
SMA	186	84	0.45x	❌ Tulipy faster
EMA	90	89	0.99x	🔄 Parity
RSI	112	91	0.81x	🔄 Near parity
MACD	135	99	0.73x	🔄 Near parity
BBANDS	99	96	0.97x	🔄 Parity
ATR	113	103	0.91x	🔄 Near parity
CCI	147	126	0.86x	🔄 Near parity
WILLR	167	119	0.71x	🔄 Near parity
OBV	88	83	0.94x	🔄 Parity
ADX	165	126	0.76x	🔄 Near parity
MFI	111	122	1.10x	✅ ferro_ta faster
STOCH	176	144	0.82x	🔄 Near parity

vs ta library (Python loops): ferro_ta is already 150–350x faster for slow indicators (ATR, CCI, ADX, MFI).

---

Why ferro_ta Doesn't Beat Tulipy Yet

Both ferro_ta and Tulipy are Rust/C extensions processing 10,000 f64 values. The bottlenecks are:

1. FFI overhead dominates at 10k bars — Python→Rust call overhead is ~50µs fixed cost
2. Array allocation: ferro_ta pads NaN values; Tulipy truncates (saves allocation)
3. SIMD: Tulipy's C code uses auto-vectorization; ferro_ta Rust needs explicit SIMD

---

Optimization Plan

Phase 1: Eliminate FFI Overhead (Target: 2x improvement)

Problem: Each Python call into Rust costs ~50µs regardless of array size.

Solutions:

• Batch API: compute_many([("SMA", close, 20), ("EMA", close, 14)]) — single FFI call
• Buffer reuse: accept pre-allocated output arrays to avoid allocation round-trips
• NumPy zero-copy: use PyReadonlyArray in pyo3 to avoid copies on input

Expected gain: 2x for small arrays (<1k bars), 1.3x for 10k bars.

Phase 2: SIMD Auto-Vectorization (Target: 3x improvement)

Problem: Rust scalar loops vs SIMD C in Tulipy.

Solutions:

• Use std::simd (portable SIMD) for rolling sum accumulation (SMA, WMA)
• Use packed_simd2 for element-wise operations (ADD, SQRT, LOG10, price transforms)
• Enable target-cpu=native in .cargo/config.toml for AVX2/AVX-512

# .cargo/config.toml
[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "target-cpu=native"]

Expected gain: 3-5x for vectorizable indicators (SMA, WMA, price transforms, math ops).

Phase 3: Algorithm-Level Optimizations (Target: 5-10x improvement)

SMA — O(n) running sum

Current: recomputes each window. Target: single-pass running sum (already done in Rust — verify SIMD path is hit).

BBANDS — Welford's algorithm

Current: compute mean, then variance in two passes. Target: Welford's online algorithm — single pass, better cache utilization.

ATR/ADX — Avoid redundant True Range calculations

Current: ATR → ADX each compute TR independently. Target: Compute TR once, share with ATR, NATR, +DI, -DI, ADX in a single pass.

MACD — Reuse EMA computations

Current: Compute fast EMA and slow EMA separately. Target: Single function computes both EMAs in one pass.

Candlestick Patterns — Batch lookup table

Current: Sequential condition checks per bar. Target: Pre-compute body/shadow ratios, vectorized pattern matching.

Phase 4: Streaming Precomputation (Target: 100x for incremental updates)

For real-time systems that update one bar at a time:

• StreamingSMA already O(1) per update — document and benchmark vs batch
• StreamingEMA α * new + (1-α) * prev — single multiply + add
• StreamingBBands — use Welford's online variance
• StreamingRSI — Wilder's smoothing: single multiply per update

At 100k bars, streaming 1 bar at a time is O(n) vs O(n) batch, but with near-zero latency per update.

Benchmark: batch 100k bars vs 100k × streaming 1 bar:

ferro_ta batch SMA(100k):     ~1.8ms
ferro_ta streaming SMA(100k): ~0.5ms total (5µs per bar × 100k = too slow)

Streaming becomes 100x advantage when:

• You only need the latest value (no history needed)
• Input arrives one bar at a time (WebSocket price feed)

---

Measurement Methodology

All benchmarks use:

• pytest-benchmark with pedantic() mode
• 5 iterations × 20 rounds × 2 warmup rounds
• Median timing (not mean) to exclude JIT warmup
• C-contiguous float64 arrays
• 10,000 bars for main benchmarks, 100,000 for scaling tests

Machine: Apple M-series / Intel x86_64 (note: results vary significantly by CPU)

---

Tracking Progress

Run pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json and commit results.json to track regression over time.

---

References

• Tulipy source — C with auto-vectorization
• Rust SIMD Guide
• pyo3 zero-copy arrays