扩展指标
This commit is contained in:
@@ -0,0 +1,321 @@
|
||||
# ferro-ta Benchmark Suite
|
||||
|
||||
> Reproducible speed and accuracy comparisons across 62 indicators and the
|
||||
> libraries available in your environment.
|
||||
|
||||
## Overview
|
||||
|
||||
The benchmark suite compares **ferro-ta** against other Python
|
||||
technical-analysis libraries on a common dataset and shared wrappers so the
|
||||
results are easier to reproduce and critique.
|
||||
|
||||
It is not designed to prove that ferro-ta wins everywhere. It is designed to
|
||||
show where ferro-ta is faster, where it only ties, and where another library
|
||||
still wins.
|
||||
|
||||
| Library | Notes |
|
||||
|-----------|-------|
|
||||
| **TA-Lib** | C extension; widely used comparison baseline |
|
||||
| **pandas-ta** | Pure Python; broad indicator set |
|
||||
| **ta** | Simple API; some indicators use O(n²) loops and are very slow |
|
||||
| **Tulipy** | C extension; truncated output (no leading NaN padding) |
|
||||
| **finta** | Expects DatetimeIndex DataFrame; some indicators very slow |
|
||||
|
||||
---
|
||||
|
||||
## Dataset (LARGE = 100k bars)
|
||||
|
||||
All **speed benchmarks** use the **LARGE** dataset: **100,000 bars** of OHLCV data.
|
||||
|
||||
- **Source:** `benchmarks/data_generator.py` — geometric Brownian motion for realistic prices; C-contiguous `float64` arrays for all libraries.
|
||||
- **Why 100k:** Reflects backtesting and batch workloads; stresses memory and CPU so differences between libraries are clear.
|
||||
- **Scales available:** `SMALL` (1k), `MEDIUM` (10k), `LARGE` (100k). Speed suite uses **LARGE** by default.
|
||||
|
||||
```python
|
||||
from benchmarks.data_generator import SMALL, MEDIUM, LARGE
|
||||
# SMALL = 1,000 bars
|
||||
# MEDIUM = 10,000 bars (e.g. accuracy tests)
|
||||
# LARGE = 100,000 bars (speed benchmarks)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
- **Harness:** [pytest-benchmark](https://pytest-benchmark.readthedocs.io/) with `benchmark.pedantic(..., iterations=5, rounds=20, warmup_rounds=2)`.
|
||||
- **Reported metric:** **Median time per call** in **microseconds (µs)** — lower is better.
|
||||
- **TA-Lib head-to-head JSON:** `benchmarks/bench_vs_talib.py` records per-run samples, variance stats, machine/runtime/build metadata, and Python-tracked peak allocation snapshots.
|
||||
- **Machine info:** Stored in the generated JSON artifacts for reproducibility.
|
||||
- **Libraries:** Only libraries present in the environment are benchmarked; missing ones are skipped.
|
||||
|
||||
## Current checked-in TA-Lib artifact
|
||||
|
||||
The checked-in `benchmarks/artifacts/latest/benchmark_vs_talib.json` artifact
|
||||
uses contiguous `float64` arrays at 10k and 100k bars on an Apple M3 Max,
|
||||
CPython 3.13.5, and Rust 1.91.1 with the default release profile
|
||||
(`lto = true`, `codegen-units = 1`).
|
||||
|
||||
- ferro-ta is ahead outside the tie band on 6 of 12 rows at 10k bars and 6 of 12 rows at 100k bars.
|
||||
- TA-Lib still wins in the current artifact on `STOCH` and `ADX`, and remains close on `EMA`, `RSI`, `ATR`, and `OBV` depending on size.
|
||||
- The public claim should therefore be read as "often faster on selected indicators," not "faster everywhere."
|
||||
- When publishing performance statements, point readers to the raw JSON artifact, not just the summary table.
|
||||
- The artifact now includes per-run samples, variance stats, and Python-tracked allocation snapshots for each compared indicator.
|
||||
|
||||
## Reproducible Perf Artifacts
|
||||
|
||||
Use the perf-contract runner when you want a compact set of machine-readable
|
||||
artifacts for single-series latency, batch throughput, streaming throughput,
|
||||
and hotspot attribution in one directory:
|
||||
|
||||
```bash
|
||||
uv run python benchmarks/run_perf_contract.py --output-dir benchmarks/artifacts/latest --skip-talib
|
||||
```
|
||||
|
||||
That command writes:
|
||||
|
||||
- `indicator_latency.json` — canonical-fixture timings for the benchmark suite indicators
|
||||
- `batch.json` — 2-D batch throughput plus grouped multi-indicator timings
|
||||
- `streaming.json` — streaming update throughput vs batch baselines
|
||||
- `runtime_hotspots.json` — ranked hotspot report with reference speedups
|
||||
- `manifest.json` — runtime/git metadata plus hashes for the generated artifacts
|
||||
|
||||
For CI or local guardrails, validate the hotspot report with:
|
||||
|
||||
```bash
|
||||
uv run python benchmarks/check_hotspot_regression.py --input benchmarks/artifacts/latest/runtime_hotspots.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Speed comparison (100k bars, median µs — lower is better)
|
||||
|
||||
The speed table includes **all 62 indicators**. **Number** = median µs; **N/A** = library does not support that indicator. To regenerate: run the full suite, then `uv run python benchmarks/benchmark_table.py`.
|
||||
|
||||
| Indicator | ferro_ta | talib | pandas_ta | ta | tulipy | finta |
|
||||
|-----------|--------:|--------:|--------:|--------:|--------:|--------:|
|
||||
| SMA | 256 | 327 | 425 | 798 | 338 | 856 |
|
||||
| EMA | 369 | 365 | 427 | 641 | 358 | 722 |
|
||||
| WMA | 257 | 356 | 433 | N/A | 356 | 112422 |
|
||||
| DEMA | 444 | 588 | 670 | N/A | 335 | 1830 |
|
||||
| TEMA | 437 | 768 | 866 | N/A | 358 | 3481 |
|
||||
| T3 | 462 | 407 | 478 | N/A | N/A | 496 |
|
||||
| TRIMA | 598 | 400 | 474 | N/A | 386 | 1722 |
|
||||
| KAMA | 992 | 369 | 140751 | N/A | 367 | 2501 |
|
||||
| HULL_MA | 547 | N/A | 957 | N/A | 372 | 329392 |
|
||||
| VWMA | 376 | N/A | 669 | N/A | 391 | N/A |
|
||||
| MIDPOINT | 1345 | 4685 | N/A | N/A | N/A | N/A |
|
||||
| MIDPRICE | 1273 | 831 | N/A | N/A | N/A | N/A |
|
||||
| RSI | 653 | 647 | 728 | 1762 | 404 | 2429 |
|
||||
| MACD | 833 | 793 | 1058 | 1657 | 423 | 1726 |
|
||||
| STOCH | 2445 | 941 | 1253 | 3233 | 901 | 3321 |
|
||||
| CCI | 918 | 1029 | 1122 | 367074 | 676 | 321471 |
|
||||
| WILLR | 1303 | 750 | 859 | 3409 | 775 | 3575 |
|
||||
| AROON | 1418 | 587 | 1322 | 130842 | 737 | N/A |
|
||||
| AROONOSC | 1464 | 586 | N/A | N/A | 773 | N/A |
|
||||
| ADX | 855 | 746 | 27637 | 321625 | 614 | N/A |
|
||||
| MOM | 189 | 180 | 254 | N/A | 186 | 352 |
|
||||
| ROC | 578 | 204 | 272 | 361 | 202 | 463 |
|
||||
| CMO | 876 | 634 | 707 | N/A | 312 | 2301 |
|
||||
| PPO | 391 | 538 | 1045 | N/A | 380 | 2395 |
|
||||
| TRIX | 488 | 831 | 1831 | 1891 | 426 | 1773 |
|
||||
| TSF | 1519 | 678 | N/A | N/A | 363 | N/A |
|
||||
| ULTOSC | 2069 | 619 | N/A | 14142 | 588 | N/A |
|
||||
| BOP | 249 | 228 | 361 | N/A | 226 | N/A |
|
||||
| PLUS_DI | 794 | 629 | 26792 | N/A | 690 | N/A |
|
||||
| MINUS_DI | 796 | 600 | N/A | N/A | 642 | N/A |
|
||||
| BBANDS | 345 | 581 | 1079 | 2163 | 406 | 2432 |
|
||||
| ATR | 640 | 660 | 800 | 157763 | 370 | 6835 |
|
||||
| NATR | 722 | 662 | 782 | N/A | 396 | N/A |
|
||||
| TRANGE | 217 | 205 | 374 | N/A | 199 | 6606 |
|
||||
| STDDEV | 611 | 408 | 461 | N/A | 400 | 1552 |
|
||||
| VAR | 1281 | 357 | 398 | N/A | 417 | N/A |
|
||||
| SAR | 520 | 459 | N/A | N/A | 454 | N/A |
|
||||
| KELTNER_CHANNELS | 926 | N/A | 1062 | 2369 | N/A | N/A |
|
||||
| DONCHIAN | 2399 | N/A | 3334 | 3145 | N/A | N/A |
|
||||
| SUPERTREND | 1242 | N/A | 638613 | N/A | N/A | N/A |
|
||||
| CHOPPINESS_INDEX | 2442 | N/A | 4892 | N/A | N/A | N/A |
|
||||
| OBV | 482 | 475 | 592 | 496 | 515 | 4646 |
|
||||
| AD | 271 | 282 | 424 | 615 | 291 | N/A |
|
||||
| ADOSC | 482 | 409 | 544 | N/A | 376 | N/A |
|
||||
| MFI | 350 | 779 | 925 | 433698 | 692 | 401076 |
|
||||
| VWAP | 288 | N/A | 11460 | N/A | N/A | 880 |
|
||||
| AVGPRICE | 215 | 211 | N/A | N/A | 229 | N/A |
|
||||
| MEDPRICE | 203 | 188 | N/A | N/A | 197 | 445 |
|
||||
| TYPPRICE | 195 | 205 | N/A | N/A | 204 | 435 |
|
||||
| WCLPRICE | 199 | 197 | N/A | N/A | 210 | 292 |
|
||||
| SQRT | 204 | 208 | N/A | N/A | 199 | N/A |
|
||||
| LOG10 | 434 | 408 | N/A | N/A | 411 | N/A |
|
||||
| ADD | 188 | 186 | N/A | N/A | 189 | N/A |
|
||||
| LINEARREG | 1555 | 704 | N/A | N/A | 368 | N/A |
|
||||
| LINEARREG_SLOPE | 1548 | 665 | N/A | N/A | 370 | N/A |
|
||||
| CORREL | 4277 | 413 | N/A | N/A | N/A | N/A |
|
||||
| BETA | 5226 | 483 | N/A | N/A | N/A | N/A |
|
||||
| HT_DCPERIOD | 10864 | 4187 | N/A | N/A | N/A | N/A |
|
||||
| HT_TRENDMODE | 10984 | 23020 | N/A | N/A | N/A | N/A |
|
||||
| CDLENGULFING | 308 | 617 | N/A | N/A | N/A | N/A |
|
||||
| CDLDOJI | 273 | 312 | N/A | N/A | N/A | N/A |
|
||||
| CDLHAMMER | 304 | 1418 | N/A | N/A | N/A | N/A |
|
||||
|
||||
*Apple M3 Max, Python 3.13; 273 passed, 121 skipped (unsupported = N/A). Regenerate with [Running benchmarks](#running-benchmarks).*
|
||||
|
||||
**Takeaways:**
|
||||
|
||||
- **`ta`** is 20–350× slower on ATR, CCI, ADX, MFI (O(n²) Python loops).
|
||||
- **ferro-ta** is often materially faster than **pandas-ta** on the checked-in 100k-bar table.
|
||||
- **TA-Lib** and **Tulipy** (C extensions) are strong; ferro-ta is competitive and avoids native dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Running benchmarks
|
||||
|
||||
```bash
|
||||
# Full speed suite (100k bars, all indicator × library pairs) — writes results.json
|
||||
uv run pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json -v
|
||||
|
||||
# Head-to-head only (12 indicators × ferro_ta) — quick check
|
||||
uv run pytest benchmarks/test_speed.py --benchmark-only -k "test_head_to_head" -v
|
||||
|
||||
# Large-dataset scaling only (ferro_ta at 100k)
|
||||
uv run pytest benchmarks/test_speed.py --benchmark-only -k "test_large_dataset" -v
|
||||
|
||||
# Regenerate the Speed Comparison markdown table from results.json
|
||||
uv run python benchmarks/benchmark_table.py
|
||||
|
||||
# TA-Lib head-to-head with machine/runtime/build metadata, per-run samples,
|
||||
# variance stats, and Python-tracked allocation snapshots
|
||||
uv run python benchmarks/bench_vs_talib.py --sizes 10000 100000 --json benchmark_vs_talib.json
|
||||
|
||||
# Selected derivatives analytics comparison (BSM price, IV, Greeks, Black-76)
|
||||
# against built-in analytical references plus optional installed libraries
|
||||
uv run python benchmarks/bench_derivatives_compare.py --sizes 1000 10000 --json benchmark_derivatives_compare.json
|
||||
|
||||
# Optional regression check used in CI
|
||||
uv run python benchmarks/check_vs_talib_regression.py --input benchmark_vs_talib.json
|
||||
|
||||
# Batch throughput + grouped multi-indicator calls
|
||||
uv run python benchmarks/bench_batch.py --samples 100000 --series 100 --json batch_benchmark.json
|
||||
|
||||
# Streaming update throughput vs batch baselines
|
||||
uv run python benchmarks/bench_streaming.py --bars 100000 --json streaming_benchmark.json
|
||||
|
||||
# Ranked hotspot attribution against bundled reference implementations
|
||||
uv run python benchmarks/profile_runtime_hotspots.py --json runtime_hotspots.json
|
||||
|
||||
# Portable vs SIMD-enabled build comparison
|
||||
uv run python benchmarks/bench_simd.py --json simd_benchmark.json
|
||||
|
||||
# One-shot perf artifact bundle
|
||||
uv run python benchmarks/run_perf_contract.py --output-dir benchmarks/artifacts/latest
|
||||
```
|
||||
|
||||
Without `uv`: use `pytest` and `python` from the same environment where `ferro_ta` and optional libs (e.g. `talib`, `pandas_ta`, `ta`, `tulipy`, `finta`) are installed.
|
||||
|
||||
### Derivatives analytics
|
||||
|
||||
`benchmarks/bench_derivatives_compare.py` focuses on selected options-analytics
|
||||
paths rather than the full surface area:
|
||||
|
||||
- `BSM` call pricing
|
||||
- call implied-volatility recovery
|
||||
- call Greeks
|
||||
- `Black-76` call pricing
|
||||
|
||||
The script always includes two analytical baselines:
|
||||
|
||||
- `reference_numpy` — pure NumPy formulas with vectorized IV bisection
|
||||
- `reference_python_loop` — scalar `math`-based reference for sanity checking
|
||||
|
||||
If `py_vollib` is installed, it is added automatically as an extra baseline.
|
||||
The output JSON includes runtime/build metadata, per-run timing samples,
|
||||
variance stats, and Python-tracked peak allocation snapshots.
|
||||
|
||||
### WASM
|
||||
|
||||
From the `wasm/` directory:
|
||||
|
||||
```bash
|
||||
wasm-pack build --target nodejs --out-dir pkg
|
||||
node bench.js --json ../wasm_benchmark.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Indicator coverage
|
||||
|
||||
### Overlap (12)
|
||||
`SMA` `EMA` `WMA` `DEMA` `TEMA` `T3` `TRIMA` `KAMA` `HULL_MA` `VWMA` `MIDPOINT` `MIDPRICE`
|
||||
|
||||
### Momentum (18)
|
||||
`RSI` `MACD` `STOCH` `CCI` `WILLR` `AROON` `AROONOSC` `ADX` `MOM` `ROC` `CMO` `PPO` `TRIX` `TSF` `ULTOSC` `BOP` `PLUS_DI` `MINUS_DI`
|
||||
|
||||
### Volatility (11)
|
||||
`BBANDS` `ATR` `NATR` `TRANGE` `STDDEV` `VAR` `SAR` `KELTNER_CHANNELS` `DONCHIAN` `SUPERTREND` `CHOPPINESS_INDEX`
|
||||
|
||||
### Volume (5)
|
||||
`OBV` `AD` `ADOSC` `MFI` `VWAP`
|
||||
|
||||
### Price Transform (4)
|
||||
`AVGPRICE` `MEDPRICE` `TYPPRICE` `WCLPRICE`
|
||||
|
||||
### Math (3)
|
||||
`SQRT` `LOG10` `ADD`
|
||||
|
||||
### Statistics (4)
|
||||
`LINEARREG` `LINEARREG_SLOPE` `CORREL` `BETA`
|
||||
|
||||
### Cycle (2)
|
||||
`HT_DCPERIOD` `HT_TRENDMODE`
|
||||
|
||||
### Candlestick patterns (3)
|
||||
`CDLENGULFING` `CDLDOJI` `CDLHAMMER`
|
||||
|
||||
---
|
||||
|
||||
## Accuracy results
|
||||
|
||||
Accuracy is tested separately; ferro_ta is the reference.
|
||||
|
||||
- **243 pairs pass** (allclose or correlation).
|
||||
- **138 pairs skipped** (known formula/anchoring/scaling differences).
|
||||
- **0 failures.**
|
||||
|
||||
### Known structural differences
|
||||
|
||||
| Pair | Reason |
|
||||
|------|--------|
|
||||
| CMO vs talib/pandas_ta/finta | ferro-ta CMO uses different smoothing variant |
|
||||
| BBANDS vs finta | finta normalizes bands differently |
|
||||
| ATR vs finta | finta uses simple TR instead of Wilder smoothing |
|
||||
| VWAP vs pandas_ta | pandas_ta anchors to session start |
|
||||
| HT_TRENDMODE vs talib | Hilbert Transform seed divergence |
|
||||
| RSI vs ta/finta | ta/finta use SMA warmup vs Wilder EMA |
|
||||
| Tulipy ROC | Fraction (0.01 = 1%) vs ferro-ta (1.0 = 1%) |
|
||||
| Tulipy BBANDS | (lower, mid, upper) order differs from ferro-ta |
|
||||
|
||||
```bash
|
||||
# Accuracy tests (62 indicators × 6 libraries)
|
||||
uv run pytest benchmarks/test_accuracy.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data generator
|
||||
|
||||
`benchmarks/data_generator.py`:
|
||||
|
||||
- **`generate_ohlcv(size)`** — dict of C-contiguous `float64` arrays: `open`, `high`, `low`, `close`, `volume`. High ≥ close ≥ low > 0; volume > 0.
|
||||
- **`get_pandas_ohlcv(data)`** — DataFrame with DatetimeIndex for pandas-ta and finta.
|
||||
|
||||
Pre-built: `SMALL`, `MEDIUM`, `LARGE` (and `*_DF` variants).
|
||||
|
||||
---
|
||||
|
||||
## Library compatibility
|
||||
|
||||
Detailed notes per library:
|
||||
|
||||
- [TA-Lib](../docs/compatibility/talib.md)
|
||||
- [pandas-ta](../docs/compatibility/pandas_ta.md)
|
||||
- [ta](../docs/compatibility/ta.md)
|
||||
- [Tulipy](../docs/compatibility/tulipy.md)
|
||||
- [finta](../docs/compatibility/finta.md)
|
||||
@@ -0,0 +1 @@
|
||||
"""benchmarks package — cross-library accuracy and speed comparison suite."""
|
||||
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "batch",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:25:58.345834+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"n_samples": 100000,
|
||||
"n_series": 100,
|
||||
"total_bars": 10000000,
|
||||
"seed": 42
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"indicator": "SMA",
|
||||
"parallel_ms": 37.86,
|
||||
"sequential_ms": 43.5625,
|
||||
"loop_ms": 17.8136,
|
||||
"parallel_speedup_vs_loop": 0.4705,
|
||||
"sequential_speedup_vs_loop": 0.4089
|
||||
},
|
||||
{
|
||||
"indicator": "RSI",
|
||||
"parallel_ms": 40.9229,
|
||||
"sequential_ms": 79.5345,
|
||||
"loop_ms": 53.3368,
|
||||
"parallel_speedup_vs_loop": 1.3033,
|
||||
"sequential_speedup_vs_loop": 0.6706
|
||||
},
|
||||
{
|
||||
"indicator": "ATR",
|
||||
"parallel_ms": 91.76,
|
||||
"sequential_ms": 130.1404,
|
||||
"loop_ms": 99.5885,
|
||||
"parallel_speedup_vs_loop": 1.0853,
|
||||
"sequential_speedup_vs_loop": 0.7652
|
||||
},
|
||||
{
|
||||
"indicator": "ADX",
|
||||
"parallel_ms": 100.1362,
|
||||
"sequential_ms": 149.3412,
|
||||
"loop_ms": 125.3319,
|
||||
"parallel_speedup_vs_loop": 1.2516,
|
||||
"sequential_speedup_vs_loop": 0.8392
|
||||
}
|
||||
],
|
||||
"grouped_results": [
|
||||
{
|
||||
"case": "close_bundle_3",
|
||||
"grouped_ms": 0.652,
|
||||
"separate_ms": 0.9124,
|
||||
"speedup_vs_separate": 1.3994
|
||||
},
|
||||
{
|
||||
"case": "hlc_bundle_3",
|
||||
"grouped_ms": 1.4784,
|
||||
"separate_ms": 3.3724,
|
||||
"speedup_vs_separate": 2.2811
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "backtest",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-27T16:31:53.866252+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"python_implementation": "CPython",
|
||||
"python_executable": "/Users/pratikbhadane/Work/Projects/ferro-ta/.venv/bin/python3",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"system": "Darwin",
|
||||
"release": "25.3.0",
|
||||
"machine": "arm64",
|
||||
"processor": "arm",
|
||||
"cpu_model": "Apple M3 Max",
|
||||
"cpu_count_logical": 14,
|
||||
"total_memory_bytes": 38654705664
|
||||
},
|
||||
"git": {
|
||||
"commit": "2d776b6f908fd1a4f30a696972b7df5e5fe2ca00",
|
||||
"dirty": true,
|
||||
"branch": "main"
|
||||
},
|
||||
"build": {
|
||||
"rustc": "rustc 1.93.1 (01f6ddf75 2026-02-11)\nbinary: rustc\ncommit-hash: 01f6ddf7588f42ae2d7eb0a2f21d44e8e96674cf\ncommit-date: 2026-02-11\nhost: aarch64-apple-darwin\nrelease: 1.93.1\nLLVM version: 21.1.8",
|
||||
"cargo": "cargo 1.93.1 (083ac5135 2025-12-15)",
|
||||
"cargo_release_profile": {
|
||||
"lto": true,
|
||||
"codegen-units": 1
|
||||
},
|
||||
"rustflags": null,
|
||||
"cargo_build_rustflags": null,
|
||||
"maturin_flags": null
|
||||
},
|
||||
"packages": {
|
||||
"numpy": "2.2.6",
|
||||
"ferro-ta": "1.0.6"
|
||||
}
|
||||
},
|
||||
"results": {
|
||||
"backtest_core_single": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"ferro_ta_ms": 0.024,
|
||||
"ferro_ta_mbars_s": 415.9388,
|
||||
"vectorbt_ms": 1.2843,
|
||||
"speedup_vs_vectorbt": 53.4187
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"ferro_ta_ms": 0.1964,
|
||||
"ferro_ta_mbars_s": 509.1209,
|
||||
"vectorbt_ms": 3.047,
|
||||
"speedup_vs_vectorbt": 15.5129
|
||||
}
|
||||
],
|
||||
"backtest_ohlcv_core": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"ferro_ta_ms": 0.0573,
|
||||
"ferro_ta_mbars_s": 174.4166
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"ferro_ta_ms": 0.7068,
|
||||
"ferro_ta_mbars_s": 141.4927
|
||||
}
|
||||
],
|
||||
"performance_metrics": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"ferro_ta_ms": 0.2182,
|
||||
"numpy_partial_ms": 0.0496,
|
||||
"speedup_vs_numpy": 0.2272,
|
||||
"note": "numpy_partial only computes sharpe+max_dd (2/23 metrics)"
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"ferro_ta_ms": 3.0303,
|
||||
"numpy_partial_ms": 0.351,
|
||||
"speedup_vs_numpy": 0.1158,
|
||||
"note": "numpy_partial only computes sharpe+max_dd (2/23 metrics)"
|
||||
}
|
||||
],
|
||||
"multi_asset": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"n_assets": 50,
|
||||
"parallel_ms": 2.4245,
|
||||
"serial_ms": 4.1751,
|
||||
"loop_ms": 2.0349,
|
||||
"parallel_speedup_vs_loop": 0.8393,
|
||||
"parallel_speedup_vs_serial": 1.722
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"n_assets": 50,
|
||||
"parallel_ms": 24.0349,
|
||||
"serial_ms": 47.9311,
|
||||
"loop_ms": 24.7476,
|
||||
"parallel_speedup_vs_loop": 1.0297,
|
||||
"parallel_speedup_vs_serial": 1.9942
|
||||
}
|
||||
],
|
||||
"monte_carlo": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"n_sims": 500,
|
||||
"ferro_ta_ms": 3.862,
|
||||
"numpy_loop_ms": 51.1589,
|
||||
"speedup_vs_numpy": 13.2469
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"n_sims": 500,
|
||||
"ferro_ta_ms": 26.0019,
|
||||
"numpy_loop_ms": 310.582,
|
||||
"speedup_vs_numpy": 11.9446
|
||||
}
|
||||
],
|
||||
"engine_full_pipeline": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"ferro_ta_ms": 0.4402,
|
||||
"description": "Full pipeline: signals + OHLCV fill + 23 metrics + trades + drawdown"
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"ferro_ta_ms": 4.445,
|
||||
"description": "Full pipeline: signals + OHLCV fill + 23 metrics + trades + drawdown"
|
||||
}
|
||||
],
|
||||
"walk_forward_indices": [
|
||||
{
|
||||
"n_bars": 10000,
|
||||
"train_bars": 2000,
|
||||
"test_bars": 500,
|
||||
"ferro_ta_us": 0.333
|
||||
},
|
||||
{
|
||||
"n_bars": 100000,
|
||||
"train_bars": 20000,
|
||||
"test_bars": 5000,
|
||||
"ferro_ta_us": 0.292
|
||||
}
|
||||
],
|
||||
"kelly_fraction": [
|
||||
{
|
||||
"n_calls": 1000,
|
||||
"ferro_ta_us": 86.458
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,163 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "indicator_latency",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:25:52.160357+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"fixtures": [
|
||||
{
|
||||
"path": "/Users/pratikbhadane/Work/Projects/ferro-ta/benchmarks/fixtures/canonical_ohlcv.npz",
|
||||
"size_bytes": 75586,
|
||||
"sha256": "60192f8349fb06cd59ef7f70fd77aa8280399e819d7cc5eed3ca95cf5ee1a89c"
|
||||
}
|
||||
],
|
||||
"dataset": {
|
||||
"fixture": "/Users/pratikbhadane/Work/Projects/ferro-ta/benchmarks/fixtures/canonical_ohlcv.npz",
|
||||
"bars": 2000,
|
||||
"rounds": 5
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"name": "VAR_20",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 20
|
||||
},
|
||||
"elapsed_ms": 0.0229
|
||||
},
|
||||
{
|
||||
"name": "STOCH",
|
||||
"inputs": "hlc",
|
||||
"kwargs": {},
|
||||
"elapsed_ms": 0.0201
|
||||
},
|
||||
{
|
||||
"name": "WILLR_14",
|
||||
"inputs": "hlc",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0201
|
||||
},
|
||||
{
|
||||
"name": "CCI_14",
|
||||
"inputs": "hlc",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0163
|
||||
},
|
||||
{
|
||||
"name": "ADX_14",
|
||||
"inputs": "hlc",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.015
|
||||
},
|
||||
{
|
||||
"name": "MACD",
|
||||
"inputs": "close",
|
||||
"kwargs": {},
|
||||
"elapsed_ms": 0.0135
|
||||
},
|
||||
{
|
||||
"name": "ATR_14",
|
||||
"inputs": "hlc",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0105
|
||||
},
|
||||
{
|
||||
"name": "RSI_14",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0098
|
||||
},
|
||||
{
|
||||
"name": "STDDEV_20",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 20
|
||||
},
|
||||
"elapsed_ms": 0.009
|
||||
},
|
||||
{
|
||||
"name": "BETA_5",
|
||||
"inputs": "pair_hl",
|
||||
"kwargs": {
|
||||
"timeperiod": 5
|
||||
},
|
||||
"elapsed_ms": 0.0083
|
||||
},
|
||||
{
|
||||
"name": "CORREL_30",
|
||||
"inputs": "pair_hl",
|
||||
"kwargs": {
|
||||
"timeperiod": 30
|
||||
},
|
||||
"elapsed_ms": 0.0076
|
||||
},
|
||||
{
|
||||
"name": "BBANDS_20",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 20
|
||||
},
|
||||
"elapsed_ms": 0.0055
|
||||
},
|
||||
{
|
||||
"name": "LINEARREG_14",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0055
|
||||
},
|
||||
{
|
||||
"name": "TSF_14",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0054
|
||||
},
|
||||
{
|
||||
"name": "LINEARREG_SLOPE_14",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 14
|
||||
},
|
||||
"elapsed_ms": 0.0052
|
||||
},
|
||||
{
|
||||
"name": "EMA_20",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 20
|
||||
},
|
||||
"elapsed_ms": 0.005
|
||||
},
|
||||
{
|
||||
"name": "SMA_20",
|
||||
"inputs": "close",
|
||||
"kwargs": {
|
||||
"timeperiod": 20
|
||||
},
|
||||
"elapsed_ms": 0.0026
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "perf_contract",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:26:40.776130+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"fixtures": [
|
||||
{
|
||||
"path": "/Users/pratikbhadane/Work/Projects/ferro-ta/benchmarks/fixtures/canonical_ohlcv.npz",
|
||||
"size_bytes": 75586,
|
||||
"sha256": "60192f8349fb06cd59ef7f70fd77aa8280399e819d7cc5eed3ca95cf5ee1a89c"
|
||||
}
|
||||
],
|
||||
"output_dir": "benchmarks/artifacts/latest"
|
||||
},
|
||||
"artifacts": {
|
||||
"indicator_latency": {
|
||||
"path": "benchmarks/artifacts/latest/indicator_latency.json",
|
||||
"size_bytes": 3217,
|
||||
"sha256": "43b88a50a4d7f91e30ff8e57dbf859ae5e76ecabaaf05cfcb7d8db67df920f7f"
|
||||
},
|
||||
"batch": {
|
||||
"path": "benchmarks/artifacts/latest/batch.json",
|
||||
"size_bytes": 1701,
|
||||
"sha256": "bc900c885c48ec1903ea4870ca1de8cb9f33c609d2cefa4688cbdd18cb977f11"
|
||||
},
|
||||
"streaming": {
|
||||
"path": "benchmarks/artifacts/latest/streaming.json",
|
||||
"size_bytes": 1944,
|
||||
"sha256": "925ba1be66d0d499daa81dfc148b03ac325ad685ce0c71e28ca1fc6927f16415"
|
||||
},
|
||||
"runtime_hotspots": {
|
||||
"path": "benchmarks/artifacts/latest/runtime_hotspots.json",
|
||||
"size_bytes": 2366,
|
||||
"sha256": "920553b14b545f211b119c099ec59885de8b9e8056271cb2d2ac34c0c69b0906"
|
||||
},
|
||||
"simd": {
|
||||
"path": "benchmarks/artifacts/latest/simd.json",
|
||||
"size_bytes": 7700,
|
||||
"sha256": "d48943a5dfcf4f8d8d2ca42f0004f02f9fc894de7477791b686231da665e3335"
|
||||
},
|
||||
"benchmark_vs_talib": {
|
||||
"path": "benchmarks/artifacts/latest/benchmark_vs_talib.json",
|
||||
"size_bytes": 5923,
|
||||
"sha256": "8a4e847517f1334255353982a5266c0323bf433a1eb78dafeff808d5ad3bf7f0"
|
||||
},
|
||||
"wasm": {
|
||||
"path": "benchmarks/artifacts/latest/wasm.json",
|
||||
"size_bytes": 935,
|
||||
"sha256": "f31fd871990c44e24a2259d618ae40a52866d20b95aa6047af3d38b9371c2ab7"
|
||||
},
|
||||
"bench_backtest": {
|
||||
"path": "benchmarks/artifacts/latest/bench_backtest_results.json",
|
||||
"size_bytes": 4022,
|
||||
"sha256": "acf27cd5d5077aff51194e31936aba2b9304a8a62d993b2ec496d6f347545316"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "runtime_hotspots",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:26:02.236710+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"price_bars": 20000,
|
||||
"iv_bars": 50000,
|
||||
"window": 252
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_zscore",
|
||||
"fast_ms": 35.984,
|
||||
"reference_ms": 944.5804,
|
||||
"speedup_vs_reference": 26.25,
|
||||
"share_of_suite_pct": 77.27
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_percentile",
|
||||
"fast_ms": 7.6624,
|
||||
"reference_ms": 81.581,
|
||||
"speedup_vs_reference": 10.6469,
|
||||
"share_of_suite_pct": 16.45
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_rank",
|
||||
"fast_ms": 2.2905,
|
||||
"reference_ms": 198.2937,
|
||||
"speedup_vs_reference": 86.5738,
|
||||
"share_of_suite_pct": 4.92
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "feature_matrix",
|
||||
"fast_ms": 0.2872,
|
||||
"reference_ms": 0.2377,
|
||||
"speedup_vs_reference": 0.8275,
|
||||
"share_of_suite_pct": 0.62
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "compute_many_close",
|
||||
"fast_ms": 0.1448,
|
||||
"reference_ms": 0.1505,
|
||||
"speedup_vs_reference": 1.0391,
|
||||
"share_of_suite_pct": 0.31
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "BETA",
|
||||
"fast_ms": 0.0637,
|
||||
"reference_ms": 164.1752,
|
||||
"speedup_vs_reference": 2575.2975,
|
||||
"share_of_suite_pct": 0.14
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "CORREL",
|
||||
"fast_ms": 0.0553,
|
||||
"reference_ms": 159.6473,
|
||||
"speedup_vs_reference": 2885.1573,
|
||||
"share_of_suite_pct": 0.12
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "LINEARREG",
|
||||
"fast_ms": 0.0415,
|
||||
"reference_ms": 47.4665,
|
||||
"speedup_vs_reference": 1143.77,
|
||||
"share_of_suite_pct": 0.09
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "TSF",
|
||||
"fast_ms": 0.0414,
|
||||
"reference_ms": 47.921,
|
||||
"speedup_vs_reference": 1157.036,
|
||||
"share_of_suite_pct": 0.09
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,285 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "simd",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:26:40.566511+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"price_bars": 20000,
|
||||
"iv_bars": 50000,
|
||||
"window": 252
|
||||
},
|
||||
"variants": [
|
||||
"portable_release",
|
||||
"simd_release"
|
||||
]
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"name": "BETA",
|
||||
"category": "rust_kernel",
|
||||
"portable_ms": 0.0635,
|
||||
"simd_ms": 0.0636,
|
||||
"speedup_simd_vs_portable": 0.9984
|
||||
},
|
||||
{
|
||||
"name": "TSF",
|
||||
"category": "rust_kernel",
|
||||
"portable_ms": 0.0415,
|
||||
"simd_ms": 0.0417,
|
||||
"speedup_simd_vs_portable": 0.9952
|
||||
},
|
||||
{
|
||||
"name": "compute_many_close",
|
||||
"category": "ffi_grouping",
|
||||
"portable_ms": 0.1548,
|
||||
"simd_ms": 0.1572,
|
||||
"speedup_simd_vs_portable": 0.9847
|
||||
},
|
||||
{
|
||||
"name": "iv_zscore",
|
||||
"category": "python_analysis",
|
||||
"portable_ms": 36.0643,
|
||||
"simd_ms": 37.2041,
|
||||
"speedup_simd_vs_portable": 0.9694
|
||||
},
|
||||
{
|
||||
"name": "feature_matrix",
|
||||
"category": "ffi_grouping",
|
||||
"portable_ms": 0.2556,
|
||||
"simd_ms": 0.2667,
|
||||
"speedup_simd_vs_portable": 0.9584
|
||||
},
|
||||
{
|
||||
"name": "iv_percentile",
|
||||
"category": "python_analysis",
|
||||
"portable_ms": 7.7548,
|
||||
"simd_ms": 8.1565,
|
||||
"speedup_simd_vs_portable": 0.9508
|
||||
},
|
||||
{
|
||||
"name": "LINEARREG",
|
||||
"category": "rust_kernel",
|
||||
"portable_ms": 0.0416,
|
||||
"simd_ms": 0.0443,
|
||||
"speedup_simd_vs_portable": 0.9391
|
||||
},
|
||||
{
|
||||
"name": "iv_rank",
|
||||
"category": "python_analysis",
|
||||
"portable_ms": 2.2813,
|
||||
"simd_ms": 2.4386,
|
||||
"speedup_simd_vs_portable": 0.9355
|
||||
},
|
||||
{
|
||||
"name": "CORREL",
|
||||
"category": "rust_kernel",
|
||||
"portable_ms": 0.0552,
|
||||
"simd_ms": 0.0633,
|
||||
"speedup_simd_vs_portable": 0.872
|
||||
}
|
||||
],
|
||||
"reports": {
|
||||
"portable_release": {
|
||||
"metadata": {
|
||||
"suite": "runtime_hotspots",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:26:06.920513+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"price_bars": 20000,
|
||||
"iv_bars": 50000,
|
||||
"window": 252
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_zscore",
|
||||
"fast_ms": 36.0643,
|
||||
"reference_ms": 908.5403,
|
||||
"speedup_vs_reference": 25.1922,
|
||||
"share_of_suite_pct": 77.2
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_percentile",
|
||||
"fast_ms": 7.7548,
|
||||
"reference_ms": 82.2352,
|
||||
"speedup_vs_reference": 10.6045,
|
||||
"share_of_suite_pct": 16.6
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_rank",
|
||||
"fast_ms": 2.2813,
|
||||
"reference_ms": 202.5375,
|
||||
"speedup_vs_reference": 88.7803,
|
||||
"share_of_suite_pct": 4.88
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "feature_matrix",
|
||||
"fast_ms": 0.2556,
|
||||
"reference_ms": 0.2252,
|
||||
"speedup_vs_reference": 0.8812,
|
||||
"share_of_suite_pct": 0.55
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "compute_many_close",
|
||||
"fast_ms": 0.1548,
|
||||
"reference_ms": 0.1508,
|
||||
"speedup_vs_reference": 0.9742,
|
||||
"share_of_suite_pct": 0.33
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "BETA",
|
||||
"fast_ms": 0.0635,
|
||||
"reference_ms": 162.8972,
|
||||
"speedup_vs_reference": 2563.6148,
|
||||
"share_of_suite_pct": 0.14
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "CORREL",
|
||||
"fast_ms": 0.0552,
|
||||
"reference_ms": 163.1357,
|
||||
"speedup_vs_reference": 2952.6826,
|
||||
"share_of_suite_pct": 0.12
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "LINEARREG",
|
||||
"fast_ms": 0.0416,
|
||||
"reference_ms": 48.0097,
|
||||
"speedup_vs_reference": 1153.3863,
|
||||
"share_of_suite_pct": 0.09
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "TSF",
|
||||
"fast_ms": 0.0415,
|
||||
"reference_ms": 47.9395,
|
||||
"speedup_vs_reference": 1155.1696,
|
||||
"share_of_suite_pct": 0.09
|
||||
}
|
||||
]
|
||||
},
|
||||
"simd_release": {
|
||||
"metadata": {
|
||||
"suite": "runtime_hotspots",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:26:25.789478+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"price_bars": 20000,
|
||||
"iv_bars": 50000,
|
||||
"window": 252
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_zscore",
|
||||
"fast_ms": 37.2041,
|
||||
"reference_ms": 930.7842,
|
||||
"speedup_vs_reference": 25.0183,
|
||||
"share_of_suite_pct": 76.81
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_percentile",
|
||||
"fast_ms": 8.1565,
|
||||
"reference_ms": 88.3639,
|
||||
"speedup_vs_reference": 10.8336,
|
||||
"share_of_suite_pct": 16.84
|
||||
},
|
||||
{
|
||||
"category": "python_analysis",
|
||||
"name": "iv_rank",
|
||||
"fast_ms": 2.4386,
|
||||
"reference_ms": 221.0389,
|
||||
"speedup_vs_reference": 90.6424,
|
||||
"share_of_suite_pct": 5.03
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "feature_matrix",
|
||||
"fast_ms": 0.2667,
|
||||
"reference_ms": 0.2436,
|
||||
"speedup_vs_reference": 0.9134,
|
||||
"share_of_suite_pct": 0.55
|
||||
},
|
||||
{
|
||||
"category": "ffi_grouping",
|
||||
"name": "compute_many_close",
|
||||
"fast_ms": 0.1572,
|
||||
"reference_ms": 0.1593,
|
||||
"speedup_vs_reference": 1.0135,
|
||||
"share_of_suite_pct": 0.32
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "BETA",
|
||||
"fast_ms": 0.0636,
|
||||
"reference_ms": 172.9198,
|
||||
"speedup_vs_reference": 2717.7961,
|
||||
"share_of_suite_pct": 0.13
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "CORREL",
|
||||
"fast_ms": 0.0633,
|
||||
"reference_ms": 170.0262,
|
||||
"speedup_vs_reference": 2686.3776,
|
||||
"share_of_suite_pct": 0.13
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "LINEARREG",
|
||||
"fast_ms": 0.0443,
|
||||
"reference_ms": 50.5614,
|
||||
"speedup_vs_reference": 1141.5474,
|
||||
"share_of_suite_pct": 0.09
|
||||
},
|
||||
{
|
||||
"category": "rust_kernel",
|
||||
"name": "TSF",
|
||||
"fast_ms": 0.0417,
|
||||
"reference_ms": 50.9599,
|
||||
"speedup_vs_reference": 1221.8259,
|
||||
"share_of_suite_pct": 0.09
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "streaming",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:25:58.628657+00:00",
|
||||
"python_version": "3.13.5",
|
||||
"platform": "macOS-26.3.1-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"processor": "arm"
|
||||
},
|
||||
"git": {
|
||||
"commit": "9011250f992119170242cf17a67834c67b91bcdb",
|
||||
"dirty": true,
|
||||
"branch": "feat/performace-1.0.2"
|
||||
},
|
||||
"dataset": {
|
||||
"n_bars": 100000,
|
||||
"seed": 2026
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"indicator": "StreamingSMA",
|
||||
"inputs": "close",
|
||||
"stream_total_ms": 4.5729,
|
||||
"batch_total_ms": 0.0685,
|
||||
"stream_ns_per_update": 45.73,
|
||||
"batch_ns_per_bar": 0.68,
|
||||
"updates_per_second": 21867879.95,
|
||||
"stream_over_batch_ratio": 66.7989
|
||||
},
|
||||
{
|
||||
"indicator": "StreamingEMA",
|
||||
"inputs": "close",
|
||||
"stream_total_ms": 4.535,
|
||||
"batch_total_ms": 0.1969,
|
||||
"stream_ns_per_update": 45.35,
|
||||
"batch_ns_per_bar": 1.97,
|
||||
"updates_per_second": 22050716.65,
|
||||
"stream_over_batch_ratio": 23.0301
|
||||
},
|
||||
{
|
||||
"indicator": "StreamingRSI",
|
||||
"inputs": "close",
|
||||
"stream_total_ms": 4.6421,
|
||||
"batch_total_ms": 0.4597,
|
||||
"stream_ns_per_update": 46.42,
|
||||
"batch_ns_per_bar": 4.6,
|
||||
"updates_per_second": 21541858.52,
|
||||
"stream_over_batch_ratio": 10.098
|
||||
},
|
||||
{
|
||||
"indicator": "StreamingATR",
|
||||
"inputs": "hlc",
|
||||
"stream_total_ms": 10.2098,
|
||||
"batch_total_ms": 0.4599,
|
||||
"stream_ns_per_update": 102.1,
|
||||
"batch_ns_per_bar": 4.6,
|
||||
"updates_per_second": 9794518.83,
|
||||
"stream_over_batch_ratio": 22.2012
|
||||
},
|
||||
{
|
||||
"indicator": "StreamingVWAP",
|
||||
"inputs": "hlcv",
|
||||
"stream_total_ms": 12.5109,
|
||||
"batch_total_ms": 0.1027,
|
||||
"stream_ns_per_update": 125.11,
|
||||
"batch_ns_per_bar": 1.03,
|
||||
"updates_per_second": 7993046.05,
|
||||
"stream_over_batch_ratio": 121.7603
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"metadata": {
|
||||
"suite": "wasm",
|
||||
"runtime": {
|
||||
"generated_at_utc": "2026-03-23T20:15:04.885Z",
|
||||
"node_version": "v25.8.1",
|
||||
"platform": "darwin",
|
||||
"arch": "arm64"
|
||||
},
|
||||
"dataset": {
|
||||
"bars": 100000
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"indicator": "SMA",
|
||||
"elapsed_ms": 0.1702,
|
||||
"ns_per_bar": 1.7,
|
||||
"million_bars_per_second": 587.52
|
||||
},
|
||||
{
|
||||
"indicator": "EMA",
|
||||
"elapsed_ms": 0.2923,
|
||||
"ns_per_bar": 2.92,
|
||||
"million_bars_per_second": 342.08
|
||||
},
|
||||
{
|
||||
"indicator": "RSI",
|
||||
"elapsed_ms": 0.5962,
|
||||
"ns_per_bar": 5.96,
|
||||
"million_bars_per_second": 167.73
|
||||
},
|
||||
{
|
||||
"indicator": "ATR",
|
||||
"elapsed_ms": 0.642,
|
||||
"ns_per_bar": 6.42,
|
||||
"million_bars_per_second": 155.75
|
||||
},
|
||||
{
|
||||
"indicator": "BBANDS",
|
||||
"elapsed_ms": 1.7411,
|
||||
"ns_per_bar": 17.41,
|
||||
"million_bars_per_second": 57.43
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
"""
|
||||
ferro_ta backtesting engine speed benchmark.
|
||||
|
||||
Measures throughput for single-asset, multi-asset, and analytics functions
|
||||
across multiple bar sizes. Optional competitor comparison (vectorbt, backtrader)
|
||||
is guarded behind try/except.
|
||||
|
||||
Usage:
|
||||
python benchmarks/bench_backtest.py
|
||||
python benchmarks/bench_backtest.py --sizes 10000 100000
|
||||
python benchmarks/bench_backtest.py --skip-competitors --json benchmarks/artifacts/bench_backtest_results.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from ferro_ta._ferro_ta import (
|
||||
backtest_core,
|
||||
backtest_multi_asset_core,
|
||||
backtest_ohlcv_core,
|
||||
compute_performance_metrics,
|
||||
kelly_fraction,
|
||||
monte_carlo_bootstrap,
|
||||
walk_forward_indices,
|
||||
)
|
||||
|
||||
from ferro_ta.analysis.backtest import BacktestEngine
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata
|
||||
except ModuleNotFoundError: # pragma: no cover
|
||||
from metadata import benchmark_metadata # type: ignore[no-redef]
|
||||
|
||||
# Optional competitors -------------------------------------------------------
|
||||
try:
|
||||
import vectorbt as vbt # type: ignore[import]
|
||||
|
||||
VECTORBT_AVAILABLE = True
|
||||
except ImportError:
|
||||
VECTORBT_AVAILABLE = False
|
||||
vbt = None # type: ignore[assignment]
|
||||
|
||||
try:
|
||||
import backtrader as bt # type: ignore[import]
|
||||
|
||||
BACKTRADER_AVAILABLE = True
|
||||
except ImportError:
|
||||
BACKTRADER_AVAILABLE = False
|
||||
bt = None # type: ignore[assignment]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
N_WARMUP = 1
|
||||
N_RUNS = 5
|
||||
DEFAULT_SIZES = [10_000, 100_000, 1_000_000]
|
||||
N_ASSETS = 50
|
||||
N_SIMS = 500
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Timer helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _time_fn(
|
||||
fn, *args, n_warmup: int = N_WARMUP, n_runs: int = N_RUNS, **kwargs
|
||||
) -> float:
|
||||
for _ in range(n_warmup):
|
||||
fn(*args, **kwargs)
|
||||
times: list[float] = []
|
||||
for _ in range(n_runs):
|
||||
t0 = time.perf_counter()
|
||||
fn(*args, **kwargs)
|
||||
times.append(time.perf_counter() - t0)
|
||||
return float(np.median(times))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data generators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_ohlcv(n: int, seed: int = 0) -> tuple[np.ndarray, ...]:
|
||||
rng = np.random.default_rng(seed)
|
||||
close = np.cumprod(1 + rng.standard_normal(n) * 0.01) * 100.0
|
||||
high = close + rng.uniform(0.1, 1.5, n)
|
||||
low = close - rng.uniform(0.1, 1.5, n)
|
||||
open_ = close + rng.standard_normal(n) * 0.3
|
||||
return open_, high, low, close
|
||||
|
||||
|
||||
def _make_signals(n: int, seed: int = 1) -> np.ndarray:
|
||||
rng = np.random.default_rng(seed)
|
||||
raw = np.sign(rng.standard_normal(n))
|
||||
raw[raw == 0] = 1.0
|
||||
return raw.astype(np.float64)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Benchmark functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def bench_backtest_core_single(n: int) -> dict[str, Any]:
|
||||
_, _, _, close = _make_ohlcv(n)
|
||||
signals = _make_signals(n)
|
||||
|
||||
t_ferro = _time_fn(backtest_core, close, signals)
|
||||
|
||||
row: dict[str, Any] = {
|
||||
"n_bars": n,
|
||||
"ferro_ta_ms": round(t_ferro * 1000, 4),
|
||||
"ferro_ta_mbars_s": round(n / t_ferro / 1e6, 4),
|
||||
}
|
||||
|
||||
if VECTORBT_AVAILABLE:
|
||||
import pandas as pd # noqa: PLC0415
|
||||
|
||||
close_s = pd.Series(close)
|
||||
sig_s = pd.Series(signals.astype(bool))
|
||||
|
||||
def _vbt():
|
||||
pf = vbt.Portfolio.from_signals(close_s, sig_s, ~sig_s, freq="1D")
|
||||
return pf.total_return()
|
||||
|
||||
t_vbt = _time_fn(_vbt)
|
||||
row["vectorbt_ms"] = round(t_vbt * 1000, 4)
|
||||
row["speedup_vs_vectorbt"] = round(t_vbt / t_ferro, 4)
|
||||
|
||||
return row
|
||||
|
||||
|
||||
def bench_backtest_ohlcv_core(n: int) -> dict[str, Any]:
|
||||
open_, high, low, close = _make_ohlcv(n)
|
||||
signals = _make_signals(n)
|
||||
|
||||
t_ferro = _time_fn(
|
||||
backtest_ohlcv_core,
|
||||
open_,
|
||||
high,
|
||||
low,
|
||||
close,
|
||||
signals,
|
||||
fill_mode="market_open",
|
||||
stop_loss_pct=0.02,
|
||||
take_profit_pct=0.04,
|
||||
)
|
||||
|
||||
return {
|
||||
"n_bars": n,
|
||||
"ferro_ta_ms": round(t_ferro * 1000, 4),
|
||||
"ferro_ta_mbars_s": round(n / t_ferro / 1e6, 4),
|
||||
}
|
||||
|
||||
|
||||
def bench_performance_metrics(n: int) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(42)
|
||||
returns = rng.standard_normal(n) * 0.01
|
||||
equity = np.cumprod(1 + returns)
|
||||
|
||||
t_ferro = _time_fn(compute_performance_metrics, returns, equity)
|
||||
|
||||
def _numpy_sharpe():
|
||||
mean_r = np.mean(returns)
|
||||
std_r = np.std(returns, ddof=1)
|
||||
_ = mean_r / std_r * np.sqrt(252)
|
||||
rolling_max = np.maximum.accumulate(equity)
|
||||
drawdown = (equity - rolling_max) / rolling_max
|
||||
_ = float(drawdown.min())
|
||||
|
||||
t_numpy = _time_fn(_numpy_sharpe)
|
||||
|
||||
return {
|
||||
"n_bars": n,
|
||||
"ferro_ta_ms": round(t_ferro * 1000, 4),
|
||||
"numpy_partial_ms": round(t_numpy * 1000, 4),
|
||||
"speedup_vs_numpy": round(t_numpy / t_ferro, 4),
|
||||
"note": "numpy_partial only computes sharpe+max_dd (2/23 metrics)",
|
||||
}
|
||||
|
||||
|
||||
def bench_multi_asset(n: int, n_assets: int = N_ASSETS) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(7)
|
||||
close_2d = np.ascontiguousarray(
|
||||
np.cumprod(1 + rng.standard_normal((n, n_assets)) * 0.01, axis=0) * 100.0
|
||||
)
|
||||
weights_2d = np.full((n, n_assets), 1.0 / n_assets)
|
||||
|
||||
t_parallel = _time_fn(
|
||||
backtest_multi_asset_core, close_2d, weights_2d, parallel=True
|
||||
)
|
||||
t_serial = _time_fn(backtest_multi_asset_core, close_2d, weights_2d, parallel=False)
|
||||
|
||||
def _numpy_loop():
|
||||
results = []
|
||||
for j in range(n_assets):
|
||||
col = np.ascontiguousarray(close_2d[:, j])
|
||||
sig = np.ones(n)
|
||||
_, _, sr, _ = backtest_core(col, sig)
|
||||
results.append(sr)
|
||||
return np.stack(results, axis=1)
|
||||
|
||||
t_loop = _time_fn(_numpy_loop)
|
||||
|
||||
return {
|
||||
"n_bars": n,
|
||||
"n_assets": n_assets,
|
||||
"parallel_ms": round(t_parallel * 1000, 4),
|
||||
"serial_ms": round(t_serial * 1000, 4),
|
||||
"loop_ms": round(t_loop * 1000, 4),
|
||||
"parallel_speedup_vs_loop": round(t_loop / t_parallel, 4),
|
||||
"parallel_speedup_vs_serial": round(t_serial / t_parallel, 4),
|
||||
}
|
||||
|
||||
|
||||
def bench_monte_carlo(n: int, n_sims: int = N_SIMS) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(3)
|
||||
returns = rng.standard_normal(n) * 0.01
|
||||
|
||||
t_ferro = _time_fn(monte_carlo_bootstrap, returns, n_sims=n_sims, seed=42)
|
||||
|
||||
def _numpy_mc():
|
||||
out = np.empty((n_sims, n))
|
||||
for i in range(n_sims):
|
||||
idx = np.random.choice(len(returns), size=len(returns), replace=True)
|
||||
out[i] = np.cumprod(1 + returns[idx])
|
||||
return out
|
||||
|
||||
t_numpy = _time_fn(_numpy_mc)
|
||||
|
||||
return {
|
||||
"n_bars": n,
|
||||
"n_sims": n_sims,
|
||||
"ferro_ta_ms": round(t_ferro * 1000, 4),
|
||||
"numpy_loop_ms": round(t_numpy * 1000, 4),
|
||||
"speedup_vs_numpy": round(t_numpy / t_ferro, 4),
|
||||
}
|
||||
|
||||
|
||||
def bench_engine_pipeline(n: int) -> dict[str, Any]:
|
||||
_, high, low, open_ = _make_ohlcv(n)
|
||||
_, _, _, close = _make_ohlcv(n, seed=10)
|
||||
|
||||
engine = (
|
||||
BacktestEngine()
|
||||
.with_commission(0.001)
|
||||
.with_slippage(5.0)
|
||||
.with_ohlcv(high=high, low=low, open_=open_)
|
||||
.with_stop_loss(0.02)
|
||||
.with_take_profit(0.04)
|
||||
)
|
||||
|
||||
t_ferro = _time_fn(engine.run, close, "sma_crossover")
|
||||
|
||||
return {
|
||||
"n_bars": n,
|
||||
"ferro_ta_ms": round(t_ferro * 1000, 4),
|
||||
"description": "Full pipeline: signals + OHLCV fill + 23 metrics + trades + drawdown",
|
||||
}
|
||||
|
||||
|
||||
def bench_walk_forward_indices(n: int) -> dict[str, Any]:
|
||||
train = max(n // 5, 100)
|
||||
test = max(n // 20, 20)
|
||||
t = _time_fn(walk_forward_indices, n, train, test)
|
||||
return {
|
||||
"n_bars": n,
|
||||
"train_bars": train,
|
||||
"test_bars": test,
|
||||
"ferro_ta_us": round(t * 1_000_000, 4),
|
||||
}
|
||||
|
||||
|
||||
def bench_kelly_fraction() -> dict[str, Any]:
|
||||
win_rates = np.linspace(0.3, 0.7, 1000)
|
||||
avg_wins = np.linspace(0.01, 0.05, 1000)
|
||||
avg_losses = np.linspace(0.005, 0.03, 1000)
|
||||
|
||||
def _loop():
|
||||
for w, a, b in zip(win_rates, avg_wins, avg_losses):
|
||||
kelly_fraction(w, a, b)
|
||||
|
||||
t = _time_fn(_loop)
|
||||
return {"n_calls": 1000, "ferro_ta_us": round(t * 1_000_000, 4)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Runner
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_all(
|
||||
sizes: list[int],
|
||||
skip_competitors: bool,
|
||||
n_assets: int,
|
||||
n_sims: int,
|
||||
) -> dict[str, Any]:
|
||||
results: dict[str, list[dict[str, Any]]] = {
|
||||
"backtest_core_single": [],
|
||||
"backtest_ohlcv_core": [],
|
||||
"performance_metrics": [],
|
||||
"multi_asset": [],
|
||||
"monte_carlo": [],
|
||||
"engine_full_pipeline": [],
|
||||
"walk_forward_indices": [],
|
||||
}
|
||||
|
||||
for n in sizes:
|
||||
print(f"\n--- {n:,} bars ---")
|
||||
|
||||
r = bench_backtest_core_single(n)
|
||||
results["backtest_core_single"].append(r)
|
||||
print(
|
||||
f" backtest_core_single: {r['ferro_ta_ms']:.2f} ms ({r['ferro_ta_mbars_s']:.2f} M bars/s)"
|
||||
)
|
||||
|
||||
r = bench_backtest_ohlcv_core(n)
|
||||
results["backtest_ohlcv_core"].append(r)
|
||||
print(
|
||||
f" backtest_ohlcv_core: {r['ferro_ta_ms']:.2f} ms ({r['ferro_ta_mbars_s']:.2f} M bars/s)"
|
||||
)
|
||||
|
||||
r = bench_performance_metrics(n)
|
||||
results["performance_metrics"].append(r)
|
||||
print(
|
||||
f" performance_metrics: {r['ferro_ta_ms']:.2f} ms (numpy partial: {r['numpy_partial_ms']:.2f} ms, {r['speedup_vs_numpy']:.2f}x)"
|
||||
)
|
||||
|
||||
r = bench_multi_asset(n, n_assets)
|
||||
results["multi_asset"].append(r)
|
||||
print(
|
||||
f" multi_asset ({n_assets}): parallel={r['parallel_ms']:.1f} ms serial={r['serial_ms']:.1f} ms loop={r['loop_ms']:.1f} ms ({r['parallel_speedup_vs_loop']:.2f}x vs loop)"
|
||||
)
|
||||
|
||||
r = bench_monte_carlo(n, n_sims)
|
||||
results["monte_carlo"].append(r)
|
||||
print(
|
||||
f" monte_carlo ({n_sims} sims): {r['ferro_ta_ms']:.2f} ms (numpy: {r['numpy_loop_ms']:.2f} ms, {r['speedup_vs_numpy']:.2f}x)"
|
||||
)
|
||||
|
||||
r = bench_engine_pipeline(n)
|
||||
results["engine_full_pipeline"].append(r)
|
||||
print(f" engine_full_pipeline: {r['ferro_ta_ms']:.2f} ms")
|
||||
|
||||
r = bench_walk_forward_indices(n)
|
||||
results["walk_forward_indices"].append(r)
|
||||
print(f" walk_forward_indices: {r['ferro_ta_us']:.1f} µs")
|
||||
|
||||
kelly_row = bench_kelly_fraction()
|
||||
results["kelly_fraction"] = [kelly_row]
|
||||
print(f"\n kelly_fraction (1k calls): {kelly_row['ferro_ta_us']:.1f} µs")
|
||||
|
||||
return {
|
||||
"metadata": benchmark_metadata("backtest"),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Benchmark ferro-ta backtesting engine."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sizes",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=DEFAULT_SIZES,
|
||||
metavar="N",
|
||||
help="Bar counts to benchmark (default: 10000 100000 1000000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-competitors",
|
||||
action="store_true",
|
||||
help="Skip optional competitor benchmarks",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--assets",
|
||||
type=int,
|
||||
default=N_ASSETS,
|
||||
help="Number of assets for multi-asset benchmark",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sims",
|
||||
type=int,
|
||||
default=N_SIMS,
|
||||
help="Number of simulations for Monte Carlo benchmark",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--json", dest="json_path", help="Write JSON results to this path"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
print(
|
||||
f"ferro-ta backtest benchmark | sizes={args.sizes} | assets={args.assets} | sims={args.sims}"
|
||||
)
|
||||
print("=" * 72)
|
||||
|
||||
payload = run_all(
|
||||
sizes=args.sizes,
|
||||
skip_competitors=args.skip_competitors,
|
||||
n_assets=args.assets,
|
||||
n_sims=args.sims,
|
||||
)
|
||||
|
||||
if args.json_path:
|
||||
json_path = Path(args.json_path)
|
||||
json_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
json_path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
print(f"\nWrote JSON results to {json_path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,227 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import ferro_ta
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from metadata import benchmark_metadata
|
||||
|
||||
|
||||
def _time_fn(fn, *args, rounds: int = 5, **kwargs) -> float:
|
||||
fn(*args, **kwargs)
|
||||
times: list[float] = []
|
||||
for _ in range(rounds):
|
||||
t0 = time.perf_counter()
|
||||
fn(*args, **kwargs)
|
||||
times.append(time.perf_counter() - t0)
|
||||
return min(times)
|
||||
|
||||
|
||||
def run_batch_benchmark(
|
||||
*,
|
||||
n_samples: int = 100_000,
|
||||
n_series: int = 100,
|
||||
seed: int = 42,
|
||||
) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(seed)
|
||||
close2d = rng.uniform(100.0, 200.0, (n_samples, n_series))
|
||||
high2d = close2d + rng.uniform(0.1, 2.0, (n_samples, n_series))
|
||||
low2d = close2d - rng.uniform(0.1, 2.0, (n_samples, n_series))
|
||||
close1d = close2d[:, 0]
|
||||
high1d = high2d[:, 0]
|
||||
low1d = low2d[:, 0]
|
||||
|
||||
batch_rows: list[dict[str, Any]] = []
|
||||
grouped_rows: list[dict[str, Any]] = []
|
||||
|
||||
indicators = [
|
||||
(
|
||||
"SMA",
|
||||
lambda: ferro_ta.batch.batch_sma(close2d, timeperiod=14, parallel=True),
|
||||
lambda: ferro_ta.batch.batch_sma(close2d, timeperiod=14, parallel=False),
|
||||
lambda: [
|
||||
ferro_ta.SMA(close2d[:, j], timeperiod=14) for j in range(n_series)
|
||||
],
|
||||
),
|
||||
(
|
||||
"RSI",
|
||||
lambda: ferro_ta.batch.batch_rsi(close2d, timeperiod=14, parallel=True),
|
||||
lambda: ferro_ta.batch.batch_rsi(close2d, timeperiod=14, parallel=False),
|
||||
lambda: [
|
||||
ferro_ta.RSI(close2d[:, j], timeperiod=14) for j in range(n_series)
|
||||
],
|
||||
),
|
||||
(
|
||||
"ATR",
|
||||
lambda: ferro_ta.batch.batch_atr(
|
||||
high2d, low2d, close2d, timeperiod=14, parallel=True
|
||||
),
|
||||
lambda: ferro_ta.batch.batch_atr(
|
||||
high2d, low2d, close2d, timeperiod=14, parallel=False
|
||||
),
|
||||
lambda: [
|
||||
ferro_ta.ATR(high2d[:, j], low2d[:, j], close2d[:, j], timeperiod=14)
|
||||
for j in range(n_series)
|
||||
],
|
||||
),
|
||||
(
|
||||
"ADX",
|
||||
lambda: ferro_ta.batch.batch_adx(
|
||||
high2d, low2d, close2d, timeperiod=14, parallel=True
|
||||
),
|
||||
lambda: ferro_ta.batch.batch_adx(
|
||||
high2d, low2d, close2d, timeperiod=14, parallel=False
|
||||
),
|
||||
lambda: [
|
||||
ferro_ta.ADX(high2d[:, j], low2d[:, j], close2d[:, j], timeperiod=14)
|
||||
for j in range(n_series)
|
||||
],
|
||||
),
|
||||
]
|
||||
|
||||
for name, parallel_fn, sequential_fn, loop_fn in indicators:
|
||||
batch_parallel_s = _time_fn(parallel_fn)
|
||||
batch_sequential_s = _time_fn(sequential_fn)
|
||||
loop_s = _time_fn(loop_fn)
|
||||
batch_rows.append(
|
||||
{
|
||||
"indicator": name,
|
||||
"parallel_ms": round(batch_parallel_s * 1000, 4),
|
||||
"sequential_ms": round(batch_sequential_s * 1000, 4),
|
||||
"loop_ms": round(loop_s * 1000, 4),
|
||||
"parallel_speedup_vs_loop": round(loop_s / batch_parallel_s, 4),
|
||||
"sequential_speedup_vs_loop": round(loop_s / batch_sequential_s, 4),
|
||||
}
|
||||
)
|
||||
|
||||
grouped_cases = [
|
||||
(
|
||||
"close_bundle_3",
|
||||
lambda: ferro_ta.batch.compute_many(
|
||||
[
|
||||
("SMA", {"timeperiod": 10}),
|
||||
("EMA", {"timeperiod": 12}),
|
||||
("RSI", {"timeperiod": 14}),
|
||||
],
|
||||
close=close1d,
|
||||
),
|
||||
lambda: (
|
||||
ferro_ta.SMA(close1d, timeperiod=10),
|
||||
ferro_ta.EMA(close1d, timeperiod=12),
|
||||
ferro_ta.RSI(close1d, timeperiod=14),
|
||||
),
|
||||
),
|
||||
(
|
||||
"hlc_bundle_3",
|
||||
lambda: ferro_ta.batch.compute_many(
|
||||
[
|
||||
("ATR", {"timeperiod": 14}),
|
||||
("ADX", {"timeperiod": 14}),
|
||||
("CCI", {"timeperiod": 14}),
|
||||
],
|
||||
close=close1d,
|
||||
high=high1d,
|
||||
low=low1d,
|
||||
),
|
||||
lambda: (
|
||||
ferro_ta.ATR(high1d, low1d, close1d, timeperiod=14),
|
||||
ferro_ta.ADX(high1d, low1d, close1d, timeperiod=14),
|
||||
ferro_ta.CCI(high1d, low1d, close1d, timeperiod=14),
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
for name, grouped_fn, separate_fn in grouped_cases:
|
||||
grouped_s = _time_fn(grouped_fn)
|
||||
separate_s = _time_fn(separate_fn)
|
||||
grouped_rows.append(
|
||||
{
|
||||
"case": name,
|
||||
"grouped_ms": round(grouped_s * 1000, 4),
|
||||
"separate_ms": round(separate_s * 1000, 4),
|
||||
"speedup_vs_separate": round(separate_s / grouped_s, 4),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"metadata": benchmark_metadata(
|
||||
"batch",
|
||||
extra={
|
||||
"dataset": {
|
||||
"n_samples": n_samples,
|
||||
"n_series": n_series,
|
||||
"total_bars": n_samples * n_series,
|
||||
"seed": seed,
|
||||
}
|
||||
},
|
||||
),
|
||||
"results": batch_rows,
|
||||
"grouped_results": grouped_rows,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Benchmark batch indicator execution.")
|
||||
parser.add_argument("--samples", type=int, default=100_000)
|
||||
parser.add_argument("--series", type=int, default=100)
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
parser.add_argument("--json", dest="json_path")
|
||||
args = parser.parse_args()
|
||||
|
||||
payload = run_batch_benchmark(
|
||||
n_samples=args.samples,
|
||||
n_series=args.series,
|
||||
seed=args.seed,
|
||||
)
|
||||
|
||||
dataset = payload["metadata"]["dataset"]
|
||||
print(
|
||||
"Batch Benchmark: "
|
||||
f"{dataset['n_samples']} bars, {dataset['n_series']} series "
|
||||
f"(Total: {dataset['total_bars'] / 1e6:.1f} M bars)"
|
||||
)
|
||||
print("-" * 74)
|
||||
print(
|
||||
f"{'Indicator':<12} {'Parallel (ms)':>14} {'Sequential (ms)':>16} "
|
||||
f"{'Loop (ms)':>12} {'P speedup':>10}"
|
||||
)
|
||||
print("-" * 74)
|
||||
for row in payload["results"]:
|
||||
print(
|
||||
f"{row['indicator']:<12} {row['parallel_ms']:14.1f} "
|
||||
f"{row['sequential_ms']:16.1f} {row['loop_ms']:12.1f} "
|
||||
f"{row['parallel_speedup_vs_loop']:10.2f}x"
|
||||
)
|
||||
|
||||
if payload["grouped_results"]:
|
||||
print("\nGrouped Multi-Indicator Calls")
|
||||
print("-" * 64)
|
||||
print(
|
||||
f"{'Case':<18} {'Grouped (ms)':>14} {'Separate (ms)':>16} {'Speedup':>12}"
|
||||
)
|
||||
print("-" * 64)
|
||||
for row in payload["grouped_results"]:
|
||||
print(
|
||||
f"{row['case']:<18} {row['grouped_ms']:14.1f} "
|
||||
f"{row['separate_ms']:16.1f} {row['speedup_vs_separate']:12.2f}x"
|
||||
)
|
||||
|
||||
if args.json_path:
|
||||
json_path = Path(args.json_path)
|
||||
json_path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
print(f"\nWrote JSON results to {json_path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,105 @@
|
||||
"""
|
||||
GPU vs CPU benchmark for ferro_ta.gpu (SMA, EMA, RSI).
|
||||
|
||||
Requires:
|
||||
pip install "ferro-ta[gpu]" # or pip install torch
|
||||
|
||||
Run:
|
||||
python benchmarks/bench_gpu.py
|
||||
|
||||
The script compares wall-clock time for 1M-element arrays and prints a
|
||||
summary table. If PyTorch is not installed or no GPU is found, GPU columns are skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
# Try to import PyTorch
|
||||
try:
|
||||
import torch
|
||||
|
||||
TORCH_AVAILABLE = True
|
||||
if torch.cuda.is_available():
|
||||
DEVICE = "cuda"
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
DEVICE = "mps"
|
||||
else:
|
||||
DEVICE = None
|
||||
except ImportError:
|
||||
torch = None # type: ignore[assignment]
|
||||
TORCH_AVAILABLE = False
|
||||
DEVICE = None
|
||||
|
||||
from ferro_ta.gpu import ema, rsi, sma
|
||||
|
||||
N = 1_000_000
|
||||
REPEATS = 10
|
||||
|
||||
|
||||
def _time_fn(fn, *args, **kwargs) -> float:
|
||||
"""Return minimum wall time (seconds) over REPEATS calls."""
|
||||
times = []
|
||||
for _ in range(REPEATS):
|
||||
t0 = time.perf_counter()
|
||||
fn(*args, **kwargs)
|
||||
if DEVICE == "cuda":
|
||||
torch.cuda.synchronize()
|
||||
elif DEVICE == "mps":
|
||||
torch.mps.synchronize()
|
||||
times.append(time.perf_counter() - t0)
|
||||
return min(times)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
rng = np.random.default_rng(42)
|
||||
close_cpu = rng.uniform(100.0, 200.0, N)
|
||||
|
||||
print(f"Array size: {N:,} elements")
|
||||
print(f"Repeats: {REPEATS}")
|
||||
print(f"Device: {DEVICE if DEVICE else 'CPU'}")
|
||||
print()
|
||||
|
||||
header = f"{'Indicator':<20} {'CPU (ms)':>10}"
|
||||
if DEVICE:
|
||||
header += f" {'GPU (ms)':>10} {'Speedup':>10}"
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
|
||||
for name, fn, kwargs in [
|
||||
("sma(period=30)", sma, {"timeperiod": 30}),
|
||||
("ema(period=30)", ema, {"timeperiod": 30}),
|
||||
("rsi(period=14)", rsi, {"timeperiod": 14}),
|
||||
]:
|
||||
cpu_time = _time_fn(fn, close_cpu, **kwargs) * 1000 # ms
|
||||
|
||||
row = f"{name:<20} {cpu_time:>10.3f}"
|
||||
if DEVICE:
|
||||
dtype = torch.float32 if DEVICE == "mps" else torch.float64
|
||||
close_gpu = torch.tensor(close_cpu, dtype=dtype, device=DEVICE)
|
||||
# Warm-up
|
||||
fn(close_gpu, **kwargs)
|
||||
if DEVICE == "cuda":
|
||||
torch.cuda.synchronize()
|
||||
elif DEVICE == "mps":
|
||||
torch.mps.synchronize()
|
||||
gpu_time = _time_fn(fn, close_gpu, **kwargs) * 1000 # ms
|
||||
speedup = cpu_time / gpu_time
|
||||
row += f" {gpu_time:>10.3f} {speedup:>10.2f}×"
|
||||
print(row)
|
||||
|
||||
if not TORCH_AVAILABLE:
|
||||
print()
|
||||
print("PyTorch not available — GPU columns skipped.")
|
||||
print("Install with: pip install 'ferro_ta[gpu]'")
|
||||
elif not DEVICE:
|
||||
print()
|
||||
print(
|
||||
"PyTorch found, but no CUDA or MPS device detected — GPU columns skipped."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,157 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from metadata import benchmark_metadata
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def _run(cmd: list[str], *, cwd: Path = ROOT) -> None:
|
||||
subprocess.run(cmd, cwd=cwd, check=True)
|
||||
|
||||
|
||||
def _profile_variant(
|
||||
*,
|
||||
label: str,
|
||||
maturin_args: list[str],
|
||||
price_bars: int,
|
||||
iv_bars: int,
|
||||
window: int,
|
||||
) -> dict[str, Any]:
|
||||
_run([sys.executable, "-m", "maturin", "develop", "--release", *maturin_args])
|
||||
with tempfile.TemporaryDirectory(prefix=f"ferro_ta_{label}_") as tmp_dir:
|
||||
json_path = Path(tmp_dir) / "runtime_hotspots.json"
|
||||
_run(
|
||||
[
|
||||
sys.executable,
|
||||
"benchmarks/profile_runtime_hotspots.py",
|
||||
"--price-bars",
|
||||
str(price_bars),
|
||||
"--iv-bars",
|
||||
str(iv_bars),
|
||||
"--window",
|
||||
str(window),
|
||||
"--json",
|
||||
str(json_path),
|
||||
]
|
||||
)
|
||||
payload = json.loads(json_path.read_text(encoding="utf-8"))
|
||||
return payload
|
||||
|
||||
|
||||
def run_simd_benchmark(
|
||||
*,
|
||||
price_bars: int = 20_000,
|
||||
iv_bars: int = 50_000,
|
||||
window: int = 252,
|
||||
) -> dict[str, Any]:
|
||||
# `simd` is a default feature, so a pure-scalar baseline must explicitly
|
||||
# opt out via --no-default-features; otherwise both builds would be
|
||||
# identical and every reported speedup would collapse to 1.0.
|
||||
variants = [
|
||||
("portable_release", ["--no-default-features"]),
|
||||
("simd_release", ["--features", "simd"]),
|
||||
]
|
||||
reports = {
|
||||
label: _profile_variant(
|
||||
label=label,
|
||||
maturin_args=args,
|
||||
price_bars=price_bars,
|
||||
iv_bars=iv_bars,
|
||||
window=window,
|
||||
)
|
||||
for label, args in variants
|
||||
}
|
||||
|
||||
portable_rows = {row["name"]: row for row in reports["portable_release"]["results"]}
|
||||
simd_rows = {row["name"]: row for row in reports["simd_release"]["results"]}
|
||||
|
||||
comparison: list[dict[str, Any]] = []
|
||||
for name in sorted(portable_rows):
|
||||
portable = portable_rows[name]
|
||||
simd = simd_rows.get(name)
|
||||
if simd is None:
|
||||
continue
|
||||
portable_ms = float(portable["fast_ms"])
|
||||
simd_ms = float(simd["fast_ms"])
|
||||
comparison.append(
|
||||
{
|
||||
"name": name,
|
||||
"category": portable["category"],
|
||||
"portable_ms": round(portable_ms, 4),
|
||||
"simd_ms": round(simd_ms, 4),
|
||||
"speedup_simd_vs_portable": round(
|
||||
portable_ms / simd_ms if simd_ms > 0.0 else float("inf"), 4
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
comparison.sort(
|
||||
key=lambda row: float(row["speedup_simd_vs_portable"]), reverse=True
|
||||
)
|
||||
|
||||
# Restore the default portable editable build so the workspace ends in the
|
||||
# distributable configuration.
|
||||
_run([sys.executable, "-m", "maturin", "develop", "--release"])
|
||||
|
||||
return {
|
||||
"metadata": benchmark_metadata(
|
||||
"simd",
|
||||
extra={
|
||||
"dataset": {
|
||||
"price_bars": price_bars,
|
||||
"iv_bars": iv_bars,
|
||||
"window": window,
|
||||
},
|
||||
"variants": [label for label, _ in variants],
|
||||
},
|
||||
),
|
||||
"results": comparison,
|
||||
"reports": reports,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Benchmark portable vs SIMD-enabled ferro-ta builds."
|
||||
)
|
||||
parser.add_argument("--price-bars", type=int, default=20_000)
|
||||
parser.add_argument("--iv-bars", type=int, default=50_000)
|
||||
parser.add_argument("--window", type=int, default=252)
|
||||
parser.add_argument("--json", dest="json_path")
|
||||
args = parser.parse_args()
|
||||
|
||||
payload = run_simd_benchmark(
|
||||
price_bars=args.price_bars,
|
||||
iv_bars=args.iv_bars,
|
||||
window=args.window,
|
||||
)
|
||||
|
||||
print(f"{'Case':<20} {'Portable (ms)':>14} {'SIMD (ms)':>12} {'SIMD speedup':>14}")
|
||||
print("-" * 64)
|
||||
for row in payload["results"]:
|
||||
print(
|
||||
f"{row['name']:<20} {row['portable_ms']:14.4f} "
|
||||
f"{row['simd_ms']:12.4f} {row['speedup_simd_vs_portable']:14.2f}x"
|
||||
)
|
||||
|
||||
if args.json_path:
|
||||
path = Path(args.json_path)
|
||||
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
print(f"\nWrote JSON results to {path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,193 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import ferro_ta as ft
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from metadata import benchmark_metadata
|
||||
|
||||
|
||||
def _time_min(fn: Callable[[], object], rounds: int = 5) -> float:
|
||||
fn()
|
||||
samples: list[float] = []
|
||||
for _ in range(rounds):
|
||||
t0 = time.perf_counter()
|
||||
fn()
|
||||
samples.append(time.perf_counter() - t0)
|
||||
return min(samples)
|
||||
|
||||
|
||||
def _stream_close(close: np.ndarray, factory: Callable[[], Any]) -> float:
|
||||
streamer = factory()
|
||||
last = np.nan
|
||||
for value in close:
|
||||
last = streamer.update(float(value))
|
||||
return float(last) if not np.isnan(last) else np.nan
|
||||
|
||||
|
||||
def _stream_hlc(
|
||||
high: np.ndarray,
|
||||
low: np.ndarray,
|
||||
close: np.ndarray,
|
||||
factory: Callable[[], Any],
|
||||
) -> float:
|
||||
streamer = factory()
|
||||
last = np.nan
|
||||
for high_value, low_value, close_value in zip(high, low, close):
|
||||
last = streamer.update(float(high_value), float(low_value), float(close_value))
|
||||
return float(last) if not np.isnan(last) else np.nan
|
||||
|
||||
|
||||
def _stream_hlcv(
|
||||
high: np.ndarray,
|
||||
low: np.ndarray,
|
||||
close: np.ndarray,
|
||||
volume: np.ndarray,
|
||||
factory: Callable[[], Any],
|
||||
) -> float:
|
||||
streamer = factory()
|
||||
last = np.nan
|
||||
for high_value, low_value, close_value, volume_value in zip(
|
||||
high, low, close, volume
|
||||
):
|
||||
last = streamer.update(
|
||||
float(high_value),
|
||||
float(low_value),
|
||||
float(close_value),
|
||||
float(volume_value),
|
||||
)
|
||||
return float(last) if not np.isnan(last) else np.nan
|
||||
|
||||
|
||||
def run_streaming_benchmark(
|
||||
*,
|
||||
n_bars: int = 100_000,
|
||||
seed: int = 2026,
|
||||
) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(seed)
|
||||
close = 100.0 + np.cumsum(rng.normal(0.0, 1.0, n_bars)).astype(np.float64)
|
||||
high = close + rng.uniform(0.1, 2.0, n_bars)
|
||||
low = close - rng.uniform(0.1, 2.0, n_bars)
|
||||
volume = rng.uniform(1_000.0, 100_000.0, n_bars)
|
||||
|
||||
cases = [
|
||||
(
|
||||
"StreamingSMA",
|
||||
"close",
|
||||
lambda: _stream_close(close, lambda: ft.StreamingSMA(period=20)),
|
||||
lambda: ft.SMA(close, timeperiod=20),
|
||||
),
|
||||
(
|
||||
"StreamingEMA",
|
||||
"close",
|
||||
lambda: _stream_close(close, lambda: ft.StreamingEMA(period=20)),
|
||||
lambda: ft.EMA(close, timeperiod=20),
|
||||
),
|
||||
(
|
||||
"StreamingRSI",
|
||||
"close",
|
||||
lambda: _stream_close(close, lambda: ft.StreamingRSI(period=14)),
|
||||
lambda: ft.RSI(close, timeperiod=14),
|
||||
),
|
||||
(
|
||||
"StreamingATR",
|
||||
"hlc",
|
||||
lambda: _stream_hlc(
|
||||
high,
|
||||
low,
|
||||
close,
|
||||
lambda: ft.StreamingATR(period=14),
|
||||
),
|
||||
lambda: ft.ATR(high, low, close, timeperiod=14),
|
||||
),
|
||||
(
|
||||
"StreamingVWAP",
|
||||
"hlcv",
|
||||
lambda: _stream_hlcv(
|
||||
high,
|
||||
low,
|
||||
close,
|
||||
volume,
|
||||
lambda: ft.StreamingVWAP(),
|
||||
),
|
||||
lambda: ft.VWAP(high, low, close, volume),
|
||||
),
|
||||
]
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
for name, input_kind, stream_fn, batch_fn in cases:
|
||||
stream_s = _time_min(stream_fn)
|
||||
batch_s = _time_min(batch_fn)
|
||||
rows.append(
|
||||
{
|
||||
"indicator": name,
|
||||
"inputs": input_kind,
|
||||
"stream_total_ms": round(stream_s * 1000.0, 4),
|
||||
"batch_total_ms": round(batch_s * 1000.0, 4),
|
||||
"stream_ns_per_update": round(stream_s * 1e9 / n_bars, 2),
|
||||
"batch_ns_per_bar": round(batch_s * 1e9 / n_bars, 2),
|
||||
"updates_per_second": round(n_bars / stream_s, 2),
|
||||
"stream_over_batch_ratio": round(stream_s / batch_s, 4),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"metadata": benchmark_metadata(
|
||||
"streaming",
|
||||
extra={
|
||||
"dataset": {
|
||||
"n_bars": n_bars,
|
||||
"seed": seed,
|
||||
}
|
||||
},
|
||||
),
|
||||
"results": rows,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Benchmark streaming indicator execution."
|
||||
)
|
||||
parser.add_argument("--bars", type=int, default=100_000)
|
||||
parser.add_argument("--seed", type=int, default=2026)
|
||||
parser.add_argument("--json", dest="json_path")
|
||||
args = parser.parse_args()
|
||||
|
||||
payload = run_streaming_benchmark(n_bars=args.bars, seed=args.seed)
|
||||
|
||||
dataset = payload["metadata"]["dataset"]
|
||||
print(f"Streaming Benchmark: {dataset['n_bars']} bars")
|
||||
print("-" * 86)
|
||||
print(
|
||||
f"{'Indicator':<16} {'Stream (ms)':>12} {'Batch (ms)':>12} "
|
||||
f"{'ns/update':>12} {'upd/s':>12} {'ratio':>10}"
|
||||
)
|
||||
print("-" * 86)
|
||||
for row in payload["results"]:
|
||||
print(
|
||||
f"{row['indicator']:<16} {row['stream_total_ms']:12.2f} "
|
||||
f"{row['batch_total_ms']:12.2f} {row['stream_ns_per_update']:12.2f} "
|
||||
f"{row['updates_per_second']:12.1f} {row['stream_over_batch_ratio']:10.2f}"
|
||||
)
|
||||
|
||||
if args.json_path:
|
||||
path = Path(args.json_path)
|
||||
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
print(f"\nWrote JSON results to {path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,481 @@
|
||||
"""
|
||||
ferro_ta vs TA-Lib speed comparison.
|
||||
|
||||
Measures throughput (M bars/s) for both libraries on the same synthetic data
|
||||
and parameters. The output is intentionally evidence-heavy:
|
||||
|
||||
- median timings
|
||||
- per-run timing samples
|
||||
- variability stats
|
||||
- Python-tracked peak allocation snapshots
|
||||
- machine, runtime, and build metadata
|
||||
|
||||
This is meant to support a narrow claim: ferro-ta is often faster on selected
|
||||
indicators, not universally faster.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
import time
|
||||
import tracemalloc
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
import talib # noqa: F401
|
||||
|
||||
TALIB_AVAILABLE = True
|
||||
except ImportError:
|
||||
TALIB_AVAILABLE = False
|
||||
talib = None # type: ignore[assignment]
|
||||
|
||||
import ferro_ta
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata, package_versions
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from metadata import benchmark_metadata, package_versions
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
N_WARMUP = 1
|
||||
N_RUNS = 7
|
||||
DEFAULT_SIZES = [10_000, 100_000, 1_000_000]
|
||||
TIE_EPSILON = 0.05
|
||||
|
||||
_rng = np.random.default_rng(42)
|
||||
|
||||
|
||||
def _median(values: list[float]) -> float:
|
||||
ordered = sorted(values)
|
||||
mid = len(ordered) // 2
|
||||
if len(ordered) % 2:
|
||||
return ordered[mid]
|
||||
return (ordered[mid - 1] + ordered[mid]) / 2.0
|
||||
|
||||
|
||||
def _summary_stats(samples_ms: list[float]) -> dict[str, float]:
|
||||
if not samples_ms:
|
||||
return {
|
||||
"median_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"min_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"stddev_ms": 0.0,
|
||||
"cv_pct": 0.0,
|
||||
}
|
||||
|
||||
mean_ms = sum(samples_ms) / len(samples_ms)
|
||||
variance = (
|
||||
sum((sample - mean_ms) ** 2 for sample in samples_ms) / (len(samples_ms) - 1)
|
||||
if len(samples_ms) > 1
|
||||
else 0.0
|
||||
)
|
||||
stddev_ms = math.sqrt(variance)
|
||||
cv_pct = (stddev_ms / mean_ms * 100.0) if mean_ms else 0.0
|
||||
return {
|
||||
"median_ms": round(_median(samples_ms), 4),
|
||||
"mean_ms": round(mean_ms, 4),
|
||||
"min_ms": round(min(samples_ms), 4),
|
||||
"max_ms": round(max(samples_ms), 4),
|
||||
"stddev_ms": round(stddev_ms, 4),
|
||||
"cv_pct": round(cv_pct, 3),
|
||||
}
|
||||
|
||||
|
||||
def _outcome(speedup: float) -> str:
|
||||
if speedup > 1.0 + TIE_EPSILON:
|
||||
return "ferro_ta_win"
|
||||
if speedup < 1.0 - TIE_EPSILON:
|
||||
return "talib_win"
|
||||
return "tie"
|
||||
|
||||
|
||||
def _summary_for_size(results: list[dict[str, Any]], size: int) -> dict[str, Any]:
|
||||
rows = [row for row in results if row.get("size") == size and "speedup" in row]
|
||||
if not rows:
|
||||
return {"size": size, "rows": 0}
|
||||
|
||||
speedups = [float(row["speedup"]) for row in rows]
|
||||
wins = sum(1 for row in rows if row.get("outcome") == "ferro_ta_win")
|
||||
ties = sum(1 for row in rows if row.get("outcome") == "tie")
|
||||
losses = sum(1 for row in rows if row.get("outcome") == "talib_win")
|
||||
return {
|
||||
"size": size,
|
||||
"rows": len(rows),
|
||||
"wins": wins,
|
||||
"ties": ties,
|
||||
"losses": losses,
|
||||
"win_rate": round(wins / len(rows), 4),
|
||||
"non_loss_rate": round((wins + ties) / len(rows), 4),
|
||||
"median_speedup": round(_median(speedups), 4),
|
||||
"min_speedup": round(min(speedups), 4),
|
||||
"max_speedup": round(max(speedups), 4),
|
||||
"talib_wins_or_ties": [
|
||||
row["indicator"]
|
||||
for row in rows
|
||||
if row.get("outcome") in {"talib_win", "tie"}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _synthetic_ohlcv(
|
||||
n: int,
|
||||
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
# Generate OHLCV so that ta crate DataItem constraints hold: low >= 0,
|
||||
# volume >= 0, and low <= open, close <= high, high >= open.
|
||||
close = 100.0 + np.cumsum(_rng.standard_normal(n) * 0.5)
|
||||
open_ = close + _rng.standard_normal(n) * 0.2
|
||||
high = np.maximum(open_, close) + np.abs(_rng.standard_normal(n) * 0.3)
|
||||
low = np.minimum(open_, close) - np.abs(_rng.standard_normal(n) * 0.3)
|
||||
high = np.maximum(high, low)
|
||||
low = np.maximum(low, 0.0)
|
||||
high = np.maximum(high, low)
|
||||
open_ = np.clip(open_, low, high)
|
||||
close = np.clip(close, low, high)
|
||||
volume = np.abs(_rng.standard_normal(n) * 1_000_000) + 500_000
|
||||
return open_, high, low, close, volume
|
||||
|
||||
|
||||
def _timed_runs_ms(fn, *args, **kwargs) -> list[float]:
|
||||
for _ in range(N_WARMUP):
|
||||
fn(*args, **kwargs)
|
||||
|
||||
samples_ms: list[float] = []
|
||||
for _ in range(N_RUNS):
|
||||
t0 = time.perf_counter()
|
||||
fn(*args, **kwargs)
|
||||
samples_ms.append((time.perf_counter() - t0) * 1000.0)
|
||||
return samples_ms
|
||||
|
||||
|
||||
def _python_peak_bytes(fn, *args, **kwargs) -> int | None:
|
||||
try:
|
||||
tracemalloc.start()
|
||||
tracemalloc.reset_peak()
|
||||
fn(*args, **kwargs)
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
return int(peak)
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
|
||||
|
||||
def _throughput_m_bars_s(size: int, median_ms: float) -> float:
|
||||
if median_ms <= 0:
|
||||
return 0.0
|
||||
return (size / 1e6) / (median_ms / 1000.0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Benchmarked callables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_ft_sma(o, h, l, c, v, n):
|
||||
return ferro_ta.SMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_sma(o, h, l, c, v, n):
|
||||
return talib.SMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_ema(o, h, l, c, v, n):
|
||||
return ferro_ta.EMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_ema(o, h, l, c, v, n):
|
||||
return talib.EMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_rsi(o, h, l, c, v, n):
|
||||
return ferro_ta.RSI(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_rsi(o, h, l, c, v, n):
|
||||
return talib.RSI(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_bbands(o, h, l, c, v, n):
|
||||
return ferro_ta.BBANDS(c[:n], timeperiod=20, nbdevup=2.0, nbdevdn=2.0)
|
||||
|
||||
|
||||
def _run_ta_bbands(o, h, l, c, v, n):
|
||||
return talib.BBANDS(c[:n], timeperiod=20, nbdevup=2.0, nbdevdn=2.0)
|
||||
|
||||
|
||||
def _run_ft_macd(o, h, l, c, v, n):
|
||||
return ferro_ta.MACD(c[:n], fastperiod=12, slowperiod=26, signalperiod=9)
|
||||
|
||||
|
||||
def _run_ta_macd(o, h, l, c, v, n):
|
||||
return talib.MACD(c[:n], fastperiod=12, slowperiod=26, signalperiod=9)
|
||||
|
||||
|
||||
def _run_ft_atr(o, h, l, c, v, n):
|
||||
return ferro_ta.ATR(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_atr(o, h, l, c, v, n):
|
||||
return talib.ATR(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_stoch(o, h, l, c, v, n):
|
||||
return ferro_ta.STOCH(h[:n], l[:n], c[:n])
|
||||
|
||||
|
||||
def _run_ta_stoch(o, h, l, c, v, n):
|
||||
return talib.STOCH(h[:n], l[:n], c[:n])
|
||||
|
||||
|
||||
def _run_ft_adx(o, h, l, c, v, n):
|
||||
return ferro_ta.ADX(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_adx(o, h, l, c, v, n):
|
||||
return talib.ADX(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_cci(o, h, l, c, v, n):
|
||||
return ferro_ta.CCI(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_cci(o, h, l, c, v, n):
|
||||
return talib.CCI(h[:n], l[:n], c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_obv(o, h, l, c, v, n):
|
||||
return ferro_ta.OBV(c[:n], v[:n])
|
||||
|
||||
|
||||
def _run_ta_obv(o, h, l, c, v, n):
|
||||
return talib.OBV(c[:n], v[:n])
|
||||
|
||||
|
||||
def _run_ft_mfi(o, h, l, c, v, n):
|
||||
return ferro_ta.MFI(h[:n], l[:n], c[:n], v[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_mfi(o, h, l, c, v, n):
|
||||
return talib.MFI(h[:n], l[:n], c[:n], v[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ft_wma(o, h, l, c, v, n):
|
||||
return ferro_ta.WMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
def _run_ta_wma(o, h, l, c, v, n):
|
||||
return talib.WMA(c[:n], timeperiod=14)
|
||||
|
||||
|
||||
COMPARISON_CASES = [
|
||||
("SMA", _run_ft_sma, _run_ta_sma),
|
||||
("EMA", _run_ft_ema, _run_ta_ema),
|
||||
("RSI", _run_ft_rsi, _run_ta_rsi),
|
||||
("BBANDS", _run_ft_bbands, _run_ta_bbands),
|
||||
("MACD", _run_ft_macd, _run_ta_macd),
|
||||
("ATR", _run_ft_atr, _run_ta_atr),
|
||||
("STOCH", _run_ft_stoch, _run_ta_stoch),
|
||||
("ADX", _run_ft_adx, _run_ta_adx),
|
||||
("CCI", _run_ft_cci, _run_ta_cci),
|
||||
("OBV", _run_ft_obv, _run_ta_obv),
|
||||
("MFI", _run_ft_mfi, _run_ta_mfi),
|
||||
("WMA", _run_ft_wma, _run_ta_wma),
|
||||
]
|
||||
|
||||
SKIP_1M_FOR = {"STOCH", "ADX"}
|
||||
|
||||
|
||||
def run_comparison(sizes: list[int], json_path: str | None) -> list[dict[str, Any]]:
|
||||
max_size = max(sizes)
|
||||
open_, high, low, close, volume = _synthetic_ohlcv(max_size)
|
||||
results: list[dict[str, Any]] = []
|
||||
|
||||
col_label = 10
|
||||
col_size = 10
|
||||
col_ft_ms = 12
|
||||
col_ta_ms = 12
|
||||
col_speedup = 10
|
||||
col_ft_m = 12
|
||||
col_ta_m = 12
|
||||
|
||||
if not TALIB_AVAILABLE:
|
||||
print("Note: ta-lib not installed. Reporting ferro_ta timings only.")
|
||||
print(
|
||||
"Install with: pip install ta-lib (or conda install ta-lib) for comparison.\n"
|
||||
)
|
||||
|
||||
print(
|
||||
f"\nferro_ta vs TA-Lib — median of {N_RUNS} measured runs after {N_WARMUP} warmup"
|
||||
)
|
||||
print(f"Sizes: {sizes}")
|
||||
print()
|
||||
|
||||
header = (
|
||||
f"{'Indicator':<{col_label}} {'Size':<{col_size}} "
|
||||
f"{'ferro_ta(ms)':<{col_ft_ms}} {'TA-Lib(ms)':<{col_ta_ms}} "
|
||||
f"{'Speedup':<{col_speedup}} {'ferro_ta(M/s)':<{col_ft_m}} {'TA-Lib(M/s)':<{col_ta_m}}"
|
||||
)
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
|
||||
for name, ft_run, ta_run in COMPARISON_CASES:
|
||||
for size in sizes:
|
||||
if size == 1_000_000 and name in SKIP_1M_FOR:
|
||||
continue
|
||||
|
||||
ft_samples_ms = _timed_runs_ms(
|
||||
ft_run, open_, high, low, close, volume, size
|
||||
)
|
||||
ft_stats = _summary_stats(ft_samples_ms)
|
||||
ft_median_ms = float(ft_stats["median_ms"])
|
||||
ft_m_bars_s = _throughput_m_bars_s(size, ft_median_ms)
|
||||
ft_peak_bytes = _python_peak_bytes(
|
||||
ft_run, open_, high, low, close, volume, size
|
||||
)
|
||||
|
||||
row: dict[str, Any] = {
|
||||
"indicator": name,
|
||||
"size": size,
|
||||
"input_layout": {
|
||||
"dtype": "float64",
|
||||
"contiguous": True,
|
||||
},
|
||||
"ferro_ta_ms": round(ft_median_ms, 4),
|
||||
"ferro_ta_m_bars_s": round(ft_m_bars_s, 2),
|
||||
"ferro_ta_runs_ms": [round(sample, 4) for sample in ft_samples_ms],
|
||||
"ferro_ta_stats": ft_stats,
|
||||
"python_peak_allocation_bytes": {
|
||||
"ferro_ta": ft_peak_bytes,
|
||||
},
|
||||
}
|
||||
|
||||
if TALIB_AVAILABLE:
|
||||
ta_samples_ms = _timed_runs_ms(
|
||||
ta_run, open_, high, low, close, volume, size
|
||||
)
|
||||
ta_stats = _summary_stats(ta_samples_ms)
|
||||
ta_median_ms = float(ta_stats["median_ms"])
|
||||
ta_m_bars_s = _throughput_m_bars_s(size, ta_median_ms)
|
||||
speedup = (
|
||||
ta_median_ms / ft_median_ms if ft_median_ms > 0 else float("inf")
|
||||
)
|
||||
outcome = _outcome(speedup)
|
||||
ta_peak_bytes = _python_peak_bytes(
|
||||
ta_run, open_, high, low, close, volume, size
|
||||
)
|
||||
|
||||
print(
|
||||
f"{name:<{col_label}} {size:<{col_size}} "
|
||||
f"{ft_median_ms:<{col_ft_ms}.3f} {ta_median_ms:<{col_ta_ms}.3f} "
|
||||
f"{speedup:<{col_speedup}.2f}x {ft_m_bars_s:<{col_ft_m}.1f} {ta_m_bars_s:<{col_ta_m}.1f}"
|
||||
)
|
||||
|
||||
row.update(
|
||||
{
|
||||
"talib_ms": round(ta_median_ms, 4),
|
||||
"talib_m_bars_s": round(ta_m_bars_s, 2),
|
||||
"talib_runs_ms": [round(sample, 4) for sample in ta_samples_ms],
|
||||
"talib_stats": ta_stats,
|
||||
"speedup": round(speedup, 4),
|
||||
"outcome": outcome,
|
||||
}
|
||||
)
|
||||
row["python_peak_allocation_bytes"]["talib"] = ta_peak_bytes
|
||||
else:
|
||||
print(
|
||||
f"{name:<{col_label}} {size:<{col_size}} "
|
||||
f"{ft_median_ms:<{col_ft_ms}.3f} {'N/A':<{col_ta_ms}} "
|
||||
f"{'N/A':<{col_speedup}} {ft_m_bars_s:<{col_ft_m}.1f} {'N/A':<{col_ta_m}}"
|
||||
)
|
||||
|
||||
results.append(row)
|
||||
|
||||
print()
|
||||
if TALIB_AVAILABLE and results:
|
||||
wins = sum(1 for row in results if row.get("outcome") == "ferro_ta_win")
|
||||
total = len([row for row in results if "speedup" in row])
|
||||
print(f"Summary: ferro_ta ahead outside the tie band on {wins}/{total} rows.")
|
||||
print()
|
||||
|
||||
if json_path:
|
||||
metadata = benchmark_metadata(
|
||||
"benchmark_vs_talib",
|
||||
extra={
|
||||
"dataset": {
|
||||
"generator": "synthetic_ohlcv",
|
||||
"sizes": sizes,
|
||||
"dtype": "float64",
|
||||
"array_layout": "C-contiguous",
|
||||
"seed": 42,
|
||||
},
|
||||
"methodology": {
|
||||
"warmup_runs": N_WARMUP,
|
||||
"measured_runs": N_RUNS,
|
||||
"reported_metric": "median_ms",
|
||||
"speedup_definition": "talib_median_ms / ferro_ta_median_ms",
|
||||
"tie_band": f"{1.0 - TIE_EPSILON:.2f} to {1.0 + TIE_EPSILON:.2f}",
|
||||
"input_layout_notes": (
|
||||
"Benchmarks use contiguous float64 arrays. If your workload "
|
||||
"passes non-contiguous arrays or other dtypes, benchmark that "
|
||||
"separately because wrapper overhead can dominate."
|
||||
),
|
||||
"allocation_notes": (
|
||||
"python_peak_allocation_bytes is a tracemalloc snapshot of "
|
||||
"Python-tracked allocations only; it is not a full native RSS "
|
||||
"or allocator profile."
|
||||
),
|
||||
},
|
||||
"packages": package_versions("numpy", "ferro-ta", "TA-Lib"),
|
||||
},
|
||||
)
|
||||
out = {
|
||||
"schema_version": 2,
|
||||
"command": " ".join(["python", *sys.argv]),
|
||||
"n_warmup": N_WARMUP,
|
||||
"n_runs": N_RUNS,
|
||||
"sizes": sizes,
|
||||
"talib_available": TALIB_AVAILABLE,
|
||||
"runtime": metadata["runtime"],
|
||||
"git": metadata["git"],
|
||||
"metadata": metadata,
|
||||
"summary": {
|
||||
"total_rows": len(results),
|
||||
"by_size": [_summary_for_size(results, size) for size in sizes],
|
||||
},
|
||||
"results": results,
|
||||
}
|
||||
if not TALIB_AVAILABLE:
|
||||
out["note"] = "ferro_ta only; ta-lib not installed"
|
||||
with open(json_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(out, handle, indent=2)
|
||||
print(f"Results written to {json_path}")
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="ferro_ta vs TA-Lib speed comparison")
|
||||
parser.add_argument("--json", default=None, help="Write results to JSON file")
|
||||
parser.add_argument(
|
||||
"--sizes",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=DEFAULT_SIZES,
|
||||
help="Bar counts to benchmark (default: 10000 100000 1000000)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
run_comparison(args.sizes, args.json)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate the Speed Comparison markdown table from benchmarks/results.json.
|
||||
|
||||
Requires results from the full suite:
|
||||
pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json -v
|
||||
|
||||
Reads results.json and prints a markdown table: all indicators × all libraries.
|
||||
Unsupported (indicator, library) pairs show N/A. Supported pairs missing benchmark
|
||||
data show ERR (indicating the benchmark run was incomplete or failed).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure project root is on path when run as script
|
||||
_root = Path(__file__).resolve().parent.parent
|
||||
if _root not in (Path(p).resolve() for p in sys.path):
|
||||
sys.path.insert(0, str(_root))
|
||||
|
||||
from benchmarks.wrapper_registry import (
|
||||
INDICATOR_CATEGORIES,
|
||||
is_supported,
|
||||
)
|
||||
from benchmarks.wrapper_registry import (
|
||||
LIBRARY_NAMES as LIBS,
|
||||
)
|
||||
|
||||
|
||||
def _all_indicators() -> list[str]:
|
||||
"""All indicators in category order (matches test_speed parametrization)."""
|
||||
return [ind for cat in INDICATOR_CATEGORIES for ind in INDICATOR_CATEGORIES[cat]]
|
||||
|
||||
|
||||
def main():
|
||||
p = Path(__file__).parent / "results.json"
|
||||
if not p.exists():
|
||||
print(
|
||||
"Run: pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json -v",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
raw = p.read_text().strip()
|
||||
if not raw:
|
||||
print(
|
||||
"results.json is empty. Run the full benchmark suite first.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Invalid JSON in results.json: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
benchmarks = data.get("benchmarks", [])
|
||||
|
||||
# Collect test_speed[Category/Indicator/library] -> median µs
|
||||
table: dict[str, dict[str, float]] = {}
|
||||
for b in benchmarks:
|
||||
name = b.get("name") or ""
|
||||
if "test_speed[" not in name:
|
||||
continue
|
||||
params = b.get("params") or {}
|
||||
ind = params.get("indicator")
|
||||
lib = params.get("library")
|
||||
if not ind or not lib or lib not in LIBS:
|
||||
continue
|
||||
median_sec = (b.get("stats") or {}).get("median")
|
||||
if median_sec is None:
|
||||
continue
|
||||
if ind not in table:
|
||||
table[ind] = {}
|
||||
table[ind][lib] = median_sec * 1e6 # to µs
|
||||
|
||||
all_indicators = _all_indicators()
|
||||
if not all_indicators:
|
||||
print("No indicators from INDICATOR_CATEGORIES.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Header: Indicator | ferro_ta | talib | ...
|
||||
lib_header = " | ".join(LIBS)
|
||||
print(f"| Indicator | {lib_header} |")
|
||||
print("|-----------|" + "|".join(["--------:" for _ in LIBS]) + "|")
|
||||
|
||||
for ind in all_indicators:
|
||||
row = table.get(ind, {})
|
||||
cells = []
|
||||
for lib in LIBS:
|
||||
if lib in row:
|
||||
cells.append(str(round(row[lib])))
|
||||
elif not is_supported(lib, ind):
|
||||
cells.append("N/A")
|
||||
else:
|
||||
cells.append("ERR")
|
||||
print(f"| {ind} | {' | '.join(cells)} |")
|
||||
|
||||
print()
|
||||
print(
|
||||
"(Median time in µs, lower is better. N/A = unsupported pair. "
|
||||
"ERR = supported pair missing benchmark data. Source: results.json from full test_speed run.)"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Validate hotspot benchmark JSON against conservative speedup floors.
|
||||
|
||||
This gate is intentionally lightweight: it checks that the optimized paths
|
||||
remain faster than their bundled reference implementations and that all
|
||||
expected cases were present in the report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _parse_threshold_items(items: list[str]) -> dict[str, float]:
|
||||
thresholds: dict[str, float] = {}
|
||||
for item in items:
|
||||
if "=" not in item:
|
||||
raise ValueError(f"Invalid threshold '{item}', expected NAME=VALUE")
|
||||
name, value_s = item.split("=", 1)
|
||||
thresholds[name] = float(value_s)
|
||||
return thresholds
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check hotspot benchmark JSON against regression thresholds."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
default="runtime_hotspots.json",
|
||||
help="Path to JSON produced by benchmarks/profile_runtime_hotspots.py",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--min-speedup",
|
||||
action="append",
|
||||
default=[
|
||||
"CORREL=2.0",
|
||||
"BETA=2.0",
|
||||
"LINEARREG=2.0",
|
||||
"TSF=2.0",
|
||||
"iv_rank=1.1",
|
||||
"iv_percentile=1.1",
|
||||
"iv_zscore=1.05",
|
||||
"compute_many_close=0.85",
|
||||
"feature_matrix=0.40",
|
||||
],
|
||||
help="Required minimum speedup per named case, e.g. CORREL=5.0 (repeatable)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--min-cases",
|
||||
type=int,
|
||||
default=9,
|
||||
help="Minimum number of benchmark rows expected in the report",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
path = Path(args.input)
|
||||
if not path.exists():
|
||||
print(f"ERROR: hotspot benchmark file not found: {path}")
|
||||
return 1
|
||||
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
rows = payload.get("results", [])
|
||||
if len(rows) < args.min_cases:
|
||||
print(
|
||||
f"ERROR: hotspot report contains {len(rows)} rows, expected at least {args.min_cases}"
|
||||
)
|
||||
return 1
|
||||
|
||||
thresholds = _parse_threshold_items(args.min_speedup)
|
||||
rows_by_name = {str(row.get("name")): row for row in rows}
|
||||
failures: list[str] = []
|
||||
|
||||
for name, floor in thresholds.items():
|
||||
row = rows_by_name.get(name)
|
||||
if row is None:
|
||||
failures.append(f"missing row for {name}")
|
||||
continue
|
||||
|
||||
speedup = float(row.get("speedup_vs_reference", 0.0))
|
||||
fast_ms = float(row.get("fast_ms", 0.0))
|
||||
reference_ms = float(row.get("reference_ms", 0.0))
|
||||
print(
|
||||
f"{name}: fast_ms={fast_ms:.4f}, reference_ms={reference_ms:.4f}, "
|
||||
f"speedup={speedup:.4f}"
|
||||
)
|
||||
|
||||
if fast_ms <= 0.0 or reference_ms <= 0.0:
|
||||
failures.append(f"{name} has non-positive timing values")
|
||||
if speedup < floor:
|
||||
failures.append(f"{name} speedup {speedup:.4f} < floor {floor:.4f}")
|
||||
|
||||
if failures:
|
||||
print("FAILED hotspot regression policy:")
|
||||
for failure in failures:
|
||||
print(f" - {failure}")
|
||||
return 1
|
||||
|
||||
print("PASS hotspot regression policy.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Validate benchmark-vs-TA-Lib results against guardrail thresholds.
|
||||
|
||||
This is intentionally conservative: it catches severe regressions and incomplete
|
||||
benchmark outputs, without overfitting to one machine.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _parse_threshold_items(items: list[str]) -> dict[int, float]:
|
||||
thresholds: dict[int, float] = {}
|
||||
for item in items:
|
||||
if "=" not in item:
|
||||
raise ValueError(f"Invalid threshold '{item}', expected SIZE=VALUE")
|
||||
size_s, value_s = item.split("=", 1)
|
||||
thresholds[int(size_s)] = float(value_s)
|
||||
return thresholds
|
||||
|
||||
|
||||
def _percentile(values: list[float], q: float) -> float:
|
||||
"""Return the q percentile using linear interpolation."""
|
||||
if not values:
|
||||
raise ValueError("Cannot compute percentile of empty sequence")
|
||||
if q <= 0:
|
||||
return min(values)
|
||||
if q >= 100:
|
||||
return max(values)
|
||||
|
||||
values = sorted(values)
|
||||
rank = (len(values) - 1) * (q / 100.0)
|
||||
lower = int(rank)
|
||||
upper = min(lower + 1, len(values) - 1)
|
||||
weight = rank - lower
|
||||
return values[lower] * (1.0 - weight) + values[upper] * weight
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check TA-Lib benchmark JSON against regression thresholds."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
default="benchmark_vs_talib.json",
|
||||
help="Path to benchmark JSON produced by benchmarks/bench_vs_talib.py",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--min-rows",
|
||||
type=int,
|
||||
default=10,
|
||||
help="Minimum benchmark rows required per size",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--median-floor",
|
||||
action="append",
|
||||
default=["10000=0.35", "100000=0.35"],
|
||||
help="Required minimum median speedup per size, e.g. 100000=0.5 (repeatable)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--min-speedup-floor",
|
||||
action="append",
|
||||
default=["10000=0.10", "100000=0.10"],
|
||||
help="Hard minimum per-row speedup floor per size, e.g. 100000=0.1 (repeatable)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tail-percentile",
|
||||
type=float,
|
||||
default=10.0,
|
||||
help="Tail percentile used for distribution-based slowdown checks (default: 10)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tail-speedup-floor",
|
||||
action="append",
|
||||
default=["10000=0.20", "100000=0.20"],
|
||||
help="Required minimum tail percentile speedup per size, e.g. 100000=0.2 (repeatable)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
path = Path(args.input)
|
||||
if not path.exists():
|
||||
print(f"ERROR: benchmark file not found: {path}")
|
||||
return 1
|
||||
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not data.get("talib_available", False):
|
||||
print(
|
||||
"ERROR: TA-Lib was not available; cannot enforce TA-Lib regression policy."
|
||||
)
|
||||
return 1
|
||||
|
||||
summary_by_size = {
|
||||
int(entry.get("size")): entry
|
||||
for entry in data.get("summary", {}).get("by_size", [])
|
||||
if entry.get("size") is not None
|
||||
}
|
||||
results_by_size: dict[int, list[dict[str, object]]] = {}
|
||||
for row in data.get("results", []):
|
||||
if "speedup" not in row or row.get("size") is None:
|
||||
continue
|
||||
size = int(row["size"])
|
||||
results_by_size.setdefault(size, []).append(row)
|
||||
|
||||
median_floor = _parse_threshold_items(args.median_floor)
|
||||
min_speedup_floor = _parse_threshold_items(args.min_speedup_floor)
|
||||
tail_speedup_floor = _parse_threshold_items(args.tail_speedup_floor)
|
||||
required_sizes = sorted(
|
||||
set(median_floor) | set(min_speedup_floor) | set(tail_speedup_floor)
|
||||
)
|
||||
|
||||
failures: list[str] = []
|
||||
for size in required_sizes:
|
||||
entry = summary_by_size.get(size)
|
||||
if entry is None:
|
||||
failures.append(f"missing summary for size={size}")
|
||||
continue
|
||||
rows_for_size = results_by_size.get(size, [])
|
||||
if not rows_for_size:
|
||||
failures.append(f"missing detailed rows for size={size}")
|
||||
continue
|
||||
|
||||
rows = int(entry.get("rows", 0))
|
||||
med = float(entry.get("median_speedup", 0.0))
|
||||
min_s = float(entry.get("min_speedup", 0.0))
|
||||
speedups = [float(row["speedup"]) for row in rows_for_size]
|
||||
tail_s = _percentile(speedups, args.tail_percentile)
|
||||
print(
|
||||
"size="
|
||||
f"{size}: rows={rows}, median_speedup={med:.4f}, "
|
||||
f"p{args.tail_percentile:g}_speedup={tail_s:.4f}, min_speedup={min_s:.4f}"
|
||||
)
|
||||
|
||||
if rows < args.min_rows:
|
||||
failures.append(f"size={size} rows {rows} < min_rows {args.min_rows}")
|
||||
if med < median_floor.get(size, float("-inf")):
|
||||
failures.append(
|
||||
f"size={size} median_speedup {med:.4f} < floor {median_floor[size]:.4f}"
|
||||
)
|
||||
if tail_s < tail_speedup_floor.get(size, float("-inf")):
|
||||
failures.append(
|
||||
"size="
|
||||
f"{size} p{args.tail_percentile:g}_speedup {tail_s:.4f} "
|
||||
f"< floor {tail_speedup_floor[size]:.4f}"
|
||||
)
|
||||
if min_s < min_speedup_floor.get(size, float("-inf")):
|
||||
failures.append(
|
||||
f"size={size} min_speedup {min_s:.4f} < floor {min_speedup_floor[size]:.4f}"
|
||||
)
|
||||
|
||||
if failures:
|
||||
print("FAILED benchmark regression policy:")
|
||||
for failure in failures:
|
||||
print(f" - {failure}")
|
||||
return 1
|
||||
|
||||
print("PASS benchmark regression policy.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,68 @@
|
||||
"""
|
||||
Benchmark data generator for cross-library comparison.
|
||||
|
||||
Produces C-contiguous float64 NumPy arrays that work correctly with all
|
||||
six libraries (ferro-ta, TA-Lib, pandas-ta, ta, Tulipy, finta).
|
||||
Critical: every array is np.ascontiguousarray(..., dtype=np.float64) to
|
||||
prevent memory segmentation faults in C-extension libraries.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
_RNG = np.random.default_rng(42)
|
||||
|
||||
|
||||
def generate_ohlcv(size: int = 10_000) -> dict[str, np.ndarray]:
|
||||
"""Return a dict of C-contiguous float64 OHLCV arrays.
|
||||
|
||||
Uses a geometric Brownian motion walk so values are realistic (no
|
||||
negatives, bounded intraday spread). Every array satisfies:
|
||||
high >= close >= low > 0
|
||||
open > 0
|
||||
volume > 0
|
||||
"""
|
||||
# Geometric random walk for close
|
||||
returns = _RNG.normal(0.0002, 0.01, size)
|
||||
close = 100.0 * np.exp(np.cumsum(returns))
|
||||
|
||||
noise_hi = np.abs(_RNG.normal(0, 0.005, size)) * close
|
||||
noise_lo = np.abs(_RNG.normal(0, 0.005, size)) * close
|
||||
|
||||
high = close + noise_hi
|
||||
low = np.maximum(close - noise_lo, 0.01) # never negative
|
||||
open_ = low + _RNG.random(size) * (high - low)
|
||||
volume = _RNG.uniform(1e5, 1e7, size)
|
||||
|
||||
def _c(arr: np.ndarray) -> np.ndarray:
|
||||
return np.ascontiguousarray(arr, dtype=np.float64)
|
||||
|
||||
return {
|
||||
"open": _c(open_),
|
||||
"high": _c(high),
|
||||
"low": _c(low),
|
||||
"close": _c(close),
|
||||
"volume": _c(volume),
|
||||
}
|
||||
|
||||
|
||||
def get_pandas_ohlcv(data: dict[str, np.ndarray]) -> pd.DataFrame:
|
||||
"""Convert an OHLCV dict to a DataFrame with a DatetimeIndex.
|
||||
|
||||
pandas-ta and finta both require a datetime-indexed DataFrame with
|
||||
lowercase column names (open/high/low/close/volume).
|
||||
"""
|
||||
idx = pd.date_range("2015-01-01", periods=len(data["close"]), freq="D")
|
||||
return pd.DataFrame(data, index=idx)
|
||||
|
||||
|
||||
# Pre-built datasets at several scales so benchmarks can import them directly
|
||||
SMALL = generate_ohlcv(1_000)
|
||||
MEDIUM = generate_ohlcv(10_000)
|
||||
LARGE = generate_ohlcv(100_000)
|
||||
|
||||
SMALL_DF = get_pandas_ohlcv(SMALL)
|
||||
MEDIUM_DF = get_pandas_ohlcv(MEDIUM)
|
||||
LARGE_DF = get_pandas_ohlcv(LARGE)
|
||||
Binary file not shown.
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the canonical OHLCV benchmark fixture.
|
||||
|
||||
This script creates benchmarks/fixtures/canonical_ohlcv.npz — a fixed,
|
||||
deterministic dataset used by the benchmark suite for both numerical-regression
|
||||
and performance tests.
|
||||
|
||||
Run once (or when you want to regenerate):
|
||||
python benchmarks/fixtures/generate_canonical.py
|
||||
|
||||
The fixture is checked into the repository so that CI does not need to
|
||||
regenerate it every run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
|
||||
import numpy as np
|
||||
|
||||
SEED = 20240101
|
||||
N = 2000 # number of bars
|
||||
|
||||
RNG = np.random.default_rng(SEED)
|
||||
|
||||
# Simulate a GBM-style price series
|
||||
returns = RNG.normal(0, 0.01, N)
|
||||
close = np.cumprod(1 + returns) * 100.0
|
||||
|
||||
open_ = close * RNG.uniform(0.998, 1.002, N)
|
||||
high = np.maximum(close, open_) + np.abs(RNG.normal(0, 0.2, N))
|
||||
low = np.minimum(close, open_) - np.abs(RNG.normal(0, 0.2, N))
|
||||
volume = RNG.uniform(500_000, 2_000_000, N)
|
||||
|
||||
out_path = pathlib.Path(__file__).parent / "canonical_ohlcv.npz"
|
||||
np.savez_compressed(
|
||||
out_path,
|
||||
open=open_,
|
||||
high=high,
|
||||
low=low,
|
||||
close=close,
|
||||
volume=volume,
|
||||
)
|
||||
print(f"Written {out_path} (N={N}, seed={SEED})")
|
||||
@@ -0,0 +1,192 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from importlib import metadata as importlib_metadata
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ImportError: # pragma: no cover
|
||||
try:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
except ImportError: # pragma: no cover
|
||||
tomllib = None # type: ignore[assignment]
|
||||
|
||||
|
||||
_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def _run_cmd(command: list[str]) -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
command,
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _read_toml(path: Path) -> dict[str, Any] | None:
|
||||
if tomllib is None or not path.exists():
|
||||
return None
|
||||
try:
|
||||
with path.open("rb") as handle:
|
||||
return tomllib.load(handle)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _cpu_model() -> str | None:
|
||||
if sys.platform == "darwin":
|
||||
return (
|
||||
_run_cmd(["sysctl", "-n", "machdep.cpu.brand_string"])
|
||||
or _run_cmd(["sysctl", "-n", "hw.model"])
|
||||
or platform.processor()
|
||||
or None
|
||||
)
|
||||
if sys.platform.startswith("linux"):
|
||||
cpuinfo = Path("/proc/cpuinfo")
|
||||
if cpuinfo.exists():
|
||||
text = cpuinfo.read_text(encoding="utf-8", errors="ignore")
|
||||
for pattern in (r"model name\s+:\s+(.+)", r"Hardware\s+:\s+(.+)"):
|
||||
match = re.search(pattern, text)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return platform.processor() or None
|
||||
if sys.platform.startswith("win"):
|
||||
return os.environ.get("PROCESSOR_IDENTIFIER") or platform.processor() or None
|
||||
return platform.processor() or None
|
||||
|
||||
|
||||
def _total_memory_bytes() -> int | None:
|
||||
if sys.platform == "darwin":
|
||||
raw = _run_cmd(["sysctl", "-n", "hw.memsize"])
|
||||
return int(raw) if raw and raw.isdigit() else None
|
||||
|
||||
if sys.platform.startswith("linux"):
|
||||
meminfo = Path("/proc/meminfo")
|
||||
if meminfo.exists():
|
||||
text = meminfo.read_text(encoding="utf-8", errors="ignore")
|
||||
match = re.search(r"MemTotal:\s+(\d+)\s+kB", text)
|
||||
if match:
|
||||
return int(match.group(1)) * 1024
|
||||
return None
|
||||
|
||||
if sys.platform.startswith("win"): # pragma: no cover
|
||||
try:
|
||||
import ctypes
|
||||
|
||||
class MEMORYSTATUSEX(ctypes.Structure):
|
||||
_fields_ = [
|
||||
("dwLength", ctypes.c_ulong),
|
||||
("dwMemoryLoad", ctypes.c_ulong),
|
||||
("ullTotalPhys", ctypes.c_ulonglong),
|
||||
("ullAvailPhys", ctypes.c_ulonglong),
|
||||
("ullTotalPageFile", ctypes.c_ulonglong),
|
||||
("ullAvailPageFile", ctypes.c_ulonglong),
|
||||
("ullTotalVirtual", ctypes.c_ulonglong),
|
||||
("ullAvailVirtual", ctypes.c_ulonglong),
|
||||
("ullAvailExtendedVirtual", ctypes.c_ulonglong),
|
||||
]
|
||||
|
||||
status = MEMORYSTATUSEX()
|
||||
status.dwLength = ctypes.sizeof(MEMORYSTATUSEX)
|
||||
ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(status))
|
||||
return int(status.ullTotalPhys)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _cargo_release_profile() -> dict[str, Any] | None:
|
||||
cargo_toml = _read_toml(_ROOT / "Cargo.toml")
|
||||
if not cargo_toml:
|
||||
return None
|
||||
profile = cargo_toml.get("profile", {}).get("release")
|
||||
return profile if isinstance(profile, dict) else None
|
||||
|
||||
|
||||
def git_info() -> dict[str, Any]:
|
||||
"""Best-effort git metadata for reproducible benchmark artifacts."""
|
||||
return {
|
||||
"commit": _run_cmd(["git", "rev-parse", "HEAD"]),
|
||||
"dirty": bool(_run_cmd(["git", "status", "--porcelain"]) or ""),
|
||||
"branch": _run_cmd(["git", "rev-parse", "--abbrev-ref", "HEAD"]),
|
||||
}
|
||||
|
||||
|
||||
def runtime_info() -> dict[str, Any]:
|
||||
return {
|
||||
"generated_at_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"python_version": sys.version.split()[0],
|
||||
"python_implementation": platform.python_implementation(),
|
||||
"python_executable": sys.executable,
|
||||
"platform": platform.platform(),
|
||||
"system": platform.system(),
|
||||
"release": platform.release(),
|
||||
"machine": platform.machine(),
|
||||
"processor": platform.processor() or None,
|
||||
"cpu_model": _cpu_model(),
|
||||
"cpu_count_logical": os.cpu_count(),
|
||||
"total_memory_bytes": _total_memory_bytes(),
|
||||
}
|
||||
|
||||
|
||||
def build_info() -> dict[str, Any]:
|
||||
return {
|
||||
"rustc": _run_cmd(["rustc", "-Vv"]),
|
||||
"cargo": _run_cmd(["cargo", "-VV"]) or _run_cmd(["cargo", "-V"]),
|
||||
"cargo_release_profile": _cargo_release_profile(),
|
||||
"rustflags": os.environ.get("RUSTFLAGS"),
|
||||
"cargo_build_rustflags": os.environ.get("CARGO_BUILD_RUSTFLAGS"),
|
||||
"maturin_flags": os.environ.get("MATURIN_EXTRA_ARGS"),
|
||||
}
|
||||
|
||||
|
||||
def package_versions(*names: str) -> dict[str, str | None]:
|
||||
versions: dict[str, str | None] = {}
|
||||
for name in names:
|
||||
try:
|
||||
versions[name] = importlib_metadata.version(name)
|
||||
except importlib_metadata.PackageNotFoundError:
|
||||
versions[name] = None
|
||||
return versions
|
||||
|
||||
|
||||
def file_info(path: str | Path) -> dict[str, Any]:
|
||||
file_path = Path(path)
|
||||
data = file_path.read_bytes()
|
||||
return {
|
||||
"path": str(file_path),
|
||||
"size_bytes": file_path.stat().st_size,
|
||||
"sha256": hashlib.sha256(data).hexdigest(),
|
||||
}
|
||||
|
||||
|
||||
def benchmark_metadata(
|
||||
suite: str,
|
||||
*,
|
||||
fixtures: list[str | Path] | None = None,
|
||||
extra: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
metadata: dict[str, Any] = {
|
||||
"suite": suite,
|
||||
"runtime": runtime_info(),
|
||||
"git": git_info(),
|
||||
"build": build_info(),
|
||||
"packages": package_versions("numpy", "ferro-ta"),
|
||||
}
|
||||
if fixtures:
|
||||
metadata["fixtures"] = [file_info(path) for path in fixtures]
|
||||
if extra:
|
||||
metadata.update(extra)
|
||||
return metadata
|
||||
@@ -0,0 +1,284 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import ferro_ta as ft
|
||||
from ferro_ta.analysis.features import feature_matrix
|
||||
from ferro_ta.analysis.options import iv_percentile, iv_rank, iv_zscore
|
||||
from ferro_ta.data.batch import compute_many
|
||||
|
||||
try:
|
||||
from benchmarks.metadata import benchmark_metadata
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from metadata import benchmark_metadata
|
||||
|
||||
|
||||
def _time_min(fn: Callable[[], object], rounds: int = 5) -> float:
|
||||
fn()
|
||||
samples: list[float] = []
|
||||
for _ in range(rounds):
|
||||
t0 = time.perf_counter()
|
||||
fn()
|
||||
samples.append(time.perf_counter() - t0)
|
||||
return min(samples) * 1000.0
|
||||
|
||||
|
||||
def _naive_correl(x: np.ndarray, y: np.ndarray, window: int) -> np.ndarray:
|
||||
out = np.full(len(x), np.nan, dtype=np.float64)
|
||||
for end in range(window - 1, len(x)):
|
||||
x_window = x[end + 1 - window : end + 1]
|
||||
y_window = y[end + 1 - window : end + 1]
|
||||
mean_x = float(np.sum(x_window)) / window
|
||||
mean_y = float(np.sum(y_window)) / window
|
||||
cov = float(np.sum((x_window - mean_x) * (y_window - mean_y)))
|
||||
std_x = float(np.sqrt(np.sum((x_window - mean_x) ** 2)))
|
||||
std_y = float(np.sqrt(np.sum((y_window - mean_y) ** 2)))
|
||||
denom = std_x * std_y
|
||||
out[end] = cov / denom if denom != 0.0 else np.nan
|
||||
return out
|
||||
|
||||
|
||||
def _naive_beta(x: np.ndarray, y: np.ndarray, window: int) -> np.ndarray:
|
||||
out = np.full(len(x), np.nan, dtype=np.float64)
|
||||
for end in range(window, len(x)):
|
||||
start = end - window
|
||||
rx = np.array(
|
||||
[
|
||||
x[idx + 1] / x[idx] - 1.0 if x[idx] != 0.0 else np.nan
|
||||
for idx in range(start, end)
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
ry = np.array(
|
||||
[
|
||||
y[idx + 1] / y[idx] - 1.0 if y[idx] != 0.0 else np.nan
|
||||
for idx in range(start, end)
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
mean_x = float(np.sum(rx)) / window
|
||||
mean_y = float(np.sum(ry)) / window
|
||||
cov = float(np.sum((rx - mean_x) * (ry - mean_y))) / window
|
||||
var_x = float(np.sum((rx - mean_x) ** 2)) / window
|
||||
out[end] = cov / var_x if var_x != 0.0 else np.nan
|
||||
return out
|
||||
|
||||
|
||||
def _naive_linearreg(series: np.ndarray, timeperiod: int, x_value: float) -> np.ndarray:
|
||||
out = np.full(len(series), np.nan, dtype=np.float64)
|
||||
xs = np.arange(timeperiod, dtype=np.float64)
|
||||
sum_x = float(np.sum(xs))
|
||||
sum_x2 = float(np.sum(xs * xs))
|
||||
for end in range(timeperiod - 1, len(series)):
|
||||
window = series[end + 1 - timeperiod : end + 1]
|
||||
sum_y = float(np.sum(window))
|
||||
sum_xy = float(np.sum(xs * window))
|
||||
denom = timeperiod * sum_x2 - sum_x * sum_x
|
||||
slope = (timeperiod * sum_xy - sum_x * sum_y) / denom if denom != 0.0 else 0.0
|
||||
intercept = (sum_y - slope * sum_x) / timeperiod
|
||||
out[end] = intercept + slope * x_value
|
||||
return out
|
||||
|
||||
|
||||
def _old_iv_rank(iv: np.ndarray, window: int) -> np.ndarray:
|
||||
out = np.full(len(iv), np.nan, dtype=np.float64)
|
||||
for idx in range(window - 1, len(iv)):
|
||||
win = iv[idx - window + 1 : idx + 1]
|
||||
lower = float(np.nanmin(win))
|
||||
upper = float(np.nanmax(win))
|
||||
out[idx] = 0.0 if upper == lower else (iv[idx] - lower) / (upper - lower)
|
||||
return out
|
||||
|
||||
|
||||
def _old_iv_percentile(iv: np.ndarray, window: int) -> np.ndarray:
|
||||
out = np.full(len(iv), np.nan, dtype=np.float64)
|
||||
for idx in range(window - 1, len(iv)):
|
||||
win = iv[idx - window + 1 : idx + 1]
|
||||
out[idx] = float(np.sum(win <= iv[idx])) / window
|
||||
return out
|
||||
|
||||
|
||||
def _old_iv_zscore(iv: np.ndarray, window: int) -> np.ndarray:
|
||||
out = np.full(len(iv), np.nan, dtype=np.float64)
|
||||
for idx in range(window - 1, len(iv)):
|
||||
win = iv[idx - window + 1 : idx + 1]
|
||||
mean = float(np.nanmean(win))
|
||||
std = float(np.nanstd(win, ddof=0))
|
||||
out[idx] = np.nan if std == 0.0 else (iv[idx] - mean) / std
|
||||
return out
|
||||
|
||||
|
||||
def build_hotspot_report(
|
||||
*,
|
||||
price_bars: int = 20_000,
|
||||
iv_bars: int = 50_000,
|
||||
window: int = 252,
|
||||
) -> dict[str, Any]:
|
||||
rng = np.random.default_rng(2026)
|
||||
close = 100 + np.cumsum(rng.normal(0, 1, price_bars)).astype(np.float64)
|
||||
high = close + rng.uniform(0.1, 2.0, price_bars)
|
||||
low = close - rng.uniform(0.1, 2.0, price_bars)
|
||||
iv = rng.uniform(10.0, 40.0, iv_bars).astype(np.float64)
|
||||
ohlcv = {
|
||||
"close": close,
|
||||
"high": high,
|
||||
"low": low,
|
||||
"volume": np.full(price_bars, 1000.0),
|
||||
}
|
||||
|
||||
rows = [
|
||||
(
|
||||
"rust_kernel",
|
||||
"CORREL",
|
||||
lambda: ft.CORREL(high, low, timeperiod=30),
|
||||
lambda: _naive_correl(high, low, 30),
|
||||
),
|
||||
(
|
||||
"rust_kernel",
|
||||
"BETA",
|
||||
lambda: ft.BETA(high, low, timeperiod=5),
|
||||
lambda: _naive_beta(high, low, 5),
|
||||
),
|
||||
(
|
||||
"rust_kernel",
|
||||
"LINEARREG",
|
||||
lambda: ft.LINEARREG(close, timeperiod=14),
|
||||
lambda: _naive_linearreg(close, 14, 13.0),
|
||||
),
|
||||
(
|
||||
"rust_kernel",
|
||||
"TSF",
|
||||
lambda: ft.TSF(close, timeperiod=14),
|
||||
lambda: _naive_linearreg(close, 14, 14.0),
|
||||
),
|
||||
(
|
||||
"python_analysis",
|
||||
"iv_rank",
|
||||
lambda: iv_rank(iv, window),
|
||||
lambda: _old_iv_rank(iv, window),
|
||||
),
|
||||
(
|
||||
"python_analysis",
|
||||
"iv_percentile",
|
||||
lambda: iv_percentile(iv, window),
|
||||
lambda: _old_iv_percentile(iv, window),
|
||||
),
|
||||
(
|
||||
"python_analysis",
|
||||
"iv_zscore",
|
||||
lambda: iv_zscore(iv, window),
|
||||
lambda: _old_iv_zscore(iv, window),
|
||||
),
|
||||
(
|
||||
"ffi_grouping",
|
||||
"compute_many_close",
|
||||
lambda: compute_many(
|
||||
[
|
||||
("SMA", {"timeperiod": 10}),
|
||||
("EMA", {"timeperiod": 12}),
|
||||
("RSI", {"timeperiod": 14}),
|
||||
],
|
||||
close=close,
|
||||
),
|
||||
lambda: (
|
||||
ft.SMA(close, timeperiod=10),
|
||||
ft.EMA(close, timeperiod=12),
|
||||
ft.RSI(close, timeperiod=14),
|
||||
),
|
||||
),
|
||||
(
|
||||
"ffi_grouping",
|
||||
"feature_matrix",
|
||||
lambda: feature_matrix(
|
||||
ohlcv,
|
||||
[
|
||||
("SMA", {"timeperiod": 10}),
|
||||
("ATR", {"timeperiod": 14}),
|
||||
("ADX", {"timeperiod": 14}),
|
||||
],
|
||||
),
|
||||
lambda: {
|
||||
"SMA": ft.SMA(close, timeperiod=10),
|
||||
"ATR": ft.ATR(high, low, close, timeperiod=14),
|
||||
"ADX": ft.ADX(high, low, close, timeperiod=14),
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for category, name, fast_fn, reference_fn in rows:
|
||||
fast_ms = _time_min(fast_fn)
|
||||
reference_ms = _time_min(reference_fn, rounds=1)
|
||||
results.append(
|
||||
{
|
||||
"category": category,
|
||||
"name": name,
|
||||
"fast_ms": round(fast_ms, 4),
|
||||
"reference_ms": round(reference_ms, 4),
|
||||
"speedup_vs_reference": round(reference_ms / fast_ms, 4),
|
||||
}
|
||||
)
|
||||
|
||||
results.sort(key=lambda row: row["fast_ms"], reverse=True)
|
||||
total_fast_ms = sum(float(row["fast_ms"]) for row in results) or 1.0
|
||||
for row in results:
|
||||
row["share_of_suite_pct"] = round(
|
||||
float(row["fast_ms"]) / total_fast_ms * 100.0, 2
|
||||
)
|
||||
|
||||
return {
|
||||
"metadata": benchmark_metadata(
|
||||
"runtime_hotspots",
|
||||
extra={
|
||||
"dataset": {
|
||||
"price_bars": price_bars,
|
||||
"iv_bars": iv_bars,
|
||||
"window": window,
|
||||
}
|
||||
},
|
||||
),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Profile ferro-ta runtime hotspots.")
|
||||
parser.add_argument("--price-bars", type=int, default=20_000)
|
||||
parser.add_argument("--iv-bars", type=int, default=50_000)
|
||||
parser.add_argument("--window", type=int, default=252)
|
||||
parser.add_argument("--json", dest="json_path")
|
||||
args = parser.parse_args()
|
||||
|
||||
payload = build_hotspot_report(
|
||||
price_bars=args.price_bars,
|
||||
iv_bars=args.iv_bars,
|
||||
window=args.window,
|
||||
)
|
||||
|
||||
print(
|
||||
f"{'Category':<16} {'Case':<18} {'Fast (ms)':>10} {'Ref (ms)':>10} {'Speedup':>10}"
|
||||
)
|
||||
print("-" * 70)
|
||||
for row in payload["results"]:
|
||||
print(
|
||||
f"{row['category']:<16} {row['name']:<18} {row['fast_ms']:10.2f} "
|
||||
f"{row['reference_ms']:10.2f} {row['speedup_vs_reference']:10.2f}x"
|
||||
)
|
||||
|
||||
if args.json_path:
|
||||
path = Path(args.json_path)
|
||||
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
print(f"\nWrote JSON results to {path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,210 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
from benchmarks.bench_batch import run_batch_benchmark
|
||||
from benchmarks.bench_simd import run_simd_benchmark
|
||||
from benchmarks.bench_streaming import run_streaming_benchmark
|
||||
from benchmarks.bench_vs_talib import run_comparison
|
||||
from benchmarks.metadata import benchmark_metadata, file_info
|
||||
from benchmarks.profile_runtime_hotspots import build_hotspot_report
|
||||
from benchmarks.test_benchmark_suite import (
|
||||
FIXTURE_PATH,
|
||||
INDICATOR_SUITE,
|
||||
_run_indicator,
|
||||
)
|
||||
except ModuleNotFoundError: # pragma: no cover - script execution fallback
|
||||
from bench_batch import run_batch_benchmark
|
||||
from bench_simd import run_simd_benchmark
|
||||
from bench_streaming import run_streaming_benchmark
|
||||
from bench_vs_talib import run_comparison
|
||||
from metadata import benchmark_metadata, file_info
|
||||
from profile_runtime_hotspots import build_hotspot_report
|
||||
from test_benchmark_suite import FIXTURE_PATH, INDICATOR_SUITE, _run_indicator
|
||||
|
||||
|
||||
def _time_min(fn, rounds: int = 5) -> float:
|
||||
fn()
|
||||
samples: list[float] = []
|
||||
for _ in range(rounds):
|
||||
t0 = time.perf_counter()
|
||||
fn()
|
||||
samples.append(time.perf_counter() - t0)
|
||||
return min(samples) * 1000.0
|
||||
|
||||
|
||||
def build_indicator_latency_report(*, rounds: int = 5) -> dict[str, Any]:
|
||||
if not FIXTURE_PATH.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Canonical fixture not found: {FIXTURE_PATH}. "
|
||||
"Run benchmarks/fixtures/generate_canonical.py first."
|
||||
)
|
||||
|
||||
fixture = np.load(FIXTURE_PATH)
|
||||
ohlcv = {key: fixture[key] for key in fixture.files}
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
for entry in INDICATOR_SUITE:
|
||||
elapsed_ms = _time_min(
|
||||
lambda entry=entry: _run_indicator(entry, ohlcv), rounds=rounds
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"name": entry["name"],
|
||||
"inputs": entry["inputs"],
|
||||
"kwargs": entry["kwargs"],
|
||||
"elapsed_ms": round(elapsed_ms, 4),
|
||||
}
|
||||
)
|
||||
|
||||
rows.sort(key=lambda row: float(row["elapsed_ms"]), reverse=True)
|
||||
return {
|
||||
"metadata": benchmark_metadata(
|
||||
"indicator_latency",
|
||||
fixtures=[FIXTURE_PATH],
|
||||
extra={
|
||||
"dataset": {
|
||||
"fixture": str(FIXTURE_PATH),
|
||||
"bars": len(ohlcv["close"]),
|
||||
"rounds": rounds,
|
||||
}
|
||||
},
|
||||
),
|
||||
"results": rows,
|
||||
}
|
||||
|
||||
|
||||
def _write_json(path: Path, payload: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate reproducible performance baseline artifacts."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
default="benchmarks/artifacts/latest",
|
||||
help="Directory where benchmark JSON artifacts are written",
|
||||
)
|
||||
parser.add_argument("--indicator-rounds", type=int, default=5)
|
||||
parser.add_argument("--batch-samples", type=int, default=100_000)
|
||||
parser.add_argument("--batch-series", type=int, default=100)
|
||||
parser.add_argument("--batch-seed", type=int, default=42)
|
||||
parser.add_argument("--streaming-bars", type=int, default=100_000)
|
||||
parser.add_argument("--streaming-seed", type=int, default=2026)
|
||||
parser.add_argument("--price-bars", type=int, default=20_000)
|
||||
parser.add_argument("--iv-bars", type=int, default=50_000)
|
||||
parser.add_argument("--window", type=int, default=252)
|
||||
parser.add_argument(
|
||||
"--skip-simd",
|
||||
action="store_true",
|
||||
help="Skip portable-vs-SIMD comparison",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--talib-sizes",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[10_000, 100_000],
|
||||
help="Bar counts used for the TA-Lib comparison suite",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-talib",
|
||||
action="store_true",
|
||||
help="Skip the TA-Lib comparison artifact",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
artifacts: dict[str, str] = {}
|
||||
|
||||
indicator_path = output_dir / "indicator_latency.json"
|
||||
_write_json(
|
||||
indicator_path,
|
||||
build_indicator_latency_report(rounds=args.indicator_rounds),
|
||||
)
|
||||
artifacts["indicator_latency"] = str(indicator_path)
|
||||
|
||||
batch_path = output_dir / "batch.json"
|
||||
_write_json(
|
||||
batch_path,
|
||||
run_batch_benchmark(
|
||||
n_samples=args.batch_samples,
|
||||
n_series=args.batch_series,
|
||||
seed=args.batch_seed,
|
||||
),
|
||||
)
|
||||
artifacts["batch"] = str(batch_path)
|
||||
|
||||
streaming_path = output_dir / "streaming.json"
|
||||
_write_json(
|
||||
streaming_path,
|
||||
run_streaming_benchmark(
|
||||
n_bars=args.streaming_bars,
|
||||
seed=args.streaming_seed,
|
||||
),
|
||||
)
|
||||
artifacts["streaming"] = str(streaming_path)
|
||||
|
||||
hotspot_path = output_dir / "runtime_hotspots.json"
|
||||
_write_json(
|
||||
hotspot_path,
|
||||
build_hotspot_report(
|
||||
price_bars=args.price_bars,
|
||||
iv_bars=args.iv_bars,
|
||||
window=args.window,
|
||||
),
|
||||
)
|
||||
artifacts["runtime_hotspots"] = str(hotspot_path)
|
||||
|
||||
if not args.skip_simd:
|
||||
simd_path = output_dir / "simd.json"
|
||||
_write_json(
|
||||
simd_path,
|
||||
run_simd_benchmark(
|
||||
price_bars=args.price_bars,
|
||||
iv_bars=args.iv_bars,
|
||||
window=args.window,
|
||||
),
|
||||
)
|
||||
artifacts["simd"] = str(simd_path)
|
||||
|
||||
if not args.skip_talib:
|
||||
talib_path = output_dir / "benchmark_vs_talib.json"
|
||||
run_comparison(args.talib_sizes, str(talib_path))
|
||||
artifacts["benchmark_vs_talib"] = str(talib_path)
|
||||
|
||||
wasm_path = output_dir / "wasm.json"
|
||||
if wasm_path.exists():
|
||||
artifacts["wasm"] = str(wasm_path)
|
||||
|
||||
manifest = {
|
||||
"metadata": benchmark_metadata(
|
||||
"perf_contract",
|
||||
fixtures=[FIXTURE_PATH],
|
||||
extra={"output_dir": str(output_dir)},
|
||||
),
|
||||
"artifacts": {name: file_info(path) for name, path in artifacts.items()},
|
||||
}
|
||||
manifest_path = output_dir / "manifest.json"
|
||||
_write_json(manifest_path, manifest)
|
||||
|
||||
print(f"Generated performance contract artifacts in {output_dir}")
|
||||
for name, path in artifacts.items():
|
||||
print(f" - {name}: {path}")
|
||||
print(f" - manifest: {manifest_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,200 @@
|
||||
"""
|
||||
Cross-library accuracy tests.
|
||||
|
||||
For each indicator we compare ferro_ta output against every available reference library.
|
||||
Tolerances are based on known algorithmic differences (e.g. Wilder vs SMA seed).
|
||||
We only compare the overlapping (valid) suffix of each output array.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from benchmarks.data_generator import MEDIUM
|
||||
from benchmarks.wrapper_registry import (
|
||||
BINARY_INDICATORS,
|
||||
CUMULATIVE_INDICATORS,
|
||||
INDICATOR_CATEGORIES,
|
||||
INDICATOR_NAMES,
|
||||
available_libraries,
|
||||
execute_indicator,
|
||||
is_supported,
|
||||
)
|
||||
|
||||
# Reference = ferro_ta; compare against each library that has a non-empty result.
|
||||
REFERENCE_LIB = "ferro_ta"
|
||||
COMPARISON_LIBS = [
|
||||
library for library in available_libraries() if library != REFERENCE_LIB
|
||||
]
|
||||
|
||||
# Per-indicator tolerances (rtol, atol)
|
||||
_TOLERANCES: dict[str, tuple[float, float]] = {
|
||||
"ATR": (1e-3, 0.05), # Wilder's smoothing seed differs
|
||||
"NATR": (1e-3, 0.10),
|
||||
"BBANDS": (1e-3, 0.20), # ddof=0 vs ddof=1
|
||||
"STDDEV": (1e-3, 0.20),
|
||||
"VAR": (1e-3, 0.50),
|
||||
"MACD": (1e-3, 1.00), # seed differences across libraries
|
||||
"KAMA": (1e-3, 1e-3),
|
||||
"STOCH": (1e-3, 0.10), # smoothing method differences
|
||||
"SAR": (1e-3, 0.20),
|
||||
"ADOSC": (1e-3, 0.20),
|
||||
"ADX": (1e-3, 0.50), # Wilder's ADX
|
||||
"PLUS_DI": (1e-3, 0.50),
|
||||
"MINUS_DI": (1e-3, 0.50),
|
||||
"PPO": (1e-2, 1e-3),
|
||||
"CMO": (1e-3, 0.10),
|
||||
"TRIX": (1e-3, 0.05),
|
||||
"CCI": (1e-3, 0.10),
|
||||
"SUPERTREND": (1e-2, 0.50),
|
||||
"KELTNER_CHANNELS": (1e-2, 0.50),
|
||||
"DONCHIAN": (1e-4, 1e-4),
|
||||
"HT_DCPERIOD": (1e-2, 2.0),
|
||||
"VWAP": (1e-3, 0.10),
|
||||
"AROON": (1e-4, 1e-3),
|
||||
"LINEARREG": (1e-4, 1e-4),
|
||||
"LINEARREG_SLOPE": (1e-4, 1e-4),
|
||||
"CORREL": (1e-4, 1e-3),
|
||||
"BETA": (1e-3, 1e-3),
|
||||
"TSF": (1e-4, 1e-4),
|
||||
"EMA": (1e-3, 0.30), # ta library uses different EMA seed
|
||||
"DEMA": (1e-3, 0.50),
|
||||
"TEMA": (1e-3, 0.50),
|
||||
"T3": (1e-3, 0.50),
|
||||
"HULL_MA": (1e-3, 0.10),
|
||||
"WMA": (1e-4, 1e-4),
|
||||
"TRIMA": (1e-4, 1e-4),
|
||||
}
|
||||
|
||||
_DEFAULT_TOL = (1e-4, 1e-5)
|
||||
|
||||
# Pairs that use correlation check (>=0.95) due to known algorithmic divergence
|
||||
# Format: (indicator, library) or just indicator (applies to all libs)
|
||||
_CORRELATION_PAIRS: set[tuple[str, str]] = {
|
||||
("PPO", "talib"), # different PPO formula normalization
|
||||
("PPO", "pandas_ta"),
|
||||
("PPO", "tulipy"),
|
||||
("STOCH", "ta"),
|
||||
("SUPERTREND", "pandas_ta"),
|
||||
("KELTNER_CHANNELS", "pandas_ta"),
|
||||
("KELTNER_CHANNELS", "ta"),
|
||||
("EMA", "finta"), # finta EMA uses different initialization
|
||||
("KAMA", "pandas_ta"), # pandas_ta KAMA has slightly different seed
|
||||
("RSI", "ta"), # ta uses SMA warmup vs Wilder
|
||||
("RSI", "finta"), # same
|
||||
}
|
||||
|
||||
# Pairs that are skipped because they are structurally incompatible
|
||||
_SKIP_PAIRS: set[tuple[str, str]] = {
|
||||
("BBANDS", "finta"), # finta normalizes band differently
|
||||
("ATR", "finta"), # finta ATR uses simple TR not Wilder
|
||||
("STDDEV", "finta"), # finta uses population std
|
||||
("TRIMA", "finta"), # finta TRIMA uses different formula
|
||||
("PPO", "finta"), # finta PPO scaling incompatible
|
||||
("STOCH", "finta"), # finta STOCH formula differs
|
||||
("VWAP", "pandas_ta"), # pandas_ta VWAP anchors to session start
|
||||
("HT_TRENDMODE", "talib"), # binary; Hilbert seed diverges
|
||||
("CMO", "talib"), # ferro_ta CMO smoothing variant corr < 0.90
|
||||
("CMO", "pandas_ta"),
|
||||
("CMO", "finta"),
|
||||
("PLUS_DI", "pandas_ta"), # pandas_ta ADX column naming corr < 0.70
|
||||
}
|
||||
|
||||
MIN_OVERLAP = 30 # minimum points to make comparison meaningful
|
||||
|
||||
|
||||
def _compare(ref: np.ndarray, cmp: np.ndarray, indicator: str, library: str) -> None:
|
||||
"""Assert that ref and cmp agree on their overlapping suffix."""
|
||||
if (indicator, library) in _SKIP_PAIRS:
|
||||
pytest.skip(f"Known structural incompatibility: {indicator} vs {library}")
|
||||
if len(ref) < MIN_OVERLAP or len(cmp) < MIN_OVERLAP:
|
||||
pytest.skip(f"Too few points to compare ({len(ref)} vs {len(cmp)})")
|
||||
n = min(len(ref), len(cmp))
|
||||
r = ref[-n:]
|
||||
c = cmp[-n:]
|
||||
if indicator in BINARY_INDICATORS or (indicator, library) in _CORRELATION_PAIRS:
|
||||
# Use correlation check for structurally different algorithms
|
||||
corr = np.corrcoef(r, c)[0, 1] if indicator not in BINARY_INDICATORS else None
|
||||
if indicator in BINARY_INDICATORS:
|
||||
agree = np.mean(r == c)
|
||||
assert agree >= 0.80, f"Binary agreement {agree:.1%} < 80%"
|
||||
else:
|
||||
assert corr >= 0.90, (
|
||||
f"Correlation {corr:.4f} < 0.90 (structural divergence)"
|
||||
)
|
||||
elif indicator in CUMULATIVE_INDICATORS:
|
||||
dr, dc = np.diff(r), np.diff(c)
|
||||
if len(dr) < 5 or len(dc) < 5:
|
||||
return
|
||||
corr = np.corrcoef(dr, dc)[0, 1]
|
||||
assert corr >= 0.999, f"Cumulative corr {corr:.6f} < 0.999"
|
||||
else:
|
||||
rtol, atol = _TOLERANCES.get(indicator, _DEFAULT_TOL)
|
||||
assert np.allclose(r, c, rtol=rtol, atol=atol), (
|
||||
f"max diff = {np.max(np.abs(r - c)):.6g}, "
|
||||
f"mean diff = {np.mean(np.abs(r - c)):.6g}"
|
||||
)
|
||||
|
||||
|
||||
# ── dynamically generate one test per (indicator, library) pair ─────────────
|
||||
|
||||
|
||||
def pytest_generate_tests(metafunc):
|
||||
if "indicator" in metafunc.fixturenames and "library" in metafunc.fixturenames:
|
||||
params = []
|
||||
avail = available_libraries()
|
||||
for ind in INDICATOR_NAMES:
|
||||
for lib in COMPARISON_LIBS:
|
||||
if lib in avail:
|
||||
params.append(pytest.param(ind, lib, id=f"{ind}-{lib}"))
|
||||
metafunc.parametrize("indicator,library", params)
|
||||
|
||||
|
||||
class TestAccuracy:
|
||||
"""Compare ferro_ta vs every other library for all indicators."""
|
||||
|
||||
def test_accuracy(self, indicator, library):
|
||||
"""ferro_ta and {library} should agree on {indicator}."""
|
||||
if not is_supported(REFERENCE_LIB, indicator):
|
||||
pytest.fail(f"{REFERENCE_LIB} does not implement {indicator}")
|
||||
if not is_supported(library, indicator):
|
||||
pytest.skip(f"{library} does not implement {indicator}")
|
||||
|
||||
ref = execute_indicator(REFERENCE_LIB, indicator, MEDIUM)
|
||||
cmp = execute_indicator(library, indicator, MEDIUM)
|
||||
|
||||
if len(cmp) == 0:
|
||||
pytest.fail(
|
||||
f"{library} returned empty output for supported indicator {indicator}"
|
||||
)
|
||||
if len(ref) == 0:
|
||||
pytest.fail(f"{REFERENCE_LIB} returned empty for {indicator}")
|
||||
|
||||
_compare(ref, cmp, indicator, library)
|
||||
|
||||
|
||||
# ── quick smoke tests that always run (no skip) ──────────────────────────────
|
||||
|
||||
|
||||
class TestSmoke:
|
||||
"""Sanity checks that ferro_ta returns non-empty finite arrays."""
|
||||
|
||||
@pytest.mark.parametrize("indicator", INDICATOR_NAMES)
|
||||
def test_ferro_ta_returns_finite(self, indicator):
|
||||
if not is_supported("ferro_ta", indicator):
|
||||
pytest.fail(f"ferro_ta does not implement {indicator}")
|
||||
|
||||
arr = execute_indicator("ferro_ta", indicator, MEDIUM)
|
||||
assert len(arr) > 0, f"ferro_ta {indicator} returned empty array"
|
||||
assert np.all(np.isfinite(arr)), (
|
||||
f"ferro_ta {indicator} has non-finite values: {arr[~np.isfinite(arr)][:5]}"
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("category,indicators", INDICATOR_CATEGORIES.items())
|
||||
def test_category_coverage(self, category, indicators):
|
||||
for ind in indicators:
|
||||
if not is_supported("ferro_ta", ind):
|
||||
pytest.fail(f"Category {category}: ferro_ta does not implement {ind}")
|
||||
arr = execute_indicator("ferro_ta", ind, MEDIUM)
|
||||
assert len(arr) > 0, f"Category {category}: {ind} returned empty"
|
||||
@@ -0,0 +1,370 @@
|
||||
"""
|
||||
Benchmark suite
|
||||
===========================
|
||||
|
||||
Numerical-regression and performance benchmarks that run against the canonical
|
||||
OHLCV fixture in ``benchmarks/fixtures/canonical_ohlcv.npz``.
|
||||
|
||||
Numerical regression checks
|
||||
----------------------------
|
||||
For each (indicator, params) pair in ``INDICATOR_SUITE``, the test:
|
||||
1. Loads the canonical dataset.
|
||||
2. Runs the indicator.
|
||||
3. Compares the last N non-NaN values to stored baselines (or tolerance-based).
|
||||
|
||||
To regenerate baselines after an intentional indicator change::
|
||||
|
||||
pytest benchmarks/test_benchmark_suite.py --update-baselines
|
||||
|
||||
Performance checks
|
||||
------------------
|
||||
Each indicator is timed over the canonical dataset. If a ``baselines.npz``
|
||||
file exists in this directory, the run compares to that; otherwise timing is
|
||||
reported only.
|
||||
|
||||
Run locally::
|
||||
|
||||
pytest benchmarks/test_benchmark_suite.py -v
|
||||
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
FIXTURE_PATH = pathlib.Path(__file__).parent / "fixtures" / "canonical_ohlcv.npz"
|
||||
BASELINE_PATH = pathlib.Path(__file__).parent / "baselines.npz"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Load fixture
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def ohlcv() -> dict[str, np.ndarray]:
|
||||
"""Load canonical OHLCV fixture."""
|
||||
if not FIXTURE_PATH.exists():
|
||||
pytest.skip(f"Canonical fixture not found: {FIXTURE_PATH}")
|
||||
data = np.load(FIXTURE_PATH)
|
||||
return {k: data[k] for k in data.files}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Indicator suite definition
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Each entry: (name, callable, kwargs)
|
||||
# The callable receives (close,) or (high, low, close,) based on 'inputs' key.
|
||||
INDICATOR_SUITE: list[dict[str, Any]] = [
|
||||
{
|
||||
"name": "SMA_20",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "SMA",
|
||||
"kwargs": {"timeperiod": 20},
|
||||
},
|
||||
{
|
||||
"name": "EMA_20",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "EMA",
|
||||
"kwargs": {"timeperiod": 20},
|
||||
},
|
||||
{
|
||||
"name": "RSI_14",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "RSI",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "ATR_14",
|
||||
"inputs": "hlc",
|
||||
"fn": None,
|
||||
"fn_name": "ATR",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "ADX_14",
|
||||
"inputs": "hlc",
|
||||
"fn": None,
|
||||
"fn_name": "ADX",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "STDDEV_20",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "STDDEV",
|
||||
"kwargs": {"timeperiod": 20},
|
||||
},
|
||||
{
|
||||
"name": "MACD",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "MACD",
|
||||
"kwargs": {},
|
||||
},
|
||||
{
|
||||
"name": "BBANDS_20",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "BBANDS",
|
||||
"kwargs": {"timeperiod": 20},
|
||||
},
|
||||
{
|
||||
"name": "STOCH",
|
||||
"inputs": "hlc",
|
||||
"fn": None,
|
||||
"fn_name": "STOCH",
|
||||
"kwargs": {},
|
||||
},
|
||||
{
|
||||
"name": "LINEARREG_14",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "LINEARREG",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "LINEARREG_SLOPE_14",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "LINEARREG_SLOPE",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "TSF_14",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "TSF",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "VAR_20",
|
||||
"inputs": "close",
|
||||
"fn": None,
|
||||
"fn_name": "VAR",
|
||||
"kwargs": {"timeperiod": 20},
|
||||
},
|
||||
{
|
||||
"name": "CORREL_30",
|
||||
"inputs": "pair_hl",
|
||||
"fn": None,
|
||||
"fn_name": "CORREL",
|
||||
"kwargs": {"timeperiod": 30},
|
||||
},
|
||||
{
|
||||
"name": "BETA_5",
|
||||
"inputs": "pair_hl",
|
||||
"fn": None,
|
||||
"fn_name": "BETA",
|
||||
"kwargs": {"timeperiod": 5},
|
||||
},
|
||||
{
|
||||
"name": "CCI_14",
|
||||
"inputs": "hlc",
|
||||
"fn": None,
|
||||
"fn_name": "CCI",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
{
|
||||
"name": "WILLR_14",
|
||||
"inputs": "hlc",
|
||||
"fn": None,
|
||||
"fn_name": "WILLR",
|
||||
"kwargs": {"timeperiod": 14},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _load_fn(fn_name: str) -> Callable[..., Any]:
|
||||
import ferro_ta as ft
|
||||
|
||||
return getattr(ft, fn_name)
|
||||
|
||||
|
||||
def _run_indicator(entry: dict[str, Any], data: dict[str, np.ndarray]) -> np.ndarray:
|
||||
fn = _load_fn(entry["fn_name"])
|
||||
if entry["inputs"] == "close":
|
||||
result = fn(data["close"], **entry["kwargs"])
|
||||
elif entry["inputs"] == "hlc":
|
||||
result = fn(data["high"], data["low"], data["close"], **entry["kwargs"])
|
||||
else: # pair_hl
|
||||
result = fn(data["high"], data["low"], **entry["kwargs"])
|
||||
if isinstance(result, tuple):
|
||||
result = result[0]
|
||||
return np.asarray(result, dtype=np.float64)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Numerical regression tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestNumericalRegression:
|
||||
"""Verify indicator outputs match stored baselines (or tolerance)."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_output_shape(
|
||||
self, entry: dict[str, Any], ohlcv: dict[str, np.ndarray]
|
||||
) -> None:
|
||||
"""Indicator output length must equal input length."""
|
||||
out = _run_indicator(entry, ohlcv)
|
||||
assert len(out) == len(ohlcv["close"]), (
|
||||
f"{entry['name']}: expected len {len(ohlcv['close'])}, got {len(out)}"
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_warmup_is_nan(
|
||||
self, entry: dict[str, Any], ohlcv: dict[str, np.ndarray]
|
||||
) -> None:
|
||||
"""First bar must be NaN (warm-up)."""
|
||||
out = _run_indicator(entry, ohlcv)
|
||||
assert np.isnan(out[0]), f"{entry['name']}: expected NaN at bar 0, got {out[0]}"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_no_inf(self, entry: dict[str, Any], ohlcv: dict[str, np.ndarray]) -> None:
|
||||
"""Output must not contain infinities."""
|
||||
out = _run_indicator(entry, ohlcv)
|
||||
assert not np.any(np.isinf(out)), f"{entry['name']}: output contains Inf"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_last_values_stable(
|
||||
self, entry: dict[str, Any], ohlcv: dict[str, np.ndarray]
|
||||
) -> None:
|
||||
"""Last 10 non-NaN values must be finite and stable (no sudden jumps)."""
|
||||
out = _run_indicator(entry, ohlcv)
|
||||
valid = out[~np.isnan(out)]
|
||||
assert len(valid) >= 10, f"{entry['name']}: fewer than 10 valid output values"
|
||||
last10 = valid[-10:]
|
||||
assert np.all(np.isfinite(last10)), (
|
||||
f"{entry['name']}: non-finite in last 10 values"
|
||||
)
|
||||
|
||||
@pytest.mark.skipif(not BASELINE_PATH.exists(), reason="No baselines.npz found")
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_regression_vs_baseline(
|
||||
self, entry: dict[str, Any], ohlcv: dict[str, np.ndarray]
|
||||
) -> None:
|
||||
"""Compare last 10 values to stored baselines."""
|
||||
baselines = np.load(BASELINE_PATH)
|
||||
key = entry["name"]
|
||||
if key not in baselines:
|
||||
pytest.skip(f"No baseline stored for {key}")
|
||||
out = _run_indicator(entry, ohlcv)
|
||||
valid = out[~np.isnan(out)]
|
||||
last10 = valid[-10:]
|
||||
stored = baselines[key]
|
||||
np.testing.assert_allclose(
|
||||
last10,
|
||||
stored,
|
||||
rtol=1e-5,
|
||||
atol=1e-8,
|
||||
err_msg=f"Numerical regression for {key}",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Performance benchmarks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPerformance:
|
||||
"""Timing benchmarks — record wall time and compare to baselines if present."""
|
||||
|
||||
PERF_THRESHOLD_FACTOR = 2.0 # fail if run is > 2× slower than baseline
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"entry", INDICATOR_SUITE, ids=[e["name"] for e in INDICATOR_SUITE]
|
||||
)
|
||||
def test_timing(
|
||||
self,
|
||||
entry: dict[str, Any],
|
||||
ohlcv: dict[str, np.ndarray],
|
||||
request: pytest.FixtureRequest,
|
||||
) -> None:
|
||||
"""Time the indicator on the canonical dataset."""
|
||||
# Warm-up run
|
||||
_run_indicator(entry, ohlcv)
|
||||
|
||||
# Timed run
|
||||
t0 = time.perf_counter()
|
||||
for _ in range(5):
|
||||
_run_indicator(entry, ohlcv)
|
||||
elapsed = (time.perf_counter() - t0) / 5.0 # average over 5 runs
|
||||
|
||||
# Store timing in request node for reporting
|
||||
request.node._ferro_ta_timing = elapsed # type: ignore[attr-defined]
|
||||
|
||||
# Compare to baseline if available
|
||||
if BASELINE_PATH.exists():
|
||||
baselines = np.load(BASELINE_PATH, allow_pickle=True)
|
||||
key = f"timing_{entry['name']}"
|
||||
if key in baselines:
|
||||
baseline_time = float(baselines[key])
|
||||
if elapsed > baseline_time * self.PERF_THRESHOLD_FACTOR:
|
||||
pytest.fail(
|
||||
f"{entry['name']}: timing regression — "
|
||||
f"current {elapsed * 1000:.2f}ms vs "
|
||||
f"baseline {baseline_time * 1000:.2f}ms "
|
||||
f"(>{self.PERF_THRESHOLD_FACTOR}×)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Baseline update helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def update_baselines(ohlcv_data: dict[str, np.ndarray]) -> None:
|
||||
"""Write current indicator outputs and timings to baselines.npz.
|
||||
|
||||
Call this after intentional changes to update the stored baselines::
|
||||
|
||||
python -c "
|
||||
import numpy as np
|
||||
from benchmarks.test_benchmark_suite import update_baselines, FIXTURE_PATH
|
||||
data = {k: v for k, v in np.load(FIXTURE_PATH).items()}
|
||||
update_baselines(data)
|
||||
"
|
||||
"""
|
||||
store: dict[str, np.ndarray] = {}
|
||||
for entry in INDICATOR_SUITE:
|
||||
out = _run_indicator(entry, ohlcv_data)
|
||||
valid = out[~np.isnan(out)]
|
||||
store[entry["name"]] = valid[-10:]
|
||||
|
||||
# Timing
|
||||
t0 = time.perf_counter()
|
||||
for _ in range(5):
|
||||
_run_indicator(entry, ohlcv_data)
|
||||
store[f"timing_{entry['name']}"] = np.array([(time.perf_counter() - t0) / 5.0])
|
||||
|
||||
np.savez_compressed(BASELINE_PATH, **store)
|
||||
print(f"Baselines written to {BASELINE_PATH}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if not FIXTURE_PATH.exists():
|
||||
print(f"Fixture not found: {FIXTURE_PATH}")
|
||||
print("Run: python benchmarks/fixtures/generate_canonical.py")
|
||||
else:
|
||||
data = {k: v for k, v in np.load(FIXTURE_PATH).items()}
|
||||
update_baselines(data)
|
||||
@@ -0,0 +1,124 @@
|
||||
"""
|
||||
Derivatives benchmark hooks.
|
||||
|
||||
These are intentionally optional and skip when `py_vollib` is unavailable.
|
||||
Run with:
|
||||
|
||||
uv run pytest benchmarks/test_derivatives_speed.py --benchmark-only -v
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
# Ensure direct benchmark test runs can import local package from `python/`.
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
PYTHON_SRC = ROOT / "python"
|
||||
if str(PYTHON_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(PYTHON_SRC))
|
||||
|
||||
HAS_FERRO_EXTENSION = True
|
||||
try:
|
||||
from ferro_ta.analysis.options import implied_volatility, option_price
|
||||
except ModuleNotFoundError:
|
||||
HAS_FERRO_EXTENSION = False
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not HAS_FERRO_EXTENSION, reason="ferro_ta extension is not built"
|
||||
)
|
||||
|
||||
|
||||
def _sample_chain(n: int = 1000) -> tuple[np.ndarray, ...]:
|
||||
spot = np.linspace(90.0, 110.0, n)
|
||||
strike = np.full(n, 100.0)
|
||||
rate = np.full(n, 0.02)
|
||||
time_to_expiry = np.full(n, 0.5)
|
||||
volatility = np.full(n, 0.2)
|
||||
return spot, strike, rate, time_to_expiry, volatility
|
||||
|
||||
|
||||
def test_ferro_ta_option_price_speed(benchmark):
|
||||
spot, strike, rate, time_to_expiry, volatility = _sample_chain()
|
||||
|
||||
benchmark.pedantic(
|
||||
lambda: option_price(
|
||||
spot,
|
||||
strike,
|
||||
rate,
|
||||
time_to_expiry,
|
||||
volatility,
|
||||
option_type="call",
|
||||
model="bsm",
|
||||
),
|
||||
iterations=5,
|
||||
rounds=20,
|
||||
warmup_rounds=2,
|
||||
)
|
||||
|
||||
|
||||
def test_ferro_ta_implied_vol_speed(benchmark):
|
||||
spot, strike, rate, time_to_expiry, volatility = _sample_chain()
|
||||
prices = option_price(
|
||||
spot,
|
||||
strike,
|
||||
rate,
|
||||
time_to_expiry,
|
||||
volatility,
|
||||
option_type="call",
|
||||
model="bsm",
|
||||
)
|
||||
|
||||
benchmark.pedantic(
|
||||
lambda: implied_volatility(
|
||||
prices,
|
||||
spot,
|
||||
strike,
|
||||
rate,
|
||||
time_to_expiry,
|
||||
option_type="call",
|
||||
model="bsm",
|
||||
),
|
||||
iterations=5,
|
||||
rounds=20,
|
||||
warmup_rounds=2,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
importlib.util.find_spec("py_vollib") is None,
|
||||
reason="py_vollib is optional",
|
||||
)
|
||||
def test_py_vollib_scalar_loop_baseline(benchmark):
|
||||
from py_vollib.black_scholes_merton import black_scholes_merton as py_vollib_bsm
|
||||
from py_vollib.black_scholes_merton.implied_volatility import (
|
||||
implied_volatility as py_vollib_iv,
|
||||
)
|
||||
|
||||
spot, strike, rate, time_to_expiry, volatility = _sample_chain(250)
|
||||
prices = [
|
||||
py_vollib_bsm("c", float(s), float(k), float(t), float(r), float(vol), 0.0)
|
||||
for s, k, r, t, vol in zip(spot, strike, rate, time_to_expiry, volatility)
|
||||
]
|
||||
|
||||
benchmark.pedantic(
|
||||
lambda: [
|
||||
py_vollib_iv(
|
||||
float(price),
|
||||
"c",
|
||||
float(s),
|
||||
float(k),
|
||||
float(t),
|
||||
float(r),
|
||||
0.0,
|
||||
)
|
||||
for price, s, k, r, t in zip(prices, spot, strike, rate, time_to_expiry)
|
||||
],
|
||||
iterations=3,
|
||||
rounds=10,
|
||||
warmup_rounds=1,
|
||||
)
|
||||
@@ -0,0 +1,102 @@
|
||||
"""
|
||||
Cross-library speed benchmarks using pytest-benchmark.
|
||||
|
||||
Run: pytest benchmarks/test_speed.py --benchmark-only -v
|
||||
pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json
|
||||
|
||||
Streaming benchmarks are in test_streaming_speed.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmarks.data_generator import LARGE
|
||||
from benchmarks.wrapper_registry import (
|
||||
INDICATOR_CATEGORIES,
|
||||
available_libraries,
|
||||
execute_indicator,
|
||||
is_supported,
|
||||
)
|
||||
|
||||
BENCH_DATA = LARGE # 100k bars for main benchmarks
|
||||
BENCH_LIBS = available_libraries()
|
||||
|
||||
|
||||
def _make_bench(indicator: str, library: str):
|
||||
"""Return a benchmark function that runs indicator on library (uses BENCH_DATA)."""
|
||||
|
||||
def _fn():
|
||||
execute_indicator(library, indicator, BENCH_DATA)
|
||||
|
||||
_fn.__name__ = f"{library}_{indicator}"
|
||||
return _fn
|
||||
|
||||
|
||||
# ── Parametrize over all (indicator, library) combinations ───────────────────
|
||||
|
||||
|
||||
def pytest_generate_tests(metafunc):
|
||||
if "indicator" in metafunc.fixturenames and "library" in metafunc.fixturenames:
|
||||
params = []
|
||||
for cat, inds in INDICATOR_CATEGORIES.items():
|
||||
for ind in inds:
|
||||
for lib in BENCH_LIBS:
|
||||
params.append(pytest.param(ind, lib, id=f"{cat}/{ind}/{lib}"))
|
||||
metafunc.parametrize("indicator,library", params)
|
||||
|
||||
|
||||
class TestSpeed:
|
||||
"""One benchmark per (indicator, library) pair — all at 100k bars (LARGE dataset)."""
|
||||
|
||||
def test_speed(self, benchmark, indicator, library):
|
||||
if not is_supported(library, indicator):
|
||||
pytest.skip(f"{library} does not implement {indicator}")
|
||||
fn = _make_bench(indicator, library)
|
||||
benchmark.pedantic(fn, iterations=5, rounds=20, warmup_rounds=2)
|
||||
|
||||
|
||||
# ── Standalone head-to-head for the most important indicators ─────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"indicator,libs",
|
||||
[
|
||||
("SMA", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("EMA", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("RSI", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("MACD", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("BBANDS", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("ATR", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("CCI", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("WILLR", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("OBV", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("ADX", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("MFI", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
("STOCH", ["ferro_ta", "talib", "tulipy", "pandas_ta", "ta", "finta"]),
|
||||
],
|
||||
)
|
||||
def test_head_to_head(benchmark, indicator, libs):
|
||||
"""Benchmark ferro_ta vs all peers — for README table generation."""
|
||||
if not is_supported("ferro_ta", indicator):
|
||||
pytest.skip(f"ferro_ta does not implement {indicator}")
|
||||
fn = _make_bench(indicator, "ferro_ta")
|
||||
benchmark.pedantic(fn, iterations=5, rounds=20, warmup_rounds=2)
|
||||
|
||||
|
||||
# ── Large dataset benchmarks (100k bars) ─────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"indicator",
|
||||
["SMA", "EMA", "RSI", "MACD", "ATR", "BBANDS", "OBV", "CCI", "ADX", "MFI"],
|
||||
)
|
||||
def test_large_dataset(benchmark, indicator):
|
||||
"""Scaling benchmark at 100k bars for ferro_ta."""
|
||||
if not is_supported("ferro_ta", indicator):
|
||||
pytest.skip(f"ferro_ta does not implement {indicator}")
|
||||
|
||||
def _fn():
|
||||
execute_indicator("ferro_ta", indicator, LARGE)
|
||||
|
||||
benchmark.pedantic(_fn, iterations=3, rounds=10, warmup_rounds=1)
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user