Pratik Bhadane
436954138f
Update version numbers across Rust, Python, and documentation files to 1.1.0. Enhance the .gitignore to include macOS dSYM files and plans directory. Introduce new dependencies in the Rust core library and update the README to reflect recent performance benchmarks and backtesting engine capabilities. Add new artifacts to the benchmarks manifest and improve documentation for the backtesting engine API.
247 lines
6.4 KiB
ReStructuredText
247 lines
6.4 KiB
ReStructuredText
Benchmarks
|
||
==========
|
||
|
||
The benchmark suite is meant to support a narrow claim: ferro-ta is often
|
||
faster on selected indicators, and the evidence is published in a reproducible
|
||
form.
|
||
|
||
What is published
|
||
-----------------
|
||
|
||
The authoritative benchmark workflow lives in ``benchmarks/``:
|
||
|
||
- Cross-library speed suite: ``benchmarks/test_speed.py``
|
||
- Cross-library accuracy suite: ``benchmarks/test_accuracy.py``
|
||
- TA-Lib head-to-head script: ``benchmarks/bench_vs_talib.py``
|
||
- Backtesting engine benchmark: ``benchmarks/bench_backtest.py``
|
||
- Table generation from benchmark JSON: ``benchmarks/benchmark_table.py``
|
||
- Perf-contract artifact bundle: ``benchmarks/run_perf_contract.py``
|
||
|
||
Backtesting engine — competitor comparison
|
||
------------------------------------------
|
||
|
||
Measured on Apple M-series, Python 3.13, Rust 1.91, using an SMA(20/50)
|
||
crossover strategy with 0.1% commission and 5 bps slippage. Median of 5 runs.
|
||
|
||
.. list-table:: Speed vs backtesting libraries (signal → equity curve)
|
||
:header-rows: 1
|
||
|
||
* - Library
|
||
- 1k bars
|
||
- 10k bars
|
||
- 100k bars
|
||
- vs ferro-ta core (100k)
|
||
* - **ferro-ta** ``backtest_core``
|
||
- 0.004 ms
|
||
- 0.033 ms
|
||
- 0.286 ms
|
||
- —
|
||
* - **ferro-ta** ``backtest_ohlcv_core``
|
||
- 0.004 ms
|
||
- 0.037 ms
|
||
- 0.332 ms
|
||
- ~same
|
||
* - NumPy vectorized (manual)
|
||
- 0.013 ms
|
||
- 0.042 ms
|
||
- 0.459 ms
|
||
- 1.6× slower
|
||
* - vectorbt 0.28
|
||
- 1.32 ms
|
||
- 1.31 ms
|
||
- 2.90 ms
|
||
- **10× slower**
|
||
* - backtesting.py
|
||
- 10.5 ms
|
||
- 42.3 ms
|
||
- 319.6 ms
|
||
- **1,117× slower**
|
||
* - backtrader 1.9
|
||
- 53.9 ms
|
||
- 518 ms
|
||
- n/a (skipped)
|
||
- **>15,000× slower**
|
||
|
||
Accuracy: ferro-ta positions and bar-returns are **bit-exact** against the NumPy
|
||
reference implementation (max per-bar equity diff = 0.00e+00 with zero
|
||
commission/slippage).
|
||
|
||
Additional ferro-ta capabilities not present in the libraries above:
|
||
|
||
.. list-table::
|
||
:header-rows: 1
|
||
|
||
* - Capability
|
||
- ferro-ta result
|
||
- NumPy baseline
|
||
- Speedup
|
||
* - Monte Carlo 1,000 sims (100k bars)
|
||
- 50 ms (parallel Rayon + LCG)
|
||
- 612 ms (Python loop)
|
||
- **12×**
|
||
* - 23 performance metrics, single call (100k bars)
|
||
- 2.8 ms
|
||
- 0.36 ms (2 metrics only)
|
||
- 0.12 ms / metric
|
||
* - Multi-asset 100 assets (100k bars)
|
||
- 43 ms parallel / 88 ms serial
|
||
- —
|
||
- 2× parallel speedup
|
||
* - Walk-forward fold indices (100k bars)
|
||
- 0.3 µs
|
||
- —
|
||
- —
|
||
|
||
Reproduce the backtest benchmark:
|
||
|
||
.. code-block:: bash
|
||
|
||
python benchmarks/bench_backtest.py --sizes 10000 100000 \
|
||
--json benchmarks/artifacts/latest/bench_backtest_results.json
|
||
|
||
Latest checked-in TA-Lib artifact
|
||
---------------------------------
|
||
|
||
The current checked-in TA-Lib comparison artifact benchmarks contiguous
|
||
``float64`` arrays at 10k and 100k bars on an ``Apple M3 Max`` with 14 logical
|
||
cores, about 38.7 GB RAM, ``CPython 3.13.5``, and ``Rust 1.91.1`` using the
|
||
default release profile (``lto = true``, ``codegen-units = 1``).
|
||
|
||
Summary from ``benchmarks/artifacts/latest/benchmark_vs_talib.json``:
|
||
|
||
.. list-table::
|
||
:header-rows: 1
|
||
|
||
* - Size
|
||
- Rows
|
||
- ferro-ta wins
|
||
- Median speedup
|
||
- TA-Lib wins or ties
|
||
* - ``10,000``
|
||
- 12
|
||
- 6
|
||
- ``1.0850x``
|
||
- ``EMA``, ``RSI``, ``ATR``, ``STOCH``, ``ADX``, ``OBV``
|
||
* - ``100,000``
|
||
- 12
|
||
- 6
|
||
- ``1.0784x``
|
||
- ``EMA``, ``RSI``, ``ATR``, ``STOCH``, ``ADX``, ``OBV``
|
||
|
||
Examples from the 100k-bar run:
|
||
|
||
.. list-table::
|
||
:header-rows: 1
|
||
|
||
* - Indicator
|
||
- ferro-ta
|
||
- TA-Lib
|
||
- Speedup
|
||
- Read
|
||
* - ``SMA``
|
||
- ``0.0985 ms``
|
||
- ``0.2241 ms``
|
||
- ``2.2751x``
|
||
- clear ferro-ta win
|
||
* - ``BBANDS``
|
||
- ``0.2122 ms``
|
||
- ``0.4966 ms``
|
||
- ``2.3402x``
|
||
- clear ferro-ta win
|
||
* - ``MACD``
|
||
- ``0.5152 ms``
|
||
- ``0.7111 ms``
|
||
- ``1.3801x``
|
||
- ferro-ta win
|
||
* - ``STOCH``
|
||
- ``1.7064 ms``
|
||
- ``0.7603 ms``
|
||
- ``0.4455x``
|
||
- TA-Lib win
|
||
* - ``ADX``
|
||
- ``0.7910 ms``
|
||
- ``0.5769 ms``
|
||
- ``0.7294x``
|
||
- TA-Lib win
|
||
* - ``ATR``
|
||
- ``0.5087 ms``
|
||
- ``0.5147 ms``
|
||
- ``1.0118x``
|
||
- tie on this machine
|
||
|
||
Methodology notes
|
||
-----------------
|
||
|
||
- The head-to-head script uses the same synthetic OHLCV generator, the same
|
||
parameters, and the same contiguous ``float64`` array layout for both
|
||
libraries.
|
||
- Reported speedup is ``TA-Lib median time / ferro-ta median time``.
|
||
- The script uses 1 warmup run and 7 measured runs per case, and now records
|
||
the full per-run timing samples, not just one selected number.
|
||
- Published JSON artifacts include machine/runtime metadata, git metadata, Rust
|
||
toolchain and build-profile metadata, per-run variance statistics, and
|
||
Python-tracked peak allocation snapshots.
|
||
- Allocation snapshots are based on ``tracemalloc`` and capture Python-tracked
|
||
allocations only; they are not full native RSS profiles.
|
||
- If your workload uses non-contiguous arrays, different dtypes, or different
|
||
batch sizes, benchmark that exact workload. Those factors can materially
|
||
change the result.
|
||
|
||
Reproduce the TA-Lib comparison
|
||
-------------------------------
|
||
|
||
.. code-block:: bash
|
||
|
||
pip install ta-lib
|
||
python benchmarks/bench_vs_talib.py --sizes 10000 100000 --json benchmark_vs_talib.json
|
||
|
||
The JSON output is the main artifact to review when publishing performance
|
||
claims.
|
||
|
||
Cross-library suite
|
||
-------------------
|
||
|
||
Run the broader speed suite on 100,000 bars:
|
||
|
||
.. code-block:: bash
|
||
|
||
uv run pytest benchmarks/test_speed.py --benchmark-only --benchmark-json=benchmarks/results.json -v
|
||
|
||
Selected throughput examples from the checked-in table:
|
||
|
||
.. list-table::
|
||
:header-rows: 1
|
||
|
||
* - Indicator
|
||
- Throughput
|
||
* - ``ADD``
|
||
- 1.9 G bars/s
|
||
* - ``CDLENGULFING``
|
||
- 454 M bars/s
|
||
* - ``EMA``
|
||
- 444 M bars/s
|
||
* - ``SMA``
|
||
- 259 M bars/s
|
||
* - ``RSI``
|
||
- 145 M bars/s
|
||
* - ``ATR``
|
||
- 70 M bars/s
|
||
* - ``MACD``
|
||
- 104 M bars/s
|
||
* - ``STOCH``
|
||
- 33 M bars/s
|
||
|
||
Perf-contract artifacts
|
||
-----------------------
|
||
|
||
Use the perf-contract runner when you want a compact, machine-readable artifact
|
||
bundle for single-series latency, batch throughput, streaming throughput, and
|
||
hotspot attribution:
|
||
|
||
.. code-block:: bash
|
||
|
||
uv run python benchmarks/run_perf_contract.py --output-dir benchmarks/artifacts/latest
|
||
|
||
See ``benchmarks/README.md`` for the detailed benchmark playbook and the
|
||
checked-in comparison tables.
|