feat: broaden CPU and platform coverage across PyPI, nodes, and crate… (#26)
* feat: broaden CPU and platform coverage across PyPI, nodes, and crates.io Replace static SIMD with runtime CPU-feature dispatch and expand the release wheel matrix so one set of artifacts runs on any target CPU and platform without illegal-instruction crashes. Rust core: - Add multiversion runtime dispatch (crates/ferro_ta_core/src/simd.rs); drop compile-time `wide`. `simd` feature is now default-on and forwarded through the pyo3 crate, and stays compatible with #![forbid(unsafe_code)]. Packaging: - abi3-py310: one cp310-abi3 wheel per platform (covers CPython 3.10+). - CI matrix adds Linux aarch64 + musllinux (x86_64/aarch64) and Windows arm64. Node/Docker + docs: - api/Dockerfile: document baseline+dispatch (no target-cpu pin) and add a fail-fast import check; aarch64 containers now install cleanly. - Rewrite docs/guides/simd.md; fix stale `wide` mention in ADR 0003. - Add ADR 0006 (CPU coverage strategy). Also bundles in-flight release prep already staged in the tree (DTW exception types, SBOM/provenance security, supporting docs). * fix(ci): clear cargo-deny and pip-audit failures; apply dependency bumps cargo-deny (advisories): - Ignore pyo3 RUSTSEC-2026-0176 / RUSTSEC-2026-0177 in deny.toml with a documented rationale: ferro-ta uses neither affected code path (PyList/PyTuple nth iterators; PyCFunction::new_closure). Upstream fix needs pyo3 >=0.29 (large API migration), tracked as a follow-up. pip-audit: - Bump dev lockfile idna 3.18, pytest 9.1.1, urllib3 2.7.0 to clear PYSEC-2026-215, CVE-2025-71176, PYSEC-2026-141/142. Dependency bumps (supersede open dependabot PRs; they auto-close on merge): - cargo: log 0.4.32, serde_json 1.0.150, rayon 1.12.0 - api/requirements.txt: uvicorn>=0.49.0, pydantic>=2.13.4, ferro-ta>=1.1.4 - CI actions: deploy-pages v5, upload-pages-artifact v5, action-gh-release v3 The open `wide` 1.5.0 bump (PR #24) is obsolete — the crate is removed in this branch. * chore: address CodeRabbit review; remove docs/adr section CodeRabbit findings: - CI sbom job: add `attestations: write` so attest-build-provenance can run (it had only contents:write + id-token:write). - simd.rs: vectorize `wma_seed` with lane-local accumulators — it was scalar behind the multiversion wrapper, adding dispatch overhead for no SIMD gain. - CHANGELOG: consolidate the duplicate `### Changed` heading. - python/ferro_ta/__init__.py: also re-export the `FerroTaError` alias. - docs/guides/dtw.md: soften "byte-for-byte" parity to within-tolerance. Remove docs/adr/ at maintainer request and clean up the ADR links in the SIMD and DTW guides. The ADR files remain in commit 9506a30 if ever needed.
This commit is contained in:
Pratik Bhadane
@@ -0,0 +1,80 @@
|
||||
# Dynamic Time Warping
|
||||
|
||||
ferro-ta ships three DTW entry points. Pick the one that matches your
|
||||
workload — the distance-only path is measurably faster than the one that
|
||||
reconstructs the warping path, and `BATCH_DTW` parallelises over rows.
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Function | Returns | When to use |
|
||||
|---|---|---|
|
||||
| `DTW_DISTANCE(a, b, window=None)` | `float` | You only need the distance. Fastest. |
|
||||
| `DTW(a, b, window=None)` | `(float, ndarray[N, 2])` | You need the alignment path for plotting or downstream analysis. |
|
||||
| `BATCH_DTW(matrix, reference, window=None)` | `ndarray[N]` | You have N candidate series and one reference; uses rayon. |
|
||||
|
||||
## Distance convention
|
||||
|
||||
ferro-ta's DTW uses squared-Euclidean local cost accumulated along the
|
||||
optimal path, with a single `sqrt()` applied at the end. This matches
|
||||
`dtaidistance.dtw.distance()` to within floating-point tolerance (parity
|
||||
tests assert numerical agreement, not bitwise identity). Example:
|
||||
|
||||
```python
|
||||
>>> import ferro_ta as fta
|
||||
>>> fta.DTW_DISTANCE([0.0, 1.0, 2.0], [1.0, 2.0, 3.0])
|
||||
1.4142135623730951 # == sqrt(2), same as dtaidistance
|
||||
```
|
||||
|
||||
If you are migrating from a library that uses absolute-difference local
|
||||
cost without the final sqrt (e.g. `fastdtw`'s default), your numbers will
|
||||
not line up. That is a choice ferro-ta made for parity with the
|
||||
scientific-Python ecosystem.
|
||||
|
||||
## Window constraint (Sakoe-Chiba band)
|
||||
|
||||
Passing `window=w` constrains the DP to cells where `|i - j| < w`. This
|
||||
turns the O(n·m) cost into O(n·w), which is typically a 5–20× speedup for
|
||||
realistic `w`. A narrower band can only *increase* the distance, so
|
||||
`window=` is safe to use whenever your series are roughly aligned.
|
||||
|
||||
```python
|
||||
# Unconstrained
|
||||
fta.DTW_DISTANCE(a, b)
|
||||
|
||||
# Constrained: warping may shift up to 5 positions
|
||||
fta.DTW_DISTANCE(a, b, window=5)
|
||||
```
|
||||
|
||||
## Batch usage
|
||||
|
||||
`BATCH_DTW` compares each row of a 2-D matrix against one reference
|
||||
series, in parallel:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import ferro_ta as fta
|
||||
|
||||
reference = np.random.random(500)
|
||||
candidates = np.random.random((1000, 500))
|
||||
|
||||
distances = fta.BATCH_DTW(candidates, reference, window=20)
|
||||
nearest = int(np.argmin(distances))
|
||||
```
|
||||
|
||||
Parallelism is via rayon; no thread-pool configuration is needed on the
|
||||
Python side. For the sequence lengths ferro-ta targets (thousands of
|
||||
bars, hundreds to low thousands of candidates), batch-parallel classic
|
||||
DTW beats FastDTW-style approximations.
|
||||
|
||||
## Edge cases
|
||||
|
||||
- **Empty input:** raises `FerroTAInputError`.
|
||||
- **NaN in input:** propagates to the output (matches IEEE 754). Call
|
||||
`ferro_ta.core.exceptions.check_finite()` first if you want to fail
|
||||
loudly instead.
|
||||
- **Different-length series:** fully supported. The path array length
|
||||
is bounded by `max(n, m) <= len(path) <= n + m - 1`.
|
||||
|
||||
## See also
|
||||
|
||||
- `tests/unit/indicators/test_statistic.py` — parity tests against `dtaidistance`.
|
||||
@@ -0,0 +1,92 @@
|
||||
# SIMD acceleration
|
||||
|
||||
ferro-ta accelerates hot reductions with **runtime CPU-feature dispatch**
|
||||
via the [`multiversion`](https://crates.io/crates/multiversion) crate. Each
|
||||
dispatched function is compiled into several variants — baseline, SSE,
|
||||
AVX2/FMA, AVX-512 on x86_64; NEON on aarch64 — and the fastest one the
|
||||
**current** CPU supports is chosen at load time via CPUID.
|
||||
|
||||
## Why dispatch instead of `-C target-cpu`
|
||||
|
||||
A static `RUSTFLAGS=-C target-cpu=x86-64-v3` build *requires* AVX2 on the
|
||||
running CPU; on an older chip it crashes with an illegal instruction
|
||||
(SIGILL). Runtime dispatch instead ships every code path in one binary and
|
||||
picks at runtime, so a single artifact:
|
||||
|
||||
- runs on **any** CPU of the target architecture (no SIGILL on pre-AVX2
|
||||
hardware), and
|
||||
- still uses wide vector units where the hardware has them.
|
||||
|
||||
That property is what lets the **same** wheel / Docker image / crate run
|
||||
across a heterogeneous fleet.
|
||||
|
||||
## When it helps
|
||||
|
||||
SIMD helps indicators whose inner loop is a reduction over contiguous
|
||||
`f64` data — e.g. the initial window sum that seeds SMA, the `(T, S)` seed
|
||||
for WMA, and similar fixed-window reductions. It does **not** help:
|
||||
|
||||
- The O(n) streaming recurrences (`window_sum += new - old`): each step
|
||||
depends on the previous one, so they are inherently sequential.
|
||||
- Branchy inner loops (SAR, candlestick patterns).
|
||||
- Streaming classes (a single-bar update is one or two ops).
|
||||
|
||||
The shared primitives live in `crates/ferro_ta_core/src/simd.rs`
|
||||
(`sum`, `wma_seed`). They accumulate into independent lanes before a final
|
||||
horizontal combine — that lane independence is what allows the optimizer to
|
||||
vectorize each CPU-feature variant. A consequence is that results differ
|
||||
from a strict left-to-right sum by a few ULPs, well inside every
|
||||
indicator's documented tolerance.
|
||||
|
||||
## The `simd` feature
|
||||
|
||||
Dispatch is gated behind the `simd` Cargo feature, which is **on by
|
||||
default**:
|
||||
|
||||
```bash
|
||||
# default build — runtime dispatch enabled
|
||||
cargo build -p ferro_ta_core --release
|
||||
|
||||
# pure-scalar build (debugging / baseline benchmarking)
|
||||
cargo build -p ferro_ta_core --release --no-default-features
|
||||
```
|
||||
|
||||
For Python, wheels published to PyPI are built with the default features,
|
||||
so `pip install ferro-ta` ships the dispatched fast path with no action on
|
||||
your part. To build a pure-scalar extension from source:
|
||||
|
||||
```bash
|
||||
maturin develop --release --no-default-features
|
||||
```
|
||||
|
||||
## Measured speedups
|
||||
|
||||
The nightly `benchmarks/bench_simd.py` job (see
|
||||
`.github/workflows/nightly-bench.yml`) builds the extension twice — once
|
||||
with `--no-default-features` (pure scalar) and once with `--features simd`
|
||||
(dispatch) — and reports the per-indicator delta. Numbers are regenerated
|
||||
on every run and vary with hardware; treat any table in a PR as a snapshot,
|
||||
not a contract. The dispatched kernels here target correctness-preserving
|
||||
reductions, so gains are modest on the sliding-window indicators and larger
|
||||
on full-array reductions.
|
||||
|
||||
## Adding a SIMD-optimized indicator
|
||||
|
||||
1. Write and test the **scalar** implementation first — it is the ground
|
||||
truth.
|
||||
2. If the hot path is a contiguous `f64` reduction, route it through a
|
||||
`crate::simd` primitive, or wrap a new helper in
|
||||
`#[multiversion::multiversion(targets = "simd")]` with the loop body
|
||||
accumulating into independent lanes.
|
||||
3. Add a parity test comparing the dispatched result against the strict
|
||||
scalar reference within tolerance (see `simd.rs` tests for the pattern).
|
||||
4. Benchmark scalar vs dispatch via `bench_simd.py`. Only keep the SIMD
|
||||
path if it wins — alignment and tail-handling overhead can make a naive
|
||||
vectorization *lose* to scalar.
|
||||
|
||||
## See also
|
||||
|
||||
- `crates/ferro_ta_core/src/simd.rs` — dispatched primitives and tests.
|
||||
- `benches/indicators.rs` — criterion suite.
|
||||
- `crates/ferro_ta_core/Cargo.toml` `[features] simd = ["dep:multiversion"]`
|
||||
— the gate (default-on).
|
||||
Reference in New Issue
Block a user