gavindiaz/ferro-ta
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: broaden CPU and platform coverage across PyPI, nodes, and crate… (#26)

Browse Source 
* feat: broaden CPU and platform coverage across PyPI, nodes, and crates.io

Replace static SIMD with runtime CPU-feature dispatch and expand the release
wheel matrix so one set of artifacts runs on any target CPU and platform
without illegal-instruction crashes.

Rust core:
- Add multiversion runtime dispatch (crates/ferro_ta_core/src/simd.rs); drop
  compile-time `wide`. `simd` feature is now default-on and forwarded through
  the pyo3 crate, and stays compatible with #![forbid(unsafe_code)].

Packaging:
- abi3-py310: one cp310-abi3 wheel per platform (covers CPython 3.10+).
- CI matrix adds Linux aarch64 + musllinux (x86_64/aarch64) and Windows arm64.

Node/Docker + docs:
- api/Dockerfile: document baseline+dispatch (no target-cpu pin) and add a
  fail-fast import check; aarch64 containers now install cleanly.
- Rewrite docs/guides/simd.md; fix stale `wide` mention in ADR 0003.
- Add ADR 0006 (CPU coverage strategy).

Also bundles in-flight release prep already staged in the tree (DTW exception
types, SBOM/provenance security, supporting docs).

* fix(ci): clear cargo-deny and pip-audit failures; apply dependency bumps

cargo-deny (advisories):
- Ignore pyo3 RUSTSEC-2026-0176 / RUSTSEC-2026-0177 in deny.toml with a
  documented rationale: ferro-ta uses neither affected code path
  (PyList/PyTuple nth iterators; PyCFunction::new_closure). Upstream fix
  needs pyo3 >=0.29 (large API migration), tracked as a follow-up.

pip-audit:
- Bump dev lockfile idna 3.18, pytest 9.1.1, urllib3 2.7.0 to clear
  PYSEC-2026-215, CVE-2025-71176, PYSEC-2026-141/142.

Dependency bumps (supersede open dependabot PRs; they auto-close on merge):
- cargo: log 0.4.32, serde_json 1.0.150, rayon 1.12.0
- api/requirements.txt: uvicorn>=0.49.0, pydantic>=2.13.4, ferro-ta>=1.1.4
- CI actions: deploy-pages v5, upload-pages-artifact v5, action-gh-release v3

The open `wide` 1.5.0 bump (PR #24) is obsolete — the crate is removed in
this branch.

* chore: address CodeRabbit review; remove docs/adr section

CodeRabbit findings:
- CI sbom job: add `attestations: write` so attest-build-provenance can run
  (it had only contents:write + id-token:write).
- simd.rs: vectorize `wma_seed` with lane-local accumulators — it was scalar
  behind the multiversion wrapper, adding dispatch overhead for no SIMD gain.
- CHANGELOG: consolidate the duplicate `### Changed` heading.
- python/ferro_ta/__init__.py: also re-export the `FerroTaError` alias.
- docs/guides/dtw.md: soften "byte-for-byte" parity to within-tolerance.

Remove docs/adr/ at maintainer request and clean up the ADR links in the
SIMD and DTW guides. The ADR files remain in commit 9506a30 if ever needed.
This commit is contained in:
Pratik Bhadane
2026-06-29 18:21:22 +05:30
committed by GitHub
parent fd1bb137d6
commit 288b1546b2
23 changed files with 839 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/CI.yml
+110 -42
View File
@@ -87,7 +87,7 @@ jobs:
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@v5
# -------------------------------------------------------------------------
# CI gate — all required jobs must pass before this job succeeds.
@@ -136,88 +136,123 @@ jobs:
# Keep release jobs here because trusted-publisher configuration points to
# CI.yml specifically.
# -------------------------------------------------------------------------
# One abi3 wheel per (platform, arch). abi3-py310 means a single wheel
# covers CPython 3.10+ on each target, so there is no python-version axis.
# extension-module + abi3 link no libpython, which is what lets the linux
# jobs cross-compile aarch64/musl from an x86_64 runner via maturin-action.
build-wheels-linux:
name: Build wheels (linux / py${{ matrix.python-version }})
name: Build wheels (linux-gnu / ${{ matrix.target }})
runs-on: ubuntu-latest
if: (github.event_name == 'release' && github.event.action == 'published') || (github.event_name == 'workflow_dispatch' && inputs.release)
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
target: [x86_64, aarch64]
steps:
- uses: actions/checkout@v6
- name: Build wheel
- name: Build abi3 manylinux wheel
uses: PyO3/maturin-action@v1
with:
command: build
args: --release --out dist --compatibility pypi -i python${{ matrix.python-version }}
target: ${{ matrix.target }}
args: --release --out dist
manylinux: "2_17"
- name: Upload wheels as artifact
uses: actions/upload-artifact@v7
with:
name: wheels-linux-py${{ matrix.python-version }}
name: wheels-linux-gnu-${{ matrix.target }}
path: dist/*.whl
build-wheels-musllinux:
name: Build wheels (linux-musl / ${{ matrix.target }})
runs-on: ubuntu-latest
if: (github.event_name == 'release' && github.event.action == 'published') || (github.event_name == 'workflow_dispatch' && inputs.release)
strategy:
fail-fast: false
matrix:
target: [x86_64, aarch64]
steps:
- uses: actions/checkout@v6
- name: Build abi3 musllinux wheel
uses: PyO3/maturin-action@v1
with:
command: build
target: ${{ matrix.target }}
args: --release --out dist
manylinux: musllinux_1_2
- name: Upload wheels as artifact
uses: actions/upload-artifact@v7
with:
name: wheels-linux-musl-${{ matrix.target }}
path: dist/*.whl
build-wheels-macos:
name: Build wheels (macos / py${{ matrix.python-version }})
name: Build wheels (macos / universal2)
runs-on: macos-latest
if: (github.event_name == 'release' && github.event.action == 'published') || (github.event_name == 'workflow_dispatch' && inputs.release)
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v6
- name: Set up Python ${{ matrix.python-version }}
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
python-version: "3.10"
- name: Build universal2 wheel
- name: Build universal2 abi3 wheel
uses: PyO3/maturin-action@v1
with:
command: build
args: --release --out dist --compatibility pypi -i python
target: universal2-apple-darwin
args: --release --out dist
- name: Upload wheels as artifact
uses: actions/upload-artifact@v7
with:
name: wheels-macos-py${{ matrix.python-version }}
name: wheels-macos-universal2
path: dist/*.whl
build-wheels-windows:
name: Build wheels (windows / py${{ matrix.python-version }})
runs-on: windows-latest
name: Build wheels (windows / ${{ matrix.platform.arch }})
runs-on: ${{ matrix.platform.runner }}
if: (github.event_name == 'release' && github.event.action == 'published') || (github.event_name == 'workflow_dispatch' && inputs.release)
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
platform:
- runner: windows-latest
arch: x64
target: x64
- runner: windows-11-arm
arch: arm64
target: aarch64
steps:
- uses: actions/checkout@v6
- name: Set up Python ${{ matrix.python-version }}
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
python-version: "3.11"
architecture: ${{ matrix.platform.arch }}
- name: Build wheel
- name: Build abi3 wheel
uses: PyO3/maturin-action@v1
with:
command: build
args: --release --out dist --compatibility pypi -i python
target: ${{ matrix.platform.target }}
args: --release --out dist
- name: Upload wheels as artifact
uses: actions/upload-artifact@v7
with:
name: wheels-windows-py${{ matrix.python-version }}
name: wheels-windows-${{ matrix.platform.arch }}
path: dist/*.whl
build-sdist:
@@ -247,6 +282,7 @@ jobs:
runs-on: ubuntu-latest
needs:
- build-wheels-linux
- build-wheels-musllinux
- build-wheels-macos
- build-wheels-windows
- build-sdist
@@ -256,6 +292,8 @@ jobs:
url: https://pypi.org/p/ferro-ta
permissions:
id-token: write
attestations: write
contents: read
steps:
- name: Download all wheels
uses: actions/download-artifact@v8
@@ -270,6 +308,11 @@ jobs:
name: sdist
path: dist
- name: Generate SLSA build provenance attestations
uses: actions/attest-build-provenance@v2
with:
subject-path: "dist/*"
- name: Verify distribution coverage
run: |
python3 - <<'PY'
@@ -283,19 +326,15 @@ jobs:
for name in files:
print(f" - {name}")
# One abi3 wheel (cp310-abi3) per platform/arch — covers CPython 3.10+.
expected = [
"ferro_ta-*-cp310-cp310-manylinux*_x86_64.whl",
"ferro_ta-*-cp311-cp311-manylinux*_x86_64.whl",
"ferro_ta-*-cp312-cp312-manylinux*_x86_64.whl",
"ferro_ta-*-cp313-cp313-manylinux*_x86_64.whl",
"ferro_ta-*-cp310-cp310-win_amd64.whl",
"ferro_ta-*-cp311-cp311-win_amd64.whl",
"ferro_ta-*-cp312-cp312-win_amd64.whl",
"ferro_ta-*-cp313-cp313-win_amd64.whl",
"ferro_ta-*-cp310-cp310-macosx*_universal2.whl",
"ferro_ta-*-cp311-cp311-macosx*_universal2.whl",
"ferro_ta-*-cp312-cp312-macosx*_universal2.whl",
"ferro_ta-*-cp313-cp313-macosx*_universal2.whl",
"ferro_ta-*-cp310-abi3-manylinux*_x86_64.whl",
"ferro_ta-*-cp310-abi3-manylinux*_aarch64.whl",
"ferro_ta-*-cp310-abi3-musllinux*_x86_64.whl",
"ferro_ta-*-cp310-abi3-musllinux*_aarch64.whl",
"ferro_ta-*-cp310-abi3-macosx*_universal2.whl",
"ferro_ta-*-cp310-abi3-win_amd64.whl",
"ferro_ta-*-cp310-abi3-win_arm64.whl",
"ferro_ta-*.tar.gz",
]
@@ -346,6 +385,7 @@ jobs:
permissions:
contents: write
id-token: write
attestations: write
steps:
- uses: actions/checkout@v6
@@ -380,12 +420,40 @@ jobs:
- name: Generate Rust SBOM (CycloneDX)
run: cargo cyclonedx --format json --override-filename ferro-ta-rust-sbom.cdx
- name: Upload Python SBOM to release
uses: softprops/action-gh-release@v2
- name: Validate SBOMs (schema check)
run: |
python3 -c "import json, sys; json.load(open('ferro-ta-python-sbom.spdx.json')); print('Python SBOM valid JSON')"
python3 -c "import json, sys; data=json.load(open('ferro-ta-rust-sbom.cdx.json')); assert data.get('bomFormat')=='CycloneDX', 'not CycloneDX'; print('Rust SBOM valid CycloneDX')"
- name: Install cosign
uses: sigstore/cosign-installer@v3
- name: Sign SBOMs with cosign (keyless)
env:
COSIGN_EXPERIMENTAL: "1"
run: |
cosign sign-blob --yes ferro-ta-python-sbom.spdx.json --output-signature ferro-ta-python-sbom.spdx.json.sig --output-certificate ferro-ta-python-sbom.spdx.json.pem
cosign sign-blob --yes ferro-ta-rust-sbom.cdx.json --output-signature ferro-ta-rust-sbom.cdx.json.sig --output-certificate ferro-ta-rust-sbom.cdx.json.pem
- name: Attest SBOM provenance
uses: actions/attest-build-provenance@v2
with:
files: ferro-ta-python-sbom.spdx.json
subject-path: |
ferro-ta-python-sbom.spdx.json
ferro-ta-rust-sbom.cdx.json
- name: Upload Python SBOM to release
uses: softprops/action-gh-release@v3
with:
files: |
ferro-ta-python-sbom.spdx.json
ferro-ta-python-sbom.spdx.json.sig
ferro-ta-python-sbom.spdx.json.pem
- name: Upload Rust SBOM to release
uses: softprops/action-gh-release@v2
uses: softprops/action-gh-release@v3
with:
files: ferro-ta-rust-sbom.cdx.json
files: |
ferro-ta-rust-sbom.cdx.json
ferro-ta-rust-sbom.cdx.json.sig
ferro-ta-rust-sbom.cdx.json.pem
.github/workflows/ci-docs.yml
+1 -1
View File
@@ -44,6 +44,6 @@ jobs:
- name: Upload GitHub Pages artifact
if: ${{ inputs.upload-pages-artifact }}
uses: actions/upload-pages-artifact@v4
uses: actions/upload-pages-artifact@v5
with:
path: docs/_build/
.github/workflows/nightly-bench.yml
+100
View File
@@ -0,0 +1,100 @@
name: Nightly benchmarks
on:
schedule:
# 03:15 UTC every day — off peak, avoids colliding with the regular CI surge.
- cron: "15 3 * * *"
workflow_dispatch:
permissions:
contents: read
issues: write
jobs:
bench:
name: Run perf contract and regression checks
runs-on: ubuntu-latest
timeout-minutes: 60
env:
RUSTFLAGS: "" # override target-cpu=native from .cargo/config.toml
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@v1
with:
toolchain: stable
- uses: Swatinem/rust-cache@v2
- uses: actions/setup-python@v6
with:
python-version: "3.12"
- name: Install uv
run: pip install uv
- name: Build ferro-ta wheel
run: |
uv sync --extra dev
uv run maturin build --release --out dist
uv run pip install --force-reinstall dist/*.whl
- name: Run vs-TA-Lib benchmark
run: uv run python benchmarks/bench_vs_talib.py --output benchmarks/artifacts/nightly_vs_talib.json
- name: Run SIMD benchmark matrix
run: uv run python benchmarks/bench_simd.py --output benchmarks/artifacts/nightly_simd.json
- name: Check hotspot regression
id: hotspot
continue-on-error: true
run: uv run python benchmarks/check_hotspot_regression.py --tolerance 0.05
- name: Check vs-TA-Lib regression
id: vs_talib
continue-on-error: true
run: uv run python benchmarks/check_vs_talib_regression.py --tolerance 0.05
- name: Run criterion bench (build only)
run: cargo bench -p ferro_ta_core --no-run
- name: Upload nightly artifacts
uses: actions/upload-artifact@v7
if: always()
with:
name: nightly-bench-${{ github.run_id }}
path: benchmarks/artifacts/
retention-days: 30
- name: Open issue on regression
if: steps.hotspot.outcome == 'failure' || steps.vs_talib.outcome == 'failure'
uses: actions/github-script@v8
env:
HOTSPOT_OUTCOME: ${{ steps.hotspot.outcome }}
VS_TALIB_OUTCOME: ${{ steps.vs_talib.outcome }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
with:
script: |
const hotspot = process.env.HOTSPOT_OUTCOME;
const vsTalib = process.env.VS_TALIB_OUTCOME;
const runUrl = process.env.RUN_URL;
const failed = [];
if (hotspot === 'failure') failed.push('hotspot');
if (vsTalib === 'failure') failed.push('vs-TA-Lib');
const title = `Nightly benchmark regression: ${failed.join(' + ')}`;
const body = [
`The nightly benchmark workflow reported a >5% regression in: **${failed.join(', ')}**.`,
'',
`Run: ${runUrl}`,
'',
'Artifacts with the raw JSON are attached to the run above.',
'If this is a known regression, update the baseline JSON and close.',
'If it is unexpected, investigate the last commit on `main` before the nightly ran.',
].join('\n');
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title,
body,
labels: ['performance', 'regression', 'nightly'],
});
.github/workflows/release.yml
+1 -1
View File
@@ -52,7 +52,7 @@ jobs:
PY
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
uses: softprops/action-gh-release@v3
with:
tag_name: ${{ github.ref_name }}
name: v${{ steps.version.outputs.version }}