gavindiaz/wickra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Per-binding throughput benchmarks + test-coverage gaps (#246)

Browse Source 
Adds a `throughput` benchmark to every target and closes two small
test-coverage documentation/QA gaps. One PR, no merge of binding code beyond
the additive benchmarks and one C test.

## 1. Per-binding throughput benchmarks (all 9 targets)

Each benchmark feeds a deterministic synthetic OHLCV series through three
indicators chosen by **FFI call-signature archetype** (not algorithm — the same
Rust core runs underneath all bindings):

- `SMA(20)` — 1-in → 1-out (baseline boundary cost)
- `ATR(14)` — multi-in → 1-out (input marshalling)
- `MACD(12,26,9)` — 1-in → multi-out (output marshalling)

Streaming is timed for all three; batch for the single-output SMA and ATR
(median of 3 runs, after a warmup pass).

New: Python (PyO3), WASM, C (CMake), C# (Stopwatch), Go, Java (FFM), R, and the
Rust core baseline (`examples/rust/.../throughput.rs`, **no FFI** — the ceiling
the bindings are measured against and the value their batch paths converge
towards). Node already had `throughput.js`.

**Not a speed claim:** there is no comparable streaming TA library for C, C#,
Go, Java, R or WASM to compare against, so these are raw per-binding throughput
numbers documenting each language's FFI overhead — see BENCHMARKS.md §3. The
"Wickra is fast" claim still lives in §1/§2 (Rust core + the Python/Rust
cross-library runs).

## 2. README `## Testing`: C# and C bullets

The section listed every layer except C# and C, even though both have suites.
Adds the two missing bullets.

## 3. C archetype ctest

`examples/c/archetypes.c` drives one indicator per FFI archetype through the
real C boundary (scalar + batch==streaming, multi-output, bars, profile, array
input) plus reset, invalid-parameter and NULL-safety — the C counterpart of the
Go/R/Java archetype suites. Runs on three OSes via the existing CMake/ctest.

## Notes

- Benchmarks are not CI-gated (manual-run scripts, like the existing
  `throughput.js`); no `ci.yml`/`release.yml` changes.
- Docs: BENCHMARKS.md §3, a `## Benchmark` section in every binding README, a
  CHANGELOG entry.
- Verified locally by running: Rust, Python, C, C#, Go, Java (real numbers); the
  C archetype ctest with `-Wall -Wextra -Wpedantic -Werror`. WASM and R are
  API-correct and syntax-checked but need their own toolchains to run.
This commit is contained in:
kingchenc
2026-06-10 03:46:38 +02:00
committed by GitHub
parent b31a9a3624
commit 3ebcb3f758
25 changed files with 1529 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+72
View File
@@ -94,3 +94,75 @@ cargo bench -p wickra-bench # Rust core vs kand / ta-rs / yata
pip install -e bindings/python[bench] # Python peers
python -m benchmarks.compare_libraries
```
## 3. Per-binding throughput — the cost of the boundary
The sections above compare Wickra against other libraries, which only exists for
Python and Rust (there is no comparable streaming TA library for C, C#, Go, Java,
R or WebAssembly to benchmark against). Every binding calls the **same** Rust
core, so these per-binding benchmarks are **not** a speed claim and **not** a
cross-library ratio — they document the raw cost of crossing each language's FFI
boundary, in million updates per second (Mupd/s).
Each binding ships a small `throughput` benchmark that feeds a synthetic OHLCV
series through three indicators chosen by call-signature archetype — `SMA(20)`
(1-in → 1-out), `ATR(14)` (multi-in → 1-out) and `MACD(12,26,9)` (1-in →
multi-out). Two things fall out of the numbers:
- **Batch converges.** A `batch` call crosses the boundary once and the Rust core
computes the whole series internally, so batch throughput is roughly the same
in every binding — close to the core speed.
- **Streaming reveals the boundary.** A per-tick `update` crosses the boundary
once per value, so streaming throughput is where the bindings differ: the raw C
ABI and P/Invoke-style calls are nearly free, while managed or interpreted
per-call marshalling (cgo, FFM, the R/WASM boundary) costs more per tick.
The Rust core ships the same benchmark with **no** FFI boundary
(`examples/rust/.../throughput.rs`) — it is the ceiling each binding is measured
against and the value the batch paths converge towards.
`SMA(20)`, 200 000 bars, median of 3 runs, on the reference machine (Windows 11,
AMD Ryzen 9 9950X):
| Target | streaming (Mupd/s) | batch (Mupd/s) |
|----------------------|-------------------:|---------------:|
| Rust core (no FFI) | 391 | 500 |
| C | 383 | 330 |
| C# / .NET | 337 | 244 |
| Python | 33 | 488 |
| Java | 28 | 175 |
| Go | 24 | 400 |
| WebAssembly | 19 | 167 |
| Node.js | 17 | 10 |
| R | 0.1 | 193 |
Streaming spans more than three orders of magnitude — the raw C ABI (383) is
nearly the FFI-free Rust ceiling (391), while R's per-call interpreter overhead
(0.1) makes streaming ~2000× slower than its own batch. Batch converges near the
core speed for the zero-copy bindings (numpy, slices, typed arrays); the two
outliers are Node — whose napi `batch` boxes every element into a JS `Array` —
and R. These are machine-dependent and reflect FFI overhead, not algorithm
speed.
These are throughput numbers, not competitive numbers — the "Wickra is fast"
claim lives in sections 1 and 2 (Rust core + the Python/Rust cross-library runs).
Run any target's benchmark (build the C ABI library first where it links one):
```bash
cargo run -p wickra-examples --release --bin throughput # Rust core baseline (no FFI)
node bindings/node/benchmarks/throughput.js # native napi-rs
( cd bindings/python && python -m benchmarks.throughput ) # native PyO3
( cd bindings/wasm && wasm-pack build --target nodejs --out-dir pkg-node --release ) \
&& node bindings/wasm/benchmarks/throughput.mjs # wasm boundary
cargo build -p wickra-c --release # the C ABI hub
cmake -S bindings/c/benchmarks -B build/cbench && cmake --build build/cbench \
&& ./build/cbench/throughput # raw C ABI
dotnet run -c Release --project bindings/csharp/benchmarks # C# / .NET (P/Invoke)
( cd bindings/go/benchmarks && go run . ) # Go (cgo)
mvn -q -f bindings/java install -DskipTests \
&& mvn -q -f bindings/java/benchmarks exec:exec -Dexec.mainClass=org.wickra.benchmarks.Throughput
Rscript bindings/r/benchmarks/throughput.R # R (.Call)
```