mirror of
https://github.com/KhizarImran/backtestingfx.git
synced 2026-11-02 03:26:11 +00:00
perf: reduce Python callback overhead and verify matched benchmarks
Reuse the Python callback method-name string and expose constant-time position count and presence. Preserve dynamic callbacks, Bar identity, and snapshot behavior with regression tests. Add isolated callback measurements and replace approximate cross-library comparisons with matched trades and equity assertions, separate setup/run timings, and reproducible reports. Track the remaining ingestion bottleneck in #53. Refs #48, #38, #39, #53 Codex 🤖, On behalf of Khizar
This commit is contained in:
Khizar Imran
@@ -3,12 +3,22 @@
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- Constant-time Python `Strategy` and `Broker` properties `has_positions` and
|
||||
`position_count`, avoiding position-list copies for presence and count checks.
|
||||
- A callback benchmark with separate conversion, engine construction,
|
||||
initialization, and simulation timings, native and Python no-op baselines,
|
||||
indicator and position-access workloads, correctness checks, and peak RSS.
|
||||
- Documented and regression-tested the stop-first intrabar SL/TP policy,
|
||||
inclusive touches and next-bar checks for new entries.
|
||||
- Contributor setup and validation guidance, bug/feature issue templates, an
|
||||
API/platform compatibility policy, and a prioritized FX roadmap.
|
||||
|
||||
### Fixed
|
||||
- Python strategy dispatch reuses the Python method-name string for `next`,
|
||||
while preserving per-bar method lookup, distinct Bar objects, and exceptions.
|
||||
- The existing cross-library comparison and benchmark now use matched fixed-unit
|
||||
next-open strategies, verify trades and full equity curves, and report setup,
|
||||
simulation, and total timings separately. Spread is zero; commission is matched.
|
||||
- Python constructors reject nonfinite or out-of-range broker settings. Order,
|
||||
partial-close, stop-update, closing-price, and equity-price inputs are checked
|
||||
before account mutation. Finite nonpositive partial sizes remain no-ops.
|
||||
|
||||
+4
-2
@@ -71,8 +71,10 @@ SMA and optimization examples use the included `data/EURUSD_1H.csv`.
|
||||
The HTML example generates synthetic data, writes
|
||||
`backtestingfx-report.html`, and attempts to open a browser. It requires the
|
||||
`report` extra. Comparison and benchmark scripts additionally require
|
||||
`python -m pip install backtesting`; report their versions and execution
|
||||
assumptions when sharing results.
|
||||
`python -m pip install backtesting==0.6.6`; report their versions and execution
|
||||
assumptions when sharing results. See [the matched comparison](examples/benchmark.md)
|
||||
for correctness checks and separate setup/run timings. Reference-comparison tests
|
||||
skip when the optional dependency is absent.
|
||||
|
||||
## Pull requests
|
||||
|
||||
|
||||
@@ -103,6 +103,21 @@ class MyStrategy(Strategy):
|
||||
| `self.close_partial(id, lot_size)` | Close part of a position, leaving the rest open |
|
||||
| `self.update_sl(id, stop_loss)` | Move a position's stop loss (returns `False` if the id is gone) |
|
||||
|
||||
### Position inspection
|
||||
|
||||
Use `self.has_positions` to check whether any positions are open, or
|
||||
`self.position_count` to read their count. Both read the current broker state
|
||||
in constant time without copying positions. Before a run, they return `False`
|
||||
and `0`. The direct Python `Broker` API provides the same read-only properties.
|
||||
|
||||
`self.positions` still returns a list of position snapshots. Use it when you
|
||||
need IDs, sizes, or stop levels. For example:
|
||||
|
||||
```python
|
||||
if not self.has_positions:
|
||||
self.buy(0.1)
|
||||
```
|
||||
|
||||
### Trailing stops and scaling out
|
||||
|
||||
`update_sl` moves the stop on an open position in place, so a trailing stop costs
|
||||
@@ -113,7 +128,7 @@ keep trailing it.
|
||||
```python
|
||||
class TrailingBreakout(Strategy):
|
||||
def next(self):
|
||||
if not self.positions:
|
||||
if not self.has_positions:
|
||||
self.buy(1.0, stop_loss=self._bar.close - 0.0050)
|
||||
return
|
||||
|
||||
@@ -181,6 +196,12 @@ stops, pyramiding, "exit after N bars") still needs `next()` and a plain loop. I
|
||||
warmup must come out as `0.0`, not `NaN` — a `NaN` signal is rejected rather than
|
||||
silently treated as "hold".
|
||||
|
||||
Precompute indicators once in `init()` when their values depend only on market
|
||||
data, then read the current value in `next()`. Python callbacks remain useful
|
||||
for decisions that depend on fills, cash, trailing stops, or partial closes.
|
||||
The [callback benchmark](examples/benchmark_callbacks.md) separates conversion,
|
||||
initialization, and simulation costs and compares these workloads.
|
||||
|
||||
## Backtest Parameters
|
||||
|
||||
```python
|
||||
@@ -360,6 +381,10 @@ in `Strategy::init()` or construct a fresh strategy for each run.
|
||||
|
||||
The backtesting engine is written in Rust and compiled as a native Python extension via [PyO3](https://pyo3.rs). This means the event loop, broker simulation, and stats computation run at native speed while your strategy stays in plain Python.
|
||||
|
||||
The [matched backtesting.py comparison](examples/benchmark.md) checks trades and
|
||||
full equity curves before timing both libraries. Current local results show a
|
||||
faster FX run phase but slower total runtime because data conversion dominates.
|
||||
|
||||
## Contributing and project direction
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for local setup, checks, and examples,
|
||||
|
||||
@@ -44,6 +44,14 @@ class Strategy:
|
||||
def positions(self):
|
||||
return self._broker.positions() if self._broker else []
|
||||
|
||||
@property
|
||||
def position_count(self):
|
||||
return self._broker.position_count if self._broker is not None else 0
|
||||
|
||||
@property
|
||||
def has_positions(self):
|
||||
return self._broker.has_positions if self._broker is not None else False
|
||||
|
||||
@property
|
||||
def data(self):
|
||||
if not self._bars:
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Matched comparison with backtesting.py
|
||||
|
||||
`compare_bt.py` verifies equivalent accounting before `benchmark.py` compares
|
||||
speed. These replace the earlier scripts that used different position sizes,
|
||||
fill timing, and SMA calculations and only printed approximate results.
|
||||
|
||||
## Run the comparison
|
||||
|
||||
From the repository root, with your Python environment activated:
|
||||
|
||||
```bash
|
||||
python -m pip install ".[report]" "backtesting==0.6.6"
|
||||
repo_dir="$PWD"
|
||||
cd /tmp
|
||||
python "$repo_dir/examples/compare_bt.py" --commission 7
|
||||
python "$repo_dir/examples/benchmark.py" \
|
||||
--sizes 2000 10000 50000 --commissions 0 7 --repeats 5 \
|
||||
--output matched-comparison.json
|
||||
python -m unittest discover -s "$repo_dir/tests" -p test_reference_comparison.py
|
||||
```
|
||||
|
||||
The FX extension uses maturin's default release build. Reinstall after source
|
||||
changes and run outside the checkout to avoid loading an old local extension.
|
||||
The reference dependency is optional; the comparison tests skip when it is not
|
||||
installed. The JSON report records library versions, CPU, OS, Python, Rust,
|
||||
source HEAD, every timing sample, medians, and matching trade counts/equity.
|
||||
|
||||
## Shared strategy and execution rules
|
||||
|
||||
Both libraries use identical OHLCV values and UTC timestamps from the bundled
|
||||
EURUSD data. Larger datasets tile that sample and regenerate a monotonic hourly
|
||||
index. They are scaling fixtures, not additional independent historical data.
|
||||
|
||||
- Compute the same trailing SMA 10/50 decisions once with pandas before either
|
||||
library runs. The precomputation uses past and current closes only. The warmup
|
||||
is flat. Both callbacks read this shared array without recalculating indicators.
|
||||
- Start with `$10,000` and use exactly `1,000` units, equivalent to `0.01` FX lots
|
||||
with contract size `100,000`. Keep at most one long position. No shorting,
|
||||
pyramiding, stops, or targets are used.
|
||||
- Enter when the fast SMA is above the slow SMA while flat. Exit when it is
|
||||
below while long. Decisions at bar `i` fill at bar `i+1`'s open.
|
||||
- Queue a final close from the penultimate bar so both libraries finish flat at
|
||||
the final bar's open. Do not open a new position from the final two decisions.
|
||||
- Use USD quote/account currency with conversion rate `1`. Test commission `0`
|
||||
and `$7` per lot per side. The reference commission callable charges
|
||||
`abs(units) * 7 / 100000` per fill, matching the FX commission model.
|
||||
- Set spread to zero. The libraries define nonzero spread differently, so this
|
||||
comparison does not claim equivalence for nonzero spread.
|
||||
|
||||
backtesting.py uses its ordinary queued market orders with
|
||||
`trade_on_close=False`, `hedging=False`, and no automatic finalization. A
|
||||
standalone opposite order flattens the long. Backtestingfx's normal `buy()` and
|
||||
`close_all()` fill immediately at the current close, so the comparison strategy
|
||||
explicitly applies the previous decision through `Broker.buy()` or
|
||||
`Broker.close_all()` at the current open. This is comparison strategy code,
|
||||
not a new next-open engine option. Neither library's broker implementation is
|
||||
patched. Both execute via their normal public `Backtest.run()` entry point.
|
||||
See the [reference execution options](https://kernc.github.io/backtesting.py/doc/backtesting/backtesting.html).
|
||||
|
||||
The reference enforces margin while FX currently does not. Fixed exposure on
|
||||
these datasets remains affordable, and the equivalence checks fail if that
|
||||
difference affects executions. This does not establish margin-model equivalence.
|
||||
|
||||
## What must agree
|
||||
|
||||
Before timing, and again after every measured pair, assertions compare trade
|
||||
count, direction, size, entry/exit timestamps, entry/exit prices, per-trade net
|
||||
PnL, the entire bar-by-bar equity curve, final equity, return, win rate for
|
||||
nonempty trades, and maximum drawdown. Monetary/price comparisons use absolute
|
||||
tolerance `1e-8`, not rounded output. Unit sizes use `1e-10` tolerance.
|
||||
|
||||
FX prepends an initial-cash observation, so its first extra observation is
|
||||
removed when aligning curves. Reference drawdown is negative while FX reports
|
||||
its positive magnitude. No prices, timestamps, or curve observations are shifted
|
||||
to make the assertions pass.
|
||||
|
||||
A seven-bar fixture has distinct opens and closes and two hand-calculated trades.
|
||||
It checks next-open fills, final-open liquidation, commission, and every equity
|
||||
observation. With commission `7`, trades enter at `1.2` and `1.5`, exit at `1.3`
|
||||
and `1.6`, and each nets `$99.86`, ending at `$10,199.72`. A separate test
|
||||
deliberately corrupts an entry price and proves the checker rejects it despite
|
||||
the unchanged trade count.
|
||||
|
||||
This establishes equivalent accounting for this strategy and these settings.
|
||||
It does not establish equivalence for arbitrary strategies, SL/TP policies,
|
||||
leverage, nonzero spread, financing, or all reported statistics. For example,
|
||||
the libraries use different Sharpe/annualization definitions.
|
||||
|
||||
## Timing boundaries
|
||||
|
||||
Load/normalize the input and compute the shared SMA decisions once outside
|
||||
both library timers. Record this common preparation separately in JSON.
|
||||
For each sample, construct a fresh `Backtest` and strategy and call `run()`.
|
||||
Alternate library order between repetitions. There is one untimed warmup and
|
||||
five measured pairs per dataset and commission setting. Report medians.
|
||||
|
||||
- **Setup:** from before the `Backtest` constructor through the end of strategy
|
||||
`init()`. This includes each library's validation and data conversion, and FX's
|
||||
DataFrame-to-Bar and initialization-history conversion.
|
||||
- **Run:** from the end of `init()` until `run()` returns, including simulation,
|
||||
finalization, and that library's normal statistics calculation.
|
||||
- **Total:** the entire constructor plus `run()` call. This is the primary number
|
||||
for deciding which library completes this workload faster.
|
||||
|
||||
The timing marker executes once during initialization. Assertions and printing
|
||||
are outside the timing. The libraries compute different numbers of statistics;
|
||||
the run-phase ratio is not a pure engine-loop speedup. Neither plotting nor
|
||||
optimization is measured. Use [benchmark_callbacks.py](benchmark_callbacks.md)
|
||||
to isolate callback overhead inside backtestingfx.
|
||||
|
||||
## Local results
|
||||
|
||||
Measured on an Intel Core i5-8350U, Linux x86_64, CPython 3.14.4, Rust 1.98.1,
|
||||
PyO3 0.28.3, pandas 3.1.0, NumPy 2.5.4, and backtesting.py 0.6.6. The installed
|
||||
FX release build includes the issue #48 callback optimization and lightweight
|
||||
position properties. All equivalence checks passed for every measured pair.
|
||||
|
||||
| Bars | Commission per lot/side | Matching trades | Final equity |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 2,000 | 0 | 34 | 9975.09 |
|
||||
| 2,000 | 7 | 34 | 9970.33 |
|
||||
| 10,000 | 0 | 174 | 9862.73 |
|
||||
| 10,000 | 7 | 174 | 9838.37 |
|
||||
| 50,000 | 0 | 874 | 9300.93 |
|
||||
| 50,000 | 7 | 874 | 9178.57 |
|
||||
|
||||
Times below are milliseconds. Setup, run, and total are independently calculated
|
||||
medians, so rounded setup plus run may not exactly equal the total.
|
||||
|
||||
| Bars | Commission | backtesting.py setup | Run | Total | backtestingfx setup | Run | Total |
|
||||
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 2,000 | 0 | 1.1 | 20.0 | 21.1 | 79.3 | 1.4 | 80.7 |
|
||||
| 2,000 | 7 | 1.2 | 20.7 | 21.9 | 75.5 | 1.4 | 76.9 |
|
||||
| 10,000 | 0 | 1.3 | 56.3 | 57.6 | 393.1 | 7.2 | 400.3 |
|
||||
| 10,000 | 7 | 1.2 | 54.9 | 56.0 | 383.7 | 7.5 | 391.9 |
|
||||
| 50,000 | 0 | 1.5 | 240.0 | 241.5 | 1970.0 | 42.3 | 2011.6 |
|
||||
| 50,000 | 7 | 1.3 | 237.6 | 239.1 | 1925.1 | 40.5 | 1967.8 |
|
||||
|
||||
For 50,000 bars with commission, backtestingfx's run phase was about 5.9 times
|
||||
faster, but its total was about 8.2 times slower. Setup, especially row-by-row
|
||||
DataFrame ingestion, dominated. Optimizing that conversion is tracked in #39.
|
||||
These results do not support a claim that backtestingfx is faster end to end
|
||||
for this workload. Timing varies with versions, hardware, and system load.
|
||||
+70
-135
@@ -1,146 +1,81 @@
|
||||
"""
|
||||
Speed benchmark: backtesting.py vs backtestingfx, on the same SMA-cross strategy.
|
||||
"""Time the matched comparison from compare_bt.py only after equivalence passes.
|
||||
|
||||
Three paths are timed:
|
||||
1. backtesting.py — everything in Python
|
||||
2. backtestingfx (py) — Rust engine + broker, but next() runs in Python each bar
|
||||
3. backtestingfx (rust) — SMA strategy runs entirely in Rust, no per-bar Python call
|
||||
|
||||
The dataset is small (2000 bars), so it's tiled up to BARS to give the engines
|
||||
real work. Timestamps are regenerated as a clean hourly range so backtesting.py
|
||||
gets the sorted DatetimeIndex it needs.
|
||||
|
||||
Note: paths 1 and 2 both pay the per-bar Python cost (the strategy logic), so the
|
||||
gap between them measures only the engine/broker plumbing. Path 3 shows the
|
||||
engine's ceiling when nothing crosses into Python.
|
||||
Run against installed release packages outside the checkout. See benchmark.md.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import time
|
||||
|
||||
import pandas as pd
|
||||
from backtesting import Backtest as BtBacktest, Strategy as BtStrategy
|
||||
from backtestingfx import Backtest, Strategy
|
||||
from backtestingfx import _backtestingfx as _rust
|
||||
|
||||
FAST = 10
|
||||
SLOW = 50
|
||||
LOT = 0.1
|
||||
BARS = 50_000
|
||||
REPEATS = 2
|
||||
from benchmark_callbacks import metadata
|
||||
from compare_bt import Comparison, load_data
|
||||
|
||||
|
||||
class SmaCrossBt(BtStrategy):
|
||||
def init(self):
|
||||
pass
|
||||
|
||||
def next(self):
|
||||
if len(self.data.Close) < SLOW:
|
||||
return
|
||||
fast_sma = self.data.Close[-FAST:].mean()
|
||||
slow_sma = self.data.Close[-SLOW:].mean()
|
||||
if not self.position:
|
||||
if fast_sma > slow_sma:
|
||||
self.buy()
|
||||
elif fast_sma < slow_sma:
|
||||
self.position.close()
|
||||
def measure(size, commission, repeats):
|
||||
start = time.perf_counter()
|
||||
comparison = Comparison(load_data(size), commission=commission)
|
||||
common_setup = time.perf_counter() - start
|
||||
# Untimed equivalence check and warmup before collecting any timing samples.
|
||||
reference, _ = comparison.run("backtesting.py")
|
||||
actual, _ = comparison.run("backtestingfx")
|
||||
comparison.verify(reference, actual)
|
||||
samples = {name: [] for name in ("backtesting.py", "backtestingfx")}
|
||||
for repetition in range(repeats):
|
||||
order = list(samples)
|
||||
if repetition % 2:
|
||||
order.reverse()
|
||||
results = {}
|
||||
for name in order:
|
||||
result, timings = comparison.run(name)
|
||||
results[name] = result
|
||||
samples[name].append(timings)
|
||||
# Every measured pair must also agree. Assertions are outside timing.
|
||||
comparison.verify(results["backtesting.py"], results["backtestingfx"])
|
||||
report = {"bars": size, "commission_per_lot_per_side": commission,
|
||||
"common_preparation_seconds": common_setup,
|
||||
"matching_trades": actual.num_trades, "final_equity": actual.final_cash,
|
||||
"equivalence": "passed", "libraries": {}}
|
||||
for name, times in samples.items():
|
||||
report["libraries"][name] = {
|
||||
**{key: statistics.median(row[key] for row in times) for key in times[0]},
|
||||
"samples": times,
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
class SmaCrossFx(Strategy):
|
||||
def next(self):
|
||||
if self.index < SLOW:
|
||||
return
|
||||
closes = [b.close for b in self.data[-SLOW:]]
|
||||
fast_sma = sum(closes[-FAST:]) / FAST
|
||||
slow_sma = sum(closes) / SLOW
|
||||
if not self.positions:
|
||||
if fast_sma > slow_sma:
|
||||
self.buy(LOT)
|
||||
elif fast_sma < slow_sma:
|
||||
self.close_all()
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--sizes", type=int, nargs="+", default=[2000, 10_000, 50_000])
|
||||
parser.add_argument("--commissions", type=float, nargs="+", default=[0.0, 7.0])
|
||||
parser.add_argument("--repeats", type=int, default=5)
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.repeats < 1 or any(size < 3 for size in args.sizes):
|
||||
parser.error("sizes must be at least three and repeats must be positive")
|
||||
report = {"metadata": metadata(), "repeats": args.repeats, "results": []}
|
||||
report["metadata"]["package_versions"]["backtesting"] = importlib.metadata.version("backtesting")
|
||||
print("Times are medians; run includes simulation, finalization, and each library's statistics.")
|
||||
for size in args.sizes:
|
||||
for commission in args.commissions:
|
||||
result = measure(size, commission, args.repeats)
|
||||
report["results"].append(result)
|
||||
print(f"\nPASS: {size:,} bars, commission {commission}/lot/side, "
|
||||
f"{result['matching_trades']} matching trades, equity {result['final_equity']:.8f}",
|
||||
flush=True)
|
||||
for name, timings in result["libraries"].items():
|
||||
print(f" {name:<15} setup {timings['setup_seconds']:.4f} s "
|
||||
f"run {timings['simulation_seconds']:.4f} s "
|
||||
f"total {timings['total_seconds']:.4f} s", flush=True)
|
||||
bt = result["libraries"]["backtesting.py"]
|
||||
fx = result["libraries"]["backtestingfx"]
|
||||
print(f" backtestingfx speed ratio: run {bt['simulation_seconds'] / fx['simulation_seconds']:.2f}x; "
|
||||
f"total {bt['total_seconds'] / fx['total_seconds']:.2f}x. Above 1 means faster.", flush=True)
|
||||
if args.output:
|
||||
args.output.write_text(json.dumps(report, indent=2) + "\n")
|
||||
|
||||
|
||||
def load_tiled(n):
|
||||
"""Load the EURUSD CSV and tile it up to n rows with a fresh hourly index."""
|
||||
df = pd.read_csv("data/EURUSD_1H.csv")[["open", "high", "low", "close", "volume"]]
|
||||
reps = -(-n // len(df)) # ceil division
|
||||
df = pd.concat([df] * reps, ignore_index=True).iloc[:n].copy()
|
||||
df["timestamp"] = pd.date_range("2000-01-01", periods=n, freq="h")
|
||||
return df
|
||||
|
||||
|
||||
def best_of(fn, repeats=REPEATS):
|
||||
"""Run fn repeats times, return (best_seconds, result_of_first_run)."""
|
||||
best = float("inf")
|
||||
result = None
|
||||
for i in range(repeats):
|
||||
t0 = time.perf_counter()
|
||||
r = fn()
|
||||
dt = time.perf_counter() - t0
|
||||
if i == 0:
|
||||
result = r
|
||||
best = min(best, dt)
|
||||
return best, result
|
||||
|
||||
|
||||
df = load_tiled(BARS)
|
||||
|
||||
# backtesting.py wants a DatetimeIndex and capitalised columns
|
||||
df_bt = df.set_index("timestamp").rename(
|
||||
columns={"open": "Open", "high": "High", "low": "Low", "close": "Close", "volume": "Volume"}
|
||||
)
|
||||
|
||||
# native-rust path needs Bar objects built directly
|
||||
bars = [
|
||||
_rust.Bar(
|
||||
timestamp=int(ts.timestamp()),
|
||||
open=float(o), high=float(h), low=float(l), close=float(c), volume=float(v),
|
||||
)
|
||||
for ts, o, h, l, c, v in zip(
|
||||
df["timestamp"], df["open"], df["high"], df["low"], df["close"], df["volume"]
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def run_bt():
|
||||
return BtBacktest(df_bt, SmaCrossBt, cash=10_000, commission=0).run()
|
||||
|
||||
|
||||
def run_fx_py():
|
||||
return Backtest(df, SmaCrossFx, cash=10_000, commission=0, spread=0).run()
|
||||
|
||||
|
||||
def run_fx_rust():
|
||||
engine = _rust.Engine(bars, 10_000.0, 0.0, 0.0, 100_000.0, 1.0)
|
||||
return engine.run_native_sma(FAST, SLOW, LOT)
|
||||
|
||||
|
||||
print(f"Benchmarking on {BARS:,} bars, best of {REPEATS} runs...\n", flush=True)
|
||||
|
||||
print(" timing backtesting.py ...", flush=True)
|
||||
t_bt, r_bt = best_of(run_bt)
|
||||
print(" timing backtestingfx (py next) ...", flush=True)
|
||||
t_py, r_py = best_of(run_fx_py)
|
||||
print(" timing backtestingfx (rust) ...\n", flush=True)
|
||||
t_rs, r_rs = best_of(run_fx_rust)
|
||||
|
||||
print("=" * 62)
|
||||
print(f"{'Path':<24}{'Time (s)':>10}{'Bars/sec':>14}{'Speedup':>12}")
|
||||
print("=" * 62)
|
||||
for name, t in [
|
||||
("backtesting.py", t_bt),
|
||||
("backtestingfx (py next)", t_py),
|
||||
("backtestingfx (rust)", t_rs),
|
||||
]:
|
||||
print(f"{name:<24}{t:>10.3f}{BARS / t:>14,.0f}{t_bt / t:>11.1f}x")
|
||||
print("=" * 62)
|
||||
|
||||
print("\nCorrectness (should roughly agree):")
|
||||
print(f"{'Path':<24}{'Trades':>10}{'Return %':>12}")
|
||||
print(f"{'backtesting.py':<24}{r_bt['# Trades']:>10}{r_bt['Return [%]']:>12.2f}")
|
||||
print(f"{'backtestingfx (py next)':<24}{r_py.num_trades:>10}{r_py.total_return_pct:>12.2f}")
|
||||
print(f"{'backtestingfx (rust)':<24}{r_rs.num_trades:>10}{r_rs.total_return_pct:>12.2f}")
|
||||
print(
|
||||
"\nNote: py-next vs rust measure the same engine; the gap is the per-bar\n"
|
||||
"Python callback cost. Both use fixed lots, so Return % differs from\n"
|
||||
"backtesting.py's cash-based sizing."
|
||||
)
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# Python callback benchmark
|
||||
|
||||
This benchmark measures the Rust-to-Python callback path separately from data
|
||||
conversion. It also compares indicator work, trading, and position inspection.
|
||||
It uses the real Python `_Adapter` and the normal Rust broker, equity, statistics,
|
||||
and final-liquidation code. It does not compare different backtesting libraries.
|
||||
|
||||
## Run the benchmark
|
||||
|
||||
From the repository root, with your Python environment activated:
|
||||
|
||||
```bash
|
||||
python -m pip install .
|
||||
repo_dir="$PWD"
|
||||
cd /tmp
|
||||
python "$repo_dir/examples/benchmark_callbacks.py" \
|
||||
--sizes 1000 10000 100000 --repeats 7 --conversion-repeats 3 \
|
||||
--label after --output callback-after.json
|
||||
```
|
||||
|
||||
The normal pip build uses maturin's release profile. Reinstall after Rust or
|
||||
Python source changes. Run outside the checkout to use the installed extension.
|
||||
Keep correctness assertions enabled; do not run with `python -O`.
|
||||
Use `--cases native-noop direct-noop adapter-noop` for a narrower comparison.
|
||||
|
||||
Each dataset runs in a fresh subprocess. JSON output records all simulation
|
||||
samples, medians for every phase, peak RSS, CPU, OS, Python, Rust, dependency
|
||||
versions, the Cargo.lock PyO3 version, and the source checkout's HEAD. The HEAD
|
||||
identifies the source checkout, not the installed wheel. Use `--label` to identify
|
||||
working-tree builds, and make sure the installed wheel matches your source.
|
||||
Peak RSS is unavailable on platforms without Python's `resource` module.
|
||||
|
||||
## Timing boundaries
|
||||
|
||||
All paths use the same prepared midpoint bars and cost configuration. Synthetic
|
||||
prices are flat at `1.1`, with a `0.0001` spread, commission `7` per lot per side,
|
||||
contract size `100000`, and quote-to-account conversion `1.27`.
|
||||
|
||||
1. **Conversion:** time `Backtest._to_bars()` separately. DataFrame construction
|
||||
is outside this measurement. Use the last conversion's bars for every case.
|
||||
2. **Engine setup:** construct a fresh `Engine` from those bars for every sample.
|
||||
Record this separately from the run. Construct a fresh strategy and adapter
|
||||
outside the run timer too.
|
||||
3. **Initialization:** start the run timer before `Engine.run()`. A single timestamp
|
||||
at the end of the real adapter's `init()` marks the boundary. This includes
|
||||
Rust-to-Python conversion of the initialization history and any indicator
|
||||
precomputation. Broker creation follows this boundary and is part of the run.
|
||||
4. **Simulation:** measure from that timestamp until `Engine.run()` returns.
|
||||
Include callbacks, SL/TP scans, equity, final liquidation, statistics, and
|
||||
conversion of results back to Python. No timing calls are added per bar.
|
||||
|
||||
The native no-op has no Python initialization. Its measured run includes its
|
||||
empty native `init()` and the normal engine setup inside `Engine::run()`, plus
|
||||
the same broker, equity, finalization, and statistics paths. The private
|
||||
`Engine._run_native_noop()` method is a benchmark hook, not a supported strategy API.
|
||||
|
||||
There is one untimed warmup followed by seven measured runs per case. Reported
|
||||
times are medians. No engines, strategy instances, or brokers carry state between
|
||||
samples. Correctness assertions run after timing. They check trade count, fills,
|
||||
net PnL, final cash, equity length and endpoints, no-op equity, final indicator
|
||||
values, and position inspection. Trading uses fixed `0.001` lots.
|
||||
|
||||
Peak RSS is the whole dataset subprocess's high-water mark after all cases,
|
||||
including imports, conversion, bars, equity, trades, and correctness checks. It is not per-case
|
||||
allocation or the memory cost of a single callback. Native memory is included.
|
||||
|
||||
## Local results
|
||||
|
||||
Measured on an Intel Core i5-8350U, Linux x86_64, CPython 3.14.4, Rust 1.98.1,
|
||||
PyO3 0.28.3, pandas 3.1.0, and NumPy 2.5.4, with default Cargo release settings.
|
||||
Both versions used the same dependency versions, benchmark, data, and costs.
|
||||
The baseline used dispatch from `d9687e7` with the native no-op benchmark hook
|
||||
added. The compatible version reuses the Python method-name string and adds
|
||||
position count/presence properties. It keeps dynamic callback lookup and fresh
|
||||
callback Bar objects.
|
||||
|
||||
| Bars | Adapter no-op before (ns/bar) | After (ns/bar) | Run-time reduction |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 1,000 | 290.7 | 201.4 | 30.7% |
|
||||
| 10,000 | 294.8 | 204.4 | 30.7% |
|
||||
| 100,000 | 311.1 | 229.7 | 26.1% |
|
||||
|
||||
At 100,000 bars, the compatible build measured:
|
||||
|
||||
| Workload | Initialization (ms) | Simulation (ms) | Simulation (ns/bar) |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Native no-op | 0.00 | 1.22 | 12.2 |
|
||||
| Direct Python no-op | 6.63 | 15.77 | 157.7 |
|
||||
| Python adapter no-op | 5.23 | 22.97 | 229.7 |
|
||||
| Rolling 20-bar mean in `next()` | 4.76 | 330.45 | 3304.5 |
|
||||
| Precompute mean in `init()` | 15.80 | 36.60 | 366.0 |
|
||||
| Open and close one trade per bar | 4.98 | 83.81 | 838.1 |
|
||||
| Hold 32 positions without inspection | 3.10 | 38.71 | 387.1 |
|
||||
| Read `len(self.positions)` with 32 positions | 3.00 | 230.73 | 2307.3 |
|
||||
| Read `self.position_count` with 32 positions | 3.01 | 52.87 | 528.7 |
|
||||
| Read `self.has_positions` with 32 positions | 3.09 | 54.81 | 548.1 |
|
||||
|
||||
The direct-to-adapter difference is about 72 ns/bar here. It includes adapter
|
||||
state updates and the extra `Strategy.next()` dispatch. List inspection is much
|
||||
more expensive with 32 positions because it copies the position vector and
|
||||
creates Python position objects. The presence workload takes about 4.2 times
|
||||
less simulation time than list inspection on this fixture. The read workloads
|
||||
share the same subclass dispatch; the hold-only case omits that extra method call.
|
||||
|
||||
At this size, constructing the engine for the adapter no-op took about 4.92 ms,
|
||||
and DataFrame-to-Bar conversion took 3.93 seconds. The 26.1% reduction applies
|
||||
to the measured callback run, not to a complete `Backtest.run()`. Conversion
|
||||
still dominates that workload and is tracked separately in #39.
|
||||
|
||||
| Bars | Peak RSS before (MiB) | After (MiB) |
|
||||
| ---: | ---: | ---: |
|
||||
| 1,000 | 78.53 | 78.37 |
|
||||
| 10,000 | 83.50 | 84.19 |
|
||||
| 100,000 | 135.88 | 135.82 |
|
||||
|
||||
These whole-process high-water marks do not establish a memory improvement.
|
||||
The after run includes the new count/presence cases too. Hardware, allocator
|
||||
state, Python version, and system load affect timings. Differences between rows
|
||||
estimate workload costs; they are not additive profiler measurements.
|
||||
|
||||
## Dispatch and allocation experiments
|
||||
|
||||
Exploratory runs used 100,000 bars, nine repetitions, and one conversion sample,
|
||||
with only `native-noop`, `direct-noop`, and `adapter-noop` selected.
|
||||
|
||||
| Variant | Direct Python (ns/bar) | Adapter (ns/bar) | Decision |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Reuse method-name string | 167.3 | 214.8 | Keep |
|
||||
| Also cache the bound callable once | 141.1 | 230.7 | Reject |
|
||||
| Reuse initialization Bar objects, dynamic callable | 129.3 | 181.5 | Reject |
|
||||
|
||||
The retained change uses [PyO3's `intern!` macro](https://docs.rs/pyo3/0.28.3/pyo3/macro.intern.html)
|
||||
to reuse the Python string `"next"` instead of constructing it for every call.
|
||||
It still resolves the method on every bar.
|
||||
|
||||
Caching the bound callable broke
|
||||
`test_direct_engine_resolves_callback_replacements_on_later_bars`: the old
|
||||
callback ran three times instead of honoring the replacement on later bars.
|
||||
The adapter measurement also did not show a consistent gain over the interned
|
||||
name alone. Reusing initialization Bars measured lower callback cost, but broke
|
||||
`test_callbacks_keep_distinct_bar_objects_and_retained_values` because the
|
||||
callback received the same object as the initialization history. These changes
|
||||
were removed. Fresh callback objects and existing position snapshot semantics
|
||||
remain intact.
|
||||
|
||||
To reproduce the old dispatch in a separate checkout, keep the benchmark hook
|
||||
and replace the interned-name call in `Engine::run_py` with:
|
||||
|
||||
```rust
|
||||
strategy.bind(py).call_method(
|
||||
"next", (bar.clone(), broker_py.clone_ref(py)), None,
|
||||
)?;
|
||||
```
|
||||
|
||||
Remove the unused `next_name` declaration, rebuild the installed package, and
|
||||
run the same benchmark command with `--label before`. Do not compare a debug
|
||||
build with a release build.
|
||||
|
||||
## Choose an execution path
|
||||
|
||||
Use Python `next()` for decisions that depend on the broker: fills, current
|
||||
equity, trailing stops, or partial closes. Read `has_positions` or `position_count`
|
||||
when you only need presence or a count; use `positions` when you need snapshots.
|
||||
|
||||
Precompute indicators during initialization when they depend only on market
|
||||
data, then read the current value during callbacks. The indicator rows include
|
||||
the added initialization work and show its separate cost.
|
||||
|
||||
For broker-independent target-lot signals and parameter grids, use `optimize()`.
|
||||
It computes one signal array per combination and runs the simulations in native
|
||||
threads with the GIL released. Python callbacks still execute once per bar under
|
||||
the GIL. No batching, callback removal, or new single-run signal API is introduced
|
||||
by this change.
|
||||
@@ -0,0 +1,264 @@
|
||||
"""Isolate callback, adapter, indicator, and position-inspection costs.
|
||||
|
||||
Run against an installed release build, outside the repository. See
|
||||
benchmark_callbacks.md for commands, timing boundaries, and interpretation.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from backtestingfx import Backtest, Engine, Strategy
|
||||
from backtestingfx.backtest import _Adapter
|
||||
|
||||
|
||||
CONFIG = (10_000.0, 7.0, 0.0001, 100_000.0, 1.27)
|
||||
LOT = 0.001
|
||||
|
||||
|
||||
class Noop(Strategy):
|
||||
pass
|
||||
|
||||
|
||||
class DirectNoop:
|
||||
def init(self, bars):
|
||||
self.setup_finished = time.perf_counter_ns()
|
||||
|
||||
def next(self, bar, broker):
|
||||
pass
|
||||
|
||||
|
||||
class TimedAdapter(_Adapter):
|
||||
def init(self, bars):
|
||||
super().init(bars)
|
||||
self.setup_finished = time.perf_counter_ns()
|
||||
|
||||
|
||||
class RollingIndicator(Strategy):
|
||||
def next(self):
|
||||
self.value = sum(bar.close for bar in self.data[-20:]) / min(self.index + 1, 20)
|
||||
|
||||
|
||||
class PrecomputedIndicator(Strategy):
|
||||
def init(self):
|
||||
closes = np.array([bar.close for bar in self._bars])
|
||||
sums = np.concatenate(([0.0], np.cumsum(closes)))
|
||||
ends = np.arange(1, len(closes) + 1)
|
||||
starts = np.maximum(0, ends - 20)
|
||||
self.values = (sums[ends] - sums[starts]) / (ends - starts)
|
||||
|
||||
def next(self):
|
||||
self.value = self.values[self.index]
|
||||
|
||||
|
||||
class Hold(Strategy):
|
||||
count = 1
|
||||
|
||||
def next(self):
|
||||
if self.index == 0:
|
||||
for _ in range(self.count):
|
||||
self.buy(LOT)
|
||||
|
||||
|
||||
class ReadPositions(Hold):
|
||||
def next(self):
|
||||
super().next()
|
||||
self.observed = len(self.positions)
|
||||
|
||||
|
||||
class ReadCount(Hold):
|
||||
def next(self):
|
||||
super().next()
|
||||
self.observed = self.position_count
|
||||
|
||||
|
||||
class ReadPresence(Hold):
|
||||
def next(self):
|
||||
super().next()
|
||||
self.observed = self.has_positions
|
||||
|
||||
|
||||
class TradeEveryBar(Strategy):
|
||||
def next(self):
|
||||
self.buy(LOT)
|
||||
self.close_all()
|
||||
|
||||
|
||||
def cases():
|
||||
result = {
|
||||
"native-noop": (None, 0),
|
||||
"direct-noop": (DirectNoop, 0),
|
||||
"adapter-noop": (Noop, 0),
|
||||
"rolling-indicator": (RollingIndicator, 0),
|
||||
"precomputed-indicator": (PrecomputedIndicator, 0),
|
||||
"trade-every-bar": (TradeEveryBar, None),
|
||||
}
|
||||
for count in (1, 32):
|
||||
for name, base in (("hold", Hold), ("positions", ReadPositions),
|
||||
("count", ReadCount), ("presence", ReadPresence)):
|
||||
if base is ReadCount and not hasattr(Strategy, "position_count"):
|
||||
continue
|
||||
if base is ReadPresence and not hasattr(Strategy, "has_positions"):
|
||||
continue
|
||||
result[f"{name}-{count}"] = (type(f"{name}{count}", (base,), {"count": count}), count)
|
||||
return result
|
||||
|
||||
|
||||
def assert_result(stats, bars, trades):
|
||||
# Flat midpoint: each trade pays one full spread, two commissions, and conversion.
|
||||
pnl = -(0.0001 * 100_000 * 1.27 + 2 * 7.0) * LOT
|
||||
assert stats.num_trades == trades, (stats.num_trades, trades)
|
||||
assert math.isclose(stats.final_cash, 10_000 + trades * pnl, abs_tol=1e-5)
|
||||
assert len(stats.equity_curve) == bars + 1
|
||||
assert stats.equity_curve[0] == 10_000.0
|
||||
assert stats.equity_curve[-1] == stats.final_cash
|
||||
if trades == 0:
|
||||
assert all(value == 10_000.0 for value in stats.equity_curve)
|
||||
assert all(math.isclose(trade.entry_price, 1.10005, abs_tol=1e-12)
|
||||
and math.isclose(trade.exit_price, 1.09995, abs_tol=1e-12)
|
||||
and math.isclose(trade.pnl, pnl, abs_tol=1e-10) for trade in stats.trades)
|
||||
|
||||
|
||||
def run_dataset(size, repeats, conversion_repeats, selected):
|
||||
data = pd.DataFrame(
|
||||
{column: np.full(size, 1.1) for column in ("open", "high", "low", "close")},
|
||||
index=pd.date_range("2026-01-01", periods=size, freq="s", tz="UTC"),
|
||||
)
|
||||
backtest = Backtest(data)
|
||||
conversion = []
|
||||
for _ in range(conversion_repeats):
|
||||
start = time.perf_counter_ns()
|
||||
bars = backtest._to_bars(data)
|
||||
conversion.append((time.perf_counter_ns() - start) / 1e9)
|
||||
# Release the old list before allocating another one in the next repetition.
|
||||
if len(conversion) < conversion_repeats:
|
||||
del bars
|
||||
|
||||
results = []
|
||||
available = cases()
|
||||
for name, (strategy_type, expected_trades) in available.items():
|
||||
if selected and name not in selected:
|
||||
continue
|
||||
samples = []
|
||||
# One untimed warmup, then fresh engines and strategies for every sample.
|
||||
for repetition in range(repeats + 1):
|
||||
start_construction = time.perf_counter_ns()
|
||||
engine = Engine(bars, *CONFIG)
|
||||
constructed = time.perf_counter_ns()
|
||||
strategy = strategy_type() if strategy_type else None
|
||||
if strategy is None:
|
||||
adapter = None
|
||||
elif name == "direct-noop":
|
||||
adapter = strategy
|
||||
else:
|
||||
adapter = TimedAdapter(strategy)
|
||||
start = time.perf_counter_ns()
|
||||
if adapter is None:
|
||||
stats = engine._run_native_noop()
|
||||
else:
|
||||
stats = engine.run(adapter)
|
||||
end = time.perf_counter_ns()
|
||||
setup_finished = adapter.setup_finished if adapter else start
|
||||
if repetition:
|
||||
samples.append({
|
||||
"engine_setup_seconds": (constructed - start_construction) / 1e9,
|
||||
"init_seconds": (setup_finished - start) / 1e9,
|
||||
"simulation_seconds": (end - setup_finished) / 1e9,
|
||||
})
|
||||
assert_result(stats, size, size if expected_trades is None else expected_trades)
|
||||
if isinstance(strategy, (RollingIndicator, PrecomputedIndicator)):
|
||||
assert math.isclose(strategy.value, 1.1, abs_tol=1e-8)
|
||||
if isinstance(strategy, Hold):
|
||||
if isinstance(strategy, ReadPresence):
|
||||
assert strategy.observed is True
|
||||
elif isinstance(strategy, (ReadPositions, ReadCount)):
|
||||
assert strategy.observed == strategy.count
|
||||
del stats, engine, strategy, adapter
|
||||
results.append({"case": name, "bars": size,
|
||||
**{key: statistics.median(sample[key] for sample in samples)
|
||||
for key in samples[0]},
|
||||
"simulation_samples_seconds": [s["simulation_seconds"] for s in samples]})
|
||||
try:
|
||||
import resource
|
||||
rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
peak_rss = rss if sys.platform == "darwin" else rss * 1024
|
||||
except ImportError:
|
||||
peak_rss = None
|
||||
return {"bars": size, "conversion_seconds": statistics.median(conversion),
|
||||
"peak_rss_bytes": peak_rss, "results": results}
|
||||
|
||||
|
||||
def command_output(command):
|
||||
try:
|
||||
return subprocess.check_output(command, text=True, stderr=subprocess.DEVNULL).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return "unavailable"
|
||||
|
||||
|
||||
def metadata():
|
||||
cpu = platform.processor()
|
||||
if Path("/proc/cpuinfo").exists():
|
||||
cpu = next((line.split(":", 1)[1].strip() for line in
|
||||
Path("/proc/cpuinfo").read_text().splitlines()
|
||||
if line.startswith("model name")), cpu)
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
lock = root / "Cargo.lock"
|
||||
pyo3 = re.search(r'name = "pyo3"\nversion = "([^"]+)"', lock.read_text()) if lock.exists() else None
|
||||
return {"python": sys.version, "platform": platform.platform(), "cpu": cpu,
|
||||
"pyo3_lock_version": pyo3.group(1) if pyo3 else "unavailable",
|
||||
"rust": command_output(["rustc", "--version"]),
|
||||
"git_head": command_output(["git", "-C", str(root), "rev-parse", "HEAD"]),
|
||||
"package_versions": {name: importlib.metadata.version(name)
|
||||
for name in ("backtestingfx", "pandas", "numpy")}}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--sizes", nargs="+", type=int, default=[1000, 10_000, 100_000])
|
||||
parser.add_argument("--repeats", type=int, default=7)
|
||||
parser.add_argument("--conversion-repeats", type=int, default=3)
|
||||
parser.add_argument("--cases", nargs="+")
|
||||
parser.add_argument("--label", default="working-tree")
|
||||
parser.add_argument("--output", type=Path)
|
||||
parser.add_argument("--worker-size", type=int, help=argparse.SUPPRESS)
|
||||
args = parser.parse_args()
|
||||
if args.repeats < 1 or args.conversion_repeats < 1 or any(size < 1 for size in args.sizes):
|
||||
parser.error("sizes and repeats must be positive")
|
||||
if args.cases and set(args.cases) - set(cases()):
|
||||
parser.error(f"available cases: {', '.join(cases())}")
|
||||
if args.worker_size:
|
||||
print(json.dumps(run_dataset(args.worker_size, args.repeats, args.conversion_repeats, args.cases)))
|
||||
return
|
||||
report = {"label": args.label, "metadata": metadata(), "repeats": args.repeats,
|
||||
"conversion_repeats": args.conversion_repeats,
|
||||
"datasets": []}
|
||||
for size in args.sizes:
|
||||
command = [sys.executable, str(Path(__file__).resolve()), "--worker-size", str(size),
|
||||
"--repeats", str(args.repeats), "--conversion-repeats", str(args.conversion_repeats)]
|
||||
if args.cases:
|
||||
command.extend(["--cases", *args.cases])
|
||||
dataset = json.loads(subprocess.check_output(command, text=True))
|
||||
report["datasets"].append(dataset)
|
||||
print(f"{size:,} bars; conversion {dataset['conversion_seconds']:.4f} s; "
|
||||
f"peak RSS {dataset['peak_rss_bytes']} bytes", flush=True)
|
||||
for row in dataset["results"]:
|
||||
print(f" {row['case']:<23} init {row['init_seconds']:.6f} s "
|
||||
f"run {row['simulation_seconds']:.6f} s "
|
||||
f"{row['simulation_seconds'] / size * 1e9:.1f} ns/bar", flush=True)
|
||||
if args.output:
|
||||
args.output.write_text(json.dumps(report, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+150
-69
@@ -1,90 +1,171 @@
|
||||
"""
|
||||
Compares our SMA crossover logic against backtesting.py using the same data,
|
||||
same SMA periods, and zero commission/spread so only entry/exit logic is tested.
|
||||
"""Verify matched next-open SMA backtests against backtesting.py.
|
||||
|
||||
Trade count and win rate should match between both libraries.
|
||||
Dollar PnL will differ because the sizing models are different:
|
||||
- backtesting.py: buys fractional units based on available cash
|
||||
- backtestingfx: fixed lot size (0.1 lots = 10,000 units)
|
||||
Install backtesting==0.6.6, then run this file. See benchmark.md for the shared
|
||||
execution policy, supported comparisons, and timing commands.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from backtesting import Backtest as BtBacktest, Strategy as BtStrategy
|
||||
from backtestingfx import Backtest, Strategy
|
||||
|
||||
|
||||
CONTRACT_SIZE = 100_000.0
|
||||
CASH = 10_000.0
|
||||
UNITS = 1000
|
||||
FAST = 10
|
||||
SLOW = 50
|
||||
|
||||
|
||||
# ── backtesting.py ────────────────────────────────────────────────────────────
|
||||
|
||||
class SmaCrossBt(BtStrategy):
|
||||
fast = FAST
|
||||
slow = SLOW
|
||||
|
||||
def init(self):
|
||||
pass
|
||||
|
||||
def next(self):
|
||||
if len(self.data.Close) < self.slow:
|
||||
return
|
||||
fast_sma = self.data.Close[-self.fast:].mean()
|
||||
slow_sma = self.data.Close[-self.slow:].mean()
|
||||
|
||||
if not self.position:
|
||||
if fast_sma > slow_sma:
|
||||
self.buy()
|
||||
else:
|
||||
if fast_sma < slow_sma:
|
||||
self.position.close()
|
||||
def load_data(count=None):
|
||||
path = Path(__file__).resolve().parents[1] / "data/EURUSD_1H.csv"
|
||||
data = pd.read_csv(path)
|
||||
data.index = pd.to_datetime(data.pop("timestamp"), utc=True)
|
||||
data = data[["open", "high", "low", "close", "volume"]]
|
||||
if count is not None:
|
||||
if count < 3:
|
||||
raise ValueError("comparison needs at least three bars")
|
||||
data = pd.concat([data] * (-(-count // len(data))), ignore_index=True).iloc[:count].copy()
|
||||
data.index = pd.date_range("2000-01-01", periods=count, freq="h", tz="UTC")
|
||||
return data
|
||||
|
||||
|
||||
# ── backtestingfx ─────────────────────────────────────────────────────────────
|
||||
|
||||
class SmaCrossFx(Strategy):
|
||||
fast = FAST
|
||||
slow = SLOW
|
||||
|
||||
def next(self):
|
||||
if self.index < self.slow:
|
||||
return
|
||||
closes = [b.close for b in self.data[-self.slow:]]
|
||||
fast_sma = sum(closes[-self.fast:]) / self.fast
|
||||
slow_sma = sum(closes) / self.slow
|
||||
|
||||
if not self.positions:
|
||||
if fast_sma > slow_sma:
|
||||
self.buy(0.1)
|
||||
else:
|
||||
if fast_sma < slow_sma:
|
||||
self.close_all()
|
||||
def sma_decisions(data, fast=FAST, slow=SLOW):
|
||||
fast_sma = data["close"].rolling(fast).mean()
|
||||
slow_sma = data["close"].rolling(slow).mean()
|
||||
return np.where(fast_sma > slow_sma, 1, np.where(fast_sma < slow_sma, -1, 0))
|
||||
|
||||
|
||||
# ── run both ──────────────────────────────────────────────────────────────────
|
||||
def strategy_types(decisions, units, clocks):
|
||||
last_bar = len(decisions) - 1
|
||||
lots = units / CONTRACT_SIZE
|
||||
|
||||
df_raw = pd.read_csv("data/EURUSD_1H.csv")
|
||||
class SmaCrossBt(BtStrategy):
|
||||
def init(self):
|
||||
clocks["backtesting.py"] = time.perf_counter_ns()
|
||||
|
||||
# backtesting.py needs a DatetimeIndex and capitalised column names
|
||||
df_bt = df_raw.copy()
|
||||
df_bt["timestamp"] = pd.to_datetime(df_bt["timestamp"], utc=True)
|
||||
df_bt = df_bt.set_index("timestamp")
|
||||
df_bt.index = df_bt.index.tz_localize(None)
|
||||
df_bt = df_bt.rename(columns={"open": "Open", "high": "High", "low": "Low", "close": "Close", "volume": "Volume"})
|
||||
def next(self):
|
||||
index = len(self.data) - 1
|
||||
if index >= last_bar - 1:
|
||||
if index == last_bar - 1 and self.position:
|
||||
self.sell(size=units)
|
||||
return
|
||||
decision = decisions[index]
|
||||
if decision > 0 and not self.position:
|
||||
self.buy(size=units)
|
||||
elif decision < 0 and self.position:
|
||||
self.sell(size=units)
|
||||
|
||||
bt_result = BtBacktest(df_bt, SmaCrossBt, cash=10_000, commission=0).run()
|
||||
class SmaCrossFx(Strategy):
|
||||
def init(self):
|
||||
clocks["backtestingfx"] = time.perf_counter_ns()
|
||||
|
||||
fx_result = Backtest(df_raw, SmaCrossFx, cash=10_000, commission=0, spread=0).run()
|
||||
def next(self):
|
||||
if self.index == 0:
|
||||
return
|
||||
price = self._bar.open
|
||||
timestamp = self._bar.timestamp
|
||||
if self.index == last_bar:
|
||||
self._broker.close_all(price, timestamp)
|
||||
return
|
||||
# Broker calls implement next-open fills explicitly. Strategy.buy()
|
||||
# normally fills immediately at the current close in backtestingfx.
|
||||
decision = decisions[self.index - 1]
|
||||
if decision > 0 and not self.has_positions:
|
||||
self._broker.buy(price, lots, timestamp, None, None)
|
||||
elif decision < 0 and self.has_positions:
|
||||
self._broker.close_all(price, timestamp)
|
||||
|
||||
# ── compare ───────────────────────────────────────────────────────────────────
|
||||
return SmaCrossBt, SmaCrossFx
|
||||
|
||||
print("=" * 45)
|
||||
print(f"{'Metric':<20} {'backtesting.py':>12} {'backtestingfx':>12}")
|
||||
print("=" * 45)
|
||||
print(f"{'Trades':<20} {bt_result['# Trades']:>12} {fx_result.num_trades:>12}")
|
||||
print(f"{'Win Rate %':<20} {bt_result['Win Rate [%]']:>12.1f} {fx_result.win_rate_pct:>12.1f}")
|
||||
print(f"{'Return %':<20} {bt_result['Return [%]']:>12.2f} {fx_result.total_return_pct:>12.2f}")
|
||||
print(f"{'Max Drawdown %':<20} {bt_result['Max. Drawdown [%]']:>12.2f} {fx_result.max_drawdown_pct:>12.2f}")
|
||||
print("=" * 45)
|
||||
print()
|
||||
print("Note: Return % differs because backtesting.py sizes by available cash,")
|
||||
print(" backtestingfx uses fixed 0.1 lots. Trade count + win rate should match.")
|
||||
|
||||
class Comparison:
|
||||
def __init__(self, data, commission=0.0, units=UNITS, decisions=None):
|
||||
if len(data) < 3:
|
||||
raise ValueError("comparison needs at least three bars")
|
||||
if units < 1 or units != int(units):
|
||||
raise ValueError("units must be a positive integer")
|
||||
self.data = data
|
||||
self.commission = commission
|
||||
self.units = int(units)
|
||||
self.decisions = sma_decisions(data) if decisions is None else np.asarray(decisions)
|
||||
if self.decisions.shape != (len(data),):
|
||||
raise ValueError("decisions must contain one value per bar")
|
||||
if self.decisions[0] != 0:
|
||||
raise ValueError("the first decision must be flat because backtesting.py starts at bar 1")
|
||||
self.clocks = {}
|
||||
self.bt_strategy, self.fx_strategy = strategy_types(self.decisions, self.units, self.clocks)
|
||||
self.bt_data = data.rename(columns=str.capitalize)
|
||||
|
||||
def new_bt(self):
|
||||
commission = self.commission
|
||||
fee = (lambda size, price: abs(size) * commission / CONTRACT_SIZE) if commission else 0.0
|
||||
return BtBacktest(
|
||||
self.bt_data, self.bt_strategy, cash=CASH, commission=fee, spread=0.0,
|
||||
trade_on_close=False, hedging=False, exclusive_orders=False, finalize_trades=False,
|
||||
)
|
||||
|
||||
def new_fx(self):
|
||||
return Backtest(self.data, self.fx_strategy, cash=CASH, commission=self.commission,
|
||||
spread=0.0, contract_size=CONTRACT_SIZE, quote_to_account=1.0)
|
||||
|
||||
def run(self, name):
|
||||
start = time.perf_counter_ns()
|
||||
backtest = self.new_bt() if name == "backtesting.py" else self.new_fx()
|
||||
stats = backtest.run()
|
||||
end = time.perf_counter_ns()
|
||||
initialized = self.clocks[name]
|
||||
timings = {"setup_seconds": (initialized - start) / 1e9,
|
||||
"simulation_seconds": (end - initialized) / 1e9,
|
||||
"total_seconds": (end - start) / 1e9}
|
||||
return stats, timings
|
||||
|
||||
def verify(self, reference, actual):
|
||||
trades = reference["_trades"]
|
||||
fx_trades = actual.trades
|
||||
np.testing.assert_equal(len(trades), len(fx_trades))
|
||||
np.testing.assert_equal(actual.num_trades, len(trades))
|
||||
if len(trades):
|
||||
np.testing.assert_array_equal(trades["Size"], [self.units] * len(fx_trades))
|
||||
np.testing.assert_allclose([trade.lot_size * CONTRACT_SIZE for trade in fx_trades],
|
||||
trades["Size"], rtol=0, atol=1e-10)
|
||||
np.testing.assert_array_equal([trade.is_long for trade in fx_trades], [True] * len(fx_trades))
|
||||
for bt_field, fx_field in (("EntryPrice", "entry_price"), ("ExitPrice", "exit_price"),
|
||||
("PnL", "pnl")):
|
||||
np.testing.assert_allclose(trades[bt_field], [getattr(t, fx_field) for t in fx_trades],
|
||||
rtol=0, atol=1e-8)
|
||||
for bt_field, fx_field in (("EntryTime", "entry_timestamp"), ("ExitTime", "exit_timestamp")):
|
||||
expected = [int(stamp.timestamp()) for stamp in trades[bt_field]]
|
||||
np.testing.assert_array_equal(expected, [getattr(t, fx_field) for t in fx_trades])
|
||||
np.testing.assert_allclose(reference["Win Rate [%]"], actual.win_rate_pct, rtol=0, atol=1e-8)
|
||||
# FX prepends an initial-cash observation before the first bar.
|
||||
np.testing.assert_allclose(reference["_equity_curve"]["Equity"], actual.equity_curve[1:],
|
||||
rtol=0, atol=1e-8)
|
||||
np.testing.assert_allclose(reference["Equity Final [$]"], actual.final_cash, rtol=0, atol=1e-8)
|
||||
np.testing.assert_allclose(reference["Return [%]"], actual.total_return_pct, rtol=0, atol=1e-8)
|
||||
np.testing.assert_allclose(-reference["Max. Drawdown [%]"], actual.max_drawdown_pct,
|
||||
rtol=0, atol=1e-8)
|
||||
np.testing.assert_allclose(actual.final_cash - CASH, sum(t.pnl for t in fx_trades),
|
||||
rtol=0, atol=1e-8)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--bars", type=int)
|
||||
parser.add_argument("--commission", type=float, default=0.0)
|
||||
args = parser.parse_args()
|
||||
comparison = Comparison(load_data(args.bars), commission=args.commission)
|
||||
bt_result, _ = comparison.run("backtesting.py")
|
||||
fx_result, _ = comparison.run("backtestingfx")
|
||||
comparison.verify(bt_result, fx_result)
|
||||
print(f"PASS: {len(comparison.data):,} bars, {fx_result.num_trades} matching trades, "
|
||||
f"final equity {fx_result.final_cash:.8f}, commission {args.commission}/lot/side")
|
||||
print("Verified fills, sizes, timestamps, net PnL, full equity curve, return, win rate, and drawdown.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -286,6 +286,16 @@ impl Broker {
|
||||
self.positions()
|
||||
}
|
||||
|
||||
#[getter]
|
||||
fn position_count(&self) -> usize {
|
||||
self.positions.len()
|
||||
}
|
||||
|
||||
#[getter]
|
||||
fn has_positions(&self) -> bool {
|
||||
!self.positions.is_empty()
|
||||
}
|
||||
|
||||
#[pyo3(name = "buy")]
|
||||
fn py_buy(
|
||||
&mut self,
|
||||
|
||||
+13
-1
@@ -70,6 +70,12 @@ impl Strategy for SmaCross {
|
||||
}
|
||||
}
|
||||
|
||||
struct Noop;
|
||||
|
||||
impl Strategy for Noop {
|
||||
fn next(&mut self, _bar: &Bar, _broker: &mut Broker) {}
|
||||
}
|
||||
|
||||
impl Engine {
|
||||
pub fn new(
|
||||
data: Vec<Bar>,
|
||||
@@ -128,6 +134,7 @@ impl Engine {
|
||||
),
|
||||
)?;
|
||||
|
||||
let next_name = pyo3::intern!(py, "next");
|
||||
for bar in &self.data {
|
||||
{
|
||||
let mut b = broker_py.borrow_mut(py);
|
||||
@@ -135,7 +142,7 @@ impl Engine {
|
||||
}
|
||||
strategy
|
||||
.bind(py)
|
||||
.call_method("next", (bar.clone(), broker_py.clone_ref(py)), None)?;
|
||||
.call_method1(next_name, (bar.clone(), broker_py.clone_ref(py)))?;
|
||||
let equity = broker_py.borrow(py).equity(bar.close);
|
||||
self.equity_curve.push(equity);
|
||||
}
|
||||
@@ -164,6 +171,11 @@ impl Engine {
|
||||
};
|
||||
Ok(self.run(&mut strat))
|
||||
}
|
||||
|
||||
// Benchmark baseline with the same broker, equity, statistics, and finalization paths.
|
||||
fn _run_native_noop(&mut self) -> Stats {
|
||||
self.run(&mut Noop)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
import unittest
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from backtestingfx import Backtest, Bar, Broker, Engine, Strategy
|
||||
|
||||
|
||||
def bars():
|
||||
return [Bar(i, price, price, price, price, 0.0)
|
||||
for i, price in enumerate((1.1, 1.2, 1.3))]
|
||||
|
||||
|
||||
def engine():
|
||||
return Engine(bars(), 10_000.0, 7.0, 0.0001, 100_000.0, 1.27)
|
||||
|
||||
|
||||
class CallbackContractTest(unittest.TestCase):
|
||||
def test_empty_engine_does_not_require_a_next_method(self):
|
||||
class InitOnly:
|
||||
def init(self, data):
|
||||
if data:
|
||||
raise AssertionError("expected no bars")
|
||||
|
||||
stats = Engine([], 10_000.0, 0.0, 0.0, 100_000.0, 1.0).run(InitOnly())
|
||||
self.assertEqual(stats.num_trades, 0)
|
||||
self.assertEqual(stats.equity_curve, [10_000.0])
|
||||
|
||||
def test_strategy_position_inspection_before_run_and_after_fills(self):
|
||||
seen = []
|
||||
strategy = Strategy()
|
||||
self.assertFalse(strategy.has_positions)
|
||||
self.assertEqual(strategy.position_count, 0)
|
||||
|
||||
class Probe(Strategy):
|
||||
def next(self):
|
||||
seen.append((self.index, self.position_count, self.has_positions))
|
||||
if self.index == 0:
|
||||
self.buy(0.4, stop_loss=1.09)
|
||||
self.buy(0.6)
|
||||
self.assert_state(2)
|
||||
self.close_partial(self.positions[0].id, 0.2)
|
||||
self.assert_state(2)
|
||||
elif self.index == 1:
|
||||
# The first position stopped out before this callback.
|
||||
self.close_all()
|
||||
self.assert_state(0)
|
||||
|
||||
def assert_state(self, count):
|
||||
if self.position_count != count or self.has_positions != (count > 0):
|
||||
raise AssertionError("position inspection differs from live broker state")
|
||||
|
||||
data = pd.DataFrame(
|
||||
{column: [1.1, 1.08, 1.1] for column in ("open", "high", "low", "close")},
|
||||
index=pd.date_range("2026-01-01", periods=3, freq="h", tz="UTC"),
|
||||
)
|
||||
stats = Backtest(data, Probe).run()
|
||||
self.assertEqual(seen, [(0, 0, False), (1, 1, True), (2, 0, False)])
|
||||
self.assertEqual(stats.num_trades, 3)
|
||||
self.assertAlmostEqual(stats.final_cash - 10_000.0, sum(t.pnl for t in stats.trades))
|
||||
|
||||
def test_broker_count_and_presence_do_not_change_position_snapshots(self):
|
||||
account = Broker(10_000.0, 0.0, 0.0, 100_000.0, 1.0)
|
||||
self.assertEqual(account.position_count, 0)
|
||||
self.assertFalse(account.has_positions)
|
||||
account.buy(1.1, 1.0, 0, None, None)
|
||||
saved = account.positions()
|
||||
account.close_partial(0, 0.4, 1.1, 1)
|
||||
self.assertEqual(account.position_count, 1)
|
||||
self.assertTrue(account.has_positions)
|
||||
self.assertEqual(saved[0].lot_size, 1.0)
|
||||
self.assertAlmostEqual(account.positions()[0].lot_size, 0.6)
|
||||
account.close_all(1.1, 2)
|
||||
self.assertFalse(account.has_positions)
|
||||
self.assertEqual(account.position_count, 0)
|
||||
|
||||
def test_callbacks_keep_distinct_bar_objects_and_retained_values(self):
|
||||
class Retain:
|
||||
def init(self, data):
|
||||
self.data = data
|
||||
self.seen = []
|
||||
|
||||
def next(self, bar, broker):
|
||||
self.seen.append(bar)
|
||||
|
||||
strategy = Retain()
|
||||
stats = engine().run(strategy)
|
||||
self.assertEqual([bar.timestamp for bar in strategy.seen], [0, 1, 2])
|
||||
self.assertEqual([bar.close for bar in strategy.seen], [1.1, 1.2, 1.3])
|
||||
for received, original in zip(strategy.seen, strategy.data):
|
||||
self.assertIsNot(received, original)
|
||||
self.assertEqual(received.close, original.close)
|
||||
self.assertEqual(stats.equity_curve, [10_000.0] * 4)
|
||||
|
||||
def test_direct_engine_resolves_callback_replacements_on_later_bars(self):
|
||||
seen = []
|
||||
|
||||
class Replace:
|
||||
def init(self, data):
|
||||
pass
|
||||
|
||||
def next(self, bar, broker):
|
||||
seen.append(("first", bar.timestamp))
|
||||
self.next = self.replacement
|
||||
|
||||
def replacement(self, bar, broker):
|
||||
seen.append(("replacement", bar.timestamp))
|
||||
|
||||
engine().run(Replace())
|
||||
self.assertEqual(seen, [("first", 0), ("replacement", 1), ("replacement", 2)])
|
||||
|
||||
def test_callback_errors_propagate_and_engine_can_be_reused(self):
|
||||
failure = RuntimeError("callback failure")
|
||||
seen = []
|
||||
|
||||
class Fail:
|
||||
def init(self, data):
|
||||
pass
|
||||
|
||||
def next(self, bar, broker):
|
||||
seen.append(bar.timestamp)
|
||||
raise failure
|
||||
|
||||
simulation = engine()
|
||||
with self.assertRaises(RuntimeError) as raised:
|
||||
simulation.run(Fail())
|
||||
self.assertIs(raised.exception, failure)
|
||||
self.assertEqual(seen, [0])
|
||||
|
||||
class Noop:
|
||||
def init(self, data):
|
||||
pass
|
||||
|
||||
def next(self, bar, broker):
|
||||
pass
|
||||
|
||||
self.assertEqual(simulation.run(Noop()).equity_curve, [10_000.0] * 4)
|
||||
|
||||
def test_adapter_keeps_index_history_and_dynamic_strategy_methods(self):
|
||||
seen = []
|
||||
|
||||
class Probe(Strategy):
|
||||
def next(self):
|
||||
seen.append((self.index, len(self.data), self.data[-1].close))
|
||||
self.next = self.replacement
|
||||
|
||||
def replacement(self):
|
||||
seen.append((self.index, len(self.data), self.data[-1].close))
|
||||
with self_check.assertRaises(IndexError):
|
||||
self.data[self.index + 1]
|
||||
|
||||
self_check = self
|
||||
data = pd.DataFrame(
|
||||
{column: [1.1, 1.2, 1.3] for column in ("open", "high", "low", "close")},
|
||||
index=pd.date_range("2026-01-01", periods=3, freq="h", tz="UTC"),
|
||||
)
|
||||
Backtest(data, Probe).run()
|
||||
self.assertEqual(seen, [(0, 1, 1.1), (1, 2, 1.2), (2, 3, 1.3)])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,69 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
try:
|
||||
import backtesting
|
||||
except ImportError:
|
||||
backtesting = None
|
||||
|
||||
|
||||
@unittest.skipIf(backtesting is None, "optional comparison requires backtesting==0.6.6")
|
||||
class ReferenceComparisonTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "examples"))
|
||||
from compare_bt import Comparison
|
||||
cls.comparison_type = Comparison
|
||||
|
||||
def comparison(self, commission=7.0, decisions=None):
|
||||
opens = np.array([1.0, 1.1, 1.2, 1.4, 1.3, 1.5, 1.6])
|
||||
closes = np.array([1.05, 1.15, 1.25, 1.35, 1.4, 1.55, 1.65])
|
||||
data = pd.DataFrame(
|
||||
{"open": opens, "high": np.maximum(opens, closes) + 0.01,
|
||||
"low": np.minimum(opens, closes) - 0.01, "close": closes},
|
||||
index=pd.date_range("2026-01-01", periods=len(opens), freq="h", tz="UTC"),
|
||||
)
|
||||
if decisions is None:
|
||||
decisions = [0, 1, 0, -1, 1, 0, 0]
|
||||
return self.comparison_type(data, commission=commission, decisions=decisions)
|
||||
|
||||
def test_next_open_trades_and_equity_match_hand_calculated_fixture(self):
|
||||
comparison = self.comparison()
|
||||
reference, _ = comparison.run("backtesting.py")
|
||||
actual, _ = comparison.run("backtestingfx")
|
||||
comparison.verify(reference, actual)
|
||||
self.assertEqual(actual.num_trades, 2)
|
||||
self.assertAlmostEqual(actual.final_cash, 10_199.72)
|
||||
np.testing.assert_allclose(actual.equity_curve[1:],
|
||||
[10_000, 10_000, 10_049.93, 10_149.93,
|
||||
10_099.86, 10_149.79, 10_199.72], rtol=0, atol=1e-8)
|
||||
np.testing.assert_allclose([trade.entry_price for trade in actual.trades], [1.2, 1.5])
|
||||
np.testing.assert_allclose([trade.exit_price for trade in actual.trades], [1.3, 1.6])
|
||||
np.testing.assert_allclose([trade.pnl for trade in actual.trades], [99.86, 99.86])
|
||||
|
||||
def test_no_trade_and_zero_commission_cases_match(self):
|
||||
for decisions, expected_trades, equity in (([0] * 7, 0, 10_000),
|
||||
([0, 1, 0, -1, 1, 0, 0], 2, 10_200)):
|
||||
comparison = self.comparison(commission=0.0, decisions=decisions)
|
||||
reference, _ = comparison.run("backtesting.py")
|
||||
actual, _ = comparison.run("backtestingfx")
|
||||
comparison.verify(reference, actual)
|
||||
self.assertEqual(actual.num_trades, expected_trades)
|
||||
self.assertAlmostEqual(actual.final_cash, equity)
|
||||
|
||||
def test_checker_rejects_price_mismatch_even_when_trade_count_matches(self):
|
||||
comparison = self.comparison()
|
||||
reference, _ = comparison.run("backtesting.py")
|
||||
actual, _ = comparison.run("backtestingfx")
|
||||
reference["_trades"] = reference["_trades"].copy()
|
||||
reference["_trades"].loc[0, "EntryPrice"] += 0.01
|
||||
with self.assertRaises(AssertionError):
|
||||
comparison.verify(reference, actual)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user