gavindiaz/manifoldbt
1
0
Fork 0
mirror of https://github.com/manifoldbt/manifoldbt.git synced 2026-11-02 07:56:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

release: v0.24.0

Browse Source
This commit is contained in:
github-actions[bot]
2026-10-23 23:06:12 +02:00
committed by Jimmy7892
parent 758796f84d
commit fa33c6b06f
11 changed files with 434 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+14 -13
View File
@@ -1,14 +1,14 @@
<p align="center">
<img src="https://raw.githubusercontent.com/manifoldbt/manifoldbt/master/assets/logo.png" width="110" alt="ManifoldBT logo">
<img src="https://raw.githubusercontent.com/manifoldbt/manifoldbt/master/assets/logo.png" width="110" alt="Manifold-BT logo">
</p>
<p align="center">
<strong>ManifoldBT</strong><br>
<strong>Manifold-BT</strong><br>
Rust-powered backtesting engine for quantitative research
</p>
<p align="center">
<a href="https://discord.gg/bvU6Wjc72d"><img src="https://img.shields.io/badge/Discord-Join%20the%20community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join the ManifoldBT Discord" height="34"></a>
<a href="https://discord.gg/bvU6Wjc72d"><img src="https://img.shields.io/badge/Discord-Join%20the%20community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join the Manifold-BT Discord" height="34"></a>
</p>
<p align="center">
@@ -26,12 +26,12 @@
---
ManifoldBT is a Python backtesting library with a Rust core. Strategies are written in a
Manifold-BT is a Python backtesting library with a Rust core. Strategies are written in a
fluent Python DSL, compiled to a vectorized Rust expression graph, then run through a
sequential fill simulation with realistic fees, slippage, funding and look-ahead protection.
**Vectorized speed with event-driven execution realism.**
## Why ManifoldBT
## Why Manifold-BT
- **Fast**: 10M bars in 329 ms. 79x faster than vectorbt, and 311x once you also want drawdown and Sharpe. [Measured in public CI](#performance), every run linked.
- **Expressive**: fluent DSL with 104 indicators and 38 candlestick patterns, conditional logic, cross-asset references
@@ -358,7 +358,7 @@ a workload they disagree on gets no published timing at all.
ran on Linux x86_64, 4 vCPU (AMD EPYC 7763), Python 3.12, manifoldbt 0.18.0 /
vectorbt 0.28.4 / raptorbt 0.9.0, 3 interleaved repetitions, medians reported.
| Workload | Bars | ManifoldBT | vectorbt | raptorbt |
| Workload | Bars | Manifold-BT | vectorbt | raptorbt |
|---|---:|---:|---:|---:|
| SMA crossover | 10M | **327 ms** | 26.12 s (x79) | 913 ms (x2.8) |
| ...with drawdown, Sharpe, Sortino, volatility | 10M | **329 ms** | 102.38 s (**x311**) | 909 ms (x2.8) |
@@ -368,11 +368,11 @@ vectorbt 0.28.4 / raptorbt 0.9.0, 3 interleaved repetitions, medians reported.
| Stop-loss and take-profit bracket | 10M | **934 ms** | 26.24 s (x28) | 916 ms (**x1.0**) |
The second row is the one worth reading twice. Asking for a performance summary
costs ManifoldBT nothing measurable, because it computes one during the run
costs Manifold-BT nothing measurable, because it computes one during the run
whether you read it or not, and costs vectorbt 102 seconds, because it defers
the equity curve until a risk metric needs it and then has to build one.
The last two rows are the ones where ManifoldBT does worst, and they are
The last two rows are the ones where Manifold-BT does worst, and they are
published for that reason. Broadcasting a column per asset is close to free for
vectorbt, while walking five books is not free for anything. And on a
stop-loss/take-profit bracket, raptorbt is level with us: the intra-bar check
@@ -381,14 +381,14 @@ engines, so there is no vectorization left to win with.
### Parameter sweeps
| Bars | Combinations | ManifoldBT | vectorbt | raptorbt |
| Bars | Combinations | Manifold-BT | vectorbt | raptorbt |
|---:|---:|---:|---:|---:|
| 20,000 | 5,000 | **446 ms**, 40 MB | 5.84 s, 2.5 GB | 7.08 s |
| 200,000 | 10,000 | **9.96 s**, 79 MB | out of memory | 164.50 s |
Past a certain grid the question stops being speed. vectorbt materialises the
simulation per combination, 1.57 MB of it at 20,000 bars, so the second row
would ask a machine for tens of gigabytes. ManifoldBT runs it in ten seconds
would ask a machine for tens of gigabytes. Manifold-BT runs it in ten seconds
inside 79 MB.
Reproduce any of it yourself: fork the repository and press **Run workflow** on
@@ -401,7 +401,7 @@ The method, the parity gate and the known divergences are written up in
### Against an event-driven engine
backtrader runs the same EMA(12/26) + RSI(14) strategy on 500K 1-minute bars in
**46,944 ms**, against **13 ms** for ManifoldBT: a factor of **3,556**. Measured
**46,944 ms**, against **13 ms** for Manifold-BT: a factor of **3,556**. Measured
with `benchmarks/bench_vs_competitors.py`, median of 3 runs, on a developer
machine and not the CI runner, so it is not comparable line-for-line with the
table above.
@@ -413,7 +413,7 @@ are not doing the same thing.
### How it compares
| | ManifoldBT | vectorbt | backtrader | Nautilus |
| | Manifold-BT | vectorbt | backtrader | Nautilus |
|---|---|---|---|---|
| Engine | Rust (vectorized + sequential fills) | Numba/NumPy (vectorized) | Python (event-driven) | Rust/Python (event-driven) |
| Execution realism¹ | High | Basic | High | High |
@@ -435,7 +435,8 @@ Full API reference, indicator list, configuration guide, and best practices:
|---|---|---|
| Single backtests (`mbt.run`) | Unlimited, full speed | Unlimited, full speed |
| Parameter sweeps & batches | Up to 256 backtests per sweep | Unlimited |
| Output resolution | Daily | 1m, 5m, 15m, 1h |
| Simulation resolution (`bar_interval`) | 1 minute and coarser | Down to 1 second |
| Output resolution | Daily | Down to 1 second |
| Monte Carlo | 1K sims | Unlimited |
| Walk-Forward | - | Anchored + Rolling |
| Parameter Stability | - | Yes |
docs/strategy-authoring.md
+4
View File
@@ -236,6 +236,7 @@ config = mbt.BacktestConfig(
### Bar intervals
```python
Interval.seconds(1) # 1-second (Pro)
Interval.minutes(1) # 1-min
Interval.minutes(15) # 15-min
Interval.hours(1) # 1-hour
@@ -244,6 +245,9 @@ Interval.hours(12) # 12-hour
Interval.days(1) # daily
```
Anything below one minute is a Pro step; Community simulates at one minute at
the finest. `bar_interval` is one minute when left out.
### Accuracy mode
```python
examples/27_bars_vs_tape.py
+147 -142
View File
@@ -1,97 +1,110 @@
"""Bars and tape - the guess a candle forces, and the answer the trades hold.
"""Bars and tape - the exit a candle has to guess, and the trades that settle it.
A bracket puts two levels around a position. When a bar's range covers both,
A bracket puts two levels around a position. When one bar's range covers both,
the OHLC cannot say which one traded first: high and low carry no order. The
engine has to assume, and it assumes the stop, because that is the conservative
reading. Assuming is not knowing, and this file measures the gap.
One dataset, two resolutions. A tape is generated once, `tape_to_bars`
aggregates it into the engine's own bar schema, and the bar engine runs on
those bars. Every trade it books is then put back to the tape it came from,
which knows the sequence. Nothing is aligned, joined or approximated: the bars
ARE the tape, summarised.
`fill_resolution="ticks"` removes the assumption. Level orders - stop-loss,
take-profit, trailing stop, limit and stop entries - are then resolved against
the individual trades inside each bar, in time order: the level the market
actually printed first is the one that fills, and a stop that gaps fills at the
trade that crossed it. Market orders are unchanged.
What comes out:
Two runs, one per resolution. Same store, same strategy, same bars; the only
difference is the config field. What separates them is the phantom return, the
part of the bar run's result that the trades themselves never gave.
1. how many exits the bar engine had to guess at;
2. how often the guess was wrong;
3. what the wrong guesses cost, in the only currency that matters.
`result.tape_resolution` says how much of the run the tape actually decided:
The conservative assumption is not free: every time it is wrong it books a loss
the trade never took, and here those phantom losses are several times the whole
reported return.
bars_on_tape bars whose level orders were resolved on trades
bars_fallen_back bars with no trade in their window (bar rule used)
contested_bars bars where the bar rule saw BOTH levels at once
contested_tp_first of those, the tape found the take-profit first
contested_stop_first of those, the tape found the stop or trailing first
entries_on_tape entry orders triggered on a trade
It is `None` in bar mode, which the file prints too: that is how you tell a run
that read the tape from one that could not.
Demonstrates:
- `bt.ticks.tape_to_bars`: a tape as a drop-in bar dataset
- the same bracket resolved by a candle and by the trades inside it
- `bt.ticks.simulate_brackets(...)`: the whole trade log re-asked in one
Rust call - the tape read once, every bracket resolved in parallel
- `fill_resolution="ticks"`: level orders resolved on the bar's own trades
- `result.tape_resolution`: what the tape decided, and what fell back
- the return a bar backtest books on the bars it had to guess at
NOT RUNNABLE YET. The tick layer (`bt.ticks`) is not unlocked by any licence
sold today, a Pro one included: the code ships ahead of its availability, so
this file is here to be READ. It will run when the layer opens; nothing in it
is a placeholder, and the numbers it prints are the ones it will print then.
NOT RUNNABLE YET. Reading a stored tape is not unlocked by any licence sold
today, a Pro one included: the code ships ahead of its availability, so this
file is here to be READ. On the gate it exits with the refusal message rather
than a traceback; nothing in it is a placeholder, and the numbers it prints are
the ones it will print when the layer opens.
Requires pandas as well, alone among the files here: the comparison below
re-indexes the bars by timestamp and joins the trade log against them, which
is a DataFrame's job. `pip install manifoldbt[pandas]`.
The strategy is a device, not a claim: fees are zeroed and the bracket is a few
basis points wide, which is what makes single bars cover both levels often
enough to be countable. Read the SIGN of the gap, not its size.
Data: synthetic (seed 20260827) - generated by this file, reproducible.
The fixture is a plain random walk with no drift, so the DIRECTION of the gap
is meaningful and its size is not a forecast of any real market.
Data: self-contained (network) - one UTC day of a symbol's trades from a public
venue archive, fetched once into `data/tape/`, plus the 1-minute bars rebuilt
from that same tape. The bars ARE the trades, summarised, so the two series
cannot disagree. Real prices: the figures move with the day and the venue.
Usage:
python examples/27_bars_vs_tape.py
"""
import os
import tempfile
import numpy as np
try:
import pandas # noqa: F401 -- see the header: the join below needs it
except ModuleNotFoundError:
raise SystemExit(
"27_bars_vs_tape.py reads its results with pandas, which the engine "
"does not depend on. Install it with: pip install manifoldbt[pandas]"
)
from collections import Counter
from datetime import datetime, timedelta, timezone
import manifoldbt as mbt
from manifoldbt.helpers import Interval, Slippage
from manifoldbt.helpers import Interval, Slippage, time_range
from manifoldbt.indicators import close, sma
# -- The dataset: one tape, then the bars it summarises -----------------------
SEED = 20260827
N_TICKS = 400_000
BAR_SECONDS = 60
SL_PCT, TP_PCT = 0.6, 0.6 # symmetric, so neither level is favoured
# -- The data -----------------------------------------------------------------
# Any venue with a public trade archive will do. The feature reads whatever tape
# the store holds and does not care where it came from; these three constants
# are this file's choice, not the design.
PROVIDER = "bybit"
SYMBOL = "BTCUSDT"
SYMBOL_ID = 1
# The venues keep a rolling archive window, so a hardcoded date eventually stops
# existing and the ingest says so by name. Set DAY to a fixed "YYYY-MM-DD" to
# pin a run.
_TODAY = datetime.now(timezone.utc).date()
DAY = str(_TODAY - timedelta(days=3))
NEXT_DAY = str(_TODAY - timedelta(days=2))
_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
DATA_ROOT = os.path.join(_ROOT, "data", "tape")
METADATA_DB = os.path.join(DATA_ROOT, "metadata.sqlite")
# A stored tape is one Arrow file per UTC day under `{provider}/ticks/{symbol}/`
# (see `help(mbt.ingest_trades)`). Its presence is what tells this file the day
# has already been fetched.
TAPE_FILE = os.path.join(
DATA_ROOT, "mega", PROVIDER.lower(), "ticks", SYMBOL, f"{DAY}.arrow"
)
BAR = "1m"
SL_PCT = TP_PCT = 0.05 # symmetric, so neither level is favoured
def build_dataset(root):
"""Generate a tape, aggregate it into bars, return (tape path, store)."""
tape = os.path.join(root, "tape.csv")
mbt.ticks.generate_tape(
tape,
seed=SEED,
n_ticks=N_TICKS,
# ~120 ms between trades gives a tape long enough for ~800 one-minute
# bars, and a per-tick volatility that moves price through a 0.6%
# bracket often enough to produce contested bars.
mean_gap_ms=120.0,
vol_per_tick=0.00035,
drift_per_tick=0.0,
start_price=100.0,
)
batch = mbt.ticks.tape_to_bars(tape, BAR_SECONDS, symbol_id=1)
frame = batch.to_pandas()
store = mbt.import_dataframe(
frame[["timestamp", "open", "high", "low", "close", "volume"]],
symbol="TAPE", symbol_id=1, interval="1m",
data_root=os.path.join(root, "data"),
metadata_db=os.path.join(root, "meta.sqlite"),
)
return tape, store, frame
def open_store():
"""One day of tape, plus the bars rebuilt from that same tape.
Fetched once; a later run reuses what is on disk. Rebuilding the bars from
the tape rather than taking the venue's own means every bar is covered, so
`bars_fallen_back` should come back at zero.
"""
if os.path.exists(TAPE_FILE):
store = mbt.DataStore(data_root=DATA_ROOT, metadata_db=METADATA_DB)
else:
print(f"Fetching {SYMBOL} trades for {DAY} from {PROVIDER} (once)...")
store = mbt.ingest_trades(
PROVIDER, SYMBOL, symbol_id=SYMBOL_ID, start=DAY, end=DAY,
data_root=DATA_ROOT, metadata_db=METADATA_DB,
)
built = mbt.bars_from_trades(store, SYMBOL, DAY, NEXT_DAY, interval=BAR)
return store, built
# -- The strategy: an ordinary bracket, nothing exotic ------------------------
@@ -106,101 +119,93 @@ strategy = (
)
def config_for(frame):
last = int(frame["timestamp"].iloc[-1].value)
def config_for(fill_resolution):
"""The same config twice, one field apart."""
start, end = time_range(DAY, NEXT_DAY)
return mbt.BacktestConfig(
universe=[1],
time_range_start=0,
time_range_end=last + 60_000_000_000,
universe=[SYMBOL_ID],
time_range_start=start,
time_range_end=end,
bar_interval=Interval.minutes(1),
initial_capital=10_000,
fees=mbt.FeeConfig.zero(), # costs would only blur the comparison
fees=mbt.FeeConfig.zero(), # costs would only blur the comparison
slippage=Slippage.none(),
warmup_bars=40,
execution=mbt.ExecutionConfig(
signal_delay=1, max_position_pct=1.0, allow_short=False,
signal_delay=1,
max_position_pct=1.0,
allow_short=False,
position_sizing_mode="FractionOfEquity",
fill_resolution=fill_resolution,
),
)
REASON = {0: "signal", 1: "stop", 2: "target", 3: "trailing", 4: "expiry"}
REASON = {1: "stop", 2: "target", 3: "trailing"}
def bracket_exits(result):
"""Exits by reason, read off the Arrow trade log (no pandas needed)."""
codes = result.trades.column("exit_reason").to_pylist()
return Counter(REASON[c] for c in codes if c in REASON)
def coverage_warning(result):
"""The one warning the engine raises when a bar had no trade to read."""
for w in result.warnings:
if "no tape coverage" in w:
return w
return None
def row(label, left, right):
print(f" {label:<34}{left:>13}{right:>13}")
if __name__ == "__main__":
root = tempfile.mkdtemp(prefix="mbt_ex27_")
try:
tape, store, frame = build_dataset(root)
store, built = open_store()
bar_run = mbt.run(strategy, config_for("bar"), store)
tape_run = mbt.run(strategy, config_for("ticks"), store)
except PermissionError as exc:
raise SystemExit(f"\n {exc}\n") from None
info = mbt.ticks.tape_info(tape)
print(f"One tape: {info['n_trades']:,} trades over "
f"{info['span_seconds'] / 3600:.1f} hours")
print(f"Aggregated into {len(frame):,} bars of {BAR_SECONDS}s, "
f"in the engine's canonical schema.\n")
print(f"\n{SYMBOL} {DAY}, {built['bars']:,} bars of {BAR} rebuilt from the "
f"day's own trades.")
print(f"Bracket -{SL_PCT}% / +{TP_PCT}%, symmetric, fees and slippage off.\n")
result = mbt.run(strategy, config_for(frame), store)
trades = result.trades_df()
bars = frame.set_index("timestamp")
bar_exits, tape_exits = bracket_exits(bar_run), bracket_exits(tape_run)
tape = tape_run.tape_resolution
# -- Re-ask each bracket exit of the tape ---------------------------------
# An exit is CONTESTED when its bar's range covered both levels: that is
# exactly the bar on which the engine had to choose.
entries = trades[trades["side"] == 1].reset_index(drop=True)
exits = trades[trades["side"] == 2].reset_index(drop=True)
row("", "bar rule", "on the tape")
print(" " + "-" * 60)
row("total return",
f"{bar_run.metrics['total_return']:+.2%}",
f"{tape_run.metrics['total_return']:+.2%}")
row("exits on the stop", bar_exits["stop"], tape_exits["stop"])
row("exits on the target", bar_exits["target"], tape_exits["target"])
row("tape_resolution", str(bar_run.tape_resolution), "(below)")
print()
row("bars resolved on trades", "-", f"{tape['bars_on_tape']:,}")
row("bars fallen back to the bar rule", "-", f"{tape['bars_fallen_back']:,}")
row("bars covering BOTH levels", "-", f"{tape['contested_bars']:,}")
row(" the target traded first", "-", f"{tape['contested_tp_first']:,}")
row(" the stop traded first", "-", f"{tape['contested_stop_first']:,}")
row("entries triggered on a trade", "-", f"{tape['entries_on_tape']:,}")
disputed = [] # (bar's verdict, sl, tp, entry data)
for i in range(min(len(entries), len(exits))):
entry_px = float(entries.loc[i, "fill_price"])
reason = int(exits.loc[i, "exit_reason"])
if reason not in (1, 2): # only bracket exits are in question
continue
sl, tp = entry_px * (1 - SL_PCT / 100), entry_px * (1 + TP_PCT / 100)
bar = bars.loc[exits.loc[i, "execution_timestamp"]]
if bar["low"] <= sl and bar["high"] >= tp:
disputed.append((reason, entry_px, sl, tp,
int(entries.loc[i, "execution_timestamp"].value)))
phantom = bar_run.metrics["total_return"] - tape_run.metrics["total_return"]
print(f"\n Phantom return: {phantom:+.2%}")
print(f" The bar run minus the tape run, over {tape['contested_bars']} "
"bars where the bar rule had to")
print(" choose. It books the stop every time; on "
f"{tape['contested_tp_first']} of them the target had already")
print(" traded, so the loss went into the result and never into the trades.")
# One call resolves them all: the tape is read once and every bracket is
# replayed in parallel, in Rust. `entry=0` triggers on the first trade at
# or after `start_times_ns[i]`: the entry is not re-simulated, it already
# happened; only the race between the two levels is put to the tape.
verdicts = mbt.ticks.simulate_brackets(
tape,
sides=["long"] * len(disputed),
entries=[0.0] * len(disputed),
sls=[d[2] for d in disputed],
tps=[d[3] for d in disputed],
start_times_ns=[d[4] for d in disputed],
)
contested, agreed, overturned, pnl_gap = len(disputed), 0, 0, 0.0
for (reason, entry_px, sl, tp, _), truth in zip(disputed, verdicts):
tape_reason = {"stop_loss": 1, "take_profit": 2}.get(truth["exit_reason"])
if tape_reason == reason:
agreed += 1
elif tape_reason is not None:
overturned += 1
# The bar booked the stop; the tape says the target traded first.
pnl_gap += (tp - sl) / entry_px
bracket_exits = int(exits["exit_reason"].isin([1, 2]).sum())
print(f" {'bracket exits':<34}{bracket_exits:>8}")
print(f" {'of which the bar had to guess':<34}{contested:>8}"
f" ({contested / max(bracket_exits, 1):.0%})")
print(f" {'guessed right':<34}{agreed:>8}")
print(f" {'guessed wrong':<34}{overturned:>8}"
f" ({overturned / max(contested, 1):.0%} of the guesses)")
print(f"\n Those wrong guesses booked {pnl_gap:.1%} of return that the "
f"trades\n themselves never lost: each one recorded a "
f"{SL_PCT:.1f}% stop where the\n tape shows a {TP_PCT:.1f}% target.")
print(f" ({contested} contested exits is a small sample: read the sign of"
f" the gap,\n not its exact rate.)")
print(f"\n Reported result: {result.metrics['total_return']:+.1%} "
f"({bracket_exits} bracket exits, "
f"{contested} of them assumed rather than observed)")
missing = coverage_warning(tape_run)
if missing:
print(f"\n [{missing}]")
print(" Those bars kept the bar rule: the feature degrades, it does "
"not refuse.")
print("\n The bars are not wrong, they are incomplete: a candle records")
print(" where price went, never in what order. On a strategy whose exits")
examples/README.md
+9 -6
View File
@@ -135,15 +135,18 @@ The self-contained and synthetic examples need none of this.
| 24 | `24_option_spread.py` | a two-leg option structure | network |
| 25 | `25_lookahead_trap.py` | the look-ahead no re-run can detect | synthetic ⚠ |
| 26 | `26_fill_costs.py` | the same signal filled four ways, and what each costs | shared store |
| 27 | `27_bars_vs_tape.py` | the exit a candle has to guess, put back to the trades | synthetic, **not runnable yet** |
| 27 | `27_bars_vs_tape.py` | the same bracket resolved on the candle and on the trades inside it | network, **not runnable yet** |
⚠ marks the two files whose fixture determines the outcome, as described above.
**27 does not run today, on any licence.** The tick layer it uses is not
unlocked by anything sold today: the code ships ahead of its availability.
The file is there to be read, and it will run unchanged when the layer
opens. It is also the only example that wants pandas, for the
join between its trade log and its bars.
**27 does not run today, on any licence.** Reading a stored tape is not
unlocked by anything sold today: the code ships ahead of its availability. The
file is there to be read, and it will run unchanged when the layer opens -- on
the gate it exits with the refusal message rather than a traceback. It is the
only example that both fetches a trade tape and reads it back through `run()`:
`fill_resolution="ticks"` resolves stop-loss, take-profit, trailing and entry
levels against the individual trades of each bar instead of against its high
and low, and `result.tape_resolution` counts what the tape decided.
Two files here are not examples and carry no number:
pyproject.toml
+1 -1
View File
@@ -5,7 +5,7 @@
[project]
name = "manifoldbt"
version = "0.23.0"
version = "0.24.0"
description = "Rust-powered backtesting engine for quantitative research"
readme = "README.md"
requires-python = ">=3.9"
python/manifoldbt/__init__.py
+129 -3
View File
@@ -40,6 +40,7 @@ from manifoldbt._native import (
BatchResultLite,
DataStore,
activate,
_flush_usage as _flush_usage_native,
license_expiry as _license_expiry,
license_info as _license_info,
compile_strategy_json,
@@ -169,6 +170,13 @@ def _print_pro_summary() -> None:
import atexit
atexit.register(_print_pro_summary)
# Fold this session's anonymous usage counters into their file on the way out.
# Without it a script short enough never to reach the periodic flush (most
# scripts) would count everything and persist nothing. Order does not matter:
# this neither prints nor reads anything the hooks around it touch, and it is a
# no-op under MANIFOLDBT_NO_TELEMETRY / DO_NOT_TRACK / CI.
atexit.register(_flush_usage_native)
def check_for_update() -> Optional[str]:
"""Ask PyPI whether a newer manifoldbt exists. Returns its version, or None.
@@ -889,6 +897,116 @@ def ingest(
return store
def ingest_trades(
provider: str,
symbol: str,
symbol_id: int,
start: str,
end: str,
*,
data_root: str = "data",
metadata_db: str = "metadata/metadata.sqlite",
exchange: Optional[str] = None,
asset_class: str = "crypto_spot",
progress: bool = True,
) -> DataStore:
"""Ingest whole days of trades (a tape) from a venue's public archive.
The tick-level counterpart of :func:`ingest`, with the same shape: you name
a provider, a symbol and a range, the provider fetches, the store keeps one
Arrow file per UTC day under ``{provider}/ticks/{symbol}/``. Nothing is
fetched that you did not ask for.
Providers with a public trade archive: ``"bybit"`` (spot, one ``.csv.gz``
per day) and ``"binance"`` (spot aggTrades, one ``.zip`` per day). Both
keep a rolling window, so an old day raises a named error rather than
returning nothing. ``start`` and ``end`` are ``YYYY-MM-DD`` (an RFC 3339
instant is accepted; its date part is used), ``end`` inclusive.
Part of the tick layer: not unlocked by any licence sold today, a Pro one
included; the engine says so before touching the network.
Example::
store = bt.ingest_trades("bybit", "BTCUSDT", symbol_id=1,
start="2026-08-25", end="2026-08-25")
"""
from manifoldbt._native import py_ingest_trades as _native
cb = None
if progress:
def cb(i, n, day):
if i < n:
print(f" tape {symbol.upper()} {day} ({i + 1}/{n})", flush=True)
return _native(
provider, symbol, int(symbol_id), start, end, data_root, metadata_db,
exchange, asset_class, cb,
)
def bars_from_trades(
store: DataStore,
symbol,
start: str,
end: str,
*,
interval: str = "1s",
) -> dict:
"""Rebuild bars from the tape stored for a symbol, and write them into
the store at ``interval``.
A venue's own bars carry one total volume. Bars rebuilt from the tape
carry what the tape carries: ``buy_volume`` and ``sell_volume`` split by
the aggressor side, ``vwap`` and ``trade_count``. They are ordinary bars
for everything downstream: ``run()`` reads them at ``interval`` or
resamples them coarser, and the DSL addresses the flow columns by name
(``col("buy_volume")``). Seconds with no trade produce no bar. One second
is the finest step the store holds.
Args:
store: The store holding the symbol's tape (see :func:`ingest_trades`).
symbol: The store ticker (e.g. ``"BTCUSDT"``) or symbol id.
start: First instant, ISO date or datetime (``"2026-08-25"``).
end: Last instant, exclusive (``"2026-08-26"`` for one full day).
interval: ``"1s"``, ``"1m"``, ``"5m"``, ``"1h"``, ... (default ``"1s"``).
Returns:
``{"bars", "version", "interval_ns", "first_ts", "last_ts"}``.
Example::
store = bt.ingest_trades("bybit", "BTCUSDT", symbol_id=1,
start="2026-08-25", end="2026-08-25")
bt.bars_from_trades(store, "BTCUSDT", "2026-08-25", "2026-08-26")
config = bt.BacktestConfig(..., bar_interval=Interval.seconds(1))
result = bt.run(strategy, config, store)
"""
from manifoldbt.helpers import time_range as _time_range
# No Python-side licence check: the native side answers, with the
# accurate refusal. Same policy as manifoldbt.ticks and attach_quotes.
if isinstance(symbol, int):
symbol_id = symbol
else:
symbol_id = store.resolve_symbol(str(symbol))
start_ns, end_ns = _time_range(start, end)
if end_ns <= start_ns:
raise ValueError(
f"end {end!r} must be after start {start!r} (end is exclusive: "
"one full day is start='2026-08-25', end='2026-08-26')"
)
from manifoldbt._native import py_bars_from_trades as _native
try:
return _native(
store.data_root(), store.metadata_db(), symbol_id, interval, start_ns, end_ns
)
except (ValueError, RuntimeError) as exc:
raise _classify_error(exc) from exc
def attach_quotes(
store: DataStore,
symbol,
@@ -1023,7 +1141,8 @@ def import_csv(
path: Path to the CSV file (standard / MT4 / MT5 format).
symbol: Ticker name (e.g. ``"EURUSD"``, ``"BTCUSDT"``).
symbol_id: Unique integer ID for this symbol in the store.
interval: Bar interval of the rows (``"1m"``, ``"5m"``, ``"1h"``, ``"1d"``, ...).
interval: Bar interval of the rows (``"1s"``, ``"1m"``, ``"5m"``, ``"1h"``,
``"1d"``, ...). The store holds a tier per interval, down to one second.
data_root: Store directory (default ``"data"``).
metadata_db: Metadata SQLite path.
exchange: Exchange label for metadata (default ``"CSV"``).
@@ -1159,7 +1278,8 @@ def import_dataframe(
data: pandas/polars DataFrame or dict of columns.
symbol: Ticker name (e.g. ``"EURUSD"``, ``"BTCUSDT"``).
symbol_id: Unique integer ID for this symbol in the store.
interval: Bar interval of the rows (``"1m"``, ``"5m"``, ``"1h"``, ``"1d"``, ...).
interval: Bar interval of the rows (``"1s"``, ``"1m"``, ``"5m"``, ``"1h"``,
``"1d"``, ...). The store holds a tier per interval, down to one second.
data_root: Store directory (default ``"data"``).
metadata_db: Metadata SQLite path.
exchange: Exchange label for metadata (default ``"DATAFRAME"``).
@@ -2039,7 +2159,8 @@ start, end = bt.time_range("2022-01-01", "2025-01-01") # UNIX NANOSECONDS -- n
cfg = bt.BacktestConfig(
universe=["BTCUSDT"], # tickers, ids, or {"provider": [...]}
time_range_start=start, time_range_end=end,
bar_interval=bt.Interval.hours(1),
bar_interval=bt.Interval.hours(1), # one minute if left out; down to seconds(1)
# on Pro, minutes(1) at the finest on Community
initial_capital=100_000.0,
warmup_bars=50,
execution=bt.ExecutionConfig(signal_delay=1, # bars between signal and fill
@@ -2100,6 +2221,9 @@ Common errors
reference as "provider:TICKER" with a dict universe.
"cross-sectional op ... must be the whole signal" -> give the cs op its own
signal; thresholding one is not supported (see the recipe above).
"bar_interval below one minute requires a Pro license" -> sub-minute simulation is
Pro; Community runs at one minute at the finest. A config that leaves
bar_interval out runs at one minute.
"""
@@ -2112,6 +2236,8 @@ __all__ = [
"SweepResult",
# Data ingestion
"ingest",
"ingest_trades",
"bars_from_trades",
"attach_quotes",
"import_csv",
"import_dataframe",
python/manifoldbt/_native.pyi
+7
View File
@@ -50,6 +50,8 @@ class BacktestResult:
def trade_count(self) -> int: ...
@property
def fill_fragility(self) -> Optional[Dict[str, int]]: ...
@property
def tape_resolution(self) -> Optional[Dict[str, int]]: ...
class AlignedData:
@@ -139,3 +141,8 @@ def py_run_monte_carlo(
def py_run_stochastic(
sim_config_json: str,
) -> Dict[str, Any]: ...
# Anonymous usage counters. Private: they exist so a user can see what the
# install ping sends, and so the tests can assert on it.
def _usage_snapshot() -> Dict[str, Any]: ...
def _flush_usage() -> None: ...
python/manifoldbt/cli.py
+1 -1
View File
@@ -13,7 +13,7 @@ import sys
def main() -> None:
parser = argparse.ArgumentParser(prog="manifoldbt", description="ManifoldBT CLI")
parser = argparse.ArgumentParser(prog="manifoldbt", description="Manifold-BT CLI")
sub = parser.add_subparsers(dest="command")
# ── ingest ────────────────────────────────────────────────────────────
python/manifoldbt/config.py
+22 -3
View File
@@ -190,6 +190,14 @@ class ExecutionConfig:
conservative, and the run stays on the general loop). The result's
fill_fragility counters say how many maker fills the choice affects.
"""
fill_resolution: Optional[str] = None
"""What level orders (stop-loss, take-profit, trailing stop, limit and stop
entries) are resolved against inside a bar: "bar" (default) uses the bar's
high and low, "ticks" walks the individual trades of the bar in time order,
so the order the market actually printed decides which level came first.
Needs a stored tape for the symbol; a bar with no trade in its window falls
back to the bar rule, and the result's tape_resolution counters say how
often that happened. Merged into fill_model when set."""
orders: Optional[OrderConfig] = None
"""Order management: limit entries, stop-loss, take-profit, trailing stops.
When None (default), the engine uses the legacy market-order path."""
@@ -207,6 +215,11 @@ class ExecutionConfig:
}
if self.fill_model is not None:
d["fill_model"] = self.fill_model
if self.fill_resolution is not None:
d["fill_model"] = {
**(self.fill_model or {}),
"fill_resolution": self.fill_resolution,
}
if self.orders is not None:
d["orders"] = self.orders.to_json_dict()
return d
@@ -349,6 +362,9 @@ class BacktestConfig:
initial_capital: float = 1000.0
currency: str = "USD"
bar_interval: Any = None
"""Bar step the simulation runs on, e.g. ``Interval.hours(1)``.
``None`` means one minute. Steps below one minute are a Pro feature;
Community simulates at one minute at the finest, whatever the store holds."""
execution: ExecutionConfig = field(default_factory=ExecutionConfig)
fees: FeeConfig = field(default_factory=FeeConfig)
slippage: Any = None
@@ -362,8 +378,11 @@ class BacktestConfig:
non-zero rate (e.g. 0.025) for an excess-return Sharpe."""
output_resolution: Any = None
"""Downsample output timeseries (equity, positions).
None = auto (uses resample_to if set, else bar_interval; min 1h).
Use ``{"Hours": 1}``, ``{"Days": 1}``, etc. for explicit control."""
None = auto (uses resample_to if set, else bar_interval), so the output
keeps the step the simulation ran at.
Use ``{"Seconds": 1}``, ``{"Hours": 1}``, ``{"Days": 1}``, etc. for explicit
control. Pro returns any step down to one second; Community output is
coarsened to daily."""
resample_to: Any = None
"""Resample raw bars to this interval before simulation.
E.g. ``{"Minutes": 60}`` aggregates 1-min data into 60-min OHLCV bars.
@@ -444,7 +463,7 @@ class BacktestConfig:
"start": self.time_range_start,
"end": self.time_range_end,
},
"bar_interval": self.bar_interval or {"Seconds": 1},
"bar_interval": self.bar_interval or {"Minutes": 1},
"initial_capital": self.initial_capital,
"currency": self.currency,
"execution": self.execution.to_json_dict(),
python/manifoldbt/ticks.py
+8
View File
@@ -13,6 +13,14 @@ data.binance.vision, or one from :func:`generate_tape` for a reproducible
example. Bar-level backtests are unaffected by this module; it is a separate
layer, not a change to :func:`manifoldbt.run`.
The bar engine has its own door to a tape, and it is not this module:
``ExecutionConfig(fill_resolution="ticks")`` makes :func:`manifoldbt.run`
resolve level orders (stop-loss, take-profit, trailing stop, limit and stop
entries) against the trades stored for the symbol -- put there by
:func:`manifoldbt.ingest_trades`, not by a CSV path -- instead of against each
bar's high and low, and ``result.tape_resolution`` counts what the tape
decided. Same tier, different entry point.
Three ways to run a strategy on a tape:
* :func:`run_orderflow` - the built-in aggressor-imbalance strategy,
python/tests/test_resolution_1s.py
+92
View File
@@ -0,0 +1,92 @@
"""One-second bars: Pro simulates and returns them, Community stops at the minute.
The store holds a one-second tier since 0.24.0. Two rules sit on top of it:
* simulating on bars finer than one minute needs Pro (``resolution_1s``),
enforced by the engine core on every runner and answered early by the
bindings;
* the output floor is one second on Pro and one day on Community, so on Pro a
one-second run comes back with one point per bar.
The Pro cases run on a Pro machine and in the engine's own debug builds, which
is what the CI runs; the Community cases skip there, like ``test_combo_limit``.
"""
import os
import tempfile
import numpy as np
import pandas as pd
import pytest
import manifoldbt as bt
IS_PRO = bt.license_info()[0] == "Pro"
N_BARS = 3 * 3600 # three hours of one-second bars
@pytest.fixture(scope="module")
def one_second_store():
root = tempfile.mkdtemp()
idx = pd.date_range("2024-01-01", periods=N_BARS, freq="1s", tz="UTC")
close = 100.0 + np.cumsum(np.random.default_rng(1).normal(0, 0.01, N_BARS))
df = pd.DataFrame({
"timestamp": idx, "open": close, "high": close + 0.02,
"low": close - 0.02, "close": close, "volume": np.full(N_BARS, 10.0),
})
return bt.import_dataframe(
df, symbol="XS", symbol_id=1, interval="1s",
data_root=os.path.join(root, "data"),
metadata_db=os.path.join(root, "meta.sqlite"),
)
def _strategy():
return bt.Strategy(
name="always_long",
signals={"signal": bt.lit(1.0)},
position_sizing=bt.lit(1.0) * bt.param("size", default=1.0),
parameters={"size": bt.param("size", default=1.0)},
)
def _config(interval):
t0, t1 = bt.time_range("2024-01-01", "2024-01-02")
return bt.BacktestConfig(
universe=[1], time_range_start=t0, time_range_end=t1, bar_interval=interval,
)
def test_one_second_bars_land_in_their_own_tier(one_second_store):
root = one_second_store.data_root()
tiers = {
d for _, dirs, _ in os.walk(root) for d in dirs if d in ("1s", "1m", "bars_1s", "bars_1m")
}
assert "1s" in tiers or "bars_1s" in tiers, tiers
# Nothing was silently rewritten as one-minute data.
assert "1m" not in tiers and "bars_1m" not in tiers, tiers
@pytest.mark.skipif(not IS_PRO, reason="Pro (or the debug override): one-second simulation")
def test_pro_simulates_and_returns_every_second(one_second_store):
result = bt.run(_strategy(), _config({"Seconds": 1}), one_second_store)
# One point per bar: the output floor is one second, not one minute.
assert len(result.equity_curve) == N_BARS
@pytest.mark.skipif(not IS_PRO, reason="Pro (or the debug override): one-second simulation")
def test_one_minute_run_on_the_same_store_is_coarser(one_second_store):
result = bt.run(_strategy(), _config({"Minutes": 1}), one_second_store)
assert len(result.equity_curve) == N_BARS // 60 + 1
@pytest.mark.skipif(IS_PRO, reason="Community-only; needs a machine without a Pro licence")
def test_community_is_refused_below_one_minute(one_second_store):
with pytest.raises(PermissionError, match="Pro"):
bt.run(_strategy(), _config({"Seconds": 1}), one_second_store)
@pytest.mark.skipif(IS_PRO, reason="Community-only; needs a machine without a Pro licence")
def test_community_still_runs_at_one_minute_on_a_one_second_store(one_second_store):
result = bt.run(_strategy(), _config({"Minutes": 1}), one_second_store)
# Daily output floor: three hours collapse onto a single day.
assert len(result.equity_curve) <= 2