698 lines
36 KiB
Markdown
698 lines
36 KiB
Markdown
# EdgeEngine v2.0.0 — Open-Source MetaTrader 5 Strategy Tester Replica
|
||
|
||
> **100% Genuine Broker Data • 1:1 MT5 Accounting & Execution Lifecycle • C++17 Core with AVX2 & Python Bindings**
|
||
> An open-source, general-purpose simulation engine replicating the execution semantics, tick generation, latency queues, and optimization algorithms of the MetaTrader 5 Strategy Tester.
|
||
|
||
---
|
||
|
||
## 📑 Table of Contents
|
||
1. [Architecture & Design Philosophy](#1-architecture--design-philosophy)
|
||
2. [Directory Structure](#2-directory-structure)
|
||
3. [Installation & Build](#3-installation--build)
|
||
4. [Downloading & Storing 100% Broker Data](#4-downloading--storing-100-broker-data)
|
||
5. [Implementing Custom Strategies](#5-implementing-custom-strategies)
|
||
6. [Simulation Modes (How to Choose the Right Mode)](#6-simulation-modes-how-to-choose-the-right-mode)
|
||
- [6.1 Real Broker Tick Replay (TickMode.REAL_TICKS)](#61-real-broker-tick-replay-tickmodereal_ticks)
|
||
7. [Execution Latency & Slippage Simulation](#7-execution-latency--slippage-simulation)
|
||
8. [Overnight Financing & Rollover Swap Mechanics (24/7 vs. Traditional Markets)](#8-overnight-financing--rollover-swap-mechanics-247-vs-traditional-markets)
|
||
9. [MT5 Strategy Tester Parity Engine (Order Lifecycle & Accounting)](#9-mt5-strategy-tester-parity-engine-order-lifecycle--accounting)
|
||
10. [Interactive Step-by-Step & Visual Replay](#10-interactive-step-by-step--visual-replay)
|
||
11. [Optimization & Hyperopt Guide](#11-optimization--hyperopt-guide)
|
||
12. [Engine Updates & Porting Guide (Changelog for Clones)](#12-engine-updates--porting-guide-changelog-for-clones)
|
||
13. [Test Suite & Verification](#13-test-suite--verification)
|
||
|
||
---
|
||
|
||
## 1. Architecture & Design Philosophy
|
||
|
||
Unlike simple backtest scripts that calculate hypothetical returns by multiplying signals by bar returns, **EdgeEngine is an event-driven 1:1 software replica of the MetaTrader 5 Strategy Tester**.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Data["Historical OHLCV Bars / Broker Ticks"] --> TG["TickGenerator<br/>(OPEN_PRICES / OHLC_M1 / EVERY_TICK / REAL_TICKS)"]
|
||
TG -->|MqlTick Stream| EQ["Execution Delay Queue<br/>(Latency: NONE / FIXED / RANDOM)"]
|
||
EQ -->|Elapsed Ticks| OB["OrderBook & Matching Engine<br/>(Pending Orders, SL/TP Triggers, Deals)"]
|
||
OB <-->|Margin / Balance / Equity| AM["Account Manager<br/>(Hedging / Netting, Margin Calls, Stopouts)"]
|
||
OB -->|OnTick / OnBar Events| Strat["Strategy Instance<br/>(OnInit ➔ OnTick ➔ OnBar ➔ OnTester)"]
|
||
Strat -->|CTrade: Buy / Sell / Modify| OB
|
||
```
|
||
|
||
### Core Tenets:
|
||
1. **Transaction Ledger Integrity**:
|
||
- Follows MT5's exact transaction flow: `MqlTradeRequest` $\to$ `Order` $\to$ `Deal` $\to$ `Position`.
|
||
- Positions are never treated as abstract numbers; every open position has ticket identifiers, entry timestamps, open price, volume, SL/TP levels, swap accruals, and continuous tracking of Maximum Favorable Excursion (MFE) and Maximum Adverse Excursion (MAE).
|
||
2. **Point Specification Accuracy**:
|
||
- PnL is computed strictly from the instrument's contract specifications:
|
||
$$\text{PnL} = \frac{\Delta\text{Price}}{\text{TradeTickSize}} \times \text{TradeTickValue} \times \text{Volume}$$
|
||
- Includes full spread modeling (fixed or dynamic per-bar spread arrays), round-turn commissions, and overnight holding swaps (long/short).
|
||
3. **No Synthetic Fallbacks for Price Data**:
|
||
- Strictly adheres to the project protocol: **all testing and optimizations operate exclusively on real broker historical candles** directly downloaded from the MetaTrader 5 terminal server.
|
||
|
||
---
|
||
|
||
## 2. Directory Structure
|
||
|
||
```
|
||
edge_engine/
|
||
├── core/
|
||
│ ├── include/ # Header-only C++17 Core Engine
|
||
│ │ ├── mql_types.hpp # MT5 structs (MqlTick, MqlRates, MqlTradeRequest, Deal, etc.)
|
||
│ │ ├── account.hpp # Balance, equity, leverage, margin requirement, stop-out
|
||
│ │ ├── order_book.hpp # Matching engine, in-flight latency queue, pending orders
|
||
│ │ ├── strategy_base.hpp # Abstract Strategy base class & TradeContext (CTrade)
|
||
│ │ ├── tick_generator.hpp # MT5 intrabar tick wave synthesizers (OHLC_M1, EVERY_TICK)
|
||
│ │ ├── interactive_tester.hpp # Step-by-step visual & interactive replay engine
|
||
│ │ ├── tester.hpp # Master simulation loop orchestrator
|
||
│ │ └── types.hpp # Shared enums (TickMode, LatencyMode, OptAlgo, ExitMethod)
|
||
│ │
|
||
│ └── src/
|
||
│ ├── bindings.cpp # pybind11 module for legacy fast engine (edge_engine_core)
|
||
│ └── tester_bindings.cpp # pybind11 module for general-purpose tester (edge_tester)
|
||
│
|
||
├── python/edge_engine/ # High-Level Python Package
|
||
│ ├── __init__.py # Top-level exports (BacktestEngine, StrategyOptimizer, etc.)
|
||
│ ├── optimizer.py # Multi-core Grid & Genetic StrategyOptimizer
|
||
│ ├── symbol_spec.py # Symbol contract specifications and helpers
|
||
│ └── param_builder.py # Parameter grid utilities
|
||
│
|
||
├── tests/ # 38 Unit & Parity Tests
|
||
│ ├── test_advanced_features.py # Latency queues, OHLC_M1, Interactive replay, OnTester
|
||
│ ├── test_general_tester.py # MT5 CTrade, accounting, margin call & stopout integrity
|
||
│ └── test_parity.py # Strict numerical parity tests vs. legacy engine
|
||
│
|
||
├── CMakeLists.txt # Build configuration
|
||
└── pyproject.toml # Packaging and build instructions
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Installation & Build
|
||
|
||
### Requirements
|
||
- **OS**: Windows 10/11 64-bit
|
||
- **Compiler**: Visual Studio 2022 (MSVC C++17) with CMake
|
||
- **Python**: 3.10 – 3.13 (64-bit)
|
||
|
||
### Compiling and Installing
|
||
Open PowerShell in the `edge_engine` directory:
|
||
|
||
```powershell
|
||
# Navigate to the edge_engine directory
|
||
cd "c:\Users\Ethan\Desktop\projects\new leaf indicator\edge_engine"
|
||
|
||
# Compile and install directly into your active Python environment
|
||
pip install . --no-build-isolation --no-deps
|
||
```
|
||
|
||
Verify the installation:
|
||
```powershell
|
||
python -c "import edge_tester as et, edge_engine as ee; print('EdgeEngine installed successfully!')"
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Downloading & Storing 100% Broker Data
|
||
|
||
To guarantee 100% genuine broker data without synthetic curve-fitting, use the MetaTrader 5 Python API to download directly from the broker's terminal into high-performance Parquet storage.
|
||
|
||
### Data Downloader Script
|
||
Save and run this script to download genuine broker data:
|
||
|
||
```python
|
||
"""
|
||
download_data.py
|
||
Downloads 100% genuine broker candles directly from MetaTrader 5 and caches them as Parquet.
|
||
"""
|
||
import os
|
||
import MetaTrader5 as mt5
|
||
import pandas as pd
|
||
|
||
# List of symbols to download
|
||
SYMBOLS = [
|
||
"Jump 75 Index",
|
||
"Volatility 75 Index",
|
||
"EURUSD",
|
||
]
|
||
TIMEFRAME = mt5.TIMEFRAME_M5
|
||
BARS_COUNT = 100_000
|
||
OUTPUT_DIR = "backtest_suite/historical_data_cache"
|
||
|
||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||
|
||
if not mt5.initialize():
|
||
raise RuntimeError(f"MT5 initialization failed: {mt5.last_error()}")
|
||
|
||
for symbol in SYMBOLS:
|
||
print(f"Fetching {BARS_COUNT:,} bars for {symbol}...")
|
||
mt5.symbol_select(symbol, True)
|
||
|
||
# Ingest in chunks to prevent socket timeout
|
||
chunk_size = 50_000
|
||
pos = 0
|
||
all_chunks = []
|
||
|
||
while pos < BARS_COUNT:
|
||
to_fetch = min(chunk_size, BARS_COUNT - pos)
|
||
rates = mt5.copy_rates_from_pos(symbol, TIMEFRAME, pos, to_fetch)
|
||
if rates is None or len(rates) == 0:
|
||
break
|
||
all_chunks.append(pd.DataFrame(rates))
|
||
pos += len(rates)
|
||
if len(rates) < to_fetch:
|
||
break
|
||
|
||
if not all_chunks:
|
||
print(f" Warning: No data returned for {symbol}")
|
||
continue
|
||
|
||
df = pd.concat(all_chunks, ignore_index=True)
|
||
df = df.drop_duplicates(subset=['time']).sort_values('time').reset_index(drop=True)
|
||
|
||
clean_name = symbol.lower().replace(" ", "_")
|
||
output_path = os.path.join(OUTPUT_DIR, f"{clean_name}_100k_m5.parquet")
|
||
df.to_parquet(output_path, engine="pyarrow", compression="snappy")
|
||
print(f" Saved {len(df):,} bars to {output_path}")
|
||
|
||
mt5.shutdown()
|
||
```
|
||
|
||
### Loading the Data for Backtesting
|
||
```python
|
||
import pandas as pd
|
||
|
||
df = pd.read_parquet("backtest_suite/historical_data_cache/jump_75_index_100k_m5.parquet")
|
||
# Ensure columns: open, high, low, close, time, tick_volume, spread
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Implementing Custom Strategies
|
||
|
||
Strategies subclass `edge_tester.Strategy`. Both snake_case and MQL5 PascalCase method overrides are supported.
|
||
|
||
### Lifecycle Callbacks
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
actor Engine as Tester Engine
|
||
participant Strat as Strategy Instance
|
||
participant Trade as CTrade API
|
||
participant Book as OrderBook & Margin
|
||
|
||
Engine->>Strat: OnInit()
|
||
Note over Strat: Initialize indicators & parameters
|
||
|
||
loop For each Bar & Tick
|
||
Engine->>Book: on_price_update(tick)
|
||
Note over Book: Check pending orders & SL/TP
|
||
Engine->>Strat: OnTick()
|
||
opt Bar Completed
|
||
Engine->>Strat: OnBar()
|
||
end
|
||
opt Trade Signal Triggered
|
||
Strat->>Trade: buy() / sell() / position_modify()
|
||
Trade->>Book: order_send()
|
||
Book-->>Strat: OnTradeTransaction(trans, req, res)
|
||
end
|
||
end
|
||
|
||
Engine->>Book: close_all_positions()
|
||
Engine->>Strat: OnDeinit(reason)
|
||
Engine->>Strat: OnTester()
|
||
Strat-->>Engine: Custom Fitness Value
|
||
```
|
||
|
||
- `on_init(self)` / `OnInit()`: Initialization hook. Return `et.INIT_SUCCEEDED` (0) or `et.INIT_FAILED` (1).
|
||
- `on_tick(self)` / `OnTick()`: Called on every price tick (synthetic or real).
|
||
- `on_bar(self)` / `OnBar()`: Called at every bar close.
|
||
- `on_trade_transaction(self, trans, req, res)` / `OnTradeTransaction(...)`: Event callback whenever an order or deal changes status.
|
||
- `on_deinit(self, reason)` / `OnDeinit(reason)`: Cleanup hook called after simulation finishes.
|
||
- `on_tester(self)` / `OnTester()`: Returns a custom scalar fitness value used during optimization.
|
||
|
||
### Trade API (`self.trade`)
|
||
The strategy has access to an MT5 `CTrade` equivalent via `self.trade`:
|
||
|
||
| Method | Description |
|
||
| :--- | :--- |
|
||
| `self.trade.buy(volume, sl=0.0, tp=0.0, comment="")` | Open a market Buy order |
|
||
| `self.trade.sell(volume, sl=0.0, tp=0.0, comment="")` | Open a market Sell order |
|
||
| `self.trade.buy_limit(volume, price, sl=0.0, tp=0.0)` | Place a pending Buy Limit order |
|
||
| `self.trade.sell_limit(volume, price, sl=0.0, tp=0.0)` | Place a pending Sell Limit order |
|
||
| `self.trade.buy_stop(volume, price, sl=0.0, tp=0.0)` | Place a pending Buy Stop order |
|
||
| `self.trade.sell_stop(volume, price, sl=0.0, tp=0.0)` | Place a pending Sell Stop order |
|
||
| `self.trade.position_close(ticket, volume=0.0)` | Close position by ticket |
|
||
| `self.trade.position_modify(ticket, sl, tp)` | Modify SL and TP of open position |
|
||
| `self.trade.order_delete(ticket)` | Cancel an active pending order |
|
||
| `self.trade.positions()` | List of active open positions (`ticket`, `volume`, `price_open`, `sl`, `tp`, `profit`, etc.) |
|
||
| `self.trade.orders()` | List of active pending orders |
|
||
| `self.trade.iClose(shift=0)` | Close price of bar (0 = current, 1 = previous) |
|
||
| `self.trade.iOpen(shift=0)` | Open price of bar |
|
||
| `self.trade.iHigh(shift=0)` | High price of bar |
|
||
| `self.trade.iLow(shift=0)` | Low price of bar |
|
||
| `self.trade.ask()` / `self.trade.bid()` | Current best market quote |
|
||
| `self.trade.account_balance()` / `equity()` | Current balance and floating equity |
|
||
|
||
### Example Strategy Implementation
|
||
|
||
```python
|
||
import edge_tester as et
|
||
|
||
class MovingAverageCross(et.Strategy):
|
||
fast_period: int = 12
|
||
slow_period: int = 26
|
||
risk_lot: float = 0.1
|
||
|
||
def on_init(self):
|
||
self.fast_buffer = []
|
||
self.slow_buffer = []
|
||
return et.INIT_SUCCEEDED
|
||
|
||
def on_bar(self):
|
||
t = self.trade
|
||
if t.bars() < self.slow_period + 1:
|
||
return
|
||
|
||
# Calculate Moving Averages using historical bar accessors
|
||
fast_sma = sum(t.iClose(i) for i in range(self.fast_period)) / self.fast_period
|
||
slow_sma = sum(t.iClose(i) for i in range(self.slow_period)) / self.slow_period
|
||
|
||
prev_fast = sum(t.iClose(i + 1) for i in range(self.fast_period)) / self.fast_period
|
||
prev_slow = sum(t.iClose(i + 1) for i in range(self.slow_period)) / self.slow_period
|
||
|
||
# Bullish Crossover
|
||
if prev_fast <= prev_slow and fast_sma > slow_sma:
|
||
if t.positions_total() == 0:
|
||
t.buy(self.risk_lot, sl=t.ask() - 10.0, tp=t.ask() + 20.0)
|
||
|
||
# Bearish Crossover
|
||
elif prev_fast >= prev_slow and fast_sma < slow_sma:
|
||
for pos in t.positions():
|
||
t.position_close(pos.ticket)
|
||
|
||
def on_tester(self):
|
||
# Custom Fitness: Sharpe Ratio multiplied by Profit Factor
|
||
res = self.trade.account_balance()
|
||
return res
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Simulation Modes (How to Choose the Right Mode)
|
||
|
||
EdgeEngine supports all 4 standard MetaTrader 5 tick generation models via `et.TickMode`:
|
||
|
||
```python
|
||
cfg = et.TesterConfig()
|
||
cfg.tick_mode = et.TickMode.OPEN_PRICES # Set mode here
|
||
```
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Start{"What kind of strategy are you testing?"} --> Q1{"Does the strategy only execute on bar close?"}
|
||
Q1 -- Yes --> M1["TickMode.OPEN_PRICES<br/>(⚡ Fastest: ~500k bars/s - Ideal for Large Grid Searches)"]
|
||
Q1 -- No --> Q2{"Are there intrabar SL/TP or limit orders?"}
|
||
Q2 -- Standard SL/TP --> M2["TickMode.OHLC_M1<br/>(🚀 Fast: 4 ticks/bar - MT5 4-point progression)"]
|
||
Q2 -- Trailing Stops / Scalping --> M3["TickMode.EVERY_TICK<br/>(🎯 High Precision: 12-point synthetic fractal wave)"]
|
||
Q2 -- Tick Data / HFT --> M4["TickMode.REAL_TICKS<br/>(🔬 Broker Accurate: 100% recorded broker ticks)"]
|
||
```
|
||
|
||
### Choosing the Right Mode
|
||
|
||
| Mode | Intrabar Steps | Speed | Best Used For |
|
||
| :--- | :---: | :---: | :--- |
|
||
| **`TickMode.OPEN_PRICES`** | 1 tick / bar | **Fastest** (~500k bars/sec) | Strategies that only execute logic on bar completion (e.g., standard Moving Average, MACD, Bar-Close signals). Ideal for rapid broad grid searches and parameter optimizations. |
|
||
| **`TickMode.OHLC_M1`** | 4 ticks / bar | **Fast** (~150k bars/sec) | The standard MT5 4-point intrabar progression ($\text{Open} \to \text{High/Low} \to \text{Low/High} \to \text{Close}$). Ideal for strategies with tight Stop-Loss or Take-Profit orders that could be triggered within the bar duration. |
|
||
| **`TickMode.EVERY_TICK`** | 12 ticks / bar | **Moderate** (~50k bars/sec) | MT5's fractal price generator synthesizing 12 price states across 3 sub-intervals. Ideal for precision backtesting trailing stops, breakeven triggers, and scalping logic. |
|
||
| **`TickMode.REAL_TICKS`** | Every actual tick | **Exhaustive** | Pure replay of recorded real broker tick history (bid, ask, tick volume). Used for high-frequency strategies and final validation runs. |
|
||
|
||
### 6.1 Real Broker Tick Replay (`TickMode.REAL_TICKS`)
|
||
|
||
While synthetic models (`OPEN_PRICES`, `OHLC_M1`, `EVERY_TICK`) generate artificial intrabar paths from OHLC bar summaries, `TickMode.REAL_TICKS` streams **100% genuine broker ticks** (`time_msc`, `bid`, `ask`, `last`, `volume`, `flags`) directly into the matching engine.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
MT5["MT5 Terminal Server<br/>(copy_ticks_range)"] --> Downloader["download_broker_ticks.py"]
|
||
Downloader --> Parquet["tick_data_cache/*.parquet<br/>(Snappy Compressed)"]
|
||
Parquet --> CppEngine["EdgeEngine C++ Core<br/>(Two-Pointer Zero-Copy Replay)"]
|
||
CppEngine --> Match["Intrabar Order Matching<br/>(SL / TP / Trailing Stops / Latency)"]
|
||
```
|
||
|
||
#### Why Real Ticks are Essential
|
||
On high-volatility instruments (such as Deriv Jump & Volatility indices, or during Forex news releases), synthetic fractal models smooth out intrabar spikes. Real broker feeds, however, contain microsecond-level volatility bursts that can prematurely trigger tight Stop Losses or dynamic Trailing Stops.
|
||
|
||
In our production validation of Jump 10 Index:
|
||
- **`OPEN_PRICES`**: Signals and trailing stops only evaluate at candle close. This produces theoretical backtest profits without intrabar stopouts.
|
||
- **`REAL_TICKS` / MT5 "Every tick based on real ticks"**: Micro-spikes continuously hit trailing stops intrabar. This reduces realized profits to ~30% of open-price theoretical metrics, but reflects 100% genuine live trading dynamics.
|
||
|
||
#### C++ Zero-Copy Execution Architecture
|
||
1. **$O(N)$ Two-Pointer Grouping**: Broker ticks are aligned with OHLC candle timestamps in linear time in C++ without incurring Python per-tick overhead.
|
||
2. **First-Tick `OnBar()` Triggering**: When a new candle opens, `OnBar()` is invoked on the *very first tick* of that candle. This guarantees that new market orders execute at the true bar open price, while all subsequent intrabar ticks stream through `OrderBook::on_price_update` to manage trailing stops and pending orders.
|
||
3. **Sub-Second Throughput**: Replays over 600,000 real broker ticks in **~1.1 seconds**.
|
||
|
||
#### Automated Broker Tick Downloader
|
||
Use the automated CLI downloader to fetch genuine broker tick data directly from MetaTrader 5:
|
||
|
||
```powershell
|
||
python backtest_suite/download_broker_ticks.py `
|
||
--symbol "Jump 10 Index" `
|
||
--days 7 `
|
||
--output "backtest_suite/tick_data_cache/jump_10_index_7d_ticks.parquet"
|
||
```
|
||
|
||
#### Running Real-Tick Backtests via CLI
|
||
```powershell
|
||
# Run TrendFinder Leaf using 100% genuine broker ticks
|
||
python strategies/trendfinder_leaf/run_backtest.py `
|
||
--symbol "Jump 10 Index" `
|
||
--data "backtest_suite/historical_data_cache/jump_10_index_100k_m5.parquet" `
|
||
--tick-mode REAL_TICKS `
|
||
--ticks-path "backtest_suite/tick_data_cache/jump_10_index_7d_ticks.parquet"
|
||
```
|
||
|
||
#### Running Real-Tick Backtests in Python API
|
||
```python
|
||
import edge_tester as et
|
||
import pandas as pd
|
||
|
||
# 1. Load historical bars and real broker ticks
|
||
bars_df = pd.read_parquet("backtest_suite/historical_data_cache/jump_10_index_100k_m5.parquet")
|
||
ticks_df = pd.read_parquet("backtest_suite/tick_data_cache/jump_10_index_7d_ticks.parquet")
|
||
|
||
# 2. Configure Tester with REAL_TICKS
|
||
spec = et.SymbolSpec()
|
||
spec.point = 0.01
|
||
spec.spread_val = 0.10
|
||
|
||
cfg = et.TesterConfig()
|
||
cfg.tick_mode = et.TickMode.REAL_TICKS
|
||
|
||
# 3. Execute
|
||
tester = et.Tester(cfg, spec)
|
||
results = tester.run(bars_df, ticks_df)
|
||
|
||
print(f"Trades Executed: {results['total_trades']}")
|
||
print(f"Net Profit: ${results['net_profit']:,.2f}")
|
||
print(f"Profit Factor: {results['profit_factor']:.2f}")
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Execution Latency & Slippage Simulation
|
||
|
||
In real-world live trading, orders do not execute instantaneously. Price quotes fluctuate while an order travels across the network to the broker. EdgeEngine replicates MT5's latency queue.
|
||
|
||
### Latency Configuration
|
||
|
||
```python
|
||
cfg = et.TesterConfig()
|
||
|
||
# 1. Zero Latency (instant execution)
|
||
cfg.latency_mode = et.LatencyMode.NONE
|
||
|
||
# 2. Fixed Latency (e.g. 150ms delay)
|
||
cfg.latency_mode = et.LatencyMode.FIXED
|
||
cfg.latency_fixed_ms = 150
|
||
|
||
# 3. Random Latency (e.g. jitter between 50ms and 250ms)
|
||
cfg.latency_mode = et.LatencyMode.RANDOM
|
||
cfg.latency_min_ms = 50
|
||
cfg.latency_max_ms = 250
|
||
cfg.latency_seed = 42
|
||
```
|
||
|
||
### Mechanics
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
participant Strat as Strategy
|
||
participant Queue as In-Flight Queue
|
||
participant Market as Market Tick Stream
|
||
participant Book as OrderBook
|
||
|
||
Strat->>Queue: Submit Market Order at t0
|
||
Note over Queue: Delay = +150ms<br/>Target Execution: t0 + 150ms
|
||
Market->>Book: Tick t1 (t0 + 50ms)
|
||
Note over Book: t1 < Target Time ➔ Order remains in transit
|
||
Market->>Book: Tick t2 (t0 + 120ms)
|
||
Note over Book: t2 < Target Time ➔ Price moves, order remains in transit
|
||
Market->>Book: Tick t3 (t0 + 160ms)
|
||
Note over Book: t3 ≥ Target Time ➔ DELAY ELAPSED!
|
||
Queue->>Book: Fill Order at Tick t3 Quote
|
||
Note over Book: Order filled with realistic slippage
|
||
```
|
||
|
||
When an order is submitted at time $t$, it is assigned an execution time $t_{\text{target}} = t + \text{delay}$. The order is placed into an `in_flight_orders` queue. As market prices evolve, the order is only matched once a tick arrives with $t_{\text{tick}} \ge t_{\text{target}}$ at that future tick's price, naturally reproducing real market slippage.
|
||
|
||
---
|
||
|
||
## 8. Overnight Financing & Rollover Swap Mechanics (24/7 vs. Traditional Markets)
|
||
|
||
In MetaTrader 5, holding an open position past the end of the trading day incurs an overnight financing cost (or credit) known as **Swap**. `EdgeEngine` strictly replicates MT5's exact daily rollover schedule and financing accounting.
|
||
|
||
### Key Rules of the Swap Engine:
|
||
|
||
1. **Daily Rollover Timing (00:00 Server Time)**:
|
||
- **Naive Mistake**: Many simplistic backtesting scripts mistakenly deduct `swap * volume` on every bar or tick. When testing on M1 or M5 candles, this charges swap hundreds of times per day, completely draining account capital artificially.
|
||
- **MT5 Parity**: In `EdgeEngine`, swaps are applied strictly once per elapsed calendar day (`time / 86400`). Holding a trade for 288 M5 bars within the same day incurs zero swap until midnight rollover.
|
||
|
||
2. **Floating Swap Accumulation vs. Realized Cash Balance**:
|
||
- Accrued swaps do **not** prematurely reduce the cash balance (`account.balance`).
|
||
- Instead, swaps accumulate continuously in the position entity's floating swap field (`pos.swap`), reducing floating equity.
|
||
- The swap charge is only realized into the cash balance when an exit deal occurs (`DEAL_ENTRY_OUT` or Netting position reduction).
|
||
|
||
3. **The 3x Triple Rollover Schedule & Weekend Financing**:
|
||
- **Traditional 5-Day Markets (Forex, Metals, Commodities)**:
|
||
Traditional currency pairs operate on a **T+2 settlement cycle** (trades settle 2 business days after execution). A position held open from Wednesday to Thursday rolls settlement from Friday to Monday. Because commercial banks and interbank clearing systems are closed over the weekend, **Wednesday night carries a 3x swap multiplier** (covering Wednesday, Saturday, and Sunday), while Saturday and Sunday carry 0 swap.
|
||
- Configuration: `spec.swap_rollover3days = 3` (`DAY_WEDNESDAY`).
|
||
- **Continuous 24/7 Synthetic Markets (e.g., Deriv Synthetics)**:
|
||
Synthetic indices (Volatility, Crash, Boom, Jump) run algorithmically 24 hours a day, 7 days a week, 365 days a year without market closures or weekend bank holidays.
|
||
- **Triple swap does NOT apply to Deriv synthetic indices.**
|
||
- Swaps are charged flat at $1\times$ every day (including Saturday and Sunday).
|
||
- Configuration: `spec.swap_rollover3days = -1` (disables 3x rollover).
|
||
- **Cryptocurrency Pairs (BTC, ETH, Altcoins) Across Brokers**:
|
||
- **Native Crypto Exchanges (Binance, Bybit, OKX)**: Trade 24/7/365 with continuous 8-hour funding rates. **There is no 3x swap.**
|
||
- **Deriv Crypto**: Runs 24/7 with a flat $1\times$ daily rollover (`spec.swap_rollover3days = -1`).
|
||
- **Traditional MT5 CFD Brokers (e.g., IC Markets, Pepperstone, Exness, FP Markets)**:
|
||
> [!WARNING]
|
||
> **Crypto CFDs on Retail Brokers May Incur 3x Swap!**
|
||
> Even though cryptocurrency markets run 24/7, many retail MT5 Forex/CFD brokers settle liquidity provider credit lines and fiat currency legs (like USD in BTCUSD) through traditional banking rails that close on weekends.
|
||
> Consequently, many brokers configure their MT5 servers with **Friday 3x swap (`swap_rollover3days = 5`)** or **Wednesday 3x swap (`swap_rollover3days = 3`)** for Crypto CFDs!
|
||
> Always verify your broker's symbol specification in MT5: `Market Watch` $\to$ Right-click symbol $\to$ `Specification` $\to$ `3-days swap`.
|
||
|
||
### Python Symbol Specification Setup
|
||
|
||
```python
|
||
import edge_tester as et
|
||
|
||
spec = et.SymbolSpec()
|
||
spec.swap_mode = 2 # 0 = disabled, 1 = points, 2 = currency/money
|
||
spec.swap_long = -15.0 # -$15.00 per lot per night for long positions
|
||
spec.swap_short = 5.20 # +$5.20 per lot per night for short positions
|
||
|
||
# Rollover schedule:
|
||
# -1 = Continuous 24/7 (Deriv synthetic indices, native crypto - flat 1x every day)
|
||
# 3 = Traditional Forex (3x Wednesday rollover, 0x Saturday/Sunday)
|
||
# 5 = CFD Crypto / Indices (3x Friday rollover on traditional retail brokers)
|
||
spec.swap_rollover3days = -1
|
||
|
||
cfg = et.TesterConfig()
|
||
cfg.enable_swaps = True # Toggle swap accruals during backtests
|
||
```
|
||
|
||
---
|
||
|
||
## 9. MT5 Strategy Tester Parity Engine (Order Lifecycle & Accounting)
|
||
|
||
EdgeEngine has been reverse-engineered and audited directly against MetaTrader 5's execution binaries (`metatester64.exe` / `terminal64.exe`) to guarantee 100% execution parity.
|
||
|
||
### 1. Stop-Limit Orders (`ORDER_TYPE_BUY_STOP_LIMIT` / `ORDER_TYPE_SELL_STOP_LIMIT`)
|
||
Unlike standard stop or limit orders, Stop-Limit orders feature two distinct price targets:
|
||
1. `price_open` (Stop Trigger Price)
|
||
2. `price_stoplimit` (Limit Order Price)
|
||
|
||
When market price touches `price_open`, the order is **not** immediately executed at market. Instead, MT5 transitions the order into an active pending Limit Order (`ORDER_TYPE_BUY_LIMIT` / `ORDER_TYPE_SELL_LIMIT`) placed at `price_stoplimit`. `EdgeEngine` emulates this exact two-stage state transition and fires `TRADE_TRANSACTION_ORDER_UPDATE`.
|
||
|
||
### 2. Stop-Out Order Book Liquidation & Pending Order Purge
|
||
When account equity drops below the broker's Stop-Out threshold (`equity / margin <= stopout_level`):
|
||
1. **Pending Order Purge**: MT5 immediately cancels and deletes all active pending orders (`ORDER_STATE_CANCELED`) to release reserved margin before closing any positions.
|
||
2. **Loss-Ranked Liquidation**: Active positions are sorted by floating loss, and the position with the greatest loss is closed first (`DEAL_REASON_SO`). If margin level remains below the threshold, subsequent positions are liquidated sequentially until account health is restored.
|
||
|
||
### 3. Execution Latency Bypass in Open Prices Mode
|
||
In MT5 Strategy Tester, when testing under `TickMode.OPEN_PRICES`, simulated transit delays are automatically bypassed. All bar-open signals execute at the bar opening quote without entering the in-flight latency queue (`in_flight_orders`). This mirrors MT5's speed optimization behavior.
|
||
|
||
### 4. Netting Mode Accounting (`ACCOUNT_MARGIN_MODE_RETAIL_NETTING`)
|
||
In Netting mode, only one position can exist per symbol:
|
||
- **Adding to Position**: Computes the volume-weighted average open price:
|
||
$$\text{Open Price}_{\text{new}} = \frac{P_{\text{old}} \cdot V_{\text{old}} + P_{\text{fill}} \cdot V_{\text{fill}}}{V_{\text{old}} + V_{\text{fill}}}$$
|
||
- **Partial Reduction**: Realizes gross profit, exit commission, and pro-rata accumulated swap (`pos.swap * (close_vol / pos.volume)`) directly into the cash balance.
|
||
- **Position Reversal**: Fully closes the existing position, realizes all accumulated profit/swap, and opens a new position for the remaining lot volume in the opposite direction (`DEAL_ENTRY_INOUT`).
|
||
|
||
### 5. Intrabar SL/TP Resolution Priority (12-Point Synthetic Wave Model)
|
||
When testing on bars where both the Stop-Loss and Take-Profit boundaries are touched within the bar range, MT5 resolves priority using the candle's price progression:
|
||
- **Bullish Bar (`Close >= Open`)**: Price moves $\text{Open} \to \text{Low} \to \text{High} \to \text{Close}$. For a Buy position, the Low is reached before the High; therefore, **Stop-Loss triggers before Take-Profit**.
|
||
- **Bearish Bar (`Close < Open`)**: Price moves $\text{Open} \to \text{High} \to \text{Low} \to \text{Close}$. For a Buy position, the High is reached before the Low; therefore, **Take-Profit triggers before Stop-Loss**.
|
||
|
||
---
|
||
|
||
## 10. Interactive Step-by-Step & Visual Replay
|
||
|
||
EdgeEngine includes `et.InteractiveTester` which mirrors MT5's Visual Tester. It allows you to pause, step bar-by-bar or tick-by-tick, and inspect floating metrics in real-time.
|
||
|
||
```python
|
||
import edge_tester as et
|
||
|
||
spec = et.SymbolSpec()
|
||
spec.point = 0.01
|
||
spec.spread_val = 0.10
|
||
|
||
cfg = et.TesterConfig()
|
||
cfg.tick_mode = et.TickMode.OPEN_PRICES
|
||
|
||
strategy = MovingAverageCross()
|
||
replay = et.InteractiveTester()
|
||
|
||
# 1. Initialize
|
||
replay.init(
|
||
strategy,
|
||
df["open"].to_numpy(), df["high"].to_numpy(),
|
||
df["low"].to_numpy(), df["close"].to_numpy(),
|
||
df["time"].to_numpy(), df["tick_volume"].to_numpy(),
|
||
df["spread"].to_numpy(), spec, cfg
|
||
)
|
||
|
||
# 2. Step bar-by-bar
|
||
while not replay.is_done():
|
||
replay.step_bar()
|
||
|
||
# Query live state at any point
|
||
current_bar = replay.current_bar()
|
||
floating_equity = replay.account_equity()
|
||
balance = replay.account_balance()
|
||
margin_level = replay.account_margin_level()
|
||
positions = replay.positions()
|
||
|
||
if current_bar % 1000 == 0:
|
||
print(f"Bar {current_bar}: Balance=${balance:,.2f}, Equity=${floating_equity:,.2f}, Positions={len(positions)}")
|
||
|
||
# 3. Finalize and extract complete report
|
||
result = replay.finish()
|
||
print(f"Backtest finished! Net profit: ${result.net_profit:,.2f}")
|
||
```
|
||
|
||
---
|
||
|
||
## 11. Optimization & Hyperopt Guide
|
||
|
||
EdgeEngine includes a dedicated multi-process `StrategyOptimizer` capable of running exhaustive grid sweeps and genetic algorithms across all CPU cores without Python GIL contention.
|
||
|
||
### Optimization Criteria (`OptimizationCriterion`)
|
||
Select which metric the optimizer should maximize:
|
||
- `OptimizationCriterion.CUSTOM`: Maximizes the return value of your strategy's `on_tester()` / `OnTester()` method.
|
||
- `OptimizationCriterion.BALANCE_MAX`: Maximizes ending balance.
|
||
- `OptimizationCriterion.PROFIT_FACTOR`: Maximizes Profit Factor.
|
||
- `OptimizationCriterion.EXPECTED_PAYOFF`: Maximizes average profit per trade.
|
||
- `OptimizationCriterion.DRAWDOWN_MIN`: Minimizes Maximum Drawdown %.
|
||
- `OptimizationCriterion.SHARPE`: Maximizes Sharpe Ratio.
|
||
- `OptimizationCriterion.RECOVERY_FACTOR`: Maximizes Recovery Factor.
|
||
|
||
### Exhaustive Grid Search vs. Genetic Algorithm
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph Grid["Exhaustive Grid Search (Complete Slow)"]
|
||
G1["Define Parameter Matrix"] --> G2["Generate Cartesian Product (All Combos)"]
|
||
G2 --> G3["Distribute Across CPU Workers"]
|
||
G3 --> G4["Evaluate Every Pass"]
|
||
G4 --> G5["Sort Descending by Fitness Criterion"]
|
||
end
|
||
|
||
subgraph Genetic["Genetic Algorithm (Fast Complex Search)"]
|
||
GA1["Initialize Random Population"] --> GA2["Evaluate Population Fitness"]
|
||
GA2 --> GA3["Select Top Elites (Top 20%)"]
|
||
GA3 --> GA4{"Generations Reached?"}
|
||
GA4 -- No --> GA5["Crossover & Mutation"]
|
||
GA5 --> GA2
|
||
GA4 -- Yes --> GA6["Return Pareto Best Parameters"]
|
||
end
|
||
```
|
||
|
||
```python
|
||
from edge_engine import StrategyOptimizer
|
||
import edge_tester as et
|
||
import pandas as pd
|
||
|
||
df = pd.read_parquet("backtest_suite/historical_data_cache/jump_75_index_100k_m5.parquet")
|
||
spec = et.SymbolSpec()
|
||
spec.point = 0.01
|
||
spec.spread_val = 0.10
|
||
|
||
# 1. Configure the Optimizer
|
||
optimizer = StrategyOptimizer(
|
||
strategy_cls=MovingAverageCross,
|
||
symbol_spec=spec,
|
||
criterion=et.OptimizationCriterion.SHARPE,
|
||
algorithm=et.OptimizationAlgorithm.GENETIC, # or EXHAUSTIVE
|
||
max_workers=8, # Distribute across 8 CPU cores
|
||
)
|
||
|
||
# 2. Define Parameter Search Space
|
||
param_grid = {
|
||
"fast_period": [8, 10, 12, 14, 16],
|
||
"slow_period": [20, 24, 28, 32, 36, 40],
|
||
"risk_lot": [0.01, 0.02, 0.05],
|
||
}
|
||
|
||
# 3. Run Optimization
|
||
results = optimizer.optimize(
|
||
df,
|
||
param_grid,
|
||
generations=15, # Genetic algorithm generations
|
||
population_size=20, # Population size per generation
|
||
)
|
||
|
||
# 4. View Top 3 Parameter Sets
|
||
print("Top 3 Configurations:")
|
||
for rank, pass_res in enumerate(results[:3], start=1):
|
||
print(f"#{rank} | Fitness: {pass_res['fitness']:.3f} | Net Profit: ${pass_res['net_profit']:,.2f} | Sharpe: {pass_res['sharpe_ratio']:.2f}")
|
||
print(f" Params: {pass_res['params']}")
|
||
```
|
||
|
||
---
|
||
|
||
## 12. Engine Updates & Porting Guide (Changelog for Clones)
|
||
|
||
If you have cloned or copied `edge_engine` into a separate standalone repository to maintain independently on GitHub, use this exact guide to port the latest MT5 parity updates into your standalone clone:
|
||
|
||
| Target File | Component / Feature | Changes & Code Additions |
|
||
| :--- | :--- | :--- |
|
||
| **`core/include/types.hpp`** | **`SymbolSpec` Specification** | Added `int32_t swap_rollover3days = -1;` to `SymbolSpec`. Default `-1` treats markets as 24/7 continuous (e.g., Deriv synthetics, 24/7 crypto), charging 1x daily swap 7 days/week without artificial 3x penalties. `3` enables traditional Forex Wednesday 3x rollover. |
|
||
| **`core/src/tester_bindings.cpp`** | **Pybind11 Bindings** | Exposed `.def_readwrite("swap_rollover3days", &SymbolSpec::swap_rollover3days)` under the `SymbolSpec` pybind registration block so it is accessible from Python. |
|
||
| **`core/include/order_book.hpp`** | **Open Prices Latency Bypass** | Added `void set_open_prices_mode(bool b)` and `is_open_prices_mode_` boolean. In `TRADE_ACTION_DEAL`, execution delays are only queued if `!is_open_prices_mode_`, matching MT5 Strategy Tester's latency bypass behavior. |
|
||
| **`core/include/order_book.hpp`** | **Swap Accrual Method** | Added `apply_swap(double multiplier = 1.0)` to iterate through `active_positions_` and increment `pos.swap` according to `swap_mode` (points or money/currency). |
|
||
| **`core/include/order_book.hpp`** | **Stop-Limit Orders** | In `_evaluate_pending_orders`, added handling for `ORDER_TYPE_BUY_STOP_LIMIT` and `ORDER_TYPE_SELL_STOP_LIMIT`. When market reaches `price_open`, the order converts to an active limit order at `price_stoplimit`. |
|
||
| **`core/include/order_book.hpp`** | **Day Expiration** | In `_evaluate_pending_orders`, added checking for `ORDER_TIME_DAY`: orders expire if `current_tick.time / 86400 > placed_time / 86400`. |
|
||
| **`core/include/order_book.hpp`** | **Stop-Out Order Purge** | In `_handle_stopout`, active pending orders are now canceled and purged (`ORDER_STATE_CANCELED`) to free up reserved margin before initiating position liquidations. |
|
||
| **`core/include/order_book.hpp`** | **Netting Mode Accounting** | Implemented weighted average open pricing when adding volume to an existing position, and realized PnL, exit commission, and pro-rata swap (`pos.swap * (close_vol / pos.volume)`) on reductions or reversals. |
|
||
| **`core/include/order_book.hpp`** | **12-Point Wave SL/TP Priority** | In `evaluate_bar_extremes`, when both SL and TP are breached within the same bar, priority is assigned by candle color (`close >= open` tests Low/SL first for Buy; `close < open` tests High/TP first for Buy). |
|
||
| **`core/include/interactive_tester.hpp`** | **Daily Rollover Scheduling** | Replaced per-bar swap charging with calendar day rollover logic tracking `last_swap_day_`. Iterates elapsed days and applies `apply_swap(multiplier)` honoring `swap_rollover3days`. Also initializes `book_->set_open_prices_mode(cfg.tick_mode == TickMode::OPEN_PRICES)`. |
|
||
| **`core/include/interactive_tester.hpp`** | **Real-Tick Storage & Lifecycle** | Added `real_ticks_storage_` vector of vectors, enabled `TickMode::REAL_TICKS` validation, and bound `OnBar()` to fire strictly on the first tick of each new bar so new orders execute at candle open while subsequent ticks stream through the order book. |
|
||
| **`core/src/tester_bindings.cpp`** | **Real-Tick Pybind11 Ingestion** | Added `ticks_df` argument to `Tester.run` accepting `tick_time_msc`, `tick_bid`, `tick_ask`, `tick_last`, `tick_volume`, `tick_flags`. Implemented zero-copy $O(N)$ two-pointer bar grouping into `std::vector<std::vector<MqlTick>>` with millisecond-to-second conversion. |
|
||
| **`tests/test_advanced_features.py`** | **Unit Test Verification** | Added `test_latency_discarded_in_open_prices` and `test_swap_rollover_schedule` to ensure full regression test coverage. |
|
||
|
||
---
|
||
|
||
## 13. Test Suite & Verification
|
||
|
||
The engine is validated by a 38-test test suite:
|
||
|
||
```powershell
|
||
python -m pytest edge_engine/tests/ -v
|
||
```
|
||
|
||
### Coverage
|
||
- **Parity Tests (`test_parity.py`)**: Validates exact numerical parity against the Numba/Python implementation across trade counts, win rates, net profit, and all trade management exit methods (A, B, C, D).
|
||
- **Advanced Feature Tests (`test_advanced_features.py`)**: Validates `OHLC_M1` 4-tick generation, transit delay slippage queues, `InteractiveTester` replay methods, and `StrategyOptimizer` Grid/Genetic passes.
|
||
- **Accounting & Margin Integrity (`test_general_tester.py`)**: Tests margin calls, stop-out liquidations, Hedging vs. Netting, swap charges, and deal log integrity.
|