288 lines
15 KiB
Markdown
288 lines
15 KiB
Markdown
# 03 — Designing Your Own Bar-by-Bar Engine
|
||
|
||
This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully
|
||
enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the
|
||
design principles** — you write the code from your own EA's logic. The worked example is a **grid
|
||
martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does.
|
||
|
||
---
|
||
|
||
## 1. What an engine is (and is not)
|
||
|
||
An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each
|
||
bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told
|
||
*when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the
|
||
strategy that produced those.
|
||
|
||
- **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics,
|
||
and the position-sizing rule.
|
||
- **It does not know:** your indicators, your regime filters, your broker, why a signal fired.
|
||
|
||
Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04).
|
||
|
||
---
|
||
|
||
## 2. The intra-bar problem (the heart of fidelity)
|
||
|
||
Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute
|
||
the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated
|
||
tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only
|
||
four anchor points. **How you order those four points inside a bar decides your fills.**
|
||
|
||
### The 4-sub-tick model
|
||
|
||
Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption:
|
||
the price visits the point that hurts an open position **before** the point that helps it.
|
||
|
||
```
|
||
For a LONG position (stop below, target above): OPEN → LOW → HIGH → CLOSE
|
||
For a SHORT position (stop above, target below): OPEN → HIGH → LOW → CLOSE
|
||
```
|
||
|
||
Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order
|
||
makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before
|
||
HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your
|
||
results honest (an optimistic engine "discovers" edges that don't survive live).
|
||
|
||
At each sub-tick the price is a single number, and you run the same checks:
|
||
|
||
```
|
||
for each bar:
|
||
for each sub_tick in pessimistic_order(position_direction):
|
||
price = sub_tick_price # one of O / H / L / C
|
||
1. if a position is open:
|
||
- apply daily swap if the calendar day rolled over
|
||
- check stop hit (use BID/ASK appropriately; add spread)
|
||
- check target hit
|
||
- if both could hit this bar → pessimistic order already decided the stop
|
||
2. if flat and an entry signal is active and a next bar exists:
|
||
- open the position at the next bar's open (avoid look-ahead)
|
||
3. sample the equity curve periodically (e.g. once per hour)
|
||
```
|
||
|
||
> **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open,
|
||
> never the same bar's close. Otherwise you are trading on information you wouldn't have had.
|
||
|
||
---
|
||
|
||
## 3. Worked example: a grid martingale engine
|
||
|
||
A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple
|
||
simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you
|
||
can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose.
|
||
|
||
### 3.1 Order types you must model
|
||
|
||
| Type | Fires when |
|
||
|------|-----------|
|
||
| Market | immediately at the current ask/bid |
|
||
| Buy-Stop | ask **rises to** the level (breakout up) |
|
||
| Buy-Limit | ask **falls to** the level (pullback down) |
|
||
| (mirror for sell side) |
|
||
|
||
### 3.2 Series lifecycle (one "basket" from open to close)
|
||
|
||
```
|
||
1. FLAT. No positions, no pending orders.
|
||
└─ after a cooldown since the previous series closed …
|
||
|
||
2. PLACE FIRST PENDING.
|
||
base_price = close price of the previous series (or current price on first run)
|
||
pending = Buy-Stop at base + open_distance% # wait for momentum confirmation
|
||
|
||
3. PENDING FILLS → position #1.
|
||
first_open = fill price
|
||
first_lot = sized by the money/lot rule (section 4)
|
||
count = 1
|
||
|
||
4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale).
|
||
next_level_price = last_fill_price − first_open × (grid_step% / 100)
|
||
next_lot = previous_lot × grid_multiplier (clamped to the volume minimum)
|
||
add positions up to grid_count at step/mult set #1, then switch to set #2, etc.
|
||
(cap the total number of levels — deep martingales are where accounts die)
|
||
|
||
5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5).
|
||
compute the basket's average price across all open positions
|
||
once profit clears (break_even + trailing_stop) → move a single basket stop up
|
||
|
||
6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick.
|
||
record the series as one closed event (sum of pnl + accumulated swap)
|
||
base_price = the close price (seed for the next series)
|
||
reset to FLAT
|
||
```
|
||
|
||
### 3.3 Adaptive grid levels
|
||
|
||
A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's
|
||
price, so the spacing stays constant in points while the trigger range tracks slippage:
|
||
|
||
```
|
||
next_level = last_fill_price − first_open_price × (grid_step% / 100)
|
||
```
|
||
|
||
This is one line, but it is the difference between a grid that drifts and one that holds its spacing.
|
||
|
||
---
|
||
|
||
## 4. Position sizing — the money/lot modes
|
||
|
||
Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority
|
||
order:
|
||
|
||
1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`.
|
||
2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`).
|
||
3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is
|
||
deployed regardless of price.
|
||
|
||
After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious
|
||
pitfalls (doc 05 covers them in full):
|
||
|
||
- A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure
|
||
the fixed-lot input is zero when you intend money mode.
|
||
- For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the
|
||
EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot
|
||
against MT5 before trusting a whole run.
|
||
|
||
---
|
||
|
||
## 5. Basket exits — break-even and trailing
|
||
|
||
For a multi-position basket, the stop is computed on the **average price**, then pushed to every
|
||
position so they close together:
|
||
|
||
```
|
||
avg = average open price of all open positions, weighted by lot
|
||
|
||
# break-even: once the basket is in profit by `break_even%`, lock the stop at avg
|
||
if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100:
|
||
basket_stop = avg
|
||
|
||
# trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price
|
||
if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100:
|
||
candidate = bid − avg × trailing% / 100
|
||
if candidate > basket_stop: # SMOOTH: move up by any improvement
|
||
basket_stop = candidate # (STEP variant: only move in trailing-sized jumps)
|
||
```
|
||
|
||
When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes.
|
||
Single-position strategies are just the `count == 1` case of this.
|
||
|
||
---
|
||
|
||
## 6. Spread and swap — small models, large effects
|
||
|
||
- **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread).
|
||
Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where
|
||
recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid.
|
||
- **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap
|
||
weekday (commonly Wednesday for many CFDs). Two models:
|
||
- *fixed per lot per day:* `swap = rate × lot × multiplier`
|
||
- *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier`
|
||
Accumulate per position and add it to the trade's PnL on close.
|
||
|
||
These look minor but compound over a multi-year, many-position backtest into hundreds of currency
|
||
units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never
|
||
hard-code.
|
||
|
||
---
|
||
|
||
## 7. Fidelity — how much to trust the Python number
|
||
|
||
Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread,
|
||
same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are
|
||
predictable, and knowing them is what makes the lab trustworthy.
|
||
|
||
### What matches MT5 well
|
||
- Bar data, per-bar spread, swap (with the triple-swap day).
|
||
- Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are
|
||
tick-invariant, so they're identical.
|
||
- Fill **prices** at pending/grid levels (the level price is the fill price).
|
||
- Direction, lot sizing, money management.
|
||
|
||
### What diverges
|
||
Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs
|
||
target is touched, exact fill timing on volatile bars. The divergence **scales with
|
||
path-sensitivity × volatility**:
|
||
|
||
| Strategy character | Typical Python vs MT5 gap (bar-level engine) |
|
||
|--------------------|-------------------------------------------|
|
||
| Clean directional, few exits | small and consistent: Python reads somewhat higher |
|
||
| Tight trailing / break-even in calm years | **NOT small** — see failure mode below |
|
||
| Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves |
|
||
|
||
#### The break-even / trailing failure mode (measured, not theoretical)
|
||
|
||
A common assumption is that "calm years → small gap" applies to trailing/BE strategies. **It does
|
||
not.** A break-even + trailing-stop strategy with bar-level simulation can show a **−40% to −50%
|
||
net-profit gap even in a calm 3-week window**, while trade count matches MT5 exactly. The mechanism:
|
||
|
||
- A bar-level engine updates the BE / trailing SL using the bar's high (or low), then checks the SL
|
||
on the **same bar's opposite extreme**. If price briefly crossed the BE threshold, the SL is moved
|
||
to break-even, and the same bar's low (for a long) can trigger that just-moved SL at break-even —
|
||
booking a **micro-profit** that MT5's tick path would have booked as a small loss (the SL-trigger
|
||
tick and the BE-trigger tick are separate in MT5, and price can continue past BE to a real loss
|
||
before the SL fills).
|
||
- This inflates both the win rate and the gross profit simultaneously. The bias is **always Python
|
||
optimistic**, and concentrates in the SL/BE exit reason (mean PnL per SL-exit trades reads
|
||
positive in Python where MT5 reads negative).
|
||
|
||
**Fix: M1 tick-level exit simulation.** Load M1 bars and, inside each M5 (or higher) bar, walk the
|
||
5 M1 sub-bars as 4 synthetic ticks each in direction-aware order (see §2). This separates the
|
||
BE-update tick from the SL-trigger tick onto different M1 bars, restoring the realistic worst case.
|
||
Measured impact on a break-even scalper:
|
||
|
||
| Mode | Net gap vs MT5 | PF gap vs MT5 | Trade-count gap |
|
||
|------|----------------|---------------|------------------|
|
||
| Bar-level (4 sub-ticks) | **−48.5%** | −30.0% | 0% |
|
||
| M1 tick-level (4 sub-ticks × 5 M1 bars) | **−5.6%** | −7.8% | 0% |
|
||
|
||
Trade count is unaffected by the choice (signals still fire on the higher timeframe); only the exit
|
||
path fidelity changes. Use M1 tick-level simulation for any strategy that moves its SL during a
|
||
trade (BE, trailing, basket trailing).
|
||
|
||
### The practical policy (this is the whole point of the two-tier design)
|
||
1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all
|
||
the optimizer needs.
|
||
2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per
|
||
bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism.
|
||
3. **MT5-verify every finalist** — the MT5 number is the one you act on.
|
||
4. **Use real ticks where available** (recent history) for the truest check, accepting the limited
|
||
window.
|
||
5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python),
|
||
and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a
|
||
known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*,
|
||
assume MT5 will erase it.
|
||
|
||
---
|
||
|
||
## 8. Validating a new engine (do this before any optimization)
|
||
|
||
1. Pick **one known preset** of your EA and a short period (a few months).
|
||
2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report.
|
||
3. Run your Python engine on the **same data, same preset**. **If your EA moves its SL during a trade
|
||
(break-even, trailing, basket trailing), you MUST pass M1 bars and run tick-level exit simulation
|
||
(§7) — the bar-level engine is not trustworthy for that class of EA.**
|
||
4. Reconcile **trade by trade**, then in aggregate. Target gates depend on the EA class and engine mode:
|
||
|
||
| EA class / engine mode | Net gap | PF gap | Trade-count gap | Equity-DD gap |
|
||
|------------------------|---------|--------|------------------|----------------|
|
||
| Clean-directional, bar-level | ≤ ~2% | essentially identical | ≤ ~5% | ≤ ~3% |
|
||
| BE / trailing, **bar-level** | **unattainable** — see §7 failure mode (~−40% to −50% net gap) |
|
||
| BE / trailing, **M1 tick-level** | ≤ ~10% | ≤ ~10% | ≤ ~5% | ≤ ~10% |
|
||
|
||
The bar-level gate (≤ ~2%) applies only to setups that don't move the SL intra-trade. For BE/trailing
|
||
EAs the tick-level gate is wider (≤ ~10%) because residual spread/tick-path differences remain —
|
||
accept it and **MT5-verify every finalist** rather than chase sub-2% on a tick-sensitive EA.
|
||
|
||
5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the
|
||
engine's **baseline fidelity document** (engine mode + measured gap + the window used) and freeze the
|
||
engine (doc 04).
|
||
|
||
> If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe
|
||
> mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, sub-tick
|
||
> ordering, **or running a BE/trailing EA on the bar-level engine (use M1 tick-level instead)**. Walk
|
||
> those six before suspecting anything exotic.
|
||
|
||
Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine
|
||
validated.
|