mymt5opp/03-engine-design.md

# 03 — Designing Your Own Bar-by-Bar Engine

This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully
enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the
design principles** — you write the code from your own EA's logic. The worked example is a **grid
martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does.

---

## 1. What an engine is (and is not)

An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each
bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told
*when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the
strategy that produced those.

- **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics,
  and the position-sizing rule.
- **It does not know:** your indicators, your regime filters, your broker, why a signal fired.

Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04).

---

## 2. The intra-bar problem (the heart of fidelity)

Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute
the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated
tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only
four anchor points. **How you order those four points inside a bar decides your fills.**

### The 4-sub-tick model

Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption:
the price visits the point that hurts an open position **before** the point that helps it.

```
For a LONG position (stop below, target above):  OPEN → LOW  → HIGH → CLOSE
For a SHORT position (stop above, target below): OPEN → HIGH → LOW  → CLOSE
```

Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order
makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before
HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your
results honest (an optimistic engine "discovers" edges that don't survive live).

At each sub-tick the price is a single number, and you run the same checks:

```
for each bar:
    for each sub_tick in pessimistic_order(position_direction):
        price = sub_tick_price            # one of O / H / L / C
        1. if a position is open:
             - apply daily swap if the calendar day rolled over
             - check stop hit   (use BID/ASK appropriately; add spread)
             - check target hit
             - if both could hit this bar → pessimistic order already decided the stop
        2. if flat and an entry signal is active and a next bar exists:
             - open the position at the next bar's open (avoid look-ahead)
        3. sample the equity curve periodically (e.g. once per hour)
```

> **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open,
> never the same bar's close. Otherwise you are trading on information you wouldn't have had.

---

## 3. Worked example: a grid martingale engine

A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple
simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you
can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose.

### 3.1 Order types you must model

| Type | Fires when |
|------|-----------|
| Market | immediately at the current ask/bid |
| Buy-Stop | ask **rises to** the level (breakout up) |
| Buy-Limit | ask **falls to** the level (pullback down) |
| (mirror for sell side) |

### 3.2 Series lifecycle (one "basket" from open to close)

```
1. FLAT. No positions, no pending orders.
   └─ after a cooldown since the previous series closed …

2. PLACE FIRST PENDING.
   base_price = close price of the previous series (or current price on first run)
   pending = Buy-Stop at base + open_distance%        # wait for momentum confirmation

3. PENDING FILLS → position #1.
   first_open = fill price
   first_lot  = sized by the money/lot rule (section 4)
   count = 1

4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale).
   next_level_price = last_fill_price − first_open × (grid_step% / 100)
   next_lot         = previous_lot × grid_multiplier      (clamped to the volume minimum)
   add positions up to grid_count at step/mult set #1, then switch to set #2, etc.
   (cap the total number of levels — deep martingales are where accounts die)

5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5).
   compute the basket's average price across all open positions
   once profit clears (break_even + trailing_stop) → move a single basket stop up

6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick.
   record the series as one closed event (sum of pnl + accumulated swap)
   base_price = the close price (seed for the next series)
   reset to FLAT
```

### 3.3 Adaptive grid levels

A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's
price, so the spacing stays constant in points while the trigger range tracks slippage:

```
next_level = last_fill_price − first_open_price × (grid_step% / 100)
```

This is one line, but it is the difference between a grid that drifts and one that holds its spacing.

---

## 4. Position sizing — the money/lot modes

Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority
order:

1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`.
2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`).
3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is
   deployed regardless of price.

After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious
pitfalls (doc 05 covers them in full):

- A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure
  the fixed-lot input is zero when you intend money mode.
- For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the
  EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot
  against MT5 before trusting a whole run.

---

## 5. Basket exits — break-even and trailing

For a multi-position basket, the stop is computed on the **average price**, then pushed to every
position so they close together:

```
avg = average open price of all open positions, weighted by lot

# break-even: once the basket is in profit by `break_even%`, lock the stop at avg
if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100:
    basket_stop = avg

# trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price
if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100:
    candidate = bid − avg × trailing% / 100
    if candidate > basket_stop:            # SMOOTH: move up by any improvement
        basket_stop = candidate            # (STEP variant: only move in trailing-sized jumps)
```

When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes.
Single-position strategies are just the `count == 1` case of this.

---

## 6. Spread and swap — small models, large effects

- **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread).
  Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where
  recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid.
- **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap
  weekday (commonly Wednesday for many CFDs). Two models:
  - *fixed per lot per day:* `swap = rate × lot × multiplier`
  - *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier`
  Accumulate per position and add it to the trade's PnL on close.

These look minor but compound over a multi-year, many-position backtest into hundreds of currency
units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never
hard-code.

---

## 7. Fidelity — how much to trust the Python number

Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread,
same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are
predictable, and knowing them is what makes the lab trustworthy.

### What matches MT5 well
- Bar data, per-bar spread, swap (with the triple-swap day).
- Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are
  tick-invariant, so they're identical.
- Fill **prices** at pending/grid levels (the level price is the fill price).
- Direction, lot sizing, money management.

### What diverges
Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs
target is touched, exact fill timing on volatile bars. The divergence **scales with
path-sensitivity × volatility**:

| Strategy character | Typical Python vs MT5 gap (bar-level engine) |
|--------------------|-------------------------------------------|
| Clean directional, few exits | small and consistent: Python reads somewhat higher |
| Tight trailing / break-even in calm years | **NOT small** — see failure mode below |
| Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves |

#### The break-even / trailing failure mode (measured, not theoretical)

A common assumption is that "calm years → small gap" applies to trailing/BE strategies. **It does
not.** A break-even + trailing-stop strategy with bar-level simulation can show a **−40% to −50%
net-profit gap even in a calm 3-week window**, while trade count matches MT5 exactly. The mechanism:

- A bar-level engine updates the BE / trailing SL using the bar's high (or low), then checks the SL
  on the **same bar's opposite extreme**. If price briefly crossed the BE threshold, the SL is moved
  to break-even, and the same bar's low (for a long) can trigger that just-moved SL at break-even —
  booking a **micro-profit** that MT5's tick path would have booked as a small loss (the SL-trigger
  tick and the BE-trigger tick are separate in MT5, and price can continue past BE to a real loss
  before the SL fills).
- This inflates both the win rate and the gross profit simultaneously. The bias is **always Python
  optimistic**, and concentrates in the SL/BE exit reason (mean PnL per SL-exit trades reads
  positive in Python where MT5 reads negative).

**Fix: M1 tick-level exit simulation.** Load M1 bars and, inside each M5 (or higher) bar, walk the
5 M1 sub-bars as 4 synthetic ticks each in direction-aware order (see §2). This separates the
BE-update tick from the SL-trigger tick onto different M1 bars, restoring the realistic worst case.
Measured impact on a break-even scalper:

| Mode | Net gap vs MT5 | PF gap vs MT5 | Trade-count gap |
|------|----------------|---------------|------------------|
| Bar-level (4 sub-ticks) | **−48.5%** | −30.0% | 0% |
| M1 tick-level (4 sub-ticks × 5 M1 bars) | **−5.6%** | −7.8% | 0% |

Trade count is unaffected by the choice (signals still fire on the higher timeframe); only the exit
path fidelity changes. Use M1 tick-level simulation for any strategy that moves its SL during a
trade (BE, trailing, basket trailing).

### The practical policy (this is the whole point of the two-tier design)
1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all
   the optimizer needs.
2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per
   bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism.
3. **MT5-verify every finalist** — the MT5 number is the one you act on.
4. **Use real ticks where available** (recent history) for the truest check, accepting the limited
   window.
5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python),
   and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a
   known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*,
   assume MT5 will erase it.

---

## 8. Validating a new engine (do this before any optimization)

1. Pick **one known preset** of your EA and a short period (a few months).
2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report.
3. Run your Python engine on the **same data, same preset**. **If your EA moves its SL during a trade
   (break-even, trailing, basket trailing), you MUST pass M1 bars and run tick-level exit simulation
   (§7) — the bar-level engine is not trustworthy for that class of EA.**
4. Reconcile **trade by trade**, then in aggregate. Target gates depend on the EA class and engine mode:

   | EA class / engine mode | Net gap | PF gap | Trade-count gap | Equity-DD gap |
   |------------------------|---------|--------|------------------|----------------|
   | Clean-directional, bar-level | ≤ ~2% | essentially identical | ≤ ~5% | ≤ ~3% |
   | BE / trailing, **bar-level** | **unattainable** — see §7 failure mode (~−40% to −50% net gap) |
   | BE / trailing, **M1 tick-level** | ≤ ~10% | ≤ ~10% | ≤ ~5% | ≤ ~10% |

   The bar-level gate (≤ ~2%) applies only to setups that don't move the SL intra-trade. For BE/trailing
   EAs the tick-level gate is wider (≤ ~10%) because residual spread/tick-path differences remain —
   accept it and **MT5-verify every finalist** rather than chase sub-2% on a tick-sensitive EA.

5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the
   engine's **baseline fidelity document** (engine mode + measured gap + the window used) and freeze the
   engine (doc 04).

> If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe
> mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, sub-tick
> ordering, **or running a BE/trailing EA on the bar-level engine (use M1 tick-level instead)**. Walk
> those six before suspecting anything exotic.

Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine
validated.