mymt5opp/03-engine-design.md

03 — Designing Your Own Bar-by-Bar Engine

This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully enough to rank ideas, fast enough to run thousands of times. This doc explains the algorithm and the design principles — you write the code from your own EA's logic. The worked example is a grid martingale engine because it exercises every hard case; adapt the principles to whatever your EA does.

---

1. What an engine is (and is not)

An engine is a generic bar-by-bar simulator. It iterates historical bars in order and, at each bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told when to enter (signal arrays) and where the stops are (price arrays). It does not know the strategy that produced those.

• It knows: bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics, and the position-sizing rule.
• It does not know: your indicators, your regime filters, your broker, why a signal fired.

Keeping that boundary is what makes the engine reusable across strategies and freezable (doc 04).

---

2. The intra-bar problem (the heart of fidelity)

Your data is bars: each M1 bar is four numbers — open, high, low, close. But within that minute the price walked a path you can't see. MT5's tester, in its finer models, simulates an interpolated tick path inside each bar, so an EA's OnTick fires many times per bar. A bar-based engine has only four anchor points. How you order those four points inside a bar decides your fills.

The 4-sub-tick model

Process each bar as four sub-ticks in a fixed order. The order encodes a pessimistic assumption: the price visits the point that hurts an open position before the point that helps it.

For a LONG position (stop below, target above):  OPEN → LOW  → HIGH → CLOSE
For a SHORT position (stop above, target below): OPEN → HIGH → LOW  → CLOSE

Why: if in the same bar price could have hit both the stop and the target, the pessimistic order makes the stop win — the realistic worst case. For a long, LOW (the stop side) is visited before HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your results honest (an optimistic engine "discovers" edges that don't survive live).

At each sub-tick the price is a single number, and you run the same checks:

for each bar:
    for each sub_tick in pessimistic_order(position_direction):
        price = sub_tick_price            # one of O / H / L / C
        1. if a position is open:
             - apply daily swap if the calendar day rolled over
             - check stop hit   (use BID/ASK appropriately; add spread)
             - check target hit
             - if both could hit this bar → pessimistic order already decided the stop
        2. if flat and an entry signal is active and a next bar exists:
             - open the position at the next bar's open (avoid look-ahead)
        3. sample the equity curve periodically (e.g. once per hour)

Look-ahead guard: a signal computed from a bar's close must execute on the next bar's open, never the same bar's close. Otherwise you are trading on information you wouldn't have had.

---

3. Worked example: a grid martingale engine

A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose.

3.1 Order types you must model

Type	Fires when
Market	immediately at the current ask/bid
Buy-Stop	ask rises to the level (breakout up)
Buy-Limit	ask falls to the level (pullback down)
(mirror for sell side)

3.2 Series lifecycle (one "basket" from open to close)

1. FLAT. No positions, no pending orders.
   └─ after a cooldown since the previous series closed …

2. PLACE FIRST PENDING.
   base_price = close price of the previous series (or current price on first run)
   pending = Buy-Stop at base + open_distance%        # wait for momentum confirmation

3. PENDING FILLS → position #1.
   first_open = fill price
   first_lot  = sized by the money/lot rule (section 4)
   count = 1

4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale).
   next_level_price = last_fill_price − first_open × (grid_step% / 100)
   next_lot         = previous_lot × grid_multiplier      (clamped to the volume minimum)
   add positions up to grid_count at step/mult set #1, then switch to set #2, etc.
   (cap the total number of levels — deep martingales are where accounts die)

5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5).
   compute the basket's average price across all open positions
   once profit clears (break_even + trailing_stop) → move a single basket stop up

6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick.
   record the series as one closed event (sum of pnl + accumulated swap)
   base_price = the close price (seed for the next series)
   reset to FLAT

3.3 Adaptive grid levels

A robust grid spaces levels from the most recent fill but scales the step by the first fill's price, so the spacing stays constant in points while the trigger range tracks slippage:

next_level = last_fill_price − first_open_price × (grid_step% / 100)

This is one line, but it is the difference between a grid that drifts and one that holds its spacing.

---

4. Position sizing — the money/lot modes

Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority order:

1. Risk-on-stop: lot = max_loss_money / (stop_distance_points × tick_value).
2. Fixed lot: lot = configured_lot (optionally scaled by balance / reference_balance).
3. Money mode: lot = amount / open_price / contract_size — size so a fixed cash amount is deployed regardless of price.

After computing, round to the broker's volume step and clamp to the volume minimum. Two notorious pitfalls (doc 05 covers them in full):

• A "money mode" EA usually ignores the cash amount if a non-zero fixed lot is also set. Make sure the fixed-lot input is zero when you intend money mode.
• For a quote currency ≠ account currency pair (e.g. a JPY-quoted pair on a USD account), the EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot against MT5 before trusting a whole run.

---

5. Basket exits — break-even and trailing

For a multi-position basket, the stop is computed on the average price, then pushed to every position so they close together:

avg = average open price of all open positions, weighted by lot

# break-even: once the basket is in profit by `break_even%`, lock the stop at avg
if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100:
    basket_stop = avg

# trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price
if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100:
    candidate = bid − avg × trailing% / 100
    if candidate > basket_stop:            # SMOOTH: move up by any improvement
        basket_stop = candidate            # (STEP variant: only move in trailing-sized jumps)

When the bid touches basket_stop, every position's stop fires in that sub-tick and the series closes. Single-position strategies are just the count == 1 case of this.

---

6. Spread and swap — small models, large effects

• Spread. Prefer a per-bar spread column stored in your data (the real historical spread). Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid.
• Swap. Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap weekday (commonly Wednesday for many CFDs). Two models:
  • fixed per lot per day: swap = rate × lot × multiplier
  • annual % on notional: swap = price × annual_pct / 365 × lot × multiplier Accumulate per position and add it to the trade's PnL on close.

These look minor but compound over a multi-year, many-position backtest into hundreds of currency units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never hard-code.

---

7. Fidelity — how much to trust the Python number

Your engine is a fast 4-point approximation of MT5's finer intra-bar path. Same data, same spread, same swap, same math — the only structural difference is intra-bar granularity. The consequences are predictable, and knowing them is what makes the lab trustworthy.

What matches MT5 well

• Bar data, per-bar spread, swap (with the triple-swap day).
• Indicator gates computed from closed higher-timeframe bars (e.g. a daily RSI) — these are tick-invariant, so they're identical.
• Fill prices at pending/grid levels (the level price is the fill price).
• Direction, lot sizing, money management.

What diverges

Anything that depends on the path inside a bar: trailing-stop triggers, the order a stop vs target is touched, exact fill timing on volatile bars. The divergence scales with path-sensitivity × volatility:

Strategy character	Typical Python vs MT5 gap (bar-level engine)
Clean directional, few exits	small and consistent: Python reads somewhat higher
Tight trailing / break-even in calm years	NOT small — see failure mode below
Tight trailing / martingale grid in crash years	large — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves

The break-even / trailing failure mode (measured, not theoretical)

A common assumption is that "calm years → small gap" applies to trailing/BE strategies. It does not. A break-even + trailing-stop strategy with bar-level simulation can show a −40% to −50% net-profit gap even in a calm 3-week window, while trade count matches MT5 exactly. The mechanism:

• A bar-level engine updates the BE / trailing SL using the bar's high (or low), then checks the SL on the same bar's opposite extreme. If price briefly crossed the BE threshold, the SL is moved to break-even, and the same bar's low (for a long) can trigger that just-moved SL at break-even — booking a micro-profit that MT5's tick path would have booked as a small loss (the SL-trigger tick and the BE-trigger tick are separate in MT5, and price can continue past BE to a real loss before the SL fills).
• This inflates both the win rate and the gross profit simultaneously. The bias is always Python optimistic, and concentrates in the SL/BE exit reason (mean PnL per SL-exit trades reads positive in Python where MT5 reads negative).

Fix: M1 tick-level exit simulation. Load M1 bars and, inside each M5 (or higher) bar, walk the 5 M1 sub-bars as 4 synthetic ticks each in direction-aware order (see §2). This separates the BE-update tick from the SL-trigger tick onto different M1 bars, restoring the realistic worst case. Measured impact on a break-even scalper:

Mode	Net gap vs MT5	PF gap vs MT5	Trade-count gap
Bar-level (4 sub-ticks)	−48.5%	−30.0%	0%
M1 tick-level (4 sub-ticks × 5 M1 bars)	−5.6%	−7.8%	0%

Trade count is unaffected by the choice (signals still fire on the higher timeframe); only the exit path fidelity changes. Use M1 tick-level simulation for any strategy that moves its SL during a trade (BE, trailing, basket trailing).

The practical policy (this is the whole point of the two-tier design)

1. Use Python for fast ranking and A/B — the relative order of setups is preserved, which is all the optimizer needs.
2. Apply a pessimistic convention (the sub-tick order, plus an optional "one adverse re-touch per bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism.
3. MT5-verify every finalist — the MT5 number is the one you act on.
4. Use real ticks where available (recent history) for the truest check, accepting the limited window.
5. Expect only a modest negative gap for clean-directional setups (MT5 a little below Python), and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is thin, assume MT5 will erase it.

---

8. Validating a new engine (do this before any optimization)

1. Pick one known preset of your EA and a short period (a few months).

2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report.

3. Run your Python engine on the same data, same preset. If your EA moves its SL during a trade (break-even, trailing, basket trailing), you MUST pass M1 bars and run tick-level exit simulation (§7) — the bar-level engine is not trustworthy for that class of EA.

4. Reconcile trade by trade, then in aggregate. Target gates depend on the EA class and engine mode:

   EA class / engine mode	Net gap	PF gap	Trade-count gap	Equity-DD gap
   Clean-directional, bar-level	≤ ~2%	essentially identical	≤ ~5%	≤ ~3%
   BE / trailing, bar-level	unattainable — see §7 failure mode (~−40% to −50% net gap)
   BE / trailing, M1 tick-level	≤ ~10%	≤ ~10%	≤ ~5%	≤ ~10%

   The bar-level gate (≤ ~2%) applies only to setups that don't move the SL intra-trade. For BE/trailing EAs the tick-level gate is wider (≤ ~10%) because residual spread/tick-path differences remain — accept it and MT5-verify every finalist rather than chase sub-2% on a tick-sensitive EA.

5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the engine's baseline fidelity document (engine mode + measured gap + the window used) and freeze the engine (doc 04).

If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, sub-tick ordering, or running a BE/trailing EA on the bar-level engine (use M1 tick-level instead). Walk those six before suspecting anything exotic.

Next: 04-isolation-rules.md — the discipline that keeps a validated engine validated.