# 03 — Designing Your Own Bar-by-Bar Engine This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the design principles** — you write the code from your own EA's logic. The worked example is a **grid martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does. --- ## 1. What an engine is (and is not) An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told *when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the strategy that produced those. - **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics, and the position-sizing rule. - **It does not know:** your indicators, your regime filters, your broker, why a signal fired. Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04). --- ## 2. The intra-bar problem (the heart of fidelity) Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only four anchor points. **How you order those four points inside a bar decides your fills.** ### The 4-sub-tick model Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption: the price visits the point that hurts an open position **before** the point that helps it. ``` For a LONG position (stop below, target above): OPEN → LOW → HIGH → CLOSE For a SHORT position (stop above, target below): OPEN → HIGH → LOW → CLOSE ``` Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your results honest (an optimistic engine "discovers" edges that don't survive live). At each sub-tick the price is a single number, and you run the same checks: ``` for each bar: for each sub_tick in pessimistic_order(position_direction): price = sub_tick_price # one of O / H / L / C 1. if a position is open: - apply daily swap if the calendar day rolled over - check stop hit (use BID/ASK appropriately; add spread) - check target hit - if both could hit this bar → pessimistic order already decided the stop 2. if flat and an entry signal is active and a next bar exists: - open the position at the next bar's open (avoid look-ahead) 3. sample the equity curve periodically (e.g. once per hour) ``` > **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open, > never the same bar's close. Otherwise you are trading on information you wouldn't have had. --- ## 3. Worked example: a grid martingale engine A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose. ### 3.1 Order types you must model | Type | Fires when | |------|-----------| | Market | immediately at the current ask/bid | | Buy-Stop | ask **rises to** the level (breakout up) | | Buy-Limit | ask **falls to** the level (pullback down) | | (mirror for sell side) | ### 3.2 Series lifecycle (one "basket" from open to close) ``` 1. FLAT. No positions, no pending orders. └─ after a cooldown since the previous series closed … 2. PLACE FIRST PENDING. base_price = close price of the previous series (or current price on first run) pending = Buy-Stop at base + open_distance% # wait for momentum confirmation 3. PENDING FILLS → position #1. first_open = fill price first_lot = sized by the money/lot rule (section 4) count = 1 4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale). next_level_price = last_fill_price − first_open × (grid_step% / 100) next_lot = previous_lot × grid_multiplier (clamped to the volume minimum) add positions up to grid_count at step/mult set #1, then switch to set #2, etc. (cap the total number of levels — deep martingales are where accounts die) 5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5). compute the basket's average price across all open positions once profit clears (break_even + trailing_stop) → move a single basket stop up 6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick. record the series as one closed event (sum of pnl + accumulated swap) base_price = the close price (seed for the next series) reset to FLAT ``` ### 3.3 Adaptive grid levels A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's price, so the spacing stays constant in points while the trigger range tracks slippage: ``` next_level = last_fill_price − first_open_price × (grid_step% / 100) ``` This is one line, but it is the difference between a grid that drifts and one that holds its spacing. --- ## 4. Position sizing — the money/lot modes Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority order: 1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`. 2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`). 3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is deployed regardless of price. After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious pitfalls (doc 05 covers them in full): - A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure the fixed-lot input is zero when you intend money mode. - For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot against MT5 before trusting a whole run. --- ## 5. Basket exits — break-even and trailing For a multi-position basket, the stop is computed on the **average price**, then pushed to every position so they close together: ``` avg = average open price of all open positions, weighted by lot # break-even: once the basket is in profit by `break_even%`, lock the stop at avg if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100: basket_stop = avg # trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100: candidate = bid − avg × trailing% / 100 if candidate > basket_stop: # SMOOTH: move up by any improvement basket_stop = candidate # (STEP variant: only move in trailing-sized jumps) ``` When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes. Single-position strategies are just the `count == 1` case of this. --- ## 6. Spread and swap — small models, large effects - **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread). Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid. - **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap weekday (commonly Wednesday for many CFDs). Two models: - *fixed per lot per day:* `swap = rate × lot × multiplier` - *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier` Accumulate per position and add it to the trade's PnL on close. These look minor but compound over a multi-year, many-position backtest into hundreds of currency units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never hard-code. --- ## 7. Fidelity — how much to trust the Python number Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread, same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are predictable, and knowing them is what makes the lab trustworthy. ### What matches MT5 well - Bar data, per-bar spread, swap (with the triple-swap day). - Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are tick-invariant, so they're identical. - Fill **prices** at pending/grid levels (the level price is the fill price). - Direction, lot sizing, money management. ### What diverges Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs target is touched, exact fill timing on volatile bars. The divergence **scales with path-sensitivity × volatility**: | Strategy character | Typical Python vs MT5 gap (bar-level engine) | |--------------------|-------------------------------------------| | Clean directional, few exits | small and consistent: Python reads somewhat higher | | Tight trailing / break-even in calm years | **NOT small** — see failure mode below | | Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves | #### The break-even / trailing failure mode (measured, not theoretical) A common assumption is that "calm years → small gap" applies to trailing/BE strategies. **It does not.** A break-even + trailing-stop strategy with bar-level simulation can show a **−40% to −50% net-profit gap even in a calm 3-week window**, while trade count matches MT5 exactly. The mechanism: - A bar-level engine updates the BE / trailing SL using the bar's high (or low), then checks the SL on the **same bar's opposite extreme**. If price briefly crossed the BE threshold, the SL is moved to break-even, and the same bar's low (for a long) can trigger that just-moved SL at break-even — booking a **micro-profit** that MT5's tick path would have booked as a small loss (the SL-trigger tick and the BE-trigger tick are separate in MT5, and price can continue past BE to a real loss before the SL fills). - This inflates both the win rate and the gross profit simultaneously. The bias is **always Python optimistic**, and concentrates in the SL/BE exit reason (mean PnL per SL-exit trades reads positive in Python where MT5 reads negative). **Fix: M1 tick-level exit simulation.** Load M1 bars and, inside each M5 (or higher) bar, walk the 5 M1 sub-bars as 4 synthetic ticks each in direction-aware order (see §2). This separates the BE-update tick from the SL-trigger tick onto different M1 bars, restoring the realistic worst case. Measured impact on a break-even scalper: | Mode | Net gap vs MT5 | PF gap vs MT5 | Trade-count gap | |------|----------------|---------------|------------------| | Bar-level (4 sub-ticks) | **−48.5%** | −30.0% | 0% | | M1 tick-level (4 sub-ticks × 5 M1 bars) | **−5.6%** | −7.8% | 0% | Trade count is unaffected by the choice (signals still fire on the higher timeframe); only the exit path fidelity changes. Use M1 tick-level simulation for any strategy that moves its SL during a trade (BE, trailing, basket trailing). ### The practical policy (this is the whole point of the two-tier design) 1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all the optimizer needs. 2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism. 3. **MT5-verify every finalist** — the MT5 number is the one you act on. 4. **Use real ticks where available** (recent history) for the truest check, accepting the limited window. 5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python), and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*, assume MT5 will erase it. --- ## 8. Validating a new engine (do this before any optimization) 1. Pick **one known preset** of your EA and a short period (a few months). 2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report. 3. Run your Python engine on the **same data, same preset**. **If your EA moves its SL during a trade (break-even, trailing, basket trailing), you MUST pass M1 bars and run tick-level exit simulation (§7) — the bar-level engine is not trustworthy for that class of EA.** 4. Reconcile **trade by trade**, then in aggregate. Target gates depend on the EA class and engine mode: | EA class / engine mode | Net gap | PF gap | Trade-count gap | Equity-DD gap | |------------------------|---------|--------|------------------|----------------| | Clean-directional, bar-level | ≤ ~2% | essentially identical | ≤ ~5% | ≤ ~3% | | BE / trailing, **bar-level** | **unattainable** — see §7 failure mode (~−40% to −50% net gap) | | BE / trailing, **M1 tick-level** | ≤ ~10% | ≤ ~10% | ≤ ~5% | ≤ ~10% | The bar-level gate (≤ ~2%) applies only to setups that don't move the SL intra-trade. For BE/trailing EAs the tick-level gate is wider (≤ ~10%) because residual spread/tick-path differences remain — accept it and **MT5-verify every finalist** rather than chase sub-2% on a tick-sensitive EA. 5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the engine's **baseline fidelity document** (engine mode + measured gap + the window used) and freeze the engine (doc 04). > If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe > mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, sub-tick > ordering, **or running a BE/trailing EA on the bar-level engine (use M1 tick-level instead)**. Walk > those six before suspecting anything exotic. Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine validated.