# 03 — Designing Your Own Bar-by-Bar Engine This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the design principles** — you write the code from your own EA's logic. The worked example is a **grid martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does. --- ## 1. What an engine is (and is not) An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told *when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the strategy that produced those. - **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics, and the position-sizing rule. - **It does not know:** your indicators, your regime filters, your broker, why a signal fired. Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04). --- ## 2. The intra-bar problem (the heart of fidelity) Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only four anchor points. **How you order those four points inside a bar decides your fills.** ### The 4-sub-tick model Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption: the price visits the point that hurts an open position **before** the point that helps it. ``` For a LONG position (stop below, target above): OPEN → LOW → HIGH → CLOSE For a SHORT position (stop above, target below): OPEN → HIGH → LOW → CLOSE ``` Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your results honest (an optimistic engine "discovers" edges that don't survive live). At each sub-tick the price is a single number, and you run the same checks: ``` for each bar: for each sub_tick in pessimistic_order(position_direction): price = sub_tick_price # one of O / H / L / C 1. if a position is open: - apply daily swap if the calendar day rolled over - check stop hit (use BID/ASK appropriately; add spread) - check target hit - if both could hit this bar → pessimistic order already decided the stop 2. if flat and an entry signal is active and a next bar exists: - open the position at the next bar's open (avoid look-ahead) 3. sample the equity curve periodically (e.g. once per hour) ``` > **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open, > never the same bar's close. Otherwise you are trading on information you wouldn't have had. --- ## 3. Worked example: a grid martingale engine A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose. ### 3.1 Order types you must model | Type | Fires when | |------|-----------| | Market | immediately at the current ask/bid | | Buy-Stop | ask **rises to** the level (breakout up) | | Buy-Limit | ask **falls to** the level (pullback down) | | (mirror for sell side) | ### 3.2 Series lifecycle (one "basket" from open to close) ``` 1. FLAT. No positions, no pending orders. └─ after a cooldown since the previous series closed … 2. PLACE FIRST PENDING. base_price = close price of the previous series (or current price on first run) pending = Buy-Stop at base + open_distance% # wait for momentum confirmation 3. PENDING FILLS → position #1. first_open = fill price first_lot = sized by the money/lot rule (section 4) count = 1 4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale). next_level_price = last_fill_price − first_open × (grid_step% / 100) next_lot = previous_lot × grid_multiplier (clamped to the volume minimum) add positions up to grid_count at step/mult set #1, then switch to set #2, etc. (cap the total number of levels — deep martingales are where accounts die) 5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5). compute the basket's average price across all open positions once profit clears (break_even + trailing_stop) → move a single basket stop up 6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick. record the series as one closed event (sum of pnl + accumulated swap) base_price = the close price (seed for the next series) reset to FLAT ``` ### 3.3 Adaptive grid levels A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's price, so the spacing stays constant in points while the trigger range tracks slippage: ``` next_level = last_fill_price − first_open_price × (grid_step% / 100) ``` This is one line, but it is the difference between a grid that drifts and one that holds its spacing. --- ## 4. Position sizing — the money/lot modes Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority order: 1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`. 2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`). 3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is deployed regardless of price. After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious pitfalls (doc 05 covers them in full): - A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure the fixed-lot input is zero when you intend money mode. - For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot against MT5 before trusting a whole run. --- ## 5. Basket exits — break-even and trailing For a multi-position basket, the stop is computed on the **average price**, then pushed to every position so they close together: ``` avg = average open price of all open positions, weighted by lot # break-even: once the basket is in profit by `break_even%`, lock the stop at avg if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100: basket_stop = avg # trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100: candidate = bid − avg × trailing% / 100 if candidate > basket_stop: # SMOOTH: move up by any improvement basket_stop = candidate # (STEP variant: only move in trailing-sized jumps) ``` When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes. Single-position strategies are just the `count == 1` case of this. --- ## 6. Spread and swap — small models, large effects - **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread). Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid. - **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap weekday (commonly Wednesday for many CFDs). Two models: - *fixed per lot per day:* `swap = rate × lot × multiplier` - *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier` Accumulate per position and add it to the trade's PnL on close. These look minor but compound over a multi-year, many-position backtest into hundreds of currency units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never hard-code. --- ## 7. Fidelity — how much to trust the Python number Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread, same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are predictable, and knowing them is what makes the lab trustworthy. ### What matches MT5 well - Bar data, per-bar spread, swap (with the triple-swap day). - Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are tick-invariant, so they're identical. - Fill **prices** at pending/grid levels (the level price is the fill price). - Direction, lot sizing, money management. ### What diverges Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs target is touched, exact fill timing on volatile bars. The divergence **scales with path-sensitivity × volatility**: | Strategy character | Typical Python vs MT5 gap | |--------------------|---------------------------| | Clean directional, few exits | small and consistent: Python reads somewhat higher | | Tight trailing / martingale grid in calm years | small | | Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves | A concrete illustration: a trailing strategy might match MT5 within a few currency units in calm, choppy years, yet the Python figure can be several times the MT5 figure across a violent crash year — because MT5's finer path triggers exits at prices the 4-point model skips. The bias is almost always **Python optimistic**, and almost always concentrated in the most volatile episodes. ### The practical policy (this is the whole point of the two-tier design) 1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all the optimizer needs. 2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism. 3. **MT5-verify every finalist** — the MT5 number is the one you act on. 4. **Use real ticks where available** (recent history) for the truest check, accepting the limited window. 5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python), and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*, assume MT5 will erase it. --- ## 8. Validating a new engine (do this before any optimization) 1. Pick **one known preset** of your EA and a short period (a few months). 2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report. 3. Run your Python engine on the **same data, same preset**. 4. Reconcile **trade by trade**, then in aggregate. Target gates for a clean-directional setup: - Net difference ≤ ~2%, trade-count difference ≤ ~5%, Profit Factor essentially identical, equity-drawdown difference ≤ ~3%. 5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the engine's **baseline fidelity document** and freeze the engine (doc 04). > If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe > mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, or sub-tick > ordering. Walk those five before suspecting anything exotic. Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine validated.