# 07 — The MetaTrader 5 Bridge The bridge connects your fast Python search to the **gold-standard** MetaTrader 5 Strategy Tester. It does four things: **(1) compile** your EA, **(2) auto-run** a backtest from a generated config, **(3) parse** the HTML report, **(4) compare** Python vs MT5 side by side. The bridge has two flavors depending on **where MT5 runs** (doc `CLAUDE.md` Phase 0). Read the one that matches your topology; the parsing and comparison steps are identical for both. ``` ┌──────────────────────────── Topology A (all-Windows, simplest) ───────────────┐ Python ──► generate .set + tester.ini ──► launch terminal64.exe /config: locally ──► report.htm └────────────────────────────────────────────────────────────────────────────────┘ ┌──────────────────── Topology B/C (Unix + remote/VM Windows) ───────────────────┐ Python ──► generate .set + tester.ini ──► copy to Windows ──► scheduled-task run ──► poll ──► pull report.htm └────────────────────────────────────────────────────────────────────────────────┘ │ parse (UTF-16 HTML → metrics dict) │ comparison table: Python vs MT5 ``` --- ## 1. Compiling your EA The tester runs a compiled `.ex5`. You compile from `.mq5` with **MetaEditor** (ships with MT5): - **In the GUI:** open the `.mq5` in MetaEditor → `Compile` (F7). Fix errors, confirm a clean compile. - **From the command line** (scriptable, useful for the bridge): ```bat :: Windows "C:\Program Files\MetaTrader 5\metaeditor64.exe" /compile:"C:\path\to\Expert.mq5" /log ``` Notes that save hours: - **Custom indicators** the EA calls must be compiled too and placed in `MQL5\Indicators\`. - If your EA's `#include` files live in a non-standard folder, compile the terminal in **portable mode** (`/portable`) so MetaEditor resolves includes from that terminal's `MQL5\Include\`. - After compiling, the `.ex5` goes in `MQL5\Experts\`. The tester references it by name. --- ## 2. What MT5 needs to auto-run a test Two generated files plus the binaries already in place: ### 2a. The `.set` file (parameters) A `.set` is the EA's inputs serialized as `key=value` lines. **MT5 writes `.set` files as UTF-16-LE** — generate yours in the same encoding or the tester silently ignores them. Your bridge's **set generator** takes the *same* parameter dict you backtested in Python and writes the matching `.set`. The mapping from Python parameter names to EA input names is strategy-specific — keep a small **mapping table** documented next to the strategy so the two tiers always agree. Watch the enum-valued inputs (mode flags, timeframe codes): MT5 inputs are often integers, so map e.g. a timeframe to its `ENUM_TIMEFRAMES` integer, a mode to its enum index. > **The lot-mode guard from doc 05 applies here too:** when you generate the `.set`, make sure the > fixed-lot input is `0` if you intend money mode, and handle the quote-currency lot trap. A > mismatched `.set` is the #1 reason a verified MT5 number disagrees with Python. ### 2b. The `tester.ini` (run config) A small INI tells the tester *what* to run: ```ini [Tester] Expert=Experts\YourEA.ex5 Symbol=YOURSYMBOL Period=H1 ; chart timeframe (≤ the EA's signal timeframe) Model=1 ; tick model — see below FromDate=2024.01.01 ToDate=2026.01.01 Deposit=10000 Leverage=100 Report=report_myrun ; output HTML name ShutdownTerminal=1 ; close the terminal when done (so the bridge knows it's finished) [TesterInputs] ; either point at the .set, or inline the key=value inputs here ``` The login section (`[Common]`) carries the demo account; keep credentials out of code (section 5). ### Tick models (the speed/accuracy dial) | Model | Name | Accuracy | Speed (≈ 2-year run) | |-------|------|----------|----------------------| | 0 | Every tick based on **real ticks** | highest (truest fills) | slowest (~10–30 min) | | 1 | Every tick (generated from M1) | high | medium | | 2 | **1-minute OHLC** | good for non-tick-sensitive | fast (~2–3 min) | | 4 | Open prices only | rough | fastest (~30 s) | Use **Model 2 (1-min OHLC)** for routine verification; **Model 0 (real ticks)** for the final check on anything path-sensitive (trailing/grid). For a pure open-bar strategy even Model 4 is exact. --- ## 3. Topology A — all on Windows (recommended, simplest) No SSH, no scheduled tasks. Two ways, easiest first: ### 3a. The official `MetaTrader5` Python package `pip install MetaTrader5` (Windows-only) lets Python drive the terminal directly — initialize, pull historical bars, and read symbol info. Use it for **data download** (section 6) and terminal control. This is the cleanest path on Windows and removes most of the plumbing the remote topologies need. ### 3b. Launch the tester from a generated config Write `tester.ini`, then launch the terminal pointed at it: ```bat start "" "C:\Program Files\MetaTrader 5\terminal64.exe" /config:"C:\path\to\tester.ini" ``` `ShutdownTerminal=1` makes the terminal close itself when the run finishes; your Python wrapper waits for the process to exit (or for the report file to appear), then parses the report. Because everything is local, there is no copy/poll step — read the HTML straight from the terminal's report folder. > Topology A is the right starting point for a Windows user. Get this working before considering > anything remote. --- ## 4. Topology B/C — Python on Unix, MT5 on a remote/VM Windows You develop on macOS/Linux and MT5 lives elsewhere (a Windows VPS, or a local VM). Two quirks force a specific design: ### Quirk 1 — a second, **portable** MT5 install If that Windows box already runs a *live* MT5 for something else, the tester would fight it for chart windows. Install a **separate portable** MT5 in its own folder (e.g. `C:\MT5_Tester\`) with its own data directory. Copy `config\` (the saved login), `bases\` (symbol metadata), and `profiles\` from the main terminal so the portable one auto-logs-in and knows your symbols. One-time setup. ### Quirk 2 — SSH-launched processes die on disconnect A process you start over SSH gets killed when the SSH session closes — the tester would die after a few seconds. **Solution: a one-shot Windows Task Scheduler task.** Your wrapper SSHes in, creates a task scheduled ~20 seconds out, and disconnects. The task fires *independently*, runs the full backtest, and the terminal shuts itself down. ### The end-to-end flow ``` 1. Python: parse the .set → build tester.ini (inject demo login from the env file) 2. copy tester.ini + .set to the Windows box (scp / shared folder) 3. SSH: run a small PowerShell that registers a one-shot scheduled task ~20s out, then disconnect 4. the task fires → portable terminal64.exe /config:tester.ini → backtest → save report → shut down 5. Python: poll the Windows box for the report_*.htm file 6. when present (and the terminal has exited): copy the report back 7. parse → metrics ``` The two tiny helper scripts that live on the Windows box (a `.bat` that launches the terminal and a `.ps1` that registers the scheduled task and clears old reports) are the only Windows-side code; keep mirror copies in your repo so you can re-deploy them. ### Manual fallback If the automated run breaks, you can always: copy the `.set` over, RDP in, open `Strategy Tester` (Ctrl+R), load the preset, Start, then `Save as Report`, and copy the HTML back. Slow but unblocks you. --- ## 5. Credentials — never in code or chat Broker demo login lives in a **gitignored env file** (e.g. `.env`): ``` MT5_DEMO_LOGIN=... MT5_DEMO_PASSWORD=... MT5_DEMO_SERVER=YourBroker-Demo ``` The bridge reads these at runtime and injects them into `tester.ini`'s `[Common]` section **without printing them**. Alternatively, copying the terminal's `accounts.dat` into the portable install gives auto-login with no password in any config at all — the cleanest option. --- ## 6. Downloading historical data (feeding the Python tier) The Python engine needs the **same** bars MT5 uses. Two ways to get them: - **Via the `MetaTrader5` package (Windows):** `mt5.copy_rates_range(symbol, timeframe, from, to)` → a numpy array → save as Parquet. Pull M1 for the full window; resample to higher timeframes in Python. - **Via an export script in MT5:** a small MQL5 script that walks `CopyRates` year by year and writes CSV, which you then convert to Parquet. Useful when Python can't reach the terminal. For a brand-new symbol you must first **let MT5 cache its history** (open a chart, set `Tools → Options → Charts → Max bars: Unlimited`, scroll back) so the tester and the export have enough bars. Confirm the exact broker symbol code (it varies: indices and metals especially) before exporting. ### 6a. Pull M1 even if your signal timeframe is higher — and pass it to the engine A common mistake: download only the signal timeframe (e.g. M5/M15/H1) and assume that's enough. It is not, **if your EA moves its SL during a trade** (break-even, trailing, basket trailing). The bar-level engine produces a −40% to −50% net gap on those EAs (doc 03 §7 measured failure mode) because the BE update and the SL trigger fall on the same bar's opposite extreme. The fix is tick-level exit simulation, which needs the **M1 bars covering the same window as your signal bars**: ```python # Download BOTH timeframes. Same symbol, same window, exclusive end (match MT5 tester). m1_bars = load_bars(DATA / f"{SYMBOL}_M1_{start}_{end}.parquet") m5_bars = load_bars(DATA / f"{SYMBOL}_M5_{start}_{end}.parquet") # the signal timeframe # Slice both to the exact same window (exclusive end matches MT5 tester's ToDate). m1_bars = m1_bars[(m1_bars["timestamp"] >= start) & (m1_bars["timestamp"] < end)] m5_bars = m5_bars[(m5_bars["timestamp"] >= start) & (m5_bars["timestamp"] < end)] # Pass m1_bars to the engine — it switches to tick-level exit simulation automatically. result = engine.run(m5_bars, signals_long, signals_short, sl, tp, instrument, sizing, deposit, m1_bars=m1_bars) ``` If your EA does **not** move its SL intra-trade (clean market entries with a fixed SL/TP), `m1_bars` can be omitted — the bar-level engine is exact for that class and faster. --- ## 7. Parsing the report MT5 saves the Strategy Tester report as **HTML encoded UTF-16-LE** — not UTF-8. Read it with the right encoding, parse with `lxml`/`html5lib`, and pull the key fields: - `Total Net Profit` - `Profit Factor` - `Total Trades` - `Balance Drawdown Maximal` and **`Equity Drawdown Maximal`** (use **Maximal**, not Absolute — it's the peak-to-trough that matters) Emit a `metrics` dict and (optionally) dump it to JSON next to the run. Validate the report is real before trusting it (e.g. file size over some floor and a non-zero bar count) — a truncated report means the run failed. --- ## 8. The comparison table For every finalist, write an `auto-verification.md` that puts the two tiers side by side: | Metric | Python | MT5 | Δ | |--------|--------|-----|---| | Net | … | … | …% | | Profit Factor | … | … | … | | Equity DD max | … | … | …% | | Total trades | … | … | …% | Then judge the delta **against the expected fidelity gap** (doc 03 §7 + §8 target-gate table): a clean-directional setup (bar-level engine, SL not moved intra-trade) should show ≤ ~2% net gap; a trailing/BE setup on the **bar-level** engine is not trustworthy at all (−40% to −50% gap, doc 03 §7 measured failure mode) — **before** judging the gap "expected", confirm the Python run used M1 tick-level exit simulation (`m1_bars=` passed to the engine), which brings the gap into the ≤ ~10% range. Only a gap inside the §8 target gate counts as "expected"; a wider gap is a missing-M1 / wrong-engine-mode bug, not fidelity noise. The decision rule: if the **MT5** number still clears your bar after the gap, the finalist is real; if the edge only existed in the optimistic Python figure, discard it. > A useful warm-up calibration: pick one known preset and run the full Python-vs-MT5 comparison on it > first. It tells you *your* stack's actual gap for *your* EA, so later finalists are judged against a > measured baseline rather than a generic rule of thumb. Next: [`08-workflow-cycle.md`](08-workflow-cycle.md) — the repeatable loop that ties all of this together.