mymt5opp/07-mt5-bridge.md

# 07 — The MetaTrader 5 Bridge

The bridge connects your fast Python search to the **gold-standard** MetaTrader 5 Strategy Tester. It
does four things: **(1) compile** your EA, **(2) auto-run** a backtest from a generated config,
**(3) parse** the HTML report, **(4) compare** Python vs MT5 side by side.

The bridge has two flavors depending on **where MT5 runs** (doc `CLAUDE.md` Phase 0). Read the one
that matches your topology; the parsing and comparison steps are identical for both.

```
              ┌──────────────────────────── Topology A (all-Windows, simplest) ───────────────┐
 Python  ──►  generate .set + tester.ini  ──►  launch terminal64.exe /config: locally  ──►  report.htm
              └────────────────────────────────────────────────────────────────────────────────┘
              ┌──────────────────── Topology B/C (Unix + remote/VM Windows) ───────────────────┐
 Python  ──►  generate .set + tester.ini  ──►  copy to Windows  ──►  scheduled-task run  ──►  poll ──► pull report.htm
              └────────────────────────────────────────────────────────────────────────────────┘
                                            │
                                  parse (UTF-16 HTML → metrics dict)
                                            │
                                  comparison table: Python vs MT5
```

---

## 1. Compiling your EA

The tester runs a compiled `.ex5`. You compile from `.mq5` with **MetaEditor** (ships with MT5):

- **In the GUI:** open the `.mq5` in MetaEditor → `Compile` (F7). Fix errors, confirm a clean compile.
- **From the command line** (scriptable, useful for the bridge):

```bat
:: Windows
"C:\Program Files\MetaTrader 5\metaeditor64.exe" /compile:"C:\path\to\Expert.mq5" /log
```

Notes that save hours:
- **Custom indicators** the EA calls must be compiled too and placed in `MQL5\Indicators\`.
- If your EA's `#include` files live in a non-standard folder, compile the terminal in **portable
  mode** (`/portable`) so MetaEditor resolves includes from that terminal's `MQL5\Include\`.
- After compiling, the `.ex5` goes in `MQL5\Experts\`. The tester references it by name.

---

## 2. What MT5 needs to auto-run a test

Two generated files plus the binaries already in place:

### 2a. The `.set` file (parameters)

A `.set` is the EA's inputs serialized as `key=value` lines. **MT5 writes `.set` files as UTF-16-LE**
— generate yours in the same encoding or the tester silently ignores them.

Your bridge's **set generator** takes the *same* parameter dict you backtested in Python and writes the
matching `.set`. The mapping from Python parameter names to EA input names is strategy-specific — keep
a small **mapping table** documented next to the strategy so the two tiers always agree. Watch the
enum-valued inputs (mode flags, timeframe codes): MT5 inputs are often integers, so map e.g. a
timeframe to its `ENUM_TIMEFRAMES` integer, a mode to its enum index.

> **The lot-mode guard from doc 05 applies here too:** when you generate the `.set`, make sure the
> fixed-lot input is `0` if you intend money mode, and handle the quote-currency lot trap. A
> mismatched `.set` is the #1 reason a verified MT5 number disagrees with Python.

### 2b. The `tester.ini` (run config)

A small INI tells the tester *what* to run:

```ini
[Tester]
Expert=Experts\YourEA.ex5
Symbol=YOURSYMBOL
Period=H1                 ; chart timeframe (≤ the EA's signal timeframe)
Model=1                   ; tick model — see below
FromDate=2024.01.01
ToDate=2026.01.01
Deposit=10000
Leverage=100
Report=report_myrun       ; output HTML name
ShutdownTerminal=1        ; close the terminal when done (so the bridge knows it's finished)

[TesterInputs]
; either point at the .set, or inline the key=value inputs here
```

The login section (`[Common]`) carries the demo account; keep credentials out of code (section 5).

### Tick models (the speed/accuracy dial)

| Model | Name | Accuracy | Speed (≈ 2-year run) |
|-------|------|----------|----------------------|
| 0 | Every tick based on **real ticks** | highest (truest fills) | slowest (~10–30 min) |
| 1 | Every tick (generated from M1) | high | medium |
| 2 | **1-minute OHLC** | good for non-tick-sensitive | fast (~2–3 min) |
| 4 | Open prices only | rough | fastest (~30 s) |

Use **Model 2 (1-min OHLC)** for routine verification; **Model 0 (real ticks)** for the final check on
anything path-sensitive (trailing/grid). For a pure open-bar strategy even Model 4 is exact.

---

## 3. Topology A — all on Windows (recommended, simplest)

No SSH, no scheduled tasks. Two ways, easiest first:

### 3a. The official `MetaTrader5` Python package

`pip install MetaTrader5` (Windows-only) lets Python drive the terminal directly — initialize, pull
historical bars, and read symbol info. Use it for **data download** (section 6) and terminal control.
This is the cleanest path on Windows and removes most of the plumbing the remote topologies need.

### 3b. Launch the tester from a generated config

Write `tester.ini`, then launch the terminal pointed at it:

```bat
start "" "C:\Program Files\MetaTrader 5\terminal64.exe" /config:"C:\path\to\tester.ini"
```

`ShutdownTerminal=1` makes the terminal close itself when the run finishes; your Python wrapper waits
for the process to exit (or for the report file to appear), then parses the report. Because everything
is local, there is no copy/poll step — read the HTML straight from the terminal's report folder.

> Topology A is the right starting point for a Windows user. Get this working before considering
> anything remote.

---

## 4. Topology B/C — Python on Unix, MT5 on a remote/VM Windows

You develop on macOS/Linux and MT5 lives elsewhere (a Windows VPS, or a local VM). Two quirks force a
specific design:

### Quirk 1 — a second, **portable** MT5 install
If that Windows box already runs a *live* MT5 for something else, the tester would fight it for chart
windows. Install a **separate portable** MT5 in its own folder (e.g. `C:\MT5_Tester\`) with its own
data directory. Copy `config\` (the saved login), `bases\` (symbol metadata), and `profiles\` from the
main terminal so the portable one auto-logs-in and knows your symbols. One-time setup.

### Quirk 2 — SSH-launched processes die on disconnect
A process you start over SSH gets killed when the SSH session closes — the tester would die after a few
seconds. **Solution: a one-shot Windows Task Scheduler task.** Your wrapper SSHes in, creates a task
scheduled ~20 seconds out, and disconnects. The task fires *independently*, runs the full backtest, and
the terminal shuts itself down.

### The end-to-end flow

```
1. Python: parse the .set → build tester.ini (inject demo login from the env file)
2. copy tester.ini + .set to the Windows box  (scp / shared folder)
3. SSH: run a small PowerShell that registers a one-shot scheduled task ~20s out, then disconnect
4. the task fires → portable terminal64.exe /config:tester.ini → backtest → save report → shut down
5. Python: poll the Windows box for the report_*.htm file
6. when present (and the terminal has exited): copy the report back
7. parse → metrics
```

The two tiny helper scripts that live on the Windows box (a `.bat` that launches the terminal and a
`.ps1` that registers the scheduled task and clears old reports) are the only Windows-side code; keep
mirror copies in your repo so you can re-deploy them.

### Manual fallback
If the automated run breaks, you can always: copy the `.set` over, RDP in, open `Strategy Tester`
(Ctrl+R), load the preset, Start, then `Save as Report`, and copy the HTML back. Slow but unblocks you.

---

## 5. Credentials — never in code or chat

Broker demo login lives in a **gitignored env file** (e.g. `.env`):

```
MT5_DEMO_LOGIN=...
MT5_DEMO_PASSWORD=...
MT5_DEMO_SERVER=YourBroker-Demo
```

The bridge reads these at runtime and injects them into `tester.ini`'s `[Common]` section **without
printing them**. Alternatively, copying the terminal's `accounts.dat` into the portable install gives
auto-login with no password in any config at all — the cleanest option.

---

## 6. Downloading historical data (feeding the Python tier)

The Python engine needs the **same** bars MT5 uses. Two ways to get them:

- **Via the `MetaTrader5` package (Windows):** `mt5.copy_rates_range(symbol, timeframe, from, to)` →
  a numpy array → save as Parquet. Pull M1 for the full window; resample to higher timeframes in Python.
- **Via an export script in MT5:** a small MQL5 script that walks `CopyRates` year by year and writes
  CSV, which you then convert to Parquet. Useful when Python can't reach the terminal.

For a brand-new symbol you must first **let MT5 cache its history** (open a chart, set
`Tools → Options → Charts → Max bars: Unlimited`, scroll back) so the tester and the export have enough
bars. Confirm the exact broker symbol code (it varies: indices and metals especially) before exporting.

---

## 7. Parsing the report

MT5 saves the Strategy Tester report as **HTML encoded UTF-16-LE** — not UTF-8. Read it with the right
encoding, parse with `lxml`/`html5lib`, and pull the key fields:

- `Total Net Profit`
- `Profit Factor`
- `Total Trades`
- `Balance Drawdown Maximal` and **`Equity Drawdown Maximal`** (use **Maximal**, not Absolute — it's
  the peak-to-trough that matters)

Emit a `metrics` dict and (optionally) dump it to JSON next to the run. Validate the report is real
before trusting it (e.g. file size over some floor and a non-zero bar count) — a truncated report means
the run failed.

---

## 8. The comparison table

For every finalist, write an `auto-verification.md` that puts the two tiers side by side:

| Metric | Python | MT5 | Δ |
|--------|--------|-----|---|
| Net | … | … | …% |
| Profit Factor | … | … | … |
| Equity DD max | … | … | …% |
| Total trades | … | … | …% |

Then judge the delta **against the expected fidelity gap** (doc 03 §7): a clean-directional setup
should show only a modest negative gap (MT5 a little below Python); a trailing/grid setup in volatile
history can gap much wider, and that's *expected*, not a bug. The decision rule: if the **MT5** number still clears your bar after the gap,
the finalist is real; if the edge only existed in the optimistic Python figure, discard it.

> A useful warm-up calibration: pick one known preset and run the full Python-vs-MT5 comparison on it
> first. It tells you *your* stack's actual gap for *your* EA, so later finalists are judged against a
> measured baseline rather than a generic rule of thumb.

Next: [`08-workflow-cycle.md`](08-workflow-cycle.md) — the repeatable loop that ties all of this
together.