267 lines
13 KiB
Markdown
267 lines
13 KiB
Markdown
# 07 — The MetaTrader 5 Bridge
|
||
|
||
The bridge connects your fast Python search to the **gold-standard** MetaTrader 5 Strategy Tester. It
|
||
does four things: **(1) compile** your EA, **(2) auto-run** a backtest from a generated config,
|
||
**(3) parse** the HTML report, **(4) compare** Python vs MT5 side by side.
|
||
|
||
The bridge has two flavors depending on **where MT5 runs** (doc `CLAUDE.md` Phase 0). Read the one
|
||
that matches your topology; the parsing and comparison steps are identical for both.
|
||
|
||
```
|
||
┌──────────────────────────── Topology A (all-Windows, simplest) ───────────────┐
|
||
Python ──► generate .set + tester.ini ──► launch terminal64.exe /config: locally ──► report.htm
|
||
└────────────────────────────────────────────────────────────────────────────────┘
|
||
┌──────────────────── Topology B/C (Unix + remote/VM Windows) ───────────────────┐
|
||
Python ──► generate .set + tester.ini ──► copy to Windows ──► scheduled-task run ──► poll ──► pull report.htm
|
||
└────────────────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
parse (UTF-16 HTML → metrics dict)
|
||
│
|
||
comparison table: Python vs MT5
|
||
```
|
||
|
||
---
|
||
|
||
## 1. Compiling your EA
|
||
|
||
The tester runs a compiled `.ex5`. You compile from `.mq5` with **MetaEditor** (ships with MT5):
|
||
|
||
- **In the GUI:** open the `.mq5` in MetaEditor → `Compile` (F7). Fix errors, confirm a clean compile.
|
||
- **From the command line** (scriptable, useful for the bridge):
|
||
|
||
```bat
|
||
:: Windows
|
||
"C:\Program Files\MetaTrader 5\metaeditor64.exe" /compile:"C:\path\to\Expert.mq5" /log
|
||
```
|
||
|
||
Notes that save hours:
|
||
- **Custom indicators** the EA calls must be compiled too and placed in `MQL5\Indicators\`.
|
||
- If your EA's `#include` files live in a non-standard folder, compile the terminal in **portable
|
||
mode** (`/portable`) so MetaEditor resolves includes from that terminal's `MQL5\Include\`.
|
||
- After compiling, the `.ex5` goes in `MQL5\Experts\`. The tester references it by name.
|
||
|
||
---
|
||
|
||
## 2. What MT5 needs to auto-run a test
|
||
|
||
Two generated files plus the binaries already in place:
|
||
|
||
### 2a. The `.set` file (parameters)
|
||
|
||
A `.set` is the EA's inputs serialized as `key=value` lines. **MT5 writes `.set` files as UTF-16-LE**
|
||
— generate yours in the same encoding or the tester silently ignores them.
|
||
|
||
Your bridge's **set generator** takes the *same* parameter dict you backtested in Python and writes the
|
||
matching `.set`. The mapping from Python parameter names to EA input names is strategy-specific — keep
|
||
a small **mapping table** documented next to the strategy so the two tiers always agree. Watch the
|
||
enum-valued inputs (mode flags, timeframe codes): MT5 inputs are often integers, so map e.g. a
|
||
timeframe to its `ENUM_TIMEFRAMES` integer, a mode to its enum index.
|
||
|
||
> **The lot-mode guard from doc 05 applies here too:** when you generate the `.set`, make sure the
|
||
> fixed-lot input is `0` if you intend money mode, and handle the quote-currency lot trap. A
|
||
> mismatched `.set` is the #1 reason a verified MT5 number disagrees with Python.
|
||
|
||
### 2b. The `tester.ini` (run config)
|
||
|
||
A small INI tells the tester *what* to run:
|
||
|
||
```ini
|
||
[Tester]
|
||
Expert=Experts\YourEA.ex5
|
||
Symbol=YOURSYMBOL
|
||
Period=H1 ; chart timeframe (≤ the EA's signal timeframe)
|
||
Model=1 ; tick model — see below
|
||
FromDate=2024.01.01
|
||
ToDate=2026.01.01
|
||
Deposit=10000
|
||
Leverage=100
|
||
Report=report_myrun ; output HTML name
|
||
ShutdownTerminal=1 ; close the terminal when done (so the bridge knows it's finished)
|
||
|
||
[TesterInputs]
|
||
; either point at the .set, or inline the key=value inputs here
|
||
```
|
||
|
||
The login section (`[Common]`) carries the demo account; keep credentials out of code (section 5).
|
||
|
||
### Tick models (the speed/accuracy dial)
|
||
|
||
| Model | Name | Accuracy | Speed (≈ 2-year run) |
|
||
|-------|------|----------|----------------------|
|
||
| 0 | Every tick based on **real ticks** | highest (truest fills) | slowest (~10–30 min) |
|
||
| 1 | Every tick (generated from M1) | high | medium |
|
||
| 2 | **1-minute OHLC** | good for non-tick-sensitive | fast (~2–3 min) |
|
||
| 4 | Open prices only | rough | fastest (~30 s) |
|
||
|
||
Use **Model 2 (1-min OHLC)** for routine verification; **Model 0 (real ticks)** for the final check on
|
||
anything path-sensitive (trailing/grid). For a pure open-bar strategy even Model 4 is exact.
|
||
|
||
---
|
||
|
||
## 3. Topology A — all on Windows (recommended, simplest)
|
||
|
||
No SSH, no scheduled tasks. Two ways, easiest first:
|
||
|
||
### 3a. The official `MetaTrader5` Python package
|
||
|
||
`pip install MetaTrader5` (Windows-only) lets Python drive the terminal directly — initialize, pull
|
||
historical bars, and read symbol info. Use it for **data download** (section 6) and terminal control.
|
||
This is the cleanest path on Windows and removes most of the plumbing the remote topologies need.
|
||
|
||
### 3b. Launch the tester from a generated config
|
||
|
||
Write `tester.ini`, then launch the terminal pointed at it:
|
||
|
||
```bat
|
||
start "" "C:\Program Files\MetaTrader 5\terminal64.exe" /config:"C:\path\to\tester.ini"
|
||
```
|
||
|
||
`ShutdownTerminal=1` makes the terminal close itself when the run finishes; your Python wrapper waits
|
||
for the process to exit (or for the report file to appear), then parses the report. Because everything
|
||
is local, there is no copy/poll step — read the HTML straight from the terminal's report folder.
|
||
|
||
> Topology A is the right starting point for a Windows user. Get this working before considering
|
||
> anything remote.
|
||
|
||
---
|
||
|
||
## 4. Topology B/C — Python on Unix, MT5 on a remote/VM Windows
|
||
|
||
You develop on macOS/Linux and MT5 lives elsewhere (a Windows VPS, or a local VM). Two quirks force a
|
||
specific design:
|
||
|
||
### Quirk 1 — a second, **portable** MT5 install
|
||
If that Windows box already runs a *live* MT5 for something else, the tester would fight it for chart
|
||
windows. Install a **separate portable** MT5 in its own folder (e.g. `C:\MT5_Tester\`) with its own
|
||
data directory. Copy `config\` (the saved login), `bases\` (symbol metadata), and `profiles\` from the
|
||
main terminal so the portable one auto-logs-in and knows your symbols. One-time setup.
|
||
|
||
### Quirk 2 — SSH-launched processes die on disconnect
|
||
A process you start over SSH gets killed when the SSH session closes — the tester would die after a few
|
||
seconds. **Solution: a one-shot Windows Task Scheduler task.** Your wrapper SSHes in, creates a task
|
||
scheduled ~20 seconds out, and disconnects. The task fires *independently*, runs the full backtest, and
|
||
the terminal shuts itself down.
|
||
|
||
### The end-to-end flow
|
||
|
||
```
|
||
1. Python: parse the .set → build tester.ini (inject demo login from the env file)
|
||
2. copy tester.ini + .set to the Windows box (scp / shared folder)
|
||
3. SSH: run a small PowerShell that registers a one-shot scheduled task ~20s out, then disconnect
|
||
4. the task fires → portable terminal64.exe /config:tester.ini → backtest → save report → shut down
|
||
5. Python: poll the Windows box for the report_*.htm file
|
||
6. when present (and the terminal has exited): copy the report back
|
||
7. parse → metrics
|
||
```
|
||
|
||
The two tiny helper scripts that live on the Windows box (a `.bat` that launches the terminal and a
|
||
`.ps1` that registers the scheduled task and clears old reports) are the only Windows-side code; keep
|
||
mirror copies in your repo so you can re-deploy them.
|
||
|
||
### Manual fallback
|
||
If the automated run breaks, you can always: copy the `.set` over, RDP in, open `Strategy Tester`
|
||
(Ctrl+R), load the preset, Start, then `Save as Report`, and copy the HTML back. Slow but unblocks you.
|
||
|
||
---
|
||
|
||
## 5. Credentials — never in code or chat
|
||
|
||
Broker demo login lives in a **gitignored env file** (e.g. `.env`):
|
||
|
||
```
|
||
MT5_DEMO_LOGIN=...
|
||
MT5_DEMO_PASSWORD=...
|
||
MT5_DEMO_SERVER=YourBroker-Demo
|
||
```
|
||
|
||
The bridge reads these at runtime and injects them into `tester.ini`'s `[Common]` section **without
|
||
printing them**. Alternatively, copying the terminal's `accounts.dat` into the portable install gives
|
||
auto-login with no password in any config at all — the cleanest option.
|
||
|
||
---
|
||
|
||
## 6. Downloading historical data (feeding the Python tier)
|
||
|
||
The Python engine needs the **same** bars MT5 uses. Two ways to get them:
|
||
|
||
- **Via the `MetaTrader5` package (Windows):** `mt5.copy_rates_range(symbol, timeframe, from, to)` →
|
||
a numpy array → save as Parquet. Pull M1 for the full window; resample to higher timeframes in Python.
|
||
- **Via an export script in MT5:** a small MQL5 script that walks `CopyRates` year by year and writes
|
||
CSV, which you then convert to Parquet. Useful when Python can't reach the terminal.
|
||
|
||
For a brand-new symbol you must first **let MT5 cache its history** (open a chart, set
|
||
`Tools → Options → Charts → Max bars: Unlimited`, scroll back) so the tester and the export have enough
|
||
bars. Confirm the exact broker symbol code (it varies: indices and metals especially) before exporting.
|
||
|
||
### 6a. Pull M1 even if your signal timeframe is higher — and pass it to the engine
|
||
|
||
A common mistake: download only the signal timeframe (e.g. M5/M15/H1) and assume that's enough. It is
|
||
not, **if your EA moves its SL during a trade** (break-even, trailing, basket trailing). The bar-level
|
||
engine produces a −40% to −50% net gap on those EAs (doc 03 §7 measured failure mode) because the BE
|
||
update and the SL trigger fall on the same bar's opposite extreme. The fix is tick-level exit
|
||
simulation, which needs the **M1 bars covering the same window as your signal bars**:
|
||
|
||
```python
|
||
# Download BOTH timeframes. Same symbol, same window, exclusive end (match MT5 tester).
|
||
m1_bars = load_bars(DATA / f"{SYMBOL}_M1_{start}_{end}.parquet")
|
||
m5_bars = load_bars(DATA / f"{SYMBOL}_M5_{start}_{end}.parquet") # the signal timeframe
|
||
|
||
# Slice both to the exact same window (exclusive end matches MT5 tester's ToDate).
|
||
m1_bars = m1_bars[(m1_bars["timestamp"] >= start) & (m1_bars["timestamp"] < end)]
|
||
m5_bars = m5_bars[(m5_bars["timestamp"] >= start) & (m5_bars["timestamp"] < end)]
|
||
|
||
# Pass m1_bars to the engine — it switches to tick-level exit simulation automatically.
|
||
result = engine.run(m5_bars, signals_long, signals_short, sl, tp,
|
||
instrument, sizing, deposit, m1_bars=m1_bars)
|
||
```
|
||
|
||
If your EA does **not** move its SL intra-trade (clean market entries with a fixed SL/TP), `m1_bars`
|
||
can be omitted — the bar-level engine is exact for that class and faster.
|
||
|
||
---
|
||
|
||
## 7. Parsing the report
|
||
|
||
MT5 saves the Strategy Tester report as **HTML encoded UTF-16-LE** — not UTF-8. Read it with the right
|
||
encoding, parse with `lxml`/`html5lib`, and pull the key fields:
|
||
|
||
- `Total Net Profit`
|
||
- `Profit Factor`
|
||
- `Total Trades`
|
||
- `Balance Drawdown Maximal` and **`Equity Drawdown Maximal`** (use **Maximal**, not Absolute — it's
|
||
the peak-to-trough that matters)
|
||
|
||
Emit a `metrics` dict and (optionally) dump it to JSON next to the run. Validate the report is real
|
||
before trusting it (e.g. file size over some floor and a non-zero bar count) — a truncated report means
|
||
the run failed.
|
||
|
||
---
|
||
|
||
## 8. The comparison table
|
||
|
||
For every finalist, write an `auto-verification.md` that puts the two tiers side by side:
|
||
|
||
| Metric | Python | MT5 | Δ |
|
||
|--------|--------|-----|---|
|
||
| Net | … | … | …% |
|
||
| Profit Factor | … | … | … |
|
||
| Equity DD max | … | … | …% |
|
||
| Total trades | … | … | …% |
|
||
|
||
Then judge the delta **against the expected fidelity gap** (doc 03 §7 + §8 target-gate table):
|
||
a clean-directional setup (bar-level engine, SL not moved intra-trade) should show ≤ ~2% net gap;
|
||
a trailing/BE setup on the **bar-level** engine is not trustworthy at all (−40% to −50% gap,
|
||
doc 03 §7 measured failure mode) — **before** judging the gap "expected", confirm the Python run
|
||
used M1 tick-level exit simulation (`m1_bars=` passed to the engine), which brings the gap into the
|
||
≤ ~10% range. Only a gap inside the §8 target gate counts as "expected"; a wider gap is a
|
||
missing-M1 / wrong-engine-mode bug, not fidelity noise. The decision rule: if the **MT5** number
|
||
still clears your bar after the gap, the finalist is real; if the edge only existed in the
|
||
optimistic Python figure, discard it.
|
||
|
||
> A useful warm-up calibration: pick one known preset and run the full Python-vs-MT5 comparison on it
|
||
> first. It tells you *your* stack's actual gap for *your* EA, so later finalists are judged against a
|
||
> measured baseline rather than a generic rule of thumb.
|
||
|
||
Next: [`08-workflow-cycle.md`](08-workflow-cycle.md) — the repeatable loop that ties all of this
|
||
together.
|