mt5-quant/docs/ARCHITECTURE.md

Architecture Deep Dive

Design Philosophy

Deal-level over aggregate. MT5's built-in HTML report gives you: profit, profit factor, max DD%, trade count. That's it. You cannot tell from those numbers whether a drawdown was caused by overleveraged locking, a bad entry during a news spike, or a grid that reached L8 in a trending market.

MT5-Quant extracts every individual deal — entry price, exit price, P/L, comment string — and reconstructs what was happening when each loss event occurred. The analysis.json artifact is the result: AI-readable, stable schema, diffable between runs.

Pipeline idempotency. MT5 caches aggressively. Cached .ex5 binaries, cached .set files, stale terminal.ini flags. The pipeline exists specifically to invalidate all of these before every run. A backtest result that came from a cached EA binary is wrong in a way that's impossible to detect without the cache clear step.

Background isolation for optimization. Genetic optimizations run 2-6 hours. Running them inside a parent process that can be killed (Claude task runner, SSH session, terminal) corrupts the MT5 optimization state. The only correct pattern is nohup + disown — fully detached from all parent process trees.

---

Component Map

MT5-Quant/
├── server/
│   └── main.py               # MCP server (stdio transport) — 10 tool handlers
│
├── scripts/
│   ├── setup.sh              # Auto-detect Wine/MT5, write config, register MCP
│   ├── platform_detect.sh    # Sourced by all scripts — Wine path + headless detection
│   ├── backtest_pipeline.sh  # 5-stage pipeline orchestrator
│   ├── optimize.sh           # Genetic optimization launcher (nohup + disown)
│   └── mqlcompile.sh         # MetaEditor wrapper (Wine/CrossOver)
│
├── analytics/
│   ├── extract.py            # HTML/XML report parser → metrics.json + deals.csv
│   ├── analyze.py            # Deal-level analysis engine → analysis.json
│   └── optimize_parser.py    # Optimization result parser
│
├── config/
│   ├── MT5-Quant.example.yaml  # Template config (copy to MT5-Quant.yaml)
│   ├── MT5-Quant.yaml          # Live config (gitignored)
│   ├── baseline.json         # Production baseline metrics (gitignored, user-maintained)
│   ├── CLAUDE.template.md    # CLAUDE.md template (generated by --claude-code)
│   └── example.set           # Example optimization .set file
│
├── .claude/
│   └── hooks/
│       └── user-prompt-submit.sh  # Injects baseline.json into every Claude prompt
│
└── docs/
    ├── ARCHITECTURE.md       # This file
    ├── MCP_TOOLS.md          # Full tool spec
    └── REMOTE_AGENTS.md      # Linux agent farm setup

---

Pipeline Stages

Stage 1: COMPILE

./scripts/mqlcompile.sh src/experts/MyEA.mq5

Invokes MetaEditor via Wine with the MQL5 source file. Copies resulting .ex5 to the MT5 Experts directory. Fails the pipeline on compile errors (stderr contains error count).

Why not skip this? MT5 caches the .ex5 binary by filename. If you edit your EA and re-run without recompiling, MT5 runs the old binary silently. Always compile.

---

Stage 2: CLEAN

rm -f "${MT5_TESTER_DIR}/cache/*.tst"
rm -f "${MT5_PROFILES_DIR}/Tester/${EXPERT}.set"

Clears:

• Tester cache (.tst files): compiled test results MT5 reuses to skip re-running ticks
• Cached .set file: MT5 writes the current parameter values here after each run; if stale, next run picks up wrong params

The .set encoding trap: MT5 reads parameter files as UTF-16LE with BOM. It writes them back as UTF-16LE. If you provide a UTF-8 .set file for optimization, MT5 reads the parameters correctly (it tries multiple encodings) but when it writes the optimization variants, it uses UTF-16LE and strips the ||Y optimization flags. Every subsequent pass uses the base value. Your 500-combination optimization runs 500 identical backtests.

Solution: always write .set files as UTF-16LE with BOM, mark them read-only before MT5 starts.

# Python: write UTF-16LE with BOM
content = "\n".join(lines)
with open(set_path, 'w', encoding='utf-16-le') as f:
    f.write('\ufeff')  # BOM
    f.write(content)
os.chmod(set_path, 0o444)  # read-only

---

Stage 3: BACKTEST

arch -x86_64 "${WINE}" cmd.exe /c 'C:\_backtest.bat'

The batch file sets MT5 CLI flags and launches terminal64.exe:

"C:\Program Files\MetaTrader 5\terminal64.exe" /config:C:\backtest.ini

backtest.ini contains:

[Tester]
Expert=MyEA
Symbol=XAUUSD
Period=M5
Deposit=10000
Currency=USD
Leverage=500
Model=0
FromDate=2025.01.01
ToDate=2025.06.30
Report=C:\report
Optimization=0

MT5 runs in headless mode, writes the report, and exits.

Process isolation note: On macOS with CrossOver, arch -x86_64 is required — CrossOver ships arm64 Wine wrappers that don't support the x86_64 MT5 binary correctly. Without it, MT5 appears to start but produces no output.

---

Stage 4: EXTRACT

Single HTML/XML parse pass that produces three artifacts:

python3 analytics/extract.py report.htm
# → metrics.json  (aggregate summary)
# → deals.csv     (all deals, 13 columns)
# → deals.json    (same data, JSON)

Why single-pass? MT5 HTML reports are large (1-5MB for 14-month tests). Each regex pass over the file takes ~200ms. The old pipeline ran 5 separate grep/regex passes, one per artifact. Collapsed to one Python pass: 5× faster and no partial-read inconsistencies.

Format detection:

# MT5 Build 48+ saves SpreadsheetML XML, not HTML
if report_path.endswith('.xml') or report_path.endswith('.htm.xml'):
    tree = ET.parse(report_path)
    # parse via ElementTree
else:
    with open(report_path, 'rb') as f:
        raw = f.read()
    try:
        text = raw.decode('utf-16')
    except:
        text = raw.decode('latin-1', errors='replace')
    # parse via regex

Deal columns (13):

Time | Type | Direction | Volume | Price | S/L | T/P | Profit | Balance | Comment | Order | Magic | Entry

The Comment column is the key to grid analytics. The EA writes "Layer #3", "Locking Total", "Zombie Exit" etc. Pattern matching on comments reconstructs which position was at which layer.

---

Stage 5: ANALYZE

# Strategy subcommand (recommended)
python3 analytics/analyze.py grid    deals.csv --output-dir reports/20250101/
python3 analytics/analyze.py scalper deals.csv --output-dir reports/20250101/
python3 analytics/analyze.py trend   deals.csv --output-dir reports/20250101/
python3 analytics/analyze.py hedge   deals.csv --output-dir reports/20250101/
python3 analytics/analyze.py generic deals.csv --output-dir reports/20250101/

# Legacy positional (no subcommand) — defaults to 'grid', backward compatible
python3 analytics/analyze.py deals.csv --output-dir reports/20250101/

# Flags available with any strategy
python3 analytics/analyze.py grid deals.csv --output-dir DIR --deep --stdout
# → analysis.json

All functions operate on the 13-column deals.csv — no MT5 or Wine required.

Strategy profiles

The analysis engine is driven by a PROFILES dict. Each profile controls:

Field	Type	Controls
depth_re	regex or None	Whether/how to extract depth from comments
exit_keywords	{reason: [kw]}	Comment patterns for exit classification
dd_cause_keywords	{cause: [kw]}	Comment patterns for DD cause classification
cycle_group_by	'magic' or 'magic+direction'	How deals are grouped into cycles
cycle_gap_min	int	Minutes between opens that mark a new cycle

Built-in profiles:

Profile	depth_re	cycle_group_by	cycle_gap_min	Exit keywords
generic	—	magic	60	profit-sign only (tp/sl)
grid	Layer #N	magic+direction	60	locking, cutloss, zombie, timeout
scalper	—	magic	10	tp, sl, manual, trailing
trend	—	magic	240	breakeven, trailing, partial, tp, sl
hedge	—	magic+direction	120	tp, sl, net_close, partial

Entry points (after pip install -e .)

mt5-analyze         deals.csv   # generic
mt5-analyze-grid    deals.csv   # grid / martingale (default in pipeline)
mt5-analyze-scalper deals.csv   # scalper
mt5-analyze-trend   deals.csv   # trend following
mt5-analyze-hedge   deals.csv   # hedging

Analytics functions

Core (always run, strategy-agnostic):

Function	What it computes
monthly_pnl	P/L, trade count, green flag per calendar month
reconstruct_dd_events	Balance curve → local minima; cause from profile keywords
top_losses	Worst individual closing deals by P/L
loss_sequences	Consecutive losing closed deals (runs of length ≥ 2)
position_pairs	Match in/out by order ticket → hold time, depth at close
direction_bias	Buy vs sell win rate, total P/L, average trade
streak_analysis	Max consecutive win/loss streaks; current streak
session_breakdown	Asian (00–08h) / London (08–13h) / London-NY (13–17h) / New York (17–22h)
weekday_pnl	Mon–Sun P/L and win rate
concurrent_peak	Peak simultaneous open positions

Strategy-driven (output varies by profile):

Function	Generic	Grid	Scalper/Trend/Hedge
depth_histogram	{} (empty)	L1–L8+ counts	{} (no depth_re)
cycle_stats	magic, 60-min gap	magic+direction, 60-min gap	per-profile config
exit_reason_breakdown	tp / sl	locking / cutloss / zombie / timeout	profile-specific

Deep analytics (--deep flag):

Function	What it computes
hourly_pnl	Hour-by-hour (0–23) P/L and win rate
volume_profile	P/L breakdown by lot size tier

DD event reconstruction:

1. Walk deals chronologically, track running balance
2. At each local minimum (DD > 1%), record timestamp, depth (%), recovery date
3. Classify cause using profile['dd_cause_keywords']; returns "unknown" for generic/unmatched

Cycle statistics: Deals are grouped by cycle_group_by key. A gap greater than cycle_gap_min between consecutive opens marks a new cycle boundary. Win rate is computed per cycle (not per deal), then broken down by max depth reached.

Exit reason classification: Iterates exit_keywords in definition order — more specific patterns must appear before general ones to avoid substring false-positives (e.g. "stop" inside "breakeven stop"). Falls back to profit-sign if no keyword matches.

Loss sequence detection: Consecutive closed deals where P/L < 0 (minimum length 2). Captures clusters of losses better than any single worst-trade metric.

---

Optimization Pipeline

Why nohup + disown is mandatory

nohup ./scripts/optimize.sh ... > /tmp/opt.log 2>&1 & disown

MT5 optimization uses Unix signals to coordinate between terminal64.exe (master) and metatester64.exe instances (workers). When the parent process tree is killed:

1. SIGHUP propagates to child processes
2. metatester64.exe workers receive the signal and terminate
3. The master terminal64.exe detects worker failure and aborts the optimization
4. terminal.ini is left with OptMode=-1, requiring manual reset before next run

nohup prevents SIGHUP propagation. disown removes the process from the shell's job table so it's not killed when the shell exits. Both are required.

---

OptMode state machine

terminal.ini contains an OptMode key that MT5 uses to track optimization state:

OptMode value	Meaning
0	Normal backtest mode (ready)
1	Optimization in progress
2	Optimization complete — show results
-1	Optimization aborted / crashed

After any optimization run (complete or aborted), MT5 writes -1 or 2. On next launch with Optimization=2 in backtest.ini, MT5 reads OptMode=-1 and exits immediately without running.

Fix: Before every optimization launch, force OptMode=0 in terminal.ini:

sed -i 's/OptMode=.*/OptMode=0/' "${MT5_DIR}/terminal.ini"
# Also remove LastOptimization line if present
sed -i '/LastOptimization=/d' "${MT5_DIR}/terminal.ini"

---

Remote Agent Architecture

MT5's distributed testing works via a custom TCP protocol. The master terminal64.exe listens on a port. Remote agents (metatester64.exe) connect and receive test configurations.

Mac (master)                    Linux server (agents)
terminal64.exe                  metatester64.exe × N
    │                                   │
    └──── TCP:3000 ─────────────────────┘

Linux setup:

# On Linux server (Wine required)
wine metatester64.exe /server:MAC_IP:3000 /agents:8

MT5 shows remote agents in the agent manager as Agent-0.0.0.0-PORT entries when listening, and activates them when the remote metatester64.exe connects.

Throughput: Linear scaling with agent count. 10 local + 16 remote = 26 agents. A 17,000-combination optimization that takes 3 hours locally completes in ~70 minutes.

---

Headless Operation

MT5-Quant uses MT5's CLI mode (terminal64.exe /config:backtest.ini) — no user interaction, no clicking in the Strategy Tester GUI. Whether this is truly "headless" depends on platform:

Platform	Status	Notes
macOS + CrossOver	Near-headless	CrossOver manages the display internally. MT5 window may flash briefly or be suppressed entirely depending on bottle settings. No monitor required in practice.
Linux + Wine	Requires Xvfb	Wine needs an X11 display connection. Without one, wine64 terminal64.exe fails with cannot open display.
Linux + Wine + Xvfb	Full headless	Virtual framebuffer satisfies Wine's X11 requirement. Use on servers with no monitor.

Linux headless setup (Xvfb):

# Install Xvfb
sudo apt install xvfb

# Start virtual display on :99
Xvfb :99 -screen 0 1024x768x16 &
export DISPLAY=:99

# Now Wine can launch MT5 without a physical display
wine64 terminal64.exe /config:backtest.ini

Persistent virtual display (systemd):

# /etc/systemd/system/xvfb.service
[Unit]
Description=Virtual Display for MT5

[Service]
ExecStart=/usr/bin/Xvfb :99 -screen 0 1024x768x16
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl enable xvfb
sudo systemctl start xvfb

Then set DISPLAY=:99 in MT5-Quant's environment config.

Note: metatester64.exe (the agent worker process) is fully headless — it runs tick simulation with no display requirement. Only the master terminal64.exe needs a display to orchestrate the session. On Mac with CrossOver this is handled transparently.

---

Known Limitations

macOS-specific:

• Requires Wine. The native MT5.app from the Mac App Store (or MetaQuotes CDN) ships bundled Wine at MetaTrader 5.app/Contents/SharedSupport/wine/. CrossOver is an alternative.
• arch -x86_64 required on Apple Silicon.
• File paths must go through Wine's virtual filesystem (C:\ = inside the Wine prefix drive_c/).

Report format dependency:

• SpreadsheetML XML format (.htm.xml) has no documented schema from MetaQuotes. The parser is reverse-engineered from observed output. May break on future MT5 builds.

Comment-based analytics:

• Strategy-specific analytics (depth histogram, exit reason, DD cause) depend on EA comment strings. EAs that don't write structured comments will get generic profile results — summary metrics, session breakdown, streaks, and direction bias all still work; only keyword-classified fields fall back to "unknown" or profit-sign.
• Custom comment patterns can be supported by adding a new entry to PROFILES in analytics/analyze.py — no other code changes needed.

Single MT5 instance:

• MT5 is single-instance per Windows drive. Two backtests cannot run simultaneously on the same Wine prefix. Parallelism requires multiple Wine prefixes (separate installations).

---

Claude Code Integration

setup.sh --claude-code generates two files that give Claude persistent context about the user's trading setup:

config/CLAUDE.template.md

A project-level CLAUDE.md template the user copies to their EA project root. Encodes:

• MT5-Quant tool names and when to use them
• Baseline tracking policy (never call something an improvement without comparing to baseline.json)
• Symbol name reminder (broker-specific suffix matters — XAUUSD.cent ≠ XAUUSD)
• Backtest and optimization constraints (model 0, single instance, UTF-16LE .set files)

.claude/hooks/user-prompt-submit.sh

A Claude Code hook that runs before every prompt submission. Reads config/baseline.json and outputs a JSON context block:

{"context": "## Production Baseline (config/baseline.json)\n..."}

Claude Code injects this into the system context for every conversation turn. The result: Claude always knows the current production metrics without the user having to paste them.

Hook execution path:

User types prompt
    → user-prompt-submit.sh executes
    → reads config/baseline.json
    → outputs {"context": "..."} to stdout
    → Claude Code prepends to system context
    → Claude sees baseline in every prompt

Graceful degradation: If baseline.json doesn't exist or is malformed, the hook exits 0 silently — no prompt is blocked. The baseline section simply doesn't appear until the user creates the file.