gavindiaz/Apex_AI_MT5_EA_Optimizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update PROJECT_HANDOFF.md with Phase 2 complete status

Browse Source 
- Added GitHub URL
- Documented all 6 critical bugs found and fixed:
  * UTF-16 LE report encoding
  * Report location (appdata root not /reports/)
  * INI Period string format (H1 not 16385)
  * INI Report relative path
  * MQL5 reference syntax
  * Stale report detection
- Added full SocketIO events table
- Added baseline run results (597 trades confirmed parsed)
- Updated file structure with all Phase 2 files
- Updated next steps to Phase 3 (PyInstaller packaging)
This commit is contained in:
LEGSTECH Optimizer
2026-04-13 02:40:13 +00:00
parent 7a3e13a734
commit 2546b4a9c0
1 changed files with 285 additions and 229 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_HANDOFF.md
+285 -229
View File
@@ -1,8 +1,9 @@
# MT5 EA Optimizer — Full Project Handoff Document
**Last Updated:** 2026-04-13
**Status:** Phase 1 complete (backend engine). Phase 2 (GUI app) = NEXT STEP
**Primary EA:** LEGSTECH_EA_V2 | Symbol: XAUUSD | Timeframe: H1
**Broker Timezone:** UTC+2
**Status:** ✅ Phase 1 (Backend Engine) + ✅ Phase 2 (Live Web Dashboard) — BOTH COMPLETE
**GitHub:** https://github.com/tonnylegacy/MT5_Optimizer
**Primary EA:** LEGSTECH_EA_V2 | Symbol: XAUUSD | Timeframe: H1 | Broker TZ: UTC+2
---
@@ -10,340 +11,395 @@
Build an automated, iterative backtesting and analysis system for MT5 Expert Advisors.
It is NOT a trading bot. It is an **optimization engine** that:
- Runs MT5 strategy tester automatically with different parameter sets
- Extracts rich trade-level data (MAE/MFE) after each run
- Analyzes results to find failure patterns
- Proposes and tests parameter mutations
- Validates improvements before accepting them
- Analyzes results to find failure patterns using statistical methods
- Proposes and tests parameter mutations based on findings
- Validates improvements before accepting them (IS → WFV → OOS)
- Prevents overfitting via Walk-Forward + Out-of-Sample testing
- Displays everything live in a beautiful browser dashboard
- Writes human-readable reports to a visible `Reports\` folder
**End user:** Traders (non-technical). Must be a double-click app, not a terminal tool.
**End user:** Traders (non-technical). Double-click app, no terminal needed.
---
## 🏗️ Architecture Overview
```
MT5 Terminal (GUI)
└── Strategy Tester (automated via .ini files)
└── LEGSTECH_EA_V2.ex5 (compiled EA with TradeLogger)
└── Writes: LEGSTECH_EA_V2_XAUUSD_TradeLog.csv
Python Optimizer (backend engine)
├── MT5 Runner → launch terminal, wait for report, collect files
├── Report Parser → parse MT5 XML/HTML report into metrics + trades
├── Log Reader → merge TradeLogger CSV (MAE/MFE) into trade objects
├── Analysis Engine → 4 modules detecting failure patterns
│ ├── ReversalAnalyzer → trades that went in profit then reversed
│ ├── TimePerformanceAnalyzer → bad sessions/hours/days
│ ├── EntryExitQualityAnalyzer → MAE/MFE quality scores
│ └── EquityCurveAnalyzer → flatness, loss clusters, R²
├── Composite Scorer → Calmar-primary weighted score (0-1)
├── Mutation Engine → findings → parameter hypotheses (13 KB rules)
├── Validation Gate → IS check → Walk-Forward → OOS
└── Data Store → SQLite (metadata) + Parquet (trade arrays)
Web App (NEXT TO BUILD — Phase 2)
├── Flask backend → serves UI, runs optimization loop
├── WebSocket → pushes live updates to browser
├── HTML/JS frontend → live dashboard, charts, findings
└── Reports folder → HTML + CSV reports per run
[Trader double-clicks "Launch Optimizer.bat"]
↓
[Browser auto-opens http://localhost:5000]
↓
[Flask + SocketIO Dashboard]
↓
[Trader clicks ▶ Start]
↓
┌────────────────────────────────────────────────────┐
│ OPTIMIZATION LOOP │
│ │
│ 1. Kill any running MT5 process (psutil) │
│ 2. Clear stale report files │
│ 3. Validate environment (EA exists, paths valid) │
│ 4. Build INI file (Period=H1, Report=relative) │
│ 5. Launch fresh MT5 via subprocess /config: │
│ 6. Wait 10s for data readiness │
│ 7. Poll MT5 appdata root for Optimizer_*.htm │
│ 8. Parse report (UTF-16 LE HTML → metrics+trades) │
│ 9. Run 4 analyzers on trade data │
│ 10. Score composite (Calmar-primary) │
│ 11. Mutate parameters → Hypotheses │
│ 12. Test each hypothesis (steps 1-10 repeated) │
│ 13. Validate: IS check → WFV → promote candidate │
│ 14. Write Reports\run_XXX\ (HTML + CSV + JSON) │
│ 15. Repeat until converged or max_iterations │
└────────────────────────────────────────────────────┘
↓
[Dashboard updates live via WebSocket events]
[Reports\ folder has all results]
```
---
## 📁 File Structure (Current State)
## 📁 Complete File Structure
```
C:\Users\DELL\Desktop\MT5_Optimizer\
├── main.py ← CLI entry point (terminal-based, to be replaced by app.py)
├── config.yaml ← ALL settings (MT5 paths, symbols, scoring weights, thresholds)
│
├── app.py ← Flask + SocketIO app entry point
│ Auto-opens browser at localhost:5000
│ Routes: /, /api/start, /api/pause,
│ /api/stop, /api/skip, /reports
│
├── optimizer_loop.py ← Background thread: full optimization pipeline
│ Emits SocketIO events for every phase/run/finding
│
├── config.yaml ← ALL settings (see section below)
├── requirements.txt ← Python dependencies
├── PROJECT_HANDOFF.md ← THIS DOCUMENT
├── "Launch Optimizer.bat" ← Double-click to start (opens browser automatically)
│ Uses Python 3.11 at hardcoded path
│
├── mql5/
│ └── TradeLogger.mqh ← MQL5 include file (ALREADY DEPLOYED to MT5)
│ └── TradeLogger.mqh ← MQL5 include: per-trade MAE/MFE logger
│ DEPLOYED to: ...Terminal\...\MQL5\Include\
│ INTEGRATED in: LEGSTECH_EA_V2.mq5 via TL_OnTick()
│
├── data/
│ ├── models.py ← Pydantic v2 models: Trade, RunMetrics, Run, Finding, Hypothesis, Candidate
│ └── store.py ← DataStore: SQLite + Parquet storage layer
│ ├── models.py ← Pydantic v2: Trade, RunMetrics, Run, Finding,
│ │ Hypothesis, Candidate, RunResult
│ └── store.py ← DataStore: SQLite metadata + Parquet trade arrays
│
├── mt5/
│ ├── ini_builder.py ← Builds MT5 tester .ini files from param dicts
│ ├── runner.py ← Launches MT5 subprocess, waits for report
│ ├── report_parser.py ← Parses MT5 XML/HTML report → RunMetrics + list[Trade]
│ └── log_reader.py ← Merges TradeLogger CSV, computes sessions/quality scores
│ ├── ini_builder.py ← Builds MT5 tester .ini files
│ │ CRITICAL: Period=H1 (string), Report=relative name
│ │ Model=4 (OHLC M1), ShutdownTerminal=1
│ ├── runner.py ← Full MT5 process control:
│ │ kill_mt5() → clear_stale() → validate() →
│ │ launch() → wait_data() → poll_report() → retry
│ │ Report location: appdata_path\ root (NOT /reports/)
│ ├── report_parser.py ← Parses MT5 HTML report
│ │ CRITICAL: MT5 reports are UTF-16 LE encoded!
│ │ Uses lhtml.document_fromstring(raw_bytes)
│ │ Pairs in/out deals → complete Trade objects
│ └── log_reader.py ← Merges TradeLogger CSV (MAE/MFE) into trades
│
├── analysis/
│ ├── base.py ← BaseAnalyzer ABC + statistical helpers
│ ├── reversal.py ← ReversalAnalyzer
│ ├── time_performance.py ← TimePerformanceAnalyzer
│ ├── entry_exit_quality.py ← EntryExitQualityAnalyzer
│ └── equity_curve.py ← EquityCurveAnalyzer
│ ├── reversal.py ← Trades that went to MFE then reversed (permutation test)
│ ├── time_performance.py ← Bad sessions/hours/days (Z-score)
│ ├── entry_exit_quality.py ← MAE/MFE quality matrix (4-case diagnosis)
│ └── equity_curve.py ← Flatness, loss clusters, R² of equity curve
│
├── scoring/
│ └── composite.py ← CompositeScorer (Calmar + PF + MFE capture + stability + recovery)
│ └── composite.py ← Weighted scorer: Calmar(35%) + PF(20%) +
│ MFE capture(20%) + stability(15%) + recovery(10%)
│
├── mutation/
│ ├── engine.py ← MutationEngine: findings → Hypothesis objects
│ ├── knowledge_base.yaml ← 13 rules mapping findings to param changes
│ └── param_manifest.yaml ← Full EA parameter space with types/bounds/defaults
│ ├── engine.py ← Findings → Hypothesis objects (dedup, cascade)
│ ├── knowledge_base.yaml ← 13 rules: finding_type → param_delta
│ └── param_manifest.yaml ← Full EA parameter space: types, bounds, defaults
│
├── validation/
│ └── gate.py ← IS check + Walk-Forward + OOS + sensitivity check
│ └── gate.py ← IS check → Walk-Forward (2 folds) → OOS
│
└── tests/
└── test_analyzers.py ← 10 unit tests (all passing ✅)
├── reports/
│ └── writer.py ← Writes per-run: summary.json, summary.html,
│ trades.csv, findings.csv, parameters.json
│
├── ui/
│ ├── templates/
│ │ ├── index.html ← Live dashboard (dark premium design)
│ │ ├── report.html ← Per-run report template
│ │ └── reports_index.html ← Reports browser page (card grid)
│ └── static/
│ ├── css/style.css ← Full dark theme CSS (glassmorphism, animations)
│ └── js/dashboard.js ← SocketIO events + Chart.js + live metrics
│
├── tests/
│ └── test_analyzers.py ← 10 unit tests (all passing ✅)
│
└── runs/ ← Generated INI + local run folders (gitignored)
└── baseline_YYYYMMDD_.../
└── run.ini
```
---
## ⚙️ Configuration (config.yaml) — Key Values
## ⚙️ Configuration (config.yaml)
```yaml
ea:
name: "LEGSTECH_EA_V2"
file: "LEGSTECH_EA_V2" # no extension — MT5 adds .ex5
symbol: "XAUUSD"
timeframe: "H1"
periods:
train_start: "2022.01.01"
train_end: "2023.12.31"
train_start: "2022.01.01"
train_end: "2023.12.31"
validate_start: "2024.01.01"
validate_end: "2024.06.30"
oos_start: "2024.07.01" # LOCKED — never touch during optimization
oos_end: "2024.12.31"
oos_start: "2024.07.01" # SACRED — never optimize against this
oos_end: "2024.12.31"
mt5:
terminal_exe: "C:/Program Files/MetaTrader 5/terminal64.exe"
appdata_path: "C:/Users/DELL/AppData/Roaming/MetaQuotes/Terminal/D0E8209F77C8CF37AD8BF550E51FF075"
mql5_files_path: "C:/Users/DELL/AppData/Roaming/MetaQuotes/Tester/D0E8209F77C8CF37AD8BF550E51FF075/Agent-127.0.0.1-3000/MQL5/Files"
tester_model: 4 # 4=OHLC M1 (fast). 0=Every Tick (slow, needs tick data)
tester_timeout_seconds: 1800
shutdown_terminal: 1 # MT5 closes itself after each test
data_readiness_wait_seconds: 10
kill_on_start: true # Always kill existing MT5 before each run
broker:
timezone_offset_hours: 2 # UTC+2
scoring:
weights:
calmar: 0.35
profit_factor: 0.20
mfe_capture: 0.20
session_stability: 0.15
recovery_factor: 0.10
timezone_offset_hours: 2 # UTC+2 (HFMarkets)
deposit: 10000.0
currency: "USD"
leverage: 100
```
---
## ✅ What is Done (Phase 1 — Backend Engine)
## ✅ What Is Complete
### Phase 1 — Backend Engine
| Component | Status | Notes |
|---|---|---|
| TradeLogger.mqh | ✅ Complete + deployed | In MT5 Include folder, tested, CSV confirmed working |
| data/models.py | ✅ Complete | All Pydantic v2 models |
| data/store.py | ✅ Complete | SQLite + Parquet CRUD |
| mt5/ini_builder.py | ✅ Complete | Generates valid MT5 tester INI files |
| mt5/runner.py | ✅ Complete | Process launch + polling + timeout |
| mt5/report_parser.py | ✅ Complete | XML + HTML dual-format parser |
| mt5/log_reader.py | ✅ Complete | MAE/MFE merge + session classification |
| analysis/reversal.py | ✅ Complete | Permutation-tested |
| analysis/time_performance.py | ✅ Complete | Hour/Session/Day analysis |
| analysis/entry_exit_quality.py | ✅ Complete | 4-case diagnosis matrix |
| analysis/equity_curve.py | ✅ Complete | Flatness + R² + loss clusters |
| scoring/composite.py | ✅ Complete | Calmar-primary weighted scorer |
| mutation/engine.py | ✅ Complete | 13 KB rules, dedup, cascade |
| validation/gate.py | ✅ Complete | IS + WFV + OOS + sensitivity |
| main.py | ✅ Complete | Terminal CLI (will be replaced by app) |
| tests/ | ✅ 10/10 passing | Pure Python, no MT5 needed |
| Unit tests verified | ✅ Working | Python 3.11 required (see below) |
| TradeLogger.mqh | ✅ Deployed | In MT5 Include folder, integrated in EA |
| data/models.py | ✅ | All Pydantic v2 models |
| data/store.py | ✅ | SQLite + Parquet |
| mt5/ini_builder.py | ✅ | Period=H1 string, relative Report path |
| mt5/runner.py | ✅ | Kill→validate→launch→wait→poll→retry |
| mt5/report_parser.py | ✅ | UTF-16 LE HTML, pairs 597 trades |
| mt5/log_reader.py | ✅ | MAE/MFE merge + session labelling |
| analysis/reversal.py | ✅ | Permutation-tested |
| analysis/time_performance.py | ✅ | Z-score per hour/session/day |
| analysis/entry_exit_quality.py | ✅ | 4-case diagnosis matrix |
| analysis/equity_curve.py | ✅ | Flatness + R² + loss clusters |
| scoring/composite.py | ✅ | Calmar-primary (0-1 score) |
| mutation/engine.py | ✅ | 13 KB rules, dedup, cascade |
| validation/gate.py | ✅ | IS → WFV → OOS → sensitivity |
| tests/ | ✅ 10/10 | Pure Python, no MT5 needed |
### Phase 2 — Live Web Dashboard
| Component | Status | Notes |
|---|---|---|
| app.py | ✅ | Flask + SocketIO, all API routes |
| optimizer_loop.py | ✅ | Background thread, live event emitter |
| ui/templates/index.html | ✅ | Score chart, metrics, findings, log, candidates |
| ui/templates/report.html | ✅ | Per-run full report with trade table |
| ui/templates/reports_index.html | ✅ | Card grid browser page |
| ui/static/css/style.css | ✅ | Premium dark theme |
| ui/static/js/dashboard.js | ✅ | SocketIO + Chart.js + live updates |
| reports/writer.py | ✅ | HTML + CSV + JSON per run |
| Launch Optimizer.bat | ✅ | Double-click launcher |
| GitHub | ✅ | https://github.com/tonnylegacy/MT5_Optimizer |
---
## 🔴 What is NOT Done Yet (Phase 2 — GUI App)
## 🐛 Critical Bugs Found & Fixed (Important for Future AI)
### The Big Next Step: Web App with Live Dashboard
**Goal:** Replace `main.py` terminal interface with a beautiful browser-based app that:
1. **Double-click `MT5_Optimizer.exe`** → browser opens automatically at `http://localhost:5000`
2. **Dashboard shows live:**
- Iteration counter + current phase badge
- Score history line chart (Calmar + composite over time)
- Live MT5 run status with elapsed timer
- Current parameters being tested
3. **Analysis panel shows** findings in plain English after each run
4. **Parameter changes panel** shows before → after with reason
5. **Results folder** `MT5_Optimizer\Reports\` gets:
- `run_001\summary.html` — full backtest result in readable format
- `run_001\trades.csv` — all trades with MAE/MFE
- `run_001\findings.csv` — analysis findings
- `run_001\parameters.json` — params used
6. **Controls:** Start, Pause, Skip Hypothesis, View Report buttons
7. **PyInstaller** bundles into single `MT5_Optimizer.exe`
### Tech Stack for Phase 2
```
Flask + Flask-SocketIO → backend API + WebSocket push
Chart.js or Plotly.js → charts in browser
Bootstrap 5 (dark theme) → UI framework
Jinja2 → HTML report templates
PyInstaller → package to .exe
### 1. MT5 Report Encoding — UTF-16 LE
**Problem:** MT5 HTML reports (`.htm`) are UTF-16 LE encoded (BOM: `\xFF\xFE`).
Standard `decode('utf-8')` silently failed, lxml returned a single `<p>` element.
**Fix:** Detect BOM and use `lhtml.document_fromstring(raw_bytes)` directly.
```python
if raw[:2] == b'\xff\xfe':
tree = lhtml.document_fromstring(raw) # handles UTF-16 internally
```
### Files to create in Phase 2
### 2. MT5 Report Location
**Problem:** Runner was looking in `appdata_path/reports/` — folder doesn't exist.
**Reality:** MT5 writes reports to the **root** of `appdata_path\` directly.
```
app.py ← Flask app entry point (replaces main.py)
ui/
templates/
index.html ← Main dashboard
report.html ← Per-run report template
findings.html ← Findings detail page
static/
css/style.css
js/dashboard.js ← WebSocket + Chart.js logic
reports/ ← All run outputs go here (user-visible)
MT5_Optimizer.spec ← PyInstaller spec file
build.bat ← One-click build to .exe
C:\Users\DELL\AppData\Roaming\MetaQuotes\Terminal\D0E8209F...\Optimizer_baseline_*.htm
```
**Fix:** `self.mt5_reports_dir = self.appdata_path` (not `/ "reports"`)
### 3. INI Period Format
**Problem:** Code used `Period=16385` (ENUM integer) — MT5 CLI ignores this.
**Fix:** `Period=H1` (string name). `TIMEFRAME_NAMES = {"H1": "H1", ...}`
### 4. INI Report Path
**Problem:** `Report=C:/absolute/path/...` — MT5 ignores absolute paths in CLI mode.
**Fix:** `Report=Optimizer_{run_id}` (relative name only — MT5 writes to appdata root).
### 5. MQL5 Reference Syntax
**Problem:** `TL_Positions[idx]&.fieldname` — C++ reference syntax, invalid in MQL5.
**Fix:** `TL_Positions[idx].fieldname` — direct array index access.
### 6. Stale Report Detection
**Problem:** Old `Optimizer_*.htm` files from previous runs triggered false-positive detection.
**Fix:** `_clear_stale_reports()` deletes old files before each new run.
---
## 🖥️ How to Run (For Any User)
### Prerequisites
1. MT5 installed at `C:\Program Files\MetaTrader 5\terminal64.exe`
2. LEGSTECH_EA_V2 compiled in MetaEditor (`.ex5` file must exist)
3. Python 3.11 installed at `C:\Users\DELL\AppData\Local\Programs\Python\Python311\`
4. Dependencies installed: `python.exe -m pip install -r requirements.txt`
### Starting the App
1. **Close MT5** if it's open (the optimizer kills and relaunches it automatically from the next run, but first run needs it clean)
2. **Double-click** `Launch Optimizer.bat` on Desktop
3. Browser opens at `http://localhost:5000`
4. Click **▶ Start Optimizer**
5. Watch it run live — MT5 launches automatically, results appear in real-time
6. Reports saved to `MT5_Optimizer\Reports\run_XXX\`
### Save Changes to GitHub
```powershell
cd "C:\Users\DELL\Desktop\MT5_Optimizer"
git add .
git commit -m "describe what changed"
git push
```
---
## 🔧 Environment & Dependencies
## 🔧 Environment Details
**Python version:** 3.11 (NOT 3.13 — use `C:\Users\DELL\AppData\Local\Programs\Python\Python311\python.exe`)
| Item | Value |
|---|---|
| Python | 3.11 — `C:\Users\DELL\AppData\Local\Programs\Python\Python311\python.exe` |
| GitHub | https://github.com/tonnylegacy/MT5_Optimizer |
| MT5 Terminal ID | `D0E8209F77C8CF37AD8BF550E51FF075` |
| MT5 Broker | HFMarketsGlobal-Live3 (Build 5660) |
| Tester Agent | `Agent-127.0.0.1-3000` |
| OS | Windows 11 |
**Install command:**
**Install all dependencies:**
```powershell
C:\Users\DELL\AppData\Local\Programs\Python\Python311\python.exe -m pip install -r requirements.txt
```
**Run tests:**
```powershell
cd C:\Users\DELL\Desktop\MT5_Optimizer
C:\Users\DELL\AppData\Local\Programs\Python\Python311\python.exe -m pytest tests/ -v
**Key dependencies:**
```
**Key dependency versions:**
```
pydantic>=2.5, pandas>=2.1, pyarrow>=14.0, lxml>=4.9
loguru>=0.7, rich>=13.0, scipy>=1.11, pyyaml>=6.0
```
**Additional needed for Phase 2:**
```
flask, flask-socketio, eventlet, jinja2, pyinstaller
flask, flask-socketio, eventlet ← Web dashboard
pydantic>=2.5, pandas>=2.1, pyarrow ← Data layer
lxml>=4.9, beautifulsoup4 ← Report parsing
psutil ← MT5 process control
loguru, rich, scipy, pyyaml ← Utilities
```
---
## 🧠 Key Design Decisions (Important Context)
## 🧠 Key Design Decisions
1. **Hypothesis-driven, not brute-force** — We don't grid-search all params. We detect failure patterns, hypothesize a fix, test it, validate it.
2. **Calmar ratio is primary score metric** — Return / MaxDrawdown is most relevant for live trading.
3. **Three validation gates** — IS threshold → Walk-Forward (2 folds) → OOS (locked period). No candidate is promoted unless it passes all three.
4. **MFE/MAE is critical** — Without the TradeLogger, the analyzer still works but is less powerful. Always confirm CSV is being generated.
5. **OOS is sacred** — `2024.07.01 → 2024.12.31` is NEVER used during optimization. Only tested as final confirmation.
6. **Broker timezone = UTC+2** — All session analysis normalizes to UTC internally. Sessions: London=07-16 UTC, NY=13-22 UTC.
7. **Python 3.11 is required** — The system Python on this machine is 3.13 which doesn't have the packages. Always use the 3.11 path.
8. **MQL5 reference syntax fix** — MQL5 does not support C++ `&` references to array elements. `TradeLogger.mqh` was patched to use direct `TL_Positions[idx].field` access.
1. **Hypothesis-driven, not brute-force** — Detect failure patterns → hypothesize fix → test → validate. Not grid search.
2. **Calmar ratio = primary score** — Return/MaxDrawdown most relevant for live trading.
3. **Three validation gates** — IS → WFV (2 folds) → OOS. No candidate promoted unless all pass.
4. **OOS is sacred** — `2024.07.01 → 2024.12.31` is NEVER used during optimization. Final check only.
5. **MT5 process control** — Always kill existing instance, launch clean, retry once on failure.
6. **UTF-16 LE** — ALL MT5 HTML reports are UTF-16 LE. Always use `document_fromstring(raw_bytes)`.
7. **Report location** — Always `appdata_path\Optimizer_{run_id}.htm` (root, not subfolder).
8. **Python 3.11 required** — System Python on this machine is 3.13 which is missing packages.
9. **Broker TZ = UTC+2** — Session analysis normalizes to UTC. London=07-16, NY=13-22.
---
## 📍 Current Machine Paths (This User's System)
## 📋 Live Dashboard Events (SocketIO)
```
MT5 Terminal EXE: C:\Program Files\MetaTrader 5\terminal64.exe
MT5 Terminal Data: C:\Users\DELL\AppData\Roaming\MetaQuotes\Terminal\D0E8209F77C8CF37AD8BF550E51FF075\
MT5 Tester Files: C:\Users\DELL\AppData\Roaming\MetaQuotes\Tester\D0E8209F77C8CF37AD8BF550E51FF075\Agent-127.0.0.1-3000\MQL5\Files\
EA Source File: C:\Users\DELL\AppData\Roaming\MetaQuotes\Terminal\D0E8209F77C8CF37AD8BF550E51FF075\MQL5\Experts\LEGSTECH_EA_V2.mq5
TradeLogger (deployed): ...Terminal\...\MQL5\Include\TradeLogger.mqh
Project Folder: C:\Users\DELL\Desktop\MT5_Optimizer\
Python 3.11: C:\Users\DELL\AppData\Local\Programs\Python\Python311\python.exe
```
The optimizer emits these events to the browser in real-time:
| Event | Payload | When |
|---|---|---|
| `status_change` | `{state, phase}` | Phase transitions |
| `run_started` | `{run_id, phase, period, params}` | MT5 launches |
| `run_complete` | `{score, calmar, pf, dd, trades, ...}` | Report parsed |
| `run_failed` | `{run_id, error}` | MT5 failed |
| `finding` | `{analyzer, severity, description, confidence}` | Each finding |
| `hypotheses` | `{items: [{id, desc, delta}]}` | Mutations proposed |
| `score_update` | `{iteration, score, calmar, pf}` | Chart data point |
| `candidate_promoted` | `{score, delta, params}` | Validation passed |
| `optimization_complete` | `{candidates, best_score, iterations}` | Loop done |
| `log` | `{level, msg}` | Live log feed |
---
## 🚀 Phase 2 Build Instructions (For Next AI Session)
## 🚀 Next Steps (Phase 3 — Packaging)
When continuing this project, build Phase 2 in this order:
1. **PyInstaller EXE** — Bundle everything into `MT5_Optimizer.exe`
```bat
pyinstaller --onefile --noconsole --name MT5_Optimizer app.py
```
Note: Need `--add-data` flags for templates, static, config, yaml files
### Step 1 — Flask app skeleton
Create `app.py` with:
- Flask app + SocketIO
- Route: `GET /` → serve dashboard
- Route: `GET /api/status` → current run status JSON
- Route: `POST /api/start` → start optimization loop in background thread
- Route: `POST /api/pause` → pause loop
- SocketIO event: emit `run_update` after each test completes
2. **Auto `.set` file export** — When a candidate is promoted, export its params as a valid MT5 `.set` file the trader can load directly into the EA
### Step 2 — Dashboard HTML
Create `ui/templates/index.html`:
- Dark theme (Bootstrap 5 dark)
- Left panel: score history chart (Chart.js line chart)
- Right panel: current run metrics card
- Bottom panel: scrollable findings feed
- Top bar: Start/Pause button, iteration counter, phase badge, timer
3. **Email/Telegram notification** — Alert the trader when a candidate is promoted
### Step 3 — Report template
Create `ui/templates/report.html`:
- Full metrics table
- Findings list in plain English
- Parameter delta table (old → new → why)
- Trade table with MAE/MFE columns
4. **Multi-symbol support** — Run optimization across EURUSD, GBPUSD alongside XAUUSD
### Step 4 — Reports folder writer
Create `reports/writer.py`:
- `write_run_report(run_id, metrics, trades_df, findings)` → writes HTML + CSV
- All outputs go to `MT5_Optimizer\Reports\run_XXX\`
### Step 5 — PyInstaller packaging
Create `build.bat`:
```bat
pyinstaller --onefile --noconsole --name MT5_Optimizer app.py
```
Create `MT5_Optimizer.spec` with proper hidden imports for Flask, SocketIO, etc.
### Step 6 — Test end-to-end
- Double-click `MT5_Optimizer.exe`
- Browser opens at localhost:5000
- Click Start
- Watch live updates
- Check Reports folder
5. **Monte Carlo simulation** — Test robustness of promoted candidates
---
## 💡 Future Enhancements (v2)
## ⚠️ Known Issues & Watch Points
- Multi-symbol optimization (EURUSD, GBPUSD alongside XAUUSD)
- Full rolling window WFV (currently 2-fold MVP)
- Portfolio-level Calmar (across symbols)
- Monte Carlo simulation for robustness testing
- Email/Telegram notification when a candidate is promoted
- Parameter sensitivity heatmap visualization
- Automatic .set file export for validated candidates
1. **First run needs MT5 closed manually** — From the second run onward, the system kills MT5 automatically via psutil.
2. **Tester Agent path** — `Agent-127.0.0.1-3000` may change if MT5 port changes. If TradeLog CSV not found, check Tester folder for new agent subfolder.
3. **WFV fold count** — Currently 2 folds (hardcoded in `validation/gate.py`). Easy to increase.
4. **XAUUSD data** — If the date range 2022-2023 gives errors, ensure XAUUSD H1 history is downloaded in MT5 (Tools → History Center).
5. **Model 4 vs 0** — `tester_model: 4` (OHLC M1) is fast and reliable. `tester_model: 0` (Every Tick) is more accurate but requires full tick data download and takes ~10x longer.
6. **ShutdownTerminal=1** — MT5 closes itself after each test. Cannot be used for live trading simultaneously.
---
## ⚠️ Known Issues / Watch Points
## 📊 Baseline Run Results (Confirmed Working)
1. **MT5 tester model** — Currently config uses `tester_model: 0` (Every Tick). For XAUUSD this is slow. Use model `4` (OHLC M1) for faster iteration during development.
First successful parser test on real MT5 report (2026-04-13):
2. **ShutdownTerminal=1** — The INI closes MT5 after each test. If MT5 is also being used for live trading, this will interrupt it. Separate terminal instances are recommended.
| Metric | Value |
|---|---|
| Net Profit | -$621.15 |
| Profit Factor | 0.95 |
| Max Drawdown | 19.25% |
| Calmar Ratio | -0.32 |
| Sharpe Ratio | -1.60 |
| Total Trades | 597 |
| Win Rate | 77.7% |
| Test Period | 2022.01.01 → 2023.12.31 |
| Model | OHLC M1 (Model 4) |
| Duration | ~56 seconds |
3. **Agent path** — The Tester Agent path (`Agent-127.0.0.1-3000`) may change if the MT5 tester port changes. If CSV is not found, check the Tester folder.
4. **WFV fold count** — Currently hardcoded to 2 folds in `validation/gate.py`. Easy to increase.
5. **INI Period code** — H1 = 16385 in MT5 (ENUM_TIMEFRAMES). This is hardcoded in `config.yaml` as `mt5_period_code: 16385`. Do not change unless changing timeframe.
> This is the baseline — the optimizer's job is to improve these numbers over iterations.
---
*This document was generated by Antigravity AI on 2026-04-13.*
*Continue building from Phase 2 — GUI App.*
*Last updated by Antigravity AI — 2026-04-13*
*GitHub: https://github.com/tonnylegacy/MT5_Optimizer*
*Both Phase 1 and Phase 2 are complete. Continue from Phase 3 (packaging).*