# Backtesting + Optuna + MT5-Verification Stack — Knowledge Base

A blueprint for building a **personal trading-strategy research lab**: a fast Python
backtesting engine, a Bayesian parameter optimizer (Optuna), and an automated bridge to the
**MetaTrader 5 Strategy Tester** that cross-checks every result against the real terminal.

This is a **knowledge base, not a code drop.** It describes the *architecture, algorithms, rules,
and tooling* so you can build your own version from scratch — with **your own** Expert Advisors,
**your own** strategies, and **your own** presets. There are no ready-made engines or strategy
files here on purpose: you supply those from your own MQL5 bots (ideally open-source EAs you own
or have the right to use).

---

## What this stack does for you

You have MQL5 Expert Advisors and a pile of ideas to test. The MetaTrader Strategy Tester is
accurate but **slow** — a single multi-year backtest with a real-tick model can take 10–30 minutes,
and an exhaustive optimization can run for days. That kills iteration speed.

This stack solves it with a **two-tier model**:

1. **Tier 1 — a fast Python "mirror engine"** that reproduces your EA's trade logic bar-by-bar over
   pre-downloaded historical data. It is an *approximation* (more on fidelity below), but it runs a
   full multi-year backtest in **seconds**, so you can rank thousands of parameter combinations with
   Optuna in the time MT5 would run a handful.

2. **Tier 2 — MetaTrader 5 as the gold standard.** Only the **finalists** from the Python search are
   compiled and run in the real Strategy Tester. The Python number gets you the *shortlist*; the MT5
   number is what you trust for anything that goes live.

```
   Idea  ─►  Python mirror engine  ─►  Optuna search (thousands of trials, fast)
                                              │
                                              ▼
                                    2–3 diverse finalists
                                              │
                                              ▼
                           MetaTrader 5 Strategy Tester (gold standard)
                                              │
                                              ▼
                         Python-vs-MT5 comparison table  ─►  keep / discard / iterate
```

The whole point is **disciplined isolation**: the engine is frozen and trusted, instruments are
described by data not code, every hypothesis is an isolated experiment, and only triple-checked
results are promoted. That discipline is what keeps a research lab from rotting into a pile of
one-off scripts that nobody can reproduce.

---

## Who this is for

- You run **MetaTrader 5** and write or use **MQL5** Expert Advisors.
- You want to test and optimize strategies **much faster** than the MT5 optimizer allows.
- You are comfortable with **Python** (intermediate) and the command line.
- You can run **MT5 on Windows** — either on the same machine (simplest) or on a separate
  Windows box/VPS that your Python machine talks to.

You do **not** need to be a quant. The hard parts (engine fidelity, Bayesian search, robustness
testing) are explained from first principles.

---

## How to read this KB

Start with `CLAUDE.md` if you plan to use an AI coding assistant (Claude Code, Cursor, etc.) to
build this with you — it is an **adaptive setup playbook** that profiles *your* machine and OS and
walks the install from zero. Otherwise read the numbered docs in order:

| # | Doc | What you get |
|---|-----|--------------|
| — | [`CLAUDE.md`](CLAUDE.md) | Adaptive AI-assistant playbook: profiles your device, drives install from scratch. Doubles as `AGENTS.md`. |
| 01 | [`01-stack-and-install.md`](01-stack-and-install.md) | The exact tech stack, every library, where to get it, and how to install — per OS. |
| 02 | [`02-architecture.md`](02-architecture.md) | The layered architecture and data flow. The mental model for everything else. |
| 03 | [`03-engine-design.md`](03-engine-design.md) | How to design your own bar-by-bar engine. Grid logic used as the worked example. Fidelity vs MT5. |
| 04 | [`04-isolation-rules.md`](04-isolation-rules.md) | The isolation discipline: frozen engines, forks, separated instruments/strategies, curated registry. |
| 05 | [`05-config-and-inputs.md`](05-config-and-inputs.md) | How test inputs are kept *separate*: instrument config, parameter space, the pre-run wizard, lot/money mode. |
| 06 | [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md) | Optuna objective design, constraints, diverse top-N selection, and anti-overfit robustness layers. |
| 07 | [`07-mt5-bridge.md`](07-mt5-bridge.md) | Connecting to MT5: compiling EAs, auto-running the tester, parsing reports, comparing Python vs MT5. Local-Windows and remote variants. |
| 08 | [`08-workflow-cycle.md`](08-workflow-cycle.md) | The full repeatable cycle: hypothesis → scaffold → stats → optimize → verify → promote. |

---

## The 30-second mental model

- **The engine knows nothing about your strategy.** It takes bars + entry signals + stop/target
  prices and simulates fills. All strategy math lives in *caller* code. (Doc 02–03.)
- **The engine is frozen.** You never edit a validated engine to test an idea — you fork it,
  prove the fork reproduces the original 1:1 with the change off, then test. (Doc 04.)
- **Instruments are data, not code branches.** Tick value, spread model, swap, lot steps — all in a
  per-symbol config object. The engine reads everything from it. (Doc 05.)
- **Search inputs are declared, not scattered.** Every tunable parameter, its range, and its
  constraints live in one declared search space; one wizard captures the run settings; one YAML
  records the answers so any run is reproducible. (Doc 05–06.)
- **Python ranks, MT5 decides.** Fast Python search produces a shortlist; MT5 produces the trusted
  number; a comparison table is saved for every finalist. (Doc 03, 06, 07.)

---

## What you must bring yourself

This KB is deliberately empty of trading IP. To build a working lab you supply:

- **Your MQL5 EA(s)** — compiled `.ex5` plus source `.mq5`, and any custom indicators they call.
- **Your strategy logic** — encoded once in Python (the caller) so the mirror engine can run it.
- **Your presets** — the `.set` files / input templates you want to test and optimize.
- **Historical data** — downloaded from your broker via MT5 (the stack includes a recipe).
- **A broker demo account** — for the MT5 Strategy Tester runs.

Everything else — the architecture, the optimizer, the MT5 bridge, the robustness checks, and the
rules that hold it together — is described in the docs above.

---

*This KB is brand-free and self-contained. Drop the folder into a Git repository, open it with your
AI coding assistant, and build your own lab.*