# CLAUDE.md — Adaptive Setup Playbook for an AI Coding Assistant > This file is written **for an AI coding assistant** (Claude Code, Cursor, Copilot Chat, etc.) so > it can help its human build this backtesting lab **from scratch on their specific machine**. > It also works as a generic `AGENTS.md`. If you are a human, you can follow it too — just perform > the steps yourself instead of asking the assistant. > > **Golden rule for the assistant:** this stack is reproduced *from your human's own EAs and > presets*, not from any bundled code. Never invent a trading strategy or fabricate parameters. > When trading logic is needed, ask the human for their MQL5 source and presets. --- ## Phase 0 — Profile the device (do this first, always) Before installing anything, detect the environment. The whole setup branches on **where MT5 runs**. Run and read the output: ```bash # OS + architecture uname -a 2>/dev/null || ver # macOS/Linux print uname; Windows cmd prints `ver` python3 --version || python --version ``` Then classify the human into **one of three topologies** and confirm it with them before proceeding: | Topology | Python runs on | MT5 runs on | Bridge needed | Best for | |----------|----------------|-------------|---------------|----------| | **A — All-Windows (simplest)** | the Windows machine | the same Windows machine | none — direct, via the `MetaTrader5` pip package or local file paths | a Windows user; recommended starting point | | **B — Unix + remote Windows** | macOS/Linux | a separate Windows box or VPS | SSH + file copy + a scheduled-task trick | someone on a Mac/Linux who already has a Windows VPS | | **C — Unix + local Windows VM** | macOS/Linux host | Windows in a VM/Parallels/UTM | shared folder or SSH into the VM | a Mac/Linux user without a separate Windows box | > **Recommendation to give the human:** if they are on Windows, use **Topology A** — it is by far > the least moving parts. The official `MetaTrader5` Python package only runs on Windows and lets > Python drive the terminal directly, so no SSH/scheduled-task plumbing is needed. Topologies B/C > exist only because some people develop on a Mac/Linux and keep MT5 elsewhere. Record the chosen topology — every later step references it. --- ## Phase 1 — Install the Python stack Follow [`01-stack-and-install.md`](01-stack-and-install.md). In short: 1. Install **Python 3.12+** (3.13/3.14 fine). On Windows, install from python.org and tick *"Add Python to PATH"*. 2. Create an **isolated virtual environment** in the project root (`.venv`). Never install into the system Python. 3. Install the libraries (`pandas`, `numpy`, `pyarrow`, `optuna`, `numba`, `sqlalchemy`, `pyyaml`, `lxml`, `html5lib`, `tqdm`, `requests`). Exact commands and *why each library exists* are in doc 01. 4. On **Topology A** only, also `pip install MetaTrader5` (Windows-only package). Verify the venv imports everything before moving on. --- ## Phase 2 — Scaffold the repository skeleton Create the directory layout from [`02-architecture.md`](02-architecture.md). Build **empty modules with docstrings and type signatures first** — do not write trading logic yet. The skeleton: ``` your-lab/ ├── .venv/ # virtual environment (gitignored) ├── data/ # downloaded market data, parquet (gitignored) ├── shared/ # reusable infrastructure │ ├── core/ # backtest engines (you fill these from your EA logic) │ ├── indicators/ # indicator functions (RSI, ATR, EMA, your custom ones) │ ├── instruments/ # per-symbol/broker config objects │ ├── data/ # data loaders + MT5 report parser │ ├── optimizer/ # objective + diverse top-N selector │ ├── robustness/ # anti-overfit layers │ ├── gates/ # reusable entry-filter masks │ ├── wizard/ # pre-run interactive Q&A │ └── mt5_pipeline/ # the MT5 bridge ├── strategies/ # one folder per strategy (your EAs + presets + iterations) ├── registry/ # approved, locked results — the source of truth └── docs/ # this KB + your own notes ``` Mirror the responsibilities described in doc 02. Leave the engines for Phase 4. --- ## Phase 3 — Bring in the human's trading assets **Stop and ask the human for:** 1. The **MQL5 EA** they want to test — `.mq5` source *and* compiled `.ex5`, plus any custom indicators (`.ex5`) the EA loads. 2. A **representative `.set` preset** (or a screenshot of the EA inputs) so you can see exactly which inputs exist and their default values. 3. Their **broker + symbol(s)** and a **demo account** for the tester. From the EA source and preset you (assistant) will: - Enumerate every EA input and group them into **frozen** (not optimized) vs **tunable** (search space). - Identify the instrument mechanics (digits, point, contract size, tick value, lot step) to build the instrument config in Phase 5. These come from the **symbol specification** in MT5 (`Right-click symbol → Specification`), not from guesses. --- ## Phase 4 — Build the Python mirror of the EA (the engine) Read [`03-engine-design.md`](03-engine-design.md) carefully — it is the hardest and most important part. The engine must reproduce the EA's **fill and exit logic** bar-by-bar. - Keep the engine **strategy-agnostic**: it consumes bars + signal arrays + stop/target arrays and simulates fills. All the *strategy* math (when to enter, where to put stops) lives in the caller. - Implement the **intra-bar sub-tick model** (doc 03) and the **pessimistic ordering** convention. This is the default bar-level exit mode — fast and exact for clean-directional setups. - **If the EA moves its SL during a trade** (break-even, trailing, basket trailing), the bar-level engine is NOT trustworthy: it produces a −40% to −50% net gap vs MT5 even in a calm window (doc 03 §7 measured failure mode). You MUST implement the **M1 tick-level exit simulation** path: load M1 bars covering the same window as the signal bars, pass them to `engine.run(..., m1_bars=m1_bars)`, and the engine walks 4 synthetic ticks per M1 bar inside each higher-TF bar (direction-aware order), separating the BE-update tick from the SL-trigger tick. This brings the gap to ~−5% net. The engine should support BOTH modes and switch on whether `m1_bars` is provided. - Match the EA's **lot/money mode**, **spread model**, and **swap model** exactly (doc 05). - The first milestone is **1:1 fidelity on one known preset**: run the EA in MT5 on a short period, run your Python engine on the same data/preset, and reconcile trade-by-trade until the numbers line up within the **target gate for the EA's class** (doc 03 §8 table): - Clean-directional (SL not moved intra-trade), bar-level: ≤ ~2% net gap. - BE / trailing, **bar-level**: unattainable — do not chase this, switch to M1 tick-level. - BE / trailing, **M1 tick-level**: ≤ ~10% net gap (residual spread/tick-path differences). **Do not optimize anything until this passes** — an unvalidated engine optimizes noise. A trailing/BE EA validated only on the bar-level engine is a silently-broken engine. > The bundled docs use a **grid martingale** engine as the worked example because it exercises every > hard case (pending orders, averaging, trailing on the basket, simultaneous closes). Your EA may be > simpler (single position, signal-to-signal) or different — apply the *principles*, not the grid > specifics. --- ## Phase 5 — Wire up configuration and inputs Read [`05-config-and-inputs.md`](05-config-and-inputs.md). Build: - An **`InstrumentConfig`** object per symbol/broker (real conditions), plus optional `worst_case` and `best_case` variants for cost-stress testing. - A **declared search space** for the strategy (parameter → range → step), kept in code *and* documented in a `parameter-space.md` next to the strategy. - A **wizard** that captures run settings (period, profile, trial count, DD cap, deposit) and writes them to `wizard-answers.yaml` for reproducibility. --- ## Phase 6 — Optimizer + robustness Read [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md). Build the Optuna objective (score + hard constraints), the diverse top-N selector, and at least the first robustness checks (neighborhood/plateau, cost stress, era split). Start with a **small smoke run** (30–40 trials, short period) to validate the script, then one full run in the background. --- ## Phase 7 — The MT5 bridge Read [`07-mt5-bridge.md`](07-mt5-bridge.md) and implement the variant for the chosen topology: - **Topology A:** use the `MetaTrader5` package or generate a `tester.ini` + `.set` and launch `terminal64.exe /config:` locally. Simplest path — no SSH. - **Topology B/C:** generate `tester.ini` + `.set`, copy to the Windows box, trigger the run via a one-shot scheduled task (so it survives the SSH disconnect), poll for the report, copy it back. Then implement the **report parser** (MT5 saves HTML as UTF-16) and the **comparison table** writer. --- ## Phase 8 — Run the full cycle once, end to end Read [`08-workflow-cycle.md`](08-workflow-cycle.md) and run a complete loop on the human's real EA: hypothesis → scaffold iteration → stats → minimal-scope test → optimize → verify the finalists in MT5 → write the comparison → let the human decide. When the loop produces a number the human trusts, the lab is working. --- ## Standing rules for the assistant - **Never touch a validated engine to test an idea.** Fork it; prove the fork == original with the change disabled; only then test. (Doc 04.) - **For trailing/BE EAs, M1 tick-level exit simulation is mandatory.** A trailing/BE EA validated only on the bar-level engine has a −40% to −50% hidden gap vs MT5 — it is *not* a validated engine, no matter how good the numbers look. Always pass `m1_bars=` to the engine for EAs that move their SL intra-trade. (Doc 03 §7/§8.) - **Heavy runs go in the background.** A full-history A/B or a full Optuna study is minutes of compute — start it detached and poll, never block. Smoke-test first. (Doc 06.) - **Don't promote partial searches.** A finalist must come from a *completed* search. An interrupted Optuna study yields preliminary numbers, not verification candidates. (Doc 06, 08.) - **MT5 is the source of truth for live numbers.** Python is for ranking and A/B. Always verify finalists in MT5 before the human relies on a result. (Doc 03, 07.) - **Keep secrets out of code and chat.** Broker login goes in a gitignored env file, read at runtime, never printed. (Doc 07.) - **Ask before destructive actions** — deleting data, force-pushing, overwriting a validated preset.