commit 49be922517fc3cb44f31581a41819afa57f7f345 Author: gavindiaz Date: Fri Jun 26 18:47:35 2026 +0800 first commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dadca99 --- /dev/null +++ b/.gitignore @@ -0,0 +1,20 @@ +# Python +.venv/ +__pycache__/ +*.pyc + +# Market data — large & re-downloadable +data/ + +# Run outputs / scratch +results/ +*.db # Optuna SQLite studies +*.htm # pulled MT5 reports + +# Secrets — NEVER commit broker credentials +.env +*.env + +# OS +.DS_Store +Thumbs.db diff --git a/01-stack-and-install.md b/01-stack-and-install.md new file mode 100644 index 0000000..f3b447a --- /dev/null +++ b/01-stack-and-install.md @@ -0,0 +1,180 @@ +# 01 — Tech Stack & Installation From Scratch + +Everything you need to install, what each piece is for, where to get it, and the exact commands per +operating system. Nothing here is strategy-specific — this is the plumbing. + +--- + +## 1. The stack at a glance + +| Layer | Tool | Role | Where to get it | +|-------|------|------|-----------------| +| Language | **Python 3.12+** (3.13/3.14 work) | everything outside MT5 | | +| Data frames | **pandas** | OHLC tables, resampling, equity curves | · `pip install pandas` | +| Numerics | **numpy** | vectorized price/indicator math | · `pip install numpy` | +| Columnar storage | **pyarrow** | read/write **Parquet** market data (fast, compact) | · `pip install pyarrow` | +| JIT speed | **numba** | compiles the hot bar-by-bar loop to machine code | · `pip install numba` | +| Optimizer | **Optuna** | Bayesian hyper-parameter search | · `pip install optuna` | +| Optuna storage | **SQLAlchemy** (+ **alembic**) | persists studies to a SQLite DB so runs resume/parallelize | · `pip install sqlalchemy` | +| Config | **PyYAML** | reproducible run settings (`wizard-answers.yaml`) | · `pip install pyyaml` | +| HTML parsing | **lxml** + **html5lib** | parse the MT5 Strategy Tester HTML report | · `pip install lxml html5lib` | +| Progress / logs | **tqdm**, **colorlog** | progress bars, readable logs | `pip install tqdm colorlog` | +| HTTP | **requests** | optional data downloads / webhooks | · `pip install requests` | +| Tester | **MetaTrader 5** terminal | the gold-standard backtester + data source | from your broker, or | +| MT5 control (Windows) | **MetaTrader5** pip package | drive the terminal & pull data from Python (Windows only) | · `pip install MetaTrader5` | + +> **Optional data source.** If you want history without MT5 export, `dukascopy-python` +> (`pip install dukascopy-python`, ) pulls free tick/bar +> data for many symbols. MT5-exported data from your own broker is preferred because it matches the +> tester exactly. + +### Reference versions + +A known-good combination (early 2026) — use these or the latest stable: + +``` +python 3.14 +pandas 3.0 +numpy 2.4 +pyarrow 24.0 +numba 0.65 +optuna 4.8 +sqlalchemy 2.0 +pyyaml 6.0 +lxml 6.1 +``` + +Pin exact versions in a `requirements.txt` once your lab works, so it reproduces later. + +--- + +## 2. What each library actually does here + +- **pandas / numpy** — market data is loaded into a DataFrame of `[timestamp, open, high, low, close, + spread]`. Indicators and signals are computed as numpy arrays. Equity curves are pandas frames. +- **pyarrow + Parquet** — a few years of M1 (one-minute) bars is millions of rows. Parquet stores it + columnar and compressed: a multi-million-row file loads in well under a second and is a fraction of + CSV size. This is what makes "full-history backtest in seconds" possible. +- **numba** — the engine's inner loop walks every bar (and sub-ticks within each bar). Pure-Python + that is too slow. `@njit` compiles it to native code on first call; subsequent runs are C-fast. +- **Optuna** — instead of brute-forcing a parameter grid, Optuna uses a Bayesian sampler (TPE) that + *learns* which regions of the search space are promising and concentrates trials there. You get a + good optimum in hundreds–thousands of trials instead of an exhaustive grid of millions. +- **SQLAlchemy/SQLite** — Optuna writes each trial to a `study.db`. That means a study can be stopped + and resumed, inspected mid-run (count completed trials), and run with multiple worker processes + pointing at the same DB. +- **lxml / html5lib** — the MT5 tester exports its report as an HTML file encoded **UTF-16-LE**. These + parse it into a metrics dict (Net Profit, Profit Factor, Drawdown, trade count, …). +- **MetaTrader5 package** — on Windows, this is the clean way to (a) download historical bars and + (b) launch/script the terminal. On non-Windows you don't have it, which is why remote topologies use + SSH + a scheduled task instead. + +--- + +## 3. Install — step by step + +### 3.0 Prerequisites + +- **Python 3.12+**. Check with `python3 --version` (macOS/Linux) or `python --version` (Windows). +- **Git** (to version your lab). +- **MetaTrader 5** installed from your broker, with a **demo account** logged in. + +### 3.1 Windows (Topology A — recommended) + +```powershell +# From the project root, in PowerShell or cmd: + +# 1. Create the virtual environment +python -m venv .venv + +# 2. Activate it +.\.venv\Scripts\activate + +# 3. Upgrade pip +python -m pip install --upgrade pip + +# 4. Install the stack (MetaTrader5 included — Windows only) +pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests MetaTrader5 +``` + +### 3.2 macOS / Linux (Topology B/C — MT5 lives elsewhere) + +```bash +# 1. Create the venv +python3 -m venv .venv + +# 2. Activate +source .venv/bin/activate + +# 3. Upgrade pip +python3 -m pip install --upgrade pip + +# 4. Install the stack (NO MetaTrader5 package — it is Windows-only) +pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests +``` + +> On Apple Silicon (M-series) everything above is native arm64 and fast. `numba`/`numpy` ship arm64 +> wheels — no Rosetta needed. + +### 3.3 Verify the install + +```bash +# Use the venv's python explicitly to avoid the system interpreter +.venv/bin/python3 -c "import pandas, numpy, pyarrow, optuna, numba, yaml, lxml; print('core OK')" +# Windows: .\.venv\Scripts\python -c "..." +``` + +On Windows also verify the terminal link: + +```python +import MetaTrader5 as mt5 +print(mt5.initialize()) # True if it found & launched the terminal +print(mt5.version()) +mt5.shutdown() +``` + +--- + +## 4. Always use the venv interpreter + +A recurring source of bugs is accidentally running the **system** Python (which lacks the libraries). +Make it a habit to call the venv interpreter by path: + +```bash +# macOS/Linux +.venv/bin/python3 your_script.py + +# Windows +.\.venv\Scripts\python your_script.py +``` + +…or activate the venv at the start of every session. Pick one convention and keep it. + +--- + +## 5. `.gitignore` essentials + +Your lab will accumulate large data and secrets. Ignore them from day one: + +```gitignore +.venv/ +data/ # market data is large & re-downloadable +results/ # scratch run outputs +*.db # Optuna SQLite studies +*.htm # pulled MT5 reports +.env # broker credentials — NEVER commit +__pycache__/ +.DS_Store +``` + +--- + +## 6. Hardware notes + +- Backtesting is **CPU + RAM** bound, not GPU. A modern multi-core CPU and 16 GB+ RAM is plenty. +- Optuna parallelizes across CPU cores. The practical cap for *heavy* concurrent backtests is roughly + your number of **performance cores** — beyond that they contend and slow each other down. Prefer + **one** Optuna study with `n_jobs=N` over N separate scripts fighting for cores. +- Millions of M1 bars fit comfortably in RAM as a pandas frame; loading from Parquet is the only I/O. + +Next: [`02-architecture.md`](02-architecture.md) — the layered architecture you are about to build. diff --git a/02-architecture.md b/02-architecture.md new file mode 100644 index 0000000..e9b220e --- /dev/null +++ b/02-architecture.md @@ -0,0 +1,165 @@ +# 02 — Architecture & Data Flow + +The whole lab is a set of **layers with one-directional dependencies**. Higher layers call lower +ones; lower layers never know about higher ones. This is what lets you freeze the engine, swap +instruments, and add strategies without anything leaking sideways. + +--- + +## 1. The layer stack + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ STRATEGY / CALLER one folder per strategy │ +│ loads data, computes signals + stops, calls the engine, scores │ +└───────────────┬──────────────────────────────┬──────────────────────┘ + │ │ + ┌───────▼────────┐ ┌─────────▼──────────┐ + │ OPTIMIZER │ │ MT5 BRIDGE │ + │ objective, │ │ compile, run, │ + │ Optuna study, │ │ parse, compare │ + │ diverse top-N │ └─────────┬──────────┘ + └───────┬─────────┘ │ + │ │ (gold-standard verify) + ┌───────▼─────────┐ │ + │ ROBUSTNESS │ anti-overfit checks over results (read-only) + └───────┬─────────┘ + │ + ┌───────▼─────────────────────────────────────────────┐ + │ ENGINE (frozen) bar-by-bar fill simulator │ + │ knows: bars, signals, stop/target prices, instrument │ + │ knows NOT: your strategy, indicators, broker │ + └───┬─────────────┬────────────────┬───────────────────┘ + │ │ │ + ┌────────▼───┐ ┌──────▼──────┐ ┌──────▼───────┐ ┌──────────────┐ + │ INDICATORS │ │ INSTRUMENTS │ │ GATES │ │ DATA │ + │ RSI/ATR/.. │ │ per-symbol │ │ entry-filter │ │ loaders + │ + │ pure fns │ │ config объ. │ │ masks │ │ MT5 parser │ + └────────────┘ └─────────────┘ └──────────────┘ └──────────────┘ +``` + +### Layer responsibilities + +| Layer | Folder | Owns | Must NOT | +|-------|--------|------|----------| +| **Data** | `shared/data/` | load bars from Parquet; fetch from MT5; parse MT5 HTML reports | contain strategy logic | +| **Instruments** | `shared/instruments/` | per-symbol/broker config objects (tick value, spread, swap, lot steps) | know any strategy | +| **Indicators** | `shared/indicators/` | pure functions: RSI, ATR, EMA, SMA, your custom range filter, etc. | hold state across calls | +| **Gates** | `shared/gates/` | reusable boolean masks that *filter* entries (regime, time-of-day, exhaustion) | open/close trades | +| **Engine** | `shared/core/` | the bar-by-bar fill/exit simulator; **frozen** once validated | compute signals or stops | +| **Robustness** | `shared/robustness/` | read-only anti-overfit analyses over a result | mutate the engine or results | +| **Optimizer** | `shared/optimizer/` | objective fn, Optuna wiring, diverse top-N selection | know broker specifics | +| **MT5 bridge** | `shared/mt5_pipeline/` | generate `.set`/`.ini`, run the tester, pull & parse the report, compare | compute strategy | +| **Strategy / caller** | `strategies//` | glue: data → signals → stops → engine → metrics; the only layer that knows the *whole* picture | be reused by another strategy (copy, don't import) | +| **Registry** | `registry/` | approved, locked results — the source of truth | be edited casually | + +--- + +## 2. The core contract: engine knows nothing about strategy + +This is the single most important design decision. The engine's input is **pre-computed**: + +``` +engine.run( + bars, # DataFrame [timestamp, open, high, low, close, spread] + signals_long, # boolean array — True on the bar an entry is triggered + signals_short, # boolean array + sl_prices, # array — the stop price for an entry on that bar (NaN if none) + tp_prices, # array — the target price + instrument, # InstrumentConfig — all symbol mechanics + lot / money_mode, # position sizing inputs + initial_deposit, +) -> Result +``` + +> **Crucial:** the stop and target arrive as **finished prices** computed by the caller. The engine +> never decides *where* a stop goes — only *whether* price touched it. This is the seam that +> separates "the strategy" from "the simulator". Change your strategy → you change the caller and the +> arrays you hand in; the engine is untouched. + +The engine's output is equally generic: + +``` +Result: + trades # list of closed trades (entry/exit time, price, lots, pnl, swap) + equity_curve # DataFrame [timestamp, balance, equity], sampled (e.g. hourly) + final_balance + initial_deposit +``` + +A separate `compute_metrics(result)` turns that into the numbers you optimize on: Net Profit, Profit +Factor, Win Rate, max Balance/Equity Drawdown, Sharpe, APR, trade count. + +--- + +## 3. Data flow of one backtest + +``` +data//_M1_.parquet + │ load_bars() → DataFrame with per-bar spread column + ▼ +caller: resample M1 → the signal timeframe (e.g. H1/H4/D1) + │ indicators.* on the resampled frame + ▼ +caller: edge-detect signals (True only on the bar the condition first flips, not every bar after) + │ + optional gates (regime/time filters AND-ed into the signal) + ▼ +caller: compute SL/TP price arrays from params (ATR stop, % stop, indicator band, …) + ▼ +engine.run(bars, signals, sl/tp, instrument, sizing, deposit) + │ + ▼ +Result → compute_metrics → dict + │ + ▼ (finalists only) +MT5 bridge: build .set from the same params → run real tester → parse report → compare +``` + +Two subtleties that cause most bugs if missed (both explained in doc 03): + +- **Edge detection.** Signals must be `True` only on the *transition* bar, not forward-filled, or the + engine re-enters every bar. +- **Timeframe alignment.** Indicators computed on a higher timeframe must be mapped back onto the M1 + bars correctly (no look-ahead — a daily value is only known after that day closes). + +--- + +## 4. Why "shared" vs "strategies" + +- **`shared/`** is infrastructure you write **once** and treat as a library: engine, indicators, + instruments, optimizer, robustness, bridge. It is import-stable. Breaking changes here ripple + everywhere, so it changes rarely and deliberately. +- **`strategies//`** is where experimentation happens. Each strategy is **self-contained**: its + EA source, its presets, and its `iterations/` (each iteration = one research attempt with its own + `optimize.py`, search space, results, and verification). Strategies **copy** glue code between + iterations rather than importing it, so an old iteration always reproduces even after you change + how you do things. + +``` +strategies// +├── source/ # your EA .mq5/.ex5 + custom indicators (read-only after compile) +├── presets/ # canonical baseline .set files (frozen reference) +├── production/ # live-ready presets +├── iterations/--/ +│ ├── README.md # context, status, result summary +│ ├── parameter-space.md # the search space for this attempt +│ ├── optimize.py # the run script (a snapshot — self-contained) +│ ├── wizard-answers.yaml # the run settings (reproducibility) +│ ├── top-1/ top-2/ # the diverse finalists: params, preset, py+mt5 metrics +│ └── auto-verification.md# Python-vs-MT5 comparison table +└── archive/ # discarded attempts +``` + +--- + +## 5. The registry is the source of truth + +Nothing is "real" until it is in `registry/`. A registry entry is a result that passed **all three** +gates: Python search → MT5 verification → human approval. Once written it is **locked** — you never +silently edit a registry entry; you create a *new* one. This is what stops a research lab from +drifting: the registry is the curated, trustworthy subset of everything you ever tried. + +Doc 04 covers the isolation rules that keep these boundaries honest. Doc 03 is next — how to actually +build the engine. + +Next: [`03-engine-design.md`](03-engine-design.md). diff --git a/03-engine-design.md b/03-engine-design.md new file mode 100644 index 0000000..30a907f --- /dev/null +++ b/03-engine-design.md @@ -0,0 +1,249 @@ +# 03 — Designing Your Own Bar-by-Bar Engine + +This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully +enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the +design principles** — you write the code from your own EA's logic. The worked example is a **grid +martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does. + +--- + +## 1. What an engine is (and is not) + +An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each +bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told +*when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the +strategy that produced those. + +- **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics, + and the position-sizing rule. +- **It does not know:** your indicators, your regime filters, your broker, why a signal fired. + +Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04). + +--- + +## 2. The intra-bar problem (the heart of fidelity) + +Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute +the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated +tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only +four anchor points. **How you order those four points inside a bar decides your fills.** + +### The 4-sub-tick model + +Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption: +the price visits the point that hurts an open position **before** the point that helps it. + +``` +For a LONG position (stop below, target above): OPEN → LOW → HIGH → CLOSE +For a SHORT position (stop above, target below): OPEN → HIGH → LOW → CLOSE +``` + +Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order +makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before +HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your +results honest (an optimistic engine "discovers" edges that don't survive live). + +At each sub-tick the price is a single number, and you run the same checks: + +``` +for each bar: + for each sub_tick in pessimistic_order(position_direction): + price = sub_tick_price # one of O / H / L / C + 1. if a position is open: + - apply daily swap if the calendar day rolled over + - check stop hit (use BID/ASK appropriately; add spread) + - check target hit + - if both could hit this bar → pessimistic order already decided the stop + 2. if flat and an entry signal is active and a next bar exists: + - open the position at the next bar's open (avoid look-ahead) + 3. sample the equity curve periodically (e.g. once per hour) +``` + +> **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open, +> never the same bar's close. Otherwise you are trading on information you wouldn't have had. + +--- + +## 3. Worked example: a grid martingale engine + +A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple +simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you +can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose. + +### 3.1 Order types you must model + +| Type | Fires when | +|------|-----------| +| Market | immediately at the current ask/bid | +| Buy-Stop | ask **rises to** the level (breakout up) | +| Buy-Limit | ask **falls to** the level (pullback down) | +| (mirror for sell side) | + +### 3.2 Series lifecycle (one "basket" from open to close) + +``` +1. FLAT. No positions, no pending orders. + └─ after a cooldown since the previous series closed … + +2. PLACE FIRST PENDING. + base_price = close price of the previous series (or current price on first run) + pending = Buy-Stop at base + open_distance% # wait for momentum confirmation + +3. PENDING FILLS → position #1. + first_open = fill price + first_lot = sized by the money/lot rule (section 4) + count = 1 + +4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale). + next_level_price = last_fill_price − first_open × (grid_step% / 100) + next_lot = previous_lot × grid_multiplier (clamped to the volume minimum) + add positions up to grid_count at step/mult set #1, then switch to set #2, etc. + (cap the total number of levels — deep martingales are where accounts die) + +5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5). + compute the basket's average price across all open positions + once profit clears (break_even + trailing_stop) → move a single basket stop up + +6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick. + record the series as one closed event (sum of pnl + accumulated swap) + base_price = the close price (seed for the next series) + reset to FLAT +``` + +### 3.3 Adaptive grid levels + +A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's +price, so the spacing stays constant in points while the trigger range tracks slippage: + +``` +next_level = last_fill_price − first_open_price × (grid_step% / 100) +``` + +This is one line, but it is the difference between a grid that drifts and one that holds its spacing. + +--- + +## 4. Position sizing — the money/lot modes + +Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority +order: + +1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`. +2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`). +3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is + deployed regardless of price. + +After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious +pitfalls (doc 05 covers them in full): + +- A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure + the fixed-lot input is zero when you intend money mode. +- For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the + EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot + against MT5 before trusting a whole run. + +--- + +## 5. Basket exits — break-even and trailing + +For a multi-position basket, the stop is computed on the **average price**, then pushed to every +position so they close together: + +``` +avg = average open price of all open positions, weighted by lot + +# break-even: once the basket is in profit by `break_even%`, lock the stop at avg +if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100: + basket_stop = avg + +# trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price +if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100: + candidate = bid − avg × trailing% / 100 + if candidate > basket_stop: # SMOOTH: move up by any improvement + basket_stop = candidate # (STEP variant: only move in trailing-sized jumps) +``` + +When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes. +Single-position strategies are just the `count == 1` case of this. + +--- + +## 6. Spread and swap — small models, large effects + +- **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread). + Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where + recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid. +- **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap + weekday (commonly Wednesday for many CFDs). Two models: + - *fixed per lot per day:* `swap = rate × lot × multiplier` + - *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier` + Accumulate per position and add it to the trade's PnL on close. + +These look minor but compound over a multi-year, many-position backtest into hundreds of currency +units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never +hard-code. + +--- + +## 7. Fidelity — how much to trust the Python number + +Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread, +same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are +predictable, and knowing them is what makes the lab trustworthy. + +### What matches MT5 well +- Bar data, per-bar spread, swap (with the triple-swap day). +- Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are + tick-invariant, so they're identical. +- Fill **prices** at pending/grid levels (the level price is the fill price). +- Direction, lot sizing, money management. + +### What diverges +Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs +target is touched, exact fill timing on volatile bars. The divergence **scales with +path-sensitivity × volatility**: + +| Strategy character | Typical Python vs MT5 gap | +|--------------------|---------------------------| +| Clean directional, few exits | small and consistent: Python reads somewhat higher | +| Tight trailing / martingale grid in calm years | small | +| Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves | + +A concrete illustration: a trailing strategy might match MT5 within a few currency units in calm, +choppy years, yet the Python figure can be several times the MT5 figure across a violent crash year — +because MT5's finer path triggers exits at prices the 4-point model skips. The bias is almost always +**Python optimistic**, and almost always concentrated in the most volatile episodes. + +### The practical policy (this is the whole point of the two-tier design) +1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all + the optimizer needs. +2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per + bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism. +3. **MT5-verify every finalist** — the MT5 number is the one you act on. +4. **Use real ticks where available** (recent history) for the truest check, accepting the limited + window. +5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python), + and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a + known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*, + assume MT5 will erase it. + +--- + +## 8. Validating a new engine (do this before any optimization) + +1. Pick **one known preset** of your EA and a short period (a few months). +2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report. +3. Run your Python engine on the **same data, same preset**. +4. Reconcile **trade by trade**, then in aggregate. Target gates for a clean-directional setup: + - Net difference ≤ ~2%, trade-count difference ≤ ~5%, Profit Factor essentially identical, + equity-drawdown difference ≤ ~3%. +5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the + engine's **baseline fidelity document** and freeze the engine (doc 04). + +> If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe +> mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, or sub-tick +> ordering. Walk those five before suspecting anything exotic. + +Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine +validated. diff --git a/04-isolation-rules.md b/04-isolation-rules.md new file mode 100644 index 0000000..2682a4a --- /dev/null +++ b/04-isolation-rules.md @@ -0,0 +1,143 @@ +# 04 — Isolation Rules + +The rules that keep a research lab trustworthy as it grows. Every one exists because the alternative +quietly corrupts results. Treat them as non-negotiable; they cost a little friction and save you from +optimizing on a silently-broken engine or shipping a number you can't reproduce. + +--- + +## Rule 1 — The validated engine is FROZEN + +Once an engine reproduces your EA within the expected fidelity gap (doc 03 §8), it becomes the +**satisfactory baseline** and is frozen. Frozen means: **you do not edit it to test an idea.** + +Why so strict: the engine's validation was expensive (trade-by-trade reconciliation against MT5). +A one-line "quick tweak" to test a hypothesis can silently shift fills across millions of bars, and +now every result the engine produces is suspect — including the ones you already trusted. Freezing +protects the validation from regressions you won't notice until much later. + +--- + +## Rule 2 — Engine changes happen in an ISOLATED FORK + +Any hypothesis that needs the engine to behave differently is implemented as a **separate engine +file** — a copy of the frozen engine plus the minimal hook for the experimental change. Never by +editing the frozen file. + +``` +shared/core/grid_engine.py ← frozen, never touched +shared/core/grid_engine_.py ← fork: copy + one experimental hook +``` + +The process is fixed: + +1. **Fork** = exact copy of the frozen engine + the experimental change, **defaulting to OFF**. +2. **Regression-verify:** run the fork with the change *disabled* and confirm it reproduces the frozen + engine **1:1** — identical Net, drawdown, and trade count — on identical data. If it doesn't, your + copy isn't clean; fix that before testing anything. +3. **A/B** the hypothesis: fork-with-change vs frozen-baseline on the same data. +4. **Decide:** significant, robust improvement → consider promotion (Rule 3). Otherwise **delete the + fork** — revert is just removing a file, the core was never touched. + +The regression-verify in step 2 is the linchpin. It proves your fork differs from the baseline *only* +in the one thing you're testing, so the A/B measures the hypothesis and nothing else. + +--- + +## Rule 3 — Promoting a fork to the new baseline is a deliberate, rare event + +A fork stays a fork until it has passed a **full round**: A/B + optimization + MT5 verification + an +explicit human decision. Only then does it replace the frozen engine as the new baseline — and at that +moment **every preset/EA built on that engine must be re-validated**, because the ground shifted. + +This is never automatic. No script and no agent swaps the baseline engine on its own. It is a manual +procedure under human supervision, precisely because it invalidates prior numbers and forces +re-verification. + +--- + +## Rule 4 — Instruments are isolated as data, not code + +Everything symbol- or broker-specific lives in an `InstrumentConfig` object (doc 05), never in the +engine and never hard-coded in a strategy. The engine pulls tick value, spread model, swap, and lot +steps *from the config*. + +Consequences: +- Testing the same strategy on a different symbol = a different config, **zero engine changes**. +- Cost-stress testing = `real` / `worst_case` / `best_case` variants of the same config. +- A broker change = edit one file. + +If you ever find yourself writing `if symbol == ...` in the engine, that logic belongs in the config. + +--- + +## Rule 5 — Strategies are isolated folders; iterations copy, not import + +Each strategy is a self-contained folder (doc 02 §4). Within it, each **iteration** (one research +attempt) carries its **own snapshot** of the run script, search space, and answers. + +- Iterations **copy** glue code rather than importing a shared "current" version. That way an + iteration from months ago still runs and reproduces exactly, even after you've changed how you do + things. Reproducibility beats DRY here — these are lab notebooks, not production code. +- Shared *infrastructure* (engine, indicators, instruments, optimizer) is imported normally; it is + import-stable by Rule 1. + +--- + +## Rule 6 — Gates and robustness sit OVER the frozen engine, read-only + +Two kinds of add-on never modify the engine: + +- **Gates** (`shared/gates/`) are boolean masks AND-ed into the signal *before* it reaches the engine: + `allow = directional_signal & ~block_condition`. A regime filter, a time-of-day filter, an + exhaustion filter — all are caller-side masks. The engine still just consumes a signal array. +- **Robustness layers** (`shared/robustness/`, doc 06) are read-only analyses *over* a finished result + (the study, the trade list, the equity curve). They never touch the engine or mutate results. + +Keeping both outside the engine means you can add or remove a filter or a check without re-validating +the engine. + +--- + +## Rule 7 — The registry is curated, locked, and append-only + +`registry/` holds only results that passed **all three** gates: Python search → MT5 verification → +human approval. Rules for it: + +- **Append-only in spirit:** you don't edit an approved entry; a new finding is a new entry. +- **Locked:** registry entries are not casual scratch space. Treat them like committed releases. +- **Self-documenting:** each entry records the params, the Python metrics, the MT5 report, and the + context (period, instrument, why it was approved). + +The registry is the *trustworthy* slice of everything you ever tried. Scratch experiments you don't +intend to keep go in a `results/` scratchpad, never the registry. + +--- + +## Rule 8 — Heavy compute runs detached; never relaunch a running job + +A full-history A/B (millions of bars) or a full Optuna study is **minutes** of compute and will block +a foreground shell or hit a timeout. Discipline: + +- **Always run heavy jobs in the background**, then poll (read the study's trial count, tail the + output file). Don't sit blocked online. +- **Smoke-test first:** a tiny run (e.g. 30–40 trials, a short period) validates the script end-to-end + before you commit to one full run. +- **Dedup guard:** before launching, check whether the same job is already running. Relaunching on a + perceived timeout spawns orphan duplicates that thrash the CPU and corrupt nothing but waste + everything. One study with N workers beats N competing scripts. + +--- + +## Why these rules pay off + +Individually each rule is a small constraint. Together they guarantee three properties that a pile of +ad-hoc scripts never has: + +1. **Trust** — the engine that produced a number is the same validated engine, every time. +2. **Reproducibility** — any past result can be re-run from its own snapshot. +3. **Fast, safe revert** — a failed idea is a deleted fork, not a half-removed change festering in the + core. + +Next: [`05-config-and-inputs.md`](05-config-and-inputs.md) — how the test inputs themselves are kept +separate and declared. diff --git a/05-config-and-inputs.md b/05-config-and-inputs.md new file mode 100644 index 0000000..d05999a --- /dev/null +++ b/05-config-and-inputs.md @@ -0,0 +1,180 @@ +# 05 — Configuration & Input Rules + +Every input to a backtest comes from one of **four declared sources**, never scattered through the +code. This is what makes runs reproducible and the engine reusable. The four: + +1. **Instrument config** — everything about the symbol/broker. +2. **Search space** — every tunable parameter, its range, its step. +3. **Wizard answers** — the run settings for *this* run (period, profile, trials, deposit, …). +4. **Frozen baseline** — the parameters you are *not* optimizing, pinned to known values. + +Keep these separate and you can always answer "what produced this number?" exactly. + +--- + +## 1. Instrument config — the symbol as data + +One object describes a symbol under one broker. The engine reads **everything** about the instrument +from here; nothing about ticks, spread, or swap is ever hard-coded elsewhere. + +### Fields you need + +| Field | Meaning | Where to find it | +|-------|---------|------------------| +| `name`, `symbol` | human label + broker symbol code | your broker | +| `point` | smallest price increment (e.g. `0.01`) | MT5 symbol spec | +| `digits` | price decimal places | MT5 symbol spec | +| `tick_size`, `tick_value` | price step and **money per lot per point** | MT5 symbol spec | +| `contract_size` | units per lot | MT5 symbol spec | +| `volume_min`, `volume_step`, `volume_max` | lot rounding/clamping | MT5 symbol spec | +| `spread_mode` | `bar_column` (real per-bar) \| `fixed_points` \| `annual_avg` | your choice per data quality | +| `spread_fixed_points` | used only with `fixed_points` | broker typical spread | +| `swap_mode` | `fixed_per_lot` \| `annual_pct` | broker / your model | +| `swap_long`, `swap_short` | per-lot-per-day rates (fixed mode) | MT5 symbol spec | +| `swap_annual_pct` | annual rate on notional (percent mode) | your model | +| `triple_swap_weekday` | the day swap is charged ×3 (often Wed) | broker | + +> **Get these from MT5's symbol specification** (`Market Watch → right-click symbol → Specification`), +> not from memory. A wrong `tick_value` or `contract_size` scales every PnL by a constant and makes +> the whole backtest meaningless. + +### Real / worst / best profiles + +Keep **three variants** of each instrument config: + +- **`real`** — your broker's actual conditions. The default for ranking. +- **`worst_case`** — wider spread, harsher swap. A finalist that survives this is robust to cost. +- **`best_case`** — tighter spread. Shows the ceiling. + +Cost-stress every finalist across all three; if the edge only exists in `best_case`, it isn't real. + +### Two instrument traps (both have bitten real labs) + +- **Quote currency ≠ account currency.** For a pair quoted in a non-account currency (e.g. a JPY-quoted + pair on a USD account), an EA's "money mode" lot formula can mis-scale the lot by ~the exchange rate. + Fix: generate that pair's presets with an **explicit fixed lot** equal to the intended cash amount + ÷ contract size, and set the money-amount input to zero. Always check the **first trade's lot** + against an MT5 run before trusting the series. +- **Unreliable historical spread.** For some instruments (notably crypto) the recorded per-bar spread + is garbage. Use `fixed_points` with a realistic value (and a wider `worst_case`) instead of + `bar_column`. + +--- + +## 2. Search space — declare every tunable parameter in one place + +The parameters Optuna is allowed to vary live in **one declared structure**, with an explicit range +and step for each. Nothing tunable is hidden in the strategy body. + +A search space is a mapping of `name → (low, high, step)`, plus a note of which names are integers: + +``` +SEARCH_SPACE = { + "open_distance": (0.1, 1.0, 0.05), # float + "grid_step": (0.8, 3.0, 0.10), # float + "grid_mult": (0.5, 1.3, 0.10), # float + "grid_count": (2, 9, 1), # int + "break_even": (0.0, 0.3, 0.05), # float + "trailing_stop": (0.1, 0.5, 0.05), # float + ... +} +INT_PARAMS = {"grid_count"} +``` + +The optimizer turns each entry into an Optuna `suggest_float` / `suggest_int` (doc 06). Document the +**same** space in a human-readable `parameter-space.md` next to the strategy, with the *reasoning* for +each range — why this floor, why this ceiling, what you expect. That note is half the value: it stops +you (or your assistant) from re-litigating ranges every iteration and makes overfit ranges obvious. + +### Rules for the search space + +- **Ranges encode priors, not hope.** A range should bracket where the answer plausibly is, informed by + the instrument's character. Absurdly wide ranges waste trials and invite overfit corners. +- **Step matters.** Too fine a step explodes the space; too coarse misses optima. Match the step to + what's meaningful (e.g. a 0.05% grid step, a 1-unit count). +- **What is NOT in the space is a decision.** Explicitly list parameters you deliberately exclude + (because prior tests showed ~0 impact, or because changing them is a *different* hypothesis). An + empty exclusion list usually means you haven't thought about it. +- **Timeframe is usually fixed per iteration.** Searching across timeframes in one study is + "timeframe-shopping" and overfits; pick the primary timeframe in the stats phase (doc 08) and lock it. + +--- + +## 3. Frozen baseline — pin what you don't optimize + +The parameters outside the search space still need *values*. Pin them to the validated baseline (your +known-good preset) so the optimizer varies only the few things under test, against an otherwise-known +configuration. The run script merges sampled parameters **onto** this frozen baseline before building +the engine params. This keeps each study **small and interpretable** — you're measuring the effect of +the few tuned knobs, not drowning in a 30-dimensional search. + +--- + +## 4. Lot / money mode — the input that silently breaks results + +Position sizing deserves its own section because it is the most common source of "my Python and MT5 +disagree by a constant factor" bugs. Expose money-based sizing through **two inputs**, plus the +fixed-lot guard: + +| Input | Meaning | +|-------|---------| +| `*_LotAmount` | the cash base. Lot = `LotAmount / open_price / contract_size`. | +| `*_LotBalance` | the account balance this base is calibrated for (the scaling window). | +| `*_Lot` | a fixed lot in lots. **Must be `0`** to enable money mode — otherwise the EA uses the fixed lot and **ignores** `LotAmount`. | + +Behavior: + +| `LotAmount` | `LotBalance` | Mode | Result | +|-------------|--------------|------|--------| +| X | **0** | **static** (default for testing) | lot always sized from cash X, independent of balance | +| X | Y | **dynamic** | lot = base × (current_balance / Y) — grows/shrinks with the account | + +### Rules + +- **Test with the static lot by default** (`LotBalance = 0`). Compounding (dynamic) is a *production* + choice; for research you want a clean, balance-independent measurement. +- **Always confirm `*_Lot = 0`** when you intend money mode. A stray non-zero fixed lot is a classic + silent bug — the run "works" but sizes every trade wrong. +- For a grid, levels multiply from the **base** lot via the grid multiplier (or ATR/other); the base + comes from the money rule above. +- Mirror whatever mode you choose **identically** in the Python engine and the generated `.set`, or the + two tiers won't agree. + +--- + +## 5. Wizard — capture run settings once, save them for reproducibility + +Before a run, a small interactive **wizard** asks the handful of settings that change run-to-run and +writes them to `wizard-answers.yaml`. That YAML *is* the reproducibility record: anyone (including +future-you) can see exactly what produced an iteration's numbers. + +Common wizard questions: + +``` +period_start / period_end # the backtest window +instrument_profile # real / worst_case / best_case +trials # Optuna trial budget (e.g. 300 / 1000 / 5000) +max_dd_ccy # hard drawdown cap for the constraint +top_n_verify # how many finalists to send to MT5 +fixed_lot / initial_deposit # sizing context +``` + +Design notes: +- Give every question a sensible **default** so a fast run is just pressing Enter. +- Make the wizard **extensible**: a base set of common questions plus a per-strategy hook for extra + ones (e.g. a grid wizard adds depth/multiplier defaults). +- **Write the answers to disk** next to the iteration. A run with no saved settings is not reproducible + and shouldn't be promoted. + +--- + +## Putting it together + +A single run is fully specified by: **which instrument config** (symbol + profile), **which search +space** (the tunable parameters), **which frozen baseline** (the pinned rest), and **which wizard +answers** (the window, trials, deposit). Four declared sources, each in its own place. That is the +whole "separate inputs" discipline — and it's why any run in this lab can be explained and reproduced +exactly. + +Next: [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md) — turning the search +space into an Optuna study and not fooling yourself with the result. diff --git a/06-optimization-and-robustness.md b/06-optimization-and-robustness.md new file mode 100644 index 0000000..bd43233 --- /dev/null +++ b/06-optimization-and-robustness.md @@ -0,0 +1,168 @@ +# 06 — Optimization (Optuna) & Robustness + +How to turn the declared search space into a Bayesian search that finds good parameters fast — and how +to avoid the trap that ruins most optimization: **overfitting to the past.** A great backtest is easy; +a robust one is the job. + +--- + +## 1. Why Optuna (and not a grid) + +A parameter grid is exponential: 8 parameters × 10 values each = 100 million combinations. Even at +"seconds per backtest" that's infeasible, and most of it is wasteland. + +**Optuna** (, docs ) uses a **TPE sampler** (Tree- +structured Parzen Estimator) that *learns* the shape of the objective as it goes and spends trials +where results are promising. You get a strong optimum in **hundreds to a few thousand** trials. + +Install: `pip install optuna` (doc 01). Core concepts you'll use: + +- **Study** — one optimization. Maximizes (or minimizes) an objective. Persisted to SQLite so it can + be resumed, inspected mid-run, and parallelized. +- **Trial** — one parameter sample + its score. `trial.suggest_float/int(name, low, high, step=…)` + draws each parameter; `trial.set_user_attr(...)` stashes the metrics for later inspection. +- **Sampler** — TPE by default; fine to start. + +```python +import optuna +study = optuna.create_study( + direction="maximize", + storage="sqlite:///study.db", # persist → resume, inspect, parallelize + study_name="my_iter", + load_if_exists=True, +) +study.optimize(objective, n_trials=1000, n_jobs=4) # n_jobs = parallel workers +``` + +--- + +## 2. The objective function + +The objective receives a trial, **samples** parameters from the search space, **runs** the engine, and +returns a **score**. Structure it in four clear steps: + +``` +def objective(trial): + 1. sampled = suggest_params(trial) # from SEARCH_SPACE (doc 05) + 2. params = build_params(sampled, frozen_baseline) + 3. result = engine.run(params, bars, instrument, deposit) + metrics = compute_metrics(result) + 4. score, violations = score_metrics(metrics, sampled) + for k, v in metrics.items(): trial.set_user_attr(k, v) + trial.set_user_attr("violations", violations) + return score +``` + +### Score design + +A good single-objective score balances **reward against risk**, for example: + +``` +score = Net − DD_WEIGHT × EquityDrawdown # reward profit, punish the drawdown that earned it +``` + +The weight makes the optimizer prefer a slightly smaller, much calmer equity curve over a fragile spike. +(Optuna also supports true multi-objective studies — e.g. maximize Net *and* minimize DD on a Pareto +front — but a weighted single objective is simpler and usually enough to start.) + +### Constraints as hard rejections, not soft nudges + +Some conditions should **disqualify** a trial outright, not just dock points. Implement them as a large +penalty per violation so violators sort to the bottom: + +``` +violations = [] +if metrics.trades < MIN_TRADES: violations.append("too few trades") +if metrics.profit_factor < MIN_PF: violations.append("PF too low") +if metrics.equity_dd > HARD_DD_CAP: violations.append("DD over cap") +score = base_score − PENALTY * len(violations) # PENALTY huge, e.g. 1e6 +``` + +Typical constraints worth enforcing: + +- **Minimum trade count** (a hard floor, calibrated to the window — e.g. ≥25–30 active trades per + year). Few-trade, high-PF results are almost always overfit luck, not edge. +- **Minimum Profit Factor.** +- **Hard drawdown cap** in account currency (e.g. ≤35% of deposit). +- **A path-independent fragility guard.** For a martingale grid, compute the *worst-case floating + drawdown* if a full series accumulates straight down an N% drop, and reject if it's worse than the + baseline's. This catches "the backtest never hit the bad path, but the configuration is a time bomb" + — the failure mode a pure historical backtest can't see. + +--- + +## 3. Don't pick the top-N by score — pick the top-N **diverse** + +Optuna converges: the top 10 trials by score are usually near-clones of one peak. Verifying 3 clones in +MT5 tells you nothing about robustness. Instead, pick **meaningfully different** finalists. + +**Greedy diverse selection:** + +``` +1. filter trials by the constraints (drop violators) +2. sort by score +3. start the selected set with the single best trial +4. repeatedly add the remaining trial with the MAXIMUM parameter-distance to the already-selected set, + lightly weighted by its rank so you still prefer strong scores +5. stop at N (2–3 is plenty) +``` + +Parameter distance = average normalized difference across all parameters (normalize each to [0,1]: +numeric by min–max, boolean to 0/1, categorical by index). The result is 2–3 finalists that are each +*good* but reach the result by *different* parameter regions — if they all survive MT5, you have real +robustness, not one lucky corner. + +--- + +## 4. Robustness layers — proving the result isn't overfit + +These are **read-only analyses over a finished result** (the study, the trade list, the equity curve). +They never touch the engine. Run them on finalists *before* spending MT5 time. Treat them as +report-only signals by default; tighten into hard gates as you gain confidence. + +| Layer | Question it answers | Method | +|-------|---------------------|--------| +| **Stability region** | Is the winner on a *plateau* of good configs, or a lone spike? | Cluster the study's top trials by parameter distance; prefer a finalist from the center of a dense good region, not an isolated peak. | +| **Neighborhood / sensitivity** | Does a small parameter nudge destroy the edge? | Perturb each lever ±1 step; demand all neighbors stay profitable. A fragile optimum fails this. | +| **Walk-forward (WFE)** | Does the edge hold out-of-sample? | Split into in-sample/out-of-sample windows; with the finalist's *fixed* params (no re-fit), compute `OOS_metric / IS_metric`. Add a **purge gap** between IS and OOS so leakage can't help. | +| **Monte-Carlo permutation** | How lucky was the trade *order*? | Shuffle trade order, drop a fraction, jitter PnL; build a distribution of drawdown/Net. A finalist whose real drawdown sits in the ugly tail is fragile. | +| **Deflated Sharpe (DSR)** | Is the Sharpe real after testing thousands of configs? | Adjust the observed Sharpe for the **number of trials** (multiple-testing). Many trials inflate the best result even under pure noise; DSR estimates the probability the true Sharpe beats a deflated benchmark. | +| **Era split** | Does the edge exist in *both halves* of history? | Run the finalist on era-1 vs era-2 separately; an edge present in only one era is regime-luck. | +| **Cost stress** | Does it survive realistic and adverse costs? | Re-run on `real` / `worst_case` / `best_case` instrument profiles (doc 05). Edge only in `best_case` = no edge. | + +> These are standard quant techniques, not exotic. Walk-forward and Monte-Carlo catch the everyday +> overfit; the Deflated Sharpe specifically counters the fact that an exhaustive search *will* surface +> an impressive-looking config from noise. References worth reading: López de Prado on the Deflated +> Sharpe Ratio and the dangers of backtest overfitting. + +--- + +## 5. What qualifies as a "finalist" + +Be strict — MT5 time is the expensive resource, and a weak finalist wastes it: + +- **From a *completed* search only.** An interrupted/partial Optuna study yields *preliminary* + numbers, not verification candidates. The README status must reflect the real process state, never + "continues" if the process is dead. +- **Passes the hard constraints** (trades, PF, DD cap, fragility guard). +- **Not concentrated in one year.** If a single year carries >~40% of total Net, it's fragile — one + regime is doing all the work. +- **Honest about flat periods.** A gate that sat out a bad year "survived by not trading" — describe it + that way, don't dress it up as robust profit. +- **Survives the robustness layers** you've chosen to enforce. +- **Converge to 2–3 diverse finalists**, not 5–6. More finalists is usually indecision, not rigor. + +--- + +## 6. Running searches without melting your machine (operational discipline) + +- **Smoke first.** A 30–40 trial run on a short period validates the whole script (data load → engine → + scoring → storage) in under a minute. Only then launch the full study. +- **Run heavy studies in the background and poll** the trial count in `study.db`; don't block a shell on + a multi-minute run. +- **One study, many workers** (`n_jobs=N`) beats N competing scripts. Cap heavy concurrency at your + performance-core count. +- **Dedup guard:** never relaunch a study you think timed out without checking it isn't still running — + duplicates thrash the CPU and waste hours. + +Next: [`07-mt5-bridge.md`](07-mt5-bridge.md) — verifying the finalists in the real terminal. diff --git a/07-mt5-bridge.md b/07-mt5-bridge.md new file mode 100644 index 0000000..6ce3dea --- /dev/null +++ b/07-mt5-bridge.md @@ -0,0 +1,236 @@ +# 07 — The MetaTrader 5 Bridge + +The bridge connects your fast Python search to the **gold-standard** MetaTrader 5 Strategy Tester. It +does four things: **(1) compile** your EA, **(2) auto-run** a backtest from a generated config, +**(3) parse** the HTML report, **(4) compare** Python vs MT5 side by side. + +The bridge has two flavors depending on **where MT5 runs** (doc `CLAUDE.md` Phase 0). Read the one +that matches your topology; the parsing and comparison steps are identical for both. + +``` + ┌──────────────────────────── Topology A (all-Windows, simplest) ───────────────┐ + Python ──► generate .set + tester.ini ──► launch terminal64.exe /config: locally ──► report.htm + └────────────────────────────────────────────────────────────────────────────────┘ + ┌──────────────────── Topology B/C (Unix + remote/VM Windows) ───────────────────┐ + Python ──► generate .set + tester.ini ──► copy to Windows ──► scheduled-task run ──► poll ──► pull report.htm + └────────────────────────────────────────────────────────────────────────────────┘ + │ + parse (UTF-16 HTML → metrics dict) + │ + comparison table: Python vs MT5 +``` + +--- + +## 1. Compiling your EA + +The tester runs a compiled `.ex5`. You compile from `.mq5` with **MetaEditor** (ships with MT5): + +- **In the GUI:** open the `.mq5` in MetaEditor → `Compile` (F7). Fix errors, confirm a clean compile. +- **From the command line** (scriptable, useful for the bridge): + +```bat +:: Windows +"C:\Program Files\MetaTrader 5\metaeditor64.exe" /compile:"C:\path\to\Expert.mq5" /log +``` + +Notes that save hours: +- **Custom indicators** the EA calls must be compiled too and placed in `MQL5\Indicators\`. +- If your EA's `#include` files live in a non-standard folder, compile the terminal in **portable + mode** (`/portable`) so MetaEditor resolves includes from that terminal's `MQL5\Include\`. +- After compiling, the `.ex5` goes in `MQL5\Experts\`. The tester references it by name. + +--- + +## 2. What MT5 needs to auto-run a test + +Two generated files plus the binaries already in place: + +### 2a. The `.set` file (parameters) + +A `.set` is the EA's inputs serialized as `key=value` lines. **MT5 writes `.set` files as UTF-16-LE** +— generate yours in the same encoding or the tester silently ignores them. + +Your bridge's **set generator** takes the *same* parameter dict you backtested in Python and writes the +matching `.set`. The mapping from Python parameter names to EA input names is strategy-specific — keep +a small **mapping table** documented next to the strategy so the two tiers always agree. Watch the +enum-valued inputs (mode flags, timeframe codes): MT5 inputs are often integers, so map e.g. a +timeframe to its `ENUM_TIMEFRAMES` integer, a mode to its enum index. + +> **The lot-mode guard from doc 05 applies here too:** when you generate the `.set`, make sure the +> fixed-lot input is `0` if you intend money mode, and handle the quote-currency lot trap. A +> mismatched `.set` is the #1 reason a verified MT5 number disagrees with Python. + +### 2b. The `tester.ini` (run config) + +A small INI tells the tester *what* to run: + +```ini +[Tester] +Expert=Experts\YourEA.ex5 +Symbol=YOURSYMBOL +Period=H1 ; chart timeframe (≤ the EA's signal timeframe) +Model=1 ; tick model — see below +FromDate=2024.01.01 +ToDate=2026.01.01 +Deposit=10000 +Leverage=100 +Report=report_myrun ; output HTML name +ShutdownTerminal=1 ; close the terminal when done (so the bridge knows it's finished) + +[TesterInputs] +; either point at the .set, or inline the key=value inputs here +``` + +The login section (`[Common]`) carries the demo account; keep credentials out of code (section 5). + +### Tick models (the speed/accuracy dial) + +| Model | Name | Accuracy | Speed (≈ 2-year run) | +|-------|------|----------|----------------------| +| 0 | Every tick based on **real ticks** | highest (truest fills) | slowest (~10–30 min) | +| 1 | Every tick (generated from M1) | high | medium | +| 2 | **1-minute OHLC** | good for non-tick-sensitive | fast (~2–3 min) | +| 4 | Open prices only | rough | fastest (~30 s) | + +Use **Model 2 (1-min OHLC)** for routine verification; **Model 0 (real ticks)** for the final check on +anything path-sensitive (trailing/grid). For a pure open-bar strategy even Model 4 is exact. + +--- + +## 3. Topology A — all on Windows (recommended, simplest) + +No SSH, no scheduled tasks. Two ways, easiest first: + +### 3a. The official `MetaTrader5` Python package + +`pip install MetaTrader5` (Windows-only) lets Python drive the terminal directly — initialize, pull +historical bars, and read symbol info. Use it for **data download** (section 6) and terminal control. +This is the cleanest path on Windows and removes most of the plumbing the remote topologies need. + +### 3b. Launch the tester from a generated config + +Write `tester.ini`, then launch the terminal pointed at it: + +```bat +start "" "C:\Program Files\MetaTrader 5\terminal64.exe" /config:"C:\path\to\tester.ini" +``` + +`ShutdownTerminal=1` makes the terminal close itself when the run finishes; your Python wrapper waits +for the process to exit (or for the report file to appear), then parses the report. Because everything +is local, there is no copy/poll step — read the HTML straight from the terminal's report folder. + +> Topology A is the right starting point for a Windows user. Get this working before considering +> anything remote. + +--- + +## 4. Topology B/C — Python on Unix, MT5 on a remote/VM Windows + +You develop on macOS/Linux and MT5 lives elsewhere (a Windows VPS, or a local VM). Two quirks force a +specific design: + +### Quirk 1 — a second, **portable** MT5 install +If that Windows box already runs a *live* MT5 for something else, the tester would fight it for chart +windows. Install a **separate portable** MT5 in its own folder (e.g. `C:\MT5_Tester\`) with its own +data directory. Copy `config\` (the saved login), `bases\` (symbol metadata), and `profiles\` from the +main terminal so the portable one auto-logs-in and knows your symbols. One-time setup. + +### Quirk 2 — SSH-launched processes die on disconnect +A process you start over SSH gets killed when the SSH session closes — the tester would die after a few +seconds. **Solution: a one-shot Windows Task Scheduler task.** Your wrapper SSHes in, creates a task +scheduled ~20 seconds out, and disconnects. The task fires *independently*, runs the full backtest, and +the terminal shuts itself down. + +### The end-to-end flow + +``` +1. Python: parse the .set → build tester.ini (inject demo login from the env file) +2. copy tester.ini + .set to the Windows box (scp / shared folder) +3. SSH: run a small PowerShell that registers a one-shot scheduled task ~20s out, then disconnect +4. the task fires → portable terminal64.exe /config:tester.ini → backtest → save report → shut down +5. Python: poll the Windows box for the report_*.htm file +6. when present (and the terminal has exited): copy the report back +7. parse → metrics +``` + +The two tiny helper scripts that live on the Windows box (a `.bat` that launches the terminal and a +`.ps1` that registers the scheduled task and clears old reports) are the only Windows-side code; keep +mirror copies in your repo so you can re-deploy them. + +### Manual fallback +If the automated run breaks, you can always: copy the `.set` over, RDP in, open `Strategy Tester` +(Ctrl+R), load the preset, Start, then `Save as Report`, and copy the HTML back. Slow but unblocks you. + +--- + +## 5. Credentials — never in code or chat + +Broker demo login lives in a **gitignored env file** (e.g. `.env`): + +``` +MT5_DEMO_LOGIN=... +MT5_DEMO_PASSWORD=... +MT5_DEMO_SERVER=YourBroker-Demo +``` + +The bridge reads these at runtime and injects them into `tester.ini`'s `[Common]` section **without +printing them**. Alternatively, copying the terminal's `accounts.dat` into the portable install gives +auto-login with no password in any config at all — the cleanest option. + +--- + +## 6. Downloading historical data (feeding the Python tier) + +The Python engine needs the **same** bars MT5 uses. Two ways to get them: + +- **Via the `MetaTrader5` package (Windows):** `mt5.copy_rates_range(symbol, timeframe, from, to)` → + a numpy array → save as Parquet. Pull M1 for the full window; resample to higher timeframes in Python. +- **Via an export script in MT5:** a small MQL5 script that walks `CopyRates` year by year and writes + CSV, which you then convert to Parquet. Useful when Python can't reach the terminal. + +For a brand-new symbol you must first **let MT5 cache its history** (open a chart, set +`Tools → Options → Charts → Max bars: Unlimited`, scroll back) so the tester and the export have enough +bars. Confirm the exact broker symbol code (it varies: indices and metals especially) before exporting. + +--- + +## 7. Parsing the report + +MT5 saves the Strategy Tester report as **HTML encoded UTF-16-LE** — not UTF-8. Read it with the right +encoding, parse with `lxml`/`html5lib`, and pull the key fields: + +- `Total Net Profit` +- `Profit Factor` +- `Total Trades` +- `Balance Drawdown Maximal` and **`Equity Drawdown Maximal`** (use **Maximal**, not Absolute — it's + the peak-to-trough that matters) + +Emit a `metrics` dict and (optionally) dump it to JSON next to the run. Validate the report is real +before trusting it (e.g. file size over some floor and a non-zero bar count) — a truncated report means +the run failed. + +--- + +## 8. The comparison table + +For every finalist, write an `auto-verification.md` that puts the two tiers side by side: + +| Metric | Python | MT5 | Δ | +|--------|--------|-----|---| +| Net | … | … | …% | +| Profit Factor | … | … | … | +| Equity DD max | … | … | …% | +| Total trades | … | … | …% | + +Then judge the delta **against the expected fidelity gap** (doc 03 §7): a clean-directional setup +should show only a modest negative gap (MT5 a little below Python); a trailing/grid setup in volatile +history can gap much wider, and that's *expected*, not a bug. The decision rule: if the **MT5** number still clears your bar after the gap, +the finalist is real; if the edge only existed in the optimistic Python figure, discard it. + +> A useful warm-up calibration: pick one known preset and run the full Python-vs-MT5 comparison on it +> first. It tells you *your* stack's actual gap for *your* EA, so later finalists are judged against a +> measured baseline rather than a generic rule of thumb. + +Next: [`08-workflow-cycle.md`](08-workflow-cycle.md) — the repeatable loop that ties all of this +together. diff --git a/08-workflow-cycle.md b/08-workflow-cycle.md new file mode 100644 index 0000000..41000e1 --- /dev/null +++ b/08-workflow-cycle.md @@ -0,0 +1,140 @@ +# 08 — The Workflow Cycle + +Everything above is parts; this is the assembly line. One research idea travels through a fixed, +repeatable cycle. The discipline of the cycle — especially the **progressive scope** (test the +*smallest* thing first) — is what keeps you from overfitting and what makes results explainable. + +--- + +## 1. The cycle + +``` +1. HYPOTHESIS — state the idea in one sentence, grounded in an existing engine + preset. +2. SCAFFOLD — create the iteration folder + run script + declared search space + run notes. +3. STATS — measure signals & filters on real history. Pick the primary timeframe. No tuning yet. +4. MINIMAL SCOPE — test the smallest version of the idea. Is there ANY edge? If no → stop here. +5. EXPAND + OPTUNA— only if minimal scope showed edge: add layers one at a time, then Optuna search. +6. MT5-VERIFY — run the 2–3 diverse finalists in the real terminal. Python ranks; MT5 decides. +7. APPROVAL — a human looks at the comparison and explicitly decides. +8. PROMOTE — approved result is copied into the locked registry with full context. +``` + +The early steps are cheap and exist to **kill bad ideas before they cost compute**. Most ideas should +die at step 3 or 4. That's the system working, not failing. + +--- + +## 2. Step-by-step + +### Step 1 — Hypothesis +Write it down in one sentence and tie it to something that already exists: *"On instrument X, adding a +daily-trend filter to preset Y should cut the drawdown from the counter-trend series without killing +Net."* A hypothesis that isn't grounded in an existing engine + baseline preset is a research project, +not an iteration — narrow it first. + +### Step 2 — Scaffold +Create the iteration folder by convention (doc 02 §4): + +``` +strategies//iterations/--/ +├── README.md # the hypothesis, status, and (later) the result +├── parameter-space.md # the declared search space + the reasoning for each range +├── optimize.py # a self-contained snapshot of the run script +└── wizard-answers.yaml # written by the wizard at run time +``` + +Copy the run script from a previous iteration as a template and edit it — each iteration owns its +snapshot (doc 04 Rule 5). A data/profile pre-check belongs here: confirm the instrument's data and +config exist *before* scaffolding (for a brand-new symbol, do the data-download + config step from doc +07 §6 / doc 05 §1 first). + +### Step 3 — Stats (measure before you tune) +Before any parameter search, **characterize the signals on real history**: + +- How often does the signal fire? What's the raw win rate, the average favorable vs adverse excursion? +- **Multi-timeframe filter analysis:** if the idea is a trend/regime filter, measure how well it + separates good from bad entries on *several* timeframes (e.g. M15/H1/H4/D1) and pick the **one + primary timeframe** where the separation is cleanest. Lock it for the rest of the iteration. +- Output: a short stats note + the chosen primary timeframe + the specific levers worth tuning. **No + parameter sweep yet.** + +This step routinely ends an iteration: if the signal has no separation on any timeframe, there's +nothing to optimize, and you've spent minutes instead of hours learning that. + +### Step 4 — Minimal scope (the anti-overfit gate) +Test the **smallest possible version** of the idea against the baseline: + +- Core engine behavior + the new filter, with **exits minimal** — e.g. for a trend strategy: trailing + only, *no* break-even; for a grid: the first level only, *no* depth/martingale/second tier. +- One question: **is there any edge at all** over the baseline? A handful of A/B runs, not a search. +- If the minimal version shows no edge, **stop**. Piling parameters onto a non-edge just manufactures an + overfit that looks good in-sample and dies in MT5. Most ideas should end here. + +### Step 5 — Expand + Optuna (only if step 4 showed edge) +Now add complexity **one layer at a time**, each as an A/B against the previous best: + +- add break-even, then a second grid tier, then an ATR-based stop, then wider ranges — *in sequence*, + keeping only what improves things. +- when the structure is set, run the **Optuna study** (doc 06): smoke-test first, then the full run in + the background. Apply the constraints and the fragility guard. Select **2–3 diverse finalists** (not + top-N-by-score). Run the robustness layers on them. + +Progressive expansion keeps every result interpretable ("the second tier added X, the ATR stop added +Y") and keeps the search space small enough that Optuna isn't just mining noise. + +### Step 6 — MT5-verify (finalists only) +Run the 2–3 finalists through the bridge (doc 07). Use a fast tick model to start, real ticks for the +final path-sensitive check. Write the `auto-verification.md` comparison and judge each delta against the +expected fidelity gap (doc 03 §7). **Don't verify every step of the search** — you trust the Python +engine for ranking; MT5 is reserved for the numbers that might go live. + +### Step 7 — Approval +A human reviews the comparison and the robustness signals and **explicitly** decides: promote, discard, +or iterate. Nothing auto-promotes. This is the gate where judgment — does this make sense, is the +drawdown livable, is the trade count real — overrides any single metric. + +### Step 8 — Promote +On approval, copy the finalist into the **locked registry** (doc 04 Rule 7) with everything needed to +trust and reproduce it: the params, the Python metrics, the MT5 report, and the context (period, +instrument, why approved). Discarded iterations are archived, not deleted — a negative result is data, +and re-testing a known dead end is waste. + +--- + +## 3. When to start a new iteration vs use the scratchpad + +- **New iteration** (full folder, will be tracked): a genuine improvement idea for a specific baseline; + an engine update that requires re-verifying existing approved results; new data/period to re-test on. +- **Scratchpad** (`results/` — never promoted): a quick experiment you don't intend to keep. Don't + pollute `iterations/` or the registry with throwaways. + +--- + +## 4. Naming convention + +``` +--/ +``` + +e.g. `trend-filter-daily-ema-2026-06-15`, `grid-second-tier-atr-2026-07-01`. The *approach* word is +free-form (coin it as needed); the date is the day of creation. Consistent names make the `iterations/` +folder a readable history of what you tried and when. + +--- + +## 5. The cycle as a habit + +Run it the same way every time and three good things happen automatically: + +- **Bad ideas die cheap** (steps 3–4), so compute goes to ideas with a pulse. +- **Surviving ideas are interpretable** (step 5's one-layer-at-a-time), so you understand *why* they + work — which is what tells you whether they'll keep working. +- **Promoted results are trustworthy and reproducible** (steps 6–8), so the registry stays a source of + truth rather than a graveyard of optimistic backtests. + +That's the whole lab: a fast engine you trust, a disciplined search you can't fool yourself with, and a +real-terminal check before anything counts. + +--- + +*Back to [`README.md`](README.md) · setup playbook [`CLAUDE.md`](CLAUDE.md).* diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..c6abed8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,183 @@ +# CLAUDE.md — Adaptive Setup Playbook for an AI Coding Assistant + +> This file is written **for an AI coding assistant** (Claude Code, Cursor, Copilot Chat, etc.) so +> it can help its human build this backtesting lab **from scratch on their specific machine**. +> It also works as a generic `AGENTS.md`. If you are a human, you can follow it too — just perform +> the steps yourself instead of asking the assistant. +> +> **Golden rule for the assistant:** this stack is reproduced *from your human's own EAs and +> presets*, not from any bundled code. Never invent a trading strategy or fabricate parameters. +> When trading logic is needed, ask the human for their MQL5 source and presets. + +--- + +## Phase 0 — Profile the device (do this first, always) + +Before installing anything, detect the environment. The whole setup branches on **where MT5 runs**. + +Run and read the output: + +```bash +# OS + architecture +uname -a 2>/dev/null || ver # macOS/Linux print uname; Windows cmd prints `ver` +python3 --version || python --version +``` + +Then classify the human into **one of three topologies** and confirm it with them before proceeding: + +| Topology | Python runs on | MT5 runs on | Bridge needed | Best for | +|----------|----------------|-------------|---------------|----------| +| **A — All-Windows (simplest)** | the Windows machine | the same Windows machine | none — direct, via the `MetaTrader5` pip package or local file paths | a Windows user; recommended starting point | +| **B — Unix + remote Windows** | macOS/Linux | a separate Windows box or VPS | SSH + file copy + a scheduled-task trick | someone on a Mac/Linux who already has a Windows VPS | +| **C — Unix + local Windows VM** | macOS/Linux host | Windows in a VM/Parallels/UTM | shared folder or SSH into the VM | a Mac/Linux user without a separate Windows box | + +> **Recommendation to give the human:** if they are on Windows, use **Topology A** — it is by far +> the least moving parts. The official `MetaTrader5` Python package only runs on Windows and lets +> Python drive the terminal directly, so no SSH/scheduled-task plumbing is needed. Topologies B/C +> exist only because some people develop on a Mac/Linux and keep MT5 elsewhere. + +Record the chosen topology — every later step references it. + +--- + +## Phase 1 — Install the Python stack + +Follow [`01-stack-and-install.md`](01-stack-and-install.md). In short: + +1. Install **Python 3.12+** (3.13/3.14 fine). On Windows, install from python.org and tick + *"Add Python to PATH"*. +2. Create an **isolated virtual environment** in the project root (`.venv`). Never install into the + system Python. +3. Install the libraries (`pandas`, `numpy`, `pyarrow`, `optuna`, `numba`, `sqlalchemy`, `pyyaml`, + `lxml`, `html5lib`, `tqdm`, `requests`). Exact commands and *why each library exists* are in doc 01. +4. On **Topology A** only, also `pip install MetaTrader5` (Windows-only package). + +Verify the venv imports everything before moving on. + +--- + +## Phase 2 — Scaffold the repository skeleton + +Create the directory layout from [`02-architecture.md`](02-architecture.md). Build **empty modules +with docstrings and type signatures first** — do not write trading logic yet. The skeleton: + +``` +your-lab/ +├── .venv/ # virtual environment (gitignored) +├── data/ # downloaded market data, parquet (gitignored) +├── shared/ # reusable infrastructure +│ ├── core/ # backtest engines (you fill these from your EA logic) +│ ├── indicators/ # indicator functions (RSI, ATR, EMA, your custom ones) +│ ├── instruments/ # per-symbol/broker config objects +│ ├── data/ # data loaders + MT5 report parser +│ ├── optimizer/ # objective + diverse top-N selector +│ ├── robustness/ # anti-overfit layers +│ ├── gates/ # reusable entry-filter masks +│ ├── wizard/ # pre-run interactive Q&A +│ └── mt5_pipeline/ # the MT5 bridge +├── strategies/ # one folder per strategy (your EAs + presets + iterations) +├── registry/ # approved, locked results — the source of truth +└── docs/ # this KB + your own notes +``` + +Mirror the responsibilities described in doc 02. Leave the engines for Phase 4. + +--- + +## Phase 3 — Bring in the human's trading assets + +**Stop and ask the human for:** + +1. The **MQL5 EA** they want to test — `.mq5` source *and* compiled `.ex5`, plus any custom + indicators (`.ex5`) the EA loads. +2. A **representative `.set` preset** (or a screenshot of the EA inputs) so you can see exactly which + inputs exist and their default values. +3. Their **broker + symbol(s)** and a **demo account** for the tester. + +From the EA source and preset you (assistant) will: + +- Enumerate every EA input and group them into **frozen** (not optimized) vs **tunable** (search space). +- Identify the instrument mechanics (digits, point, contract size, tick value, lot step) to build the + instrument config in Phase 5. These come from the **symbol specification** in MT5 + (`Right-click symbol → Specification`), not from guesses. + +--- + +## Phase 4 — Build the Python mirror of the EA (the engine) + +Read [`03-engine-design.md`](03-engine-design.md) carefully — it is the hardest and most important +part. The engine must reproduce the EA's **fill and exit logic** bar-by-bar. + +- Keep the engine **strategy-agnostic**: it consumes bars + signal arrays + stop/target arrays and + simulates fills. All the *strategy* math (when to enter, where to put stops) lives in the caller. +- Implement the **intra-bar sub-tick model** (doc 03) and the **pessimistic ordering** convention. +- Match the EA's **lot/money mode**, **spread model**, and **swap model** exactly (doc 05). +- The first milestone is **1:1 fidelity on one known preset**: run the EA in MT5 on a short period, + run your Python engine on the same data/preset, and reconcile trade-by-trade until the numbers + line up within the expected gap (doc 03 "Fidelity" section). **Do not optimize anything until this + passes** — an unvalidated engine optimizes noise. + +> The bundled docs use a **grid martingale** engine as the worked example because it exercises every +> hard case (pending orders, averaging, trailing on the basket, simultaneous closes). Your EA may be +> simpler (single position, signal-to-signal) or different — apply the *principles*, not the grid +> specifics. + +--- + +## Phase 5 — Wire up configuration and inputs + +Read [`05-config-and-inputs.md`](05-config-and-inputs.md). Build: + +- An **`InstrumentConfig`** object per symbol/broker (real conditions), plus optional `worst_case` + and `best_case` variants for cost-stress testing. +- A **declared search space** for the strategy (parameter → range → step), kept in code *and* + documented in a `parameter-space.md` next to the strategy. +- A **wizard** that captures run settings (period, profile, trial count, DD cap, deposit) and writes + them to `wizard-answers.yaml` for reproducibility. + +--- + +## Phase 6 — Optimizer + robustness + +Read [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md). Build the Optuna +objective (score + hard constraints), the diverse top-N selector, and at least the first robustness +checks (neighborhood/plateau, cost stress, era split). Start with a **small smoke run** (30–40 +trials, short period) to validate the script, then one full run in the background. + +--- + +## Phase 7 — The MT5 bridge + +Read [`07-mt5-bridge.md`](07-mt5-bridge.md) and implement the variant for the chosen topology: + +- **Topology A:** use the `MetaTrader5` package or generate a `tester.ini` + `.set` and launch + `terminal64.exe /config:` locally. Simplest path — no SSH. +- **Topology B/C:** generate `tester.ini` + `.set`, copy to the Windows box, trigger the run via a + one-shot scheduled task (so it survives the SSH disconnect), poll for the report, copy it back. + +Then implement the **report parser** (MT5 saves HTML as UTF-16) and the **comparison table** writer. + +--- + +## Phase 8 — Run the full cycle once, end to end + +Read [`08-workflow-cycle.md`](08-workflow-cycle.md) and run a complete loop on the human's real EA: +hypothesis → scaffold iteration → stats → minimal-scope test → optimize → verify the finalists in +MT5 → write the comparison → let the human decide. When the loop produces a number the human trusts, +the lab is working. + +--- + +## Standing rules for the assistant + +- **Never touch a validated engine to test an idea.** Fork it; prove the fork == original with the + change disabled; only then test. (Doc 04.) +- **Heavy runs go in the background.** A full-history A/B or a full Optuna study is minutes of + compute — start it detached and poll, never block. Smoke-test first. (Doc 06.) +- **Don't promote partial searches.** A finalist must come from a *completed* search. An interrupted + Optuna study yields preliminary numbers, not verification candidates. (Doc 06, 08.) +- **MT5 is the source of truth for live numbers.** Python is for ranking and A/B. Always verify + finalists in MT5 before the human relies on a result. (Doc 03, 07.) +- **Keep secrets out of code and chat.** Broker login goes in a gitignored env file, read at runtime, + never printed. (Doc 07.) +- **Ask before destructive actions** — deleting data, force-pushing, overwriting a validated preset. diff --git a/README.md b/README.md new file mode 100644 index 0000000..78d4f03 --- /dev/null +++ b/README.md @@ -0,0 +1,117 @@ +# Backtesting + Optuna + MT5-Verification Stack — Knowledge Base + +A blueprint for building a **personal trading-strategy research lab**: a fast Python +backtesting engine, a Bayesian parameter optimizer (Optuna), and an automated bridge to the +**MetaTrader 5 Strategy Tester** that cross-checks every result against the real terminal. + +This is a **knowledge base, not a code drop.** It describes the *architecture, algorithms, rules, +and tooling* so you can build your own version from scratch — with **your own** Expert Advisors, +**your own** strategies, and **your own** presets. There are no ready-made engines or strategy +files here on purpose: you supply those from your own MQL5 bots (ideally open-source EAs you own +or have the right to use). + +--- + +## What this stack does for you + +You have MQL5 Expert Advisors and a pile of ideas to test. The MetaTrader Strategy Tester is +accurate but **slow** — a single multi-year backtest with a real-tick model can take 10–30 minutes, +and an exhaustive optimization can run for days. That kills iteration speed. + +This stack solves it with a **two-tier model**: + +1. **Tier 1 — a fast Python "mirror engine"** that reproduces your EA's trade logic bar-by-bar over + pre-downloaded historical data. It is an *approximation* (more on fidelity below), but it runs a + full multi-year backtest in **seconds**, so you can rank thousands of parameter combinations with + Optuna in the time MT5 would run a handful. + +2. **Tier 2 — MetaTrader 5 as the gold standard.** Only the **finalists** from the Python search are + compiled and run in the real Strategy Tester. The Python number gets you the *shortlist*; the MT5 + number is what you trust for anything that goes live. + +``` + Idea ─► Python mirror engine ─► Optuna search (thousands of trials, fast) + │ + ▼ + 2–3 diverse finalists + │ + ▼ + MetaTrader 5 Strategy Tester (gold standard) + │ + ▼ + Python-vs-MT5 comparison table ─► keep / discard / iterate +``` + +The whole point is **disciplined isolation**: the engine is frozen and trusted, instruments are +described by data not code, every hypothesis is an isolated experiment, and only triple-checked +results are promoted. That discipline is what keeps a research lab from rotting into a pile of +one-off scripts that nobody can reproduce. + +--- + +## Who this is for + +- You run **MetaTrader 5** and write or use **MQL5** Expert Advisors. +- You want to test and optimize strategies **much faster** than the MT5 optimizer allows. +- You are comfortable with **Python** (intermediate) and the command line. +- You can run **MT5 on Windows** — either on the same machine (simplest) or on a separate + Windows box/VPS that your Python machine talks to. + +You do **not** need to be a quant. The hard parts (engine fidelity, Bayesian search, robustness +testing) are explained from first principles. + +--- + +## How to read this KB + +Start with `CLAUDE.md` if you plan to use an AI coding assistant (Claude Code, Cursor, etc.) to +build this with you — it is an **adaptive setup playbook** that profiles *your* machine and OS and +walks the install from zero. Otherwise read the numbered docs in order: + +| # | Doc | What you get | +|---|-----|--------------| +| — | [`CLAUDE.md`](CLAUDE.md) | Adaptive AI-assistant playbook: profiles your device, drives install from scratch. Doubles as `AGENTS.md`. | +| 01 | [`01-stack-and-install.md`](01-stack-and-install.md) | The exact tech stack, every library, where to get it, and how to install — per OS. | +| 02 | [`02-architecture.md`](02-architecture.md) | The layered architecture and data flow. The mental model for everything else. | +| 03 | [`03-engine-design.md`](03-engine-design.md) | How to design your own bar-by-bar engine. Grid logic used as the worked example. Fidelity vs MT5. | +| 04 | [`04-isolation-rules.md`](04-isolation-rules.md) | The isolation discipline: frozen engines, forks, separated instruments/strategies, curated registry. | +| 05 | [`05-config-and-inputs.md`](05-config-and-inputs.md) | How test inputs are kept *separate*: instrument config, parameter space, the pre-run wizard, lot/money mode. | +| 06 | [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md) | Optuna objective design, constraints, diverse top-N selection, and anti-overfit robustness layers. | +| 07 | [`07-mt5-bridge.md`](07-mt5-bridge.md) | Connecting to MT5: compiling EAs, auto-running the tester, parsing reports, comparing Python vs MT5. Local-Windows and remote variants. | +| 08 | [`08-workflow-cycle.md`](08-workflow-cycle.md) | The full repeatable cycle: hypothesis → scaffold → stats → optimize → verify → promote. | + +--- + +## The 30-second mental model + +- **The engine knows nothing about your strategy.** It takes bars + entry signals + stop/target + prices and simulates fills. All strategy math lives in *caller* code. (Doc 02–03.) +- **The engine is frozen.** You never edit a validated engine to test an idea — you fork it, + prove the fork reproduces the original 1:1 with the change off, then test. (Doc 04.) +- **Instruments are data, not code branches.** Tick value, spread model, swap, lot steps — all in a + per-symbol config object. The engine reads everything from it. (Doc 05.) +- **Search inputs are declared, not scattered.** Every tunable parameter, its range, and its + constraints live in one declared search space; one wizard captures the run settings; one YAML + records the answers so any run is reproducible. (Doc 05–06.) +- **Python ranks, MT5 decides.** Fast Python search produces a shortlist; MT5 produces the trusted + number; a comparison table is saved for every finalist. (Doc 03, 06, 07.) + +--- + +## What you must bring yourself + +This KB is deliberately empty of trading IP. To build a working lab you supply: + +- **Your MQL5 EA(s)** — compiled `.ex5` plus source `.mq5`, and any custom indicators they call. +- **Your strategy logic** — encoded once in Python (the caller) so the mirror engine can run it. +- **Your presets** — the `.set` files / input templates you want to test and optimize. +- **Historical data** — downloaded from your broker via MT5 (the stack includes a recipe). +- **A broker demo account** — for the MT5 Strategy Tester runs. + +Everything else — the architecture, the optimizer, the MT5 bridge, the robustness checks, and the +rules that hold it together — is described in the docs above. + +--- + +*This KB is brand-free and self-contained. Drop the folder into a Git repository, open it with your +AI coding assistant, and build your own lab.*