first commit
This commit is contained in:
+20
@@ -0,0 +1,20 @@
|
||||
# Python
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Market data — large & re-downloadable
|
||||
data/
|
||||
|
||||
# Run outputs / scratch
|
||||
results/
|
||||
*.db # Optuna SQLite studies
|
||||
*.htm # pulled MT5 reports
|
||||
|
||||
# Secrets — NEVER commit broker credentials
|
||||
.env
|
||||
*.env
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,180 @@
|
||||
# 01 — Tech Stack & Installation From Scratch
|
||||
|
||||
Everything you need to install, what each piece is for, where to get it, and the exact commands per
|
||||
operating system. Nothing here is strategy-specific — this is the plumbing.
|
||||
|
||||
---
|
||||
|
||||
## 1. The stack at a glance
|
||||
|
||||
| Layer | Tool | Role | Where to get it |
|
||||
|-------|------|------|-----------------|
|
||||
| Language | **Python 3.12+** (3.13/3.14 work) | everything outside MT5 | <https://www.python.org/downloads/> |
|
||||
| Data frames | **pandas** | OHLC tables, resampling, equity curves | <https://pandas.pydata.org> · `pip install pandas` |
|
||||
| Numerics | **numpy** | vectorized price/indicator math | <https://numpy.org> · `pip install numpy` |
|
||||
| Columnar storage | **pyarrow** | read/write **Parquet** market data (fast, compact) | <https://arrow.apache.org/docs/python/> · `pip install pyarrow` |
|
||||
| JIT speed | **numba** | compiles the hot bar-by-bar loop to machine code | <https://numba.pydata.org> · `pip install numba` |
|
||||
| Optimizer | **Optuna** | Bayesian hyper-parameter search | <https://optuna.org> · `pip install optuna` |
|
||||
| Optuna storage | **SQLAlchemy** (+ **alembic**) | persists studies to a SQLite DB so runs resume/parallelize | <https://www.sqlalchemy.org> · `pip install sqlalchemy` |
|
||||
| Config | **PyYAML** | reproducible run settings (`wizard-answers.yaml`) | <https://pyyaml.org> · `pip install pyyaml` |
|
||||
| HTML parsing | **lxml** + **html5lib** | parse the MT5 Strategy Tester HTML report | <https://lxml.de> · `pip install lxml html5lib` |
|
||||
| Progress / logs | **tqdm**, **colorlog** | progress bars, readable logs | `pip install tqdm colorlog` |
|
||||
| HTTP | **requests** | optional data downloads / webhooks | <https://requests.readthedocs.io> · `pip install requests` |
|
||||
| Tester | **MetaTrader 5** terminal | the gold-standard backtester + data source | from your broker, or <https://www.metatrader5.com/en/download> |
|
||||
| MT5 control (Windows) | **MetaTrader5** pip package | drive the terminal & pull data from Python (Windows only) | <https://pypi.org/project/MetaTrader5/> · `pip install MetaTrader5` |
|
||||
|
||||
> **Optional data source.** If you want history without MT5 export, `dukascopy-python`
|
||||
> (`pip install dukascopy-python`, <https://pypi.org/project/dukascopy-python/>) pulls free tick/bar
|
||||
> data for many symbols. MT5-exported data from your own broker is preferred because it matches the
|
||||
> tester exactly.
|
||||
|
||||
### Reference versions
|
||||
|
||||
A known-good combination (early 2026) — use these or the latest stable:
|
||||
|
||||
```
|
||||
python 3.14
|
||||
pandas 3.0
|
||||
numpy 2.4
|
||||
pyarrow 24.0
|
||||
numba 0.65
|
||||
optuna 4.8
|
||||
sqlalchemy 2.0
|
||||
pyyaml 6.0
|
||||
lxml 6.1
|
||||
```
|
||||
|
||||
Pin exact versions in a `requirements.txt` once your lab works, so it reproduces later.
|
||||
|
||||
---
|
||||
|
||||
## 2. What each library actually does here
|
||||
|
||||
- **pandas / numpy** — market data is loaded into a DataFrame of `[timestamp, open, high, low, close,
|
||||
spread]`. Indicators and signals are computed as numpy arrays. Equity curves are pandas frames.
|
||||
- **pyarrow + Parquet** — a few years of M1 (one-minute) bars is millions of rows. Parquet stores it
|
||||
columnar and compressed: a multi-million-row file loads in well under a second and is a fraction of
|
||||
CSV size. This is what makes "full-history backtest in seconds" possible.
|
||||
- **numba** — the engine's inner loop walks every bar (and sub-ticks within each bar). Pure-Python
|
||||
that is too slow. `@njit` compiles it to native code on first call; subsequent runs are C-fast.
|
||||
- **Optuna** — instead of brute-forcing a parameter grid, Optuna uses a Bayesian sampler (TPE) that
|
||||
*learns* which regions of the search space are promising and concentrates trials there. You get a
|
||||
good optimum in hundreds–thousands of trials instead of an exhaustive grid of millions.
|
||||
- **SQLAlchemy/SQLite** — Optuna writes each trial to a `study.db`. That means a study can be stopped
|
||||
and resumed, inspected mid-run (count completed trials), and run with multiple worker processes
|
||||
pointing at the same DB.
|
||||
- **lxml / html5lib** — the MT5 tester exports its report as an HTML file encoded **UTF-16-LE**. These
|
||||
parse it into a metrics dict (Net Profit, Profit Factor, Drawdown, trade count, …).
|
||||
- **MetaTrader5 package** — on Windows, this is the clean way to (a) download historical bars and
|
||||
(b) launch/script the terminal. On non-Windows you don't have it, which is why remote topologies use
|
||||
SSH + a scheduled task instead.
|
||||
|
||||
---
|
||||
|
||||
## 3. Install — step by step
|
||||
|
||||
### 3.0 Prerequisites
|
||||
|
||||
- **Python 3.12+**. Check with `python3 --version` (macOS/Linux) or `python --version` (Windows).
|
||||
- **Git** (to version your lab).
|
||||
- **MetaTrader 5** installed from your broker, with a **demo account** logged in.
|
||||
|
||||
### 3.1 Windows (Topology A — recommended)
|
||||
|
||||
```powershell
|
||||
# From the project root, in PowerShell or cmd:
|
||||
|
||||
# 1. Create the virtual environment
|
||||
python -m venv .venv
|
||||
|
||||
# 2. Activate it
|
||||
.\.venv\Scripts\activate
|
||||
|
||||
# 3. Upgrade pip
|
||||
python -m pip install --upgrade pip
|
||||
|
||||
# 4. Install the stack (MetaTrader5 included — Windows only)
|
||||
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests MetaTrader5
|
||||
```
|
||||
|
||||
### 3.2 macOS / Linux (Topology B/C — MT5 lives elsewhere)
|
||||
|
||||
```bash
|
||||
# 1. Create the venv
|
||||
python3 -m venv .venv
|
||||
|
||||
# 2. Activate
|
||||
source .venv/bin/activate
|
||||
|
||||
# 3. Upgrade pip
|
||||
python3 -m pip install --upgrade pip
|
||||
|
||||
# 4. Install the stack (NO MetaTrader5 package — it is Windows-only)
|
||||
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests
|
||||
```
|
||||
|
||||
> On Apple Silicon (M-series) everything above is native arm64 and fast. `numba`/`numpy` ship arm64
|
||||
> wheels — no Rosetta needed.
|
||||
|
||||
### 3.3 Verify the install
|
||||
|
||||
```bash
|
||||
# Use the venv's python explicitly to avoid the system interpreter
|
||||
.venv/bin/python3 -c "import pandas, numpy, pyarrow, optuna, numba, yaml, lxml; print('core OK')"
|
||||
# Windows: .\.venv\Scripts\python -c "..."
|
||||
```
|
||||
|
||||
On Windows also verify the terminal link:
|
||||
|
||||
```python
|
||||
import MetaTrader5 as mt5
|
||||
print(mt5.initialize()) # True if it found & launched the terminal
|
||||
print(mt5.version())
|
||||
mt5.shutdown()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Always use the venv interpreter
|
||||
|
||||
A recurring source of bugs is accidentally running the **system** Python (which lacks the libraries).
|
||||
Make it a habit to call the venv interpreter by path:
|
||||
|
||||
```bash
|
||||
# macOS/Linux
|
||||
.venv/bin/python3 your_script.py
|
||||
|
||||
# Windows
|
||||
.\.venv\Scripts\python your_script.py
|
||||
```
|
||||
|
||||
…or activate the venv at the start of every session. Pick one convention and keep it.
|
||||
|
||||
---
|
||||
|
||||
## 5. `.gitignore` essentials
|
||||
|
||||
Your lab will accumulate large data and secrets. Ignore them from day one:
|
||||
|
||||
```gitignore
|
||||
.venv/
|
||||
data/ # market data is large & re-downloadable
|
||||
results/ # scratch run outputs
|
||||
*.db # Optuna SQLite studies
|
||||
*.htm # pulled MT5 reports
|
||||
.env # broker credentials — NEVER commit
|
||||
__pycache__/
|
||||
.DS_Store
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Hardware notes
|
||||
|
||||
- Backtesting is **CPU + RAM** bound, not GPU. A modern multi-core CPU and 16 GB+ RAM is plenty.
|
||||
- Optuna parallelizes across CPU cores. The practical cap for *heavy* concurrent backtests is roughly
|
||||
your number of **performance cores** — beyond that they contend and slow each other down. Prefer
|
||||
**one** Optuna study with `n_jobs=N` over N separate scripts fighting for cores.
|
||||
- Millions of M1 bars fit comfortably in RAM as a pandas frame; loading from Parquet is the only I/O.
|
||||
|
||||
Next: [`02-architecture.md`](02-architecture.md) — the layered architecture you are about to build.
|
||||
@@ -0,0 +1,165 @@
|
||||
# 02 — Architecture & Data Flow
|
||||
|
||||
The whole lab is a set of **layers with one-directional dependencies**. Higher layers call lower
|
||||
ones; lower layers never know about higher ones. This is what lets you freeze the engine, swap
|
||||
instruments, and add strategies without anything leaking sideways.
|
||||
|
||||
---
|
||||
|
||||
## 1. The layer stack
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ STRATEGY / CALLER one folder per strategy │
|
||||
│ loads data, computes signals + stops, calls the engine, scores │
|
||||
└───────────────┬──────────────────────────────┬──────────────────────┘
|
||||
│ │
|
||||
┌───────▼────────┐ ┌─────────▼──────────┐
|
||||
│ OPTIMIZER │ │ MT5 BRIDGE │
|
||||
│ objective, │ │ compile, run, │
|
||||
│ Optuna study, │ │ parse, compare │
|
||||
│ diverse top-N │ └─────────┬──────────┘
|
||||
└───────┬─────────┘ │
|
||||
│ │ (gold-standard verify)
|
||||
┌───────▼─────────┐ │
|
||||
│ ROBUSTNESS │ anti-overfit checks over results (read-only)
|
||||
└───────┬─────────┘
|
||||
│
|
||||
┌───────▼─────────────────────────────────────────────┐
|
||||
│ ENGINE (frozen) bar-by-bar fill simulator │
|
||||
│ knows: bars, signals, stop/target prices, instrument │
|
||||
│ knows NOT: your strategy, indicators, broker │
|
||||
└───┬─────────────┬────────────────┬───────────────────┘
|
||||
│ │ │
|
||||
┌────────▼───┐ ┌──────▼──────┐ ┌──────▼───────┐ ┌──────────────┐
|
||||
│ INDICATORS │ │ INSTRUMENTS │ │ GATES │ │ DATA │
|
||||
│ RSI/ATR/.. │ │ per-symbol │ │ entry-filter │ │ loaders + │
|
||||
│ pure fns │ │ config объ. │ │ masks │ │ MT5 parser │
|
||||
└────────────┘ └─────────────┘ └──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
### Layer responsibilities
|
||||
|
||||
| Layer | Folder | Owns | Must NOT |
|
||||
|-------|--------|------|----------|
|
||||
| **Data** | `shared/data/` | load bars from Parquet; fetch from MT5; parse MT5 HTML reports | contain strategy logic |
|
||||
| **Instruments** | `shared/instruments/` | per-symbol/broker config objects (tick value, spread, swap, lot steps) | know any strategy |
|
||||
| **Indicators** | `shared/indicators/` | pure functions: RSI, ATR, EMA, SMA, your custom range filter, etc. | hold state across calls |
|
||||
| **Gates** | `shared/gates/` | reusable boolean masks that *filter* entries (regime, time-of-day, exhaustion) | open/close trades |
|
||||
| **Engine** | `shared/core/` | the bar-by-bar fill/exit simulator; **frozen** once validated | compute signals or stops |
|
||||
| **Robustness** | `shared/robustness/` | read-only anti-overfit analyses over a result | mutate the engine or results |
|
||||
| **Optimizer** | `shared/optimizer/` | objective fn, Optuna wiring, diverse top-N selection | know broker specifics |
|
||||
| **MT5 bridge** | `shared/mt5_pipeline/` | generate `.set`/`.ini`, run the tester, pull & parse the report, compare | compute strategy |
|
||||
| **Strategy / caller** | `strategies/<name>/` | glue: data → signals → stops → engine → metrics; the only layer that knows the *whole* picture | be reused by another strategy (copy, don't import) |
|
||||
| **Registry** | `registry/` | approved, locked results — the source of truth | be edited casually |
|
||||
|
||||
---
|
||||
|
||||
## 2. The core contract: engine knows nothing about strategy
|
||||
|
||||
This is the single most important design decision. The engine's input is **pre-computed**:
|
||||
|
||||
```
|
||||
engine.run(
|
||||
bars, # DataFrame [timestamp, open, high, low, close, spread]
|
||||
signals_long, # boolean array — True on the bar an entry is triggered
|
||||
signals_short, # boolean array
|
||||
sl_prices, # array — the stop price for an entry on that bar (NaN if none)
|
||||
tp_prices, # array — the target price
|
||||
instrument, # InstrumentConfig — all symbol mechanics
|
||||
lot / money_mode, # position sizing inputs
|
||||
initial_deposit,
|
||||
) -> Result
|
||||
```
|
||||
|
||||
> **Crucial:** the stop and target arrive as **finished prices** computed by the caller. The engine
|
||||
> never decides *where* a stop goes — only *whether* price touched it. This is the seam that
|
||||
> separates "the strategy" from "the simulator". Change your strategy → you change the caller and the
|
||||
> arrays you hand in; the engine is untouched.
|
||||
|
||||
The engine's output is equally generic:
|
||||
|
||||
```
|
||||
Result:
|
||||
trades # list of closed trades (entry/exit time, price, lots, pnl, swap)
|
||||
equity_curve # DataFrame [timestamp, balance, equity], sampled (e.g. hourly)
|
||||
final_balance
|
||||
initial_deposit
|
||||
```
|
||||
|
||||
A separate `compute_metrics(result)` turns that into the numbers you optimize on: Net Profit, Profit
|
||||
Factor, Win Rate, max Balance/Equity Drawdown, Sharpe, APR, trade count.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data flow of one backtest
|
||||
|
||||
```
|
||||
data/<symbol>/<SYM>_M1_<years>.parquet
|
||||
│ load_bars() → DataFrame with per-bar spread column
|
||||
▼
|
||||
caller: resample M1 → the signal timeframe (e.g. H1/H4/D1)
|
||||
│ indicators.* on the resampled frame
|
||||
▼
|
||||
caller: edge-detect signals (True only on the bar the condition first flips, not every bar after)
|
||||
│ + optional gates (regime/time filters AND-ed into the signal)
|
||||
▼
|
||||
caller: compute SL/TP price arrays from params (ATR stop, % stop, indicator band, …)
|
||||
▼
|
||||
engine.run(bars, signals, sl/tp, instrument, sizing, deposit)
|
||||
│
|
||||
▼
|
||||
Result → compute_metrics → dict
|
||||
│
|
||||
▼ (finalists only)
|
||||
MT5 bridge: build .set from the same params → run real tester → parse report → compare
|
||||
```
|
||||
|
||||
Two subtleties that cause most bugs if missed (both explained in doc 03):
|
||||
|
||||
- **Edge detection.** Signals must be `True` only on the *transition* bar, not forward-filled, or the
|
||||
engine re-enters every bar.
|
||||
- **Timeframe alignment.** Indicators computed on a higher timeframe must be mapped back onto the M1
|
||||
bars correctly (no look-ahead — a daily value is only known after that day closes).
|
||||
|
||||
---
|
||||
|
||||
## 4. Why "shared" vs "strategies"
|
||||
|
||||
- **`shared/`** is infrastructure you write **once** and treat as a library: engine, indicators,
|
||||
instruments, optimizer, robustness, bridge. It is import-stable. Breaking changes here ripple
|
||||
everywhere, so it changes rarely and deliberately.
|
||||
- **`strategies/<name>/`** is where experimentation happens. Each strategy is **self-contained**: its
|
||||
EA source, its presets, and its `iterations/` (each iteration = one research attempt with its own
|
||||
`optimize.py`, search space, results, and verification). Strategies **copy** glue code between
|
||||
iterations rather than importing it, so an old iteration always reproduces even after you change
|
||||
how you do things.
|
||||
|
||||
```
|
||||
strategies/<strategy>/
|
||||
├── source/ # your EA .mq5/.ex5 + custom indicators (read-only after compile)
|
||||
├── presets/ # canonical baseline .set files (frozen reference)
|
||||
├── production/ # live-ready presets
|
||||
├── iterations/<base>-<approach>-<YYYY-MM-DD>/
|
||||
│ ├── README.md # context, status, result summary
|
||||
│ ├── parameter-space.md # the search space for this attempt
|
||||
│ ├── optimize.py # the run script (a snapshot — self-contained)
|
||||
│ ├── wizard-answers.yaml # the run settings (reproducibility)
|
||||
│ ├── top-1/ top-2/ # the diverse finalists: params, preset, py+mt5 metrics
|
||||
│ └── auto-verification.md# Python-vs-MT5 comparison table
|
||||
└── archive/ # discarded attempts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. The registry is the source of truth
|
||||
|
||||
Nothing is "real" until it is in `registry/`. A registry entry is a result that passed **all three**
|
||||
gates: Python search → MT5 verification → human approval. Once written it is **locked** — you never
|
||||
silently edit a registry entry; you create a *new* one. This is what stops a research lab from
|
||||
drifting: the registry is the curated, trustworthy subset of everything you ever tried.
|
||||
|
||||
Doc 04 covers the isolation rules that keep these boundaries honest. Doc 03 is next — how to actually
|
||||
build the engine.
|
||||
|
||||
Next: [`03-engine-design.md`](03-engine-design.md).
|
||||
@@ -0,0 +1,249 @@
|
||||
# 03 — Designing Your Own Bar-by-Bar Engine
|
||||
|
||||
This is the core craft of the lab: a Python engine that reproduces your EA's fills and exits faithfully
|
||||
enough to rank ideas, fast enough to run thousands of times. This doc explains the **algorithm and the
|
||||
design principles** — you write the code from your own EA's logic. The worked example is a **grid
|
||||
martingale** engine because it exercises every hard case; adapt the principles to whatever your EA does.
|
||||
|
||||
---
|
||||
|
||||
## 1. What an engine is (and is not)
|
||||
|
||||
An engine is a **generic bar-by-bar simulator**. It iterates historical bars in order and, at each
|
||||
bar, decides whether existing positions hit a stop/target and whether a new entry fills. It is told
|
||||
*when* to enter (signal arrays) and *where* the stops are (price arrays). It does **not** know the
|
||||
strategy that produced those.
|
||||
|
||||
- **It knows:** bars (OHLC + spread), entry signals, stop/target prices, the instrument's mechanics,
|
||||
and the position-sizing rule.
|
||||
- **It does not know:** your indicators, your regime filters, your broker, why a signal fired.
|
||||
|
||||
Keeping that boundary is what makes the engine reusable across strategies and **freezable** (doc 04).
|
||||
|
||||
---
|
||||
|
||||
## 2. The intra-bar problem (the heart of fidelity)
|
||||
|
||||
Your data is **bars**: each M1 bar is four numbers — open, high, low, close. But within that minute
|
||||
the price walked a *path* you can't see. MT5's tester, in its finer models, simulates an interpolated
|
||||
tick path inside each bar, so an EA's `OnTick` fires many times per bar. A bar-based engine has only
|
||||
four anchor points. **How you order those four points inside a bar decides your fills.**
|
||||
|
||||
### The 4-sub-tick model
|
||||
|
||||
Process each bar as **four sub-ticks** in a fixed order. The order encodes a *pessimistic* assumption:
|
||||
the price visits the point that hurts an open position **before** the point that helps it.
|
||||
|
||||
```
|
||||
For a LONG position (stop below, target above): OPEN → LOW → HIGH → CLOSE
|
||||
For a SHORT position (stop above, target below): OPEN → HIGH → LOW → CLOSE
|
||||
```
|
||||
|
||||
Why: if in the same bar price could have hit *both* the stop and the target, the pessimistic order
|
||||
makes the **stop** win — the realistic worst case. For a long, LOW (the stop side) is visited before
|
||||
HIGH (the target side). This matches how a careful tester resolves ambiguous bars and keeps your
|
||||
results honest (an optimistic engine "discovers" edges that don't survive live).
|
||||
|
||||
At each sub-tick the price is a single number, and you run the same checks:
|
||||
|
||||
```
|
||||
for each bar:
|
||||
for each sub_tick in pessimistic_order(position_direction):
|
||||
price = sub_tick_price # one of O / H / L / C
|
||||
1. if a position is open:
|
||||
- apply daily swap if the calendar day rolled over
|
||||
- check stop hit (use BID/ASK appropriately; add spread)
|
||||
- check target hit
|
||||
- if both could hit this bar → pessimistic order already decided the stop
|
||||
2. if flat and an entry signal is active and a next bar exists:
|
||||
- open the position at the next bar's open (avoid look-ahead)
|
||||
3. sample the equity curve periodically (e.g. once per hour)
|
||||
```
|
||||
|
||||
> **Look-ahead guard:** a signal computed *from* a bar's close must execute on the *next* bar's open,
|
||||
> never the same bar's close. Otherwise you are trading on information you wouldn't have had.
|
||||
|
||||
---
|
||||
|
||||
## 3. Worked example: a grid martingale engine
|
||||
|
||||
A grid (averaging) EA is the stress test for an engine because it juggles pending orders, multiple
|
||||
simultaneous positions, a moving basket stop, and a synchronized close. If you can mirror this, you
|
||||
can mirror anything simpler. Here is the full lifecycle in pseudocode-level prose.
|
||||
|
||||
### 3.1 Order types you must model
|
||||
|
||||
| Type | Fires when |
|
||||
|------|-----------|
|
||||
| Market | immediately at the current ask/bid |
|
||||
| Buy-Stop | ask **rises to** the level (breakout up) |
|
||||
| Buy-Limit | ask **falls to** the level (pullback down) |
|
||||
| (mirror for sell side) |
|
||||
|
||||
### 3.2 Series lifecycle (one "basket" from open to close)
|
||||
|
||||
```
|
||||
1. FLAT. No positions, no pending orders.
|
||||
└─ after a cooldown since the previous series closed …
|
||||
|
||||
2. PLACE FIRST PENDING.
|
||||
base_price = close price of the previous series (or current price on first run)
|
||||
pending = Buy-Stop at base + open_distance% # wait for momentum confirmation
|
||||
|
||||
3. PENDING FILLS → position #1.
|
||||
first_open = fill price
|
||||
first_lot = sized by the money/lot rule (section 4)
|
||||
count = 1
|
||||
|
||||
4. PRICE MOVES AGAINST THE BASKET → grid levels fire (martingale).
|
||||
next_level_price = last_fill_price − first_open × (grid_step% / 100)
|
||||
next_lot = previous_lot × grid_multiplier (clamped to the volume minimum)
|
||||
add positions up to grid_count at step/mult set #1, then switch to set #2, etc.
|
||||
(cap the total number of levels — deep martingales are where accounts die)
|
||||
|
||||
5. PRICE MOVES WITH THE BASKET → basket trailing activates (section 5).
|
||||
compute the basket's average price across all open positions
|
||||
once profit clears (break_even + trailing_stop) → move a single basket stop up
|
||||
|
||||
6. PRICE HITS THE BASKET STOP → all positions close in the same sub-tick.
|
||||
record the series as one closed event (sum of pnl + accumulated swap)
|
||||
base_price = the close price (seed for the next series)
|
||||
reset to FLAT
|
||||
```
|
||||
|
||||
### 3.3 Adaptive grid levels
|
||||
|
||||
A robust grid spaces levels from the **most recent fill** but scales the step by the **first** fill's
|
||||
price, so the spacing stays constant in points while the trigger range tracks slippage:
|
||||
|
||||
```
|
||||
next_level = last_fill_price − first_open_price × (grid_step% / 100)
|
||||
```
|
||||
|
||||
This is one line, but it is the difference between a grid that drifts and one that holds its spacing.
|
||||
|
||||
---
|
||||
|
||||
## 4. Position sizing — the money/lot modes
|
||||
|
||||
Match your EA's sizing exactly or your PnL will be off by a constant factor. Common modes, in priority
|
||||
order:
|
||||
|
||||
1. **Risk-on-stop:** `lot = max_loss_money / (stop_distance_points × tick_value)`.
|
||||
2. **Fixed lot:** `lot = configured_lot` (optionally scaled by `balance / reference_balance`).
|
||||
3. **Money mode:** `lot = amount / open_price / contract_size` — size so a fixed *cash* amount is
|
||||
deployed regardless of price.
|
||||
|
||||
After computing, **round to the broker's volume step** and clamp to the volume minimum. Two notorious
|
||||
pitfalls (doc 05 covers them in full):
|
||||
|
||||
- A "money mode" EA usually ignores the cash amount if a non-zero **fixed lot** is also set. Make sure
|
||||
the fixed-lot input is zero when you intend money mode.
|
||||
- For a **quote currency ≠ account currency** pair (e.g. a JPY-quoted pair on a USD account), the
|
||||
EA's money formula can mis-scale the lot by the exchange rate. Validate the very first trade's lot
|
||||
against MT5 before trusting a whole run.
|
||||
|
||||
---
|
||||
|
||||
## 5. Basket exits — break-even and trailing
|
||||
|
||||
For a multi-position basket, the stop is computed on the **average price**, then pushed to every
|
||||
position so they close together:
|
||||
|
||||
```
|
||||
avg = average open price of all open positions, weighted by lot
|
||||
|
||||
# break-even: once the basket is in profit by `break_even%`, lock the stop at avg
|
||||
if break_even% > 0 and (bid − avg) ≥ avg × break_even% / 100:
|
||||
basket_stop = avg
|
||||
|
||||
# trailing: once profit clears (break_even + trailing)%, trail the stop `trailing%` below price
|
||||
if trailing% > 0 and (bid − avg) ≥ avg × (break_even% + trailing%) / 100:
|
||||
candidate = bid − avg × trailing% / 100
|
||||
if candidate > basket_stop: # SMOOTH: move up by any improvement
|
||||
basket_stop = candidate # (STEP variant: only move in trailing-sized jumps)
|
||||
```
|
||||
|
||||
When the bid touches `basket_stop`, every position's stop fires in that sub-tick and the series closes.
|
||||
Single-position strategies are just the `count == 1` case of this.
|
||||
|
||||
---
|
||||
|
||||
## 6. Spread and swap — small models, large effects
|
||||
|
||||
- **Spread.** Prefer a **per-bar spread column** stored in your data (the real historical spread).
|
||||
Fall back to a fixed point value per instrument when history is unreliable (e.g. crypto, where
|
||||
recorded spreads are noisy). Apply spread on the side that costs you: buy at ask, sell at bid.
|
||||
- **Swap.** Charge it once per calendar-day rollover, with a triple charge on the broker's triple-swap
|
||||
weekday (commonly Wednesday for many CFDs). Two models:
|
||||
- *fixed per lot per day:* `swap = rate × lot × multiplier`
|
||||
- *annual % on notional:* `swap = price × annual_pct / 365 × lot × multiplier`
|
||||
Accumulate per position and add it to the trade's PnL on close.
|
||||
|
||||
These look minor but compound over a multi-year, many-position backtest into hundreds of currency
|
||||
units — enough to flip a marginal strategy. Get them from the instrument config (doc 05), never
|
||||
hard-code.
|
||||
|
||||
---
|
||||
|
||||
## 7. Fidelity — how much to trust the Python number
|
||||
|
||||
Your engine is a **fast 4-point approximation** of MT5's finer intra-bar path. Same data, same spread,
|
||||
same swap, same math — the *only* structural difference is intra-bar granularity. The consequences are
|
||||
predictable, and knowing them is what makes the lab trustworthy.
|
||||
|
||||
### What matches MT5 well
|
||||
- Bar data, per-bar spread, swap (with the triple-swap day).
|
||||
- Indicator gates computed from **closed higher-timeframe bars** (e.g. a daily RSI) — these are
|
||||
tick-invariant, so they're identical.
|
||||
- Fill **prices** at pending/grid levels (the level price is the fill price).
|
||||
- Direction, lot sizing, money management.
|
||||
|
||||
### What diverges
|
||||
Anything that depends on the **path inside a bar**: trailing-stop triggers, the *order* a stop vs
|
||||
target is touched, exact fill timing on volatile bars. The divergence **scales with
|
||||
path-sensitivity × volatility**:
|
||||
|
||||
| Strategy character | Typical Python vs MT5 gap |
|
||||
|--------------------|---------------------------|
|
||||
| Clean directional, few exits | small and consistent: Python reads somewhat higher |
|
||||
| Tight trailing / martingale grid in calm years | small |
|
||||
| Tight trailing / martingale grid **in crash years** | **large** — Python's 4 points miss the finer exits MT5 takes, over-crediting big moves |
|
||||
|
||||
A concrete illustration: a trailing strategy might match MT5 within a few currency units in calm,
|
||||
choppy years, yet the Python figure can be several times the MT5 figure across a violent crash year —
|
||||
because MT5's finer path triggers exits at prices the 4-point model skips. The bias is almost always
|
||||
**Python optimistic**, and almost always concentrated in the most volatile episodes.
|
||||
|
||||
### The practical policy (this is the whole point of the two-tier design)
|
||||
1. **Use Python for fast ranking and A/B** — the *relative order* of setups is preserved, which is all
|
||||
the optimizer needs.
|
||||
2. **Apply a pessimistic convention** (the sub-tick order, plus an optional "one adverse re-touch per
|
||||
bar after the favorable extreme" flag for tight-trailing setups) to screen out the worst optimism.
|
||||
3. **MT5-verify every finalist** — the MT5 number is the one you act on.
|
||||
4. **Use real ticks where available** (recent history) for the truest check, accepting the limited
|
||||
window.
|
||||
5. **Expect** only a modest negative gap for clean-directional setups (MT5 a little below Python),
|
||||
and a much larger gap for trailing/grid in volatile history. Measure your own stack's gap on a
|
||||
known preset (doc 07 §8) instead of trusting any rule of thumb. If a finalist's edge is *thin*,
|
||||
assume MT5 will erase it.
|
||||
|
||||
---
|
||||
|
||||
## 8. Validating a new engine (do this before any optimization)
|
||||
|
||||
1. Pick **one known preset** of your EA and a short period (a few months).
|
||||
2. Run it in MT5 (1-minute-OHLC model is fine to start) and save the report.
|
||||
3. Run your Python engine on the **same data, same preset**.
|
||||
4. Reconcile **trade by trade**, then in aggregate. Target gates for a clean-directional setup:
|
||||
- Net difference ≤ ~2%, trade-count difference ≤ ~5%, Profit Factor essentially identical,
|
||||
equity-drawdown difference ≤ ~3%.
|
||||
5. Only when this passes is the engine trustworthy enough to optimize on. Record the comparison as the
|
||||
engine's **baseline fidelity document** and freeze the engine (doc 04).
|
||||
|
||||
> If you can't reconcile, the usual culprits are: signal edge-detection (re-entry every bar), timeframe
|
||||
> mapping/look-ahead, lot-mode mismatch, spread/swap applied on the wrong side or day, or sub-tick
|
||||
> ordering. Walk those five before suspecting anything exotic.
|
||||
|
||||
Next: [`04-isolation-rules.md`](04-isolation-rules.md) — the discipline that keeps a validated engine
|
||||
validated.
|
||||
@@ -0,0 +1,143 @@
|
||||
# 04 — Isolation Rules
|
||||
|
||||
The rules that keep a research lab trustworthy as it grows. Every one exists because the alternative
|
||||
quietly corrupts results. Treat them as non-negotiable; they cost a little friction and save you from
|
||||
optimizing on a silently-broken engine or shipping a number you can't reproduce.
|
||||
|
||||
---
|
||||
|
||||
## Rule 1 — The validated engine is FROZEN
|
||||
|
||||
Once an engine reproduces your EA within the expected fidelity gap (doc 03 §8), it becomes the
|
||||
**satisfactory baseline** and is frozen. Frozen means: **you do not edit it to test an idea.**
|
||||
|
||||
Why so strict: the engine's validation was expensive (trade-by-trade reconciliation against MT5).
|
||||
A one-line "quick tweak" to test a hypothesis can silently shift fills across millions of bars, and
|
||||
now every result the engine produces is suspect — including the ones you already trusted. Freezing
|
||||
protects the validation from regressions you won't notice until much later.
|
||||
|
||||
---
|
||||
|
||||
## Rule 2 — Engine changes happen in an ISOLATED FORK
|
||||
|
||||
Any hypothesis that needs the engine to behave differently is implemented as a **separate engine
|
||||
file** — a copy of the frozen engine plus the minimal hook for the experimental change. Never by
|
||||
editing the frozen file.
|
||||
|
||||
```
|
||||
shared/core/grid_engine.py ← frozen, never touched
|
||||
shared/core/grid_engine_<idea>.py ← fork: copy + one experimental hook
|
||||
```
|
||||
|
||||
The process is fixed:
|
||||
|
||||
1. **Fork** = exact copy of the frozen engine + the experimental change, **defaulting to OFF**.
|
||||
2. **Regression-verify:** run the fork with the change *disabled* and confirm it reproduces the frozen
|
||||
engine **1:1** — identical Net, drawdown, and trade count — on identical data. If it doesn't, your
|
||||
copy isn't clean; fix that before testing anything.
|
||||
3. **A/B** the hypothesis: fork-with-change vs frozen-baseline on the same data.
|
||||
4. **Decide:** significant, robust improvement → consider promotion (Rule 3). Otherwise **delete the
|
||||
fork** — revert is just removing a file, the core was never touched.
|
||||
|
||||
The regression-verify in step 2 is the linchpin. It proves your fork differs from the baseline *only*
|
||||
in the one thing you're testing, so the A/B measures the hypothesis and nothing else.
|
||||
|
||||
---
|
||||
|
||||
## Rule 3 — Promoting a fork to the new baseline is a deliberate, rare event
|
||||
|
||||
A fork stays a fork until it has passed a **full round**: A/B + optimization + MT5 verification + an
|
||||
explicit human decision. Only then does it replace the frozen engine as the new baseline — and at that
|
||||
moment **every preset/EA built on that engine must be re-validated**, because the ground shifted.
|
||||
|
||||
This is never automatic. No script and no agent swaps the baseline engine on its own. It is a manual
|
||||
procedure under human supervision, precisely because it invalidates prior numbers and forces
|
||||
re-verification.
|
||||
|
||||
---
|
||||
|
||||
## Rule 4 — Instruments are isolated as data, not code
|
||||
|
||||
Everything symbol- or broker-specific lives in an `InstrumentConfig` object (doc 05), never in the
|
||||
engine and never hard-coded in a strategy. The engine pulls tick value, spread model, swap, and lot
|
||||
steps *from the config*.
|
||||
|
||||
Consequences:
|
||||
- Testing the same strategy on a different symbol = a different config, **zero engine changes**.
|
||||
- Cost-stress testing = `real` / `worst_case` / `best_case` variants of the same config.
|
||||
- A broker change = edit one file.
|
||||
|
||||
If you ever find yourself writing `if symbol == ...` in the engine, that logic belongs in the config.
|
||||
|
||||
---
|
||||
|
||||
## Rule 5 — Strategies are isolated folders; iterations copy, not import
|
||||
|
||||
Each strategy is a self-contained folder (doc 02 §4). Within it, each **iteration** (one research
|
||||
attempt) carries its **own snapshot** of the run script, search space, and answers.
|
||||
|
||||
- Iterations **copy** glue code rather than importing a shared "current" version. That way an
|
||||
iteration from months ago still runs and reproduces exactly, even after you've changed how you do
|
||||
things. Reproducibility beats DRY here — these are lab notebooks, not production code.
|
||||
- Shared *infrastructure* (engine, indicators, instruments, optimizer) is imported normally; it is
|
||||
import-stable by Rule 1.
|
||||
|
||||
---
|
||||
|
||||
## Rule 6 — Gates and robustness sit OVER the frozen engine, read-only
|
||||
|
||||
Two kinds of add-on never modify the engine:
|
||||
|
||||
- **Gates** (`shared/gates/`) are boolean masks AND-ed into the signal *before* it reaches the engine:
|
||||
`allow = directional_signal & ~block_condition`. A regime filter, a time-of-day filter, an
|
||||
exhaustion filter — all are caller-side masks. The engine still just consumes a signal array.
|
||||
- **Robustness layers** (`shared/robustness/`, doc 06) are read-only analyses *over* a finished result
|
||||
(the study, the trade list, the equity curve). They never touch the engine or mutate results.
|
||||
|
||||
Keeping both outside the engine means you can add or remove a filter or a check without re-validating
|
||||
the engine.
|
||||
|
||||
---
|
||||
|
||||
## Rule 7 — The registry is curated, locked, and append-only
|
||||
|
||||
`registry/` holds only results that passed **all three** gates: Python search → MT5 verification →
|
||||
human approval. Rules for it:
|
||||
|
||||
- **Append-only in spirit:** you don't edit an approved entry; a new finding is a new entry.
|
||||
- **Locked:** registry entries are not casual scratch space. Treat them like committed releases.
|
||||
- **Self-documenting:** each entry records the params, the Python metrics, the MT5 report, and the
|
||||
context (period, instrument, why it was approved).
|
||||
|
||||
The registry is the *trustworthy* slice of everything you ever tried. Scratch experiments you don't
|
||||
intend to keep go in a `results/` scratchpad, never the registry.
|
||||
|
||||
---
|
||||
|
||||
## Rule 8 — Heavy compute runs detached; never relaunch a running job
|
||||
|
||||
A full-history A/B (millions of bars) or a full Optuna study is **minutes** of compute and will block
|
||||
a foreground shell or hit a timeout. Discipline:
|
||||
|
||||
- **Always run heavy jobs in the background**, then poll (read the study's trial count, tail the
|
||||
output file). Don't sit blocked online.
|
||||
- **Smoke-test first:** a tiny run (e.g. 30–40 trials, a short period) validates the script end-to-end
|
||||
before you commit to one full run.
|
||||
- **Dedup guard:** before launching, check whether the same job is already running. Relaunching on a
|
||||
perceived timeout spawns orphan duplicates that thrash the CPU and corrupt nothing but waste
|
||||
everything. One study with N workers beats N competing scripts.
|
||||
|
||||
---
|
||||
|
||||
## Why these rules pay off
|
||||
|
||||
Individually each rule is a small constraint. Together they guarantee three properties that a pile of
|
||||
ad-hoc scripts never has:
|
||||
|
||||
1. **Trust** — the engine that produced a number is the same validated engine, every time.
|
||||
2. **Reproducibility** — any past result can be re-run from its own snapshot.
|
||||
3. **Fast, safe revert** — a failed idea is a deleted fork, not a half-removed change festering in the
|
||||
core.
|
||||
|
||||
Next: [`05-config-and-inputs.md`](05-config-and-inputs.md) — how the test inputs themselves are kept
|
||||
separate and declared.
|
||||
@@ -0,0 +1,180 @@
|
||||
# 05 — Configuration & Input Rules
|
||||
|
||||
Every input to a backtest comes from one of **four declared sources**, never scattered through the
|
||||
code. This is what makes runs reproducible and the engine reusable. The four:
|
||||
|
||||
1. **Instrument config** — everything about the symbol/broker.
|
||||
2. **Search space** — every tunable parameter, its range, its step.
|
||||
3. **Wizard answers** — the run settings for *this* run (period, profile, trials, deposit, …).
|
||||
4. **Frozen baseline** — the parameters you are *not* optimizing, pinned to known values.
|
||||
|
||||
Keep these separate and you can always answer "what produced this number?" exactly.
|
||||
|
||||
---
|
||||
|
||||
## 1. Instrument config — the symbol as data
|
||||
|
||||
One object describes a symbol under one broker. The engine reads **everything** about the instrument
|
||||
from here; nothing about ticks, spread, or swap is ever hard-coded elsewhere.
|
||||
|
||||
### Fields you need
|
||||
|
||||
| Field | Meaning | Where to find it |
|
||||
|-------|---------|------------------|
|
||||
| `name`, `symbol` | human label + broker symbol code | your broker |
|
||||
| `point` | smallest price increment (e.g. `0.01`) | MT5 symbol spec |
|
||||
| `digits` | price decimal places | MT5 symbol spec |
|
||||
| `tick_size`, `tick_value` | price step and **money per lot per point** | MT5 symbol spec |
|
||||
| `contract_size` | units per lot | MT5 symbol spec |
|
||||
| `volume_min`, `volume_step`, `volume_max` | lot rounding/clamping | MT5 symbol spec |
|
||||
| `spread_mode` | `bar_column` (real per-bar) \| `fixed_points` \| `annual_avg` | your choice per data quality |
|
||||
| `spread_fixed_points` | used only with `fixed_points` | broker typical spread |
|
||||
| `swap_mode` | `fixed_per_lot` \| `annual_pct` | broker / your model |
|
||||
| `swap_long`, `swap_short` | per-lot-per-day rates (fixed mode) | MT5 symbol spec |
|
||||
| `swap_annual_pct` | annual rate on notional (percent mode) | your model |
|
||||
| `triple_swap_weekday` | the day swap is charged ×3 (often Wed) | broker |
|
||||
|
||||
> **Get these from MT5's symbol specification** (`Market Watch → right-click symbol → Specification`),
|
||||
> not from memory. A wrong `tick_value` or `contract_size` scales every PnL by a constant and makes
|
||||
> the whole backtest meaningless.
|
||||
|
||||
### Real / worst / best profiles
|
||||
|
||||
Keep **three variants** of each instrument config:
|
||||
|
||||
- **`real`** — your broker's actual conditions. The default for ranking.
|
||||
- **`worst_case`** — wider spread, harsher swap. A finalist that survives this is robust to cost.
|
||||
- **`best_case`** — tighter spread. Shows the ceiling.
|
||||
|
||||
Cost-stress every finalist across all three; if the edge only exists in `best_case`, it isn't real.
|
||||
|
||||
### Two instrument traps (both have bitten real labs)
|
||||
|
||||
- **Quote currency ≠ account currency.** For a pair quoted in a non-account currency (e.g. a JPY-quoted
|
||||
pair on a USD account), an EA's "money mode" lot formula can mis-scale the lot by ~the exchange rate.
|
||||
Fix: generate that pair's presets with an **explicit fixed lot** equal to the intended cash amount
|
||||
÷ contract size, and set the money-amount input to zero. Always check the **first trade's lot**
|
||||
against an MT5 run before trusting the series.
|
||||
- **Unreliable historical spread.** For some instruments (notably crypto) the recorded per-bar spread
|
||||
is garbage. Use `fixed_points` with a realistic value (and a wider `worst_case`) instead of
|
||||
`bar_column`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Search space — declare every tunable parameter in one place
|
||||
|
||||
The parameters Optuna is allowed to vary live in **one declared structure**, with an explicit range
|
||||
and step for each. Nothing tunable is hidden in the strategy body.
|
||||
|
||||
A search space is a mapping of `name → (low, high, step)`, plus a note of which names are integers:
|
||||
|
||||
```
|
||||
SEARCH_SPACE = {
|
||||
"open_distance": (0.1, 1.0, 0.05), # float
|
||||
"grid_step": (0.8, 3.0, 0.10), # float
|
||||
"grid_mult": (0.5, 1.3, 0.10), # float
|
||||
"grid_count": (2, 9, 1), # int
|
||||
"break_even": (0.0, 0.3, 0.05), # float
|
||||
"trailing_stop": (0.1, 0.5, 0.05), # float
|
||||
...
|
||||
}
|
||||
INT_PARAMS = {"grid_count"}
|
||||
```
|
||||
|
||||
The optimizer turns each entry into an Optuna `suggest_float` / `suggest_int` (doc 06). Document the
|
||||
**same** space in a human-readable `parameter-space.md` next to the strategy, with the *reasoning* for
|
||||
each range — why this floor, why this ceiling, what you expect. That note is half the value: it stops
|
||||
you (or your assistant) from re-litigating ranges every iteration and makes overfit ranges obvious.
|
||||
|
||||
### Rules for the search space
|
||||
|
||||
- **Ranges encode priors, not hope.** A range should bracket where the answer plausibly is, informed by
|
||||
the instrument's character. Absurdly wide ranges waste trials and invite overfit corners.
|
||||
- **Step matters.** Too fine a step explodes the space; too coarse misses optima. Match the step to
|
||||
what's meaningful (e.g. a 0.05% grid step, a 1-unit count).
|
||||
- **What is NOT in the space is a decision.** Explicitly list parameters you deliberately exclude
|
||||
(because prior tests showed ~0 impact, or because changing them is a *different* hypothesis). An
|
||||
empty exclusion list usually means you haven't thought about it.
|
||||
- **Timeframe is usually fixed per iteration.** Searching across timeframes in one study is
|
||||
"timeframe-shopping" and overfits; pick the primary timeframe in the stats phase (doc 08) and lock it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Frozen baseline — pin what you don't optimize
|
||||
|
||||
The parameters outside the search space still need *values*. Pin them to the validated baseline (your
|
||||
known-good preset) so the optimizer varies only the few things under test, against an otherwise-known
|
||||
configuration. The run script merges sampled parameters **onto** this frozen baseline before building
|
||||
the engine params. This keeps each study **small and interpretable** — you're measuring the effect of
|
||||
the few tuned knobs, not drowning in a 30-dimensional search.
|
||||
|
||||
---
|
||||
|
||||
## 4. Lot / money mode — the input that silently breaks results
|
||||
|
||||
Position sizing deserves its own section because it is the most common source of "my Python and MT5
|
||||
disagree by a constant factor" bugs. Expose money-based sizing through **two inputs**, plus the
|
||||
fixed-lot guard:
|
||||
|
||||
| Input | Meaning |
|
||||
|-------|---------|
|
||||
| `*_LotAmount` | the cash base. Lot = `LotAmount / open_price / contract_size`. |
|
||||
| `*_LotBalance` | the account balance this base is calibrated for (the scaling window). |
|
||||
| `*_Lot` | a fixed lot in lots. **Must be `0`** to enable money mode — otherwise the EA uses the fixed lot and **ignores** `LotAmount`. |
|
||||
|
||||
Behavior:
|
||||
|
||||
| `LotAmount` | `LotBalance` | Mode | Result |
|
||||
|-------------|--------------|------|--------|
|
||||
| X | **0** | **static** (default for testing) | lot always sized from cash X, independent of balance |
|
||||
| X | Y | **dynamic** | lot = base × (current_balance / Y) — grows/shrinks with the account |
|
||||
|
||||
### Rules
|
||||
|
||||
- **Test with the static lot by default** (`LotBalance = 0`). Compounding (dynamic) is a *production*
|
||||
choice; for research you want a clean, balance-independent measurement.
|
||||
- **Always confirm `*_Lot = 0`** when you intend money mode. A stray non-zero fixed lot is a classic
|
||||
silent bug — the run "works" but sizes every trade wrong.
|
||||
- For a grid, levels multiply from the **base** lot via the grid multiplier (or ATR/other); the base
|
||||
comes from the money rule above.
|
||||
- Mirror whatever mode you choose **identically** in the Python engine and the generated `.set`, or the
|
||||
two tiers won't agree.
|
||||
|
||||
---
|
||||
|
||||
## 5. Wizard — capture run settings once, save them for reproducibility
|
||||
|
||||
Before a run, a small interactive **wizard** asks the handful of settings that change run-to-run and
|
||||
writes them to `wizard-answers.yaml`. That YAML *is* the reproducibility record: anyone (including
|
||||
future-you) can see exactly what produced an iteration's numbers.
|
||||
|
||||
Common wizard questions:
|
||||
|
||||
```
|
||||
period_start / period_end # the backtest window
|
||||
instrument_profile # real / worst_case / best_case
|
||||
trials # Optuna trial budget (e.g. 300 / 1000 / 5000)
|
||||
max_dd_ccy # hard drawdown cap for the constraint
|
||||
top_n_verify # how many finalists to send to MT5
|
||||
fixed_lot / initial_deposit # sizing context
|
||||
```
|
||||
|
||||
Design notes:
|
||||
- Give every question a sensible **default** so a fast run is just pressing Enter.
|
||||
- Make the wizard **extensible**: a base set of common questions plus a per-strategy hook for extra
|
||||
ones (e.g. a grid wizard adds depth/multiplier defaults).
|
||||
- **Write the answers to disk** next to the iteration. A run with no saved settings is not reproducible
|
||||
and shouldn't be promoted.
|
||||
|
||||
---
|
||||
|
||||
## Putting it together
|
||||
|
||||
A single run is fully specified by: **which instrument config** (symbol + profile), **which search
|
||||
space** (the tunable parameters), **which frozen baseline** (the pinned rest), and **which wizard
|
||||
answers** (the window, trials, deposit). Four declared sources, each in its own place. That is the
|
||||
whole "separate inputs" discipline — and it's why any run in this lab can be explained and reproduced
|
||||
exactly.
|
||||
|
||||
Next: [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md) — turning the search
|
||||
space into an Optuna study and not fooling yourself with the result.
|
||||
@@ -0,0 +1,168 @@
|
||||
# 06 — Optimization (Optuna) & Robustness
|
||||
|
||||
How to turn the declared search space into a Bayesian search that finds good parameters fast — and how
|
||||
to avoid the trap that ruins most optimization: **overfitting to the past.** A great backtest is easy;
|
||||
a robust one is the job.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why Optuna (and not a grid)
|
||||
|
||||
A parameter grid is exponential: 8 parameters × 10 values each = 100 million combinations. Even at
|
||||
"seconds per backtest" that's infeasible, and most of it is wasteland.
|
||||
|
||||
**Optuna** (<https://optuna.org>, docs <https://optuna.readthedocs.io>) uses a **TPE sampler** (Tree-
|
||||
structured Parzen Estimator) that *learns* the shape of the objective as it goes and spends trials
|
||||
where results are promising. You get a strong optimum in **hundreds to a few thousand** trials.
|
||||
|
||||
Install: `pip install optuna` (doc 01). Core concepts you'll use:
|
||||
|
||||
- **Study** — one optimization. Maximizes (or minimizes) an objective. Persisted to SQLite so it can
|
||||
be resumed, inspected mid-run, and parallelized.
|
||||
- **Trial** — one parameter sample + its score. `trial.suggest_float/int(name, low, high, step=…)`
|
||||
draws each parameter; `trial.set_user_attr(...)` stashes the metrics for later inspection.
|
||||
- **Sampler** — TPE by default; fine to start.
|
||||
|
||||
```python
|
||||
import optuna
|
||||
study = optuna.create_study(
|
||||
direction="maximize",
|
||||
storage="sqlite:///study.db", # persist → resume, inspect, parallelize
|
||||
study_name="my_iter",
|
||||
load_if_exists=True,
|
||||
)
|
||||
study.optimize(objective, n_trials=1000, n_jobs=4) # n_jobs = parallel workers
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. The objective function
|
||||
|
||||
The objective receives a trial, **samples** parameters from the search space, **runs** the engine, and
|
||||
returns a **score**. Structure it in four clear steps:
|
||||
|
||||
```
|
||||
def objective(trial):
|
||||
1. sampled = suggest_params(trial) # from SEARCH_SPACE (doc 05)
|
||||
2. params = build_params(sampled, frozen_baseline)
|
||||
3. result = engine.run(params, bars, instrument, deposit)
|
||||
metrics = compute_metrics(result)
|
||||
4. score, violations = score_metrics(metrics, sampled)
|
||||
for k, v in metrics.items(): trial.set_user_attr(k, v)
|
||||
trial.set_user_attr("violations", violations)
|
||||
return score
|
||||
```
|
||||
|
||||
### Score design
|
||||
|
||||
A good single-objective score balances **reward against risk**, for example:
|
||||
|
||||
```
|
||||
score = Net − DD_WEIGHT × EquityDrawdown # reward profit, punish the drawdown that earned it
|
||||
```
|
||||
|
||||
The weight makes the optimizer prefer a slightly smaller, much calmer equity curve over a fragile spike.
|
||||
(Optuna also supports true multi-objective studies — e.g. maximize Net *and* minimize DD on a Pareto
|
||||
front — but a weighted single objective is simpler and usually enough to start.)
|
||||
|
||||
### Constraints as hard rejections, not soft nudges
|
||||
|
||||
Some conditions should **disqualify** a trial outright, not just dock points. Implement them as a large
|
||||
penalty per violation so violators sort to the bottom:
|
||||
|
||||
```
|
||||
violations = []
|
||||
if metrics.trades < MIN_TRADES: violations.append("too few trades")
|
||||
if metrics.profit_factor < MIN_PF: violations.append("PF too low")
|
||||
if metrics.equity_dd > HARD_DD_CAP: violations.append("DD over cap")
|
||||
score = base_score − PENALTY * len(violations) # PENALTY huge, e.g. 1e6
|
||||
```
|
||||
|
||||
Typical constraints worth enforcing:
|
||||
|
||||
- **Minimum trade count** (a hard floor, calibrated to the window — e.g. ≥25–30 active trades per
|
||||
year). Few-trade, high-PF results are almost always overfit luck, not edge.
|
||||
- **Minimum Profit Factor.**
|
||||
- **Hard drawdown cap** in account currency (e.g. ≤35% of deposit).
|
||||
- **A path-independent fragility guard.** For a martingale grid, compute the *worst-case floating
|
||||
drawdown* if a full series accumulates straight down an N% drop, and reject if it's worse than the
|
||||
baseline's. This catches "the backtest never hit the bad path, but the configuration is a time bomb"
|
||||
— the failure mode a pure historical backtest can't see.
|
||||
|
||||
---
|
||||
|
||||
## 3. Don't pick the top-N by score — pick the top-N **diverse**
|
||||
|
||||
Optuna converges: the top 10 trials by score are usually near-clones of one peak. Verifying 3 clones in
|
||||
MT5 tells you nothing about robustness. Instead, pick **meaningfully different** finalists.
|
||||
|
||||
**Greedy diverse selection:**
|
||||
|
||||
```
|
||||
1. filter trials by the constraints (drop violators)
|
||||
2. sort by score
|
||||
3. start the selected set with the single best trial
|
||||
4. repeatedly add the remaining trial with the MAXIMUM parameter-distance to the already-selected set,
|
||||
lightly weighted by its rank so you still prefer strong scores
|
||||
5. stop at N (2–3 is plenty)
|
||||
```
|
||||
|
||||
Parameter distance = average normalized difference across all parameters (normalize each to [0,1]:
|
||||
numeric by min–max, boolean to 0/1, categorical by index). The result is 2–3 finalists that are each
|
||||
*good* but reach the result by *different* parameter regions — if they all survive MT5, you have real
|
||||
robustness, not one lucky corner.
|
||||
|
||||
---
|
||||
|
||||
## 4. Robustness layers — proving the result isn't overfit
|
||||
|
||||
These are **read-only analyses over a finished result** (the study, the trade list, the equity curve).
|
||||
They never touch the engine. Run them on finalists *before* spending MT5 time. Treat them as
|
||||
report-only signals by default; tighten into hard gates as you gain confidence.
|
||||
|
||||
| Layer | Question it answers | Method |
|
||||
|-------|---------------------|--------|
|
||||
| **Stability region** | Is the winner on a *plateau* of good configs, or a lone spike? | Cluster the study's top trials by parameter distance; prefer a finalist from the center of a dense good region, not an isolated peak. |
|
||||
| **Neighborhood / sensitivity** | Does a small parameter nudge destroy the edge? | Perturb each lever ±1 step; demand all neighbors stay profitable. A fragile optimum fails this. |
|
||||
| **Walk-forward (WFE)** | Does the edge hold out-of-sample? | Split into in-sample/out-of-sample windows; with the finalist's *fixed* params (no re-fit), compute `OOS_metric / IS_metric`. Add a **purge gap** between IS and OOS so leakage can't help. |
|
||||
| **Monte-Carlo permutation** | How lucky was the trade *order*? | Shuffle trade order, drop a fraction, jitter PnL; build a distribution of drawdown/Net. A finalist whose real drawdown sits in the ugly tail is fragile. |
|
||||
| **Deflated Sharpe (DSR)** | Is the Sharpe real after testing thousands of configs? | Adjust the observed Sharpe for the **number of trials** (multiple-testing). Many trials inflate the best result even under pure noise; DSR estimates the probability the true Sharpe beats a deflated benchmark. |
|
||||
| **Era split** | Does the edge exist in *both halves* of history? | Run the finalist on era-1 vs era-2 separately; an edge present in only one era is regime-luck. |
|
||||
| **Cost stress** | Does it survive realistic and adverse costs? | Re-run on `real` / `worst_case` / `best_case` instrument profiles (doc 05). Edge only in `best_case` = no edge. |
|
||||
|
||||
> These are standard quant techniques, not exotic. Walk-forward and Monte-Carlo catch the everyday
|
||||
> overfit; the Deflated Sharpe specifically counters the fact that an exhaustive search *will* surface
|
||||
> an impressive-looking config from noise. References worth reading: López de Prado on the Deflated
|
||||
> Sharpe Ratio and the dangers of backtest overfitting.
|
||||
|
||||
---
|
||||
|
||||
## 5. What qualifies as a "finalist"
|
||||
|
||||
Be strict — MT5 time is the expensive resource, and a weak finalist wastes it:
|
||||
|
||||
- **From a *completed* search only.** An interrupted/partial Optuna study yields *preliminary*
|
||||
numbers, not verification candidates. The README status must reflect the real process state, never
|
||||
"continues" if the process is dead.
|
||||
- **Passes the hard constraints** (trades, PF, DD cap, fragility guard).
|
||||
- **Not concentrated in one year.** If a single year carries >~40% of total Net, it's fragile — one
|
||||
regime is doing all the work.
|
||||
- **Honest about flat periods.** A gate that sat out a bad year "survived by not trading" — describe it
|
||||
that way, don't dress it up as robust profit.
|
||||
- **Survives the robustness layers** you've chosen to enforce.
|
||||
- **Converge to 2–3 diverse finalists**, not 5–6. More finalists is usually indecision, not rigor.
|
||||
|
||||
---
|
||||
|
||||
## 6. Running searches without melting your machine (operational discipline)
|
||||
|
||||
- **Smoke first.** A 30–40 trial run on a short period validates the whole script (data load → engine →
|
||||
scoring → storage) in under a minute. Only then launch the full study.
|
||||
- **Run heavy studies in the background and poll** the trial count in `study.db`; don't block a shell on
|
||||
a multi-minute run.
|
||||
- **One study, many workers** (`n_jobs=N`) beats N competing scripts. Cap heavy concurrency at your
|
||||
performance-core count.
|
||||
- **Dedup guard:** never relaunch a study you think timed out without checking it isn't still running —
|
||||
duplicates thrash the CPU and waste hours.
|
||||
|
||||
Next: [`07-mt5-bridge.md`](07-mt5-bridge.md) — verifying the finalists in the real terminal.
|
||||
@@ -0,0 +1,236 @@
|
||||
# 07 — The MetaTrader 5 Bridge
|
||||
|
||||
The bridge connects your fast Python search to the **gold-standard** MetaTrader 5 Strategy Tester. It
|
||||
does four things: **(1) compile** your EA, **(2) auto-run** a backtest from a generated config,
|
||||
**(3) parse** the HTML report, **(4) compare** Python vs MT5 side by side.
|
||||
|
||||
The bridge has two flavors depending on **where MT5 runs** (doc `CLAUDE.md` Phase 0). Read the one
|
||||
that matches your topology; the parsing and comparison steps are identical for both.
|
||||
|
||||
```
|
||||
┌──────────────────────────── Topology A (all-Windows, simplest) ───────────────┐
|
||||
Python ──► generate .set + tester.ini ──► launch terminal64.exe /config: locally ──► report.htm
|
||||
└────────────────────────────────────────────────────────────────────────────────┘
|
||||
┌──────────────────── Topology B/C (Unix + remote/VM Windows) ───────────────────┐
|
||||
Python ──► generate .set + tester.ini ──► copy to Windows ──► scheduled-task run ──► poll ──► pull report.htm
|
||||
└────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
parse (UTF-16 HTML → metrics dict)
|
||||
│
|
||||
comparison table: Python vs MT5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Compiling your EA
|
||||
|
||||
The tester runs a compiled `.ex5`. You compile from `.mq5` with **MetaEditor** (ships with MT5):
|
||||
|
||||
- **In the GUI:** open the `.mq5` in MetaEditor → `Compile` (F7). Fix errors, confirm a clean compile.
|
||||
- **From the command line** (scriptable, useful for the bridge):
|
||||
|
||||
```bat
|
||||
:: Windows
|
||||
"C:\Program Files\MetaTrader 5\metaeditor64.exe" /compile:"C:\path\to\Expert.mq5" /log
|
||||
```
|
||||
|
||||
Notes that save hours:
|
||||
- **Custom indicators** the EA calls must be compiled too and placed in `MQL5\Indicators\`.
|
||||
- If your EA's `#include` files live in a non-standard folder, compile the terminal in **portable
|
||||
mode** (`/portable`) so MetaEditor resolves includes from that terminal's `MQL5\Include\`.
|
||||
- After compiling, the `.ex5` goes in `MQL5\Experts\`. The tester references it by name.
|
||||
|
||||
---
|
||||
|
||||
## 2. What MT5 needs to auto-run a test
|
||||
|
||||
Two generated files plus the binaries already in place:
|
||||
|
||||
### 2a. The `.set` file (parameters)
|
||||
|
||||
A `.set` is the EA's inputs serialized as `key=value` lines. **MT5 writes `.set` files as UTF-16-LE**
|
||||
— generate yours in the same encoding or the tester silently ignores them.
|
||||
|
||||
Your bridge's **set generator** takes the *same* parameter dict you backtested in Python and writes the
|
||||
matching `.set`. The mapping from Python parameter names to EA input names is strategy-specific — keep
|
||||
a small **mapping table** documented next to the strategy so the two tiers always agree. Watch the
|
||||
enum-valued inputs (mode flags, timeframe codes): MT5 inputs are often integers, so map e.g. a
|
||||
timeframe to its `ENUM_TIMEFRAMES` integer, a mode to its enum index.
|
||||
|
||||
> **The lot-mode guard from doc 05 applies here too:** when you generate the `.set`, make sure the
|
||||
> fixed-lot input is `0` if you intend money mode, and handle the quote-currency lot trap. A
|
||||
> mismatched `.set` is the #1 reason a verified MT5 number disagrees with Python.
|
||||
|
||||
### 2b. The `tester.ini` (run config)
|
||||
|
||||
A small INI tells the tester *what* to run:
|
||||
|
||||
```ini
|
||||
[Tester]
|
||||
Expert=Experts\YourEA.ex5
|
||||
Symbol=YOURSYMBOL
|
||||
Period=H1 ; chart timeframe (≤ the EA's signal timeframe)
|
||||
Model=1 ; tick model — see below
|
||||
FromDate=2024.01.01
|
||||
ToDate=2026.01.01
|
||||
Deposit=10000
|
||||
Leverage=100
|
||||
Report=report_myrun ; output HTML name
|
||||
ShutdownTerminal=1 ; close the terminal when done (so the bridge knows it's finished)
|
||||
|
||||
[TesterInputs]
|
||||
; either point at the .set, or inline the key=value inputs here
|
||||
```
|
||||
|
||||
The login section (`[Common]`) carries the demo account; keep credentials out of code (section 5).
|
||||
|
||||
### Tick models (the speed/accuracy dial)
|
||||
|
||||
| Model | Name | Accuracy | Speed (≈ 2-year run) |
|
||||
|-------|------|----------|----------------------|
|
||||
| 0 | Every tick based on **real ticks** | highest (truest fills) | slowest (~10–30 min) |
|
||||
| 1 | Every tick (generated from M1) | high | medium |
|
||||
| 2 | **1-minute OHLC** | good for non-tick-sensitive | fast (~2–3 min) |
|
||||
| 4 | Open prices only | rough | fastest (~30 s) |
|
||||
|
||||
Use **Model 2 (1-min OHLC)** for routine verification; **Model 0 (real ticks)** for the final check on
|
||||
anything path-sensitive (trailing/grid). For a pure open-bar strategy even Model 4 is exact.
|
||||
|
||||
---
|
||||
|
||||
## 3. Topology A — all on Windows (recommended, simplest)
|
||||
|
||||
No SSH, no scheduled tasks. Two ways, easiest first:
|
||||
|
||||
### 3a. The official `MetaTrader5` Python package
|
||||
|
||||
`pip install MetaTrader5` (Windows-only) lets Python drive the terminal directly — initialize, pull
|
||||
historical bars, and read symbol info. Use it for **data download** (section 6) and terminal control.
|
||||
This is the cleanest path on Windows and removes most of the plumbing the remote topologies need.
|
||||
|
||||
### 3b. Launch the tester from a generated config
|
||||
|
||||
Write `tester.ini`, then launch the terminal pointed at it:
|
||||
|
||||
```bat
|
||||
start "" "C:\Program Files\MetaTrader 5\terminal64.exe" /config:"C:\path\to\tester.ini"
|
||||
```
|
||||
|
||||
`ShutdownTerminal=1` makes the terminal close itself when the run finishes; your Python wrapper waits
|
||||
for the process to exit (or for the report file to appear), then parses the report. Because everything
|
||||
is local, there is no copy/poll step — read the HTML straight from the terminal's report folder.
|
||||
|
||||
> Topology A is the right starting point for a Windows user. Get this working before considering
|
||||
> anything remote.
|
||||
|
||||
---
|
||||
|
||||
## 4. Topology B/C — Python on Unix, MT5 on a remote/VM Windows
|
||||
|
||||
You develop on macOS/Linux and MT5 lives elsewhere (a Windows VPS, or a local VM). Two quirks force a
|
||||
specific design:
|
||||
|
||||
### Quirk 1 — a second, **portable** MT5 install
|
||||
If that Windows box already runs a *live* MT5 for something else, the tester would fight it for chart
|
||||
windows. Install a **separate portable** MT5 in its own folder (e.g. `C:\MT5_Tester\`) with its own
|
||||
data directory. Copy `config\` (the saved login), `bases\` (symbol metadata), and `profiles\` from the
|
||||
main terminal so the portable one auto-logs-in and knows your symbols. One-time setup.
|
||||
|
||||
### Quirk 2 — SSH-launched processes die on disconnect
|
||||
A process you start over SSH gets killed when the SSH session closes — the tester would die after a few
|
||||
seconds. **Solution: a one-shot Windows Task Scheduler task.** Your wrapper SSHes in, creates a task
|
||||
scheduled ~20 seconds out, and disconnects. The task fires *independently*, runs the full backtest, and
|
||||
the terminal shuts itself down.
|
||||
|
||||
### The end-to-end flow
|
||||
|
||||
```
|
||||
1. Python: parse the .set → build tester.ini (inject demo login from the env file)
|
||||
2. copy tester.ini + .set to the Windows box (scp / shared folder)
|
||||
3. SSH: run a small PowerShell that registers a one-shot scheduled task ~20s out, then disconnect
|
||||
4. the task fires → portable terminal64.exe /config:tester.ini → backtest → save report → shut down
|
||||
5. Python: poll the Windows box for the report_*.htm file
|
||||
6. when present (and the terminal has exited): copy the report back
|
||||
7. parse → metrics
|
||||
```
|
||||
|
||||
The two tiny helper scripts that live on the Windows box (a `.bat` that launches the terminal and a
|
||||
`.ps1` that registers the scheduled task and clears old reports) are the only Windows-side code; keep
|
||||
mirror copies in your repo so you can re-deploy them.
|
||||
|
||||
### Manual fallback
|
||||
If the automated run breaks, you can always: copy the `.set` over, RDP in, open `Strategy Tester`
|
||||
(Ctrl+R), load the preset, Start, then `Save as Report`, and copy the HTML back. Slow but unblocks you.
|
||||
|
||||
---
|
||||
|
||||
## 5. Credentials — never in code or chat
|
||||
|
||||
Broker demo login lives in a **gitignored env file** (e.g. `.env`):
|
||||
|
||||
```
|
||||
MT5_DEMO_LOGIN=...
|
||||
MT5_DEMO_PASSWORD=...
|
||||
MT5_DEMO_SERVER=YourBroker-Demo
|
||||
```
|
||||
|
||||
The bridge reads these at runtime and injects them into `tester.ini`'s `[Common]` section **without
|
||||
printing them**. Alternatively, copying the terminal's `accounts.dat` into the portable install gives
|
||||
auto-login with no password in any config at all — the cleanest option.
|
||||
|
||||
---
|
||||
|
||||
## 6. Downloading historical data (feeding the Python tier)
|
||||
|
||||
The Python engine needs the **same** bars MT5 uses. Two ways to get them:
|
||||
|
||||
- **Via the `MetaTrader5` package (Windows):** `mt5.copy_rates_range(symbol, timeframe, from, to)` →
|
||||
a numpy array → save as Parquet. Pull M1 for the full window; resample to higher timeframes in Python.
|
||||
- **Via an export script in MT5:** a small MQL5 script that walks `CopyRates` year by year and writes
|
||||
CSV, which you then convert to Parquet. Useful when Python can't reach the terminal.
|
||||
|
||||
For a brand-new symbol you must first **let MT5 cache its history** (open a chart, set
|
||||
`Tools → Options → Charts → Max bars: Unlimited`, scroll back) so the tester and the export have enough
|
||||
bars. Confirm the exact broker symbol code (it varies: indices and metals especially) before exporting.
|
||||
|
||||
---
|
||||
|
||||
## 7. Parsing the report
|
||||
|
||||
MT5 saves the Strategy Tester report as **HTML encoded UTF-16-LE** — not UTF-8. Read it with the right
|
||||
encoding, parse with `lxml`/`html5lib`, and pull the key fields:
|
||||
|
||||
- `Total Net Profit`
|
||||
- `Profit Factor`
|
||||
- `Total Trades`
|
||||
- `Balance Drawdown Maximal` and **`Equity Drawdown Maximal`** (use **Maximal**, not Absolute — it's
|
||||
the peak-to-trough that matters)
|
||||
|
||||
Emit a `metrics` dict and (optionally) dump it to JSON next to the run. Validate the report is real
|
||||
before trusting it (e.g. file size over some floor and a non-zero bar count) — a truncated report means
|
||||
the run failed.
|
||||
|
||||
---
|
||||
|
||||
## 8. The comparison table
|
||||
|
||||
For every finalist, write an `auto-verification.md` that puts the two tiers side by side:
|
||||
|
||||
| Metric | Python | MT5 | Δ |
|
||||
|--------|--------|-----|---|
|
||||
| Net | … | … | …% |
|
||||
| Profit Factor | … | … | … |
|
||||
| Equity DD max | … | … | …% |
|
||||
| Total trades | … | … | …% |
|
||||
|
||||
Then judge the delta **against the expected fidelity gap** (doc 03 §7): a clean-directional setup
|
||||
should show only a modest negative gap (MT5 a little below Python); a trailing/grid setup in volatile
|
||||
history can gap much wider, and that's *expected*, not a bug. The decision rule: if the **MT5** number still clears your bar after the gap,
|
||||
the finalist is real; if the edge only existed in the optimistic Python figure, discard it.
|
||||
|
||||
> A useful warm-up calibration: pick one known preset and run the full Python-vs-MT5 comparison on it
|
||||
> first. It tells you *your* stack's actual gap for *your* EA, so later finalists are judged against a
|
||||
> measured baseline rather than a generic rule of thumb.
|
||||
|
||||
Next: [`08-workflow-cycle.md`](08-workflow-cycle.md) — the repeatable loop that ties all of this
|
||||
together.
|
||||
@@ -0,0 +1,140 @@
|
||||
# 08 — The Workflow Cycle
|
||||
|
||||
Everything above is parts; this is the assembly line. One research idea travels through a fixed,
|
||||
repeatable cycle. The discipline of the cycle — especially the **progressive scope** (test the
|
||||
*smallest* thing first) — is what keeps you from overfitting and what makes results explainable.
|
||||
|
||||
---
|
||||
|
||||
## 1. The cycle
|
||||
|
||||
```
|
||||
1. HYPOTHESIS — state the idea in one sentence, grounded in an existing engine + preset.
|
||||
2. SCAFFOLD — create the iteration folder + run script + declared search space + run notes.
|
||||
3. STATS — measure signals & filters on real history. Pick the primary timeframe. No tuning yet.
|
||||
4. MINIMAL SCOPE — test the smallest version of the idea. Is there ANY edge? If no → stop here.
|
||||
5. EXPAND + OPTUNA— only if minimal scope showed edge: add layers one at a time, then Optuna search.
|
||||
6. MT5-VERIFY — run the 2–3 diverse finalists in the real terminal. Python ranks; MT5 decides.
|
||||
7. APPROVAL — a human looks at the comparison and explicitly decides.
|
||||
8. PROMOTE — approved result is copied into the locked registry with full context.
|
||||
```
|
||||
|
||||
The early steps are cheap and exist to **kill bad ideas before they cost compute**. Most ideas should
|
||||
die at step 3 or 4. That's the system working, not failing.
|
||||
|
||||
---
|
||||
|
||||
## 2. Step-by-step
|
||||
|
||||
### Step 1 — Hypothesis
|
||||
Write it down in one sentence and tie it to something that already exists: *"On instrument X, adding a
|
||||
daily-trend filter to preset Y should cut the drawdown from the counter-trend series without killing
|
||||
Net."* A hypothesis that isn't grounded in an existing engine + baseline preset is a research project,
|
||||
not an iteration — narrow it first.
|
||||
|
||||
### Step 2 — Scaffold
|
||||
Create the iteration folder by convention (doc 02 §4):
|
||||
|
||||
```
|
||||
strategies/<strategy>/iterations/<base-preset>-<approach>-<YYYY-MM-DD>/
|
||||
├── README.md # the hypothesis, status, and (later) the result
|
||||
├── parameter-space.md # the declared search space + the reasoning for each range
|
||||
├── optimize.py # a self-contained snapshot of the run script
|
||||
└── wizard-answers.yaml # written by the wizard at run time
|
||||
```
|
||||
|
||||
Copy the run script from a previous iteration as a template and edit it — each iteration owns its
|
||||
snapshot (doc 04 Rule 5). A data/profile pre-check belongs here: confirm the instrument's data and
|
||||
config exist *before* scaffolding (for a brand-new symbol, do the data-download + config step from doc
|
||||
07 §6 / doc 05 §1 first).
|
||||
|
||||
### Step 3 — Stats (measure before you tune)
|
||||
Before any parameter search, **characterize the signals on real history**:
|
||||
|
||||
- How often does the signal fire? What's the raw win rate, the average favorable vs adverse excursion?
|
||||
- **Multi-timeframe filter analysis:** if the idea is a trend/regime filter, measure how well it
|
||||
separates good from bad entries on *several* timeframes (e.g. M15/H1/H4/D1) and pick the **one
|
||||
primary timeframe** where the separation is cleanest. Lock it for the rest of the iteration.
|
||||
- Output: a short stats note + the chosen primary timeframe + the specific levers worth tuning. **No
|
||||
parameter sweep yet.**
|
||||
|
||||
This step routinely ends an iteration: if the signal has no separation on any timeframe, there's
|
||||
nothing to optimize, and you've spent minutes instead of hours learning that.
|
||||
|
||||
### Step 4 — Minimal scope (the anti-overfit gate)
|
||||
Test the **smallest possible version** of the idea against the baseline:
|
||||
|
||||
- Core engine behavior + the new filter, with **exits minimal** — e.g. for a trend strategy: trailing
|
||||
only, *no* break-even; for a grid: the first level only, *no* depth/martingale/second tier.
|
||||
- One question: **is there any edge at all** over the baseline? A handful of A/B runs, not a search.
|
||||
- If the minimal version shows no edge, **stop**. Piling parameters onto a non-edge just manufactures an
|
||||
overfit that looks good in-sample and dies in MT5. Most ideas should end here.
|
||||
|
||||
### Step 5 — Expand + Optuna (only if step 4 showed edge)
|
||||
Now add complexity **one layer at a time**, each as an A/B against the previous best:
|
||||
|
||||
- add break-even, then a second grid tier, then an ATR-based stop, then wider ranges — *in sequence*,
|
||||
keeping only what improves things.
|
||||
- when the structure is set, run the **Optuna study** (doc 06): smoke-test first, then the full run in
|
||||
the background. Apply the constraints and the fragility guard. Select **2–3 diverse finalists** (not
|
||||
top-N-by-score). Run the robustness layers on them.
|
||||
|
||||
Progressive expansion keeps every result interpretable ("the second tier added X, the ATR stop added
|
||||
Y") and keeps the search space small enough that Optuna isn't just mining noise.
|
||||
|
||||
### Step 6 — MT5-verify (finalists only)
|
||||
Run the 2–3 finalists through the bridge (doc 07). Use a fast tick model to start, real ticks for the
|
||||
final path-sensitive check. Write the `auto-verification.md` comparison and judge each delta against the
|
||||
expected fidelity gap (doc 03 §7). **Don't verify every step of the search** — you trust the Python
|
||||
engine for ranking; MT5 is reserved for the numbers that might go live.
|
||||
|
||||
### Step 7 — Approval
|
||||
A human reviews the comparison and the robustness signals and **explicitly** decides: promote, discard,
|
||||
or iterate. Nothing auto-promotes. This is the gate where judgment — does this make sense, is the
|
||||
drawdown livable, is the trade count real — overrides any single metric.
|
||||
|
||||
### Step 8 — Promote
|
||||
On approval, copy the finalist into the **locked registry** (doc 04 Rule 7) with everything needed to
|
||||
trust and reproduce it: the params, the Python metrics, the MT5 report, and the context (period,
|
||||
instrument, why approved). Discarded iterations are archived, not deleted — a negative result is data,
|
||||
and re-testing a known dead end is waste.
|
||||
|
||||
---
|
||||
|
||||
## 3. When to start a new iteration vs use the scratchpad
|
||||
|
||||
- **New iteration** (full folder, will be tracked): a genuine improvement idea for a specific baseline;
|
||||
an engine update that requires re-verifying existing approved results; new data/period to re-test on.
|
||||
- **Scratchpad** (`results/` — never promoted): a quick experiment you don't intend to keep. Don't
|
||||
pollute `iterations/` or the registry with throwaways.
|
||||
|
||||
---
|
||||
|
||||
## 4. Naming convention
|
||||
|
||||
```
|
||||
<base-preset>-<approach>-<YYYY-MM-DD>/
|
||||
```
|
||||
|
||||
e.g. `trend-filter-daily-ema-2026-06-15`, `grid-second-tier-atr-2026-07-01`. The *approach* word is
|
||||
free-form (coin it as needed); the date is the day of creation. Consistent names make the `iterations/`
|
||||
folder a readable history of what you tried and when.
|
||||
|
||||
---
|
||||
|
||||
## 5. The cycle as a habit
|
||||
|
||||
Run it the same way every time and three good things happen automatically:
|
||||
|
||||
- **Bad ideas die cheap** (steps 3–4), so compute goes to ideas with a pulse.
|
||||
- **Surviving ideas are interpretable** (step 5's one-layer-at-a-time), so you understand *why* they
|
||||
work — which is what tells you whether they'll keep working.
|
||||
- **Promoted results are trustworthy and reproducible** (steps 6–8), so the registry stays a source of
|
||||
truth rather than a graveyard of optimistic backtests.
|
||||
|
||||
That's the whole lab: a fast engine you trust, a disciplined search you can't fool yourself with, and a
|
||||
real-terminal check before anything counts.
|
||||
|
||||
---
|
||||
|
||||
*Back to [`README.md`](README.md) · setup playbook [`CLAUDE.md`](CLAUDE.md).*
|
||||
@@ -0,0 +1,183 @@
|
||||
# CLAUDE.md — Adaptive Setup Playbook for an AI Coding Assistant
|
||||
|
||||
> This file is written **for an AI coding assistant** (Claude Code, Cursor, Copilot Chat, etc.) so
|
||||
> it can help its human build this backtesting lab **from scratch on their specific machine**.
|
||||
> It also works as a generic `AGENTS.md`. If you are a human, you can follow it too — just perform
|
||||
> the steps yourself instead of asking the assistant.
|
||||
>
|
||||
> **Golden rule for the assistant:** this stack is reproduced *from your human's own EAs and
|
||||
> presets*, not from any bundled code. Never invent a trading strategy or fabricate parameters.
|
||||
> When trading logic is needed, ask the human for their MQL5 source and presets.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Profile the device (do this first, always)
|
||||
|
||||
Before installing anything, detect the environment. The whole setup branches on **where MT5 runs**.
|
||||
|
||||
Run and read the output:
|
||||
|
||||
```bash
|
||||
# OS + architecture
|
||||
uname -a 2>/dev/null || ver # macOS/Linux print uname; Windows cmd prints `ver`
|
||||
python3 --version || python --version
|
||||
```
|
||||
|
||||
Then classify the human into **one of three topologies** and confirm it with them before proceeding:
|
||||
|
||||
| Topology | Python runs on | MT5 runs on | Bridge needed | Best for |
|
||||
|----------|----------------|-------------|---------------|----------|
|
||||
| **A — All-Windows (simplest)** | the Windows machine | the same Windows machine | none — direct, via the `MetaTrader5` pip package or local file paths | a Windows user; recommended starting point |
|
||||
| **B — Unix + remote Windows** | macOS/Linux | a separate Windows box or VPS | SSH + file copy + a scheduled-task trick | someone on a Mac/Linux who already has a Windows VPS |
|
||||
| **C — Unix + local Windows VM** | macOS/Linux host | Windows in a VM/Parallels/UTM | shared folder or SSH into the VM | a Mac/Linux user without a separate Windows box |
|
||||
|
||||
> **Recommendation to give the human:** if they are on Windows, use **Topology A** — it is by far
|
||||
> the least moving parts. The official `MetaTrader5` Python package only runs on Windows and lets
|
||||
> Python drive the terminal directly, so no SSH/scheduled-task plumbing is needed. Topologies B/C
|
||||
> exist only because some people develop on a Mac/Linux and keep MT5 elsewhere.
|
||||
|
||||
Record the chosen topology — every later step references it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Install the Python stack
|
||||
|
||||
Follow [`01-stack-and-install.md`](01-stack-and-install.md). In short:
|
||||
|
||||
1. Install **Python 3.12+** (3.13/3.14 fine). On Windows, install from python.org and tick
|
||||
*"Add Python to PATH"*.
|
||||
2. Create an **isolated virtual environment** in the project root (`.venv`). Never install into the
|
||||
system Python.
|
||||
3. Install the libraries (`pandas`, `numpy`, `pyarrow`, `optuna`, `numba`, `sqlalchemy`, `pyyaml`,
|
||||
`lxml`, `html5lib`, `tqdm`, `requests`). Exact commands and *why each library exists* are in doc 01.
|
||||
4. On **Topology A** only, also `pip install MetaTrader5` (Windows-only package).
|
||||
|
||||
Verify the venv imports everything before moving on.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Scaffold the repository skeleton
|
||||
|
||||
Create the directory layout from [`02-architecture.md`](02-architecture.md). Build **empty modules
|
||||
with docstrings and type signatures first** — do not write trading logic yet. The skeleton:
|
||||
|
||||
```
|
||||
your-lab/
|
||||
├── .venv/ # virtual environment (gitignored)
|
||||
├── data/ # downloaded market data, parquet (gitignored)
|
||||
├── shared/ # reusable infrastructure
|
||||
│ ├── core/ # backtest engines (you fill these from your EA logic)
|
||||
│ ├── indicators/ # indicator functions (RSI, ATR, EMA, your custom ones)
|
||||
│ ├── instruments/ # per-symbol/broker config objects
|
||||
│ ├── data/ # data loaders + MT5 report parser
|
||||
│ ├── optimizer/ # objective + diverse top-N selector
|
||||
│ ├── robustness/ # anti-overfit layers
|
||||
│ ├── gates/ # reusable entry-filter masks
|
||||
│ ├── wizard/ # pre-run interactive Q&A
|
||||
│ └── mt5_pipeline/ # the MT5 bridge
|
||||
├── strategies/ # one folder per strategy (your EAs + presets + iterations)
|
||||
├── registry/ # approved, locked results — the source of truth
|
||||
└── docs/ # this KB + your own notes
|
||||
```
|
||||
|
||||
Mirror the responsibilities described in doc 02. Leave the engines for Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Bring in the human's trading assets
|
||||
|
||||
**Stop and ask the human for:**
|
||||
|
||||
1. The **MQL5 EA** they want to test — `.mq5` source *and* compiled `.ex5`, plus any custom
|
||||
indicators (`.ex5`) the EA loads.
|
||||
2. A **representative `.set` preset** (or a screenshot of the EA inputs) so you can see exactly which
|
||||
inputs exist and their default values.
|
||||
3. Their **broker + symbol(s)** and a **demo account** for the tester.
|
||||
|
||||
From the EA source and preset you (assistant) will:
|
||||
|
||||
- Enumerate every EA input and group them into **frozen** (not optimized) vs **tunable** (search space).
|
||||
- Identify the instrument mechanics (digits, point, contract size, tick value, lot step) to build the
|
||||
instrument config in Phase 5. These come from the **symbol specification** in MT5
|
||||
(`Right-click symbol → Specification`), not from guesses.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Build the Python mirror of the EA (the engine)
|
||||
|
||||
Read [`03-engine-design.md`](03-engine-design.md) carefully — it is the hardest and most important
|
||||
part. The engine must reproduce the EA's **fill and exit logic** bar-by-bar.
|
||||
|
||||
- Keep the engine **strategy-agnostic**: it consumes bars + signal arrays + stop/target arrays and
|
||||
simulates fills. All the *strategy* math (when to enter, where to put stops) lives in the caller.
|
||||
- Implement the **intra-bar sub-tick model** (doc 03) and the **pessimistic ordering** convention.
|
||||
- Match the EA's **lot/money mode**, **spread model**, and **swap model** exactly (doc 05).
|
||||
- The first milestone is **1:1 fidelity on one known preset**: run the EA in MT5 on a short period,
|
||||
run your Python engine on the same data/preset, and reconcile trade-by-trade until the numbers
|
||||
line up within the expected gap (doc 03 "Fidelity" section). **Do not optimize anything until this
|
||||
passes** — an unvalidated engine optimizes noise.
|
||||
|
||||
> The bundled docs use a **grid martingale** engine as the worked example because it exercises every
|
||||
> hard case (pending orders, averaging, trailing on the basket, simultaneous closes). Your EA may be
|
||||
> simpler (single position, signal-to-signal) or different — apply the *principles*, not the grid
|
||||
> specifics.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Wire up configuration and inputs
|
||||
|
||||
Read [`05-config-and-inputs.md`](05-config-and-inputs.md). Build:
|
||||
|
||||
- An **`InstrumentConfig`** object per symbol/broker (real conditions), plus optional `worst_case`
|
||||
and `best_case` variants for cost-stress testing.
|
||||
- A **declared search space** for the strategy (parameter → range → step), kept in code *and*
|
||||
documented in a `parameter-space.md` next to the strategy.
|
||||
- A **wizard** that captures run settings (period, profile, trial count, DD cap, deposit) and writes
|
||||
them to `wizard-answers.yaml` for reproducibility.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Optimizer + robustness
|
||||
|
||||
Read [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md). Build the Optuna
|
||||
objective (score + hard constraints), the diverse top-N selector, and at least the first robustness
|
||||
checks (neighborhood/plateau, cost stress, era split). Start with a **small smoke run** (30–40
|
||||
trials, short period) to validate the script, then one full run in the background.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — The MT5 bridge
|
||||
|
||||
Read [`07-mt5-bridge.md`](07-mt5-bridge.md) and implement the variant for the chosen topology:
|
||||
|
||||
- **Topology A:** use the `MetaTrader5` package or generate a `tester.ini` + `.set` and launch
|
||||
`terminal64.exe /config:` locally. Simplest path — no SSH.
|
||||
- **Topology B/C:** generate `tester.ini` + `.set`, copy to the Windows box, trigger the run via a
|
||||
one-shot scheduled task (so it survives the SSH disconnect), poll for the report, copy it back.
|
||||
|
||||
Then implement the **report parser** (MT5 saves HTML as UTF-16) and the **comparison table** writer.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Run the full cycle once, end to end
|
||||
|
||||
Read [`08-workflow-cycle.md`](08-workflow-cycle.md) and run a complete loop on the human's real EA:
|
||||
hypothesis → scaffold iteration → stats → minimal-scope test → optimize → verify the finalists in
|
||||
MT5 → write the comparison → let the human decide. When the loop produces a number the human trusts,
|
||||
the lab is working.
|
||||
|
||||
---
|
||||
|
||||
## Standing rules for the assistant
|
||||
|
||||
- **Never touch a validated engine to test an idea.** Fork it; prove the fork == original with the
|
||||
change disabled; only then test. (Doc 04.)
|
||||
- **Heavy runs go in the background.** A full-history A/B or a full Optuna study is minutes of
|
||||
compute — start it detached and poll, never block. Smoke-test first. (Doc 06.)
|
||||
- **Don't promote partial searches.** A finalist must come from a *completed* search. An interrupted
|
||||
Optuna study yields preliminary numbers, not verification candidates. (Doc 06, 08.)
|
||||
- **MT5 is the source of truth for live numbers.** Python is for ranking and A/B. Always verify
|
||||
finalists in MT5 before the human relies on a result. (Doc 03, 07.)
|
||||
- **Keep secrets out of code and chat.** Broker login goes in a gitignored env file, read at runtime,
|
||||
never printed. (Doc 07.)
|
||||
- **Ask before destructive actions** — deleting data, force-pushing, overwriting a validated preset.
|
||||
@@ -0,0 +1,117 @@
|
||||
# Backtesting + Optuna + MT5-Verification Stack — Knowledge Base
|
||||
|
||||
A blueprint for building a **personal trading-strategy research lab**: a fast Python
|
||||
backtesting engine, a Bayesian parameter optimizer (Optuna), and an automated bridge to the
|
||||
**MetaTrader 5 Strategy Tester** that cross-checks every result against the real terminal.
|
||||
|
||||
This is a **knowledge base, not a code drop.** It describes the *architecture, algorithms, rules,
|
||||
and tooling* so you can build your own version from scratch — with **your own** Expert Advisors,
|
||||
**your own** strategies, and **your own** presets. There are no ready-made engines or strategy
|
||||
files here on purpose: you supply those from your own MQL5 bots (ideally open-source EAs you own
|
||||
or have the right to use).
|
||||
|
||||
---
|
||||
|
||||
## What this stack does for you
|
||||
|
||||
You have MQL5 Expert Advisors and a pile of ideas to test. The MetaTrader Strategy Tester is
|
||||
accurate but **slow** — a single multi-year backtest with a real-tick model can take 10–30 minutes,
|
||||
and an exhaustive optimization can run for days. That kills iteration speed.
|
||||
|
||||
This stack solves it with a **two-tier model**:
|
||||
|
||||
1. **Tier 1 — a fast Python "mirror engine"** that reproduces your EA's trade logic bar-by-bar over
|
||||
pre-downloaded historical data. It is an *approximation* (more on fidelity below), but it runs a
|
||||
full multi-year backtest in **seconds**, so you can rank thousands of parameter combinations with
|
||||
Optuna in the time MT5 would run a handful.
|
||||
|
||||
2. **Tier 2 — MetaTrader 5 as the gold standard.** Only the **finalists** from the Python search are
|
||||
compiled and run in the real Strategy Tester. The Python number gets you the *shortlist*; the MT5
|
||||
number is what you trust for anything that goes live.
|
||||
|
||||
```
|
||||
Idea ─► Python mirror engine ─► Optuna search (thousands of trials, fast)
|
||||
│
|
||||
▼
|
||||
2–3 diverse finalists
|
||||
│
|
||||
▼
|
||||
MetaTrader 5 Strategy Tester (gold standard)
|
||||
│
|
||||
▼
|
||||
Python-vs-MT5 comparison table ─► keep / discard / iterate
|
||||
```
|
||||
|
||||
The whole point is **disciplined isolation**: the engine is frozen and trusted, instruments are
|
||||
described by data not code, every hypothesis is an isolated experiment, and only triple-checked
|
||||
results are promoted. That discipline is what keeps a research lab from rotting into a pile of
|
||||
one-off scripts that nobody can reproduce.
|
||||
|
||||
---
|
||||
|
||||
## Who this is for
|
||||
|
||||
- You run **MetaTrader 5** and write or use **MQL5** Expert Advisors.
|
||||
- You want to test and optimize strategies **much faster** than the MT5 optimizer allows.
|
||||
- You are comfortable with **Python** (intermediate) and the command line.
|
||||
- You can run **MT5 on Windows** — either on the same machine (simplest) or on a separate
|
||||
Windows box/VPS that your Python machine talks to.
|
||||
|
||||
You do **not** need to be a quant. The hard parts (engine fidelity, Bayesian search, robustness
|
||||
testing) are explained from first principles.
|
||||
|
||||
---
|
||||
|
||||
## How to read this KB
|
||||
|
||||
Start with `CLAUDE.md` if you plan to use an AI coding assistant (Claude Code, Cursor, etc.) to
|
||||
build this with you — it is an **adaptive setup playbook** that profiles *your* machine and OS and
|
||||
walks the install from zero. Otherwise read the numbered docs in order:
|
||||
|
||||
| # | Doc | What you get |
|
||||
|---|-----|--------------|
|
||||
| — | [`CLAUDE.md`](CLAUDE.md) | Adaptive AI-assistant playbook: profiles your device, drives install from scratch. Doubles as `AGENTS.md`. |
|
||||
| 01 | [`01-stack-and-install.md`](01-stack-and-install.md) | The exact tech stack, every library, where to get it, and how to install — per OS. |
|
||||
| 02 | [`02-architecture.md`](02-architecture.md) | The layered architecture and data flow. The mental model for everything else. |
|
||||
| 03 | [`03-engine-design.md`](03-engine-design.md) | How to design your own bar-by-bar engine. Grid logic used as the worked example. Fidelity vs MT5. |
|
||||
| 04 | [`04-isolation-rules.md`](04-isolation-rules.md) | The isolation discipline: frozen engines, forks, separated instruments/strategies, curated registry. |
|
||||
| 05 | [`05-config-and-inputs.md`](05-config-and-inputs.md) | How test inputs are kept *separate*: instrument config, parameter space, the pre-run wizard, lot/money mode. |
|
||||
| 06 | [`06-optimization-and-robustness.md`](06-optimization-and-robustness.md) | Optuna objective design, constraints, diverse top-N selection, and anti-overfit robustness layers. |
|
||||
| 07 | [`07-mt5-bridge.md`](07-mt5-bridge.md) | Connecting to MT5: compiling EAs, auto-running the tester, parsing reports, comparing Python vs MT5. Local-Windows and remote variants. |
|
||||
| 08 | [`08-workflow-cycle.md`](08-workflow-cycle.md) | The full repeatable cycle: hypothesis → scaffold → stats → optimize → verify → promote. |
|
||||
|
||||
---
|
||||
|
||||
## The 30-second mental model
|
||||
|
||||
- **The engine knows nothing about your strategy.** It takes bars + entry signals + stop/target
|
||||
prices and simulates fills. All strategy math lives in *caller* code. (Doc 02–03.)
|
||||
- **The engine is frozen.** You never edit a validated engine to test an idea — you fork it,
|
||||
prove the fork reproduces the original 1:1 with the change off, then test. (Doc 04.)
|
||||
- **Instruments are data, not code branches.** Tick value, spread model, swap, lot steps — all in a
|
||||
per-symbol config object. The engine reads everything from it. (Doc 05.)
|
||||
- **Search inputs are declared, not scattered.** Every tunable parameter, its range, and its
|
||||
constraints live in one declared search space; one wizard captures the run settings; one YAML
|
||||
records the answers so any run is reproducible. (Doc 05–06.)
|
||||
- **Python ranks, MT5 decides.** Fast Python search produces a shortlist; MT5 produces the trusted
|
||||
number; a comparison table is saved for every finalist. (Doc 03, 06, 07.)
|
||||
|
||||
---
|
||||
|
||||
## What you must bring yourself
|
||||
|
||||
This KB is deliberately empty of trading IP. To build a working lab you supply:
|
||||
|
||||
- **Your MQL5 EA(s)** — compiled `.ex5` plus source `.mq5`, and any custom indicators they call.
|
||||
- **Your strategy logic** — encoded once in Python (the caller) so the mirror engine can run it.
|
||||
- **Your presets** — the `.set` files / input templates you want to test and optimize.
|
||||
- **Historical data** — downloaded from your broker via MT5 (the stack includes a recipe).
|
||||
- **A broker demo account** — for the MT5 Strategy Tester runs.
|
||||
|
||||
Everything else — the architecture, the optimizer, the MT5 bridge, the robustness checks, and the
|
||||
rules that hold it together — is described in the docs above.
|
||||
|
||||
---
|
||||
|
||||
*This KB is brand-free and self-contained. Drop the folder into a Git repository, open it with your
|
||||
AI coding assistant, and build your own lab.*
|
||||
Reference in New Issue
Block a user