# 02 — Architecture & Data Flow The whole lab is a set of **layers with one-directional dependencies**. Higher layers call lower ones; lower layers never know about higher ones. This is what lets you freeze the engine, swap instruments, and add strategies without anything leaking sideways. --- ## 1. The layer stack ``` ┌─────────────────────────────────────────────────────────────────────┐ │ STRATEGY / CALLER one folder per strategy │ │ loads data, computes signals + stops, calls the engine, scores │ └───────────────┬──────────────────────────────┬──────────────────────┘ │ │ ┌───────▼────────┐ ┌─────────▼──────────┐ │ OPTIMIZER │ │ MT5 BRIDGE │ │ objective, │ │ compile, run, │ │ Optuna study, │ │ parse, compare │ │ diverse top-N │ └─────────┬──────────┘ └───────┬─────────┘ │ │ │ (gold-standard verify) ┌───────▼─────────┐ │ │ ROBUSTNESS │ anti-overfit checks over results (read-only) └───────┬─────────┘ │ ┌───────▼─────────────────────────────────────────────┐ │ ENGINE (frozen) bar-by-bar fill simulator │ │ knows: bars, signals, stop/target prices, instrument │ │ knows NOT: your strategy, indicators, broker │ └───┬─────────────┬────────────────┬───────────────────┘ │ │ │ ┌────────▼───┐ ┌──────▼──────┐ ┌──────▼───────┐ ┌──────────────┐ │ INDICATORS │ │ INSTRUMENTS │ │ GATES │ │ DATA │ │ RSI/ATR/.. │ │ per-symbol │ │ entry-filter │ │ loaders + │ │ pure fns │ │ config объ. │ │ masks │ │ MT5 parser │ └────────────┘ └─────────────┘ └──────────────┘ └──────────────┘ ``` ### Layer responsibilities | Layer | Folder | Owns | Must NOT | |-------|--------|------|----------| | **Data** | `shared/data/` | load bars from Parquet; fetch from MT5; parse MT5 HTML reports | contain strategy logic | | **Instruments** | `shared/instruments/` | per-symbol/broker config objects (tick value, spread, swap, lot steps) | know any strategy | | **Indicators** | `shared/indicators/` | pure functions: RSI, ATR, EMA, SMA, your custom range filter, etc. | hold state across calls | | **Gates** | `shared/gates/` | reusable boolean masks that *filter* entries (regime, time-of-day, exhaustion) | open/close trades | | **Engine** | `shared/core/` | the bar-by-bar fill/exit simulator; **frozen** once validated | compute signals or stops | | **Robustness** | `shared/robustness/` | read-only anti-overfit analyses over a result | mutate the engine or results | | **Optimizer** | `shared/optimizer/` | objective fn, Optuna wiring, diverse top-N selection | know broker specifics | | **MT5 bridge** | `shared/mt5_pipeline/` | generate `.set`/`.ini`, run the tester, pull & parse the report, compare | compute strategy | | **Strategy / caller** | `strategies//` | glue: data → signals → stops → engine → metrics; the only layer that knows the *whole* picture | be reused by another strategy (copy, don't import) | | **Registry** | `registry/` | approved, locked results — the source of truth | be edited casually | --- ## 2. The core contract: engine knows nothing about strategy This is the single most important design decision. The engine's input is **pre-computed**: ``` engine.run( bars, # DataFrame [timestamp, open, high, low, close, spread] signals_long, # boolean array — True on the bar an entry is triggered signals_short, # boolean array sl_prices, # array — the stop price for an entry on that bar (NaN if none) tp_prices, # array — the target price instrument, # InstrumentConfig — all symbol mechanics sizing, # SizingInputs — position sizing inputs initial_deposit, *, # keyword-only from here m1_bars=None, # OPTIONAL: M1 bars for tick-level exit simulation # REQUIRED if the EA moves its SL intra-trade (BE / trailing / # basket trailing) — bar-level mode is untrustworthy for that # class (doc 03 §7 failure mode, doc 03 §8 target gates). ) -> Result ``` > **Crucial:** the stop and target arrive as **finished prices** computed by the caller. The engine > never decides *where* a stop goes — only *whether* price touched it. This is the seam that > separates "the strategy" from "the simulator". Change your strategy → you change the caller and the > arrays you hand in; the engine is untouched. > > **The `m1_bars` parameter is not optional for trailing/BE strategies.** When the EA updates its SL > during a trade, the bar-level engine can produce a −40% to −50% net gap vs MT5 (doc 03 §7 measured > failure mode). Pass M1 bars and the engine switches to tick-level exit simulation, dropping the > gap to ~−5%. Bar-level mode remains the right (and faster) choice for clean-directional setups that > don't move the SL. The engine's output is equally generic: ``` Result: trades # list of closed trades (entry/exit time, price, lots, pnl, swap) equity_curve # DataFrame [timestamp, balance, equity], sampled (e.g. hourly) final_balance initial_deposit ``` A separate `compute_metrics(result)` turns that into the numbers you optimize on: Net Profit, Profit Factor, Win Rate, max Balance/Equity Drawdown, Sharpe, APR, trade count. --- ## 3. Data flow of one backtest ``` data//_M1_.parquet data//_M5_.parquet │ load_bars() → M1 DataFrame │ load_bars() → signal-TF DataFrame │ │ (resampled from M1 if needed) │ ▼ │ caller: indicators.* on the signal timeframe │ │ │ ▼ │ caller: edge-detect signals (True only on the bar │ │ the condition first flips, not every bar after) │ │ + optional gates (regime/time filters AND-ed in) │ ▼ │ caller: compute SL/TP price arrays from params │ │ (ATR stop, % stop, indicator band, …) └──────────────┐ ┌──────────────────────────┘ ▼ ▼ engine.run(signal_bars, signals, sl/tp, instrument, sizing, deposit, m1_bars=m1_bars) ← M1 is passed back to the engine │ for tick-level exit simulation when the EA │ moves its SL intra-trade (BE / trailing). Without │ it, the bar-level engine over-credits BE exits │ (doc 03 §7 failure mode). ▼ Result → compute_metrics → dict │ ▼ (finalists only) MT5 bridge: build .set from the same params → run real tester → parse report → compare ``` Three subtleties that cause most bugs if missed (the first two explained in doc 03): - **Edge detection.** Signals must be `True` only on the *transition* bar, not forward-filled, or the engine re-enters every bar. - **Timeframe alignment.** Indicators computed on a higher timeframe must be mapped back onto the M1 bars correctly (no look-ahead — a daily value is only known after that day closes). - **M1 must reach the engine for trailing/BE EAs.** Downloading M1 only to resample it up to the signal timeframe is **not** enough if the EA moves its SL during a trade. Pass the M1 bars to `engine.run` (the `m1_bars=` kwarg); otherwise the bar-level exit simulation produces a −40% to −50% net gap (doc 03 §7) that looks like a "fidelity issue" but is actually a missing-input bug. --- ## 4. Why "shared" vs "strategies" - **`shared/`** is infrastructure you write **once** and treat as a library: engine, indicators, instruments, optimizer, robustness, bridge. It is import-stable. Breaking changes here ripple everywhere, so it changes rarely and deliberately. - **`strategies//`** is where experimentation happens. Each strategy is **self-contained**: its EA source, its presets, and its `iterations/` (each iteration = one research attempt with its own `optimize.py`, search space, results, and verification). Strategies **copy** glue code between iterations rather than importing it, so an old iteration always reproduces even after you change how you do things. ``` strategies// ├── source/ # your EA .mq5/.ex5 + custom indicators (read-only after compile) ├── presets/ # canonical baseline .set files (frozen reference) ├── production/ # live-ready presets ├── iterations/--/ │ ├── README.md # context, status, result summary │ ├── parameter-space.md # the search space for this attempt │ ├── optimize.py # the run script (a snapshot — self-contained) │ ├── wizard-answers.yaml # the run settings (reproducibility) │ ├── top-1/ top-2/ # the diverse finalists: params, preset, py+mt5 metrics │ └── auto-verification.md# Python-vs-MT5 comparison table └── archive/ # discarded attempts ``` --- ## 5. The registry is the source of truth Nothing is "real" until it is in `registry/`. A registry entry is a result that passed **all three** gates: Python search → MT5 verification → human approval. Once written it is **locked** — you never silently edit a registry entry; you create a *new* one. This is what stops a research lab from drifting: the registry is the curated, trustworthy subset of everything you ever tried. Doc 04 covers the isolation rules that keep these boundaries honest. Doc 03 is next — how to actually build the engine. Next: [`03-engine-design.md`](03-engine-design.md).