# 06 — Optimization (Optuna) & Robustness How to turn the declared search space into a Bayesian search that finds good parameters fast — and how to avoid the trap that ruins most optimization: **overfitting to the past.** A great backtest is easy; a robust one is the job. --- ## 1. Why Optuna (and not a grid) A parameter grid is exponential: 8 parameters × 10 values each = 100 million combinations. Even at "seconds per backtest" that's infeasible, and most of it is wasteland. **Optuna** (, docs ) uses a **TPE sampler** (Tree- structured Parzen Estimator) that *learns* the shape of the objective as it goes and spends trials where results are promising. You get a strong optimum in **hundreds to a few thousand** trials. Install: `pip install optuna` (doc 01). Core concepts you'll use: - **Study** — one optimization. Maximizes (or minimizes) an objective. Persisted to SQLite so it can be resumed, inspected mid-run, and parallelized. - **Trial** — one parameter sample + its score. `trial.suggest_float/int(name, low, high, step=…)` draws each parameter; `trial.set_user_attr(...)` stashes the metrics for later inspection. - **Sampler** — TPE by default; fine to start. ```python import optuna study = optuna.create_study( direction="maximize", storage="sqlite:///study.db", # persist → resume, inspect, parallelize study_name="my_iter", load_if_exists=True, ) study.optimize(objective, n_trials=1000, n_jobs=4) # n_jobs = parallel workers ``` --- ## 2. The objective function The objective receives a trial, **samples** parameters from the search space, **runs** the engine, and returns a **score**. Structure it in four clear steps: ``` def objective(trial): 1. sampled = suggest_params(trial) # from SEARCH_SPACE (doc 05) 2. params = build_params(sampled, frozen_baseline) 3. result = engine.run(params, bars, instrument, deposit) metrics = compute_metrics(result) 4. score, violations = score_metrics(metrics, sampled) for k, v in metrics.items(): trial.set_user_attr(k, v) trial.set_user_attr("violations", violations) return score ``` ### Score design A good single-objective score balances **reward against risk**, for example: ``` score = Net − DD_WEIGHT × EquityDrawdown # reward profit, punish the drawdown that earned it ``` The weight makes the optimizer prefer a slightly smaller, much calmer equity curve over a fragile spike. (Optuna also supports true multi-objective studies — e.g. maximize Net *and* minimize DD on a Pareto front — but a weighted single objective is simpler and usually enough to start.) ### Constraints as hard rejections, not soft nudges Some conditions should **disqualify** a trial outright, not just dock points. Implement them as a large penalty per violation so violators sort to the bottom: ``` violations = [] if metrics.trades < MIN_TRADES: violations.append("too few trades") if metrics.profit_factor < MIN_PF: violations.append("PF too low") if metrics.equity_dd > HARD_DD_CAP: violations.append("DD over cap") score = base_score − PENALTY * len(violations) # PENALTY huge, e.g. 1e6 ``` Typical constraints worth enforcing: - **Minimum trade count** (a hard floor, calibrated to the window — e.g. ≥25–30 active trades per year). Few-trade, high-PF results are almost always overfit luck, not edge. - **Minimum Profit Factor.** - **Hard drawdown cap** in account currency (e.g. ≤35% of deposit). - **A path-independent fragility guard.** For a martingale grid, compute the *worst-case floating drawdown* if a full series accumulates straight down an N% drop, and reject if it's worse than the baseline's. This catches "the backtest never hit the bad path, but the configuration is a time bomb" — the failure mode a pure historical backtest can't see. --- ## 3. Don't pick the top-N by score — pick the top-N **diverse** Optuna converges: the top 10 trials by score are usually near-clones of one peak. Verifying 3 clones in MT5 tells you nothing about robustness. Instead, pick **meaningfully different** finalists. **Greedy diverse selection:** ``` 1. filter trials by the constraints (drop violators) 2. sort by score 3. start the selected set with the single best trial 4. repeatedly add the remaining trial with the MAXIMUM parameter-distance to the already-selected set, lightly weighted by its rank so you still prefer strong scores 5. stop at N (2–3 is plenty) ``` Parameter distance = average normalized difference across all parameters (normalize each to [0,1]: numeric by min–max, boolean to 0/1, categorical by index). The result is 2–3 finalists that are each *good* but reach the result by *different* parameter regions — if they all survive MT5, you have real robustness, not one lucky corner. --- ## 4. Robustness layers — proving the result isn't overfit These are **read-only analyses over a finished result** (the study, the trade list, the equity curve). They never touch the engine. Run them on finalists *before* spending MT5 time. Treat them as report-only signals by default; tighten into hard gates as you gain confidence. | Layer | Question it answers | Method | |-------|---------------------|--------| | **Stability region** | Is the winner on a *plateau* of good configs, or a lone spike? | Cluster the study's top trials by parameter distance; prefer a finalist from the center of a dense good region, not an isolated peak. | | **Neighborhood / sensitivity** | Does a small parameter nudge destroy the edge? | Perturb each lever ±1 step; demand all neighbors stay profitable. A fragile optimum fails this. | | **Walk-forward (WFE)** | Does the edge hold out-of-sample? | Split into in-sample/out-of-sample windows; with the finalist's *fixed* params (no re-fit), compute `OOS_metric / IS_metric`. Add a **purge gap** between IS and OOS so leakage can't help. | | **Monte-Carlo permutation** | How lucky was the trade *order*? | Shuffle trade order, drop a fraction, jitter PnL; build a distribution of drawdown/Net. A finalist whose real drawdown sits in the ugly tail is fragile. | | **Deflated Sharpe (DSR)** | Is the Sharpe real after testing thousands of configs? | Adjust the observed Sharpe for the **number of trials** (multiple-testing). Many trials inflate the best result even under pure noise; DSR estimates the probability the true Sharpe beats a deflated benchmark. | | **Era split** | Does the edge exist in *both halves* of history? | Run the finalist on era-1 vs era-2 separately; an edge present in only one era is regime-luck. | | **Cost stress** | Does it survive realistic and adverse costs? | Re-run on `real` / `worst_case` / `best_case` instrument profiles (doc 05). Edge only in `best_case` = no edge. | > These are standard quant techniques, not exotic. Walk-forward and Monte-Carlo catch the everyday > overfit; the Deflated Sharpe specifically counters the fact that an exhaustive search *will* surface > an impressive-looking config from noise. References worth reading: López de Prado on the Deflated > Sharpe Ratio and the dangers of backtest overfitting. --- ## 5. What qualifies as a "finalist" Be strict — MT5 time is the expensive resource, and a weak finalist wastes it: - **From a *completed* search only.** An interrupted/partial Optuna study yields *preliminary* numbers, not verification candidates. The README status must reflect the real process state, never "continues" if the process is dead. - **Passes the hard constraints** (trades, PF, DD cap, fragility guard). - **Not concentrated in one year.** If a single year carries >~40% of total Net, it's fragile — one regime is doing all the work. - **Honest about flat periods.** A gate that sat out a bad year "survived by not trading" — describe it that way, don't dress it up as robust profit. - **Survives the robustness layers** you've chosen to enforce. - **Converge to 2–3 diverse finalists**, not 5–6. More finalists is usually indecision, not rigor. --- ## 6. Running searches without melting your machine (operational discipline) - **Smoke first.** A 30–40 trial run on a short period validates the whole script (data load → engine → scoring → storage) in under a minute. Only then launch the full study. - **Run heavy studies in the background and poll** the trial count in `study.db`; don't block a shell on a multi-minute run. - **One study, many workers** (`n_jobs=N`) beats N competing scripts. Cap heavy concurrency at your performance-core count. - **Dedup guard:** never relaunch a study you think timed out without checking it isn't still running — duplicates thrash the CPU and waste hours. Next: [`07-mt5-bridge.md`](07-mt5-bridge.md) — verifying the finalists in the real terminal.