169 lines
8.7 KiB
Markdown
169 lines
8.7 KiB
Markdown
# 06 — Optimization (Optuna) & Robustness
|
||
|
||
How to turn the declared search space into a Bayesian search that finds good parameters fast — and how
|
||
to avoid the trap that ruins most optimization: **overfitting to the past.** A great backtest is easy;
|
||
a robust one is the job.
|
||
|
||
---
|
||
|
||
## 1. Why Optuna (and not a grid)
|
||
|
||
A parameter grid is exponential: 8 parameters × 10 values each = 100 million combinations. Even at
|
||
"seconds per backtest" that's infeasible, and most of it is wasteland.
|
||
|
||
**Optuna** (<https://optuna.org>, docs <https://optuna.readthedocs.io>) uses a **TPE sampler** (Tree-
|
||
structured Parzen Estimator) that *learns* the shape of the objective as it goes and spends trials
|
||
where results are promising. You get a strong optimum in **hundreds to a few thousand** trials.
|
||
|
||
Install: `pip install optuna` (doc 01). Core concepts you'll use:
|
||
|
||
- **Study** — one optimization. Maximizes (or minimizes) an objective. Persisted to SQLite so it can
|
||
be resumed, inspected mid-run, and parallelized.
|
||
- **Trial** — one parameter sample + its score. `trial.suggest_float/int(name, low, high, step=…)`
|
||
draws each parameter; `trial.set_user_attr(...)` stashes the metrics for later inspection.
|
||
- **Sampler** — TPE by default; fine to start.
|
||
|
||
```python
|
||
import optuna
|
||
study = optuna.create_study(
|
||
direction="maximize",
|
||
storage="sqlite:///study.db", # persist → resume, inspect, parallelize
|
||
study_name="my_iter",
|
||
load_if_exists=True,
|
||
)
|
||
study.optimize(objective, n_trials=1000, n_jobs=4) # n_jobs = parallel workers
|
||
```
|
||
|
||
---
|
||
|
||
## 2. The objective function
|
||
|
||
The objective receives a trial, **samples** parameters from the search space, **runs** the engine, and
|
||
returns a **score**. Structure it in four clear steps:
|
||
|
||
```
|
||
def objective(trial):
|
||
1. sampled = suggest_params(trial) # from SEARCH_SPACE (doc 05)
|
||
2. params = build_params(sampled, frozen_baseline)
|
||
3. result = engine.run(params, bars, instrument, deposit)
|
||
metrics = compute_metrics(result)
|
||
4. score, violations = score_metrics(metrics, sampled)
|
||
for k, v in metrics.items(): trial.set_user_attr(k, v)
|
||
trial.set_user_attr("violations", violations)
|
||
return score
|
||
```
|
||
|
||
### Score design
|
||
|
||
A good single-objective score balances **reward against risk**, for example:
|
||
|
||
```
|
||
score = Net − DD_WEIGHT × EquityDrawdown # reward profit, punish the drawdown that earned it
|
||
```
|
||
|
||
The weight makes the optimizer prefer a slightly smaller, much calmer equity curve over a fragile spike.
|
||
(Optuna also supports true multi-objective studies — e.g. maximize Net *and* minimize DD on a Pareto
|
||
front — but a weighted single objective is simpler and usually enough to start.)
|
||
|
||
### Constraints as hard rejections, not soft nudges
|
||
|
||
Some conditions should **disqualify** a trial outright, not just dock points. Implement them as a large
|
||
penalty per violation so violators sort to the bottom:
|
||
|
||
```
|
||
violations = []
|
||
if metrics.trades < MIN_TRADES: violations.append("too few trades")
|
||
if metrics.profit_factor < MIN_PF: violations.append("PF too low")
|
||
if metrics.equity_dd > HARD_DD_CAP: violations.append("DD over cap")
|
||
score = base_score − PENALTY * len(violations) # PENALTY huge, e.g. 1e6
|
||
```
|
||
|
||
Typical constraints worth enforcing:
|
||
|
||
- **Minimum trade count** (a hard floor, calibrated to the window — e.g. ≥25–30 active trades per
|
||
year). Few-trade, high-PF results are almost always overfit luck, not edge.
|
||
- **Minimum Profit Factor.**
|
||
- **Hard drawdown cap** in account currency (e.g. ≤35% of deposit).
|
||
- **A path-independent fragility guard.** For a martingale grid, compute the *worst-case floating
|
||
drawdown* if a full series accumulates straight down an N% drop, and reject if it's worse than the
|
||
baseline's. This catches "the backtest never hit the bad path, but the configuration is a time bomb"
|
||
— the failure mode a pure historical backtest can't see.
|
||
|
||
---
|
||
|
||
## 3. Don't pick the top-N by score — pick the top-N **diverse**
|
||
|
||
Optuna converges: the top 10 trials by score are usually near-clones of one peak. Verifying 3 clones in
|
||
MT5 tells you nothing about robustness. Instead, pick **meaningfully different** finalists.
|
||
|
||
**Greedy diverse selection:**
|
||
|
||
```
|
||
1. filter trials by the constraints (drop violators)
|
||
2. sort by score
|
||
3. start the selected set with the single best trial
|
||
4. repeatedly add the remaining trial with the MAXIMUM parameter-distance to the already-selected set,
|
||
lightly weighted by its rank so you still prefer strong scores
|
||
5. stop at N (2–3 is plenty)
|
||
```
|
||
|
||
Parameter distance = average normalized difference across all parameters (normalize each to [0,1]:
|
||
numeric by min–max, boolean to 0/1, categorical by index). The result is 2–3 finalists that are each
|
||
*good* but reach the result by *different* parameter regions — if they all survive MT5, you have real
|
||
robustness, not one lucky corner.
|
||
|
||
---
|
||
|
||
## 4. Robustness layers — proving the result isn't overfit
|
||
|
||
These are **read-only analyses over a finished result** (the study, the trade list, the equity curve).
|
||
They never touch the engine. Run them on finalists *before* spending MT5 time. Treat them as
|
||
report-only signals by default; tighten into hard gates as you gain confidence.
|
||
|
||
| Layer | Question it answers | Method |
|
||
|-------|---------------------|--------|
|
||
| **Stability region** | Is the winner on a *plateau* of good configs, or a lone spike? | Cluster the study's top trials by parameter distance; prefer a finalist from the center of a dense good region, not an isolated peak. |
|
||
| **Neighborhood / sensitivity** | Does a small parameter nudge destroy the edge? | Perturb each lever ±1 step; demand all neighbors stay profitable. A fragile optimum fails this. |
|
||
| **Walk-forward (WFE)** | Does the edge hold out-of-sample? | Split into in-sample/out-of-sample windows; with the finalist's *fixed* params (no re-fit), compute `OOS_metric / IS_metric`. Add a **purge gap** between IS and OOS so leakage can't help. |
|
||
| **Monte-Carlo permutation** | How lucky was the trade *order*? | Shuffle trade order, drop a fraction, jitter PnL; build a distribution of drawdown/Net. A finalist whose real drawdown sits in the ugly tail is fragile. |
|
||
| **Deflated Sharpe (DSR)** | Is the Sharpe real after testing thousands of configs? | Adjust the observed Sharpe for the **number of trials** (multiple-testing). Many trials inflate the best result even under pure noise; DSR estimates the probability the true Sharpe beats a deflated benchmark. |
|
||
| **Era split** | Does the edge exist in *both halves* of history? | Run the finalist on era-1 vs era-2 separately; an edge present in only one era is regime-luck. |
|
||
| **Cost stress** | Does it survive realistic and adverse costs? | Re-run on `real` / `worst_case` / `best_case` instrument profiles (doc 05). Edge only in `best_case` = no edge. |
|
||
|
||
> These are standard quant techniques, not exotic. Walk-forward and Monte-Carlo catch the everyday
|
||
> overfit; the Deflated Sharpe specifically counters the fact that an exhaustive search *will* surface
|
||
> an impressive-looking config from noise. References worth reading: López de Prado on the Deflated
|
||
> Sharpe Ratio and the dangers of backtest overfitting.
|
||
|
||
---
|
||
|
||
## 5. What qualifies as a "finalist"
|
||
|
||
Be strict — MT5 time is the expensive resource, and a weak finalist wastes it:
|
||
|
||
- **From a *completed* search only.** An interrupted/partial Optuna study yields *preliminary*
|
||
numbers, not verification candidates. The README status must reflect the real process state, never
|
||
"continues" if the process is dead.
|
||
- **Passes the hard constraints** (trades, PF, DD cap, fragility guard).
|
||
- **Not concentrated in one year.** If a single year carries >~40% of total Net, it's fragile — one
|
||
regime is doing all the work.
|
||
- **Honest about flat periods.** A gate that sat out a bad year "survived by not trading" — describe it
|
||
that way, don't dress it up as robust profit.
|
||
- **Survives the robustness layers** you've chosen to enforce.
|
||
- **Converge to 2–3 diverse finalists**, not 5–6. More finalists is usually indecision, not rigor.
|
||
|
||
---
|
||
|
||
## 6. Running searches without melting your machine (operational discipline)
|
||
|
||
- **Smoke first.** A 30–40 trial run on a short period validates the whole script (data load → engine →
|
||
scoring → storage) in under a minute. Only then launch the full study.
|
||
- **Run heavy studies in the background and poll** the trial count in `study.db`; don't block a shell on
|
||
a multi-minute run.
|
||
- **One study, many workers** (`n_jobs=N`) beats N competing scripts. Cap heavy concurrency at your
|
||
performance-core count.
|
||
- **Dedup guard:** never relaunch a study you think timed out without checking it isn't still running —
|
||
duplicates thrash the CPU and waste hours.
|
||
|
||
Next: [`07-mt5-bridge.md`](07-mt5-bridge.md) — verifying the finalists in the real terminal.
|