181 lines
7.3 KiB
Markdown
181 lines
7.3 KiB
Markdown
# 01 — Tech Stack & Installation From Scratch
|
||
|
||
Everything you need to install, what each piece is for, where to get it, and the exact commands per
|
||
operating system. Nothing here is strategy-specific — this is the plumbing.
|
||
|
||
---
|
||
|
||
## 1. The stack at a glance
|
||
|
||
| Layer | Tool | Role | Where to get it |
|
||
|-------|------|------|-----------------|
|
||
| Language | **Python 3.12+** (3.13/3.14 work) | everything outside MT5 | <https://www.python.org/downloads/> |
|
||
| Data frames | **pandas** | OHLC tables, resampling, equity curves | <https://pandas.pydata.org> · `pip install pandas` |
|
||
| Numerics | **numpy** | vectorized price/indicator math | <https://numpy.org> · `pip install numpy` |
|
||
| Columnar storage | **pyarrow** | read/write **Parquet** market data (fast, compact) | <https://arrow.apache.org/docs/python/> · `pip install pyarrow` |
|
||
| JIT speed | **numba** | compiles the hot bar-by-bar loop to machine code | <https://numba.pydata.org> · `pip install numba` |
|
||
| Optimizer | **Optuna** | Bayesian hyper-parameter search | <https://optuna.org> · `pip install optuna` |
|
||
| Optuna storage | **SQLAlchemy** (+ **alembic**) | persists studies to a SQLite DB so runs resume/parallelize | <https://www.sqlalchemy.org> · `pip install sqlalchemy` |
|
||
| Config | **PyYAML** | reproducible run settings (`wizard-answers.yaml`) | <https://pyyaml.org> · `pip install pyyaml` |
|
||
| HTML parsing | **lxml** + **html5lib** | parse the MT5 Strategy Tester HTML report | <https://lxml.de> · `pip install lxml html5lib` |
|
||
| Progress / logs | **tqdm**, **colorlog** | progress bars, readable logs | `pip install tqdm colorlog` |
|
||
| HTTP | **requests** | optional data downloads / webhooks | <https://requests.readthedocs.io> · `pip install requests` |
|
||
| Tester | **MetaTrader 5** terminal | the gold-standard backtester + data source | from your broker, or <https://www.metatrader5.com/en/download> |
|
||
| MT5 control (Windows) | **MetaTrader5** pip package | drive the terminal & pull data from Python (Windows only) | <https://pypi.org/project/MetaTrader5/> · `pip install MetaTrader5` |
|
||
|
||
> **Optional data source.** If you want history without MT5 export, `dukascopy-python`
|
||
> (`pip install dukascopy-python`, <https://pypi.org/project/dukascopy-python/>) pulls free tick/bar
|
||
> data for many symbols. MT5-exported data from your own broker is preferred because it matches the
|
||
> tester exactly.
|
||
|
||
### Reference versions
|
||
|
||
A known-good combination (early 2026) — use these or the latest stable:
|
||
|
||
```
|
||
python 3.14
|
||
pandas 3.0
|
||
numpy 2.4
|
||
pyarrow 24.0
|
||
numba 0.65
|
||
optuna 4.8
|
||
sqlalchemy 2.0
|
||
pyyaml 6.0
|
||
lxml 6.1
|
||
```
|
||
|
||
Pin exact versions in a `requirements.txt` once your lab works, so it reproduces later.
|
||
|
||
---
|
||
|
||
## 2. What each library actually does here
|
||
|
||
- **pandas / numpy** — market data is loaded into a DataFrame of `[timestamp, open, high, low, close,
|
||
spread]`. Indicators and signals are computed as numpy arrays. Equity curves are pandas frames.
|
||
- **pyarrow + Parquet** — a few years of M1 (one-minute) bars is millions of rows. Parquet stores it
|
||
columnar and compressed: a multi-million-row file loads in well under a second and is a fraction of
|
||
CSV size. This is what makes "full-history backtest in seconds" possible.
|
||
- **numba** — the engine's inner loop walks every bar (and sub-ticks within each bar). Pure-Python
|
||
that is too slow. `@njit` compiles it to native code on first call; subsequent runs are C-fast.
|
||
- **Optuna** — instead of brute-forcing a parameter grid, Optuna uses a Bayesian sampler (TPE) that
|
||
*learns* which regions of the search space are promising and concentrates trials there. You get a
|
||
good optimum in hundreds–thousands of trials instead of an exhaustive grid of millions.
|
||
- **SQLAlchemy/SQLite** — Optuna writes each trial to a `study.db`. That means a study can be stopped
|
||
and resumed, inspected mid-run (count completed trials), and run with multiple worker processes
|
||
pointing at the same DB.
|
||
- **lxml / html5lib** — the MT5 tester exports its report as an HTML file encoded **UTF-16-LE**. These
|
||
parse it into a metrics dict (Net Profit, Profit Factor, Drawdown, trade count, …).
|
||
- **MetaTrader5 package** — on Windows, this is the clean way to (a) download historical bars and
|
||
(b) launch/script the terminal. On non-Windows you don't have it, which is why remote topologies use
|
||
SSH + a scheduled task instead.
|
||
|
||
---
|
||
|
||
## 3. Install — step by step
|
||
|
||
### 3.0 Prerequisites
|
||
|
||
- **Python 3.12+**. Check with `python3 --version` (macOS/Linux) or `python --version` (Windows).
|
||
- **Git** (to version your lab).
|
||
- **MetaTrader 5** installed from your broker, with a **demo account** logged in.
|
||
|
||
### 3.1 Windows (Topology A — recommended)
|
||
|
||
```powershell
|
||
# From the project root, in PowerShell or cmd:
|
||
|
||
# 1. Create the virtual environment
|
||
python -m venv .venv
|
||
|
||
# 2. Activate it
|
||
.\.venv\Scripts\activate
|
||
|
||
# 3. Upgrade pip
|
||
python -m pip install --upgrade pip
|
||
|
||
# 4. Install the stack (MetaTrader5 included — Windows only)
|
||
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests MetaTrader5
|
||
```
|
||
|
||
### 3.2 macOS / Linux (Topology B/C — MT5 lives elsewhere)
|
||
|
||
```bash
|
||
# 1. Create the venv
|
||
python3 -m venv .venv
|
||
|
||
# 2. Activate
|
||
source .venv/bin/activate
|
||
|
||
# 3. Upgrade pip
|
||
python3 -m pip install --upgrade pip
|
||
|
||
# 4. Install the stack (NO MetaTrader5 package — it is Windows-only)
|
||
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests
|
||
```
|
||
|
||
> On Apple Silicon (M-series) everything above is native arm64 and fast. `numba`/`numpy` ship arm64
|
||
> wheels — no Rosetta needed.
|
||
|
||
### 3.3 Verify the install
|
||
|
||
```bash
|
||
# Use the venv's python explicitly to avoid the system interpreter
|
||
.venv/bin/python3 -c "import pandas, numpy, pyarrow, optuna, numba, yaml, lxml; print('core OK')"
|
||
# Windows: .\.venv\Scripts\python -c "..."
|
||
```
|
||
|
||
On Windows also verify the terminal link:
|
||
|
||
```python
|
||
import MetaTrader5 as mt5
|
||
print(mt5.initialize()) # True if it found & launched the terminal
|
||
print(mt5.version())
|
||
mt5.shutdown()
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Always use the venv interpreter
|
||
|
||
A recurring source of bugs is accidentally running the **system** Python (which lacks the libraries).
|
||
Make it a habit to call the venv interpreter by path:
|
||
|
||
```bash
|
||
# macOS/Linux
|
||
.venv/bin/python3 your_script.py
|
||
|
||
# Windows
|
||
.\.venv\Scripts\python your_script.py
|
||
```
|
||
|
||
…or activate the venv at the start of every session. Pick one convention and keep it.
|
||
|
||
---
|
||
|
||
## 5. `.gitignore` essentials
|
||
|
||
Your lab will accumulate large data and secrets. Ignore them from day one:
|
||
|
||
```gitignore
|
||
.venv/
|
||
data/ # market data is large & re-downloadable
|
||
results/ # scratch run outputs
|
||
*.db # Optuna SQLite studies
|
||
*.htm # pulled MT5 reports
|
||
.env # broker credentials — NEVER commit
|
||
__pycache__/
|
||
.DS_Store
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Hardware notes
|
||
|
||
- Backtesting is **CPU + RAM** bound, not GPU. A modern multi-core CPU and 16 GB+ RAM is plenty.
|
||
- Optuna parallelizes across CPU cores. The practical cap for *heavy* concurrent backtests is roughly
|
||
your number of **performance cores** — beyond that they contend and slow each other down. Prefer
|
||
**one** Optuna study with `n_jobs=N` over N separate scripts fighting for cores.
|
||
- Millions of M1 bars fit comfortably in RAM as a pandas frame; loading from Parquet is the only I/O.
|
||
|
||
Next: [`02-architecture.md`](02-architecture.md) — the layered architecture you are about to build.
|