01 — Tech Stack & Installation From Scratch

Everything you need to install, what each piece is for, where to get it, and the exact commands per operating system. Nothing here is strategy-specific — this is the plumbing.

1. The stack at a glance

Layer	Tool	Role	Where to get it
Language	Python 3.12+ (3.13/3.14 work)	everything outside MT5	https://www.python.org/downloads/
Data frames	pandas	OHLC tables, resampling, equity curves	https://pandas.pydata.org · `pip install pandas`
Numerics	numpy	vectorized price/indicator math	https://numpy.org · `pip install numpy`
Columnar storage	pyarrow	read/write Parquet market data (fast, compact)	https://arrow.apache.org/docs/python/ · `pip install pyarrow`
JIT speed	numba	compiles the hot bar-by-bar loop to machine code	https://numba.pydata.org · `pip install numba`
Optimizer	Optuna	Bayesian hyper-parameter search	https://optuna.org · `pip install optuna`
Optuna storage	SQLAlchemy (+ alembic)	persists studies to a SQLite DB so runs resume/parallelize	https://www.sqlalchemy.org · `pip install sqlalchemy`
Config	PyYAML	reproducible run settings (`wizard-answers.yaml`)	https://pyyaml.org · `pip install pyyaml`
HTML parsing	lxml + html5lib	parse the MT5 Strategy Tester HTML report	https://lxml.de · `pip install lxml html5lib`
Progress / logs	tqdm, colorlog	progress bars, readable logs	`pip install tqdm colorlog`
HTTP	requests	optional data downloads / webhooks	https://requests.readthedocs.io · `pip install requests`
Tester	MetaTrader 5 terminal	the gold-standard backtester + data source	from your broker, or https://www.metatrader5.com/en/download
MT5 control (Windows)	MetaTrader5 pip package	drive the terminal & pull data from Python (Windows only)	https://pypi.org/project/MetaTrader5/ · `pip install MetaTrader5`

Optional data source. If you want history without MT5 export, dukascopy-python (pip install dukascopy-python, https://pypi.org/project/dukascopy-python/) pulls free tick/bar data for many symbols. MT5-exported data from your own broker is preferred because it matches the tester exactly.

Reference versions

A known-good combination (early 2026) — use these or the latest stable:

python   3.14
pandas   3.0
numpy    2.4
pyarrow  24.0
numba    0.65
optuna   4.8
sqlalchemy 2.0
pyyaml   6.0
lxml     6.1

Pin exact versions in a requirements.txt once your lab works, so it reproduces later.

2. What each library actually does here

pandas / numpy — market data is loaded into a DataFrame of [timestamp, open, high, low, close, spread]. Indicators and signals are computed as numpy arrays. Equity curves are pandas frames.
pyarrow + Parquet — a few years of M1 (one-minute) bars is millions of rows. Parquet stores it columnar and compressed: a multi-million-row file loads in well under a second and is a fraction of CSV size. This is what makes "full-history backtest in seconds" possible.
numba — the engine's inner loop walks every bar (and sub-ticks within each bar). Pure-Python that is too slow. @njit compiles it to native code on first call; subsequent runs are C-fast.
Optuna — instead of brute-forcing a parameter grid, Optuna uses a Bayesian sampler (TPE) that learns which regions of the search space are promising and concentrates trials there. You get a good optimum in hundreds–thousands of trials instead of an exhaustive grid of millions.
SQLAlchemy/SQLite — Optuna writes each trial to a study.db. That means a study can be stopped and resumed, inspected mid-run (count completed trials), and run with multiple worker processes pointing at the same DB.
lxml / html5lib — the MT5 tester exports its report as an HTML file encoded UTF-16-LE. These parse it into a metrics dict (Net Profit, Profit Factor, Drawdown, trade count, …).
MetaTrader5 package — on Windows, this is the clean way to (a) download historical bars and (b) launch/script the terminal. On non-Windows you don't have it, which is why remote topologies use SSH + a scheduled task instead.

3. Install — step by step

3.0 Prerequisites

Python 3.12+. Check with python3 --version (macOS/Linux) or python --version (Windows).
Git (to version your lab).
MetaTrader 5 installed from your broker, with a demo account logged in.

3.1 Windows (Topology A — recommended)

# From the project root, in PowerShell or cmd:

# 1. Create the virtual environment
python -m venv .venv

# 2. Activate it
.\.venv\Scripts\activate

# 3. Upgrade pip
python -m pip install --upgrade pip

# 4. Install the stack (MetaTrader5 included — Windows only)
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests MetaTrader5

3.2 macOS / Linux (Topology B/C — MT5 lives elsewhere)

# 1. Create the venv
python3 -m venv .venv

# 2. Activate
source .venv/bin/activate

# 3. Upgrade pip
python3 -m pip install --upgrade pip

# 4. Install the stack (NO MetaTrader5 package — it is Windows-only)
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests

On Apple Silicon (M-series) everything above is native arm64 and fast. numba/numpy ship arm64 wheels — no Rosetta needed.

3.3 Verify the install

# Use the venv's python explicitly to avoid the system interpreter
.venv/bin/python3 -c "import pandas, numpy, pyarrow, optuna, numba, yaml, lxml; print('core OK')"
# Windows: .\.venv\Scripts\python -c "..."

On Windows also verify the terminal link:

import MetaTrader5 as mt5
print(mt5.initialize())     # True if it found & launched the terminal
print(mt5.version())
mt5.shutdown()

4. Always use the venv interpreter

A recurring source of bugs is accidentally running the system Python (which lacks the libraries). Make it a habit to call the venv interpreter by path:

# macOS/Linux
.venv/bin/python3 your_script.py

# Windows
.\.venv\Scripts\python your_script.py

…or activate the venv at the start of every session. Pick one convention and keep it.

5. `.gitignore` essentials

Your lab will accumulate large data and secrets. Ignore them from day one:

.venv/
data/                # market data is large & re-downloadable
results/             # scratch run outputs
*.db                 # Optuna SQLite studies
*.htm                # pulled MT5 reports
.env                 # broker credentials — NEVER commit
__pycache__/
.DS_Store

6. Hardware notes

Backtesting is CPU + RAM bound, not GPU. A modern multi-core CPU and 16 GB+ RAM is plenty.
Optuna parallelizes across CPU cores. The practical cap for heavy concurrent backtests is roughly your number of performance cores — beyond that they contend and slow each other down. Prefer one Optuna study with n_jobs=N over N separate scripts fighting for cores.
Millions of M1 bars fit comfortably in RAM as a pandas frame; loading from Parquet is the only I/O.

Next: 02-architecture.md — the layered architecture you are about to build.

7.3 KiB Raw Permalink Blame History Unescape Escape