gavindiaz/mymt5opp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
gavindiaz
2026-06-26 18:47:35 +08:00
commit 49be922517
11 changed files with 1781 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-stack-and-install.md
+180
View File
@@ -0,0 +1,180 @@
# 01 — Tech Stack & Installation From Scratch
Everything you need to install, what each piece is for, where to get it, and the exact commands per
operating system. Nothing here is strategy-specific — this is the plumbing.
---
## 1. The stack at a glance
| Layer | Tool | Role | Where to get it |
|-------|------|------|-----------------|
| Language | **Python 3.12+** (3.13/3.14 work) | everything outside MT5 | <https://www.python.org/downloads/> |
| Data frames | **pandas** | OHLC tables, resampling, equity curves | <https://pandas.pydata.org> · `pip install pandas` |
| Numerics | **numpy** | vectorized price/indicator math | <https://numpy.org> · `pip install numpy` |
| Columnar storage | **pyarrow** | read/write **Parquet** market data (fast, compact) | <https://arrow.apache.org/docs/python/> · `pip install pyarrow` |
| JIT speed | **numba** | compiles the hot bar-by-bar loop to machine code | <https://numba.pydata.org> · `pip install numba` |
| Optimizer | **Optuna** | Bayesian hyper-parameter search | <https://optuna.org> · `pip install optuna` |
| Optuna storage | **SQLAlchemy** (+ **alembic**) | persists studies to a SQLite DB so runs resume/parallelize | <https://www.sqlalchemy.org> · `pip install sqlalchemy` |
| Config | **PyYAML** | reproducible run settings (`wizard-answers.yaml`) | <https://pyyaml.org> · `pip install pyyaml` |
| HTML parsing | **lxml** + **html5lib** | parse the MT5 Strategy Tester HTML report | <https://lxml.de> · `pip install lxml html5lib` |
| Progress / logs | **tqdm**, **colorlog** | progress bars, readable logs | `pip install tqdm colorlog` |
| HTTP | **requests** | optional data downloads / webhooks | <https://requests.readthedocs.io> · `pip install requests` |
| Tester | **MetaTrader 5** terminal | the gold-standard backtester + data source | from your broker, or <https://www.metatrader5.com/en/download> |
| MT5 control (Windows) | **MetaTrader5** pip package | drive the terminal & pull data from Python (Windows only) | <https://pypi.org/project/MetaTrader5/> · `pip install MetaTrader5` |
> **Optional data source.** If you want history without MT5 export, `dukascopy-python`
> (`pip install dukascopy-python`, <https://pypi.org/project/dukascopy-python/>) pulls free tick/bar
> data for many symbols. MT5-exported data from your own broker is preferred because it matches the
> tester exactly.
### Reference versions
A known-good combination (early 2026) — use these or the latest stable:
```
python 3.14
pandas 3.0
numpy 2.4
pyarrow 24.0
numba 0.65
optuna 4.8
sqlalchemy 2.0
pyyaml 6.0
lxml 6.1
```
Pin exact versions in a `requirements.txt` once your lab works, so it reproduces later.
---
## 2. What each library actually does here
- **pandas / numpy** — market data is loaded into a DataFrame of `[timestamp, open, high, low, close,
spread]`. Indicators and signals are computed as numpy arrays. Equity curves are pandas frames.
- **pyarrow + Parquet** — a few years of M1 (one-minute) bars is millions of rows. Parquet stores it
columnar and compressed: a multi-million-row file loads in well under a second and is a fraction of
CSV size. This is what makes "full-history backtest in seconds" possible.
- **numba** — the engine's inner loop walks every bar (and sub-ticks within each bar). Pure-Python
that is too slow. `@njit` compiles it to native code on first call; subsequent runs are C-fast.
- **Optuna** — instead of brute-forcing a parameter grid, Optuna uses a Bayesian sampler (TPE) that
*learns* which regions of the search space are promising and concentrates trials there. You get a
good optimum in hundreds–thousands of trials instead of an exhaustive grid of millions.
- **SQLAlchemy/SQLite** — Optuna writes each trial to a `study.db`. That means a study can be stopped
and resumed, inspected mid-run (count completed trials), and run with multiple worker processes
pointing at the same DB.
- **lxml / html5lib** — the MT5 tester exports its report as an HTML file encoded **UTF-16-LE**. These
parse it into a metrics dict (Net Profit, Profit Factor, Drawdown, trade count, …).
- **MetaTrader5 package** — on Windows, this is the clean way to (a) download historical bars and
(b) launch/script the terminal. On non-Windows you don't have it, which is why remote topologies use
SSH + a scheduled task instead.
---
## 3. Install — step by step
### 3.0 Prerequisites
- **Python 3.12+**. Check with `python3 --version` (macOS/Linux) or `python --version` (Windows).
- **Git** (to version your lab).
- **MetaTrader 5** installed from your broker, with a **demo account** logged in.
### 3.1 Windows (Topology A — recommended)
```powershell
# From the project root, in PowerShell or cmd:
# 1. Create the virtual environment
python -m venv .venv
# 2. Activate it
.\.venv\Scripts\activate
# 3. Upgrade pip
python -m pip install --upgrade pip
# 4. Install the stack (MetaTrader5 included — Windows only)
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests MetaTrader5
```
### 3.2 macOS / Linux (Topology B/C — MT5 lives elsewhere)
```bash
# 1. Create the venv
python3 -m venv .venv
# 2. Activate
source .venv/bin/activate
# 3. Upgrade pip
python3 -m pip install --upgrade pip
# 4. Install the stack (NO MetaTrader5 package — it is Windows-only)
pip install pandas numpy pyarrow numba optuna sqlalchemy alembic pyyaml lxml html5lib tqdm colorlog requests
```
> On Apple Silicon (M-series) everything above is native arm64 and fast. `numba`/`numpy` ship arm64
> wheels — no Rosetta needed.
### 3.3 Verify the install
```bash
# Use the venv's python explicitly to avoid the system interpreter
.venv/bin/python3 -c "import pandas, numpy, pyarrow, optuna, numba, yaml, lxml; print('core OK')"
# Windows: .\.venv\Scripts\python -c "..."
```
On Windows also verify the terminal link:
```python
import MetaTrader5 as mt5
print(mt5.initialize()) # True if it found & launched the terminal
print(mt5.version())
mt5.shutdown()
```
---
## 4. Always use the venv interpreter
A recurring source of bugs is accidentally running the **system** Python (which lacks the libraries).
Make it a habit to call the venv interpreter by path:
```bash
# macOS/Linux
.venv/bin/python3 your_script.py
# Windows
.\.venv\Scripts\python your_script.py
```
…or activate the venv at the start of every session. Pick one convention and keep it.
---
## 5. `.gitignore` essentials
Your lab will accumulate large data and secrets. Ignore them from day one:
```gitignore
.venv/
data/ # market data is large & re-downloadable
results/ # scratch run outputs
*.db # Optuna SQLite studies
*.htm # pulled MT5 reports
.env # broker credentials — NEVER commit
__pycache__/
.DS_Store
```
---
## 6. Hardware notes
- Backtesting is **CPU + RAM** bound, not GPU. A modern multi-core CPU and 16 GB+ RAM is plenty.
- Optuna parallelizes across CPU cores. The practical cap for *heavy* concurrent backtests is roughly
your number of **performance cores** — beyond that they contend and slow each other down. Prefer
**one** Optuna study with `n_jobs=N` over N separate scripts fighting for cores.
- Millions of M1 bars fit comfortably in RAM as a pandas frame; loading from Parquet is the only I/O.
Next: [`02-architecture.md`](02-architecture.md) — the layered architecture you are about to build.