feat: complete Rust migration with modular architecture
Major changes: - Migrate Python/shell scripts to Rust modules: - analytics/extract.py → src/analytics/extract.rs (ReportExtractor) - analytics/analyze.py → src/analytics/analyze.rs (DealAnalyzer) - scripts/mqlcompile.sh → src/compile/mql_compiler.rs (MqlCompiler) - scripts/backtest_pipeline.sh → src/pipeline/backtest.rs (BacktestPipeline) - New modular structure: - src/models/ - Config, Deal, Metrics, Report structs - src/analytics/ - Report parsing and deal analysis - src/compile/ - MQL5 compilation via Wine - src/pipeline/ - 5-stage backtest orchestration - src/tools/ - 27 MCP tool definitions and handlers - Remove PyInstaller setup (now pure Rust) - Remove migrated shell scripts (backtest_pipeline.sh, mqlcompile.sh) - Add GitHub Actions CI/CD for macOS & Linux releases - Update all documentation for Rust architecture Binary size: 4.3MB (no Python dependencies) Tools: 27 MCP tools fully functional
This commit is contained in:
Devid HW
+70
-60
@@ -16,36 +16,45 @@ MT5-Quant extracts every individual deal — entry price, exit price, P/L, comme
|
||||
|
||||
```
|
||||
MT5-Quant/
|
||||
├── server/
|
||||
│ └── main.py # MCP server (stdio transport) — 10 tool handlers
|
||||
├── src/
|
||||
│ ├── main.rs # MCP server entry (stdio transport)
|
||||
│ ├── mcp_server.rs # MCP protocol handling
|
||||
│ ├── models/ # Data structures
|
||||
│ │ ├── config.rs # Configuration
|
||||
│ │ ├── deals.rs # Deal, PositionPair, DrawdownEvent, etc.
|
||||
│ │ ├── metrics.rs # Metrics parsing from HTML/XML
|
||||
│ │ └── report.rs # Report, PipelineMetadata, etc.
|
||||
│ ├── analytics/ # Report extraction & analysis (migrated from Python)
|
||||
│ │ ├── extract.rs # HTML/XML report parser → metrics.json + deals.csv
|
||||
│ │ └── analyze.rs # Deal-level analysis engine → analysis.json
|
||||
│ ├── compile/ # MQL5 compilation
|
||||
│ │ └── mql_compiler.rs # MetaEditor wrapper (Wine/CrossOver)
|
||||
│ ├── pipeline/ # Backtest orchestration
|
||||
│ │ ├── backtest.rs # 5-stage pipeline (COMPILE→CLEAN→BACKTEST→EXTRACT→ANALYZE)
|
||||
│ │ └── stages.rs # Pipeline stage definitions
|
||||
│ └── tools/ # MCP tool definitions
|
||||
│ ├── definitions.rs # 27 tool schemas
|
||||
│ └── handlers.rs # Tool dispatch
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── setup.sh # Auto-detect Wine/MT5, write config, register MCP
|
||||
│ ├── platform_detect.sh # Sourced by all scripts — Wine path + headless detection
|
||||
│ ├── backtest_pipeline.sh # 5-stage pipeline orchestrator
|
||||
│ ├── optimize.sh # Genetic optimization launcher (nohup + disown)
|
||||
│ └── mqlcompile.sh # MetaEditor wrapper (Wine/CrossOver)
|
||||
│ ├── setup.sh # Auto-detect Wine/MT5, write config, register MCP
|
||||
│ ├── platform_detect.sh # Wine path + headless detection
|
||||
│ ├── build-rust.sh # Rust build script
|
||||
│ └── optimize.sh # Genetic optimization launcher (nohup + disown)
|
||||
│
|
||||
├── analytics/
|
||||
│ ├── extract.py # HTML/XML report parser → metrics.json + deals.csv
|
||||
│ ├── analyze.py # Deal-level analysis engine → analysis.json
|
||||
│ └── optimize_parser.py # Optimization result parser
|
||||
├── analytics/ # Legacy Python (reference only)
|
||||
│ ├── extract.py
|
||||
│ ├── analyze.py
|
||||
│ └── optimize_parser.py
|
||||
│
|
||||
├── config/
|
||||
│ ├── MT5-Quant.example.yaml # Template config (copy to MT5-Quant.yaml)
|
||||
│ ├── MT5-Quant.yaml # Live config (gitignored)
|
||||
│ ├── baseline.json # Production baseline metrics (gitignored, user-maintained)
|
||||
│ ├── CLAUDE.template.md # CLAUDE.md template (generated by --claude-code)
|
||||
│ └── example.set # Example optimization .set file
|
||||
│
|
||||
├── .claude/
|
||||
│ └── hooks/
|
||||
│ └── user-prompt-submit.sh # Injects baseline.json into every Claude prompt
|
||||
│ ├── mt5-quant.example.yaml # Template config
|
||||
│ └── mt5-quant.yaml # Live config (gitignored)
|
||||
│
|
||||
└── docs/
|
||||
├── ARCHITECTURE.md # This file
|
||||
├── MCP_TOOLS.md # Full tool spec
|
||||
└── REMOTE_AGENTS.md # Linux agent farm setup
|
||||
├── ARCHITECTURE.md # This file
|
||||
├── MCP_TOOLS.md # Full tool spec
|
||||
└── REMOTE_AGENTS.md # Linux agent farm setup
|
||||
```
|
||||
|
||||
---
|
||||
@@ -54,11 +63,13 @@ MT5-Quant/
|
||||
|
||||
### Stage 1: COMPILE
|
||||
|
||||
```bash
|
||||
./scripts/mqlcompile.sh src/experts/MyEA.mq5
|
||||
```rust
|
||||
// src/compile/mql_compiler.rs
|
||||
let compiler = MqlCompiler::new(config);
|
||||
let result = compiler.compile("src/experts/MyEA.mq5")?;
|
||||
```
|
||||
|
||||
Invokes MetaEditor via Wine with the MQL5 source file. Copies resulting `.ex5` to the MT5 Experts directory. Fails the pipeline on compile errors (stderr contains error count).
|
||||
Invokes MetaEditor via Wine with the MQL5 source file. Copies resulting `.ex5` to the MT5 Experts directory. Fails the pipeline on compile errors.
|
||||
|
||||
**Why not skip this?** MT5 caches the `.ex5` binary by filename. If you edit your EA and re-run without recompiling, MT5 runs the old binary silently. Always compile.
|
||||
|
||||
@@ -128,29 +139,30 @@ MT5 runs in headless mode, writes the report, and exits.
|
||||
|
||||
Single HTML/XML parse pass that produces three artifacts:
|
||||
|
||||
```bash
|
||||
python3 analytics/extract.py report.htm
|
||||
# → metrics.json (aggregate summary)
|
||||
# → deals.csv (all deals, 13 columns)
|
||||
# → deals.json (same data, JSON)
|
||||
```rust
|
||||
// src/analytics/extract.rs
|
||||
let extractor = ReportExtractor::new();
|
||||
let result = extractor.extract(&report_path, &output_dir)?;
|
||||
// → metrics.json (aggregate summary)
|
||||
// → deals.csv (all deals, 13 columns)
|
||||
// → deals.json (same data, JSON)
|
||||
```
|
||||
|
||||
**Why single-pass?** MT5 HTML reports are large (1-5MB for 14-month tests). Each regex pass over the file takes ~200ms. The old pipeline ran 5 separate grep/regex passes, one per artifact. Collapsed to one Python pass: 5× faster and no partial-read inconsistencies.
|
||||
**Why single-pass?** MT5 HTML reports are large (1-5MB for 14-month tests). Each regex pass over the file takes ~200ms. The old pipeline ran 5 separate grep/regex passes. The Rust implementation uses a single-pass parser: 5× faster and no partial-read inconsistencies.
|
||||
|
||||
**Format detection:**
|
||||
```python
|
||||
# MT5 Build 48+ saves SpreadsheetML XML, not HTML
|
||||
if report_path.endswith('.xml') or report_path.endswith('.htm.xml'):
|
||||
tree = ET.parse(report_path)
|
||||
# parse via ElementTree
|
||||
else:
|
||||
with open(report_path, 'rb') as f:
|
||||
raw = f.read()
|
||||
try:
|
||||
text = raw.decode('utf-16')
|
||||
except:
|
||||
text = raw.decode('latin-1', errors='replace')
|
||||
# parse via regex
|
||||
```rust
|
||||
// MT5 Build 48+ saves SpreadsheetML XML, not HTML
|
||||
let ext = Path::new(&path).extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.unwrap_or("");
|
||||
|
||||
if ext == "xml" || path.ends_with(".htm.xml") {
|
||||
// Parse as SpreadsheetML XML
|
||||
let doc = roxmltree::Document::parse(&text)?;
|
||||
} else {
|
||||
// Parse as HTML with regex
|
||||
}
|
||||
```
|
||||
|
||||
**Deal columns (13):**
|
||||
@@ -164,23 +176,21 @@ The `Comment` column is the key to grid analytics. The EA writes `"Layer #3"`, `
|
||||
|
||||
### Stage 5: ANALYZE
|
||||
|
||||
```bash
|
||||
# Strategy subcommand (recommended)
|
||||
python3 analytics/analyze.py grid deals.csv --output-dir reports/20250101/
|
||||
python3 analytics/analyze.py scalper deals.csv --output-dir reports/20250101/
|
||||
python3 analytics/analyze.py trend deals.csv --output-dir reports/20250101/
|
||||
python3 analytics/analyze.py hedge deals.csv --output-dir reports/20250101/
|
||||
python3 analytics/analyze.py generic deals.csv --output-dir reports/20250101/
|
||||
|
||||
# Legacy positional (no subcommand) — defaults to 'grid', backward compatible
|
||||
python3 analytics/analyze.py deals.csv --output-dir reports/20250101/
|
||||
|
||||
# Flags available with any strategy
|
||||
python3 analytics/analyze.py grid deals.csv --output-dir DIR --deep --stdout
|
||||
# → analysis.json
|
||||
```rust
|
||||
// src/analytics/analyze.rs
|
||||
let analyzer = DealAnalyzer::new();
|
||||
let result = analyzer.analyze(&deals, &metrics, strategy, deep)?;
|
||||
// → analysis.json
|
||||
```
|
||||
|
||||
All functions operate on the 13-column `deals.csv` — no MT5 or Wine required.
|
||||
All functions operate on the parsed deal data — no MT5 or Wine required.
|
||||
|
||||
**Strategy profiles** (defined in `analyze.rs`):
|
||||
- `grid` — Layer depth tracking, locking/cutloss/zombie keywords
|
||||
- `scalper` — TP/SL/manual/trailing exit classification
|
||||
- `trend` — TP/SL/trailing/breakeven/partial exits
|
||||
- `hedge` — TP/SL/net_close/partial, magic+direction grouping
|
||||
- `generic` — Simple profit-based TP/SL classification
|
||||
|
||||
#### Strategy profiles
|
||||
|
||||
|
||||
Reference in New Issue
Block a user