diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 76d4022..f3003c7 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -23,24 +23,26 @@ MT5-Quant/ │ │ ├── metrics.rs # Metrics parsing from HTML/XML │ │ └── report.rs # Report, PipelineMetadata, etc. │ ├── analytics/ # Report extraction & analysis (migrated from Python) -│ │ ├── extract.rs # HTML/XML report parser → metrics.json + deals.csv +│ │ ├── extract.rs # HTML/XML report parser → metrics.json (deals → DB) │ │ └── analyze.rs # Deal-level analysis engine → analysis.json │ ├── compile/ # MQL5 compilation │ │ └── mql_compiler.rs # MetaEditor wrapper (Wine/CrossOver) │ ├── pipeline/ # Backtest orchestration │ │ ├── backtest.rs # 5-stage pipeline (COMPILE→CLEAN→BACKTEST→EXTRACT→ANALYZE) │ │ └── stages.rs # Pipeline stage definitions +│ ├── storage/ # SQLite persistence +│ │ └── database.rs # ReportDb: reports table + deals table │ └── tools/ # MCP tool definitions -│ ├── definitions/ # Tool schemas (9 domain modules, 89 tools) +│ ├── definitions/ # Tool schemas (9 domain modules, 90 tools) │ │ ├── mod.rs -│ │ ├── analytics.rs # 9 analysis tools -│ │ ├── backtest.rs # 4 backtest tools +│ │ ├── analytics.rs # 19 analysis tools (DB-backed) +│ │ ├── backtest.rs # 7 backtest tools │ │ ├── baseline.rs # 1 baseline tool -│ │ ├── experts.rs # 4 EA/indicator/script tools +│ │ ├── experts.rs # 9 EA/indicator/script tools │ │ ├── optimization.rs # 4 optimization tools -│ │ ├── reports.rs # 11 report management tools +│ │ ├── reports.rs # 20 report management tools │ │ ├── setfiles.rs # 8 .set file tools -│ │ └── system.rs # 3 system tools +│ │ └── system.rs # 6 system tools │ └── handlers/ # Tool dispatch (9 domain modules) │ ├── mod.rs │ ├── analysis.rs @@ -145,17 +147,26 @@ MT5 runs in headless mode, writes the report, and exits. --- -### Stage 4: EXTRACT +### Stage 4: EXTRACT + STORE -Single HTML/XML parse pass that produces three artifacts: +Single HTML/XML parse pass. Deals go directly into the SQLite database; the raw report file is deleted afterwards. ```rust // src/analytics/extract.rs let extractor = ReportExtractor::new(); let result = extractor.extract(&report_path, &output_dir)?; -// → metrics.json (aggregate summary) -// → deals.csv (all deals, 13 columns) -// → deals.json (same data, JSON) +// → metrics.json (aggregate summary — written to report_dir) +// HTML report deleted after extraction + +// src/storage/database.rs +db.insert_deals(&report_id, &result.deals)?; +// → deals table in SQLite (all deals, keyed by report_id) +``` + +On-demand CSV export is available via the `export_deals_csv` tool: +``` +export_deals_csv(report_id: "20260422_051041_DPS21_XAUUSDc_M5_1") +// → report_dir/deals.csv (written only when explicitly requested) ``` **Why single-pass?** MT5 HTML reports are large (1-5MB for 14-month tests). Each regex pass over the file takes ~200ms. The old pipeline ran 5 separate grep/regex passes. The Rust implementation uses a single-pass parser: 5× faster and no partial-read inconsistencies. @@ -175,12 +186,13 @@ if ext == "xml" || path.ends_with(".htm.xml") { } ``` -**Deal columns (13):** +**Deal columns (stored in DB):** ``` -Time | Type | Direction | Volume | Price | S/L | T/P | Profit | Balance | Comment | Order | Magic | Entry +time | deal | symbol | deal_type | entry | volume | price | order_id +commission | swap | profit | balance | comment | magic ``` -The `Comment` column is the key to grid analytics. The EA writes `"Layer #3"`, `"Locking Total"`, `"Zombie Exit"` etc. Pattern matching on comments reconstructs which position was at which layer. +The `comment` column is the key to grid analytics. The EA writes `"Layer #3"`, `"Locking Total"`, `"Zombie Exit"` etc. Pattern matching on comments reconstructs which position was at which layer. --- diff --git a/docs/MCP_TOOLS.md b/docs/MCP_TOOLS.md index 417609b..4031db5 100644 --- a/docs/MCP_TOOLS.md +++ b/docs/MCP_TOOLS.md @@ -84,8 +84,8 @@ Run a complete backtest pipeline: compile → clean cache → backtest → extra files: { metrics_json: string; analysis_json: string; - deals_csv: string; - deals_json: string; + // Note: deals are stored in SQLite DB, not on disk. + // Call export_deals_csv(report_id) to generate a CSV file on demand. }; error?: string; // Present on failure @@ -128,9 +128,7 @@ Run a complete backtest pipeline: compile → clean cache → backtest → extra }, "files": { "metrics_json": "reports/20250619_143022_MyEA_XAUUSD_M5/metrics.json", - "analysis_json": "reports/20250619_143022_MyEA_XAUUSD_M5/analysis.json", - "deals_csv": "reports/20250619_143022_MyEA_XAUUSD_M5/deals.csv", - "deals_json": "reports/20250619_143022_MyEA_XAUUSD_M5/deals.json" + "analysis_json": "reports/20250619_143022_MyEA_XAUUSD_M5/analysis.json" } } ``` @@ -157,7 +155,7 @@ Same as `run_backtest`. Backtest only: clean cache → backtest → extract. No analysis phase. -**When to call:** When you just need raw trade data (deals.csv) and don't need analytics. Fastest option for batch processing. +**When to call:** When you just need raw trade data (stored in DB) and don't need analytics. Fastest option for batch processing. Use `export_deals_csv` afterwards if you need a CSV file. ### Input schema @@ -484,13 +482,15 @@ Delete old report directories to reclaim disk space, keeping the most recent N r ## `analyze_report` -Read and summarize a completed backtest report without re-running MT5. +Read and summarize a completed backtest report without re-running MT5. Loads deals from the SQLite database. ### Input schema ```typescript { - report_dir: string; // Path to report directory + report_id?: string; // Preferred: report ID from list_reports + report_dir?: string; // Legacy: path to report directory (looks up DB entry) + // Omit both to use the latest report automatically strategy?: "grid" | "scalper" | "trend" | "hedge" | "generic"; // Strategy profile that was used (default: "grid"). // Only affects interpretation of analysis.json fields — @@ -3057,6 +3057,22 @@ Find comparable reports (same EA/symbol/timeframe). --- +## Granular Analytics — Common Input Pattern + +All granular analytics tools below (`analyze_monthly_pnl` through `analyze_efficiency`, plus `list_deals` and `search_deals_*`) share the same report resolution logic: + +```typescript +{ + report_id?: string; // Preferred: ID from list_reports + report_dir?: string; // Legacy: filesystem path (looks up DB entry) + // Omit both → uses the latest report automatically +} +``` + +Deals are loaded from **SQLite DB**, not from CSV files. The `report_dir` parameter is kept for backward compatibility — if your report is in the DB, passing `report_dir` will resolve to its `report_id` automatically. + +--- + ## `analyze_monthly_pnl` Monthly P/L breakdown only. @@ -3067,8 +3083,10 @@ Monthly P/L breakdown only. ```typescript { - report_dir: string; // Path to report directory + report_id?: string; // Preferred: from list_reports + report_dir?: string; // Legacy: path to report directory } +// Omit all args to use the latest report ``` ### Output schema @@ -3333,7 +3351,7 @@ Peak simultaneous positions. ## `list_deals` -List individual deals with filters (type, profit range, volume, dates). +List individual deals with filters (type, profit range, volume, dates). Loads from SQLite DB. **When to call:** To query individual trades with specific criteria. @@ -3341,7 +3359,9 @@ List individual deals with filters (type, profit range, volume, dates). ```typescript { - report_dir: string; + report_id?: string; // Preferred: from list_reports + report_dir?: string; // Legacy: path to report directory + // Omit all to use latest report deal_type?: "buy" | "sell" | "all"; min_profit?: number; max_profit?: number; @@ -3377,7 +3397,7 @@ List individual deals with filters (type, profit range, volume, dates). ## `search_deals_by_comment` -Full-text search in deal comments (e.g., "Layer #3"). +Full-text search in deal comments (e.g., "Layer #3"). Loads from SQLite DB. **When to call:** To find deals with specific comment patterns. @@ -3385,8 +3405,9 @@ Full-text search in deal comments (e.g., "Layer #3"). ```typescript { - report_dir: string; - query: string; // Search pattern in comment field + report_id?: string; // Preferred: from list_reports + report_dir?: string; // Legacy: path to report directory + query: string; // Search pattern in comment field (required) limit?: number; } ``` diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 2965e35..c0f9a5d 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -82,6 +82,23 @@ Check `/MQL5/Logs/`: - Check `.set` file values appropriate for symbol/broker - Confirm `OnInit()` returns `INIT_SUCCEEDED` (MT5 Journal tab) +## Analytics Tools: "No reports in DB" or "Report not found" + +Analytics tools load deals from the **SQLite database**, not from CSV files on disk. Resolution order: + +1. `report_id` (preferred) — ID from `list_reports` +2. `report_dir` (legacy) — filesystem path, looks up matching DB entry +3. No args — uses the latest report automatically + +Pre-DB reports (before this version) won't be found by `report_dir`. Re-run the backtest to get a DB-backed report. + +**`deals.csv` is no longer written automatically.** Call `export_deals_csv` to generate one on demand: +``` +export_deals_csv() # latest report → report_dir/deals.csv +export_deals_csv(report_id: "20260422_…") # specific report +export_deals_csv(output_path: "/tmp/out.csv") # custom path +``` + ## Optimization Never Finishes ```bash