Update documentation for new debugging and diagnostic tools

docs/ARCHITECTURE.md

@@ -23,24 +23,26 @@ MT5-Quant/
 │   │   ├── metrics.rs          # Metrics parsing from HTML/XML
 │   │   └── report.rs           # Report, PipelineMetadata, etc.
 │   ├── analytics/              # Report extraction & analysis (migrated from Python)
-│   │   ├── extract.rs          # HTML/XML report parser → metrics.json + deals.csv
+│   │   ├── extract.rs          # HTML/XML report parser → metrics.json (deals → DB)
 │   │   └── analyze.rs          # Deal-level analysis engine → analysis.json
 │   ├── compile/                # MQL5 compilation
 │   │   └── mql_compiler.rs   # MetaEditor wrapper (Wine/CrossOver)
 │   ├── pipeline/               # Backtest orchestration
 │   │   ├── backtest.rs         # 5-stage pipeline (COMPILE→CLEAN→BACKTEST→EXTRACT→ANALYZE)
 │   │   └── stages.rs           # Pipeline stage definitions
+│   ├── storage/                # SQLite persistence
+│   │   └── database.rs         # ReportDb: reports table + deals table
 │   └── tools/                  # MCP tool definitions
-│       ├── definitions/        # Tool schemas (9 domain modules, 89 tools)
+│       ├── definitions/        # Tool schemas (9 domain modules, 90 tools)
 │       │   ├── mod.rs
-│       │   ├── analytics.rs      # 9 analysis tools
+│       │   ├── analytics.rs      # 19 analysis tools (DB-backed)
-│       │   ├── backtest.rs       # 4 backtest tools
+│       │   ├── backtest.rs       # 7 backtest tools
 │       │   ├── baseline.rs       # 1 baseline tool
-│       │   ├── experts.rs        # 4 EA/indicator/script tools
+│       │   ├── experts.rs        # 9 EA/indicator/script tools
 │       │   ├── optimization.rs   # 4 optimization tools
-│       │   ├── reports.rs        # 11 report management tools
+│       │   ├── reports.rs        # 20 report management tools
 │       │   ├── setfiles.rs       # 8 .set file tools
-│       │   └── system.rs         # 3 system tools
+│       │   └── system.rs         # 6 system tools
 │       └── handlers/             # Tool dispatch (9 domain modules)
 │           ├── mod.rs
 │           ├── analysis.rs
@@ -145,17 +147,26 @@ MT5 runs in headless mode, writes the report, and exits.
 
 ---
 
-### Stage 4: EXTRACT
+### Stage 4: EXTRACT + STORE
 
-Single HTML/XML parse pass that produces three artifacts:
+Single HTML/XML parse pass. Deals go directly into the SQLite database; the raw report file is deleted afterwards.
 
 ```rust
 // src/analytics/extract.rs
 let extractor = ReportExtractor::new();
 let result = extractor.extract(&report_path, &output_dir)?;
-// → metrics.json  (aggregate summary)
+// → metrics.json  (aggregate summary — written to report_dir)
-// → deals.csv     (all deals, 13 columns)
+// HTML report deleted after extraction
-// → deals.json    (same data, JSON)
+
+// src/storage/database.rs
+db.insert_deals(&report_id, &result.deals)?;
+// → deals table in SQLite (all deals, keyed by report_id)
+```
+
+On-demand CSV export is available via the `export_deals_csv` tool:
+```
+export_deals_csv(report_id: "20260422_051041_DPS21_XAUUSDc_M5_1")
+// → report_dir/deals.csv  (written only when explicitly requested)
 ```
 
 **Why single-pass?** MT5 HTML reports are large (1-5MB for 14-month tests). Each regex pass over the file takes ~200ms. The old pipeline ran 5 separate grep/regex passes. The Rust implementation uses a single-pass parser: 5× faster and no partial-read inconsistencies.
@@ -175,12 +186,13 @@ if ext == "xml" || path.ends_with(".htm.xml") {
 }
 ```
 
-**Deal columns (13):**
+**Deal columns (stored in DB):**
 ```
-Time | Type | Direction | Volume | Price | S/L | T/P | Profit | Balance | Comment | Order | Magic | Entry
+time | deal | symbol | deal_type | entry | volume | price | order_id
+commission | swap | profit | balance | comment | magic
 ```
 
-The `Comment` column is the key to grid analytics. The EA writes `"Layer #3"`, `"Locking Total"`, `"Zombie Exit"` etc. Pattern matching on comments reconstructs which position was at which layer.
+The `comment` column is the key to grid analytics. The EA writes `"Layer #3"`, `"Locking Total"`, `"Zombie Exit"` etc. Pattern matching on comments reconstructs which position was at which layer.
 
 ---
 

docs/MCP_TOOLS.md

@@ -84,8 +84,8 @@ Run a complete backtest pipeline: compile → clean cache → backtest → extra
   files: {
     metrics_json: string;
     analysis_json: string;
-    deals_csv: string;
+    // Note: deals are stored in SQLite DB, not on disk.
-    deals_json: string;
+    // Call export_deals_csv(report_id) to generate a CSV file on demand.
   };
 
   error?: string;  // Present on failure
@@ -128,9 +128,7 @@ Run a complete backtest pipeline: compile → clean cache → backtest → extra
   },
   "files": {
     "metrics_json": "reports/20250619_143022_MyEA_XAUUSD_M5/metrics.json",
-    "analysis_json": "reports/20250619_143022_MyEA_XAUUSD_M5/analysis.json",
+    "analysis_json": "reports/20250619_143022_MyEA_XAUUSD_M5/analysis.json"
-    "deals_csv": "reports/20250619_143022_MyEA_XAUUSD_M5/deals.csv",
-    "deals_json": "reports/20250619_143022_MyEA_XAUUSD_M5/deals.json"
   }
 }
 ```
@@ -157,7 +155,7 @@ Same as `run_backtest`.
 
 Backtest only: clean cache → backtest → extract. No analysis phase.
 
-**When to call:** When you just need raw trade data (deals.csv) and don't need analytics. Fastest option for batch processing.
+**When to call:** When you just need raw trade data (stored in DB) and don't need analytics. Fastest option for batch processing. Use `export_deals_csv` afterwards if you need a CSV file.
 
 ### Input schema
 
@@ -484,13 +482,15 @@ Delete old report directories to reclaim disk space, keeping the most recent N r
 
 ## `analyze_report`
 
-Read and summarize a completed backtest report without re-running MT5.
+Read and summarize a completed backtest report without re-running MT5. Loads deals from the SQLite database.
 
 ### Input schema
 
 ```typescript
 {
-  report_dir: string;          // Path to report directory
+  report_id?: string;          // Preferred: report ID from list_reports
+  report_dir?: string;         // Legacy: path to report directory (looks up DB entry)
+                               // Omit both to use the latest report automatically
   strategy?: "grid" | "scalper" | "trend" | "hedge" | "generic";
                                // Strategy profile that was used (default: "grid").
                                // Only affects interpretation of analysis.json fields —
@@ -3057,6 +3057,22 @@ Find comparable reports (same EA/symbol/timeframe).
 
 ---
 
+## Granular Analytics — Common Input Pattern
+
+All granular analytics tools below (`analyze_monthly_pnl` through `analyze_efficiency`, plus `list_deals` and `search_deals_*`) share the same report resolution logic:
+
+```typescript
+{
+  report_id?: string;   // Preferred: ID from list_reports
+  report_dir?: string;  // Legacy: filesystem path (looks up DB entry)
+  // Omit both → uses the latest report automatically
+}
+```
+
+Deals are loaded from **SQLite DB**, not from CSV files. The `report_dir` parameter is kept for backward compatibility — if your report is in the DB, passing `report_dir` will resolve to its `report_id` automatically.
+
+---
+
 ## `analyze_monthly_pnl`
 
 Monthly P/L breakdown only.
@@ -3067,8 +3083,10 @@ Monthly P/L breakdown only.
 
 ```typescript
 {
-  report_dir: string;       // Path to report directory
+  report_id?: string;       // Preferred: from list_reports
+  report_dir?: string;      // Legacy: path to report directory
 }
+// Omit all args to use the latest report
 ```
 
 ### Output schema
@@ -3333,7 +3351,7 @@ Peak simultaneous positions.
 
 ## `list_deals`
 
-List individual deals with filters (type, profit range, volume, dates).
+List individual deals with filters (type, profit range, volume, dates). Loads from SQLite DB.
 
 **When to call:** To query individual trades with specific criteria.
 
@@ -3341,7 +3359,9 @@ List individual deals with filters (type, profit range, volume, dates).
 
 ```typescript
 {
-  report_dir: string;
+  report_id?: string;       // Preferred: from list_reports
+  report_dir?: string;      // Legacy: path to report directory
+  // Omit all to use latest report
   deal_type?: "buy" | "sell" | "all";
   min_profit?: number;
   max_profit?: number;
@@ -3377,7 +3397,7 @@ List individual deals with filters (type, profit range, volume, dates).
 
 ## `search_deals_by_comment`
 
-Full-text search in deal comments (e.g., "Layer #3").
+Full-text search in deal comments (e.g., "Layer #3"). Loads from SQLite DB.
 
 **When to call:** To find deals with specific comment patterns.
 
@@ -3385,8 +3405,9 @@ Full-text search in deal comments (e.g., "Layer #3").
 
 ```typescript
 {
-  report_dir: string;
+  report_id?: string;      // Preferred: from list_reports
-  query: string;           // Search pattern in comment field
+  report_dir?: string;     // Legacy: path to report directory
+  query: string;           // Search pattern in comment field (required)
   limit?: number;
 }
 ```

docs/TROUBLESHOOTING.md

@@ -82,6 +82,23 @@ Check `<terminal_dir>/MQL5/Logs/`:
 - Check `.set` file values appropriate for symbol/broker
 - Confirm `OnInit()` returns `INIT_SUCCEEDED` (MT5 Journal tab)
 
+## Analytics Tools: "No reports in DB" or "Report not found"
+
+Analytics tools load deals from the **SQLite database**, not from CSV files on disk. Resolution order:
+
+1. `report_id` (preferred) — ID from `list_reports`
+2. `report_dir` (legacy) — filesystem path, looks up matching DB entry
+3. No args — uses the latest report automatically
+
+Pre-DB reports (before this version) won't be found by `report_dir`. Re-run the backtest to get a DB-backed report.
+
+**`deals.csv` is no longer written automatically.** Call `export_deals_csv` to generate one on demand:
+```
+export_deals_csv()                              # latest report → report_dir/deals.csv
+export_deals_csv(report_id: "20260422_…")      # specific report
+export_deals_csv(output_path: "/tmp/out.csv")  # custom path
+```
+
 ## Optimization Never Finishes
 
 ```bash