Commit Graph

17 Commits

Author SHA1 Message Date
LEGSTECH Optimizer d321ea23ca docs: TODO_NEXT.md — post-hackathon roadmap (PyInstaller + first-run wizard)
Saved as a self-contained prompt + feature list so it can be picked up
in a fresh Claude Code session next weekend without re-explaining
context. Top of the list: one-click .exe installer + first-run wizard
that detects MT5 via Windows registry and walks the user through API
key + EA selection.
2026-04-25 13:33:21 +00:00
LEGSTECH Optimizer 405f769ef2 fix: replace placeholder GitHub URLs with real tonnylegacy/MT5_Optimizer
Sidebar 'Source on GitHub' link and README clone command both pointed at
the placeholder 'your-user' / '<your-user>' rather than the real
tonnylegacy account, so clicking from the dashboard 404'd.
2026-04-25 13:29:00 +00:00
LEGSTECH Optimizer d95e5102c6 fix: V4-style EAs now expose optimizable params (auto-infer ranges)
Bug: a .set file saved out of MT5's "save settings" button (rather than
"save optimization") contains bare \`Inp...=value\` lines with no
\`|min|max|step\` metadata. The parser previously marked every such param
as type="fixed", and /api/ea_params filters fixed params out, so the user
saw "No optimizable parameters found in .set file" and could not start a
run with that EA. Reproduces with LEGSTECH_EA_V4 in this user's setup.

Fix:
- SetParser.parse() now takes an auto_infer_ranges flag (default True).
  When a line has no '|' separator, the parser falls back to a heuristic
  range based on the param's name + value:
  * `*Pips`, `*Period`, `*Lookback`, `*Spread`, `*Trades` → integer ±50% / ±150%
  * `*Percent`, `*Pct`, `*Risk`, `*Buffer`, `*Multiplier`, `*Ratio` → float ±50% / ×2
  * `RRRatio` → 0.5–2.5x with 0.25 step
  * `Score` → 0.5–1.5x with 0.05 step
  * `Lot*` → ±50% with 0.01 step
  * `Hour*` / `Session*` → 0–23
  * Pure 0/1 with `Use*`/`Enable*`/`Allow*` prefix → bool
  * `true`/`false` literals → bool
  * Generic numeric fallback → ±50% with type-appropriate step
- Added timeframe enum names (htf/mtf/ltf/_tf/timeframe) and debug/log
  flags to _FORCE_FIXED_PATTERNS so MT5 internal constants like
  InpHTF=16388 don't get incorrectly inferred as scalars.

Verified on LEGSTECH_EA_V4 (raw saved-settings format): 64/71 params now
optimizable (was 0); regression-checked LEGSTECH_EA_V2 still returns 42
params from its existing optimization metadata.
2026-04-25 13:23:14 +00:00
LEGSTECH Optimizer 51a73c953b docs: Cerebral Valley Opus 4.7 hackathon submission packet
- AIReasoner default model now claude-opus-4-7 (was sonnet-4-6); pipeline
  reads ai.model from config and passes to constructor. The reasoner
  class attribute is also Opus 4.7 so any caller without explicit model
  defaults to the hackathon model
- README hero gains a "Built with Claude Opus 4.7" badge + a line linking
  to the Cerebral Valley × Anthropic hackathon page
- New SUBMISSION.md — copy-paste blocks for every field of the
  submission form: name, taglines, short / long description, "how Claude
  is used" detail, GitHub URL, demo instructions, tags, tech stack,
  team, license, and a final pre-submission checklist
2026-04-25 13:00:53 +00:00
LEGSTECH Optimizer a584f46891 feat: 6 user-facing upgrades + realistic demo metrics + animated hero
Demo + assets
- Synthetic backtests now occasionally fail (regime failures, OOS degradation)
  so verdicts span RECOMMENDED / RISKY / NOT_RELIABLE realistically. Phase 1
  has ~35% failure rate, Phase 2 AI loop ~10%, OOS has 30% chance of severe
  degradation — matches what real markets look like
- screenshots/dashboard.png + best_result_modal.png regenerated against the
  current UI; screenshots/apex_demo.gif (6-frame autonomous-run timelapse)
  embedded in the README

FEATURE 1 — Live AI token streaming
- AIReasoner._call_claude() now streams via SSE when a callback is
  registered. Each text delta forwards to the dashboard as
  `ai_thinking_chunk` events
- The Live AI Thinking Feed renders a single growing bubble with a blinking
  cursor while text streams in, finalising on `end`. Looks and feels like
  watching the AI type

FEATURE 2 — Pre-flight check on /setup
- New /api/preflight endpoint runs 5–7 probes: config readable, API key
  set, MT5 paths exist (skipped in demo), EA registered, reports folder
  writable. Returns {ok, blocking_count, checks[]}
- Setup page renders a colour-coded checklist on load and refocus.
  Replaces "click Start, wait 5s, see generic error"

FEATURE 3 — Hot-reload settings into the running pipeline
- pipeline.reload_config() applies AI model / timeout / API-key swaps to
  the live reasoner mid-run. Threshold changes surface for next run
- /api/settings POST detects a running pipeline and calls reload_config(),
  returning the changed keys plus a "hot-reloaded into the running
  optimization" note

FEATURE 4 — Replay scrubber on Best Result
- Evolution path now renders as an interactive scrubber: range slider +
  prev/next/play buttons. Each step shows the run ID, phase, score, full
  metrics grid, parameter changes for that step, and the AI's analysis
  text — auto-plays at 700ms/step

FEATURE 5 — Compare runs on /reports
- Each card has a checkbox; selecting 2–4 reveals a floating Compare bar.
  Compare modal renders a side-by-side table with metric winners
  highlighted (Calmar / PF / profit favour higher; DD favours lower)
  and a parameter-diff section showing changed values

FEATURE 6 — Discord / Slack / generic webhook on completion
- New `notifications.webhook_url` + `webhook_style` config keys
- Auto-detects Discord vs Slack from the URL host. Posts a one-line
  summary on `optimization_complete`: verdict + best run + PF/Calmar/DD/
  profit/trades/elapsed
2026-04-25 12:53:27 +00:00
LEGSTECH Optimizer c42345ea1e fix: live-activity persistence on refresh + reports page rebuild
Refresh-during-run no longer wipes the dashboard
- pipeline now keeps in-memory rolling logs of ai_thinking, param_changes,
  validation events, and the current early-termination state. Logs reset at
  the start of each new run (capped: 200 thinking / 50 param-change records /
  40 validation events)
- new GET /api/live_activity returns those logs + current phase + running
  state in one shot — the dashboard hits it on page load
- restoreHistory() in dashboard.js now replays each event into addThinking /
  addParamChanges / validationRunStart-Complete-Done / showEarlyTermination,
  and re-applies the active phase via setPhaseActive. Reload F5 mid-run no
  longer shows a fresh empty dashboard
- addThinking() preserves the original ts on replay (was using nowStr() so
  every replayed entry got the refresh time)

Reports page (/reports) rebuilt
- previous template crashed with 500 on legacy summary.json files that
  pre-date the win_rate / drawdown_pct fields. Server now backfills sane
  defaults for every metric the template touches
- runs now sorted by ts (was filesystem-iterdir order)
- new template: search bar + filter chips (All / Exploration / AI Iteration
  / Validation / AI insight / Has .set), phase tags, AI/.set badges, three
  action buttons per card (View Params / Download .set / Full Report)
- per-card detail modal shows metrics grid + full parameters table +
  AI reasoning + Download .set button — the missing "click to see params
  + download" path the user reported
- has_set detected per-run by globbing run_dir/*.set; AI insight tag shown
  when ai_insight.json exists on disk
2026-04-25 12:20:30 +00:00
LEGSTECH Optimizer 746ab8fb11 fix: hands-on bugs found during pre-submission audit
API + data integrity
- /api/settings GET no longer leaks the active Anthropic API key — returns a
  masked preview (sk-ant-XX…YYYY) plus a boolean `anthropic_api_key_set` flag
- /api/settings POST won't overwrite a real key with the masked placeholder
  the client receives back on GET (length<30 / "…" / "..." / "***" markers
  trigger a preserve-existing path)
- /api/best_result, /api/status, _make_run_dict, _result_to_dict, all AI-loop
  emits, validation_run_complete, optimization_complete, ai_iteration_complete,
  ai_targets_met, run_complete: max_drawdown and win_rate are now consistently
  emitted as PERCENTAGES (0–100), matching the dashboard's existing display
  formatters. They were previously emitted as fractions (0.13 = 13%) so the UI
  rendered "0.13%" instead of "13%"
- ResultRanker.make_result() now sets `passing` and `raw_score` on every result
  it produces. Previously these were only set during a full ranker.rank() pass,
  so individual Phase 2 / Phase 3 runs hit _make_run_dict with passing=False
  even when they cleared all gates (history showed "0 passing" when 22/22
  actually passed)
2026-04-25 11:55:22 +00:00
LEGSTECH Optimizer 6caafdb794 feat: open-source release — AI-driven autonomous optimization with live visibility
Major upgrade making the AI loop visible and the project ready for public release.

UI / UX
- Live AI Thinking Feed: streams reasoning, decisions, and outcomes per iteration
- Parameter Changes panel: prev → new + reason for every AI-driven edit
- Validation Activity panel: out-of-sample + sensitivity runs with live metrics
- Early Termination banner: surfaces why optimization stopped (targets met, no profit, budget, stuck, user stop)
- 3-phase tracker renamed Exploration / Iteration / Validation with live N/total
- Best Result modal exposes Evolution Path showing how the AI arrived at the winner
- Run-detail modal accessible from every recent run row
- Setup form validation (dates, walk-forward order, params selection, AI targets)
- Pause button removed; misleading sidebar nav consolidated to Dashboard / New Run / Reports / Source

Backend
- AIGuidedLoop streams ai_thinking, param_changes, ai_targets_met, ai_stuck
- Pipeline emits validation_start / validation_run_start / validation_run_complete / validation_done
- Pipeline emits early_termination on every early-stop path
- /api/best_result returns best run + full evolution chain
- /api/run/<id> + /api/runs sorted by ts
- AIReasoner falls back to ANTHROPIC_API_KEY env var when config is a placeholder
- Demo mode (APEX_DEMO_MODE=1) generates deterministic synthetic backtests so judges can run end-to-end without MT5

Open-source readiness
- README.md with pitch, demo flow, architecture diagram, quickstart, event reference
- LICENSE (MIT)
- config.example.yaml template (config.yaml now git-ignored)
- requirements.txt: added anthropic / requests / psutil / beautifulsoup4, capped majors
- .gitignore: secrets, *.set, scratch screenshots, ea_registry.yaml
- demo/run_demo.py: one-command offline demo runner
- 10 polished screenshots for README + judge review
2026-04-25 11:39:17 +00:00
LEGSTECH Optimizer e5dd9550b7 fix: resolve Pydantic Literal error preventing pipeline execution
The Run model used a strict Literal['baseline', 'explore', 'validate', 'wfv', 'oos']
for the phase field, but the new pipeline emits phases like 'phase1', 'phase2', etc.
This resulted in a silent validation failure inside the pipeline loop before MT5
was launched, causing the optimizer to appear 'stuck' without ever opening MT5.

Fixed by relaxing the phase field to 'str'.
2026-04-13 21:31:06 +00:00
LEGSTECH Optimizer 47f6012eb2 feat: Smart Autonomous 3-Phase Optimizer — complete redesign
PROBLEM: Old system repeated identical params, all scores flat at 0.2500,
user had zero control over symbol/TF/dates. Not smart, not dynamic.

NEW ARCHITECTURE:
  optimizer/                        (NEW package)
  ├── __init__.py
  ├── session_config.py             User choices (EA, symbol, TF, dates, budget, objective)
  ├── lhs_sampler.py                Latin Hypercube Sampling — diverse exploration
  ├── result_ranker.py              Relative scoring (best in session=1.0, worst=0.0)
  ├── budget.py                     Time budget tracker
  └── pipeline.py                   3-phase orchestrator

  Phase 1 — Broad Discovery (LHS, 20-26 runs):
    Samples FULL parameter space, not just defaults±tiny step
    LHS guarantees coverage: all 17 optimizable LEGSTECH params explored
    Relative ranking: profitable configs float top, losers score 0

  Phase 2 — Refinement (9 runs):
    Neighbor search around top 3 configs at ±20% range (not ±0.5 step)
    Keeps best of Phase1 vs Phase2 — never regresses

  Phase 3 — Validation (5 runs):
    OOS backtest on unseen data period
    Sensitivity test: nudge params ±20%, detect fragility
    Verdict: RECOMMENDED / RISKY / NOT_RELIABLE

  Output: Clean downloadable .set file via /download_set/<run_id>

UI REDESIGN:
  ui/templates/landing.html         New / homepage (was old dashboard)
  ui/templates/setup.html           New /setup — EA, symbol, TF, dates, budget, objective
  ui/templates/dashboard.html       Updated /dashboard with:
    - 5-step phase indicator
    - Real progress bar per run
    - Phase 1 results table (top 5 after phase1)
    - Verdict banner with download button
    - No-profitable-config warning
  ui/static/js/dashboard.js         Handles 8 new pipeline SocketIO events

  app.py                            New routes: /, /setup, /dashboard
                                    /api/start accepts full SessionConfig JSON
                                    /download_set/<id> serves optimized .set

  ea/registry.py                    +list_all() for setup page dropdown
  ui/templates/reports_index.html   Back to Dashboard → /dashboard (was /)
  ui/static/css/style.css           +dot-warn, dot-done, profit-pos/neg, aliases

FIXES:
  Score no longer flat 0.2500 (was: absolute thresholds on losing EA)
  User now controls: symbol, timeframe, dates, budget, objective
  Parameters now span full range (was: tiny step from defaults)
  Verdict is actionable: RECOMMENDED / RISKY / NOT_RELIABLE with reason

TESTED:
  8/8 pre-flight checks pass
  Browser test: landing ✓, setup form ✓, /dashboard ✓,
                phase indicator active ✓, /reports ✓, back link ✓
2026-04-13 21:14:47 +00:00
LEGSTECH Optimizer c4b5fe0ad0 fix: restore ea: bridge in config + wire optimizer_loop to EARegistry
Root cause: Removing ea: from config.yaml broke 4 call sites in
optimizer_loop.py (_execute_run, _build_components) and main.py that
still read cfg['ea']. Server started but MT5 never launched.

Fixes:
  config.yaml          - Restored ea: block as bridge for legacy callers
  optimizer_loop.py    - _build_components: loads EARegistry, creates
                         IniBuilder(schema=schema) instead of manifest path
                         _execute_run: uses self._profile.{name,symbol,tf}
                         instead of cfg['ea']; passes profile to runner.run()
  mt5/runner.py        - Validation + TradeLog now use EAProfile data when
                         profile is provided (legacy fallback preserved)

Tested:
  7/7 pre-flight checks pass (config, registry, schema, IniBuilder, runner,
  MutationEngine, OptimizerLoop._build_components)
  Live browser test: Start clicked → MT5 launched → logs appeared → Reports
  page loaded (9 cards) → Back to dashboard navigated in-tab
  System fully operational
2026-04-13 20:08:39 +00:00
LEGSTECH Optimizer d5a8973c05 feat(step1): Universal EA Schema Layer — replaces param_manifest.yaml
NEW FILES:
  ea/__init__.py          - EA module package
  ea/schema.py            - ParameterDef + ParameterSchema (replaces YAML manifest)
  ea/set_parser.py        - Parse ANY MT5 .set file → ParameterSchema
  ea/registry.py          - EAProfile + EARegistry (ea_registry.yaml)
  ea_registry.yaml        - LEGSTECH_EA_V2 pre-registered with automation overrides

MODIFIED:
  mt5/ini_builder.py      - Accepts ParameterSchema OR legacy manifest_path
  mt5/runner.py           - Accepts optional EAProfile for EA-agnostic validation
  config.yaml             - Removed hardcoded ea: block; added ea_registry path

KEY DESIGN:
  - .set file IS the manifest: value|min|max|step auto-detects float/int/bool/enum/fixed
  - automation_overrides in EAProfile forces headless-safe values (InpShowPanel=0)
  - Phase A optimization INI: Optimization=2 (genetic), ranges from schema
  - Phase B single backtest: Optimization=0 (unchanged behavior)
  - LEGSTECH advanced mode: fully backwards compatible via legacy manifest fallback
  - runner.py still works for legacy callers (no EAProfile needed)

TESTED:
  - 49/49 LEGSTECH params parsed correctly
  - InpShowPanel=0 automation override applied
  - INI output correct for both Phase A (Optimization=2) and Phase B (Optimization=0)
  - All imports pass
2026-04-13 19:55:16 +00:00
LEGSTECH Optimizer b70a6760ae fix: 6 workflow flaws detected via full system test
1. CRITICAL FIX - Reports 500 Internal Server Error:
   Root cause: {{}} in onclick was Jinja2 template expression
   Fix: Changed to single {} in reports_index.html onclick

2. Score history chart always empty:
   Root cause: score_update only emitted inside wfv.passed block
   Fix: Emit score_update after every hypothesis run with {promoted:false}
   Fix: Baseline run also pushes to chart via run_complete handler

3. IS gate too strict - optimizer produces 0 candidates forever:
   Root cause: min_calmar=0.35 but best EA has calmar=0.165
   Fix: Two-tier IS check - passes if composite_score improves vs baseline
   (absolute thresholds still apply as alternative pass condition)

4. Findings emitted twice (double UI display):
   Root cause: baseline analyzed once after run, then re-analyzed in iter 1
   Fix: Cache baseline findings, reuse in iteration 1 without re-emitting

5. Chart labels improved:
   'Baseline' for first point, 'It1·h1' for hypothesis runs
   Calmar normalized -0.5..2.0 -> 0..1 for chart display

6. Best score header only updates on actual promotion (not per-hypothesis)
2026-04-13 03:45:14 +00:00
LEGSTECH Optimizer cf8613eb49 fix: reports 500, NaN scores, hypothesis dedup blocking
Reports 500 (Internal Server Error):
- app.py: Replace NaN/Infinity->null before json.loads() in both
  /reports and /api/runs routes. Old summary.json files with invalid
  NaN are now handled transparently.
- reports/writer.py: Added _safe_val() helper using math.isfinite().
  All summary.json values now use safe rounding (NaN->0, Inf->0).

No new hypotheses (optimizer stops after 1 iter):
- optimizer_loop.py: Changed dedup from DB-wide history to
  session-scoped (self.session_tested_deltas). Previous runs from
  old sessions no longer block new hypothesis proposals.
- Each tested hypothesis is tracked within the session only.

Result: optimizer now runs multiple iterations, scores are valid
numbers, reports page loads without 500.
2026-04-13 03:29:33 +00:00
LEGSTECH Optimizer 26e5f9d0c4 fix: 4 critical optimizer bugs
- NaN composite score: dropna().mean() on empty series returns NaN,
  nan is truthy so (nan or 0)=nan. Fixed: use None when no MFE data
  so scorer uses 0.5 fallback instead of propagating NaN.

- Analysis stopping: gate WFV was importing from main.execute_run
  (old CLI code that uses broken runner). Fixed: gate.run_walk_forward
  now accepts an executor callable from optimizer_loop.

- Back to Dashboard opened blank tab: Reports button used
  window.open(_blank). Fixed: same-tab navigation + history.back().

- NaN in reports card: downstream of composite_score NaN above.
2026-04-13 03:05:40 +00:00
LEGSTECH Optimizer 2546b4a9c0 docs: update PROJECT_HANDOFF.md with Phase 2 complete status
- Added GitHub URL
- Documented all 6 critical bugs found and fixed:
  * UTF-16 LE report encoding
  * Report location (appdata root not /reports/)
  * INI Period string format (H1 not 16385)
  * INI Report relative path
  * MQL5 reference syntax
  * Stale report detection
- Added full SocketIO events table
- Added baseline run results (597 trades confirmed parsed)
- Updated file structure with all Phase 2 files
- Updated next steps to Phase 3 (PyInstaller packaging)
2026-04-13 02:40:13 +00:00
LEGSTECH Optimizer 7a3e13a734 Initial commit: MT5 EA Optimizer v1.0
Full optimization system for LEGSTECH_EA_V2:
- Flask + SocketIO live dashboard (dark premium UI)
- MT5 process control (auto-kill, clean launch, retry)
- HTML report parser (UTF-16 LE, 597 trades, metrics)
- Pre-run validation and actionable error messages
- Analysis engines: Reversal, TimePerfomance, EntryExit, EquityCurve
- Composite scoring (Calmar-primary)
- Mutation engine with knowledge_base.yaml
- Validation gate: IS + Walk-Forward
- Reports folder with HTML/CSV per run
- Double-click launcher batch file
2026-04-13 02:28:09 +00:00