# version: v27 # v27 — Registered the /api/chart-templates router (chart_templates_routes.py). # Companion to /api/chart-presets — named full-chart-config snapshots # the operator saves/loads via the TPL ▾ button. (2026-05-11) # v26 — /api/bands attaches `_last_modified` (unix epoch float) sourced from # the resolved JSON file's mtime (PRO local OR sync cache). Frontend # uses it to detect when bands are older than the most recent market # close and flag the BANDS toolbar button red. Best-effort: if the # cache file path can't be resolved (legacy fallback or unknown sync # layout), the field is omitted and the frontend treats the bands as # fresh (no red flag, no false alarms). """ ================================================================================ Quantum Terminal — Data Server (Consumer Build) ================================================================================ FastAPI backend for the consumer terminal. Display-only — no computation. v25 — Added env-gated install of debug_telemetry harness during lifespan startup. No effect when MK_DEBUG_TELEMETRY != "1". Disposable — see docs/specs/2026-05-05-debug-telemetry-harness-design.md. v19 — Non-blocking initial sync. Previous lifespan ran consumer_gate() end- to-end before the FastAPI HTTP server became responsive, so every retry/slow endpoint in sync_all() added to the user-visible loading screen (up to 15 min in the field). Now: 1. consumer_gate(skip_sync=True) runs synchronously — auth only, fast. Lifespan returns, HTTP routes go live immediately. 2. A daemon thread runs sync_all() in the background; when it finishes, `app.state.consumer_gate["sync_summary"]` is populated. Frontend polls /api/sync/status for the SYNCING pill (already wired) and shows the terminal UI the whole time. Net: users always see the terminal within seconds, regardless of server-side data-endpoint health. Per CLAUDE.md Rule C2 relaxed. v18 — Force OPTIONS_AVAILABLE = False unconditionally. The PRO `options_routes.create_options_router` serves `/api/options/{ticker}/full` directly from `compute_all_multi(snap)` and never applies the ETF→CFD conversion — so if its `options_data_manager` + `options_engine` dependencies happen to be bundled into the consumer's PyInstaller payload (older builds shipped them), the router takes over the route and every GEX level, wall, max pain etc. renders in raw ETF price space. Bug symptom reported in the field: XAUUSD candles stream fine (MT5 connected) but options levels are in ETF numbers (GLD ~200 vs XAUUSD ~4700). Consumer is display-only (Rule C1) — the PRO options engine should never run here, period. We keep the import attempt (so upstream module resolution doesn't change) but set OPTIONS_AVAILABLE to False after, guaranteeing the sync-cache fallback route (which applies the CFD conversion) always handles /api/options/*. v2 — Data Center transition: /api/bands/{ticker} now reads standalone {TICKER}_bands.json from the sync cache (Data Center schema), with a legacy fallback to the old nested bands key inside {TICKER}_cones.json. /api/cones was already shape-flexible (wrapped or flat) — no change. v3 — Cache directory watcher: new background task started in consumer lifespan that polls %APPDATA%\\QuantumTerminal\\cache\\ and broadcasts data_sync_complete when any {TICKER}_{category}.json changes. Enables live updates without browser refresh for manual drops and future auto-uploaders. Feature-flagged via consumer_config.ini [sync] cache_watcher_enabled (default: on). v4 — New endpoint /api/historical_cones/{ticker} for the REPLAY sub-tab of quant analysis. Serves per-anchor historical cone payloads from the sync cache (weekly + monthly anchors, each with gbm/mjd/bates models). v9 — Focused-symbol fast tick loop. Per-symbol MT5 polling is the bottleneck (≈5ms × universe_size per iteration) so dropping the global tick interval hits a wall ≈ 5-7 fps at universe=20. New focus_tick_loop polls ONLY the symbol the user is watching every FOCUS_TICK_INTERVAL (0.05s = 20 fps), while the existing tick_loop keeps the rest of the universe fresh at the normal 1s rate. Frontend sets focus via POST /api/focus/{ticker}. v8 — Tick loop reads providers.accounts.mt5_default.use_tick_data on every iteration. When True, polls MT5 every 0.2s instead of 1.0s, giving the chart MT5-style live tick movement. Toggle surfaces in Settings → PROVIDERS. Default OFF; switching it via PATCH /api/mt5/config takes effect on the next loop tick — no restart required. v7 — /api/quarterly_cones/{ticker} now also resolves the Data Center's GLOBAL__quarterly_cones.json naming via the sync client + a direct second-path fallback. Schema upgraded to quarterly_cones_v2 (4 anchors × 2 models). Frontend maps the 8 keys into the QUARTERLY row of the CONES dropdown with per-anchor sub-toggles. v6 — New endpoint /api/quarterly_cones/{ticker} — reads {TICKER}_quarterly_cones.json directly from the consumer cache. Temporary wiring: the frontend maps gbm_quarter_curr / mjd_quarter_curr / bates_quarter_curr into the existing "MANUAL" cone slots so the user can see the quarterly forecast by toggling MANUAL in the CONES dropdown. v5 — New endpoint /api/scalp_bands/{ticker} for the MICRO toggle in the bands dropdown. Serves the scalp bands JSON (per-horizon point values anchored at a recent timestamp) from the sync cache. Responsibilities: 1. MT5 Price Feed — poll ticks and bars, detect new closed bars 2. WebSocket Broadcasting — push prices and state-change events to clients 3. REST API — serve pre-computed data (cones, bands, anchors, probability field) 4. File Watching — monitor JSON state files for changes, notify clients 5. Consumer Auth — JWT login/register via license_client proxy 6. Data Sync — download pre-computed data from server via data_sync_client REMOVED from PRO (Rule C1 — no calculation triggers): - Signal engine evaluation (signal_engine_v2, signal_live_manager) - OU analysis (ou_mean_reversion) - Flow confirmation (flow_confirmation) - Auto executor (auto_executor) - Orchestrator endpoints (/api/orchestrator/*) - Calculate endpoints (/api/calculate/*) - Signal evaluate endpoints (/api/signals/evaluate/*) - Anchor calibration endpoints (/api/anchor/calibrate) - Manual cone computation (/api/cones/manual/*) - Dev endpoints (/api/dev/*) - Trade reconciliation loop - Options archive scheduler - Broker DPP cache population Usage: python data_server.py --port 8502 This file does NOT introduce any calculation triggers. ================================================================================ """ import json import asyncio import logging import time import argparse import os from pathlib import Path from datetime import datetime, timezone from typing import Dict, Set, Optional, Any from contextlib import asynccontextmanager # ── Project root (portable — Rule 3) ── PROJECT_ROOT = Path(__file__).resolve().parent import sys if str(PROJECT_ROOT) not in sys.path: sys.path.insert(0, str(PROJECT_ROOT)) from server_config import ServerConfig, PROJECT_ROOT from config_manager import ConfigManager from config_routes import create_config_router from execution_routes import create_execution_router from regime_routes import create_regime_router from system_routes import create_system_router from account_routes import create_account_router # v18: keep the import attempt (some downstream module trees expect the # symbol to resolve) but force OPTIONS_AVAILABLE = False so the PRO # router can never register over the sync-cache fallback. The fallback # at `/api/options/{ticker}/full` is the ONLY path that applies the # ETF→CFD conversion for strikes/walls/levels/GEX. try: from options_routes import create_options_router # noqa: F401 except ImportError: create_options_router = None # noqa: F811 OPTIONS_AVAILABLE = False from freshness_routes import create_freshness_router from trade_mapper_routes import create_trade_mapper_router from pulse_room.routes import create_pulse_router from chart_presets_routes import create_chart_presets_router from chart_templates_routes import create_chart_templates_router # ── Macro rotation data (yfinance sector + country ETFs) ── try: from macro_data import get_sector_data, get_country_data MACRO_DATA_AVAILABLE = True except ImportError: MACRO_DATA_AVAILABLE = False # ── Consumer terminal wiring (Project B) ── try: from consumer_startup import consumer_gate, wire_consumer_routes CONSUMER_MODE = True except ImportError: CONSUMER_MODE = False # ── Third-party imports ── from fastapi import FastAPI, WebSocket, WebSocketDisconnect, HTTPException, Request from fastapi.middleware.cors import CORSMiddleware import uvicorn # ── MT5 availability flag (for provider type detection, no direct calls) ── try: import MetaTrader5 as _mt5_check MT5_AVAILABLE = True del _mt5_check except ImportError: MT5_AVAILABLE = False # ── File watcher ── try: from watchfiles import awatch, Change WATCHFILES_AVAILABLE = True except ImportError: WATCHFILES_AVAILABLE = False # ── Signal live manager — REMOVED in consumer (Rule C1) ── SIGNAL_MANAGER_AVAILABLE = False lifecycle_mgr = None signal_evaluator = None # ── Auto executor — REMOVED in consumer (Rule C1) ── AUTO_EXECUTOR_AVAILABLE = False log = logging.getLogger("data_server") # ── Session file logging ── try: from session_logger import setup_session_logging, create_logs_router _session_log_path = setup_session_logging() log.info(f"Session log: {_session_log_path}") except ImportError: log.info("session_logger not available — console only") create_logs_router = None logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(name)s] %(levelname)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S", ) # ============================================================ # 3. WEBSOCKET CONNECTION MANAGER # ============================================================ class ConnectionManager: """Manages WebSocket connections across channels.""" def __init__(self): self._price_clients: Set[WebSocket] = set() self._event_clients: Set[WebSocket] = set() async def connect_prices(self, ws: WebSocket): await ws.accept() self._price_clients.add(ws) log.info(f"Price client connected ({len(self._price_clients)} total)") async def connect_events(self, ws: WebSocket): await ws.accept() self._event_clients.add(ws) log.info(f"Event client connected ({len(self._event_clients)} total)") def disconnect_prices(self, ws: WebSocket): self._price_clients.discard(ws) log.info(f"Price client disconnected ({len(self._price_clients)} remaining)") def disconnect_events(self, ws: WebSocket): self._event_clients.discard(ws) log.info(f"Event client disconnected ({len(self._event_clients)} remaining)") async def broadcast_prices(self, message: dict): """Send to all price channel subscribers.""" if not self._price_clients: return data = json.dumps(message) dead = [] for ws in self._price_clients: try: await ws.send_text(data) except Exception: dead.append(ws) for ws in dead: self._price_clients.discard(ws) async def broadcast_event(self, message: dict): """Send to all event channel subscribers.""" if not self._event_clients: return data = json.dumps(message) dead = [] for ws in self._event_clients: try: await ws.send_text(data) except Exception: dead.append(ws) for ws in dead: self._event_clients.discard(ws) @property def price_client_count(self) -> int: return len(self._price_clients) @property def event_client_count(self) -> int: return len(self._event_clients) # ============================================================ # 4. FILE STATE READER # ============================================================ def read_json_safe(path: Path) -> Optional[dict]: """Read a JSON file, returning None on any error.""" try: if not path.exists(): return None with open(path, "r") as f: return json.load(f) except (json.JSONDecodeError, IOError, OSError) as e: log.warning(f"Failed to read {path.name}: {e}") return None # ============================================================ # 5. BACKGROUND TASKS # ============================================================ # ── Tick price cache (populated by tick_loop, read by signal lifecycle) ── _latest_ticks: Dict[str, float] = {} FAST_TICK_INTERVAL = 1.0 # v9: full-universe loop stays at config rate when # USE TICK DATA is on (fast loop now handles the # focused chart at FOCUS_TICK_INTERVAL). FOCUS_TICK_INTERVAL = 0.1 # v9/v10: focused-symbol fast loop — 10 fps. # Was 0.05 (20 fps) but that saturated React's # main thread with re-renders, causing chart drag # to freeze. 10 fps still feels live and leaves # event-loop budget for mouse input. True MT5-style # fluidity needs Option C (bypass React state on # the price hot path) — defer until requested. # v9: focused symbol the user is actively watching. None = no focus → fast # loop idles. Set via POST /api/focus/{ticker}. _focused_symbol: Optional[str] = None def set_focused_symbol(ticker: Optional[str]): global _focused_symbol _focused_symbol = ticker.upper() if ticker else None log.info(f"Focused symbol = {_focused_symbol or '(none)'}") def get_focused_symbol() -> Optional[str]: return _focused_symbol def _resolve_tick_interval(cfg_manager: ConfigManager, base: float) -> float: """v9: Main full-universe tick loop always uses base config.tick_interval. The fast path is now focus_tick_loop, which polls only the focused symbol at FOCUS_TICK_INTERVAL. (Kept for backward-compat / future hooks.)""" return base async def tick_loop(app_state, manager: ConnectionManager, config: ServerConfig, cfg_manager: ConfigManager): """ Polls MT5 for latest ticks. Default interval = config.tick_interval (1.0s). v8: When user enables 'USE TICK DATA' in Settings → PROVIDERS, switches to FAST_TICK_INTERVAL (0.2s) on the next iteration — no restart needed. Broadcasts to all /ws/prices subscribers. """ log.info(f"Tick loop started (base interval={config.tick_interval}s, " f"fast interval={FAST_TICK_INTERVAL}s when use_tick_data=True)") last_mode = None while True: try: provider = app_state.provider if provider and provider.connected and manager.price_client_count > 0: universe = cfg_manager.get_active_universe() ticks = await asyncio.to_thread(provider.get_latest_ticks, universe) if ticks: # Update tick cache for signal lifecycle price checks for t, tick in ticks.items(): _latest_ticks[t] = tick.last # Send all ticks as one batch — avoids React render-frame drops await manager.broadcast_prices({ "type": "ticks_batch", "ticks": {t: tick.to_dict() for t, tick in ticks.items()}, }) elif provider and provider.connected: # Still update cache even if no WS clients (signals need prices) universe = cfg_manager.get_active_universe() ticks = await asyncio.to_thread(provider.get_latest_ticks, universe) if ticks: for t, tick in ticks.items(): _latest_ticks[t] = tick.last except Exception as e: log.error(f"Tick loop error: {e}") # v8: dynamic sleep — re-read each iteration so the user toggle is live. interval = _resolve_tick_interval(cfg_manager, config.tick_interval) if interval != last_mode: log.info(f"Tick interval = {interval}s " f"({'FAST / use_tick_data ON' if interval == FAST_TICK_INTERVAL else 'NORMAL'})") last_mode = interval await asyncio.sleep(interval) async def focus_tick_loop(app_state, manager: ConnectionManager, cfg_manager: ConfigManager): """v9: Fast loop dedicated to the symbol the user is actively watching. Polls ONLY the focused symbol every FOCUS_TICK_INTERVAL when USE TICK DATA is on. This avoids the universe-size ceiling that caps the main tick loop. Sleeps 1s when: - USE TICK DATA is off - No focused symbol set - No WS price clients connected - MT5 disconnected """ log.info(f"Focus tick loop started (interval={FOCUS_TICK_INTERVAL}s when active)") last_active = None while True: try: mt5_cfg = (cfg_manager.get_config().get("providers", {}) .get("accounts", {}).get("mt5_default", {}) or {}) use_tick = bool(mt5_cfg.get("use_tick_data", False)) sym = get_focused_symbol() provider = app_state.provider active = ( use_tick and sym is not None and provider is not None and provider.connected and manager.price_client_count > 0 ) if active != last_active: log.info( f"Focus tick loop: {'ACTIVE' if active else 'IDLE'} " f"(use_tick={use_tick}, sym={sym}, " f"connected={provider.connected if provider else False}, " f"clients={manager.price_client_count})" ) last_active = active if active: ticks = await asyncio.to_thread(provider.get_latest_ticks, [sym]) if ticks: for t, tick in ticks.items(): _latest_ticks[t] = tick.last await manager.broadcast_prices({ "type": "ticks_batch", "ticks": {t: tick.to_dict() for t, tick in ticks.items()}, }) await asyncio.sleep(FOCUS_TICK_INTERVAL) else: await asyncio.sleep(1.0) except Exception as e: log.error(f"Focus tick loop error: {e}") await asyncio.sleep(1.0) async def bar_check_loop( app_state, manager: ConnectionManager, config: ServerConfig, cfg_manager: ConfigManager ): """ Checks for newly closed M15 bars at config.bar_check_interval. Broadcasts new bars to /ws/prices subscribers. """ log.info(f"Bar check loop started (interval={config.bar_check_interval}s)") while True: try: provider = app_state.provider if provider and provider.connected: universe = cfg_manager.get_active_universe() new_bars = await asyncio.to_thread( provider.check_new_bars, universe, config.default_timeframe ) for bar_msg in new_bars: await manager.broadcast_prices(bar_msg) log.debug( f"New {bar_msg['timeframe']} bar: " f"{bar_msg['ticker']} @ {bar_msg['bar']['time']}" ) except Exception as e: log.error(f"Bar check error: {e}") await asyncio.sleep(config.bar_check_interval) async def mt5_reconnect_loop(app_state, config: ServerConfig): """ Monitors MT5 connection. Reconnects with backoff if dropped. """ while True: await asyncio.sleep(10) provider = app_state.provider if provider and not provider.connected: log.info("MT5 disconnected — attempting reconnect...") for attempt in range(config.mt5_max_reconnect_attempts): success = await asyncio.to_thread(provider.reconnect) if success: log.info("MT5 reconnected successfully") break delay = min(config.mt5_reconnect_delay * (2 ** attempt), 60) log.warning( f"MT5 reconnect attempt {attempt + 1} failed, " f"retrying in {delay:.0f}s" ) await asyncio.sleep(delay) elif provider: # Heartbeat check — ask the provider to verify its connection try: alive = await asyncio.to_thread(provider.heartbeat) if not alive: log.warning("Provider heartbeat failed — will reconnect next cycle") except Exception: log.warning("Provider heartbeat exception — will reconnect next cycle") async def file_watch_loop(manager: ConnectionManager, config: ServerConfig): """ Watches state JSON files for changes and pushes events to /ws/events. Falls back to polling if watchfiles is unavailable. """ watched_paths = config.watched_files path_to_source = { config.resolve_path(config.terminal_payload): "terminal_payload", config.resolve_path(config.daily_signals): "daily_signals", config.resolve_path(config.signal_lifecycle): "signal_lifecycle", config.resolve_path(config.weekly_state): "weekly_state", } if WATCHFILES_AVAILABLE: log.info(f"File watcher started (watchfiles, {len(watched_paths)} paths)") # Watch the project root directory, filter for our files watch_dir = PROJECT_ROOT async for changes in awatch(watch_dir, debounce=1000): for change_type, changed_path in changes: changed = Path(changed_path) source = path_to_source.get(changed) if source is None: continue log.info(f"File change detected: {changed.name} ({change_type.name})") now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S") if source == "terminal_payload": # Read which tickers were updated data = read_json_safe(changed) tickers = list(data.get("assets", {}).keys()) if data else [] await manager.broadcast_event({ "type": "payload_update", "source": source, "timestamp": now, "tickers_updated": tickers, }) elif source in ("daily_signals", "signal_lifecycle"): data = read_json_safe(changed) summary = {} if data and isinstance(data, dict): # Count signals per state if available all_signals = [] for ticker_signals in data.values(): if isinstance(ticker_signals, list): all_signals.extend(ticker_signals) summary = { "total_signals": len(all_signals), "tickers_affected": list(data.keys()), } await manager.broadcast_event({ "type": "state_update", "source": source, "timestamp": now, "summary": summary, }) else: await manager.broadcast_event({ "type": "state_update", "source": source, "timestamp": now, "summary": {}, }) else: # Fallback: polling-based file watcher log.info("File watcher started (polling mode — install watchfiles for efficiency)") mtimes: Dict[str, float] = {} while True: for path in watched_paths: if not path.exists(): continue mtime = path.stat().st_mtime key = str(path) if key in mtimes and mtime > mtimes[key]: source = path_to_source.get(path, path.stem) log.info(f"File change detected (poll): {path.name}") now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S") await manager.broadcast_event({ "type": "state_update", "source": source, "timestamp": now, "summary": {}, }) mtimes[key] = mtime await asyncio.sleep(config.file_watch_debounce) # ============================================================ # 6. FASTAPI APPLICATION # ============================================================ server_config = ServerConfig() cfg_manager = ConfigManager() manager = ConnectionManager() start_time = time.time() def _get_current_prices() -> dict: """Return latest known prices for all tickers from tick cache.""" return dict(_latest_ticks) @asynccontextmanager async def lifespan(app: FastAPI): """Startup and shutdown logic.""" # ── Startup ── log.info("=" * 60) log.info(" Quantum Terminal Data Server — Starting") # ── Consumer startup gate (Rule C2, relaxed — v19) ── # Run auth synchronously (need the result for login routing); spawn the # slow sync_all() in a daemon thread so this lifespan returns in <1s and # the HTTP server starts serving API routes immediately. if CONSUMER_MODE: gate_result = consumer_gate(skip_sync=True) app.state.consumer_gate = gate_result log.info( f"Consumer gate (auth phase): ok={gate_result['ok']}, " f"login={gate_result.get('needs_login')}, " f"offline={gate_result.get('offline_mode')}" ) # Spawn background sync only when auth succeeded (no login needed). # If login is required, sync waits — user logs in, periodic_sync.py # or a post-login hook picks it up. if gate_result.get("ok") and not gate_result.get("needs_login"): import threading def _run_initial_sync(): try: from data_sync_client import get_sync_client sc = get_sync_client() log.info("[initial-sync] thread started") summary = sc.sync_all() # Persist summary onto app.state so /api/sync/status can surface it. try: app.state.consumer_gate["sync_summary"] = summary if summary.get("offline_mode"): app.state.consumer_gate["offline_mode"] = True # Rule C6 — format version mismatch surfaces after sync too. if not summary.get("format_version_ok", True): app.state.consumer_gate["needs_update"] = True except Exception: pass log.info( f"[initial-sync] complete: " f"synced={summary.get('synced', 0)} " f"skipped={summary.get('skipped', 0)} " f"offline={summary.get('offline_mode', False)}" ) except Exception as e: log.error(f"[initial-sync] failed: {e}") try: app.state.consumer_gate["sync_summary"] = {"error": str(e)} except Exception: pass threading.Thread( target=_run_initial_sync, daemon=True, name="consumer-initial-sync", ).start() log.info("Consumer gate: initial sync spawned in background") log.info("=" * 60) # Initialize providers from config (graceful — server starts even if MT5 fails) provider = None connected = False try: providers = await asyncio.to_thread(cfg_manager.init_providers) provider = cfg_manager.get_provider() # Active data provider if provider: try: connected = await asyncio.wait_for( asyncio.to_thread(provider.connect), timeout=15.0, # Don't hang forever if MT5 is stuck ) except asyncio.TimeoutError: log.warning("MT5 connection timed out (15s) — continuing without live prices") connected = False except Exception as e: log.warning(f"MT5 connection failed: {e} — continuing without live prices") connected = False if connected: # Resolve universe symbols through the provider universe = cfg_manager.get_active_universe() resolved = await asyncio.to_thread(provider.resolve_universe, universe) log.info(f"Provider ready: {len(resolved)} symbols resolved") else: log.warning("Running in DEMO MODE — provider failed to connect") else: log.warning("Running in DEMO MODE — no provider configured") except Exception as e: log.warning(f"Provider initialization failed: {e} — server will start without MT5") provider = None connected = False # Store provider reference for background tasks app.state.provider = provider app.state.cfg_manager = cfg_manager # Launch background tasks (consumer: tick loop + bar check + file watch only) tasks = [ asyncio.create_task(tick_loop(app.state, manager, server_config, cfg_manager)), asyncio.create_task(focus_tick_loop(app.state, manager, cfg_manager)), # v9 asyncio.create_task(bar_check_loop(app.state, manager, server_config, cfg_manager)), asyncio.create_task(file_watch_loop(manager, server_config)), ] if provider and provider.connected: tasks.append(asyncio.create_task(mt5_reconnect_loop(app.state, server_config))) # ── Consumer periodic sync (Rule C1: display-only refresh, 5 min default) ── # Polls the VPS at a configurable interval and broadcasts data_sync_complete # WS event when fresh files arrive. Frontend listens and refreshes panels. if CONSUMER_MODE: try: from periodic_sync import create_periodic_sync_task sync_task_factory = create_periodic_sync_task(manager.broadcast_event) tasks.append(asyncio.create_task(sync_task_factory())) log.info("Consumer periodic sync task started") except Exception as e: log.warning(f"Failed to start periodic sync task: {e}") # ── Cache directory watcher (v3) ─────────────────────────────────── # Live updates whenever a file in the cache dir changes on disk. # Covers manual drops and future automated DC uploaders. Disable via # consumer_config.ini → [sync] cache_watcher_enabled = false. try: from cache_watcher import create_cache_watcher_task watcher_factory = create_cache_watcher_task(manager.broadcast_event) if watcher_factory is not None: tasks.append(asyncio.create_task(watcher_factory())) log.info("Cache watcher task started") except Exception as e: log.warning(f"Failed to start cache watcher task: {e}") # NOTE: Signal lifecycle, auto executor, options archive, trade reconciliation # are all PRO-only features — removed in consumer (Rule C1). log.info( f"Data server ready — " f"{len(cfg_manager.get_active_universe())} assets, " f"Provider {'connected' if (provider and provider.connected) else 'disconnected (demo mode)'}" ) log.info(f"Listening on http://{server_config.host}:{server_config.port}") # --- Disposable debug telemetry harness (env-gated) --- if os.environ.get("MK_DEBUG_TELEMETRY") == "1": import debug_telemetry debug_telemetry.install(app) yield # ── Shutdown ── log.info("Shutting down...") for t in tasks: t.cancel() # Disconnect all providers for pid, prov in cfg_manager.get_all_providers().items(): prov.disconnect() log.info("Data server stopped") app = FastAPI( title="Quantum Terminal Data Server", version="1.0.0", lifespan=lifespan, ) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=False, allow_methods=["*"], allow_headers=["*"], ) # ── Wire consumer routes (auth + sync + status) ── if CONSUMER_MODE: wire_consumer_routes(app) log.info("Consumer routes wired") # ── Wire API routers (must be at module level, not inside lifespan) ── app.include_router(create_config_router(cfg_manager, manager.broadcast_event)) app.include_router(create_execution_router(cfg_manager, app, manager.broadcast_event)) app.include_router(create_regime_router(cfg_manager, app)) app.include_router(create_system_router()) app.include_router(create_account_router(manager.broadcast_event)) app.include_router(create_freshness_router(cfg_manager)) app.include_router(create_trade_mapper_router(app)) app.include_router(create_pulse_router()) app.include_router(create_chart_presets_router()) app.include_router(create_chart_templates_router()) # ── MT5 connection management routes (consumer settings panel) ── try: from mt5_routes import create_mt5_router app.include_router(create_mt5_router(cfg_manager, app)) log.info("MT5 management routes registered (/api/mt5/*)") except ImportError: log.warning("mt5_routes not available — MT5 settings panel won't work") # ── Tradovate POC routes (consumer settings panel) ── # v13: standalone proof-of-work integration. Removing this block + the file # backend/tradovate_routes.py + providers/tradovate_provider.py fully # disables Tradovate without touching anything else. try: from tradovate_routes import create_tradovate_router app.include_router(create_tradovate_router(cfg_manager)) log.info("Tradovate POC routes registered (/api/tradovate/*)") except ImportError as _e: log.warning(f"tradovate_routes not available: {_e}") if create_logs_router is not None: app.include_router(create_logs_router()) if OPTIONS_AVAILABLE: try: app.include_router(create_options_router()) log.info("Options quant routes registered (/api/options/*)") except Exception as e: log.warning(f"Options routes failed to load: {e}") OPTIONS_AVAILABLE = False # ── Fallback options from sync cache (consumer mode) ── # v12: convert ETF strikes → live CFD prices server-side so frontend never # does ratio math. Ratio = mt5_live_price / options.spot (ETF). Frozen # per request so both MainChart's sauce lines and OptionsPanel eat the # exact same numbers. When MT5 is offline we pass the raw payload # through with _conversion_applied=false and the frontend can flag it. def _mul_options_prices_inplace(data: dict, ratio: float) -> None: """Multiply known strike/level/price fields by ratio in-place.""" def mul(v): return v * ratio if isinstance(v, (int, float)) and v is not None else v for k in ("hvl", "max_pain", "gex_flip"): if k in data and data[k] is not None: data[k] = mul(data[k]) w = data.get("walls") or {} for k in ("primary_call_wall", "primary_put_wall"): if k in w and w[k] is not None: w[k] = mul(w[k]) for arr_key in ("call_walls", "put_walls"): for item in (w.get(arr_key) or []): if isinstance(item, dict) and item.get("strike") is not None: item["strike"] = mul(item["strike"]) # v15: convert both swing `expected_move` and the new `expected_move_0dte` # block (shipped by DC 2026-04-21). Same shape — only `pct` is # dimensionless and stays untouched. for em_key in ("expected_move", "expected_move_0dte"): em = data.get(em_key) or {} for k in ("atm_strike", "move_up", "move_down", "straddle"): if k in em and em[k] is not None: em[k] = mul(em[k]) for iv_key in ("iv_divergence", "iv_skew"): iv = data.get(iv_key) or {} for k in ("atm_strike", "call_25d_strike", "put_25d_strike"): if k in iv and iv[k] is not None: iv[k] = mul(iv[k]) dh = data.get("dealer_heatmap") or {} for k in ("max_buy_strike", "max_sell_strike"): if k in dh and dh[k] is not None: dh[k] = mul(dh[k]) if isinstance(dh.get("strikes"), list): dh["strikes"] = [mul(s) for s in dh["strikes"]] for gex_key in ("gex", "gex_0dte", "gex_stack"): g = data.get(gex_key) or {} if isinstance(g.get("strikes"), list): g["strikes"] = [mul(s) for s in g["strikes"]] # v16: convert both `sauce` (swing) and the new `sauce_0dte` block # (shipped by DC 2026-04-21). Same per-level strike conversion — no # mixing. sauce_0dte is null when today ∉ chain (expected). for sauce_key in ("sauce", "sauce_0dte"): s_block = data.get(sauce_key) or {} for s_key in ("volatility_gravity", "structural_integrity"): s = s_block.get(s_key) or {} for item in (s.get("levels") or []): if isinstance(item, dict) and item.get("strike") is not None: item["strike"] = mul(item["strike"]) def _mt5_mid_for(ticker: str) -> Optional[float]: """Fetch current MT5 mid price ((bid+ask)/2) for a canonical ticker. Returns None if provider offline or tick unavailable.""" prov = app.state.provider if not prov or not getattr(prov, "connected", False): return None try: ticks = prov.get_latest_ticks([ticker]) t = ticks.get(ticker) if ticks else None if t is None: return None bid = getattr(t, "bid", None) ask = getattr(t, "ask", None) if bid and ask: return (bid + ask) / 2.0 return getattr(t, "last", None) except Exception: return None if not OPTIONS_AVAILABLE: @app.get("/api/options/{ticker}/full") async def get_options_from_sync(ticker: str, n: int = 8): """Serve options data from consumer sync cache with ETF→CFD conversion baked in (v12). All strikes / levels / walls in the returned JSON are in MT5 CFD price space — frontend consumes as-is.""" canonical = ticker.upper() try: import copy from data_sync_client import get_sync_client raw = get_sync_client().get_options(canonical) if not raw: raise HTTPException(404, f"No options data for {canonical}") # v12 fix: sync_client returns the live cached reference — mutating # it double-multiplies strikes on every subsequent request. Deep # copy so conversion only applies to the response, not the cache. data = copy.deepcopy(raw) # Normalize keys: sync cache may use spot_etf; ensure we have an ETF spot. etf_spot = data.get("spot_etf") or data.get("spot") if etf_spot is None: raise HTTPException(502, f"Options data for {canonical} missing spot price") if "canonical" not in data: data["canonical"] = canonical if "cboe_ticker" not in data: cboe_map = {"US500": "SPY", "USTEC": "QQQ", "XAUUSD": "GLD"} data["cboe_ticker"] = cboe_map.get(canonical) mt5_price = await asyncio.to_thread(_mt5_mid_for, canonical) levels_space = data.get("levels_space") if levels_space == "broker": # Producer (v2.0+) pre-multiplied levels to broker space; consumer no-op. if mt5_price: data["spot"] = mt5_price data["_conversion_applied"] = True data["_conversion_source"] = "broker_native" elif mt5_price and etf_spot > 0: ratio = mt5_price / etf_spot _mul_options_prices_inplace(data, ratio) # Always preserve the raw ETF spot; replace top-level spot with the # live MT5 price so frontend sees CFD space consistently. data["spot_etf"] = etf_spot data["spot"] = mt5_price data["_conversion_applied"] = True data["_conversion_ratio"] = ratio data["_conversion_source"] = "mt5_live" else: # No MT5 price available — fall through with raw ETF values so # the panel still renders, but flagged so the UI can warn. if "spot" not in data: data["spot"] = etf_spot data["_conversion_applied"] = False data["_conversion_source"] = "mt5_unavailable" return data except HTTPException: raise except Exception as e: log.warning(f"options fallback failed for {canonical}: {e}") raise HTTPException(404, f"No options data for {canonical}") log.info("Options fallback route registered (sync cache, v12 CFD-converted)") # ── Macro Rotation Endpoints (Money Flow Tabs 2 & 3) ── if MACRO_DATA_AVAILABLE: @app.get("/api/macro/sectors") async def api_macro_sectors(refresh: bool = False): try: from data_sync_client import get_sync_client data = get_sync_client().get_macro_sectors() if data: return data except Exception as e: log.warning(f"[MACRO] macro_sectors failed: {e}") raise HTTPException(404, "No sector data available") @app.get("/api/macro/countries") async def api_macro_countries(refresh: bool = False): try: from data_sync_client import get_sync_client data = get_sync_client().get_macro_countries() if data: return data except Exception as e: log.warning(f"[MACRO] macro_countries failed: {e}") raise HTTPException(404, "No country data available") @app.get("/api/macro/flows") async def api_macro_flows(period: str = "1W"): try: from data_sync_client import get_sync_client data = get_sync_client().get_macro_flows() if data: key = "4W" if period.upper() == "4W" else "1W" flows_obj = data.get(key) if flows_obj and "flows" in flows_obj: return flows_obj except Exception as e: log.warning(f"[MACRO] macro_flows failed: {e}") raise HTTPException(404, "No country flow data available") log.info("Macro rotation routes registered (/api/macro/sectors, /api/macro/countries, /api/macro/flows)") else: log.info("macro_data not available -- yfinance may not be installed") # -- Money Flow from sync cache (consumer mode) -- @app.get("/api/fundamental/money-flow") async def get_money_flow(period: str = "macro"): """Serve capital flow data from consumer sync cache.""" try: from data_sync_client import get_sync_client data = get_sync_client().get_global("macro_sector_flows") if data: # Data is keyed by period (1W, 4W). Extract requested period. period_map = {"macro": "macro", "weekly": "weekly", "1w": "macro", "4w": "weekly"} key = period_map.get(period.lower(), period) if key in data: return data[key] # If only one period exists, return it for k, v in data.items(): if isinstance(v, dict) and "flows" in v: return v return data except Exception: pass raise HTTPException(404, "No money flow data available") # -- Fundamental state (macro regime, liquidity, yields, COT, scores) -- @app.get("/api/fundamental_state") async def get_fundamental_state(): """Serve fundamental_state.json from consumer sync cache (Rule C1: read-only).""" try: from data_sync_client import get_sync_client data = get_sync_client().get_fundamentals() if data: return data except Exception as e: log.warning(f"[FUNDAMENTAL] fundamental_state failed: {e}") raise HTTPException(404, "No fundamental state data available") # ── REST ENDPOINTS ────────────────────────────────────────── @app.get("/api/health") async def health(): """Server health check — backwards-compatible top-level fields plus a v11+ `components` block where each subsystem reports its own status. Frontends that want detailed diagnostics read components; legacy clients that just check `status == "ok"` still work.""" uptime = time.time() - start_time provider = app.state.provider # ── Component health ── components: Dict[str, Any] = {} # MT5 / data provider mt5_connected = bool(provider and provider.connected) components["mt5"] = { "status": "ok" if mt5_connected else "disconnected", "type": provider.provider_type if provider else None, "label": provider.label if provider else None, "connected": mt5_connected, "active_symbols": len(cfg_manager.get_active_universe()), } # License / auth — Quantum Terminal: always free components[\"license\"] = { \"status\": \"ok\", \"user_email\": \"free@quantumterminal\", \"subscription_status\": \"active\", \"subscription_type\": \"free\", \"offline\": False, } # Sync status try: from data_sync_client import get_sync_client as _gsc _sc = _gsc() sync_in_progress = bool(getattr(_sc, "_sync_in_progress", False)) sync_total = int(getattr(_sc, "_sync_total", 0)) sync_done = int(getattr(_sc, "_sync_done", 0)) sync_errors = list(getattr(_sc, "_sync_errors", []) or []) last_synced = getattr(_sc, "last_synced_at", None) # Status: error if errors and not in progress, syncing if in progress, ok otherwise if sync_in_progress: sync_status = "syncing" elif sync_errors: sync_status = "degraded" else: sync_status = "ok" components["sync"] = { "status": sync_status, "in_progress": sync_in_progress, "progress": (sync_done, sync_total), "last_synced_at": last_synced, "format_version_ok": bool(getattr(_sc, "format_version_ok", True)), "errors_count": len(sync_errors), "offline": bool(getattr(_sc, "is_offline", False)), } except Exception as e: components["sync"] = {"status": "error", "error": str(e)} # WebSocket clients components["websockets"] = { "status": "ok", "price_clients": manager.price_client_count, "event_clients": manager.event_client_count, } # Build version version = None try: from pathlib import Path as _Path for vp in [_Path(__file__).parent / "VERSION", _Path(__file__).parent.parent.parent / "VERSION", _Path(__file__).parent.parent / "VERSION"]: if vp.exists(): version = vp.read_text(encoding="utf-8").strip() break except Exception: pass # Roll up overall status — error if any component errored, degraded # if any is in a non-ok-but-recoverable state, otherwise ok. overall = "ok" for c in components.values(): s = c.get("status") if s == "error": overall = "error" break if s in ("disconnected", "auth_required", "degraded", "syncing"): overall = "degraded" return { # Back-compat top-level fields (v1 and earlier readers) "status": "ok" if overall != "error" else "error", "mt5_connected": mt5_connected, "uptime_seconds": round(uptime, 1), "active_symbols": len(cfg_manager.get_active_universe()), "ws_price_clients": manager.price_client_count, "ws_event_clients": manager.event_client_count, "universe": cfg_manager.get_display_order(), "signal_engine_available": False, "active_signals": 0, "provider": { "type": provider.provider_type if provider else None, "label": provider.label if provider else None, "connected": mt5_connected, }, # v11+ structured component view "overall": overall, "version": version, "components": components, } @app.get("/api/ticks") async def get_all_ticks(): """ Latest tick for every symbol in the universe. Used by the frontend on initial load to seed prices (before the WebSocket starts streaming). """ provider = app.state.provider if not provider or not provider.connected: return {"ticks": {}, "note": "MT5 not connected"} universe = cfg_manager.get_active_universe() ticks = await asyncio.to_thread(provider.get_latest_ticks, universe) return { "ticks": {t: tick.to_dict() for t, tick in ticks.items()}, } @app.post("/api/focus/{ticker}") async def post_focus(ticker: str): """v9: Tell the backend which symbol the user is actively watching. The focus_tick_loop polls only this symbol at FOCUS_TICK_INTERVAL when USE TICK DATA is on, giving the chart MT5-style live tick movement without paying universe-size MT5 polling cost.""" set_focused_symbol(ticker) return {"focused": get_focused_symbol(), "interval": FOCUS_TICK_INTERVAL} @app.delete("/api/focus") async def delete_focus(): """v9: Clear the focused symbol (focus_tick_loop goes idle).""" set_focused_symbol(None) return {"focused": None} @app.get("/api/symbol-info/{ticker}") async def get_symbol_info(ticker: str): """ Real broker symbol metadata — tick_value, contract_size, etc. Used by OrderTicket for accurate P&L preview. """ provider = app.state.provider if not provider or not provider.connected: raise HTTPException(503, "Provider not connected") canonical = ticker.upper() info = await asyncio.to_thread(provider.get_symbol_info, canonical) if info is None: raise HTTPException(404, f"Symbol info not found for {canonical}") return info.to_dict() @app.get("/api/symbol-info") async def get_all_symbol_info(): """Batch symbol info for the entire universe.""" provider = app.state.provider if not provider or not provider.connected: return {"symbols": {}, "note": "Provider not connected"} universe = cfg_manager.get_active_universe() infos = await asyncio.to_thread(provider.get_all_symbol_info, universe) return { "symbols": {t: info.to_dict() for t, info in infos.items()}, } @app.get("/api/payload") async def get_payload(): """Full terminal_payload.json contents. Returns empty in consumer mode.""" path = server_config.resolve_path(server_config.terminal_payload) data = read_json_safe(path) if data is None: # Consumer mode: no local payload file — data comes via sync cache return {"assets": {}, "note": "Consumer mode — data served via /api/cones and /api/sync"} return data @app.get("/api/payload/{ticker}") async def get_payload_ticker(ticker: str): """Single ticker's payload entry.""" path = server_config.resolve_path(server_config.terminal_payload) data = read_json_safe(path) if data is None: return {"note": "Consumer mode — use /api/sync/cones/{ticker}"} assets = data.get("assets", {}) canonical = ticker.upper() if canonical not in assets: return {"note": f"No payload entry for {canonical}"} return assets[canonical] @app.get("/api/signals") async def get_signals(): """All signals from daily_signals.json.""" path = server_config.resolve_path(server_config.daily_signals) data = read_json_safe(path) if data is None: return {"signals": {}, "note": "No signals file found"} return data @app.get("/api/signals/{ticker}") async def get_signals_ticker(ticker: str): """Signals for one ticker.""" path = server_config.resolve_path(server_config.daily_signals) data = read_json_safe(path) if data is None: raise HTTPException(404, "daily_signals.json not found") canonical = ticker.upper() if canonical not in data: return {"ticker": canonical, "signals": []} return {"ticker": canonical, "signals": data[canonical]} # ── Stress Lab (v10) ───────────────────────────────────────────────────── # Phase 1: consume DC's stress_lab JSON files from the standard AppData # cache. Files land there either via data_sync_client (future wiring) or # manual copy for testing. No computation, no synthesis — display-only (C1). # # Expected files in %APPDATA%\QuantumTerminal\cache\ : # _stress_lab_index.json — watchlist # {TICKER}_stress_lab.json — per-asset thesis def _stress_lab_cache_dir() -> Path: """Same cache dir the sync_client writes to — flat layout.""" from data_sync_client import _get_cache_dir return _get_cache_dir() @app.get("/api/stress_lab/index") async def get_stress_lab_index(): """Watchlist — list of assets with conviction + spot + one-liner.""" path = _stress_lab_cache_dir() / "_stress_lab_index.json" data = read_json_safe(path) if data is None: raise HTTPException(404, "Stress Lab index not available yet — waiting on first sync") return data @app.get("/api/stress_lab/{ticker}") async def get_stress_lab_thesis(ticker: str): """Full per-asset thesis JSON produced by DC stress_lab_engine.""" canonical = ticker.upper() path = _stress_lab_cache_dir() / f"{canonical}_stress_lab.json" data = read_json_safe(path) if data is None: raise HTTPException(404, f"No Stress Lab thesis for {canonical}") return data # v11: Historical backtest positions for the "last filled signal" replay panel. @app.get("/api/stress_lab/backtest/positions") async def get_stress_lab_backtest_positions(): path = _stress_lab_cache_dir() / "_stress_lab_backtest_positions.json" data = read_json_safe(path) if data is None: raise HTTPException(404, "Backtest positions not available yet") return data # v11: Historical bar range — replay chart on Stress Lab last-signal panel. # `from` is a Python keyword, so we accept it via Query(alias="from"). from fastapi import Query as _Query from datetime import datetime as _dt, timezone as _tz @app.get("/api/bars_range/{ticker}") async def get_bars_range( ticker: str, timeframe: str = "M15", from_: str = _Query(..., alias="from"), to: str = _Query(...), ): """OHLCV bars between two ISO-8601 UTC datetimes. Used by the Stress Lab last-signal replay chart.""" canonical = ticker.upper() provider = app.state.provider if not provider or not provider.connected: raise HTTPException(503, "No data provider connected") try: f_dt = _dt.fromisoformat(from_.replace("Z", "+00:00")) t_dt = _dt.fromisoformat(to.replace("Z", "+00:00")) except ValueError as e: raise HTTPException(400, f"bad ISO datetime: {e}") if f_dt.tzinfo is None: f_dt = f_dt.replace(tzinfo=_tz.utc) if t_dt.tzinfo is None: t_dt = t_dt.replace(tzinfo=_tz.utc) if not hasattr(provider, "get_bars_range"): raise HTTPException(501, "provider does not support range fetch") bars = await asyncio.to_thread( provider.get_bars_range, canonical, timeframe, f_dt, t_dt ) if not bars: return {"bars": [], "ticker": canonical, "timeframe": timeframe, "from": from_, "to": to} # Normalize to the shape frontend expects (LWC candles). return { "ticker": canonical, "timeframe": timeframe, "from": from_, "to": to, "bars": [{ "time": b.time, "open": b.open, "high": b.high, "low": b.low, "close": b.close, "volume": b.volume, } for b in bars], } @app.get("/api/bands/{ticker}") async def get_bands(ticker: str): """Probability bands data for a ticker. Falls back to sync cache.""" canonical = ticker.upper() # Try PRO local file first bands_dir = server_config.resolve_path(server_config.bands_data_dir) bands_file = bands_dir / f"{canonical}_bands.json" data = read_json_safe(bands_file) if data: # v26: stamp file mtime for staleness detection on the frontend. try: data["_last_modified"] = bands_file.stat().st_mtime except Exception: pass return data # Fallback 1 (v2 — Data Center): standalone {TICKER}_bands.json in sync cache. try: from data_sync_client import get_sync_client client = get_sync_client() bands_data = client.get_bands(canonical) if bands_data: # v26: best-effort mtime from the sync cache file. Naming # patterns observed: _bands.json or GLOBAL__bands.json. try: cache_dir = getattr(client, "cache_dir", None) if cache_dir is not None: from pathlib import Path as _P cache_dir_p = _P(cache_dir) if not isinstance(cache_dir, _P) else cache_dir for name in (f"{canonical}_bands.json", f"GLOBAL_{canonical.lower()}_bands.json"): candidate = cache_dir_p / name if candidate.exists(): bands_data["_last_modified"] = candidate.stat().st_mtime break except Exception: pass return bands_data except Exception: pass # Fallback 2 (legacy): bands nested inside the cones sync blob. # Retained so any cache still holding pre-transition files keeps working. try: sync_data = client.get_cones(canonical) if sync_data and "bands" in sync_data: return sync_data["bands"] except Exception: pass raise HTTPException(404, f"No bands data for {canonical}") def _sanitize_json(data): """Replace NaN/Infinity with None — these are not valid JSON.""" raw = json.dumps(data, allow_nan=True, default=str) raw = raw.replace(": NaN", ": null").replace(":NaN", ":null") raw = raw.replace(": Infinity", ": null").replace(":Infinity", ":null") raw = raw.replace(": -Infinity", ": null").replace(":-Infinity", ":null") return json.loads(raw) @app.get("/api/state") async def get_state(): """Weekly state (regime, anchors, WF params).""" path = server_config.resolve_path(server_config.weekly_state) data = read_json_safe(path) if data is None: return {"assets": {}, "note": "No weekly state found — run orchestrator weekly tier"} return _sanitize_json(data) @app.get("/api/bars/{ticker}") async def get_bars(ticker: str, timeframe: str = "M15", count: int = 200): """ Fetch historical bars directly from MT5. Used for initial chart load and timeframe switches. """ canonical = ticker.upper() if canonical not in cfg_manager.get_active_universe(): raise HTTPException(404, f"Ticker {canonical} not in universe") provider = app.state.provider if not provider or not provider.connected: raise HTTPException(503, "No data provider connected") bars = await asyncio.to_thread(provider.get_bars, canonical, timeframe, count) if not bars: raise HTTPException(404, f"No bar data for {canonical} {timeframe}") return { "ticker": canonical, "timeframe": timeframe, "count": len(bars), "bars": [b.to_dict() for b in bars], } @app.get("/api/lifecycle") async def get_lifecycle(): """Signal lifecycle state.""" path = server_config.resolve_path(server_config.signal_lifecycle) data = read_json_safe(path) if data is None: return {"signals": {}, "note": "No lifecycle file found"} return data # REMOVED: /api/signals/evaluate, /api/orchestrator, /api/calculate (Rule C1) @app.get("/api/anchors/{ticker}") async def get_anchors(ticker: str): """Institutional anchors + extremes. Falls back to sync cache.""" canonical = ticker.upper() # Try PRO local file first anchors_path = PROJECT_ROOT / "terminal_anchors.json" data = read_json_safe(anchors_path) if data and canonical in data: return data[canonical] # Fallback: consumer sync cache — anchors are inside the cones response try: from data_sync_client import get_sync_client sync_data = get_sync_client().get_cones(canonical) if sync_data: result = {} if "anchors" in sync_data: result["anchors"] = sync_data["anchors"] if "extremes" in sync_data: result["extremes"] = sync_data["extremes"] if result: return result except Exception: pass raise HTTPException(404, f"No anchor data for {canonical}") @app.get("/api/anchors") async def get_all_anchors(): """All tickers' anchor data. Falls back to sync cache.""" anchors_path = PROJECT_ROOT / "terminal_anchors.json" data = read_json_safe(anchors_path) if data: return data # Fallback: consumer sync cache try: from data_sync_client import get_sync_client client = get_sync_client() result = {} for ticker in client.synced_tickers: sync_data = client.get_cones(ticker) if sync_data: entry = {} if "anchors" in sync_data: entry["anchors"] = sync_data["anchors"] if "extremes" in sync_data: entry["extremes"] = sync_data["extremes"] if entry: result[ticker] = entry if result: return result except Exception: pass return {"anchors": {}, "note": "No anchor data found"} @app.get("/api/regime_live/{ticker}") async def get_regime_live_stub(ticker: str, timeframe: str = "M15", bars: int = 500): """Stub — regime_live is a PRO-only endpoint (Rule C1). Returns empty payload so the frontend stops 404-retrying.""" return { "ticker": ticker.upper(), "timeframe": timeframe, "data": [], "note": "regime_live unavailable in consumer build" } @app.get("/api/cones/{ticker}") async def get_cones(ticker: str): """Probability cone data for a ticker. Falls back to sync cache in consumer mode. Also merges GEX cones from the options sync cache (gex_cones lives in options JSON, not cones JSON, on the producer side).""" canonical = ticker.upper() # Try PRO local file first cones_file = server_config.resolve_path("terminal_cones.json") data = read_json_safe(cones_file) if data and canonical in data: return data[canonical] # Fallback: consumer sync cache (Rule C5) result = None try: from data_sync_client import get_sync_client client = get_sync_client() sync_data = client.get_cones(canonical) if sync_data and "cones" in sync_data: result = dict(sync_data.get("cones", {})) elif sync_data: result = dict(sync_data) # Merge gex_cones from the options sync cache. The producer writes # gex_now / gex_weekly_1 / gex_weekly_2 into the options JSON, not # the cones JSON. Their shape (median, sd1_high, sd1_low, sd2_high, # sd2_low, sd3_high, sd3_low, dates) matches the standard cone shape # the frontend renderer expects, so a flat merge is enough. opt_data = client.get_options(canonical) if opt_data and isinstance(opt_data.get("gex_cones"), dict): if result is None: result = {} for k, v in opt_data["gex_cones"].items(): # Don't clobber any pre-existing key (cones JSON wins if both exist) if k not in result: result[k] = v except Exception: pass if result: return result raise HTTPException(404, f"No cone data for {canonical}") @app.get("/api/quarterly_cones/{ticker}") async def get_quarterly_cones(ticker: str): """Quarterly cone forecast (183-day+ horizon, 4 anchors × 2 models). v6 — Reads {TICKER}_quarterly_cones.json from the consumer cache. v7 — Also handles the Data Center's GLOBAL__quarterly_cones.json naming convention (same legacy pattern as historical_cones / scalp_bands). Routes through the sync client first (in-memory store populated by resolve_cache_filename), then falls back to direct file reads. Shape: median + sd1/2/3 high/low + dates per model, keys prefixed gbm_quarter_*, mjd_quarter_*.""" import json as _json canonical = ticker.upper() try: from data_sync_client import get_sync_client, _get_cache_dir client = get_sync_client() # (1) in-memory store (populated by sync_all + _load_all_from_cache) data = client._get_data(canonical, "quarterly_cones") if data: return data # (2) direct disk fallback — try both naming conventions explicitly, # in case the cache scan hasn't run yet for this session. cache_dir = _get_cache_dir() for path in ( cache_dir / f"{canonical}_quarterly_cones.json", cache_dir / f"GLOBAL_{canonical.lower()}_quarterly_cones.json", ): if path.is_file(): return _json.loads(path.read_text(encoding="utf-8")) except Exception as e: log.warning(f"quarterly_cones load failed for {canonical}: {e}") raise HTTPException(404, f"No quarterly cone data for {canonical}") # v14: TEMPORARY dev-only route. Reads the producer's bands-replay JSON # directly from D:\MK_DATA_CENTER\temp_out so the replay view can # render the new per-day anchor schema before the real sync pipeline # exists. Delete this function (and its import side-effects) to revert. @app.get("/api/bands_replay/{ticker}") async def get_bands_replay(ticker: str): """Bands-replay prototype — reads {TICKER}_bands_replay.json from D:\\MK_DATA_CENTER\\temp_out (dev path only). Schema matches /api/historical_cones but with a 'daily' bucket, a 'head' warm-start block, and fusion_instructions metadata. Returns 404 if the file isn't present (non-XAUUSD tickers, production boxes, etc).""" import json as _json from pathlib import Path as _Path canonical = ticker.upper() test_dir = _Path(r"D:\MK_DATA_CENTER\temp_out") path = test_dir / f"{canonical}_bands_replay.json" if not path.is_file(): raise HTTPException(404, f"No bands_replay data for {canonical}") try: return _json.loads(path.read_text(encoding="utf-8")) except Exception as e: log.warning(f"bands_replay load failed for {canonical}: {e}") raise HTTPException(500, f"bands_replay parse failed for {canonical}") @app.get("/api/scalp_bands/{ticker}") async def get_scalp_bands(ticker: str): """Scalp bands (5-min anchor + per-horizon projections) for the MICRO toggle in the BANDS dropdown. Served from the sync cache. Source file: GLOBAL__scalp_bands.json (resolver-routed).""" canonical = ticker.upper() try: from data_sync_client import get_sync_client client = get_sync_client() data = client._get_data(canonical, "scalp_bands") if data: return data except Exception: pass raise HTTPException(404, f"No scalp bands data for {canonical}") @app.get("/api/historical_cones/{ticker}") async def get_historical_cones(ticker: str): """Historical cone anchors (weekly + monthly) for the REPLAY sub-tab. Each anchor date maps to a dict with gbm / mjd / bates model payloads (median + sd1/2/3 high/low + dates). Served from the sync cache. v17: mirrors the quarterly_cones endpoint — falls back to direct file reads (both `{TICKER}_*.json` and `GLOBAL__*.json` shapes) when the in-memory store hasn't been populated yet. Without this, a cold-boot or stalled-sync session would 404 on tickers that have a file-on-disk but no store entry yet (e.g. EURUSD). """ import json as _json canonical = ticker.upper() try: from data_sync_client import get_sync_client, _get_cache_dir client = get_sync_client() # (1) in-memory store (populated by sync_all + _load_all_from_cache) data = client._get_data(canonical, "historical_cones") if data: return data # (2) direct disk fallback — handles legacy GLOBAL_ naming. cache_dir = _get_cache_dir() for path in ( cache_dir / f"{canonical}_historical_cones.json", cache_dir / f"GLOBAL_{canonical.lower()}_historical_cones.json", ): if path.is_file(): return _json.loads(path.read_text(encoding="utf-8")) except Exception as e: log.warning(f"historical_cones load failed for {canonical}: {e}") raise HTTPException(404, f"No historical cone data for {canonical}") @app.get("/api/cones") async def get_all_cones(): """All computed cone data. Falls back to sync cache.""" cones_file = server_config.resolve_path("terminal_cones.json") data = read_json_safe(cones_file) if data: return data # Fallback: consumer sync cache try: from data_sync_client import get_sync_client client = get_sync_client() result = {} for ticker in client.synced_tickers: sync_data = client.get_cones(ticker) if sync_data and "cones" in sync_data: result[ticker] = sync_data["cones"] elif sync_data: result[ticker] = sync_data if result: return result except Exception: pass return {"note": "No cones data found"} # ── PROBABILITY FIELD ENDPOINT ──────────────────────────── @app.get("/api/probability-field/{ticker}") async def get_probability_field(ticker: str): """ Probability field (v2.0) — consumer version. Serves the full pre-computed payload from sync cache. Both modes (month_curr, month_prev) are bundled in one response — the frontend picks which mode to render from the cached payload. Rule C1: no live computation. Rule C5: cache-only serving. """ canonical = ticker.upper() try: from data_sync_client import get_sync_client probfield = get_sync_client().get_probfield(canonical) except Exception as e: log.warning(f"[probfield] get_probfield failed for {canonical}: {e}") probfield = None if not probfield: raise HTTPException( 404, f"No probability field data for {canonical}. " f"Data may not have synced yet." ) # Return the full v2.0 payload unchanged. # Frontend reads probfield.modes.month_curr / month_prev directly. return probfield # ── PATH OUTCOME FOREST (PRO) ───────────────────────────── # v20: serve the per-ticker path_forest forecast from sync cache. Producer # writes GLOBAL__path_forest.json once live; meanwhile DC has shipped # 5 scenario fixtures (trend / range / stress / mixed / miscal) so the # frontend can develop against varied conditions. The `scenario` query param # picks among the fixtures; live filename takes priority when present. @app.get("/api/forest") async def list_forest_assets(): """ v21: List every ticker that has a path_forest forecast in cache. Used by the Stress Lab → Forest Path sub-tab so its asset selector auto-populates with whatever DC has shipped instead of relying on a hardcoded probe list. Returns ISO sorted upper-case tickers. """ import re from data_sync_client import _get_cache_dir cache_dir = _get_cache_dir() if not cache_dir or not cache_dir.exists(): return {"tickers": []} pattern = re.compile(r"^GLOBAL_([a-z0-9_]+)_path_forest(?:_[a-z]+)?\.json$") found = set() try: for p in cache_dir.iterdir(): if not p.is_file(): continue m = pattern.match(p.name) if m: found.add(m.group(1).upper()) except Exception as e: log.warning(f"[forest] index scan failed: {e}") return {"tickers": sorted(found)} @app.get("/api/forest/{ticker}") async def get_forest(ticker: str, scenario: str = "mixed"): """ Path Outcome Forest forecast (PRO feature). Reads the cached path_forest JSON for the given ticker. Rule C1: no live computation. Rule C5: cache-only. """ from data_sync_client import _get_cache_dir canonical = (ticker or "").lower() if not canonical: raise HTTPException(400, "ticker required") valid = {"trend", "range", "stress", "mixed", "miscal"} if scenario not in valid: scenario = "mixed" cache_dir = _get_cache_dir() candidates = [ cache_dir / f"GLOBAL_{canonical}_path_forest.json", cache_dir / f"GLOBAL_{canonical}_path_forest_{scenario}.json", ] for p in candidates: try: if p.exists(): with open(p, "r", encoding="utf-8") as f: return json.load(f) except Exception as e: log.warning(f"[forest] read failed for {p}: {e}") continue raise HTTPException( 404, f"No path_forest data for {ticker} (tried scenario '{scenario}'). " f"Producer may not have fired yet." ) # ── MANUAL CONE ENDPOINTS ────────────────────────────────── # REMOVED: /api/cones/manual (Rule C1 — computation endpoint) @app.get("/api/performance/summary") async def get_performance_summary(days: int = 90): """ Comprehensive performance summary from auto_executor_log.json. Includes win/loss metrics in $ and pips, gate funnel, daily timeline. Used by the PerformanceDashboard component. Note: win/loss are EXPECTED (based on TP/SL), not realised (no outcome tracking yet). """ log_path = PROJECT_ROOT / "auto_executor_log.json" if not log_path.exists(): return {"total": 0, "days": days} try: with open(log_path) as f: all_records = json.load(f) except Exception: return {"total": 0, "error": "Failed to read log"} from datetime import timedelta cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat() records = [r for r in all_records if r.get("timestamp", "") >= cutoff] if not records: return {"total": 0, "days": days} # ── Gate funnel (all records) ── funnel: dict = {} for r in records: action = r.get("action", "unknown") gate = r.get("gate", action) key = gate if action in ("skipped", "blocked") else action funnel[key] = funnel.get(key, 0) + 1 # ── Executed trades ONLY for P&L metrics ── executed = [r for r in records if r.get("action") == "executed"] # Dry-run trades for activity display only dry_run = [r for r in records if r.get("action") == "dry_run"] trades = executed + dry_run # combined for non-P&L distributions ne = len(executed) # count of real executions # ── Win/loss helpers (expected, based on TP/SL distances) ── def _win_usd(r): return round(r.get("risk_usd", 0) * r.get("risk_reward", 0), 2) def _loss_usd(r): return round(r.get("risk_usd", 0), 2) def _win_pips(r): entry = r.get("fill_price") or r.get("entry_price", 0) target = r.get("take_profit", 0) if not entry or not target: return 0.0 return _to_pips(r.get("ticker", ""), abs(target - entry)) def _loss_pips(r): entry = r.get("fill_price") or r.get("entry_price", 0) sl = r.get("stop_loss", 0) if not entry or not sl: return 0.0 return _to_pips(r.get("ticker", ""), abs(entry - sl)) avg_win_usd = round(sum(_win_usd(r) for r in executed) / ne, 2) if ne else 0 avg_loss_usd = round(sum(_loss_usd(r) for r in executed) / ne, 2) if ne else 0 avg_win_pips = round(sum(_win_pips(r) for r in executed) / ne, 1) if ne else 0 avg_loss_pips = round(sum(_loss_pips(r) for r in executed) / ne, 1) if ne else 0 total_risk_usd = round(sum(_loss_usd(r) for r in executed), 2) avg_risk_pct = round(sum(r.get("risk_pct", 0) for r in executed) / ne, 4) if ne else 0 avg_conf = round(sum(r.get("confidence", 0) for r in executed) / ne, 4) if ne else 0 avg_rr = round(sum(r.get("risk_reward", 0) for r in executed) / ne, 2) if ne else 0 # ── By ticker (executed only) ── by_ticker: dict = {} for r in executed: t = r.get("ticker", "?") by_ticker.setdefault(t, { "count": 0, "risk_usd": 0.0, "win_usd": 0.0, "win_pips": 0.0, "loss_pips": 0.0, "confidence": [], }) by_ticker[t]["count"] += 1 by_ticker[t]["risk_usd"] = round(by_ticker[t]["risk_usd"] + _loss_usd(r), 2) by_ticker[t]["win_usd"] = round(by_ticker[t]["win_usd"] + _win_usd(r), 2) by_ticker[t]["win_pips"] = round(by_ticker[t]["win_pips"] + _win_pips(r), 1) by_ticker[t]["loss_pips"] = round(by_ticker[t]["loss_pips"] + _loss_pips(r), 1) by_ticker[t]["confidence"].append(r.get("confidence", 0)) for t, d in by_ticker.items(): c = d.pop("confidence") cnt = d["count"] d["avg_confidence"] = round(sum(c) / len(c), 4) if c else 0 d["avg_win_usd"] = round(d["win_usd"] / cnt, 2) if cnt else 0 d["avg_loss_usd"] = round(d["risk_usd"] / cnt, 2) if cnt else 0 d["avg_win_pips"] = round(d["win_pips"] / cnt, 1) if cnt else 0 d["avg_loss_pips"] = round(d["loss_pips"] / cnt, 1) if cnt else 0 # ── By signal type (executed only) ── by_type: dict = {} for r in executed: st = r.get("signal_type", "?") by_type[st] = by_type.get(st, 0) + 1 # ── By calibration grade (executed only) ── by_grade: dict = {} for r in executed: g = r.get("calibration_grade", "?") or "?" by_grade[g] = by_grade.get(g, 0) + 1 # ── By regime (executed only) ── by_regime: dict = {} for r in executed: reg = r.get("regime", "?") or "?" by_regime[reg] = by_regime.get(reg, 0) + 1 # ── By direction (executed only) ── by_direction: dict = {"long": 0, "short": 0} for r in executed: d = r.get("direction", "") if d in by_direction: by_direction[d] += 1 # ── Daily timeline (all records for activity view) ── daily: dict = {} for r in records: ts = r.get("timestamp", "")[:10] if not ts: continue if ts not in daily: daily[ts] = {"executed": 0, "dry_run": 0, "skipped": 0, "blocked": 0, "awaiting": 0} action = r.get("action", "") if action == "executed": daily[ts]["executed"] += 1 elif action == "dry_run": daily[ts]["dry_run"] += 1 elif action == "skipped": daily[ts]["skipped"] += 1 elif action == "blocked": daily[ts]["blocked"] += 1 elif action == "awaiting_confirmation": daily[ts]["awaiting"] += 1 timeline = [{"date": d, **counts} for d, counts in sorted(daily.items())] # ── Skipped by gate ── by_gate: dict = {} for r in records: if r.get("action") == "skipped": g = r.get("gate", "unknown") by_gate[g] = by_gate.get(g, 0) + 1 return { "days": days, "total_records": len(records), "total_executed": ne, "total_dry_run": len(dry_run), "avg_confidence": avg_conf, "avg_risk_reward": avg_rr, "avg_win_usd": avg_win_usd, "avg_loss_usd": avg_loss_usd, "avg_win_pips": avg_win_pips, "avg_loss_pips": avg_loss_pips, "total_risk_usd": total_risk_usd, "avg_risk_pct": avg_risk_pct, "funnel": funnel, "by_gate": by_gate, "timeline": timeline, "by_ticker": by_ticker, "by_type": by_type, "by_grade": by_grade, "by_regime": by_regime, "by_direction": by_direction, } # ── WEBSOCKET ENDPOINTS ───────────────────────────────────── @app.websocket("/ws/prices") async def ws_prices(ws: WebSocket): """ Live price stream. Client can optionally send subscribe messages to filter tickers. By default, all universe tickers are streamed. """ await manager.connect_prices(ws) try: while True: # Keep connection alive; process client messages try: raw = await asyncio.wait_for(ws.receive_text(), timeout=30.0) # Parse client commands (subscribe/unsubscribe) try: msg = json.loads(raw) if msg.get("action") == "subscribe": # Future: per-client filtering log.debug(f"Client subscribe: {msg.get('tickers', [])}") except json.JSONDecodeError: pass except asyncio.TimeoutError: # Send keepalive ping try: await ws.send_text(json.dumps({"type": "ping"})) except Exception: break except WebSocketDisconnect: pass finally: manager.disconnect_prices(ws) @app.websocket("/ws/events") async def ws_events(ws: WebSocket): """ State change event stream. Pushes notifications when state files change. """ await manager.connect_events(ws) try: while True: try: # Just keep alive — events are pushed server-side await asyncio.wait_for(ws.receive_text(), timeout=30.0) except asyncio.TimeoutError: try: await ws.send_text(json.dumps({"type": "ping"})) except Exception: break except WebSocketDisconnect: pass finally: manager.disconnect_events(ws) # ============================================================ # 7. ENTRY POINT # ============================================================ def main(): parser = argparse.ArgumentParser(description="Quantum Terminal Data Server") parser.add_argument("--host", default=server_config.host, help="Bind address") parser.add_argument("--port", type=int, default=server_config.port, help="Port number") parser.add_argument("--reload", action="store_true", help="Auto-reload on code changes") args = parser.parse_args() server_config.host = args.host server_config.port = args.port uvicorn.run( "data_server:app", host=args.host, port=args.port, reload=args.reload, log_level="info", ) if __name__ == "__main__": main()