Quantum Terminal
1378 lines
55 KiB
Python
1378 lines
55 KiB
Python
# version: v12
|
|
"""
|
|
================================================================================
|
|
Quantum Terminal Consumer — Data Sync Client
|
|
================================================================================
|
|
Downloads pre-computed data from the Quantum Terminal VPS and caches it locally
|
|
for offline resilience.
|
|
|
|
v11 — Parallel sync via httpx + asyncio.gather. The manifest path now
|
|
issues all per-file GETs concurrently (semaphore-bounded at 8) using
|
|
a single httpx.AsyncClient. On a typical fresh install (~150 files),
|
|
drops total sync time from ~3 min serial to ~20-30 s. Public
|
|
sync_all() stays a synchronous method; it spins up its own event
|
|
loop via asyncio.run() so existing callers (lifespan thread,
|
|
asyncio.to_thread wrappers in route handlers, periodic_sync) keep
|
|
working unchanged. The legacy path (manifest 404) stays serial
|
|
using `requests` — it's rarely hit in production and not worth the
|
|
additional surface area.
|
|
|
|
v10 — Download-speed instrumentation. Adds `sync_bytes_total` (cumulative
|
|
bytes synced this run) to /api/sync/status. The frontend remembers
|
|
the previous poll's value + timestamp via useRef and computes a
|
|
live KB/s or MB/s readout next to the progress bar. Each successful
|
|
download adds the JSON-encoded payload size — an estimate (no
|
|
transport headers / compression accounted for) but close enough
|
|
for a user-visible speed indicator.
|
|
|
|
v9 — Sync-progress instrumentation. Tracks `_sync_total` (files the
|
|
manifest says we need) and `_sync_done` (files completed so far),
|
|
plus exposes both alongside `sync_in_progress` in /api/sync/status.
|
|
Drives the new top-toolbar download progress bar in the consumer UI.
|
|
|
|
v8 — Async-handler fix. The /api/sync/refresh, /api/sync/refresh/{ticker}
|
|
and /api/sync/clear-cache routes were declared `async def` but called
|
|
the synchronous `client.sync_all()` / `client.sync_ticker()` /
|
|
`client.clear_cache()` directly. That FROZE the FastAPI event loop
|
|
for the entire duration of sync (5+ min on first install), starving
|
|
every other endpoint — including /api/sync/status, which is what the
|
|
LoginScreen v9 fast-path polls to detect format-version mismatch.
|
|
Symptom: user passes login, sees "Verifying data version…" forever
|
|
because the status poll was queued behind the blocked refresh.
|
|
Fix: wrap every blocking sync call in `asyncio.to_thread()` so it
|
|
runs on the worker thread pool while the event loop stays free to
|
|
serve other requests.
|
|
|
|
v7 — Startup-latency fix. Previous behavior made the initial sync take up
|
|
to 15 minutes for users whose servers returned persistent 5xx on
|
|
specific endpoints (notably /data/{ticker}/probfield returning 500
|
|
instead of 404 — known server bug). Each failing endpoint cost up to
|
|
70 s (30 s timeout + 10 s sleep + 30 s retry). Changes:
|
|
• Per-file `timeout` dropped 30 s → 8 s. Consumer is display-only;
|
|
a healthy server responds in well under a second.
|
|
• Retry sleep on 5xx dropped 10 s → 1 s (retains transient-blip
|
|
protection without stalling).
|
|
• Manifest fetch timeout dropped 15 s → 8 s (same reasoning).
|
|
Worst case per failing endpoint now ~17 s (8 + 1 + 8) vs prior 70 s.
|
|
Across a universe with several persistent 500s this turns a 15-min
|
|
startup into roughly ~2 min.
|
|
|
|
v2.1 — Incremental sync with full data category support:
|
|
- Manifest-first sync (only download what changed)
|
|
- If-Modified-Since headers (server returns 304)
|
|
- Categories: cones, options, probfield, bands, fundamentals, signals,
|
|
lifecycle, weeklystate
|
|
- Global data (fundamentals, signals, etc.) stored under ticker "GLOBAL"
|
|
- Falls back to legacy full sync if manifest unavailable
|
|
|
|
v2 — Data Center transition: bands externalized to {TICKER}_bands.json
|
|
(previously nested inside {TICKER}_cones.json). Bands is now a first-class
|
|
per-ticker category synced alongside cones/options/probfield.
|
|
|
|
v6 — Fix: `total` counter in _load_all_from_cache was referenced but never
|
|
initialized after the v5 rewrite, causing an UnboundLocalError the first
|
|
time the sweep matched a file. This crashed the end-of-sync_all orphan
|
|
sweep (added in v5), so misnamed cache files like GLOBAL_eurusd_bands.json
|
|
never got loaded → bands empty for those tickers. Just adds `total = 0`
|
|
above the loop.
|
|
|
|
v5 — _load_all_from_cache() is now also invoked at the END of every
|
|
successful sync_all() (online path), not just on offline/failure. The
|
|
manifest-driven sync only knows about files the server advertises —
|
|
orphan cache files (e.g. legacy GLOBAL_<tkr>_bands.json drops that
|
|
aren't in the manifest) were never getting into the in-memory stores.
|
|
Now they are, via the shared resolver. HTTP-fresh data is still written
|
|
first, so the reload confirms the same content for advertised files
|
|
and adds any orphans as a second pass. Slot guard prevents the reload
|
|
from clobbering fresher HTTP data.
|
|
|
|
v4 — Filename-resolution helper `resolve_cache_filename()` added and used by
|
|
`_load_all_from_cache()` (and reused from cache_watcher.py). Handles:
|
|
- bare globals e.g. "money_flow.json"
|
|
- prefixed globals e.g. "GLOBAL_probability_state.json"
|
|
(fixes mis-parse of multi-word categories)
|
|
- legacy per-ticker misnaming e.g. "GLOBAL_eurusd_bands.json" →
|
|
reroutes in-memory to (EURUSD, bands)
|
|
- normal per-ticker e.g. "EURUSD_bands.json"
|
|
Keeps stale/mislabeled cache files usable without touching disk.
|
|
|
|
v3 — Data Center transition: five new GLOBAL categories registered for the
|
|
forward-looking probability layer (shipped under temp_out/probability/
|
|
on the DC side, served as /data/GLOBAL/<key> and cached as
|
|
GLOBAL_<key>.json):
|
|
- probability_state (aggregator containing the 4 below)
|
|
- regime_transitions (Markov model between regimes)
|
|
- score_cones (per-asset probability cones for 15 tickers)
|
|
- leading_composite (leading vs coincident indicator composite)
|
|
- liquidity_forecast (6-mo net-liquidity projection with cones)
|
|
Each file is independently synced, watched, and served — lets the DC
|
|
recompute components at different cadences without redownloading the
|
|
whole bundle. Exposed to the frontend via /api/data/global/{key}.
|
|
|
|
Storage:
|
|
%APPDATA%\\QuantumTerminal\\cache\\{TICKER}_cones.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\{TICKER}_bands.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\{TICKER}_options.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\{TICKER}_probfield.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\GLOBAL_fundamentals.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\GLOBAL_signals.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\GLOBAL_lifecycle.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\GLOBAL_weeklystate.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\_data_status.json
|
|
%APPDATA%\\QuantumTerminal\\cache\\_sync_timestamps.json
|
|
|
|
This module does NOT introduce any calculation triggers.
|
|
================================================================================
|
|
"""
|
|
|
|
import asyncio # v8: needed for asyncio.to_thread to keep async handlers from blocking
|
|
import json
|
|
import logging
|
|
import os
|
|
import time
|
|
import configparser
|
|
from pathlib import Path
|
|
|
|
# AppData directory name.
|
|
APP_DIR = os.environ.get("MK_APP_DIR_NAME", "QuantumTerminal")
|
|
from datetime import datetime, timezone, timedelta
|
|
from email.utils import formatdate
|
|
from typing import Optional, Dict, List, Any, Tuple
|
|
from dataclasses import dataclass, field, asdict
|
|
from enum import Enum
|
|
|
|
log = logging.getLogger("mk.data_sync")
|
|
|
|
PROJECT_ROOT = Path(__file__).resolve().parent
|
|
|
|
# -- Format version this client expects (Rule C6) --
|
|
EXPECTED_MAJOR_VERSION = 1
|
|
|
|
# Per-ticker data categories (well-known types)
|
|
TICKER_CATEGORIES = ("cones", "options", "probfield", "bands")
|
|
|
|
# Known global categories — but the system accepts ANY category from the manifest.
|
|
# New global files are auto-discovered; no code change needed.
|
|
KNOWN_GLOBAL = ("fundamentals", "fundamental_state", "daily_signals",
|
|
"signal_lifecycle", "weekly_state", "money_flow",
|
|
"central_bank_rates", "macro_sectors", "macro_countries", "macro_flows",
|
|
"macro_sector_flows",
|
|
# v3: probability layer (Data Center forward-looking globals)
|
|
"probability_state", "regime_transitions", "score_cones",
|
|
"leading_composite", "liquidity_forecast")
|
|
|
|
# ── Global data endpoints (universe-level, not per-ticker) ──────────────────
|
|
GLOBAL_ENDPOINTS = {
|
|
"fundamentals": "/data/GLOBAL/fundamentals",
|
|
"fundamental_state": "/data/GLOBAL/fundamental_state",
|
|
"money_flow": "/data/GLOBAL/money_flow",
|
|
"macro_sectors": "/data/GLOBAL/macro_sectors",
|
|
"macro_countries": "/data/GLOBAL/macro_countries",
|
|
"macro_flows": "/data/GLOBAL/macro_flows",
|
|
"macro_sector_flows": "/data/GLOBAL/global_macro_sector_flows",
|
|
"central_bank_rates": "/data/GLOBAL/central_bank_rates",
|
|
# v3: probability layer
|
|
"probability_state": "/data/GLOBAL/probability_state",
|
|
"regime_transitions": "/data/GLOBAL/regime_transitions",
|
|
"score_cones": "/data/GLOBAL/score_cones",
|
|
"leading_composite": "/data/GLOBAL/leading_composite",
|
|
"liquidity_forecast": "/data/GLOBAL/liquidity_forecast",
|
|
}
|
|
|
|
# Cache filenames for global data
|
|
GLOBAL_CACHE_NAMES = {
|
|
"fundamentals": "GLOBAL_fundamentals.json",
|
|
"fundamental_state": "GLOBAL_fundamental_state.json",
|
|
"money_flow": "GLOBAL_money_flow.json",
|
|
"macro_sectors": "GLOBAL_macro_sectors.json",
|
|
"macro_countries": "GLOBAL_macro_countries.json",
|
|
"macro_flows": "GLOBAL_macro_flows.json",
|
|
"macro_sector_flows": "GLOBAL_macro_sector_flows.json",
|
|
"central_bank_rates": "GLOBAL_central_bank_rates.json",
|
|
# v3: probability layer
|
|
"probability_state": "GLOBAL_probability_state.json",
|
|
"regime_transitions": "GLOBAL_regime_transitions.json",
|
|
"score_cones": "GLOBAL_score_cones.json",
|
|
"leading_composite": "GLOBAL_leading_composite.json",
|
|
"liquidity_forecast": "GLOBAL_liquidity_forecast.json",
|
|
}
|
|
|
|
|
|
# ============================================================
|
|
# 1. STALENESS LEVELS (Rule C4)
|
|
# ============================================================
|
|
|
|
class FreshnessLevel(str, Enum):
|
|
FRESH = "fresh"
|
|
WARNING = "warning"
|
|
ERROR = "error"
|
|
UNKNOWN = "unknown"
|
|
|
|
|
|
@dataclass
|
|
class StalenessInfo:
|
|
ticker: str
|
|
computed_at: Optional[str] = None
|
|
age_hours: float = 0.0
|
|
level: FreshnessLevel = FreshnessLevel.UNKNOWN
|
|
display_date: str = "N/A"
|
|
|
|
|
|
# ============================================================
|
|
# 2. CONFIG
|
|
# ============================================================
|
|
|
|
def _read_config() -> configparser.ConfigParser:
|
|
import platform as _plat
|
|
config = configparser.ConfigParser()
|
|
|
|
search_paths = [
|
|
PROJECT_ROOT / "consumer_config.ini",
|
|
PROJECT_ROOT.parent / "consumer_config.ini",
|
|
]
|
|
|
|
import sys as _sys
|
|
if getattr(_sys, 'frozen', False):
|
|
search_paths.append(Path(_sys.executable).parent / "consumer_config.ini")
|
|
|
|
if _plat.system() == "Windows":
|
|
import os as _os
|
|
appdata = _os.environ.get("APPDATA", "")
|
|
if appdata:
|
|
search_paths.append(Path(appdata) / APP_DIR / "consumer_config.ini")
|
|
|
|
for p in _sys.path:
|
|
candidate = Path(p) / "consumer_config.ini"
|
|
if candidate not in search_paths:
|
|
search_paths.append(candidate)
|
|
|
|
for path in search_paths:
|
|
if path.exists():
|
|
config.read(str(path), encoding="utf-8-sig")
|
|
return config
|
|
|
|
config.read_dict({
|
|
"data": {
|
|
"cache_dir": f"%APPDATA%\\{APP_DIR}\\cache",
|
|
"stale_warning_hours": "48",
|
|
"stale_error_hours": "72",
|
|
},
|
|
})
|
|
return config
|
|
|
|
|
|
def _get_stale_thresholds() -> Tuple[float, float]:
|
|
cfg = _read_config()
|
|
warn = float(cfg.get("data", "stale_warning_hours", fallback="48"))
|
|
err = float(cfg.get("data", "stale_error_hours", fallback="72"))
|
|
return warn, err
|
|
|
|
|
|
# ============================================================
|
|
# 3. CACHE DIRECTORY
|
|
# ============================================================
|
|
|
|
def _get_cache_dir() -> Path:
|
|
import platform, os
|
|
if platform.system() == "Windows":
|
|
appdata = os.environ.get("APPDATA", "")
|
|
if appdata:
|
|
base = Path(appdata) / APP_DIR / "cache"
|
|
else:
|
|
base = Path.home() / "AppData" / "Roaming" / APP_DIR / "cache"
|
|
else:
|
|
base = Path.home() / ".QuantumTerminal" / "cache"
|
|
|
|
base.mkdir(parents=True, exist_ok=True)
|
|
return base
|
|
|
|
|
|
def _cache_path(ticker: str, data_type: str) -> Path:
|
|
return _get_cache_dir() / f"{ticker.upper()}_{data_type}.json"
|
|
|
|
|
|
def _status_cache_path() -> Path:
|
|
return _get_cache_dir() / "_data_status.json"
|
|
|
|
|
|
def _sync_timestamps_path() -> Path:
|
|
return _get_cache_dir() / "_sync_timestamps.json"
|
|
|
|
|
|
# ============================================================
|
|
# 4. CACHE I/O
|
|
# ============================================================
|
|
|
|
def _write_cache(path: Path, data: dict):
|
|
try:
|
|
path.write_text(json.dumps(data, separators=(",", ":")), encoding="utf-8")
|
|
except Exception as e:
|
|
log.warning(f"Failed to write cache {path.name}: {e}")
|
|
|
|
|
|
def _read_cache(path: Path) -> Optional[dict]:
|
|
if not path.exists():
|
|
return None
|
|
try:
|
|
return json.loads(path.read_text(encoding="utf-8"))
|
|
except Exception as e:
|
|
log.warning(f"Failed to read cache {path.name}: {e}")
|
|
return None
|
|
|
|
|
|
# ============================================================
|
|
# 5. SYNC TIMESTAMP TRACKER
|
|
# ============================================================
|
|
|
|
def _load_sync_timestamps() -> dict:
|
|
data = _read_cache(_sync_timestamps_path())
|
|
return data if data else {}
|
|
|
|
|
|
def _save_sync_timestamps(timestamps: dict):
|
|
_write_cache(_sync_timestamps_path(), timestamps)
|
|
|
|
|
|
def _ts_key(category: str, ticker: str) -> str:
|
|
return f"{category}:{ticker.upper()}"
|
|
|
|
|
|
# v4: single source of truth for parsing cache filenames. Returns (ticker, category)
|
|
# or None. Handles bare globals, prefixed globals (incl. multi-word categories),
|
|
# legacy GLOBAL_<lowercase_ticker>_<category> misnaming, and normal per-ticker.
|
|
def resolve_cache_filename(stem: str):
|
|
"""Map a cache filename stem (without .json) to (ticker, category).
|
|
|
|
Examples:
|
|
"EURUSD_bands" → ("EURUSD", "bands")
|
|
"GLOBAL_fundamentals" → ("GLOBAL", "fundamentals")
|
|
"GLOBAL_fundamental_state" → ("GLOBAL", "fundamental_state")
|
|
"GLOBAL_probability_state" → ("GLOBAL", "probability_state")
|
|
"GLOBAL_money_flow" → ("GLOBAL", "money_flow")
|
|
"GLOBAL_eurusd_bands" → ("EURUSD", "bands") [legacy reroute]
|
|
"GLOBAL_xauusd_historical_cones" → ("XAUUSD", "historical_cones") [legacy reroute]
|
|
"money_flow" → ("GLOBAL", "money_flow") [bare global]
|
|
"probability_state" → ("GLOBAL", "probability_state") [bare global]
|
|
"""
|
|
import re as _re
|
|
if not stem:
|
|
return None
|
|
low = stem.lower()
|
|
known_global_cats = set(KNOWN_GLOBAL) | set(GLOBAL_ENDPOINTS.keys())
|
|
|
|
# (a) Bare global — DC writes "money_flow.json" instead of "GLOBAL_money_flow.json".
|
|
if low in known_global_cats:
|
|
return ("GLOBAL", low)
|
|
|
|
# (b) Prefixed global — "GLOBAL_<known_cat>". Must come BEFORE the greedy
|
|
# regex (which would misparse multi-word categories).
|
|
if low.startswith("global_"):
|
|
suffix = low[len("global_"):]
|
|
if suffix in known_global_cats:
|
|
return ("GLOBAL", suffix)
|
|
# (c) Legacy misnaming — "GLOBAL_<lowercase_ticker>_<category>".
|
|
m = _re.match(r'^([a-z0-9]+)_([a-z_]+)$', suffix)
|
|
if m:
|
|
return (m.group(1).upper(), m.group(2))
|
|
return None
|
|
|
|
# (d) Normal per-ticker — greedy split on the last "_<category>" suffix.
|
|
m = _re.match(r'^(.+)_([a-z_]+)$', stem, _re.IGNORECASE)
|
|
if m:
|
|
return (m.group(1).upper(), m.group(2).lower())
|
|
return None
|
|
|
|
|
|
# ============================================================
|
|
# 6. FORMAT VERSION CHECK (Rule C6)
|
|
# ============================================================
|
|
|
|
def _check_format_version(data: dict) -> bool:
|
|
"""Telemetry only — logs format_version but never gates (Rule C6 reversal 2026-05-03)."""
|
|
version_str = data.get("format_version", "1.0")
|
|
try:
|
|
major = int(version_str.split(".")[0])
|
|
if major != EXPECTED_MAJOR_VERSION:
|
|
log.info(
|
|
f"format_version telemetry: server={version_str}, baseline major={EXPECTED_MAJOR_VERSION}"
|
|
)
|
|
except (ValueError, IndexError):
|
|
log.warning(f"Could not parse format_version: {version_str}")
|
|
return True
|
|
|
|
|
|
# ============================================================
|
|
# 7. STALENESS CALCULATOR (Rule C4)
|
|
# ============================================================
|
|
|
|
def _compute_staleness(computed_at: Optional[str], ticker: str) -> StalenessInfo:
|
|
if not computed_at:
|
|
return StalenessInfo(ticker=ticker)
|
|
|
|
try:
|
|
ts = datetime.fromisoformat(computed_at.replace("Z", "+00:00"))
|
|
if ts.tzinfo is None:
|
|
ts = ts.replace(tzinfo=timezone.utc)
|
|
now = datetime.now(timezone.utc)
|
|
age = (now - ts).total_seconds() / 3600.0
|
|
|
|
warn_h, err_h = _get_stale_thresholds()
|
|
|
|
if age > err_h:
|
|
level = FreshnessLevel.ERROR
|
|
elif age > warn_h:
|
|
level = FreshnessLevel.WARNING
|
|
else:
|
|
level = FreshnessLevel.FRESH
|
|
|
|
display_date = ts.strftime("%Y-%m-%d")
|
|
|
|
return StalenessInfo(
|
|
ticker=ticker,
|
|
computed_at=computed_at,
|
|
age_hours=round(age, 1),
|
|
level=level,
|
|
display_date=display_date,
|
|
)
|
|
except Exception:
|
|
return StalenessInfo(ticker=ticker, computed_at=computed_at)
|
|
|
|
|
|
# ============================================================
|
|
# 8. DATA SYNC CLIENT
|
|
# ============================================================
|
|
|
|
class DataSyncClient:
|
|
"""
|
|
Singleton data sync client with incremental sync.
|
|
|
|
v2.1 flow (sync_all):
|
|
1. GET /data/manifest -> per-file metadata with updated_at
|
|
2. Compare each file's updated_at against local _sync_timestamps.json
|
|
3. Only download files where server version is newer
|
|
4. Each request includes If-Modified-Since header (server returns 304)
|
|
5. Global data (fundamentals, signals, etc.) fetched as GLOBAL ticker
|
|
6. On any failure -> fall back to cache (Rule C5)
|
|
"""
|
|
|
|
def __init__(self):
|
|
# In-memory data stores by category (dynamic — no hardcoded list)
|
|
self._stores: Dict[str, Dict[str, dict]] = {}
|
|
self._data_status: Optional[dict] = None
|
|
self._sync_errors: List[str] = []
|
|
self._format_version_ok = True
|
|
self._offline_mode = False
|
|
self._last_synced_at: Optional[str] = None
|
|
# Tracks background sync activity so the terminal top-bar pill can
|
|
# show "SYNCING" / "READY" without the user being stuck on a loader.
|
|
self._sync_in_progress: bool = False
|
|
# v9: progress counters — drive the toolbar progress bar.
|
|
self._sync_total: int = 0
|
|
self._sync_done: int = 0
|
|
# v10: cumulative bytes downloaded this sync — frontend computes
|
|
# speed from the delta between two consecutive /api/sync/status polls.
|
|
self._sync_bytes_total: int = 0
|
|
self._sync_timestamps: Dict[str, str] = _load_sync_timestamps()
|
|
# Track which categories we've seen (from manifest or cache)
|
|
self._known_categories: set = set(TICKER_CATEGORIES)
|
|
|
|
def _get_store(self, category: str) -> Dict[str, dict]:
|
|
if category not in self._stores:
|
|
self._stores[category] = {}
|
|
return self._stores[category]
|
|
|
|
# ---- Public: sync all (manifest-first) ----
|
|
|
|
def sync_all(self) -> dict:
|
|
"""Public entry — runs the async core in a fresh event loop via
|
|
asyncio.run() so existing synchronous callers stay unchanged.
|
|
Caller MUST NOT be inside a running event loop; wrap with
|
|
asyncio.to_thread(client.sync_all) if calling from async context."""
|
|
self._sync_in_progress = True
|
|
# v9: reset counters on each entry so a re-sync starts at 0.
|
|
self._sync_total = 0
|
|
self._sync_done = 0
|
|
# v10: reset cumulative bytes too.
|
|
self._sync_bytes_total = 0
|
|
try:
|
|
return asyncio.run(self._sync_all_async())
|
|
finally:
|
|
self._sync_in_progress = False
|
|
# v9: clamp `done` to `total` so the bar reads 100% on completion
|
|
# even if the manifest path bailed out before populating total.
|
|
if self._sync_total > 0:
|
|
self._sync_done = self._sync_total
|
|
|
|
async def _sync_all_async(self) -> dict:
|
|
"""v11: parallel manifest-based sync. Issues all per-file GETs
|
|
concurrently via httpx.AsyncClient + asyncio.gather, semaphore-
|
|
bounded at 8 to avoid hammering the server. Falls back to the
|
|
synchronous legacy path on manifest 404."""
|
|
import httpx
|
|
|
|
self._sync_errors = []
|
|
summary = {"offline_mode": False, "format_version_ok": True,
|
|
"synced": 0, "skipped": 0}
|
|
|
|
# Quantum Terminal: No auth required — load from cache directly.
|
|
jwt = ""
|
|
|
|
if not jwt:
|
|
log.warning("No JWT — loading from cache (Rule C5)")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
base_url = _read_config().get("server", "base_url", fallback="").rstrip("/")
|
|
headers = {"Authorization": f"Bearer {jwt}"}
|
|
timeout = httpx.Timeout(connect=8.0, read=8.0, write=8.0, pool=8.0)
|
|
# Pool limits — keep headroom over the semaphore concurrency so we
|
|
# never starve.
|
|
limits = httpx.Limits(max_connections=20, max_keepalive_connections=12)
|
|
|
|
async with httpx.AsyncClient(base_url=base_url, headers=headers,
|
|
timeout=timeout, limits=limits) as http:
|
|
# Step 1: manifest
|
|
try:
|
|
r = await http.get("/data/manifest")
|
|
except Exception as e:
|
|
log.warning(f"Manifest unreachable: {e}")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
m_status = r.status_code
|
|
|
|
if m_status in (401, 403):
|
|
log.warning(f"Auth error during data sync (HTTP {m_status})")
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
if m_status == 404:
|
|
log.info("Manifest not found — falling back to legacy serial sync")
|
|
# Legacy path is sync (uses requests) — run it on a worker thread.
|
|
return await asyncio.to_thread(self._sync_all_legacy_sync, summary)
|
|
|
|
if m_status != 200:
|
|
log.warning(f"Manifest failed (HTTP {m_status}) — using cache")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
try:
|
|
manifest = r.json()
|
|
except Exception as e:
|
|
log.warning(f"Manifest JSON parse failed: {e}")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
# Step 2: format-version check at the manifest level.
|
|
if not _check_format_version(manifest):
|
|
self._format_version_ok = False
|
|
summary["format_version_ok"] = False
|
|
return summary
|
|
|
|
# Step 3: parallel downloads.
|
|
server_files = manifest.get("files", {})
|
|
self._sync_total = sum(len(c) for c in server_files.values())
|
|
self._sync_done = 0
|
|
|
|
sem = asyncio.Semaphore(8)
|
|
tasks = []
|
|
|
|
async def fetch_one(ticker: str, category: str, server_updated: str):
|
|
async with sem:
|
|
local_key = _ts_key(category, ticker)
|
|
local_updated = self._sync_timestamps.get(local_key, "")
|
|
# Skip if local is current
|
|
if local_updated and server_updated and local_updated >= server_updated:
|
|
cached = _read_cache(_cache_path(ticker, category))
|
|
if cached:
|
|
self._get_store(category)[ticker.upper()] = cached
|
|
summary["skipped"] += 1
|
|
self._sync_done += 1
|
|
return
|
|
result = await self._sync_one_async(http, ticker, category)
|
|
if result in ("synced", "not_modified"):
|
|
self._sync_timestamps[local_key] = server_updated
|
|
if result == "not_modified":
|
|
cached = _read_cache(_cache_path(ticker, category))
|
|
if cached:
|
|
self._get_store(category)[ticker.upper()] = cached
|
|
summary["skipped"] += 1
|
|
else:
|
|
summary["synced"] += 1
|
|
self._sync_done += 1
|
|
|
|
for category, cat_files in server_files.items():
|
|
self._known_categories.add(category)
|
|
for ticker, meta in cat_files.items():
|
|
tasks.append(fetch_one(ticker, category, meta.get("updated_at", "")))
|
|
|
|
await asyncio.gather(*tasks, return_exceptions=True)
|
|
|
|
_save_sync_timestamps(self._sync_timestamps)
|
|
# Sweep cache for orphan files (legacy misnaming, manual drops).
|
|
self._load_all_from_cache(skip_existing=True)
|
|
|
|
log.info(f"Parallel sync: {summary['synced']} downloaded, {summary['skipped']} skipped (concurrency=8)")
|
|
self._last_synced_at = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
|
return summary
|
|
|
|
async def _sync_one_async(self, http, ticker: str, data_type: str) -> str:
|
|
"""v11: async equivalent of _sync_one using httpx. Same retry / 304 /
|
|
404 / 5xx behavior as the sync version."""
|
|
canonical = ticker.upper()
|
|
endpoint = f"/data/{canonical}/{data_type}"
|
|
|
|
extra_headers = {}
|
|
local_key = _ts_key(data_type, canonical)
|
|
local_ts = self._sync_timestamps.get(local_key)
|
|
if local_ts:
|
|
try:
|
|
ts_dt = datetime.fromisoformat(local_ts)
|
|
extra_headers["If-Modified-Since"] = formatdate(ts_dt.timestamp(), usegmt=True)
|
|
except (ValueError, TypeError):
|
|
pass
|
|
|
|
async def _do_request():
|
|
return await http.get(endpoint, headers=extra_headers)
|
|
|
|
try:
|
|
r = await _do_request()
|
|
except Exception as e:
|
|
log.warning(f"Sync request failed for {canonical}/{data_type}: {e}")
|
|
await asyncio.sleep(1)
|
|
try:
|
|
r = await _do_request()
|
|
except Exception as e2:
|
|
log.warning(f"Sync retry failed: {e2}")
|
|
cached = _read_cache(_cache_path(canonical, data_type))
|
|
if cached:
|
|
self._get_store(data_type)[canonical] = cached
|
|
return "cached"
|
|
return "failed"
|
|
|
|
status = r.status_code
|
|
|
|
# v8: retry once on 5xx with a short sleep (matches sync version).
|
|
if status >= 500:
|
|
log.warning(f"Server error {status} for {endpoint} — retrying in 1s")
|
|
await asyncio.sleep(1)
|
|
try:
|
|
r = await _do_request()
|
|
status = r.status_code
|
|
except Exception:
|
|
pass
|
|
|
|
if status == 304:
|
|
return "not_modified"
|
|
|
|
if status == 200:
|
|
try:
|
|
data = r.json()
|
|
except Exception as e:
|
|
log.warning(f"JSON parse failed for {canonical}/{data_type}: {e}")
|
|
return "failed"
|
|
if not _check_format_version(data):
|
|
self._format_version_ok = False
|
|
return "failed"
|
|
self._get_store(data_type)[canonical] = data
|
|
_write_cache(_cache_path(canonical, data_type), data)
|
|
try:
|
|
self._sync_bytes_total += len(json.dumps(data).encode("utf-8"))
|
|
except Exception:
|
|
pass
|
|
return "synced"
|
|
|
|
if status == 404:
|
|
return "cached"
|
|
|
|
# Other / unexpected — fall back to disk cache if present.
|
|
log.warning(f"Failed to sync {data_type} for {canonical} (HTTP {status})")
|
|
self._sync_errors.append(f"{canonical}/{data_type}: HTTP {status}")
|
|
cached = _read_cache(_cache_path(canonical, data_type))
|
|
if cached:
|
|
self._get_store(data_type)[canonical] = cached
|
|
return "cached"
|
|
return "failed"
|
|
|
|
def _sync_all_legacy_sync(self, summary: dict) -> dict:
|
|
"""v11: thin wrapper invoking the existing sync legacy path on the
|
|
worker thread. The legacy path uses `requests` and is rarely hit
|
|
(only when the server returns 404 on /data/manifest)."""
|
|
log.warning("Legacy sync unavailable in Quantum Terminal — using cache")
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
def _sync_all_unchecked(self) -> dict:
|
|
# Quantum Terminal: no license client needed
|
|
client = None
|
|
|
|
self._sync_errors = []
|
|
summary = {"offline_mode": False, "format_version_ok": True,
|
|
"synced": 0, "skipped": 0}
|
|
|
|
# Step 1: Try manifest
|
|
manifest, m_status = client.auth_request("GET", "/data/manifest", timeout=8) # v7
|
|
|
|
if m_status == 0:
|
|
log.warning("Server unreachable -- loading from cache (Rule C5)")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
if m_status in (401, 403):
|
|
log.warning(f"Auth error during data sync (HTTP {m_status})")
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
if m_status == 404:
|
|
log.info("Manifest not found -- falling back to legacy sync")
|
|
return self._sync_all_legacy(client, summary)
|
|
|
|
if m_status != 200 or manifest is None:
|
|
log.warning(f"Manifest failed (HTTP {m_status}) -- using cache")
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
# Step 2: Format version check
|
|
if not _check_format_version(manifest):
|
|
self._format_version_ok = False
|
|
summary["format_version_ok"] = False
|
|
return summary
|
|
|
|
# Step 3: Download only changed files — iterate ALL categories from manifest
|
|
server_files = manifest.get("files", {})
|
|
|
|
# v9: count total files in this sync up front so the toolbar
|
|
# progress bar can render `done / total`.
|
|
self._sync_total = sum(len(c) for c in server_files.values())
|
|
self._sync_done = 0
|
|
|
|
for category, cat_files in server_files.items():
|
|
self._known_categories.add(category)
|
|
|
|
for ticker, meta in cat_files.items():
|
|
server_updated = meta.get("updated_at", "")
|
|
local_key = _ts_key(category, ticker)
|
|
local_updated = self._sync_timestamps.get(local_key, "")
|
|
|
|
# Skip if local is up to date
|
|
if local_updated and server_updated and local_updated >= server_updated:
|
|
cached = _read_cache(_cache_path(ticker, category))
|
|
if cached:
|
|
self._get_store(category)[ticker.upper()] = cached
|
|
summary["skipped"] += 1
|
|
self._sync_done += 1 # v9
|
|
continue
|
|
|
|
# Download this file
|
|
result = self._sync_one(client, ticker, category)
|
|
if result in ("synced", "not_modified"):
|
|
self._sync_timestamps[local_key] = server_updated
|
|
if result == "not_modified":
|
|
cached = _read_cache(_cache_path(ticker, category))
|
|
if cached:
|
|
self._get_store(category)[ticker.upper()] = cached
|
|
summary["skipped"] += 1
|
|
else:
|
|
summary["synced"] += 1
|
|
self._sync_done += 1 # v9 — count even on failed/cached fall-through
|
|
|
|
_save_sync_timestamps(self._sync_timestamps)
|
|
|
|
# v5: sweep cache for orphan files not in the manifest (legacy
|
|
# misnaming, manually dropped files, older categories). HTTP-fresh
|
|
# entries already in the store are preserved via skip_existing=True.
|
|
self._load_all_from_cache(skip_existing=True)
|
|
|
|
log.info(f"Incremental sync: {summary['synced']} downloaded, {summary['skipped']} skipped")
|
|
self._last_synced_at = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
|
return summary
|
|
|
|
# ---- Legacy full sync (fallback) ----
|
|
|
|
def _sync_all_legacy(self, client, summary: dict) -> dict:
|
|
data, status = client.auth_request("GET", "/data/status")
|
|
|
|
if status != 200 or data is None:
|
|
self._offline_mode = True
|
|
summary["offline_mode"] = True
|
|
self._load_all_from_cache()
|
|
return summary
|
|
|
|
if not _check_format_version(data):
|
|
self._format_version_ok = False
|
|
summary["format_version_ok"] = False
|
|
return summary
|
|
|
|
self._data_status = data
|
|
_write_cache(_status_cache_path(), data)
|
|
|
|
for ticker in data.get("assets_available", []):
|
|
for cat in TICKER_CATEGORIES:
|
|
result = self._sync_one(client, ticker, cat)
|
|
if result == "synced":
|
|
summary["synced"] = summary.get("synced", 0) + 1
|
|
|
|
# Try known global data too
|
|
for cat in KNOWN_GLOBAL:
|
|
self._sync_one(client, "GLOBAL", cat)
|
|
|
|
# Sync global datasets (fundamentals, macro, money flow, rates)
|
|
self.sync_global()
|
|
|
|
self._last_synced_at = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
|
return summary
|
|
|
|
# ---- Public: sync single ticker ----
|
|
|
|
def sync_ticker(self, ticker: str) -> dict:
|
|
# Quantum Terminal: no license client needed
|
|
client = None
|
|
|
|
result = {}
|
|
if ticker.upper() == "GLOBAL":
|
|
# Sync all known global categories
|
|
for cat in self._known_categories:
|
|
if cat not in TICKER_CATEGORIES:
|
|
result[cat] = self._sync_one(client, ticker, cat)
|
|
else:
|
|
for cat in TICKER_CATEGORIES:
|
|
result[cat] = self._sync_one(client, ticker, cat)
|
|
return result
|
|
|
|
# ---- Public: read data ----
|
|
|
|
def get_cones(self, ticker: str) -> Optional[dict]:
|
|
return self._get_data(ticker, "cones")
|
|
|
|
def get_bands(self, ticker: str) -> Optional[dict]:
|
|
return self._get_data(ticker, "bands")
|
|
|
|
def get_options(self, ticker: str) -> Optional[dict]:
|
|
return self._get_data(ticker, "options")
|
|
|
|
def get_probfield(self, ticker: str) -> Optional[dict]:
|
|
return self._get_data(ticker, "probfield")
|
|
|
|
def get_fundamentals(self) -> Optional[dict]:
|
|
"""Get global fundamental data (regime, liquidity, yields, COT, etc.)."""
|
|
return self._get_data("GLOBAL", "fundamental_state")
|
|
|
|
def get_signals(self) -> Optional[dict]:
|
|
"""Get daily signals data."""
|
|
return self._get_data("GLOBAL", "daily_signals")
|
|
|
|
def get_lifecycle(self) -> Optional[dict]:
|
|
"""Get signal lifecycle data."""
|
|
return self._get_data("GLOBAL", "signal_lifecycle")
|
|
|
|
def get_weekly_state(self) -> Optional[dict]:
|
|
"""Get weekly state data."""
|
|
return self._get_data("GLOBAL", "weekly_state")
|
|
|
|
def get_global(self, category: str) -> Optional[dict]:
|
|
"""
|
|
Get any global data by category name.
|
|
Works for any file synced from the server — no code changes needed.
|
|
Examples: get_global("central_bank_rates"), get_global("macro_flows")
|
|
"""
|
|
return self._get_data("GLOBAL", category)
|
|
|
|
# ── Global data getters ──────────────────────────────────────────────────────
|
|
|
|
def _fetch_global(self, key: str) -> dict | None:
|
|
"""
|
|
Fetch one global (GLOBAL ticker) dataset from the server.
|
|
Falls back to cache on any error. Returns None if unavailable.
|
|
"""
|
|
endpoint = GLOBAL_ENDPOINTS.get(key)
|
|
cache_file = _get_cache_dir() / GLOBAL_CACHE_NAMES.get(key, f"GLOBAL_{key}.json")
|
|
|
|
if not endpoint:
|
|
log.warning(f"[DataSync] Unknown global key: {key}")
|
|
return None
|
|
|
|
try:
|
|
jwt = self._get_jwt()
|
|
url = f"{self.base_url}{endpoint}"
|
|
resp = requests.get(
|
|
url,
|
|
headers={"Authorization": f"Bearer {jwt}"},
|
|
timeout=8, # v7: was 15 — consumer is display-only, fast fail beats long wait
|
|
)
|
|
|
|
if resp.status_code == 200:
|
|
data = resp.json()
|
|
# Write to cache
|
|
cache_file.parent.mkdir(parents=True, exist_ok=True)
|
|
cache_file.write_text(json.dumps(data), encoding="utf-8")
|
|
log.info(f"[DataSync] Global {key}: fetched and cached ({len(resp.content)} bytes)")
|
|
return data
|
|
|
|
elif resp.status_code == 304:
|
|
log.info(f"[DataSync] Global {key}: 304 Not Modified — using cache")
|
|
return self._load_global_cache(key)
|
|
|
|
elif resp.status_code == 404:
|
|
log.warning(f"[DataSync] Global {key}: 404 — not on server yet")
|
|
return self._load_global_cache(key)
|
|
|
|
else:
|
|
log.warning(f"[DataSync] Global {key}: HTTP {resp.status_code} — using cache")
|
|
return self._load_global_cache(key)
|
|
|
|
except Exception as e:
|
|
log.warning(f"[DataSync] Global {key}: fetch failed ({e}) — using cache")
|
|
return self._load_global_cache(key)
|
|
|
|
def _load_global_cache(self, key: str) -> dict | None:
|
|
"""Load global data from AppData cache. Returns None if not cached."""
|
|
cache_file = _get_cache_dir() / GLOBAL_CACHE_NAMES.get(key, f"GLOBAL_{key}.json")
|
|
if cache_file.exists():
|
|
try:
|
|
return json.loads(cache_file.read_text(encoding="utf-8"))
|
|
except Exception as e:
|
|
log.warning(f"[DataSync] Cache read failed for {key}: {e}")
|
|
return None
|
|
|
|
def sync_global(self) -> dict:
|
|
"""
|
|
Fetch all global datasets from server.
|
|
Called inside sync_all() — runs once at startup, not per-ticker.
|
|
Returns dict of {key: success_bool}.
|
|
"""
|
|
results = {}
|
|
for key in GLOBAL_ENDPOINTS:
|
|
data = self._fetch_global(key)
|
|
results[key] = data is not None
|
|
return results
|
|
|
|
def get_macro_sectors(self) -> dict | None:
|
|
"""Get macro sector rotation data."""
|
|
return self._load_global_cache("macro_sectors") or self._fetch_global("macro_sectors")
|
|
|
|
def get_macro_countries(self) -> dict | None:
|
|
"""Get macro country data."""
|
|
return self._load_global_cache("macro_countries") or self._fetch_global("macro_countries")
|
|
|
|
def get_macro_flows(self) -> dict | None:
|
|
"""Get country capital flow data (1W and 4W periods)."""
|
|
return self._load_global_cache("macro_flows") or self._fetch_global("macro_flows")
|
|
|
|
def get_macro_sector_flows(self) -> dict | None:
|
|
"""Get inter-sector money flow data (macro + weekly periods)."""
|
|
return self._load_global_cache("macro_sector_flows") or self._fetch_global("macro_sector_flows")
|
|
|
|
def get_central_bank_rates(self) -> dict | None:
|
|
"""Get central bank rate differential data."""
|
|
return self._load_global_cache("central_bank_rates") or self._fetch_global("central_bank_rates")
|
|
|
|
def _get_data(self, ticker: str, category: str) -> Optional[dict]:
|
|
canonical = ticker.upper()
|
|
store = self._get_store(category)
|
|
if canonical in store:
|
|
return store[canonical]
|
|
cached = _read_cache(_cache_path(canonical, category))
|
|
if cached:
|
|
store[canonical] = cached
|
|
return cached
|
|
|
|
def get_data_status(self) -> Optional[dict]:
|
|
if self._data_status:
|
|
return self._data_status
|
|
return _read_cache(_status_cache_path())
|
|
|
|
# ---- Public: staleness (Rule C4) ----
|
|
|
|
def get_staleness_info(self) -> List[StalenessInfo]:
|
|
results = []
|
|
for ticker, data in self._get_store("cones").items():
|
|
computed_at = data.get("computed_at")
|
|
results.append(_compute_staleness(computed_at, ticker))
|
|
return results
|
|
|
|
def get_overall_staleness(self) -> StalenessInfo:
|
|
infos = self.get_staleness_info()
|
|
# Use last_synced_at as the authoritative freshness signal
|
|
sync_time = getattr(self, "_last_synced_at", None) or getattr(self, "last_synced_at", None)
|
|
if sync_time:
|
|
return _compute_staleness(sync_time, "ALL")
|
|
if not infos:
|
|
return StalenessInfo(ticker="ALL", level=FreshnessLevel.FRESH)
|
|
worst = max(infos, key=lambda x: x.age_hours)
|
|
worst.ticker = "ALL"
|
|
return worst
|
|
|
|
# ---- Public: status flags ----
|
|
|
|
@property
|
|
def is_offline(self) -> bool:
|
|
return self._offline_mode
|
|
|
|
@property
|
|
def format_version_ok(self) -> bool:
|
|
return self._format_version_ok
|
|
|
|
@property
|
|
def sync_errors(self) -> List[str]:
|
|
return list(self._sync_errors)
|
|
|
|
@property
|
|
def is_syncing(self) -> bool:
|
|
"""True while sync_all is running — top-bar pill polls via
|
|
/api/consumer/status.sync_in_progress."""
|
|
return bool(getattr(self, "_sync_in_progress", False))
|
|
|
|
@property
|
|
def synced_tickers(self) -> List[str]:
|
|
return sorted(self._get_store("cones").keys())
|
|
|
|
@property
|
|
def last_synced_at(self) -> Optional[str]:
|
|
return self._last_synced_at
|
|
|
|
# ---- Internal: sync one file ----
|
|
|
|
def _sync_one(self, client, ticker: str, data_type: str) -> str:
|
|
canonical = ticker.upper()
|
|
endpoint = f"/data/{canonical}/{data_type}"
|
|
|
|
extra_headers = {}
|
|
local_key = _ts_key(data_type, canonical)
|
|
local_ts = self._sync_timestamps.get(local_key)
|
|
if local_ts:
|
|
try:
|
|
ts_dt = datetime.fromisoformat(local_ts)
|
|
extra_headers["If-Modified-Since"] = formatdate(
|
|
ts_dt.timestamp(), usegmt=True
|
|
)
|
|
except (ValueError, TypeError):
|
|
pass
|
|
|
|
data, status = client.auth_request(
|
|
"GET", endpoint, timeout=8, extra_headers=extra_headers # v7: was 30
|
|
)
|
|
|
|
if status == 304:
|
|
log.info(f"304 Not Modified: {canonical}/{data_type}")
|
|
return "not_modified"
|
|
|
|
# v7: retry once on 5xx with a SHORT sleep (was 10s). A genuine
|
|
# transient blip recovers in <1 s; a persistent 500 (e.g. a
|
|
# mis-served /probfield) would previously cost the user ~70 s per
|
|
# failure, stacking into multi-minute startup delays. Now worst
|
|
# case per failing endpoint is ~17 s (8 + 1 + 8).
|
|
if status >= 500:
|
|
log.warning(f"Server error {status} for {endpoint} -- retrying in 1s")
|
|
time.sleep(1)
|
|
data, status = client.auth_request(
|
|
"GET", endpoint, timeout=8, extra_headers=extra_headers
|
|
)
|
|
|
|
if status == 304:
|
|
return "not_modified"
|
|
|
|
if status == 200 and data is not None:
|
|
if not _check_format_version(data):
|
|
self._format_version_ok = False
|
|
return "failed"
|
|
|
|
self._get_store(data_type)[canonical] = data
|
|
_write_cache(_cache_path(canonical, data_type), data)
|
|
# v10: rough byte count for the toolbar speed readout. Uses the
|
|
# serialized JSON length — close enough to wire bytes for a UX
|
|
# indicator (no headers / compression accounted for).
|
|
try:
|
|
self._sync_bytes_total += len(json.dumps(data).encode("utf-8"))
|
|
except Exception:
|
|
pass
|
|
return "synced"
|
|
|
|
elif status == 404:
|
|
log.info(f"No {data_type} data for {canonical} (404)")
|
|
return "cached"
|
|
|
|
else:
|
|
log.warning(f"Failed to sync {data_type} for {canonical} (HTTP {status})")
|
|
self._sync_errors.append(f"{canonical}/{data_type}: HTTP {status}")
|
|
cached = _read_cache(_cache_path(canonical, data_type))
|
|
if cached:
|
|
self._get_store(data_type)[canonical] = cached
|
|
log.info(f"Loaded {data_type} for {canonical} from cache")
|
|
return "cached"
|
|
return "failed"
|
|
|
|
# ---- Internal: load from cache (offline) ----
|
|
|
|
def _load_all_from_cache(self, skip_existing: bool = False):
|
|
"""Load all cache files into in-memory stores via resolve_cache_filename().
|
|
|
|
If skip_existing=True, slots already populated (typically by a fresh
|
|
HTTP sync in the same tick) are preserved — only orphan/missing slots
|
|
get filled. Used by v5's end-of-sync_all sweep.
|
|
"""
|
|
cache_dir = _get_cache_dir()
|
|
if not cache_dir.exists():
|
|
return
|
|
|
|
self._data_status = _read_cache(_status_cache_path())
|
|
|
|
total = 0
|
|
for f in cache_dir.glob("*.json"):
|
|
if f.name.startswith("_"):
|
|
continue
|
|
if not f.name.lower().endswith(".json"):
|
|
continue
|
|
stem = f.name[:-5]
|
|
resolved = resolve_cache_filename(stem)
|
|
if not resolved:
|
|
continue
|
|
ticker, category = resolved
|
|
if skip_existing and ticker in self._get_store(category):
|
|
continue
|
|
data = _read_cache(f)
|
|
if data:
|
|
self._get_store(category)[ticker] = data
|
|
self._known_categories.add(category)
|
|
total += 1
|
|
|
|
if total:
|
|
log.info(f"Loaded {total} data files from cache (offline mode)")
|
|
|
|
# ---- Public: clear cache (Rule C9 — reversible via _trash) ----
|
|
|
|
def clear_cache(self) -> dict:
|
|
"""
|
|
Move all cached data files to a timestamped trash folder.
|
|
Resets in-memory stores and sync timestamps so the next sync_all()
|
|
pulls everything fresh from the server.
|
|
|
|
Files are NEVER deleted — they are preserved in
|
|
%APPDATA%\\QuantumTerminal\\_trash\\cache_<timestamp>\\ for recovery
|
|
per Rule C9. Operator can manually purge _trash/ when confident.
|
|
|
|
Returns:
|
|
{"moved": int, "trash_dir": str | None}
|
|
"""
|
|
cache_dir = _get_cache_dir()
|
|
if not cache_dir.exists():
|
|
return {"moved": 0, "trash_dir": None}
|
|
|
|
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
|
|
trash_dir = cache_dir.parent / "_trash" / f"cache_{ts}"
|
|
try:
|
|
trash_dir.mkdir(parents=True, exist_ok=True)
|
|
except Exception as e:
|
|
log.error(f"Failed to create trash dir {trash_dir}: {e}")
|
|
return {"moved": 0, "trash_dir": None}
|
|
|
|
moved = 0
|
|
for f in cache_dir.glob("*.json"):
|
|
try:
|
|
f.replace(trash_dir / f.name)
|
|
moved += 1
|
|
except Exception as e:
|
|
log.warning(f"Failed to move {f.name} to trash: {e}")
|
|
|
|
# Reset in-memory state so next sync_all is a clean pull
|
|
self._stores.clear()
|
|
self._sync_timestamps.clear()
|
|
try:
|
|
_save_sync_timestamps(self._sync_timestamps)
|
|
except Exception:
|
|
pass
|
|
self._data_status = None
|
|
self._sync_errors = []
|
|
self._offline_mode = False
|
|
self._last_synced_at = None
|
|
|
|
log.info(f"Cleared cache: {moved} files moved to {trash_dir}")
|
|
return {"moved": moved, "trash_dir": str(trash_dir)}
|
|
|
|
|
|
# ============================================================
|
|
# 9. SINGLETON + REST API ROUTES
|
|
# ============================================================
|
|
|
|
_sync_client: Optional[DataSyncClient] = None
|
|
|
|
|
|
def get_sync_client() -> DataSyncClient:
|
|
global _sync_client
|
|
if _sync_client is None:
|
|
_sync_client = DataSyncClient()
|
|
return _sync_client
|
|
|
|
|
|
def create_data_sync_routes():
|
|
from fastapi import APIRouter
|
|
|
|
router = APIRouter(tags=["data_sync"])
|
|
|
|
@router.get("/api/sync/status")
|
|
async def api_sync_status():
|
|
client = get_sync_client()
|
|
staleness = client.get_overall_staleness()
|
|
# v9: surface in-progress flag + progress counters so the top-toolbar
|
|
# download bar can render `done / total` and the post-sync "please
|
|
# refresh" prompt can fire on transition true → false.
|
|
return {
|
|
"offline_mode": client.is_offline,
|
|
"format_version_ok": client.format_version_ok,
|
|
"synced_tickers": client.synced_tickers,
|
|
"sync_errors": client.sync_errors,
|
|
"analysis_date": staleness.display_date,
|
|
"staleness_level": staleness.level.value,
|
|
"staleness_hours": staleness.age_hours,
|
|
"last_synced_at": client.last_synced_at,
|
|
"sync_in_progress": bool(getattr(client, "_sync_in_progress", False)),
|
|
"sync_total": int(getattr(client, "_sync_total", 0)),
|
|
"sync_done": int(getattr(client, "_sync_done", 0)),
|
|
"sync_bytes_total": int(getattr(client, "_sync_bytes_total", 0)), # v10
|
|
}
|
|
|
|
@router.get("/api/sync/cones/{ticker}")
|
|
async def api_get_cones(ticker: str):
|
|
client = get_sync_client()
|
|
data = client.get_cones(ticker.upper())
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, f"No cone data for {ticker}")
|
|
return data
|
|
|
|
@router.get("/api/sync/options/{ticker}")
|
|
async def api_get_options(ticker: str):
|
|
client = get_sync_client()
|
|
data = client.get_options(ticker.upper())
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, f"No options data for {ticker}")
|
|
return data
|
|
|
|
@router.get("/api/sync/probfield/{ticker}")
|
|
async def api_get_probfield(ticker: str):
|
|
client = get_sync_client()
|
|
data = client.get_probfield(ticker.upper())
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, f"No probfield data for {ticker}")
|
|
return data
|
|
|
|
@router.get("/api/sync/fundamentals")
|
|
async def api_get_fundamentals():
|
|
client = get_sync_client()
|
|
data = client.get_fundamentals()
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, "No fundamentals data")
|
|
return data
|
|
|
|
@router.get("/api/sync/signals")
|
|
async def api_get_signals():
|
|
client = get_sync_client()
|
|
data = client.get_signals()
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, "No signals data")
|
|
return data
|
|
|
|
@router.get("/api/sync/lifecycle")
|
|
async def api_get_lifecycle():
|
|
client = get_sync_client()
|
|
data = client.get_lifecycle()
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, "No lifecycle data")
|
|
return data
|
|
|
|
@router.get("/api/sync/weeklystate")
|
|
async def api_get_weekly_state():
|
|
client = get_sync_client()
|
|
data = client.get_weekly_state()
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(404, "No weekly state data")
|
|
return data
|
|
|
|
@router.get("/api/sync/staleness")
|
|
async def api_staleness():
|
|
client = get_sync_client()
|
|
infos = client.get_staleness_info()
|
|
overall = client.get_overall_staleness()
|
|
return {
|
|
"overall": asdict(overall),
|
|
"per_ticker": [asdict(i) for i in infos],
|
|
}
|
|
|
|
@router.post("/api/sync/refresh")
|
|
async def api_refresh_sync():
|
|
# v8: run on the worker thread pool so the event loop is free to
|
|
# serve concurrent requests (notably /api/sync/status, which the
|
|
# LoginScreen polls to detect format-version mismatch). Without
|
|
# this, sync_all blocks ALL other endpoints for ~5 min on a
|
|
# first-install full pull.
|
|
client = get_sync_client()
|
|
summary = await asyncio.to_thread(client.sync_all)
|
|
return summary
|
|
|
|
@router.post("/api/sync/refresh/{ticker}")
|
|
async def api_refresh_ticker(ticker: str):
|
|
# v8: same off-loop reasoning as /api/sync/refresh.
|
|
client = get_sync_client()
|
|
result = await asyncio.to_thread(client.sync_ticker, ticker.upper())
|
|
return result
|
|
|
|
@router.post("/api/sync/clear-cache")
|
|
async def api_clear_cache():
|
|
"""
|
|
Move cache to _trash and re-sync from server.
|
|
Rule C9: files are preserved in _trash/, never deleted.
|
|
"""
|
|
# v8: both clear_cache and sync_all are blocking I/O — keep them
|
|
# off the event loop so /api/sync/status keeps responding.
|
|
client = get_sync_client()
|
|
clear_result = await asyncio.to_thread(client.clear_cache)
|
|
sync_summary = await asyncio.to_thread(client.sync_all)
|
|
return {
|
|
"cleared": clear_result,
|
|
"resync": sync_summary,
|
|
}
|
|
|
|
@router.get("/api/sync/poll-config")
|
|
async def api_poll_config():
|
|
"""Returns the current periodic sync interval (for UI display)."""
|
|
try:
|
|
from periodic_sync import _read_interval
|
|
return {"interval_seconds": _read_interval()}
|
|
except Exception:
|
|
return {"interval_seconds": 300}
|
|
|
|
@router.get("/api/data/global/{key}")
|
|
async def get_global_data(key: str):
|
|
"""Proxy global data from local cache to React frontend."""
|
|
valid_keys = set(GLOBAL_ENDPOINTS.keys())
|
|
if key not in valid_keys:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(status_code=404, detail=f"Unknown global key: {key}")
|
|
client = get_sync_client()
|
|
data = client._load_global_cache(key)
|
|
if data is None:
|
|
# Try live fetch as fallback
|
|
data = client._fetch_global(key)
|
|
if data is None:
|
|
from fastapi import HTTPException
|
|
raise HTTPException(status_code=503, detail=f"{key} not available yet")
|
|
return data
|
|
|
|
return router |