Dashboard changes:
- SSE stream now emits per-source events as each API completes instead of
waiting for all 47 sources to finish before rendering anything
- Map and UI appear within seconds; each data layer populates progressively
- Loading indicator shows feed progress (LOADING FEEDS 12/47)
- Spinner hidden after first 3 sources arrive
Backend (app.py):
- Replaced monolithic _fetch_overview gather with asyncio.Queue-based
progressive emission — each source fires as a task, results stream
to the client as they complete
- Fixed Starlette 1.0 incompatibility: replaced removed on_startup kwarg
with asynccontextmanager lifespan pattern
- Added proper vector store shutdown in lifespan teardown
Frontend (index.html):
- SSE handler now accumulates partial events into _accumulatedData
- Each incoming source merges and triggers updateAll() progressively
- Backwards compatible with legacy full-payload mode
- Exponential backoff: retry waits are now 1s, 2s, 4s (was linear 1s, 2s, 3s)
- Cache poisoning prevention: validate Content-Type is not HTML before JSON parse,
reject non-dict/list responses before caching
- Add 18 live smoke tests against real public APIs (USGS, CoinGecko, Yahoo Finance,
SEC EDGAR, FAA, NOAA SWPC, Hacker News, NASA EONET, etc.)
- Smoke tests excluded from default pytest run; opt-in with: pytest -m smoke
- All 220 unit tests + 18 smoke tests pass
- intel_cross_correlate: finds related signals across all domains for a topic,
groups by category, shows how events ripple across intelligence streams
- intel_domain_summary: per-category aggregate of stored intelligence with
data point counts, unique sources, latest/earliest timestamps
- intel_trend_detection: compares recent vs baseline activity rates per
category, identifies SURGE/ELEVATED/DECLINING/DROP patterns for early warning
- All three leverage accumulated Qdrant vector store data
- 109 tools total across 30+ domains
Phase 16: Vector intelligence (+3 tools = 104 total)
New tools:
- intel_semantic_search: Natural language search across all stored intelligence
- intel_similar_events: Find historically similar events for pattern matching
- intel_timeline: Chronological timeline of stored data with domain filtering
Architecture:
- vector_store.py: Qdrant client with FastEmbed (ONNX, bge-small-en-v1.5)
- Background worker queue (asyncio) for non-blocking storage
- Auto-populates from every Fetcher.get_json() call
- 40+ domain categories mapped for filtered search
- Graceful degradation: vector store is optional, all 101 existing tools work without it
Integration:
- Fetcher: new vector_store parameter, stores data after each successful fetch
- server.py: vector store init + worker lifecycle + 3 dispatch cases
- dashboard/app.py: vector store worker starts on dashboard boot
- intel_status: now includes vector store statistics
- pyproject.toml: [vector] optional dependency (qdrant-client, fastembed)
186/186 tests passing.
When both ACLED and UCDP fail, the dashboard falls back to static
INTEL_HOTSPOTS data. Previously the fallback events had empty fields
(country, location, actors, date, source, notes). Now maps ISO codes
to country names via TIER1_COUNTRIES, populates all fields the
frontend modal expects, and includes escalation severity labels.
New tools:
- intel_btc_technicals: SMA-50/200, Mayer Multiple, golden/death cross, ATH distance
- intel_central_bank_rates: 15 banks (Fed/ECB/BoE via FRED, 12 curated fallback)
- intel_trade_routes: 19 maritime chokepoints with oil flow and vessel transit data
- intel_cloud_regions: 28 AWS/Azure/GCP regions with coordinates
- intel_financial_centers: GFCI top 20 with rankings and specializations
Dashboard additions:
- BTC Technicals drawer section (price, SMA, Mayer, cross signal)
- Central Bank Rates drawer section (15 banks, sorted by rate)
- Trade route markers on infrastructure map layer (chokepoint/canal/route icons)
- Regional presets (Globe/Americas/Europe/MENA/Asia-Pac/Africa) with localStorage persistence
- Time window filter (1h/6h/24h/7d/All) filtering earthquakes and news by timestamp
- Static datasets served via /api/static for instant boot
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Captured via Playwright with live SSE data showing all 14 map layers,
HUD stats, alert banner, drawer with posture/markets/crypto, and
the full layer panel including new Climate and News layers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Client-side geolocation scans article titles/summaries for country
mentions and plots them at country centroids with jitter. Integrated
into layer toggles, search, HUD click-to-zoom, and detail panel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_syncLayersTop() now measures both topbar height and alert banner,
positioning layers panel below both. Alert banner also uses
--layers-top CSS var so it tracks topbar height changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Climate map layer: 15 zones with temp-anomaly colored markers (hot/warm/cool/cold)
with significance animation and full detail panel
- Topbar: flex-wrap with max-height cap on HUD pills, dynamic --layers-top CSS var
ensures layer panel never overlaps topbar regardless of row count
- Mobile: collapsible layer panel (tap title to expand/collapse)
- Search and HUD pill click-to-zoom extended for climate layer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Search: type country/city/keyword to zoom map to matching markers
- ISO3→name mapping for 40+ countries (search "ukraine" matches UKR)
- Searches all data sources: earthquakes, military, conflict, fires, webcams, bases, etc.
- / keyboard shortcut to focus search, Esc to clear
- Visual feedback: cyan border on match, red on no match
- Layer persistence: toggle states saved to localStorage, restored on reload
- Drawer section collapse: click any section header to collapse/expand
- Collapsed state persists across SSE updates and page reloads via localStorage
- 19 collapsible sections with rotate indicator
- HUD pill click-to-zoom: click any stat pill to zoom map to that layer's markers
- Auto-enables layer if toggled off
- Webcam auto-refresh: preview image refreshes every 15s while detail panel is open
- Interval properly cleared on panel close (no memory leak)
- Webcam drawer items now clickable (opens detail panel with preview)
- Loading state: spinner + "LOADING LIVE FEEDS" shown until first SSE data arrives
- Fixed keyboard shortcuts: D toggle now properly uses drawer 'closed' class
- Fixed drawer event delegation: attached once at boot instead of re-adding every 30s SSE update
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Marker clustering (leaflet.markercluster) for air traffic, military,
conflict, wildfires, and nav warnings — clusters expand on click
- Cable corridor rectangles set to interactive:false so clicks pass
through to markers underneath (fixes blocked clicks in Med/Middle East)
- Batch addLayers() for air traffic performance with 9500+ markers
- Keyboard shortcuts: Esc closes panels, D toggles drawer, R resets view
- Custom dark-theme cluster styles with per-layer color coding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Webcam clicks now show the live camera feed image from Windy API
- Added img to DOMPurify allowed tags for preview rendering
- Road Traffic layer dot → white (#e0e0e0), Webcams → grey (#909090)
- All 12 layers now have visually distinct indicator colors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map marker clicks were showing headers but empty bodies because the type
handlers were only in showDrawerDetail, not the primary showDetail function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Air Traffic: sample every 10th aircraft from OpenSky for ~950 map markers
- Nav Warnings: parse DD-MM.mmN DDD-MM.mmW coordinates from NGA warning text
- Road Traffic: TomTom city congestion circles (when API key set)
- Webcams: Windy camera positions (when API key set)
- 12 toggleable layers total (up from 8)
- Detail panels for all new marker types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four new intelligence domains for the dashboard:
1. Domestic flights (OpenSky) — global airborne aircraft count by
region with commercial/general breakdown. No API key needed.
2. Road traffic (TomTom) — real-time congestion % for 20 major world
cities + traffic incidents in 5 strategic regions. Needs
TOMTOM_API_KEY (free 2500 req/day at developer.tomtom.com).
3. CCTV webcams (Windy) — public traffic camera locations worldwide.
Needs WINDY_API_KEY (free 100 req/day at api.windy.com).
4. AI situation brief (Ollama) — LLM-generated 3-paragraph
intelligence brief synthesizing all dashboard data. Uses local
Ollama (llama3.2). Falls back to structured metrics summary.
New files: sources/traffic.py, sources/webcams.py, analysis/situation.py
Modified: sources/aviation.py (+fetch_domestic_flights), dashboard/app.py,
dashboard/index.html (drawer sections + HUD pills for all 4 domains).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UCDP now requires x-ucdp-access-token header (previously open access).
Updated from v24.1 to v25.1 (latest stable). Token passed via
UCDP_ACCESS_TOKEN env var; gracefully degrades to 0 events without it
(conflict zone fallback uses static INTEL_HOTSPOTS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ACLED retired api.acleddata.com and moved to acleddata.com/api/ with
OAuth2 password-grant authentication. New flow:
- POST to /oauth/token with email+password → Bearer token (24h TTL)
- Automatic refresh via refresh_token (14-day TTL)
- Module-level token cache with async lock
Shared acled_query() helper in conflict.py used by all 9 call sites
in intelligence.py. Env vars: ACLED_EMAIL + ACLED_PASSWORD (replaces
ACLED_ACCESS_TOKEN).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Matches the sidebar fix — computes cluster count from fires_by_region
top_clusters rather than displaying total_fires (raw FIRMS detections).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wildfire sidebar counter now shows rendered cluster markers (180) instead
of raw FIRMS detections (74,901); raw total preserved in tooltip.
Transpacific cable corridor rectangle split into two halves at the
antimeridian so it renders over the Pacific instead of across landmasses.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tested all 80+ RSS feeds — 22 returned 403/404/timeout. Changes:
- AP Top News: rsshub → feedx.net (working wire proxy)
- Reuters World: reutersagency.com → Google News RSS filter
- Defense One: /rss/ → /rss/all/ (correct endpoint)
- Military Times replaces Stars and Stripes (404, no RSS found)
- The National UAE: /rss → Arc outbound feed (working)
- Nikkei Asia: /rss → /rss/feed/nar (working)
- Lowy Interpreter replaces East Asia Forum (403 WAF block)
- Dialogo Americas replaces Americas Quarterly + Brazil Wire
- Remove 13 dead feeds with no working replacement:
Threatpost, CSIS, CFR, Chatham House, Atlantic Council, IISS,
Al Monitor, ACAPS, EU External Action, NATO News,
Energy Intelligence, The Africa Report, African Arguments
Reduces feed count from 80 to 62, eliminating wasted timeout cycles
that were blocking the dashboard SSE first frame.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add /api/static endpoint for instant geospatial data on boot (158 items)
- Add per-coroutine 45s timeout to prevent SSE first-frame blocking
- Dismiss loading overlay after static fetch instead of waiting for SSE
- Render 24 oil/gas/hydrogen pipelines as colored polylines on map
- Fix theaters dict-vs-list bug in posture.py and fleet.py
- Add exposure detail modal and color/label entries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Population exposure was showing 0 because exposure.py parsed wildfire
data using wrong keys ("regions" instead of "fires_by_region", "detections"
instead of "top_clusters"). Now correctly matches NASA FIRMS response format.
Dashboard updateAll() now wraps each map/UI update in try/catch to prevent
a single failed layer from blocking all subsequent layer renders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Strategic Posture section (top of drawer): composite score, 9 domain
grid with color-coded scores, top 5 threats
- Fleet Report section: readiness score/level, waterway status table,
aircraft count, active surges
- Population Exposure map layer: circle markers sized by population,
colored by threat type (red=conflict, orange=quake, yellow=fire)
- Population Exposure drawer section: total exposed, by event type,
city table with distance
- HUD pills: Posture score + Exposed population
- Layer toggle for Pop Exposure (purple, default on)
- Backend: exposure API now includes lat/lon for map rendering
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 10: News Intelligence & NLP
- intel_extract_entities: Regex-based NER for countries, leaders, orgs,
companies, CVEs, APT groups (28 leaders, 41 orgs, 25 companies, 36 APTs)
- intel_classify_event: Keyword-based threat classification into 14
categories with severity scoring (1-10)
- intel_news_clusters: Jaccard similarity topic clustering for news
articles with keyword extraction
- intel_keyword_spikes: Welford's algorithm baseline comparison with
CVE/APT pattern extraction from headlines
No new dependencies — all pure Python with regex and SQLite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Iframes blocked by most sites (X-Frame-Options/CSP). External links
now open in a new tab via window.open with noopener,noreferrer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Marc Shade