Commit Graph

32 Commits

Author SHA1 Message Date
Marc Shade 71fbb1cda4 feat: waterway diamonds, cable corridor zones, cache resilience
- Render 9 strategic waterway chokepoints as cyan diamond markers
  (Hormuz, Malacca, Suez, Panama, Bab-el-Mandeb, Taiwan, Gibraltar,
  GIUK Gap, Bosphorus) with throughput tooltips
- Render 6 submarine cable corridors as dashed blue rectangles
  (transatlantic N/S, transpacific, asia-europe, red sea, med)
- Add waterways + cable_corridors to /api/static and SSE payloads
- Make cache.set() and evict_expired() swallow sqlite3 write errors
  instead of crashing callers (fixes "readonly database" in dashboard)
- Infrastructure layer: 134 → 173 items

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 10:37:03 -05:00
Marc Shade 95b149ad55 fix: replace 22 dead/blocked RSS feeds with working alternatives
Tested all 80+ RSS feeds — 22 returned 403/404/timeout. Changes:
- AP Top News: rsshub → feedx.net (working wire proxy)
- Reuters World: reutersagency.com → Google News RSS filter
- Defense One: /rss/ → /rss/all/ (correct endpoint)
- Military Times replaces Stars and Stripes (404, no RSS found)
- The National UAE: /rss → Arc outbound feed (working)
- Nikkei Asia: /rss → /rss/feed/nar (working)
- Lowy Interpreter replaces East Asia Forum (403 WAF block)
- Dialogo Americas replaces Americas Quarterly + Brazil Wire
- Remove 13 dead feeds with no working replacement:
  Threatpost, CSIS, CFR, Chatham House, Atlantic Council, IISS,
  Al Monitor, ACAPS, EU External Action, NATO News,
  Energy Intelligence, The Africa Report, African Arguments

Reduces feed count from 80 to 62, eliminating wasted timeout cycles
that were blocking the dashboard SSE first frame.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 10:26:47 -05:00
Marc Shade a72fdc50f7 fix: dashboard instant boot, pipeline map layer, theater data bugs
- Add /api/static endpoint for instant geospatial data on boot (158 items)
- Add per-coroutine 45s timeout to prevent SSE first-frame blocking
- Dismiss loading overlay after static fetch instead of waiting for SSE
- Render 24 oil/gas/hydrogen pipelines as colored polylines on map
- Fix theaters dict-vs-list bug in posture.py and fleet.py
- Add exposure detail modal and color/label entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 10:09:14 -05:00
Marc Shade 2d1c95389a fix: wildfire data parsing + dashboard cascade failure protection
Population exposure was showing 0 because exposure.py parsed wildfire
data using wrong keys ("regions" instead of "fires_by_region", "detections"
instead of "top_clusters"). Now correctly matches NASA FIRMS response format.

Dashboard updateAll() now wraps each map/UI update in try/catch to prevent
a single failed layer from blocking all subsequent layer renders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:36:29 -05:00
Marc Shade de2cbeaa1a feat: add strategic synthesis to dashboard map + drawer
- Strategic Posture section (top of drawer): composite score, 9 domain
  grid with color-coded scores, top 5 threats
- Fleet Report section: readiness score/level, waterway status table,
  aircraft count, active surges
- Population Exposure map layer: circle markers sized by population,
  colored by threat type (red=conflict, orange=quake, yellow=fire)
- Population Exposure drawer section: total exposed, by event type,
  city table with distance
- HUD pills: Posture score + Exposed population
- Layer toggle for Pop Exposure (purple, default on)
- Backend: exposure API now includes lat/lon for map rendering

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:21:03 -05:00
Marc Shade 2969dfc950 feat: add strategic synthesis layer — Phase 11 (+4 = 68 tools)
- intel_strategic_posture: composite risk from 9 weighted domains
  (military, political, conflict, infrastructure, economic, cyber,
  health, climate, space) with per-domain scoring and top threats
- intel_world_brief: structured daily intelligence summary aggregating
  posture, focal points, news clusters, anomalies, trending threats
- intel_fleet_report: naval fleet activity combining theater posture,
  waterway status, military surge, and readiness scoring
- intel_population_exposure: population at risk near active events
  (earthquakes, wildfires, conflict) using 105-city dataset (1B pop)

New files: analysis/posture.py, analysis/world_brief.py,
analysis/exposure.py, sources/fleet.py, config/population.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:02:14 -05:00
Marc Shade 7359c03275 docs: update all docs for 64 tools, Phase 9 NLP complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 08:44:28 -05:00
Marc Shade 69f1a6d44f feat: NLP intelligence — entity extraction, classification, clustering, spikes (+4 = 64 tools)
Phase 10: News Intelligence & NLP
- intel_extract_entities: Regex-based NER for countries, leaders, orgs,
  companies, CVEs, APT groups (28 leaders, 41 orgs, 25 companies, 36 APTs)
- intel_classify_event: Keyword-based threat classification into 14
  categories with severity scoring (1-10)
- intel_news_clusters: Jaccard similarity topic clustering for news
  articles with keyword extraction
- intel_keyword_spikes: Welford's algorithm baseline comparison with
  CVE/APT pattern extraction from headlines

No new dependencies — all pure Python with regex and SQLite.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 08:41:40 -05:00
Marc Shade 2aa5fd1425 fix: replace iframe modal with window.open for external links
Iframes blocked by most sites (X-Frame-Options/CSP). External links
now open in a new tab via window.open with noopener,noreferrer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 08:17:54 -05:00
Marc Shade 084a386645 docs: update README, pyproject for 60 tools across 25 domains
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 08:05:58 -05:00
Marc Shade 6065eabb82 docs: update ROADMAP to reflect 60 tools and completed phases
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 07:46:47 -05:00
Marc Shade 11fc331ba4 feat: geospatial datasets — 70 bases, 40 ports, 24 pipelines, 24 nuclear (+4 = 60 tools)
Static geospatial intelligence datasets with map visualization:
- 70 military bases from 9 operators (USA, RUS, CHN, GBR, FRA, NATO, etc.)
- 40 strategic ports (container, oil, LNG, naval, bulk)
- 24 oil/gas/hydrogen pipelines with capacity and status
- 24 nuclear facilities (power, enrichment, research, reprocessing)
- 4 new MCP tools: intel_military_bases, intel_strategic_ports,
  intel_pipelines, intel_nuclear_facilities
- New "Infrastructure" map layer with base/port/facility markers
- Pipeline summary table in drawer with click-to-detail
- All geospatial items have detail modals on click

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 07:42:08 -05:00
Marc Shade a3dabb2c9f feat: modal detail system, 80+ RSS feeds, service status tool (#56)
- All external links open in iframe modal instead of navigating away
- 9 drawer item types (cyber, health, AI, social, elections, etc.)
  expand into rich detail panel on click
- RSS feeds expanded from ~20 to ~80 across 14 categories with
  source tier reliability scoring
- New intel_service_status tool monitoring AWS/Azure/GCP/CF/GitHub
- Cloud service incidents rendered in dashboard Security section
- Zero target=_blank links remain in dashboard

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 07:29:31 -05:00
Marc Shade 2d33ea136a feat: IODA fallback, UCDP timeout fix, PDF export, 18 tests
- infrastructure.py: Add IODA (Georgia Tech) as fallback when Cloudflare
  Radar returns 403 (no API token). Internet outage data now available
  without CLOUDFLARE_API_TOKEN.
- conflict.py: Increase UCDP GED API timeout from 15s to 30s to handle
  slow responses from ucdpapi.pcr.uu.se.
- dashboard/app.py: Add /api/report/pdf endpoint for PDF daily brief
  export via weasyprint (optional dependency).
- pyproject.toml: Add dashboard + pdf optional deps, intel-dashboard
  script entry point, update description for 55 tools.
- tests: Add 11 new tests covering health, sanctions, elections, shipping,
  social, nuclear, infrastructure (IODA fallback + cable health), and
  UCDP conflict. Fix cross-loop asyncio lock issue in conftest.
- Total: 50 tests passing, 55 tools verified.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 07:05:58 -05:00
Marc Shade 6fee3feae5 fix: dashboard layout — alerts clear layers, zoom above ticker
Alert banner moved right (left: 220px) to clear layers panel,
reduced strip size (0.58rem, 3px padding), text-overflow ellipsis.
Zoom controls pushed up 50px margin-bottom to clear bottom ticker.
Responsive rule for alert banner on small screens.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 05:40:44 -05:00
Marc Shade dd3c4f6951 fix: conflict zones always visible via INTEL_HOTSPOTS fallback
When both ACLED (no API key) and UCDP (timeout/empty) fail, the
dashboard now falls back to 22 pre-coded conflict hotspots from
INTEL_HOTSPOTS with severity levels (critical/high/moderate/low).
Frontend updated with 3-tier fallback: ACLED → UCDP → conflict_zones.
Marker sizing uses escalation level for hotspots, tooltips show severity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:25:05 -05:00
Marc Shade 2d46b064dd feat: add adsb.lol as primary military aircraft source
OpenSky anonymous access returns 429 consistently. Added adsb.lol
free community API (/v2/mil endpoint) as primary source with OpenSky
as fallback.

- _fetch_adsblol_military: dedicated military endpoint, no filtering needed
- _fetch_opensky_military: original OpenSky logic as fallback
- _icao_to_country: derives country from ICAO hex prefix (US, UK, FR, etc.)
- adsblol rate limit: 5s between calls (community API, be polite)
- Cache TTL increased to 300s for both sources
- bbox filtering applied client-side for adsb.lol data

Result: 127 military aircraft from multiple nations now visible on map.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:23:11 -05:00
Marc Shade 6c2159d06d fix: conflict map uses UCDP as fallback when ACLED unavailable
- updateMapConflict supports both ACLED and UCDP data fields
  (fatalities/best, event_type/type_of_violence_label)
- updateAll passes UCDP events when ACLED errors (no API key)
- HUD conflict pill also falls back to UCDP source
- Increased conflict marker limit from 100 to 200

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:03:17 -05:00
Marc Shade e47f0a70b2 fix: stale-data fallback, per-source rate limits, broken feeds
Cache: add get_stale() for last-known-good fallback when APIs fail.
Fetcher: on API failure or tripped circuit breaker, serve stale cached
data instead of returning None — dashboards never go blank after first
successful fetch. Add per-source rate limits (11 sources configured).

Fix Yahoo Finance: compute change_pct from previousClose when
regularMarketChangePercent is absent (Yahoo v8 API change).
Fix health RSS: replace dead WHO DON/ProMED/CIDRAP URLs with working
WHO news, CDC outbreaks, and Outbreak News Today feeds.
Fix wildfires: extend FIRMS query from 1-day to 2-day (NRT data lag),
include nominal-confidence fires (was filtering to high-only).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 20:34:36 -05:00
Marc Shade 0fdd0814d7 feat: wire Phase 7 feeds to dashboard — 7 new visualizations + nuclear map layer
Backend: add 7 new data feeds to _fetch_overview() (33 total SSE streams).
Frontend: alert banner (critical/high/medium), health outbreak tracker,
election calendar with risk bars, shipping stress gauge, social signals
from Reddit, nuclear test site map layer with concern scoring, weekly
trends with anomaly tracking. New HUD pills for alerts, shipping stress,
outbreaks, nuclear flags.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 20:18:09 -05:00
Marc Shade d5bf09f342 feat: add health, sanctions, elections, shipping, social, nuclear intelligence — Phase 7 (+10 = 55 tools)
New source modules: disease outbreaks (WHO/ProMED/CIDRAP), OFAC sanctions
search, election calendar with risk scoring, shipping stress index,
Reddit social signals, nuclear test site seismic monitor. Cross-domain
alert digest and weekly trend analysis. Registers orphaned space_weather
and ai_watch modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 20:06:05 -05:00
Marc Shade 9f7a81812b feat: add military & infrastructure intelligence — Phase 6 (+6 = 45 tools)
New analysis modules: surge detection (8 sensitive regions), cascade
simulation (6 cable corridors), hotspot escalation scoring (22 hotspots).
New tools: commodity quotes, unrest events, hotspot escalation, military
surge, vessel snapshot, cascade analysis.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 19:04:49 -05:00
Marc Shade 8d60a9a220 feat: add core analysis engine — focal points, temporal anomalies, CII v2
Phase 5 of world-intel-mcp: intelligence synthesis layer that makes raw
data actionable.

New modules:
- config/countries.py: 22 tier-1 countries with event multipliers, 22
  intel hotspots, 9 strategic waterways, 6 conflict zones
- analysis/focal_points.py: detect convergence of signals on entities
  using signal count, type diversity, and recency weighting
- analysis/temporal.py: Welford's algorithm + SQLite for streaming
  anomaly detection with seasonal baselines (weekday+month)

Upgraded modules:
- analysis/instability.py: CII v2 with 4 weighted domains (unrest 0.25,
  conflict 0.30, security 0.20, information 0.25), UCDP floors, event
  multipliers. Full v1 backward compat preserved.
- analysis/signals.py: v2 aggregator accepts 7 data sources (added
  outages, military, protests) with convergence scoring
- sources/intelligence.py: 3 new fetch functions (focal_points,
  signal_summary, temporal_anomalies), instability upgraded to CII v2

3 new MCP tools: intel_focal_points, intel_signal_summary,
intel_temporal_anomalies. Total: 39 tools.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:39:57 -05:00
Marc Shade 561ed8e277 feat: add space weather + AI watch feeds, fix data binding issues
- Add space_weather source (NOAA SWPC: Kp index, X-ray flux, alerts)
- Add ai_watch source (arXiv cs.AI/LG/CL, HuggingFace, lab trending)
- Fix RSS circuit breaker: per-feed source names prevent cascade trips
- Fix displacement field: internally_displaced (was reading idps)
- Fix energy prices: handle nested oil.brent/wti structure
- Fix climate anomalies: use temp_anomaly_c and precip_anomaly_pct
- Fix news ticker: add feed_name to source fallback chain
- Remove defunct Reuters RSS feed, update War Zone URL
- Disable HTML caching for dev-friendly reloads

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:28:38 -05:00
Marc Shade affaf4b200 feat: redesign intelligence dashboard as map-first ops center
Complete v3 rewrite: full-viewport Leaflet map as primary interface
with glassmorphic floating panels. All geolocated events (earthquakes,
military aircraft, ACLED conflict, wildfire clusters, signal convergence)
are clickable map markers with detailed slide-in modals.

Design: Chakra Petch + Share Tech Mono typography, vignette overlay,
CSS custom properties, animated markers with pulse/glow on critical
events. Data drawer with sticky section headers for non-geospatial
data (markets, crypto, cyber, predictions, displacement). Scrolling
news ticker, HUD stat pills, layer toggle switches.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:08:09 -05:00
Marc Shade 15ecc8e3e1 feat: expand intelligence dashboard with 7 new feeds, tabbed UI, map, and charts
Backend: wire stablecoin_status, etf_flows, acled_events, ucdp_events,
displacement, risk_scores, and signal_convergence into the SSE overview
endpoint (23 total feeds, up from 16).

Frontend: complete rewrite with 7-tab navigation (Overview, Markets,
Security, Conflict & Humanitarian, Natural & Climate, Geospatial Map,
Trend Charts), Leaflet dark-tile map with toggleable layers, Chart.js
rolling time-series, feed health counter, and DOMPurify sanitization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:49:11 -05:00
Marc Shade 5af243d7a5 fix: handle object-shaped climate anomaly data in dashboard
Climate API may return zones as {zone_name: data} object instead of
array. Added Object.values() fallback to normalize both shapes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:25:06 -05:00
Marc Shade 7a4036dc8d feat: add live intelligence dashboard with SSE streaming
Starlette app serving a real-time dashboard at http://127.0.0.1:8501.
17 intelligence domains updated every 30s via Server-Sent Events:
markets, crypto, sectors, macro, cyber, military, earthquakes,
wildfires, climate, news, predictions, energy, aviation, infra,
cables, nav warnings, trending keywords.

- DOMPurify for XSS-safe DOM updates
- Exponential backoff SSE reconnection
- Responsive dark-theme grid layout with Chart.js
- CLI: `intel dashboard [--port 8501]`

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:20:48 -05:00
Marc Shade c12f45eaa8 fix: Jinja2 None-crash in report templates — |default(0) → or 0
Jinja2's |default(0) only replaces undefined, not None values from API
data. When used with comparison operators (>=) or format(), None causes
TypeError crashes. Replaced with Python `or` operator which handles
None correctly across market_overview.html (7 patterns) and
country_dossier.html (1 pattern).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:05:10 -05:00
Marc Shade a4d2b546a3 refactor: replace Wingbits (paid) with hexdb.io (free) for aircraft lookups
hexdb.io provides ICAO hex aircraft details with no API key required,
eliminating the 30-day trial limitation of Wingbits.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:12:09 -05:00
Marc Shade 3131ccbcdc fix: wildfire parser uses dynamic CSV headers + cache key collision
- Parse FIRMS CSV column indices from header row instead of hardcoded
  offsets. FIRMS added 'instrument' column, shifting confidence/frp.
- Fix cache key collision: _fetch_region and fetch_wildfires both used
  'wildfire:fires:{region}' — now CSV uses 'wildfire:csv:{region}'.
- Add type guard for non-string input to _parse_fires_csv.
- Remove unused _CSV_COLUMNS constant.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:55:05 -05:00
Marc Shade 01cb4255ef feat: add world-intel-mcp server — 36 tools across 14 intelligence domains
New MCP server providing real-time global intelligence: financial markets
(Yahoo Finance, CoinGecko), economic indicators (FRED, EIA, World Bank),
conflict tracking (ACLED, UCDP, HDX), military flights (OpenSky),
infrastructure monitoring (Cloudflare Radar, NGA cable health), maritime
warnings (NGA), climate anomalies (Open-Meteo), news aggregation (RSS,
GDELT), prediction markets (Polymarket), displacement data (UNHCR),
aviation delays (FAA), cyber threats (Feodo, CISA KEV, SANS, URLhaus),
country intelligence briefs (Ollama LLM), and HTML report generation
(Jinja2 + Chart.js).

Includes: SQLite TTL cache, per-source circuit breakers, async HTTP
fetcher with retry/rate-limiting, Click CLI with 27 commands, 4 analysis
modules (instability scoring, geo-convergence, signal aggregation, news
clustering), 4 HTML report templates, and 28 unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:33:57 -05:00