diff --git a/docs/superpowers/plans/2026-05-27-redis-stream-realtime-event-architecture.md b/docs/superpowers/plans/2026-05-27-redis-stream-realtime-event-architecture.md new file mode 100644 index 00000000..1653da9c --- /dev/null +++ b/docs/superpowers/plans/2026-05-27-redis-stream-realtime-event-architecture.md @@ -0,0 +1,105 @@ +# Redis Stream Realtime Event Architecture Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add Redis Stream as the production realtime event store and live fanout source while preserving the existing SSE patch protocol. + +**Architecture:** Keep `city_observation_patch.v1` and numeric `revision` as the browser contract. Add a Redis-backed event store beside the existing SQLite store, select it through an event-store factory, and let Redis-backed deployments fan out live events through a Redis subscriber loop instead of direct in-process ingest broadcast. + +**Tech Stack:** FastAPI, redis-py, Redis Stream, SQLite fallback, pytest, existing frontend EventSource hook. + +--- + +### Task 1: Redis Store Contract + +**Files:** +- Create: `tests/test_redis_realtime_event_store.py` +- Create: `web/redis_realtime_event_store.py` + +- [ ] **Step 1: Write failing tests** + +Cover append, replay by city, replay gap, and fallback-independent event shape. + +- [ ] **Step 2: Verify tests fail** + +Run: `python -m pytest tests/test_redis_realtime_event_store.py -q` + +Expected: fails because `web.redis_realtime_event_store` does not exist. + +- [ ] **Step 3: Implement Redis store** + +Add `RedisRealtimeEventStore` with `append_event`, `latest_revision`, `replay_events`, `replay_requires_resync`, and idempotent `start_live_subscription`. + +- [ ] **Step 4: Verify tests pass** + +Run: `python -m pytest tests/test_redis_realtime_event_store.py -q` + +### Task 2: Store Factory And SSE Router + +**Files:** +- Create: `web/realtime_event_store_factory.py` +- Modify: `web/routers/sse_router.py` +- Test: `tests/test_sse_replay.py` + +- [ ] **Step 1: Write failing tests** + +Cover `POLYWEATHER_EVENT_STORE=redis` selecting Redis, SQLite fallback when Redis is not configured, and Redis ingest not doing direct local broadcast. + +- [ ] **Step 2: Verify tests fail** + +Run: `python -m pytest tests/test_sse_replay.py tests/test_realtime_event_store_factory.py -q` + +- [ ] **Step 3: Implement factory and router integration** + +Use Redis store when configured; otherwise keep existing SQLite store. Ingest broadcasts directly only for stores that do not provide external live fanout. + +- [ ] **Step 4: Verify tests pass** + +Run: `python -m pytest tests/test_sse_replay.py tests/test_realtime_event_store_factory.py -q` + +### Task 3: Dependency And Operational Config + +**Files:** +- Modify: `requirements.txt` +- Modify: `docs/superpowers/specs/2026-05-27-redis-stream-realtime-event-architecture-design.md` + +- [ ] **Step 1: Add redis-py dependency** + +Add `redis>=5.0.0` to `requirements.txt`. + +- [ ] **Step 2: Add final config notes** + +Ensure the design doc names the runtime env vars used by code. + +### Task 4: Verification + +**Files:** +- No production edits unless tests reveal a gap. + +- [ ] **Step 1: Run backend realtime tests** + +Run: + +```powershell +python -m pytest tests/test_realtime_patch_schema.py tests/test_realtime_event_store.py tests/test_redis_realtime_event_store.py tests/test_realtime_event_store_factory.py tests/test_sse_replay.py -q +``` + +- [ ] **Step 2: Run broader frontend/backend checks** + +Run: + +```powershell +cd frontend +npm run test:business +npm run typecheck +npm run build +``` + +- [ ] **Step 3: Inspect diff** + +Run: + +```powershell +git diff --check +git status --short +``` diff --git a/docs/superpowers/specs/2026-05-27-redis-stream-realtime-event-architecture-design.md b/docs/superpowers/specs/2026-05-27-redis-stream-realtime-event-architecture-design.md new file mode 100644 index 00000000..f385159e --- /dev/null +++ b/docs/superpowers/specs/2026-05-27-redis-stream-realtime-event-architecture-design.md @@ -0,0 +1,407 @@ +# Redis Stream Realtime Event Architecture Design + +> 日期: 2026-05-27 +> 范围: PolyWeather 网站图表实时观测事件层 +> 目标服务器: 2 vCPU / 8 GB RAM / 50 GB 系统盘 + +## 背景 + +当前实时层已经具备生产化雏形: + +- 后端已有 `city_observation_patch.v1` schema。 +- `web/realtime_event_store.py` 使用 SQLite 保存 `observation_patch_events`,支持 `since_revision` replay。 +- `/api/events?cities=...&since_revision=...&replay_limit=...` 已经是前端 SSE 入口。 +- 前端 `frontend/hooks/use-sse-patches.ts` 使用 `EventSource`,并按当前可见城市订阅。 + +这解决了单进程、单实例下的实时推送和短窗口 replay,但还不够“一步到位”: + +- 多 worker / 多实例时,进程内广播不能共享。 +- SQLite replay log 可以补数据,但不适合作为跨实例 live fanout。 +- 服务重启、浏览器后台恢复、SSE 断线后,需要更稳定的事件源。 +- 后续如果把采集器拆成独立进程,需要一个明确的后端事件总线。 + +## 目标 + +升级为“Redis Stream 主事件日志 + SSE 浏览器传输 + SQLite fallback”的架构: + +- Redis Stream 是生产环境实时事件主通道。 +- SSE 仍然是浏览器唯一实时入口。 +- 前端协议保持 `city_observation_patch.v1`,不让浏览器感知 Redis。 +- 继续使用 numeric `revision`,兼容现有前端 `since_revision` 逻辑。 +- 支持按城市订阅、断线 replay、多 worker fanout。 +- 事件只保留短窗口,默认 24 小时;生产不建议超过 24 小时。 + +## 非目标 + +这次不做以下事情: + +- 不引入 Kafka、RabbitMQ 或复杂 consumer group。 +- 不把天气历史数据全部 event sourcing。 +- 不让浏览器直接连接 Redis。 +- 不把 DEB 预测、模型曲线、概率分布都改成事件流。 +- 不做永久行情归档;超过 replay 窗口时继续走 HTTP full detail resync。 + +## 推荐架构 + +```mermaid +flowchart LR + Collector["WeatherDataCollector / city refresh"] --> Ingest["POST /api/internal/collector-patch"] + Ingest --> Normalize["city_observation_patch.v1 normalizer"] + Normalize --> Store["RealtimeEventStore interface"] + Store --> Redis["Redis Stream: stream:city_observation"] + Store --> SQLite["SQLite fallback: observation_patch_events"] + Redis --> WorkerSub["Redis XREAD subscriber per API worker"] + WorkerSub --> LocalFanout["Local SSE fanout"] + SQLite --> ReplayFallback["SQLite replay fallback"] + SSE["GET /api/events?cities=...&since_revision=..."] --> Browser["Frontend EventSource"] + LocalFanout --> SSE + Redis --> SSE + ReplayFallback --> SSE + Browser --> Chart["Chart applies patch"] + Browser -->|resync_required| Detail["HTTP /api/city/{city}/detail"] +``` + +关键原则: + +- Redis Stream 负责后端事件保存、短窗口 replay、多实例共享。 +- SSE 负责服务端到浏览器的单向推送。 +- SQLite 只做 fallback 和本地开发,不再作为生产主事件总线。 +- 前端只认 patch event,不关心事件来自 Redis 还是 SQLite。 + +## 事件模型 + +继续使用现有 schema: + +```json +{ + "type": "city_observation_patch.v1", + "revision": 12345, + "city": "taipei", + "source": "cwa", + "obs_time": "2026-05-27T10:00:00+08:00", + "observed_at_utc": "2026-05-27T02:00:00Z", + "observed_at_local": "2026-05-27T10:00:00+08:00", + "city_local_date": "2026-05-27", + "city_timezone": "Asia/Taipei", + "source_cadence_sec": 600, + "ts": 1780000000000, + "payload": { + "temp": 34.2, + "max_so_far": 34.2, + "station_code": "466920", + "station_label": "中央气象署台北站", + "series_key": "settlement", + "unit": "celsius" + } +} +``` + +`revision` 仍然是前端公开排序键,必须是全局单调递增整数。 + +Redis Stream 自身的 stream ID,例如 `1780000000000-0`,只作为后端内部定位信息,不暴露给前端作为主 revision。 + +## Redis Stream 设计 + +Stream key: + +```text +stream:city_observation +``` + +Revision key: + +```text +counter:city_observation_revision +``` + +每条 stream entry 字段: + +- `revision`: 全局递增整数,公开给 SSE/前端。 +- `type`: `city_observation_patch.v1` +- `city`: normalized city key。 +- `source`: source code,例如 `cwa`, `metar`, `amos`, `amsc_awos`。 +- `obs_time`: 原始观测时间。 +- `payload_json`: compact JSON。 +- `created_at_ms`: 写入 Redis 的 UTC epoch ms。 +- `producer_id`: API worker / collector instance id,便于排查。 + +写入必须通过 Lua 脚本原子完成: + +1. `INCR counter:city_observation_revision` +2. `XADD stream:city_observation MAXLEN ~ {maxlen} * revision ... payload_json ...` +3. 返回 `{revision, stream_id}` + +原因: + +- 避免 `INCR` 成功但 `XADD` 失败后造成非必要 gap。 +- 让 revision 分配和事件入流成为一个原子操作。 +- 保持现有前端 numeric `since_revision` 不变。 + +## Replay 策略 + +`GET /api/events` 收到 `since_revision` 后: + +1. 读取 Redis Stream 最旧 entry,拿到 oldest revision。 +2. 如果 `since_revision > 0` 且 `since_revision < oldest_revision - 1`,发送 `resync_required`。 +3. 从 Redis Stream 顺序扫描事件。 +4. 过滤 `revision > since_revision`。 +5. 过滤 `city IN subscribed_cities`。 +6. 返回最多 `replay_limit + 1` 条。 +7. 如果超过 `replay_limit`,发送 `resync_required`,让前端 HTTP resync。 + +当前事件量很小,单个 stream 顺序扫描 24 小时事件是可以接受的: + +- 30 城市 * 1 条/分钟 * 24 小时 = 43,200 条。 +- replay 通常只发生在断线重连,不是高频查询。 + +如果后续城市量扩大,或者确实需要超过 24 小时的 replay,再加 per-city stream 或 Redis sorted index。 + +## Live Fanout + +每个 API worker 启动一个 Redis subscriber loop: + +```text +XREAD BLOCK 5000 STREAMS stream:city_observation {last_seen_stream_id} +``` + +读取到事件后: + +1. 解析 `city_observation_patch.v1`。 +2. 按本进程 SSE 连接的 `cities` subscription 过滤。 +3. 推送给本进程匹配的浏览器连接。 +4. 更新 worker 内存中的 `last_seen_stream_id`。 + +ingest endpoint 写入 Redis 后不直接广播本地连接,由 subscriber loop 统一 fanout。这样避免同一个 worker 上“写入后本地广播一次、subscriber 又广播一次”的重复事件。 + +前端仍然用 revision 去重。即使极端情况下收到重复 revision,也应忽略旧 revision。 + +## SQLite Fallback + +保留当前 SQLite store,但角色调整为: + +- 本地开发无需 Redis 时使用。 +- Redis 不可用且允许降级时使用。 +- 单实例临时运行时可用。 + +新增环境变量: + +```text +POLYWEATHER_EVENT_STORE=redis +POLYWEATHER_EVENT_STORE_FALLBACK=sqlite +POLYWEATHER_REDIS_URL=redis://127.0.0.1:6379/0 +POLYWEATHER_REDIS_STREAM_KEY=stream:city_observation +POLYWEATHER_REDIS_STREAM_MAXLEN=50000 +POLYWEATHER_PATCH_EVENT_RETENTION_HOURS=24 +POLYWEATHER_REDIS_REQUIRED=true +``` + +推荐生产策略: + +- `POLYWEATHER_EVENT_STORE=redis` +- `POLYWEATHER_REDIS_REQUIRED=true` +- Redis 写入失败时 ingest 返回 503,不广播不可 replay 的事件。 + +推荐本地策略: + +- 不配置 Redis,自动使用 SQLite。 +- 或 `POLYWEATHER_EVENT_STORE=sqlite`。 + +## Redis 容量与配置 + +当前 VPS 可以跑本机 Redis。 + +建议: + +```text +bind 127.0.0.1 +protected-mode yes +appendonly yes +appendfsync everysec +maxmemory 512mb +maxmemory-policy noeviction +``` + +默认 `MAXLEN ~ 50000` 足够约 24 小时事件: + +- 30 城市 * 1/min * 24h = 43,200 条。 +- payload 约 1-3 KB,实际内存取决于 Redis Stream overhead。 +- 512 MB Redis maxmemory 对当前事件量足够。 + +注意:这个事件流不是永久数据仓库。超过 24 小时窗口后,前端应 HTTP resync 当前城市 detail。 + +## 模块边界 + +新增/调整后端接口: + +### `web/realtime_event_store.py` + +定义统一接口: + +```python +class RealtimeEventStore: + def append_event(self, event: dict) -> dict: ... + def replay_events(self, *, cities: set[str] | None, since_revision: int, limit: int) -> list[dict]: ... + def replay_requires_resync(self, *, cities: set[str] | None, since_revision: int, replay_count: int, limit: int) -> bool: ... + def latest_revision(self) -> int: ... +``` + +保留现有 SQLite 实现,新增 Redis 实现。 + +### `web/redis_realtime_event_store.py` + +职责: + +- Redis 连接与健康检查。 +- Lua append script。 +- Redis Stream replay。 +- maxlen trim。 +- entry 到 SSE event 的反序列化。 + +### `web/sse_manager.py` + +职责: + +- 管理本进程 SSE 连接。 +- 记录每个连接订阅城市集合。 +- 将 Redis subscriber 读到的 event fanout 给匹配连接。 + +### `web/routers/sse_router.py` + +职责: + +- `/api/internal/collector-patch` 规范化并写入 event store。 +- `/api/events` 处理 connected、replay、heartbeat、live stream。 +- 当 replay 不完整时发送 `resync_required`。 + +### 前端 + +前端原则上不需要大改: + +- `use-sse-patches.ts` 继续使用 `EventSource`。 +- `lastRevision` 仍然是 number。 +- `resync_required` 继续触发当前可见城市 HTTP detail refresh。 +- 继续按可见城市列表构建 `cities` 参数。 + +## 运行流程 + +### 首屏 + +1. 前端 HTTP 加载 terminal rows。 +2. 可见图表 HTTP 加载 full detail。 +3. 前端连接 `/api/events?cities=...&since_revision=...&replay_limit=500`。 +4. 后端先 replay Redis Stream 中缺失事件,再进入 live stream。 + +### 新观测 + +1. 采集器产生 `city_patch` 或 v1 patch。 +2. ingest endpoint 规范化成 `city_observation_patch.v1`。 +3. Redis Lua script 写入 stream 并生成 revision。 +4. 每个 API worker 的 subscriber loop 读到事件。 +5. worker fanout 给订阅对应城市的 SSE 连接。 +6. 前端 apply patch,图表无痛追加点。 + +### 浏览器后台恢复 + +1. EventSource 如果断线,前端按指数退避重连。 +2. URL 带 `since_revision=lastRevision`。 +3. Redis replay 补齐后台期间错过的事件。 +4. 如果 revision 太旧或 replay 超限,前端收到 `resync_required`,HTTP 重拉 full detail。 + +## 错误处理 + +- Redis append 失败且 `POLYWEATHER_REDIS_REQUIRED=true`:ingest 返回 503,不广播。 +- Redis append 失败且允许 fallback:写入 SQLite,并在 health 状态标记 degraded。 +- Redis replay 失败:SSE 发送 `resync_required`,然后继续尝试 live stream。 +- Redis subscriber loop 断开:指数退避重连;期间 SSE heartbeat 仍可维持连接,但不会有 live patch。 +- 前端收到 revision 倒退或重复:忽略。 +- 前端收到未知 event type:忽略并记录 debug。 + +## 监控与诊断 + +新增健康指标: + +- 当前 event store mode: `redis` / `sqlite` / `degraded_sqlite` +- Redis ping latency +- latest revision +- stream length +- oldest revision +- subscriber connected +- SSE active connection count +- dropped/resync_required count + +建议暴露到现有 health endpoint 或日志: + +```json +{ + "realtime": { + "store": "redis", + "redis_connected": true, + "stream_len": 43210, + "latest_revision": 123456, + "oldest_revision": 80200, + "subscriber_connected": true, + "sse_connections": 9 + } +} +``` + +## 测试策略 + +后端: + +- Redis store append 会生成全局递增 numeric revision。 +- Redis replay 支持 city 过滤。 +- Redis replay 支持 `since_revision`。 +- Redis replay 超出 retention/limit 返回 resync required。 +- Redis 不可用时,根据 env 严格失败或 fallback SQLite。 +- SQLite store 现有测试保持通过。 +- SSE router 在 Redis store 下仍发送 connected、replay、heartbeat。 + +前端: + +- `use-sse-patches.ts` 保持 `revision` numeric。 +- 重连 URL 保留 `since_revision`。 +- 收到 `resync_required` 会触发 visible city refresh。 +- 重复 revision 不重复追加图表点。 + +验收命令: + +```powershell +python -m pytest tests/test_realtime_patch_schema.py tests/test_realtime_event_store.py tests/test_sse_replay.py +python -m pytest tests/test_redis_realtime_event_store.py +cd frontend +npm run test:business +npm run typecheck +npm run build +``` + +## 迁移步骤 + +1. 保留现有 SQLite implementation,抽出 event store factory。 +2. 新增 Redis implementation 和单元测试。 +3. 新增 Redis subscriber loop,但先在本地开发环境跑。 +4. `/api/events` 接入 event store factory。 +5. 线上安装 Redis,只监听 `127.0.0.1`。 +6. 先用 `POLYWEATHER_EVENT_STORE=redis`、`POLYWEATHER_REDIS_REQUIRED=false` 灰度。 +7. 观察 stream length、latest revision、SSE error、resync count。 +8. 稳定后切换 `POLYWEATHER_REDIS_REQUIRED=true`。 +9. 保留 SQLite fallback 代码,但生产不主动降级,避免多实例状态分裂。 + +## 验收标准 + +- 断开浏览器网络 1-5 分钟后恢复,图表自动补齐期间 patch。 +- 后台挂页面再回来,图表不需要等待 full loading 才更新当前点。 +- 多 worker 下,任意 worker 写入的 patch 都能推到其他 worker 的 SSE 连接。 +- `/api/events?cities=taipei,shanghai&since_revision=...` 只 replay 指定城市。 +- Redis 重启后,前端收到 `resync_required` 并 HTTP 重建画面。 +- 前端无需知道 Redis 存在。 +- 现有 `city_observation_patch.v1` schema 不破坏。 + +## 结论 + +这是一种“一步到位但不过度设计”的方案: + +- 一步到位的是事件契约、replay、跨实例 fanout 和生产部署边界。 +- 不过度设计的是不用 Kafka、不做永久行情库、不重写前端协议。 + +对当前产品最关键的是:图表像股票行情一样无痛刷新,断线可补,后台恢复可追上,多实例以后不用再推倒实时层。 diff --git a/frontend/components/dashboard/scan-terminal/LiveTemperatureThresholdChart.tsx b/frontend/components/dashboard/scan-terminal/LiveTemperatureThresholdChart.tsx index 4f6fab6f..ad9d8885 100644 --- a/frontend/components/dashboard/scan-terminal/LiveTemperatureThresholdChart.tsx +++ b/frontend/components/dashboard/scan-terminal/LiveTemperatureThresholdChart.tsx @@ -30,6 +30,7 @@ import { normObs, prefersHighFrequencyRunwayResolution, readSessionCache, + selectDisplayRunwayTemp, seedHourlyForecastFromRow, shouldPollLiveChart, validNumber, @@ -478,7 +479,7 @@ export function LiveTemperatureThresholdChart({ () => getObservationDisplayMetrics(row, chartHourly, settlementPlate), [row, chartHourly, settlementPlate], ); - const displayRunwayTemp = liveTemp ?? currentRunwayTemp; + const displayRunwayTemp = selectDisplayRunwayTemp(liveTemp, currentRunwayTemp, hasRunwayData); const wundergroundDailyHigh = validNumber(chartHourly?.airportCurrent?.max_so_far ?? chartHourly?.airportPrimary?.max_so_far) ?? null; const localDateStr = chartLocalDate || new Date().toISOString().slice(0, 10); @@ -786,3 +787,4 @@ export const __getObservationDisplayMetricsForTest = getObservationDisplayMetric export const __getPeakGlowStateForTest = getPeakGlowState; export const __shouldPollLiveChartForTest = shouldPollLiveChart; export const __mergePatchIntoHourlyForTest = mergePatchIntoHourly; +export const __selectDisplayRunwayTempForTest = selectDisplayRunwayTemp; diff --git a/frontend/components/dashboard/scan-terminal/__tests__/ssePatchArchitecture.test.ts b/frontend/components/dashboard/scan-terminal/__tests__/ssePatchArchitecture.test.ts index 2d2e029f..4cb70d25 100644 --- a/frontend/components/dashboard/scan-terminal/__tests__/ssePatchArchitecture.test.ts +++ b/frontend/components/dashboard/scan-terminal/__tests__/ssePatchArchitecture.test.ts @@ -42,6 +42,20 @@ export function runTests() { assert(store.includes("replay_events"), "event store must expose replay_events"); assert(store.includes("replay_requires_resync"), "event store must detect incomplete replay windows"); + const redisStorePath = path.join(repoRoot, "web", "redis_realtime_event_store.py"); + assert(fs.existsSync(redisStorePath), "backend must define a Redis Stream realtime event store"); + const redisStore = fs.readFileSync(redisStorePath, "utf8"); + assert(redisStore.includes("RedisRealtimeEventStore"), "Redis store must expose RedisRealtimeEventStore"); + assert(redisStore.includes("XADD") && redisStore.includes("MAXLEN"), "Redis store must append patches to a bounded Redis Stream"); + assert(redisStore.includes("xread"), "Redis store must support live fanout through Redis Stream reads"); + assert(redisStore.includes("counter:city_observation_revision"), "Redis store must keep a numeric revision counter for frontend compatibility"); + + const storeFactoryPath = path.join(repoRoot, "web", "realtime_event_store_factory.py"); + assert(fs.existsSync(storeFactoryPath), "backend must define a realtime event store factory"); + const storeFactory = fs.readFileSync(storeFactoryPath, "utf8"); + assert(storeFactory.includes("POLYWEATHER_EVENT_STORE"), "event store factory must select sqlite/redis from runtime config"); + assert(storeFactory.includes("POLYWEATHER_REDIS_REQUIRED"), "event store factory must support strict Redis mode"); + const sseRouterPath = path.join(repoRoot, "web", "routers", "sse_router.py"); assert(fs.existsSync(sseRouterPath), "FastAPI backend must define web/routers/sse_router.py"); const sseRouter = fs.readFileSync(sseRouterPath, "utf8"); @@ -53,6 +67,9 @@ export function runTests() { assert(sseRouter.includes('"/api/internal/collector-patch"'), "SSE router must expose collector patch ingest endpoint"); assert(sseRouter.includes("StreamingResponse"), "SSE route must return StreamingResponse"); assert(sseRouter.includes('"text/event-stream"'), "SSE route must use text/event-stream media type"); + assert(sseRouter.includes("create_realtime_event_store"), "SSE router must use the realtime event store factory"); + assert(sseRouter.includes("_ensure_live_subscription"), "SSE router must start external live fanout when the store provides it"); + assert(sseRouter.includes("uses_external_live_fanout"), "Redis-backed ingest must not directly broadcast duplicate local events"); const appFactory = readRepoFile("web", "app_factory.py"); assert(appFactory.includes("sse_router"), "FastAPI app factory must register the SSE router"); diff --git a/frontend/components/dashboard/scan-terminal/__tests__/temperatureDefaultVisibilityPolicy.test.ts b/frontend/components/dashboard/scan-terminal/__tests__/temperatureDefaultVisibilityPolicy.test.ts index a4a92497..c5f6e402 100644 --- a/frontend/components/dashboard/scan-terminal/__tests__/temperatureDefaultVisibilityPolicy.test.ts +++ b/frontend/components/dashboard/scan-terminal/__tests__/temperatureDefaultVisibilityPolicy.test.ts @@ -8,6 +8,7 @@ import { __getVisibleTemperatureSeriesForTest, __isTemperatureSeriesVisibleByDefaultForTest, __mergePatchIntoHourlyForTest, + __selectDisplayRunwayTempForTest, } from "@/components/dashboard/scan-terminal/LiveTemperatureThresholdChart"; function assert(condition: unknown, message: string): asserts condition { @@ -396,6 +397,46 @@ export function runTests() { "Shenzhen HKO observation series should include the airportPrimaryTodayObs curve points", ); + const hongKongCowinAndHko = __buildTemperatureChartDataForTest( + { + city: "hong kong", + local_date: "2026-05-27", + local_time: "10:42", + tz_offset_seconds: 8 * 60 * 60, + temp_symbol: "°C", + } as any, + { + localTime: "10:42", + times: ["00:00", "12:00", "18:00"], + temps: [27.2, 30.9, 27.6], + airportPrimary: { + source_code: "cowin_obs", + source_label: "CoWIN 6087", + station_label: "保良局陳守仁小學 1min (CoWIN)", + temp: 31.3, + obs_time: "2026-05-27T02:42:00Z", + }, + airportPrimaryTodayObs: [ + ["2026-05-27T02:40:00Z", 31.1], + ["2026-05-27T02:41:00Z", 31.2], + ["2026-05-27T02:42:00Z", 31.3], + ], + settlementTodayObs: [ + { time: "2026-05-27T02:30:00Z", temp: 31.0 }, + { time: "2026-05-27T02:40:00Z", temp: 31.2 }, + ], + } as any, + "1D", + ); + const hongKongCowinSeries = seriesByKey(hongKongCowinAndHko.series, "settlement") as any; + const hongKongHkoSeries = seriesByKey(hongKongCowinAndHko.series, "madis") as any; + assert(hongKongCowinSeries?.label === "CoWIN 6087", "Hong Kong should render CoWIN 6087 as the reference-station curve"); + assert( + hongKongCowinSeries.values.filter((value: number | null) => value !== null).length >= 2, + "Hong Kong CoWIN 6087 curve should use airportPrimaryTodayObs history points", + ); + assert(hongKongHkoSeries?.label === "HKO", "Hong Kong HKO settlement observations should remain visible as the HKO curve"); + const chengduFromAmosSnapshot = __buildTemperatureChartDataForTest( { city: "chengdu", @@ -663,6 +704,49 @@ export function runTests() { "AMOS temp/dew tuples should not be misread as two runway temperature samples", ); + const seoulRunwayMetrics = __getObservationDisplayMetricsForTest( + { + city: "seoul", + local_date: "2026-05-27", + local_time: "11:45", + tz_offset_seconds: 9 * 60 * 60, + current_temp: 23.0, + temp_symbol: "°C", + } as any, + { + localTime: "11:45", + times: ["00:00", "12:00", "18:00", "23:00"], + temps: [22.6, 22.6, 22.0, 21.4], + runwayPlateHistory: { + "15R/33L": [ + { time: "2026-05-27T02:40:00Z", temp: 23.9 }, + { time: "2026-05-27T02:45:00Z", temp: 24.3 }, + ], + "16L/34R": [ + { time: "2026-05-27T02:40:00Z", temp: 24.1 }, + { time: "2026-05-27T02:45:00Z", temp: 24.7 }, + ], + }, + amos: { + source: "amos", + temp_c: 23.0, + }, + } as any, + { maxTemp: 23.0 }, + ); + assert( + seoulRunwayMetrics.currentRunwayTemp === 24.3, + "runway header should use the latest settlement runway point instead of AMOS/METAR aggregate temp", + ); + assert( + seoulRunwayMetrics.observedHighRunway === 24.3, + "runway high should follow settlement runway history before AMOS/METAR aggregate temp", + ); + assert( + __selectDisplayRunwayTempForTest(23.0, 24.3, true) === 24.3, + "live aggregate temp should not override runway-history current temp when runway data is rendered", + ); + const newYorkMetrics = __getObservationDisplayMetricsForTest( { city: "new york", diff --git a/frontend/components/dashboard/scan-terminal/temperature-chart-logic.ts b/frontend/components/dashboard/scan-terminal/temperature-chart-logic.ts index d1aec61c..60c56482 100644 --- a/frontend/components/dashboard/scan-terminal/temperature-chart-logic.ts +++ b/frontend/components/dashboard/scan-terminal/temperature-chart-logic.ts @@ -548,6 +548,35 @@ function maxObservationValue(obs: Array<{ ts: number; value: number }>) { return Math.max(...obs.map((point) => point.value)); } +function getRunwayHistoryObservationMetrics( + row: ScanOpportunityRow | null, + hourly: HourlyForecast, +) { + const tzOffset = row?.tz_offset_seconds ?? 0; + const localDateStr = resolveChartLocalDate(row, hourly); + const localDayBounds = getLocalDayBounds(localDateStr); + const runwayHistorySeries = buildRunwayHistorySeries(row, hourly, tzOffset, localDateStr, 1) + .map((item) => ({ + ...item, + points: filterTimelinePointsToLocalDay(item.points, localDayBounds), + })) + .filter((item) => item.points.length > 0); + + const settlementSeries = runwayHistorySeries.filter((item) => item.isSettlement); + const candidateSeries = settlementSeries.length ? settlementSeries : runwayHistorySeries; + const points = candidateSeries.flatMap((item) => item.points); + if (!points.length) return { latest: null, high: null }; + + const latestTs = Math.max(...points.map((point) => point.ts)); + const latestValues = points + .filter((point) => point.ts === latestTs) + .map((point) => point.value); + return { + latest: latestValues.length ? Math.max(...latestValues) : null, + high: Math.max(...points.map((point) => point.value)), + }; +} + function hasRenderableLineSeries(series: EvidenceSeries[]) { return series.some( (item) => item.values.filter((value) => validNumber(value) !== null).length >= 2, @@ -588,6 +617,7 @@ function getObservationDisplayMetrics( const airportCurrentTemp = validNumber(hourly?.airportCurrent?.temp) ?? validNumber(hourly?.airportPrimary?.temp); const airportHigh = validNumber(hourly?.airportCurrent?.max_so_far) ?? validNumber(hourly?.airportPrimary?.max_so_far); const rowMetarHigh = validNumber(row?.metar_context?.airport_max_so_far ?? row?.metar_context?.max_temp ?? row?.current_max_so_far); + const runwayHistoryMetrics = getRunwayHistoryObservationMetrics(row, hourly); const settlementCityKey = normalizeCityKey(row?.city); const isShenzhen = settlementCityKey === 'shenzhen'; @@ -616,14 +646,16 @@ function getObservationDisplayMetrics( null; } else { currentRunwayTemp = - validNumber(hourly?.amos?.temp_c) ?? + runwayHistoryMetrics.latest ?? settlementPlate?.maxTemp ?? + validNumber(hourly?.amos?.temp_c) ?? latestSettlement ?? latestMetar ?? airportCurrentTemp ?? validNumber(row?.current_temp) ?? null; observedHighRunway = + runwayHistoryMetrics.high ?? settlementPlate?.maxTemp ?? highSettlement ?? airportHigh ?? @@ -638,6 +670,17 @@ function getObservationDisplayMetrics( return { currentRunwayTemp, observedHighMetar, observedHighRunway }; } +function selectDisplayRunwayTemp( + liveTemp: number | null, + currentRunwayTemp: number | null, + hasRunwayData: boolean, +) { + if (hasRunwayData && currentRunwayTemp !== null) { + return currentRunwayTemp; + } + return liveTemp ?? currentRunwayTemp; +} + function isSettlementRunway(row: ScanOpportunityRow | null, rwy: string) { const cityKey = normalizeCityKey(row?.city); const settlementPairs = SETTLEMENT_RUNWAY_PAIRS[cityKey] || []; @@ -1071,6 +1114,7 @@ function buildRunwayHistorySeries( hourly: HourlyForecast, tzOffset: number, localDateStr: string, + minPoints = 2, ): RunwayHistorySeries[] { const directHistory = hourly?.runwayPlateHistory ?? @@ -1100,7 +1144,7 @@ function buildRunwayHistorySeries( points, }; }) - .filter((series) => series.points.length > 1); + .filter((series) => series.points.length >= minPoints); if (directSeries.length) return directSeries; } @@ -1153,7 +1197,7 @@ function buildRunwayHistorySeries( }; }) .filter((point) => validNumber(point.value) !== null); - if (values.length <= 1) return null; + if (values.length < minPoints) return null; return { key: runwaySeriesKey(rwy), label: `${rwy}${isSettlement ? " 结算跑道" : ""}`, @@ -2047,6 +2091,7 @@ export { normalizeCityKey, prefersHighFrequencyRunwayResolution, readSessionCache, + selectDisplayRunwayTemp, seedHourlyForecastFromRow, seriesStats, shouldPollLiveChart, diff --git a/requirements.txt b/requirements.txt index 8534c0f0..b93aee76 100644 --- a/requirements.txt +++ b/requirements.txt @@ -9,3 +9,4 @@ web3 fastapi uvicorn websockets +redis>=5.0.0 diff --git a/src/data_collection/country_networks.py b/src/data_collection/country_networks.py index 1f35526b..6b366780 100644 --- a/src/data_collection/country_networks.py +++ b/src/data_collection/country_networks.py @@ -443,15 +443,30 @@ def _airport_primary_from_raw(city: str, raw: Dict[str, Any]) -> Dict[str, Any]: cowin = raw.get("cowin_current") or {} if cowin.get("temp") is not None: + cowin_station_code = str( + cowin.get("istNo") + or cowin.get("station_code") + or cowin.get("station_id") + or "" + ).strip() + if not cowin_station_code: + raw_icao = str(cowin.get("icao") or "").strip().upper() + if raw_icao.startswith("COWIN") and raw_icao[5:]: + cowin_station_code = raw_icao[5:] return _normalize_station_row( - station_code=meta.get("icao") or str(cowin.get("icao") or "COWIN6087"), - station_label=meta.get("airport_name") or cowin.get("station_label") or meta.get("icao"), + station_code=cowin_station_code or "6087", + station_label=( + cowin.get("station_label") + or cowin.get("station_name") + or meta.get("airport_name") + or "CoWIN 6087" + ), temp=_safe_float(cowin["temp"]), obs_time=str(cowin.get("obs_time") or metar.get("observation_time") or ""), source_code="cowin_obs", source_label="CoWIN 6087", is_official=True, - is_airport_station=True, + is_airport_station=False, is_settlement_anchor=False, extra={ "max_so_far": _safe_float(current.get("max_temp_so_far")), diff --git a/tests/test_country_networks.py b/tests/test_country_networks.py index d69f9dbd..2fa96c85 100644 --- a/tests/test_country_networks.py +++ b/tests/test_country_networks.py @@ -306,6 +306,58 @@ def test_hko_provider_marks_explicit_official_station_as_anchor(): assert snapshot["official_nearby"][0]["station_code"] == "LFS" +def test_hong_kong_cowin_primary_uses_station_6087_history(monkeypatch): + from src.database import runtime_state + + requested = {} + + class FakeOfficialIntradayObservationRepository: + def load_points(self, *, source_code, station_code, target_date): + requested["source_code"] = source_code + requested["station_code"] = station_code + requested["target_date"] = target_date + if source_code == "cowin_obs" and station_code == "6087": + return [ + {"time": "10:40", "temp": 31.1}, + {"time": "10:41", "temp": 31.3}, + ] + return [] + + monkeypatch.setattr( + runtime_state, + "OfficialIntradayObservationRepository", + FakeOfficialIntradayObservationRepository, + ) + + snapshot = build_country_network_snapshot( + "hong kong", + { + "cowin_current": { + "temp": 31.3, + "obs_time": "2026-05-27T02:41:00Z", + "istNo": "6087", + "icao": "COWIN6087", + "station_label": "保良局陳守仁小學 1min (CoWIN)", + }, + "settlement_current": { + "station_code": "HKO", + "station_name": "HK Observatory", + "observation_time": "2026-05-27T10:40:00+08:00", + "current": {"temp": 31.2}, + }, + }, + ) + + assert snapshot["airport_primary_current"]["source_code"] == "cowin_obs" + assert snapshot["airport_primary_current"]["station_code"] == "6087" + assert "陳守仁" in snapshot["airport_primary_current"]["station_label"] + assert requested["station_code"] == "6087" + assert snapshot["airport_primary_today_obs"] == [ + {"time": "10:40", "temp": 31.1}, + {"time": "10:41", "temp": 31.3}, + ] + + def test_moscow_provider_uses_realtime_metar_cluster_not_station_archive_rows(): raw = { "metar": { diff --git a/tests/test_realtime_event_store_factory.py b/tests/test_realtime_event_store_factory.py new file mode 100644 index 00000000..77343bf7 --- /dev/null +++ b/tests/test_realtime_event_store_factory.py @@ -0,0 +1,55 @@ +import pytest + +from web.realtime_event_store import RealtimeEventStore +from web.realtime_event_store_factory import create_realtime_event_store +from web.redis_realtime_event_store import RedisRealtimeEventStore + + +class FakeRedis: + pass + + +def test_event_store_factory_uses_sqlite_by_default(monkeypatch, tmp_path): + monkeypatch.delenv("POLYWEATHER_EVENT_STORE", raising=False) + + store = create_realtime_event_store(db_path=str(tmp_path / "polyweather.db")) + + assert isinstance(store, RealtimeEventStore) + + +def test_event_store_factory_uses_redis_when_configured(monkeypatch): + monkeypatch.setenv("POLYWEATHER_EVENT_STORE", "redis") + + store = create_realtime_event_store(redis_client=FakeRedis()) + + assert isinstance(store, RedisRealtimeEventStore) + + +def test_event_store_factory_falls_back_to_sqlite_when_redis_is_optional(monkeypatch, tmp_path): + monkeypatch.setenv("POLYWEATHER_EVENT_STORE", "redis") + monkeypatch.setenv("POLYWEATHER_REDIS_REQUIRED", "false") + + def broken_redis_store(**_kwargs): + raise RuntimeError("redis down") + + store = create_realtime_event_store( + db_path=str(tmp_path / "polyweather.db"), + redis_store_builder=broken_redis_store, + ) + + assert isinstance(store, RealtimeEventStore) + assert getattr(store, "degraded_from", None) == "redis" + + +def test_event_store_factory_raises_when_redis_is_required(monkeypatch, tmp_path): + monkeypatch.setenv("POLYWEATHER_EVENT_STORE", "redis") + monkeypatch.setenv("POLYWEATHER_REDIS_REQUIRED", "true") + + def broken_redis_store(**_kwargs): + raise RuntimeError("redis down") + + with pytest.raises(RuntimeError, match="redis down"): + create_realtime_event_store( + db_path=str(tmp_path / "polyweather.db"), + redis_store_builder=broken_redis_store, + ) diff --git a/tests/test_redis_realtime_event_store.py b/tests/test_redis_realtime_event_store.py new file mode 100644 index 00000000..4e162ec7 --- /dev/null +++ b/tests/test_redis_realtime_event_store.py @@ -0,0 +1,125 @@ +from web.redis_realtime_event_store import RedisRealtimeEventStore +from web.realtime_patch_schema import normalize_observation_patch + + +class FakeRedis: + def __init__(self): + self.counter = 0 + self.entries = [] + + def eval(self, _script, _numkeys, stream_key, counter_key, *args): + ( + maxlen, + event_type, + schema_type, + schema_version, + city, + source, + obs_time, + payload_json, + created_at_ms, + ts, + producer_id, + ) = args + self.counter += 1 + stream_id = f"{created_at_ms}-{self.counter}" + fields = { + "revision": str(self.counter), + "type": event_type, + "schema_type": schema_type, + "schema_version": str(schema_version), + "city": city, + "source": source, + "obs_time": obs_time, + "payload_json": payload_json, + "created_at_ms": str(created_at_ms), + "ts": str(ts), + "producer_id": producer_id, + } + self.entries.append((stream_id, fields)) + trim_to = int(maxlen) + if trim_to > 0 and len(self.entries) > trim_to: + self.entries = self.entries[-trim_to:] + return [self.counter, stream_id] + + def get(self, key): + if key == "counter:city_observation_revision": + return str(self.counter) + return None + + def xrange(self, stream_key, min="-", max="+", count=None): + rows = list(self.entries) + if count is not None: + rows = rows[: int(count)] + return rows + + +def _event(city: str, temp: float, source: str = "cwa"): + return normalize_observation_patch( + { + "city": city, + "changes": { + "temp": temp, + "obs_time": "2026-05-27T10:00:00+08:00", + "source": source, + }, + } + ) + + +def test_redis_event_store_appends_monotonic_revisions_and_replays_by_city(): + store = RedisRealtimeEventStore(redis_client=FakeRedis(), maxlen=10, producer_id="test") + + taipei = store.append_event(_event("taipei", 34.2)) + seoul = store.append_event(_event("seoul", 21.5, source="amos")) + taipei_next = store.append_event(_event("taipei", 34.4)) + + assert taipei["revision"] == 1 + assert seoul["revision"] == 2 + assert taipei_next["revision"] == 3 + assert store.latest_revision() == 3 + + replay = store.replay_events(cities={"taipei"}, since_revision=1, limit=10) + + assert [event["revision"] for event in replay] == [3] + assert replay[0]["city"] == "taipei" + assert replay[0]["payload"]["temp"] == 34.4 + + +def test_redis_event_store_preserves_time_contract_on_replay(): + store = RedisRealtimeEventStore(redis_client=FakeRedis(), maxlen=10, producer_id="test") + + stored = store.append_event( + normalize_observation_patch( + { + "city": "toronto", + "changes": { + "temp": 26, + "obs_time": "2026-05-27T23:16:00Z", + "source": "metar", + }, + } + ) + ) + replayed = store.replay_events(cities={"toronto"}, since_revision=0, limit=10)[0] + + assert stored["observed_at_utc"] == "2026-05-27T23:16:00Z" + assert replayed["observed_at_local"] == "2026-05-27T19:16:00-04:00" + assert replayed["city_timezone"] == "America/Toronto" + + +def test_redis_event_store_reports_replay_gap_when_limit_is_exceeded(): + store = RedisRealtimeEventStore(redis_client=FakeRedis(), maxlen=10, producer_id="test") + + for temp in [30.0, 30.5, 31.0]: + store.append_event(_event("hong kong", temp, source="hko")) + + replay = store.replay_events(cities={"hong kong"}, since_revision=0, limit=2) + + assert [event["revision"] for event in replay] == [1, 2] + assert store.replay_requires_resync( + cities={"hong kong"}, + since_revision=0, + replay_count=len(replay), + limit=2, + ) diff --git a/tests/test_sse_replay.py b/tests/test_sse_replay.py index a3759f52..d3e0d40d 100644 --- a/tests/test_sse_replay.py +++ b/tests/test_sse_replay.py @@ -131,3 +131,52 @@ def test_replay_limit_is_bounded(): assert sse_router._bounded_replay_limit(0) == 1 assert sse_router._bounded_replay_limit(500) == 500 assert sse_router._bounded_replay_limit(5000) == 2000 + + +def test_ingest_patch_uses_external_fanout_without_direct_broadcast(monkeypatch): + class FakeExternalStore: + uses_external_live_fanout = True + + def __init__(self): + self.started = 0 + + def start_live_subscription(self, callback): + self.started += 1 + self.callback = callback + + def append_event(self, event): + return { + **event, + "revision": 12, + } + + class FakeManager: + def __init__(self): + self.broadcasted = [] + + def broadcast_event(self, event): + self.broadcasted.append(event) + return event + + store = FakeExternalStore() + manager = FakeManager() + monkeypatch.setattr(sse_router, "event_store", store) + monkeypatch.setattr(sse_router, "sse_manager", manager) + monkeypatch.setattr(sse_router, "_live_subscription_started", False) + + response = TestClient(app).post( + "/api/internal/collector-patch", + json={ + "city": "taipei", + "changes": { + "temp": 34.2, + "source": "cwa", + "obs_time": "2026-05-27T10:00:00+08:00", + }, + }, + ) + + assert response.status_code == 200 + assert response.json()["revision"] == 12 + assert store.started == 1 + assert manager.broadcasted == [] diff --git a/tests/test_web_observability.py b/tests/test_web_observability.py index 910f6a0c..20dc0cc2 100644 --- a/tests/test_web_observability.py +++ b/tests/test_web_observability.py @@ -36,6 +36,10 @@ def test_system_status_returns_summary_shape(): assert payload['probability']['engine_mode'] == 'legacy' assert 'training_data' in payload assert 'station_networks' in payload + assert 'realtime' in payload + assert payload['realtime']['store'] in {'sqlite', 'redis', 'degraded_sqlite'} + assert 'latest_revision' in payload['realtime'] + assert 'sse_connections' in payload['realtime'] assert 'truth_records' in payload['training_data'] assert 'training_features' in payload['training_data'] assert 'city_coverage' in payload['training_data'] diff --git a/web/realtime_event_store_factory.py b/web/realtime_event_store_factory.py new file mode 100644 index 00000000..2bd0f35a --- /dev/null +++ b/web/realtime_event_store_factory.py @@ -0,0 +1,45 @@ +"""Factory for selecting the realtime observation event store.""" + +from __future__ import annotations + +import os +from typing import Any, Callable, Optional + +from loguru import logger + +from web.realtime_event_store import RealtimeEventStore +from web.redis_realtime_event_store import RedisRealtimeEventStore + + +def _truthy(value: Optional[str], *, default: bool = False) -> bool: + raw = str(value or "").strip().lower() + if not raw: + return default + return raw in {"1", "true", "yes", "on"} + + +def create_realtime_event_store( + *, + db_path: Optional[str] = None, + redis_client: Any = None, + redis_store_builder: Optional[Callable[..., Any]] = None, +) -> Any: + mode = str(os.getenv("POLYWEATHER_EVENT_STORE") or "sqlite").strip().lower() + if mode in {"", "sqlite"}: + return RealtimeEventStore(db_path=db_path) + + if mode != "redis": + logger.warning(f"Unknown POLYWEATHER_EVENT_STORE={mode!r}; using sqlite event store") + return RealtimeEventStore(db_path=db_path) + + builder = redis_store_builder or RedisRealtimeEventStore + try: + kwargs = {"redis_client": redis_client} if redis_client is not None else {} + return builder(**kwargs) + except Exception: + if _truthy(os.getenv("POLYWEATHER_REDIS_REQUIRED"), default=True): + raise + logger.exception("Redis realtime event store unavailable; falling back to sqlite") + fallback = RealtimeEventStore(db_path=db_path) + setattr(fallback, "degraded_from", "redis") + return fallback diff --git a/web/redis_realtime_event_store.py b/web/redis_realtime_event_store.py new file mode 100644 index 00000000..d64baa4e --- /dev/null +++ b/web/redis_realtime_event_store.py @@ -0,0 +1,284 @@ +"""Redis Stream-backed realtime observation event store.""" + +from __future__ import annotations + +import json +import os +import socket +import threading +import time +from typing import Any, Callable, Dict, Iterable, List, Optional, Set + +from loguru import logger + +from web.realtime_event_store import MAX_REPLAY_LIMIT, TIME_CONTRACT_KEYS +from web.realtime_patch_schema import EVENT_TYPE + + +DEFAULT_STREAM_KEY = "stream:city_observation" +DEFAULT_COUNTER_KEY = "counter:city_observation_revision" +DEFAULT_MAXLEN = 50000 + +APPEND_EVENT_SCRIPT = """ +local revision = redis.call('INCR', KEYS[2]) +local stream_id = redis.call( + 'XADD', KEYS[1], 'MAXLEN', '~', ARGV[1], '*', + 'revision', revision, + 'type', ARGV[2], + 'schema_type', ARGV[3], + 'schema_version', ARGV[4], + 'city', ARGV[5], + 'source', ARGV[6], + 'obs_time', ARGV[7], + 'payload_json', ARGV[8], + 'created_at_ms', ARGV[9], + 'ts', ARGV[10], + 'producer_id', ARGV[11] +) +return {revision, stream_id} +""" + + +def _decode(value: Any) -> str: + if isinstance(value, bytes): + return value.decode("utf-8") + return str(value or "") + + +def _normalize_city_set(cities: Optional[Iterable[str]]) -> Set[str]: + return {str(city or "").strip().lower() for city in (cities or set()) if str(city or "").strip()} + + +def _time_contract_from_payload(payload: Dict[str, Any]) -> Dict[str, Any]: + return {key: payload[key] for key in TIME_CONTRACT_KEYS if key in payload} + + +def _int_or_zero(value: Any) -> int: + try: + return int(value) + except (TypeError, ValueError): + return 0 + + +class RedisRealtimeEventStore: + """Persist replayable observation patch events in a Redis Stream.""" + + uses_external_live_fanout = True + + def __init__( + self, + *, + redis_url: Optional[str] = None, + redis_client: Any = None, + stream_key: Optional[str] = None, + counter_key: Optional[str] = None, + maxlen: Optional[int] = None, + producer_id: Optional[str] = None, + ) -> None: + self.stream_key = stream_key or os.getenv("POLYWEATHER_REDIS_STREAM_KEY") or DEFAULT_STREAM_KEY + self.counter_key = counter_key or os.getenv("POLYWEATHER_REDIS_COUNTER_KEY") or DEFAULT_COUNTER_KEY + self.maxlen = max(1, int(maxlen or os.getenv("POLYWEATHER_REDIS_STREAM_MAXLEN") or DEFAULT_MAXLEN)) + self.producer_id = producer_id or os.getenv("POLYWEATHER_INSTANCE_ID") or socket.gethostname() + self._client = redis_client or self._build_client(redis_url) + self._subscriber_lock = threading.Lock() + self._subscriber_thread: Optional[threading.Thread] = None + self._subscriber_stop: Optional[threading.Event] = None + + @staticmethod + def _build_client(redis_url: Optional[str]) -> Any: + try: + import redis # type: ignore + except ImportError as exc: + raise RuntimeError("redis package is required for Redis realtime event store") from exc + + url = redis_url or os.getenv("POLYWEATHER_REDIS_URL") or "redis://127.0.0.1:6379/0" + client = redis.Redis.from_url( + url, + socket_timeout=5, + socket_connect_timeout=5, + health_check_interval=30, + ) + client.ping() + return client + + def append_event(self, event: Dict[str, Any]) -> Dict[str, Any]: + if event.get("type") != EVENT_TYPE: + raise ValueError("unsupported realtime event type") + payload = event.get("payload") + if not isinstance(payload, dict): + raise ValueError("event payload must be an object") + + created_at_ms = int(time.time() * 1000) + ts = int(event.get("ts") or created_at_ms) + payload_json = json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + result = self._client.eval( + APPEND_EVENT_SCRIPT, + 2, + self.stream_key, + self.counter_key, + self.maxlen, + str(event["type"]), + str(event["schema_type"]), + int(event["schema_version"]), + str(event["city"]), + str(event["source"]), + str(event.get("obs_time") or ""), + payload_json, + created_at_ms, + ts, + self.producer_id, + ) + revision = int(_decode(result[0] if isinstance(result, (list, tuple)) else result)) + return { + "type": event["type"], + "revision": revision, + "city": str(event["city"]), + "source": str(event["source"]), + "obs_time": event.get("obs_time"), + **_time_contract_from_payload(payload), + "ts": ts, + "payload": payload, + } + + def latest_revision(self) -> int: + value = self._client.get(self.counter_key) + revision = _int_or_zero(_decode(value)) + if revision: + return revision + return max((event["revision"] for event in self._all_events()), default=0) + + def status(self) -> Dict[str, Any]: + out: Dict[str, Any] = { + "store": "redis", + "redis_connected": False, + "stream_key": self.stream_key, + "latest_revision": 0, + "stream_len": None, + "oldest_revision": None, + "subscriber_connected": bool( + self._subscriber_thread and self._subscriber_thread.is_alive() + ), + } + try: + ping = getattr(self._client, "ping", None) + if callable(ping): + ping() + out["redis_connected"] = True + out["latest_revision"] = self.latest_revision() + xlen = getattr(self._client, "xlen", None) + if callable(xlen): + out["stream_len"] = int(xlen(self.stream_key)) + events = self._all_events() + if events: + out["oldest_revision"] = min(int(event["revision"]) for event in events) + except Exception as exc: + out["error"] = str(exc) + return out + + def replay_events( + self, + *, + cities: Optional[Set[str]], + since_revision: int, + limit: int, + ) -> List[Dict[str, Any]]: + city_set = _normalize_city_set(cities) + since = max(0, int(since_revision or 0)) + bounded_limit = max(1, min(MAX_REPLAY_LIMIT, int(limit or 1))) + replay: List[Dict[str, Any]] = [] + for event in self._all_events(): + if int(event.get("revision") or 0) <= since: + continue + if city_set and str(event.get("city") or "").strip().lower() not in city_set: + continue + replay.append(event) + if len(replay) >= bounded_limit: + break + return replay + + def replay_requires_resync( + self, + *, + cities: Optional[Set[str]], + since_revision: int, + replay_count: int, + limit: int, + ) -> bool: + city_set = _normalize_city_set(cities) + since = max(0, int(since_revision or 0)) + matching_events = [ + event + for event in self._all_events() + if not city_set or str(event.get("city") or "").strip().lower() in city_set + ] + if not matching_events: + return False + + min_revision = min(int(event["revision"]) for event in matching_events) + if since > 0 and since < min_revision - 1: + return True + bounded_limit = max(1, int(limit or 1)) + if int(replay_count or 0) < bounded_limit: + return False + return sum(1 for event in matching_events if int(event["revision"]) > since) > bounded_limit + + def start_live_subscription(self, callback: Callable[[Dict[str, Any]], None]) -> None: + with self._subscriber_lock: + if self._subscriber_thread and self._subscriber_thread.is_alive(): + return + self._subscriber_stop = threading.Event() + self._subscriber_thread = threading.Thread( + target=self._live_subscription_loop, + args=(callback, self._subscriber_stop), + name="polyweather-redis-realtime-subscriber", + daemon=True, + ) + self._subscriber_thread.start() + + def stop_live_subscription(self) -> None: + with self._subscriber_lock: + if self._subscriber_stop: + self._subscriber_stop.set() + self._subscriber_thread = None + self._subscriber_stop = None + + def _live_subscription_loop( + self, + callback: Callable[[Dict[str, Any]], None], + stop_event: threading.Event, + ) -> None: + last_seen_id = "$" + while not stop_event.is_set(): + try: + rows = self._client.xread({self.stream_key: last_seen_id}, count=100, block=5000) + for _stream_name, entries in rows or []: + for entry_id, fields in entries: + last_seen_id = _decode(entry_id) + callback(self._entry_to_event(entry_id, fields)) + except Exception as exc: + logger.warning(f"Redis realtime subscriber disconnected: {exc}") + stop_event.wait(2.0) + + def _all_events(self) -> List[Dict[str, Any]]: + rows = self._client.xrange(self.stream_key, min="-", max="+") + return [self._entry_to_event(entry_id, fields) for entry_id, fields in rows or []] + + @staticmethod + def _entry_to_event(entry_id: Any, fields: Dict[Any, Any]) -> Dict[str, Any]: + normalized = {_decode(key): _decode(value) for key, value in dict(fields or {}).items()} + payload = json.loads(normalized.get("payload_json") or "{}") + schema_type = normalized.get("schema_type") or "city_observation_patch" + schema_version = int(normalized.get("schema_version") or 1) + created_at_ms = _int_or_zero(normalized.get("created_at_ms")) or int(time.time() * 1000) + ts = _int_or_zero(normalized.get("ts")) or created_at_ms + obs_time = normalized.get("obs_time") or None + return { + "type": normalized.get("type") or f"{schema_type}.v{schema_version}", + "revision": int(normalized["revision"]), + "city": normalized.get("city") or "", + "source": normalized.get("source") or "", + "obs_time": obs_time, + **_time_contract_from_payload(payload if isinstance(payload, dict) else {}), + "ts": ts, + "payload": payload if isinstance(payload, dict) else {}, + } diff --git a/web/routers/sse_router.py b/web/routers/sse_router.py index d5d2ccc8..3234a1e4 100644 --- a/web/routers/sse_router.py +++ b/web/routers/sse_router.py @@ -3,18 +3,22 @@ from __future__ import annotations import time +import threading from typing import Any, Optional, Set from fastapi import APIRouter, HTTPException, Query, Request from fastapi.responses import StreamingResponse -from web.realtime_event_store import RealtimeEventStore, MAX_REPLAY_LIMIT +from web.realtime_event_store import MAX_REPLAY_LIMIT +from web.realtime_event_store_factory import create_realtime_event_store from web.realtime_patch_schema import PatchValidationError, normalize_observation_patch from web.sse_manager import sse_manager router = APIRouter(tags=["events"]) -event_store = RealtimeEventStore() +event_store = create_realtime_event_store() +_live_subscription_lock = threading.Lock() +_live_subscription_started = False def _parse_cities_param(cities: str) -> Set[str]: @@ -33,6 +37,18 @@ def _bounded_replay_limit(value: int) -> int: return max(1, min(MAX_REPLAY_LIMIT, limit)) +def _ensure_live_subscription() -> None: + starter = getattr(event_store, "start_live_subscription", None) + if not callable(starter): + return + global _live_subscription_started + with _live_subscription_lock: + if _live_subscription_started: + return + starter(sse_manager.broadcast_event) + _live_subscription_started = True + + @router.options("/api/events") async def sse_events_preflight(request: Request): return {"ok": True} @@ -50,6 +66,7 @@ async def sse_events( allowed = origin in {"https://polyweather.top", "https://www.polyweather.top", "http://localhost:3000"} city_set = _parse_cities_param(cities) limit = _bounded_replay_limit(replay_limit) + _ensure_live_subscription() latest_revision = event_store.latest_revision() replay_events = [] resync_event = None @@ -107,10 +124,13 @@ async def ingest_patch(patch: dict[str, Any]): except PatchValidationError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc + _ensure_live_subscription() + try: event = event_store.append_event(normalized) except Exception as exc: raise HTTPException(status_code=500, detail="event log write failed") from exc - sse_manager.broadcast_event(event) + if not bool(getattr(event_store, "uses_external_live_fanout", False)): + sse_manager.broadcast_event(event) return {"ok": True, "revision": event["revision"]} diff --git a/web/services/system_api.py b/web/services/system_api.py index a59e2899..eb2cd429 100644 --- a/web/services/system_api.py +++ b/web/services/system_api.py @@ -23,7 +23,35 @@ def get_health_payload() -> Dict[str, Any]: async def get_system_status_payload() -> Dict[str, Any]: - return await run_in_threadpool(build_system_status_payload) + payload = await run_in_threadpool(build_system_status_payload) + payload["realtime"] = await run_in_threadpool(_realtime_status_payload) + return payload + + +def _realtime_status_payload() -> Dict[str, Any]: + try: + from web.routers import sse_router + + store = sse_router.event_store + status_fn = getattr(store, "status", None) + if callable(status_fn): + status = dict(status_fn()) + else: + store_name = "degraded_sqlite" if getattr(store, "degraded_from", None) == "redis" else "sqlite" + status = { + "store": store_name, + "latest_revision": int(store.latest_revision()), + } + connection_count = getattr(sse_router.sse_manager, "connection_count", None) + status["sse_connections"] = int(connection_count()) if callable(connection_count) else 0 + return status + except Exception as exc: + return { + "store": "unknown", + "latest_revision": 0, + "sse_connections": 0, + "error": str(exc), + } def get_system_cache_status(request: Request, cities: Optional[str] = None) -> Dict[str, Any]: diff --git a/web/sse_manager.py b/web/sse_manager.py index 475f71ef..c74b3006 100644 --- a/web/sse_manager.py +++ b/web/sse_manager.py @@ -18,6 +18,7 @@ class SseManager: def __init__(self) -> None: self._queues: DefaultDict[str, set[asyncio.Queue[dict[str, Any]]]] = defaultdict(set) self._queue_cities: dict[int, frozenset[str]] = {} + self._queue_loops: dict[int, asyncio.AbstractEventLoop] = {} self._lock = threading.RLock() self._revision = 0 @@ -49,6 +50,10 @@ class SseManager: if revision > self._revision: self._revision = revision + def connection_count(self) -> int: + with self._lock: + return sum(len(queue_set) for queue_set in self._queues.values()) + def broadcast(self, city: str, changes: dict[str, Any]) -> dict[str, Any]: event = { "type": "city_patch", @@ -69,26 +74,37 @@ class SseManager: with self._lock: queue_items = [ - (queue, self._queue_cities.get(id(queue), frozenset())) + ( + queue, + self._queue_cities.get(id(queue), frozenset()), + self._queue_loops.get(id(queue)), + ) for queue_set in self._queues.values() for queue in queue_set ] - for queue, subscribed_cities in queue_items: + for queue, subscribed_cities, loop in queue_items: if subscribed_cities and city not in subscribed_cities: continue + if loop and loop.is_running(): + loop.call_soon_threadsafe(self._put_queue_event, queue, event) + continue + self._put_queue_event(queue, event) + return event + + @staticmethod + def _put_queue_event(queue: asyncio.Queue[dict[str, Any]], event: dict[str, Any]) -> None: + try: + queue.put_nowait(event) + except asyncio.QueueFull: + try: + queue.get_nowait() + except asyncio.QueueEmpty: + pass try: queue.put_nowait(event) except asyncio.QueueFull: - try: - queue.get_nowait() - except asyncio.QueueEmpty: - pass - try: - queue.put_nowait(event) - except asyncio.QueueFull: - pass - return event + pass async def event_stream( self, @@ -102,9 +118,11 @@ class SseManager: user_key = str(user_id or "anon") city_set = frozenset(self._normalize_city_set(cities)) queue: asyncio.Queue[dict[str, Any]] = asyncio.Queue(maxsize=QUEUE_MAXSIZE) + loop = asyncio.get_running_loop() with self._lock: self._queues[user_key].add(queue) self._queue_cities[id(queue)] = city_set + self._queue_loops[id(queue)] = loop if connected_revision is not None: self._revision = max(self._revision, int(connected_revision or 0)) @@ -138,6 +156,7 @@ class SseManager: with self._lock: self._queues[user_key].discard(queue) self._queue_cities.pop(id(queue), None) + self._queue_loops.pop(id(queue), None) if not self._queues[user_key]: self._queues.pop(user_key, None)