Restore DEB training settlement worker

docs/superpowers/plans/2026-06-23-deb-training-settlement-worker.md

@@ -0,0 +1,138 @@
+# DEB Training Settlement Worker Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Restore automatic DEB training data freshness after the realtime collector/canonical-cache split.
+
+**Architecture:** Add a dedicated low-frequency training settlement service that runs city analysis for forecast/DEB snapshots and reconciles recent settled actual highs. Keep it separate from the high-frequency observation collector so user-facing realtime updates stay light.
+
+**Tech Stack:** Python, FastAPI service modules, Docker Compose, pytest, existing `update_daily_record()` and `reconcile_recent_actual_highs()` paths.
+
+---
+
+### Task 1: Training Settlement Service
+
+**Files:**
+- Create: `web/training_settlement_service.py`
+- Create: `tests/test_training_settlement_service.py`
+
+- [ ] **Step 1: Write the failing service test**
+
+```python
+def test_training_settlement_cycle_runs_analysis_and_reconciles_supported_cities():
+    calls = {"analysis": [], "reconcile": []}
+
+    def analysis_runner(city):
+        calls["analysis"].append(city)
+        return {"city": city, "deb": {"prediction": 31.2}}
+
+    def reconciler(city, *, lookback_days):
+        calls["reconcile"].append((city, lookback_days))
+        return {"ok": True, "updated": 1}
+
+    result = run_training_settlement_cycle(
+        city_registry={
+            "shanghai": {"icao": "ZSSS", "settlement_source": "metar"},
+            "legacy": {"settlement_source": "wunderground"},
+        },
+        analysis_runner=analysis_runner,
+        actual_reconciler=reconciler,
+        lookback_days=9,
+    )
+
+    assert result["ok"] is True
+    assert result["processed"] == 1
+    assert calls["analysis"] == ["shanghai"]
+    assert calls["reconcile"] == [("shanghai", 9)]
+```
+
+- [ ] **Step 2: Run the test to verify it fails**
+
+Run: `python -m pytest tests/test_training_settlement_service.py::test_training_settlement_cycle_runs_analysis_and_reconciles_supported_cities -q`
+
+Expected: FAIL because `web.training_settlement_service` does not exist.
+
+- [ ] **Step 3: Implement the service**
+
+Create `run_training_settlement_cycle()` with injectable `city_registry`, `analysis_runner`, and `actual_reconciler`. Default analysis runner calls `web.analysis_service._analyze(city, force_refresh=False, detail_mode="panel")`; default reconciler calls `src.analysis.deb_algorithm.reconcile_recent_actual_highs()`.
+
+- [ ] **Step 4: Run the test to verify it passes**
+
+Run: `python -m pytest tests/test_training_settlement_service.py -q`
+
+Expected: PASS.
+
+### Task 2: Worker Entrypoint And Compose
+
+**Files:**
+- Create: `web/training_settlement_worker.py`
+- Modify: `docker-compose.yml`
+- Modify: `tests/test_deployment_runtime_config.py`
+
+- [ ] **Step 1: Write failing deployment tests**
+
+Assert the compose file contains `polyweather_training_settlement`, command `python -m web.training_settlement_worker`, role `training_settlement`, and conservative interval/lookback environment variables.
+
+- [ ] **Step 2: Run the deployment test to verify it fails**
+
+Run: `python -m pytest tests/test_deployment_runtime_config.py::test_runtime_compose_splits_realtime_workers -q`
+
+Expected: FAIL because the worker service is absent.
+
+- [ ] **Step 3: Implement worker and compose service**
+
+The worker loops `run_training_settlement_cycle()` every `POLYWEATHER_TRAINING_SETTLEMENT_INTERVAL_SEC` seconds, with initial delay and lookback from environment.
+
+- [ ] **Step 4: Run focused tests**
+
+Run: `python -m pytest tests/test_training_settlement_service.py tests/test_deployment_runtime_config.py -q`
+
+Expected: PASS.
+
+### Task 3: Stale Monitoring
+
+**Files:**
+- Modify: `web/diagnostics/health.py`
+- Modify: `web/services/system_api.py`
+- Modify: `tests/test_web_observability.py`
+
+- [ ] **Step 1: Write failing observability tests**
+
+Assert system status training summaries include `stale_days` for daily/truth/features, and Prometheus exports stale gauges for training data.
+
+- [ ] **Step 2: Run focused observability tests to verify failure**
+
+Run: `python -m pytest tests/test_web_observability.py::test_system_status_includes_training_data tests/test_web_observability.py::test_metrics_endpoint_returns_prometheus_payload_for_ops_admin -q`
+
+Expected: FAIL because stale fields/gauges are absent.
+
+- [ ] **Step 3: Implement stale summary and gauges**
+
+Add date-diff calculation against UTC today. Export `polyweather_daily_records_stale_days`, `polyweather_truth_records_stale_days`, `polyweather_training_features_stale_days`, and `polyweather_training_data_stale`.
+
+- [ ] **Step 4: Run focused observability tests**
+
+Run: `python -m pytest tests/test_web_observability.py::test_system_status_includes_training_data tests/test_web_observability.py::test_metrics_endpoint_returns_prometheus_payload_for_ops_admin -q`
+
+Expected: PASS.
+
+### Task 4: Verification And Backfill
+
+**Files:**
+- Optional create: `scripts/backfill_training_settlement.py`
+
+- [ ] **Step 1: Run backend checks**
+
+Run: `python -m ruff check .`
+
+Run: `python -m pytest tests/test_training_settlement_service.py tests/test_deployment_runtime_config.py tests/test_web_observability.py -q`
+
+- [ ] **Step 2: Run local one-shot cycle**
+
+Run: `python -m web.training_settlement_worker --once --lookback-days 10 --cities shanghai`
+
+Expected: JSON-like log output with `ok=True`; local DB should get a fresh row for the current local target date if analysis succeeds.
+
+- [ ] **Step 3: Production note**
+
+Missed forecast snapshots from 2026-06-15 to 2026-06-22 cannot be reconstructed honestly unless archived city analysis payloads exist. The worker restores forward automatic samples; actual-high truth can still be reconciled for supported settlement sources.