PolyWeather/docs/superpowers/plans/2026-06-23-deb-training-settlement-worker.md

DEB Training Settlement Worker Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Restore automatic DEB training data freshness after the realtime collector/canonical-cache split.

Architecture: Add a dedicated low-frequency training settlement service that runs city analysis for forecast/DEB snapshots and reconciles recent settled actual highs. Keep it separate from the high-frequency observation collector so user-facing realtime updates stay light.

Tech Stack: Python, FastAPI service modules, Docker Compose, pytest, existing update_daily_record() and reconcile_recent_actual_highs() paths.

---

Task 1: Training Settlement Service

Files:

• Create: web/training_settlement_service.py

• Create: tests/test_training_settlement_service.py

• Step 1: Write the failing service test

def test_training_settlement_cycle_runs_analysis_and_reconciles_supported_cities():
    calls = {"analysis": [], "reconcile": []}

    def analysis_runner(city):
        calls["analysis"].append(city)
        return {"city": city, "deb": {"prediction": 31.2}}

    def reconciler(city, *, lookback_days):
        calls["reconcile"].append((city, lookback_days))
        return {"ok": True, "updated": 1}

    result = run_training_settlement_cycle(
        city_registry={
            "shanghai": {"icao": "ZSSS", "settlement_source": "metar"},
            "legacy": {"settlement_source": "wunderground"},
        },
        analysis_runner=analysis_runner,
        actual_reconciler=reconciler,
        lookback_days=9,
    )

    assert result["ok"] is True
    assert result["processed"] == 1
    assert calls["analysis"] == ["shanghai"]
    assert calls["reconcile"] == [("shanghai", 9)]

• Step 2: Run the test to verify it fails

Run: python -m pytest tests/test_training_settlement_service.py::test_training_settlement_cycle_runs_analysis_and_reconciles_supported_cities -q

Expected: FAIL because web.training_settlement_service does not exist.

• Step 3: Implement the service

Create run_training_settlement_cycle() with injectable city_registry, analysis_runner, and actual_reconciler. Default analysis runner calls web.analysis_service._analyze(city, force_refresh=False, detail_mode="panel"); default reconciler calls src.analysis.deb_algorithm.reconcile_recent_actual_highs().

• Step 4: Run the test to verify it passes

Run: python -m pytest tests/test_training_settlement_service.py -q

Expected: PASS.

Task 2: Worker Entrypoint And Compose

Files:

• Create: web/training_settlement_worker.py

• Modify: docker-compose.yml

• Modify: tests/test_deployment_runtime_config.py

• Step 1: Write failing deployment tests

Assert the compose file contains polyweather_training_settlement, command python -m web.training_settlement_worker, role training_settlement, and conservative interval/lookback environment variables.

• Step 2: Run the deployment test to verify it fails

Run: python -m pytest tests/test_deployment_runtime_config.py::test_runtime_compose_splits_realtime_workers -q

Expected: FAIL because the worker service is absent.

• Step 3: Implement worker and compose service

The worker loops run_training_settlement_cycle() every POLYWEATHER_TRAINING_SETTLEMENT_INTERVAL_SEC seconds, with initial delay and lookback from environment.

• Step 4: Run focused tests

Run: python -m pytest tests/test_training_settlement_service.py tests/test_deployment_runtime_config.py -q

Expected: PASS.

Task 3: Stale Monitoring

Files:

• Modify: web/diagnostics/health.py

• Modify: web/services/system_api.py

• Modify: tests/test_web_observability.py

• Step 1: Write failing observability tests

Assert system status training summaries include stale_days for daily/truth/features, and Prometheus exports stale gauges for training data.

• Step 2: Run focused observability tests to verify failure

Run: python -m pytest tests/test_web_observability.py::test_system_status_includes_training_data tests/test_web_observability.py::test_metrics_endpoint_returns_prometheus_payload_for_ops_admin -q

Expected: FAIL because stale fields/gauges are absent.

• Step 3: Implement stale summary and gauges

Add date-diff calculation against UTC today. Export polyweather_daily_records_stale_days, polyweather_truth_records_stale_days, polyweather_training_features_stale_days, and polyweather_training_data_stale.

• Step 4: Run focused observability tests

Run: python -m pytest tests/test_web_observability.py::test_system_status_includes_training_data tests/test_web_observability.py::test_metrics_endpoint_returns_prometheus_payload_for_ops_admin -q

Expected: PASS.

Task 4: Verification And Backfill

Files:

• Optional create: scripts/backfill_training_settlement.py

• Step 1: Run backend checks

Run: python -m ruff check .

Run: python -m pytest tests/test_training_settlement_service.py tests/test_deployment_runtime_config.py tests/test_web_observability.py -q

• Step 2: Run local one-shot cycle

Run: python -m web.training_settlement_worker --once --lookback-days 10 --cities shanghai

Expected: JSON-like log output with ok=True; local DB should get a fresh row for the current local target date if analysis succeeds.

• Step 3: Production note

Missed forecast snapshots from 2026-06-15 to 2026-06-22 cannot be reconstructed honestly unless archived city analysis payloads exist. The worker restores forward automatic samples; actual-high truth can still be reconciled for supported settlement sources.