feat: Enhance real-time data fetching with cache control and primary MGM sourcing for Ankara, and add technical debt documentation.
This commit is contained in:
2569718930@qq.com
@@ -4,7 +4,7 @@
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://deepwiki.com/yangyuan-zhen/PolyWeather)
|
||||
|
||||
PolyWeather is a weather analysis tool built for prediction markets like **Polymarket**. It aggregates multi-source forecasts, real-time airport METAR observations, a math-based probability engine, and AI-driven decision support to help users evaluate weather trading risks more scientifically.
|
||||
PolyWeather is a multi-source weather analysis and quantification tool. It aggregates high-precision forecasts, real-time airport METAR observations, a math-based probability engine, and AI-driven decision support to provide deep insights for weather-related risk assessment and data-driven decision making.
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/images/demo_ankara.png" alt="PolyWeather Demo - Ankara Live Analysis" width="420">
|
||||
@@ -27,7 +27,8 @@ PolyWeather is a weather analysis tool built for prediction markets like **Polym
|
||||
- **Global Overview**: Real-time Leaflet-based dark-themed map pinpointed to official Polymarket settlement airport coordinates.
|
||||
- **Progressive Background Loading**: Intelligently fetches multi-source data across all cities without hitting API rate limits.
|
||||
- **Rich Visualization**: Chart.js-powered temperature trends with METAR scatter overlay, multi-model comparison bars, Gaussian probability distribution, and dynamic risk badges.
|
||||
- **Cinematic Interaction & Sync**: City selection triggers a smooth fly-to zoom animation. The **Multi-Model Forecast** panel automatically synchronizes with the selected day in the 5-day forecast table.
|
||||
- **Cinematic Interaction & Sync**: City selection triggers a smooth fly-to zoom animation. The **Multi-Model Forecast** panel automatically synchronizes with the selected day in the 5-day forecast table, showing historical model performance and future projections.
|
||||
- **Forced Sync & Cache Control**: For specific regions like Ankara, the dashboard supports a 60-second real-time cache TTL with a manual "Force Refresh" button to bypass global caches and fetch the absolute latest MGM/METAR data.
|
||||
- **Dual-Engine Architecture**: Runs concurrently with the Telegram bot via a FastAPI backend, sharing the same data collection, analysis logic (`analyze_weather_trend`), and AI prompt pipeline.
|
||||
|
||||
### 2. 🧬 Dynamic Ensemble Blending (DEB Algorithm)
|
||||
@@ -36,8 +37,8 @@ The system automatically tracks the historical performance of weather models (EC
|
||||
|
||||
- **Error-Based Weighting**: Dynamically adjusts model weights based on their Mean Absolute Error (MAE) over the past 7 days. Lower error = higher weight.
|
||||
- **Blended Forecast**: Provides a bias-corrected "DEB Blended High Temperature" recommendation.
|
||||
- **Self-Learning**: Requires at least 2 days of observations before activating weight differentiation. Uses equal-weight averaging during cold start.
|
||||
- **Accuracy Tracking**: Use the `/deb` command to view DEB's historical WU settlement hit rate and MAE, compared against individual models.
|
||||
- **Multi-Source Training**: Integrates official regional sources (like Turkey's MGM) into the training pipeline alongside international models (ECMWF, GFS, etc.).
|
||||
- **Accuracy Tracking**: Use the `/deb` command to view DEB's historical settlement hit rate and MAE, compared against individual models.
|
||||
- **Auto-Cleanup**: Only retains the last 14 days of records to prevent unbounded data growth.
|
||||
|
||||
### 3. 🎲 Math Probability Engine (Settlement Probability)
|
||||
@@ -78,9 +79,9 @@ Feeds all weather data into LLaMA 70B, analyzed via a **P0→P4 Priority Chain**
|
||||
### 5. ⏱️ Real-time Airport Observations (Zero-Cache METAR)
|
||||
|
||||
- **Precise Timing**: Extracts actual observation time from raw METAR text (`rawOb`), not the API's rounded `reportTime`. Accurate to the minute.
|
||||
- **Live Passthrough**: Bypasses CDN caching via dynamic headers to obtain first-hand METAR reports.
|
||||
- **Settlement Warning**: Automatically calculates the settlement boundary (X.5 rounding line).
|
||||
- **MGM Fallback**: For Turkish cities (Ankara), falls back to MGM data when METAR is unavailable.
|
||||
- **Live Passthrough**: Bypasses CDN caching via dynamic headers and randomized timestamps to obtain first-hand METAR/MGM reports.
|
||||
- **Settlement Warning**: Automatically calculates the rounding boundary for integer-based settlement (X.5 line).
|
||||
- **MGM Primary (Ankara)**: For Turkish cities like Ankara, PolyWeather uses official MGM data as a primary source for both real-time observations and 5-day hourly forecasts, ensuring maximum local accuracy.
|
||||
- **Anomaly Filtering**: Automatically filters out -9999 sentinel values to prevent garbage data in output.
|
||||
|
||||
### 6. 📈 Historical Data Collection
|
||||
@@ -160,7 +161,7 @@ graph TD
|
||||
Collector --> OM[Open-Meteo Forecast/Ensemble]
|
||||
Collector --> MM[Multi-Model ECMWF/GFS/ICON/GEM/JMA]
|
||||
Collector --> METAR["Live Airport METAR (rawOb)"]
|
||||
Collector --> MGM["MGM Fallback (Turkey)"]
|
||||
Collector --> MGM["MGM Official (Ankara)"]
|
||||
end
|
||||
|
||||
subgraph Algorithm Layer
|
||||
|
||||
+10
-9
@@ -4,7 +4,7 @@
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://deepwiki.com/yangyuan-zhen/PolyWeather)
|
||||
|
||||
PolyWeather 是一款专为 **Polymarket** 等预测市场打造的天气分析工具。它通过聚合多源气象预报、实时机场 METAR 观测，并引入数学概率模型与 AI 决策支持，帮助用户更科学地评估天气博弈风险。
|
||||
PolyWeather 是一款多模型气象分析与量化工具。它通过聚合高精度气象预报、实时机场 METAR 观测，并引入数学概率模型与 AI 决策支持，为气象风险评估和数据驱动的交易决策提供深度洞察。
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/images/demo_ankara.png" alt="PolyWeather 效果展示 - 安卡拉实时分析" width="420">
|
||||
@@ -26,9 +26,10 @@ PolyWeather 是一款专为 **Polymarket** 等预测市场打造的天气分析
|
||||
|
||||
- **全球纵览**：基于 Leaflet 的暗黑全屏实时地图，直接采用官方结算机场经纬度进行追踪与显示。
|
||||
- **渐进式数据流加载**：进图后智能在后台无感知拉取，不触发 API 频率限制。
|
||||
- **丰富的数据可视化**：Chart.js 温度走势图叠加 METAR 实测散点，多模型对比条形图，高斯概率分布条，实时风险等级色彩系统。
|
||||
- **镜头与日期联动**：点击城市平滑飞入。**多模型预报区域**会根据“逐日预报”选中的日期自动刷新（支持未来 3-5 天的各模型对比与 DEB 融合值切换）。
|
||||
- **双引擎共生**：FastAPI 后端与 Telegram Bot 共享同一份分析逻辑（`analyze_weather_trend`）和 AI Prompt 管线，确保两端输出完全一致。
|
||||
- **数据可视化**：Chart.js 温度走势图叠加 METAR 实测散点，多模型对比条形图，高斯概率分布条，实时风险等级色彩系统。
|
||||
- **镜头与日期联动**：点击城市平滑飞入面板。**多模型预报区域**会根据“逐日预报”选中的日期自动刷新，支持查看未来 5 天各模型的历史表现与融合预测值。
|
||||
- **强制同步与缓存控制**：针对安卡拉等重点区域，Web 端支持 60 秒极速缓存 TTL，并提供手动“强制刷新”按钮，可穿透全局缓存获取最及时的 MGM/METAR 实测数据。
|
||||
- **双引擎共生**：FastAPI 后端与 Telegram Bot 共享同一份分析逻辑（`analyze_weather_trend`）和 AI Prompt 管线。
|
||||
|
||||
### 2. 🧬 动态权重集合预报 (DEB 算法)
|
||||
|
||||
@@ -36,8 +37,8 @@ PolyWeather 是一款专为 **Polymarket** 等预测市场打造的天气分析
|
||||
|
||||
- **误差加权**：根据过去 7 天的平均绝对误差（MAE），动态调整各模型的权重。误差越小的模型，话语权越大。
|
||||
- **融合预报**：给出经过历史偏差修正后的"DEB 融合最高温"建议值。
|
||||
- **自学习机制**：系统需要至少 2 天的实测记录才会启动权重分化。冷启动期间以等权平均过渡。
|
||||
- **准确率追踪**：通过 `/deb` 命令查看 DEB 融合预测的历史 WU 结算命中率和 MAE，并与各个单一模型对比。
|
||||
- **多源训练集成**：将区域性官方数据源（如土耳其 MGM）同步纳入 DEB 训练管线，与 ECMWF、GFS 等国际模型共同参与权重博弈。
|
||||
- **准确率追踪**：通过 `/deb` 命令查看 DEB 融合预测的历史结算命中率和 MAE，并与各个单一模型对比。
|
||||
- **自动清理**：只保留最近 14 天的记录，防止数据无限增长。
|
||||
|
||||
### 3. 🎲 数学概率引擎 (Settlement Probability)
|
||||
@@ -78,9 +79,9 @@ PolyWeather 是一款专为 **Polymarket** 等预测市场打造的天气分析
|
||||
### 5. ⏱️ 实时机场观测 (Zero-Cache METAR)
|
||||
|
||||
- **精确时间**：从 METAR 原始报文 (`rawOb`) 中提取真实观测时间，精确到分钟。
|
||||
- **实时穿透**：通过动态请求头绕过 CDN 缓存，获取机场第一手 METAR 报文。
|
||||
- **实时穿透**：通过动态请求头和随机时间戳绕过 CDN 缓存，获取机场第一手 METAR/MGM 报文。
|
||||
- **结算预警**：自动计算结算边界（X.5 进位线），提醒潜在波动。
|
||||
- **MGM 回退**：土耳其城市（安卡拉）METAR 不可用时回退至 MGM 数据。
|
||||
- **MGM 官方直连 (安卡拉)**：针对安卡拉，PolyWeather 已将 MGM 提升为一级数据源，同步采集实时观测数据与 5 天逐小时预报，确保本地精度最大化。
|
||||
- **异常过滤**：自动过滤 -9999 等哨兵值，避免垃圾数据污染输出。
|
||||
|
||||
### 6. 📈 历史数据采集
|
||||
@@ -160,7 +161,7 @@ graph TD
|
||||
Collector --> OM[Open-Meteo 预报/集合]
|
||||
Collector --> MM[多模型 ECMWF/GFS/ICON/GEM/JMA]
|
||||
Collector --> METAR["机场实时 METAR (rawOb)"]
|
||||
Collector --> MGM["MGM 回退 (土耳其)"]
|
||||
Collector --> MGM["MGM 官方直连 (安卡拉)"]
|
||||
end
|
||||
|
||||
subgraph 算法层
|
||||
|
||||
+15
-15
@@ -15,7 +15,7 @@
|
||||
| 概率引擎 | **基本可用** | Gaussian 拟合 + Shock Score + 时间衰减 + 实况锚定 μ + 死盘覆盖 |
|
||||
| AI 决策 | **基本可用** | P0→P4 分级框架 + 预报失准分级 + 高可用降级 |
|
||||
| Telegram Bot | **稳定运行** | 当前最成熟的交互端 |
|
||||
| Web 仪表盘 | **基本可用** | 地图 + 面板 + 图表均可运作，完善度不如 Bot |
|
||||
| Web 仪表盘 | **基本可用** | 全功能地图 + 面板 + 图表，支持 5rd/多模型联动与强制刷新 |
|
||||
| 部署 | **可用** | Docker + 传统 VPS 双通道，`update.sh` 一键部署 |
|
||||
|
||||
### ⚠️ 真实能力边界
|
||||
@@ -24,7 +24,7 @@
|
||||
2. **AI 决策是"结构化的猜测"**。Prompt 再精巧，底层仍是 LLM 概率续写。AI 有时自相矛盾，置信度是自评的，不代表真实概率。
|
||||
3. **Shock Score 是启发式指标**。风向/云量/气压权重（0.4/0.35/0.25）未经回归验证。
|
||||
4. **DEB 自学习窗口只有 7 天**，无法捕捉季节性趋势。
|
||||
5. **Web 端图表数据来自 Open-Meteo 分析值**，已过时段与 METAR 实测存在 1-2°C 偏差（已标注为"OM 模型"并叠加 METAR 散点）。
|
||||
5. **Web 端图表已对齐实测**。利用 MGM 逐小时预报和实测数据，安卡拉等重点城市的日内曲线已实现预报与实况的实时拟合叠加。
|
||||
|
||||
---
|
||||
|
||||
@@ -40,11 +40,11 @@
|
||||
|
||||
### 🟡 中优先级
|
||||
|
||||
| 问题 | 影响 | 建议 |
|
||||
| ------------------ | --------------------------------------------------------------------------- | -------------------------------------- |
|
||||
| 无回测框架 | 无法用历史数据验证算法改动是否真的提升了准确率，改代码全凭直觉 | 用 `fetch_history.py` 的数据搭回测管线 |
|
||||
| 硬编码阈值散落各处 | 死盘 (3°C/1.5°C)、forecast bust (2°C)、Shock Score 权重等直接写在业务代码里 | 提取到配置文件或 `constants.py` |
|
||||
| MGM 数据源不稳定 | 经常 403，fallback 逻辑虽有但增加延迟 | 考虑增加本地缓存或二次 fallback |
|
||||
| 问题 | 影响 | 建议 |
|
||||
| ------------------ | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| 无回测框架 | 无法用历史数据验证算法改动是否真的提升了准确率，改代码全凭直觉 | 用 `fetch_history.py` 的数据搭回测管线 |
|
||||
| 硬编码阈值散落各处 | 死盘 (3°C/1.5°C)、forecast bust (2°C)、Shock Score 权重等直接写在业务代码里 | 提取到配置文件或 `constants.py` |
|
||||
| MGM 数据源不稳定 | 经常 403，数据频率受限 | ✅ **已解决**：采用自定义 Header + 随机时间戳 + 5级 API 遍历抓取，稳定性大幅提升 |
|
||||
|
||||
### 🟢 低优先级
|
||||
|
||||
@@ -58,14 +58,14 @@
|
||||
|
||||
## 三、未完成功能
|
||||
|
||||
| # | 功能 | 说明 |
|
||||
| --- | ------------------- | -------------------------------------------------------- |
|
||||
| 1 | 历史数据消费 | `fetch_history.py` 可采集 3 年数据，但没有任何模型在使用 |
|
||||
| 2 | 结算自动对账 | 系统不知道昨天的预测对不对（`/deb` 需人工触发） |
|
||||
| 3 | 交易信号输出 | 系统只给分析建议，不输出可执行的交易信号 |
|
||||
| 4 | Web 身份认证 | 任何人都可以访问 Web 面板 |
|
||||
| 5 | 主动推送 | 预报崩盘或结算边界预警时不会主动通知用户 |
|
||||
| 6 | 完整日内 METAR 走势 | `trend.recent` 只保留最近 4 条，无法展示完整日内观测曲线 |
|
||||
| # | 功能 | 说明 |
|
||||
| --- | ------------------- | ---------------------------------------------------------------- |
|
||||
| 1 | 历史数据消费 | `fetch_history.py` 可采集 3 年数据，但没有任何模型在使用 |
|
||||
| 2 | 结算自动对账 | 系统已支持通过 `/deb` 查看命中率，但尚未实现自动生成昨日总结推送 |
|
||||
| 3 | 交易信号输出 | 系统只给分析建议，不输出可执行的交易信号 |
|
||||
| 4 | Web 身份认证 | 任何人都可以访问 Web 面板 |
|
||||
| 5 | 主动推送 | 预报崩盘或结算边界预警时不会主动通知用户 |
|
||||
| 6 | 完整日内 METAR 走势 | `trend.recent` 只保留最近 4 条，无法展示完整日内观测曲线 |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user