diff --git a/README.md b/README.md index 8b081c04..12e070c0 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ [![个人开源](https://img.shields.io/badge/%E4%B8%AA%E4%BA%BA%E5%BC%80%E6%BA%90-green?style=for-the-badge&labelColor=red)](https://github.com/shy3130/tick-stock-panel) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](./LICENSE) [![Engine: Polars](https://img.shields.io/badge/Engine-Polars-6f42c1?style=for-the-badge)](https://pola.rs/) -[![Open API](https://img.shields.io/badge/Open_API-61_%E7%AB%AF%E7%82%B9-00b386?style=for-the-badge)](./docs/open-platform-plan.md) +[![Open API](https://img.shields.io/badge/Open_API-65_%E7%AB%AF%E7%82%B9-00b386?style=for-the-badge)](./docs/open-platform-plan.md) [![MCP](https://img.shields.io/badge/MCP-12_%E5%B7%A5%E5%85%B7-7c3aed?style=for-the-badge)](./mcp-server/README.md) [![Build: Docker CI](https://github.com/shy3130/tick-stock-panel/actions/workflows/docker.yml/badge.svg)](https://github.com/shy3130/tick-stock-panel/actions/workflows/docker.yml) @@ -24,7 +24,7 @@ shy3130%2Ftick-stock-panel | Trendshift - + @@ -50,7 +50,7 @@ About RunningHub API — 产品介绍 -**本项目由 [RunningHub](https://www.runninghub.ai/call-api?source=github&inviteCode=edt5wh7c) 提供支持** · 单一接口直连 400+ 主流大模型 · 免费测试 +**本项目由 [RunningHub](https://www.runninghub.ai/call-api?source=github&inviteCode=edt5wh7c) 提供支持** · 单一接口直连 400+ 主流大模型 · [DeepSeek v4.1 Flash 6 折](https://www.runninghub.ai/call-api?source=github&inviteCode=edt5wh7c) · 免费测试 @@ -71,7 +71,7 @@ | 策略、回测、监控各用一套工具,口径对不上 | 全站统一 **enriched 数据口径**:策略 → 回测 → 监控 → 复盘一条链 | | 盘中异动靠人盯盘,错过就是错过 | **竞价/盘中/偏移**全时段异动 + 实时弹窗、语音播报、飞书推送 | | 想查个数据要在几个页面之间来回点 | **AI 对话助手**:一句话问出全站数据,取数过程逐条可见、可展开核对;还能经确认卡放行生成信号、跑回测、补数据 | -| 想基于面板数据做自己的工具/机器人,只能爬页面 | **开放接口 + MCP**:Token 六档权限、61 端点契约化、SSE 事件流,AI 客户端即插即用 | +| 想基于面板数据做自己的工具/机器人,只能爬页面 | **开放接口 + MCP**:Token 六档权限、65 端点契约化、SSE 事件流,AI 客户端即插即用 | | 付费终端贵、云端平台数据出不了本地 | **自托管**:Docker 单容器,数据全部落在本地 Parquet,零运维 | ## ✨ 核心功能 @@ -141,14 +141,14 @@ + + + + +
61
开放 API 端点
65
开放 API 端点		 6
Token 权限档		 12
MCP 工具		 2400+
测试全绿 · CI 守护		 -**🧰 数据扩展**
插件化数据源, 扩展字段成页, 按日历史回补 +**🧰 数据扩展**
插件化数据源, 扩展字段成页, 两级拉取配方, 按日历史回补
-**🌐 开放接口**
Token 六档权限, 61 端点契约化, 读+写闭环 +**🌐 开放接口**
Token 六档权限, 65 端点契约化, 读+写闭环 		@@ -160,61 +160,28 @@ **🧠 AI 对话助手**
一句话问全站数据, 工具足迹逐条可核对 +
+ +**🧩 扩展页面**
11 个官方页面单文件化, 删目录即卸载、复制即导入 + +		 + +**📦 页面包**
页面打包产物上传即挂载, 免重建宿主, 刷新生效 + +		 + +**🎛️ 菜单设置**
显隐/排序 + 内置/自定义/扩展三类管理与架构徽标 +
-
-📦 主要页面与功能 - -**📊 行情总览** - -- **看板** Dashboard — 市场情绪评分 + 涨跌/成交额榜单 + 概念/行业领涨领跌(点击板块直达成分股,领涨股带涨跌幅) + 大盘异动事件流,一日全貌; **布局可自定义** — 12 列吸附网格, 组件拖拽换位/角柄自由调宽高, 可增删组件、嵌入外部链接(iframe 沙箱), 跨设备同步, 一键恢复默认 -- **自选** Watchlist — 自选池,多分组管理(M:N),表格/卡片双视图,换手/量比/RSI 等实时指标,按档位分流实时刷新 -- **指数** Indices — 沪深指数浏览与同步 - -**🔍 策略与回测** - -- **策略** Screener — Polars 毫秒级扫描全 A 股,日线/分钟策略统一单池,按策略声明周期自动路由执行 -- **回测** Backtest — 四种研究视图: - - **因子回测** — IC/IR、分层收益、多空组合,62+ 因子目录先筛掉无效指标 - - **策略回测** — 净值曲线、回撤、夏普、胜率、盈亏比、蒙卡回撤,T+1/手续费/滑点/止损,SSE 流式进度;评分因子策略附带「因子归因」(胜/败单入场信号日因子对比) - - **分钟策略回测** — 逐交易日回放信号、分钟收盘入场,分钟级成交明细 - - **验证** — 参数敏感性与滚动样本外 - - 研究闭环:结果导出 CSV(概要/净值/交易明细/分标的统计) → 保存候选 → **一键载入复测** -- **因子** Factors — 检验/因子库/编辑器/组合四 tab:IC·分层·Newey-West 检验、自定义 DSL 因子(25 算子点选、双语字段、我的因子模板)、版本与生命周期管理;因子库可**一键生成排名策略**,策略触发器可直接引用因子条件 -- **挖掘** Mining — 嵌套样本外因子与策略挖掘:训练区间因子方向重估 + 相关性去重 + 多因子排名组合搜索,自有策略作对照轨;候选入库,显式确认后才发布,永不自动上线 - -**📈 个股与板块分析** - -- **个股分析** Stock Analysis (Beta) — 日K + 9 类关键价位 + AI 四维分析(技术/基本面/财务/消息面) -- **财务分析** Financials — 利润表/资负表/现金流/关键指标(多源并集合并,fuyao 财务四表适配) + AI 解读 -- **概念分析 / 行业分析** — ths 维度涨幅轮动矩阵 + 领涨/领跌主线 + 个股穿透 -- **市场环境** Regime — 情绪周期 6 阶段(冰点/启动/主升/高潮/退潮/修复,连板梯队驱动,EMA 平滑 + 2 日确认)+ 概念/行业主线排名,与 5 档环境分并存 -- **连板梯队** Limit Up Ladder — 连板层级统计 + 概念/行业分布 + 封单监控(可切换连跌梯队) - -**🔔 监控与复盘** - -- **监控中心** Monitor — 策略/个股信号/价格/异动四类规则,支持自选分组作用域,盘中实时弹窗 + 语音播报(播报个股名称与信号) + 触发记录持久化 -- **持仓提醒** Lots — 记录个股/ETF 买入批次,自动生成止盈止损/到期监控规则 -- **信号库** Signals — 全部为用户定义的条件信号(字段+运算符+阈值,支持因子条件、涨跌停判定价与 AI 生成),供策略触发器/回测/监控统一取用;新装为空库,可自建或导入 -- **异动监控** Abnormal Moves — 按交易时间线三 tab: - - **竞价异动** — 同花顺盘前风向标(含当日/次日真实收益对照、追高风险标记)+ 全市场竞价扫描(待采集任务) - - **盘中异动** — 涨停/炸板/翘板/跌停/新高/新低/放量当日信号聚合,零新增采集 - - **偏移异动** — 交易所异动偏离值口径(主板 3 日 ±20%、创业板/科创板 ±30%、北交所 ±40%;10 日 +100%/−50%、30 日 +200%/−70%),实时接近度 -- **复盘** Review (Beta) — 盘后 AI 自动生成市场复盘,注入龙虎榜资金动向与盘前风向标对照;可定时执行、推送飞书、下载 Markdown - -**🗄️ 数据与扩展** - -- **数据** Data — 本地数据画像与同步状态(维表/日K/除权/Enriched/指数/ETF/分钟K/财务),盘后管道与历史扩展 -- **扩展分析** (动态菜单) — 把任意第三方/扩展数据字段配成一级菜单,与内置数据同台分析 -- **设置** Settings — 数据源与能力检测(能力路由矩阵、档位徽章)、AI 接口、实时监控、扩展页面、开放接口(Token 管理)、菜单与系统设置 - -**🤖 AI 助手** - -- **AI 对话助手** — 悬浮球 / 侧栏 AI 徽标旁入口 / ⌘K 呼出; 21 个查询工具覆盖个股·大盘·板块·自选·持仓·信号·策略·因子·数据完整性·扩展数据表, 外加 4 个**动作工具**(生成信号策略 / 回测 / 加自选 / 数据补全)——写操作先弹**确认卡**(完整待执行参数可见, 确认/取消/120s 超时自动取消)才执行; 逐字流式输出 + 工具调用足迹卡(参数与耗时可展开核对), 每条回答附风险与数据口径提示; 完全解耦的扩展模块, 删除目录即卸载 - -
+> 📄 **19 个页面的完整功能细节**见 [docs/features.md](./docs/features.md) 与 [screenshots/](./screenshots/README.md) —— 上表之外, 回测四种研究视图、异动三时间线、页面包运行时分发等都在其中。 --- @@ -257,7 +224,13 @@ ## 🏗️ 架构 -**TSP 是一个"数据工程优先"的本地量化工作台** —— 数据源、口径、执行、开放通道做成一套自洽的架构,功能只是长在上面的应用。八个特点,一句话一个: +**TSP 的一切设计围绕一条主线:核心底座 + 扩展层。** + +- **核心域(8 个)** 拥有数据与语义 —— 数据底座、能力路由、指标与执行、事件与开放通道。它们是自洽的最小系统,不依赖任何外围。 +- **扩展域(5 条通道)** 只消费核心契约 —— 数据源插件、扩展数据表、自定义策略/信号、页面扩展、后端 custom 模块。**可整体替换或移除而不伤核心**:删除目录即卸载、复制目录即导入。 +- 你看到的 19 个页面里,**11 个官方页面本身就是扩展**(单文件 `frontend/src/custom//extension.tsx`)—— 监控中心页面可以整个换掉,规则引擎与推送管道属核心照常运行。**连 AI 对话助手都是扩展域的参考实现。** + +> 一句话:核心是引擎,其余皆插件。下面这张图,先看懂虚线框与实线框的边界,就看懂了 TSP。 | 架构特点 | 一句话说明 | | :--- | :--- | @@ -265,10 +238,8 @@ | 🧬 **单一 enriched 口径** | 只存 15 列基础数据、现算 68 列指标信号;策略 → 回测 → 监控 → 复盘吃同一份数据 | | 📁 **文件型零运维存储** | Parquet + DuckDB,单容器零外部数据库,数据 100% 落在本地 | | 🔁 **回测 = 实盘同路径** | 策略执行只此一条 `StrategyEngine.run`;分钟策略逐日回放 +「当日已知」纪律,杜绝未来函数 | -| 🏛️ **网关居中的开放单体** | 面板密码会话与 Token 网关双通道并行;61 端点契约快照 + CI 守护,SSE 事件流与 MCP 平等开放 | -| 🧩 **核心域 / 扩展域边界** | 8 个核心域拥有数据与语义,5 条扩展通道只消费契约——删除目录即整体卸载,零核心改动 | +| 🏛️ **网关居中的开放单体** | 面板密码会话与 Token 网关双通道并行;65 端点契约快照 + CI 守护,SSE 事件流与 MCP 平等开放 | | 🚀 **重活进程隔离** | 回测跑在 spawn worker 子进程,刷新/切页重连不丢任务,主服务永不被计算卡住 | -| 📅 **交易日探针** | 节假日自动停掉实时轮询与分钟增量,零无效请求 | ### 总览 · 网关居中的开放单体 @@ -284,33 +255,36 @@ flowchart TB APPC["外部程序
脚本 · 看板"] end - GW{{"开放网关
Token · 六档 scope
限流 120/min · 61 端点契约"}} + GW{{"开放网关
Token · 六档 scope
限流 120/min · 65 端点契约"}} API["应用层 · ⚡ FastAPI
REST · SSE 事件流"] - subgraph CORE["核心域 · 数据与语义的所有者"] - S1[("能力路由 · 插件化数据源
enriched 分区存储
Parquet · DuckDB")] - S2["指标流水线 · 存15现算68列
策略 · 因子 · 回测 · 挖掘"] - S3["监控引擎 · 事件总线 · SSE 出站
扩展数据 · 认证配置"] - S1 -->|"enriched 口径"| S2 - S2 -->|"信号 · 事件"| S3 - end - - subgraph EXT["🧩 扩展域 · 5 条接入通道 —— 只消费开放契约, 零核心改动"] + subgraph BODY[""] direction LR - X1["数据源插件"] - X2["扩展数据表
读 + 写"] - X3["自定义策略
与信号"] - X4["声明式页面
前端插槽挂载"] - X5["custom 后端模块
★ AI 助手参考实现"] + subgraph CORE["⬛ 核心域 — 数据与语义的所有者 (8 个)"] + direction TB + S1[("① 数据底座
能力路由 · 插件化数据源
Parquet · DuckDB")] + S2["② 计算与执行
指标流水线(存15现算68列)
策略 · 因子 · 回测 · 挖掘"] + S3["③ 事件与开放
监控引擎 · 事件总线 · SSE"] + S1 -->|"enriched 口径"| S2 + S2 -->|"信号 · 事件"| S3 + end + subgraph EXT["🧩 扩展域 — 5 条通道 · 只消费契约, 可整体替换/移除"] + direction TB + X1["数据源插件
TickFlow / fuyao / TdxQuant"] + X2["扩展数据表
两级拉取 · 读+写"] + X3["自定义策略
与信号"] + X4["页面扩展
11 官方页单文件 + 运行时页面包"] + X5["custom 后端模块
★ AI 助手参考实现"] + end end UI -->|"密码会话"| API MCP -->|"Bearer + scope"| GW APPC -->|"Bearer + scope"| GW GW -->|"契约放行"| API - API ==> S1 - S3 -.-> X3 + API ==>|"数据与语义"| CORE + EXT ==>|"只消费开放契约 · 零核心改动"| CORE classDef client fill:#eef2ff,stroke:#6366f1,color:#312e81 classDef gate fill:#faf5ff,stroke:#8b5cf6,color:#6b21a8,stroke-width:2px @@ -331,8 +305,9 @@ flowchart TB class X5 star style CLIENTS fill:#f5f3ff,stroke:#c4b5fd,stroke-width:1.5px,color:#4c1d95 - style CORE fill:#f8fafc,stroke:#cbd5e1,stroke-width:1.5px,color:#0f172a - style EXT fill:#f8fafc,stroke:#94a3b8,stroke-width:1.5px,stroke-dasharray:6 4,color:#334155 + style BODY fill:transparent,stroke:transparent + style CORE fill:#f8fafc,stroke:#334155,stroke-width:2px,color:#0f172a + style EXT fill:#fefce8,stroke:#a16207,stroke-width:1.5px,stroke-dasharray:7 4,color:#713f12 ``` ### 核心域 / 扩展域边界 @@ -342,23 +317,22 @@ flowchart TB | | 内容 | 划线依据 | | :------------------ | :--------------------------------------------------------------------------------------------------------------------- | :---------------------------------------- | | **核心域 · 8** | 数据底座 / 能力路由 / 订阅锚点(自选) / 策略与回测 / 因子与挖掘 / 环境数据生产 / 监控推送管道 / 扩展数据存储 + 认证配置 | 被多个模块消费其数据与语义,自身不依赖外围 | -| **扩展域 · 5 通道** | 数据源插件 · 扩展数据表 · 自定义策略/信号 · 声明式页面/前端插槽 · 后端 custom 模块 | 只消费核心契约,可整体替换/移除而不伤核心 | +| **扩展域 · 5 通道** | 数据源插件 · 扩展数据表 · 自定义策略/信号 · 页面扩展(单文件源码 + 运行时页面包) · 后端 custom 模块 | 只消费核心契约,可整体替换/移除而不伤核心 | -设置 → 菜单设置里每个页面带 `核心 / 扩展` 徽标与归属原因;11 个内置页面(概念/行业/个股/财务分析、监控中心、异动、持仓提醒、模拟盘、信号库、复盘、连板梯队)都是**可被二开替换的消费页** —— 例如监控中心页面可换,规则引擎与推送管道属核心照常运行。详见 [docs/open-platform-plan.md](./docs/open-platform-plan.md)。 +设置 → 菜单设置里每个页面带 `核心 / 扩展` 架构徽标与 `内置 / 自定义 / 扩展` 类型徽标;11 个官方页面(概念/行业/个股/财务分析、监控中心、异动、持仓提醒、模拟盘、信号库、复盘、连板梯队)已全部落地为 `frontend/src/custom//extension.tsx` **单文件扩展** —— 删除目录即卸载、复制目录即导入;例如监控中心页面可换,规则引擎与推送管道属核心照常运行。新装默认隐藏这些扩展页面(菜单设置一键开启),老用户升级自动保持原菜单位置与排序。免宿主重建的运行时分发走**页面包**(设置 → 页面包,契约见 [docs/page-sdk.md](./docs/page-sdk.md))。详见 [docs/open-platform-plan.md](./docs/open-platform-plan.md)。 ### 关键机制 | 机制 | 说明 | | :------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **能力路由矩阵** | 各数据集按源声明能力独立路由,注册表集中定义、可扩展:TICKFLOW 档位探测(None/Free/Starter/Pro/Expert)+ 插件源能力声明,fail-closed(声明 `pct_unit` 未声明即拒)。同一数据集可随时换源,指标与回测口径不变 | -| **交易日探针** | fuyao 交易日历(确定性,含调休)→ tickflow 全市场行情时间戳探针(OR 语义)→ 工作日兜底;节假日自动停掉实时轮询与分钟增量,零无效请求 | | **财务多源合并** | 按 `(symbol, period_end)` 报告期累积,多源取并集、逐列按公告日取最新(PIT);公告前一律空值,绝不填 0 | | **非路由数据集直连** | 龙虎榜/盘前风向标/交易日历等 fuyao 专有能力不进路由矩阵,由独立服务直连消费——按日 JSON 缓存(历史不可变)、交易日回退、四态降级 | | **回测执行隔离** | 回测在 spawn worker 子进程运行,持久 run ID,刷新/切页重连不丢任务;子进程结果消息经锁保护回传 | -| **分层缓存** | enriched 读取时现算指标(存储仅 15 列基础数据,现算 68 列指标与信号)+ 进程内快照缓存;扩展字段按日分区快照,页面即配即用 | | **开放网关** | Token 通道与面板密码会话并行互不影响:网络门禁(默认仅本机/内网, `API_TOKEN_LOCAL_ONLY=0` 放开) → 认证 → 六档 scope 校验 → 每 Token 滑动窗口限流(默认 120 次/分,O(1) 内存)→ 放行;管理面(数据同步/表结构/设置)永不开放给 Token | | **事件总线** | 进程内发布/订阅(慢消费者丢旧保新,广播失败不反噬主流程),告警落盘唯一入口已挂接;SSE 流按票据 scope 过滤,票据一次性 60 秒过期 | -| **完全解耦扩展** | 后端 `app/custom/<包>/` 启动时自动发现、注册独立路由(版本不符或 setup 失败即隔离跳过), 前端 `src/custom/*/extension.tsx` 构建时自动挂载到插槽; 删除目录即整体卸载, 扩展无需改动核心 —— **AI 对话助手**即该机制的参考实现 | +| **交易日探针** | fuyao 交易日历(确定性,含调休)→ tickflow 全市场行情时间戳探针(OR 语义)→ 工作日兜底;节假日自动停掉实时轮询与分钟增量,零无效请求 | +| **扩展自动发现** | 后端 `app/custom/<包>/` 启动时自动发现、注册独立路由(版本不符或 setup 失败即隔离跳过);前端 `src/custom//extension.tsx` 单文件即完整扩展(组件+路由+导航),构建时自动挂载 —— 删除目录即整体卸载、复制目录即导入;运行时分发走**页面包**(上传即挂载,免重建宿主) | ### 技术栈 @@ -367,8 +341,8 @@ flowchart TB | **后端** | ![Python](https://img.shields.io/badge/Python_3.11+-3776AB?logo=python&logoColor=white) ![FastAPI](https://img.shields.io/badge/FastAPI-009688?logo=fastapi&logoColor=white) ![Pydantic](https://img.shields.io/badge/Pydantic_v2-E92063?logo=pydantic&logoColor=white) APScheduler · sse-starlette | | **数据** | ![Polars](https://img.shields.io/badge/Polars-CD882D?logo=polars&logoColor=white)（计算）· ![DuckDB](https://img.shields.io/badge/DuckDB-FFF100?logo=duckdb&logoColor=black)（查询）· Parquet（存储） | | **回测** | 自研仓位模拟引擎(T+1/费用/滑点/分钟回放)· vectorbt(部分路径) | -| **数据源** | [TickFlow](https://tickflow.org/auth/register?ref=V3KDKGXPEA) 官方 SDK · fuyao(同花顺 REST) · 插件化扩展(YAML 自定义源) | -| **AI**(可选) | ![OpenAI兼容](https://img.shields.io/badge/OpenAI兼容-412991?logo=openai&logoColor=white) DeepSeek / 通义 / Ollama 等 · 策略生成 / 报告 / **对话助手**(助手依赖工具调用能力, 需 OpenAI 兼容接口) | +| **数据源** | [TickFlow](https://tickflow.org/auth/register?ref=V3KDKGXPEA) 官方 SDK · fuyao(同花顺 REST) · [TdxQuant](https://help.tdx.com.cn/quant/)(通达信量化,实时/五档/日K/复权/分钟) · 插件化扩展(YAML 自定义源) | +| **AI**(可选) | ![OpenAI兼容](https://img.shields.io/badge/OpenAI兼容-412991?logo=openai&logoColor=white) Chat Completions 与 Responses 双协议 · DeepSeek / 通义 / Ollama 等 · 策略生成 / 报告 / **对话助手**(助手依赖工具调用能力, 需 OpenAI 兼容接口) | | **MCP** | [mcp-server](./mcp-server/README.md)(官方 SDK 2.x, stdio) — 12 工具按 scope 暴露,权限裁决复用开放网关 | | **前端** | ![React 18](https://img.shields.io/badge/React_18-61DAFB?logo=react&logoColor=black) ![Vite](https://img.shields.io/badge/Vite-646CFF?logo=vite&logoColor=white) ![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white) ![Tailwind](https://img.shields.io/badge/Tailwind-06B6D4?logo=tailwindcss&logoColor=white) Tanstack Query · [Lightweight Charts](https://www.tradingview.com/lightweight-charts/)(TradingView 开源) · ![ECharts](https://img.shields.io/badge/ECharts-AA344D?logo=apacheecharts&logoColor=white) · dnd-kit | | **部署** | ![Docker](https://img.shields.io/badge/Docker_单容器-2496ED?logo=docker&logoColor=white) 两阶段构建,前端 dist 拷进后端镜像 | @@ -390,11 +364,10 @@ flowchart TB | 回测 | 子进程隔离不卡主服务,刷新重连不丢任务 | spawn worker + 持久 run ID + 重活并发闸 | | 部署与兼容 | 单容器**零外部数据库**(Parquet + DuckDB 文件型) · 新老 CPU 双内核 | polars rtcompat 兼容内核,运行时自动探测 | -### 开放层(0.3.2 新架构带来) +> 开放层的性能(网关裁决 O(1)、契约快照守护、亚秒级事件推送等)见下方[开放能力](#-开放能力open-api--mcp)章节。 -| 能力 | 表现 | 靠什么 | -| :----------- | :---------------------------------- | :----------------------------------------- | -| 开放 API | **61 端点** · 每 Token 120 次/分 | 进程内滑动窗口限流,O(1) 内存,零外部依赖 | +----------- | :---------------------------------- | :----------------------------------------- | +| 开放 API | **65 端点** · 每 Token 120 次/分 | 进程内滑动窗口限流,O(1) 内存,零外部依赖 | | 网关裁决 | 认证→scope→限流,纯内存单函数完成 | 单一裁决点(`evaluate`),无数据库往返 | | 契约稳定性 | 机器可读契约 + **快照测试**守护 | 开放面增删必须显式改快照,CI 拦截无意识变更 | | 数据写闭环 | 外部写入复用管理端同一校验/落盘路径 | `write:ext` 只写行数据,表结构锁死管理面 | @@ -427,7 +400,7 @@ flowchart TB | **API Token** | `设置 → 开放接口` 创建,明文只显示一次,SHA-256 哈希存储,吊销立即生效 | | **网络门禁** | Token 通道默认仅本机/内网调用,公网一律 403(`API_TOKEN_LOCAL_ONLY=0` 显式放开,远程推荐 SSH 隧道);门禁先于验签,不泄露 Token 有效性也不消耗限流额度 | | **六档 scope** | `read:market` / `read:ext` / `write:ext` / `read:analysis` / `run:backtest` / `paper:trade`,管理面永不开放给 Token | -| **61 端点契约** | `GET /api/openapi.json?tier=a` 机器可读,Postman/代码生成即用;**契约快照测试 + CI 守护**,开放面变更必须显式确认 | +| **65 端点契约** | `GET /api/openapi.json?tier=a` 机器可读,Postman/代码生成即用;**契约快照测试 + CI 守护**,开放面变更必须显式确认 | | **数据写闭环** | `write:ext` 程序化写入扩展表行数据(与内置数据同台分析),表结构锁死在管理面 | | **SSE 事件流** | 60 秒一次性票据订阅实时告警推送,票据只继承 scope 不放大权限 | | **MCP 服务器** | 12 个精选工具,按 Token scope 暴露;模拟盘交易刻意不交给 AI | @@ -452,7 +425,7 @@ curl -H "Authorization: Bearer tsp_xxxx" \ | 打开方式 | 说明 | | :---------------- | :----------------------------------- | | **悬浮球** | 可拖动, 位置记忆; 生成中带状态指示点 | -| **AI 徽标旁入口** | 侧栏顶部模型徽标右侧, 一键展开 | +| **侧栏「AI 助手」行** | 左上角状态区 (AI 配置行下方), 一键展开 | | **⌘K / Ctrl+K** | 全局快捷键随时呼出, Esc 关闭 | **能问什么** — 21 个查询工具覆盖全站页面能力, 4 个动作工具经确认卡放行: @@ -470,19 +443,13 @@ curl -H "Authorization: Bearer tsp_xxxx" \ **交互设计** -- **逐字流式输出** — 文本按 token 逐步渲染, 长回答不再"整段蹦出" -- **分组快捷建议** — 新会话页按 行情与大盘 / 我的与个股 / 策略与信号 / 数据与扩展 分组给出 18 条按文字自适应排布的建议(含策略对比、个股横向对比), 并提示扩展内容同样可以问答 - **工具足迹卡** — 每次调用的工具名、参数、耗时、结果摘要均可展开核对; 取数可核对是设计铁律 -- **走势双图** — 个股问答自动附工具真实序列的走势小卡(日K蜡烛 + 当日分时), 并排展示且日K固定左侧、分时右侧; 宽表格(>4 列)按内容取宽不折行, 超宽时横向滚动 -- **动作确认卡** — 生成信号/回测/加自选/数据补全等写操作先弹确认卡: 动作名 + 影响说明 + **完整待执行参数**, 你点「确认执行」才会真正运行; 拒绝或 120 秒超时自动取消, 模型被明确告知此时不得重试 -- **轮次检查点** — 工具调用不设硬上限, 连续 N 轮未完成时弹「继续 / 停止」选择卡(120 秒未选自动停止), 继续则再跑一个周期, 已生成内容始终保留; 检查点轮数可在 设置→AI 调整(默认 100, 最小 5, 0 = 不检查) -- **非模态面板** — 从页面右缘滑入, 默认 720px, 左缘拖拽调宽(宽度记忆), 边看行情边问 -- **会话历史** — 本地保存, 支持多会话切换与删除, 刷新不丢 -- **固定合规提示** — 每条回答完成后附风险提示与数据口径提示 +- **动作确认卡** — 生成信号/回测/加自选/数据补全等写操作先弹确认卡(完整待执行参数可见), 你点「确认执行」才真正运行; 拒绝或 120 秒超时自动取消 +- **逐字流式 + 非模态面板** — 从页面右缘滑入可拖宽(宽度记忆), 边看行情边问; 新会话带分组快捷建议, 个股问答附走势双图(日K + 分时) +- **会话历史跨设备** — 服务端注册表同步(回测/挖掘/优化/滚动任务同步接回), 本地存储降级为缓存 +- **固定合规提示** — 每条回答完成后附风险提示与数据口径提示; 轮次检查点与连续执行上限可在设置调整 -**完全解耦的扩展模块** - -助手是项目扩展系统的参考实现: 后端 `app/custom/assistant/`(启动时自动发现并注册独立路由)与前端 `src/custom/assistant/`(构建时自动挂载到插槽)各自独立, **删除目录即整体卸载**。数据补全工具与数据页**共用同一条触发路径**(为此在核心 API 层抽取了两个共用触发函数, 端点行为不变), 单飞去重与进度上报天然一致。未配置 AI Key 或使用不支持工具调用的供应商(如 Codex CLI)时 fail-closed, 直接提示前往设置页。 +**扩展域的参考实现** — 后端 `app/custom/assistant/` + 前端 `src/custom/assistant/` 各自独立, 删除目录即整体卸载(见上方[架构](#-架构)); 数据补全工具与数据页共用同一条触发路径, 单飞去重天然一致。未配置 AI Key 或供应商不支持工具调用时 fail-closed, 直接提示前往设置页。 > ⚠️ 助手是数据分析工具, 不提供买卖指令; 涉及交易决策的问题会转换为客观的技术/财务状态、关键价位、风险因素与条件情景。 @@ -549,6 +516,7 @@ CODEX_CLI_VERSION=0.144.3 docker compose up --build > Codex CLI 模式允许 TickFlow 容器读取本机 Codex 登录凭据，仅应在受信任的本机环境启用。凭据目录以只读方式挂载，不会写入镜像。 + > 📖 Docker 进阶、老 CPU 兼容、访问密码设置等见 [docs/deployment.md](./docs/deployment.md)。 @@ -602,6 +570,7 @@ xattr -dr com.apple.quarantine /Applications/TSP.app 5. **监控中心**配规则,盘中实时弹窗 + 持久化记录;**异动监控**覆盖竞价/盘中/偏移全时段 6. 配好 AI Key 后,**悬浮球 / ⌘K** 呼出 **AI 对话助手**,直接问「今天市场怎么样」「我的自选表现如何」「我的扩展数据表里有什么」,或让它「生成一个放量站上 20 日线的信号」「检查一下数据完整性」——写操作会先弹确认卡, 你点确认才执行 7. 想接自己的程序?**设置 → 开放接口**建个 Token,照着 [examples/open-api](./examples/open-api/README.md) 十分钟跑通第一个脚本;AI 玩家直接配 [MCP](./mcp-server/README.md) +8. 想定制页面?核心之外 11 个官方页面都是单文件扩展(新装默认隐藏,在**设置 → 菜单设置**开启);自己开发的页面用 [frontend/page-pack-template](./frontend/page-pack-template) 打包,经**设置 → 页面包**上传即用,宿主无需重建 --- @@ -631,6 +600,7 @@ PORT=3018 # 服务端口 | **v0.3** | 能力路由矩阵 · fuyao 数据源(财务/龙虎榜/风向标) · 分钟策略与回测 · 交易日探针 · 全时段异动中心 · 回测导出与候选复测 | ✅ | | **AI 助手** | 对话式数据问答与操作: 21 查询 + 4 动作工具(确认卡放行, 含生成信号/回测/加自选/数据补全) · 逐字流式 + 工具足迹卡 · 完全解耦扩展模块 | ✅ | | **v0.3.2 开放能力版** | API Token + 六档 scope 网关 · 61 端点契约(快照+CI 守护) · 扩展数据读写闭环 · SSE 事件流 · MCP 服务器 · 核心/扩展域边界标注 | ✅ | +| **v0.3.6** | [TdxQuant](https://help.tdx.com.cn/quant/) 数据源(realtime/五档/日K/复权/分钟) · 11 官方页面单文件扩展化 + 运行时页面包 · 扩展表两级拉取配方(概念/行业出厂即用) · AI Responses 协议与会话跨设备 · 情绪评分口径可配置 · 财务画像手动拉取 · 策略池叠加分组 · 65 端点契约 | 🚧 | --- @@ -651,31 +621,37 @@ PORT=3018 # 服务端口 | [docs/market-phase.md](./docs/market-phase.md) | 市场情绪周期 6 阶段与概念/行业主线识别的口径与设计 | | [docs/plugin-development.md](./docs/plugin-development.md) | 数据源插件开发规范(以 fuyao 为参考实现) | | [docs/secondary-development.md](./docs/secondary-development.md) | 代码二次开发、前端插槽、后端策略接口与 AI 开发模板 | +| [docs/page-sdk.md](./docs/page-sdk.md) | 页面包开发指南:产物契约、能力桥 ctx、打包与上传、安全边界 | | [backend/app/strategy/prompts/strategy-guide.md](./backend/app/strategy/prompts/strategy-guide.md) | 策略开发完整规范(AI 生成与手写) | --- -## ❤️ 支持项目 +## ❤️ 支持与交流 -
+ + + + + +
如果这个项目对你有帮助,欢迎请作者喝杯咖啡 ☕ -微信赞赏码 · 感谢道友支持 愿一路长红 +
+ +微信赞赏码 · 感谢道友支持 愿一路长红 作者精力有限,优先响应赞助回馈,希望理解 📈 - - -## 💬 交流群 - -
+
欢迎加入交流群,一起讨论交流 · 个人维护了一些个性化接口统一公布在群公告 -交流群二维码 · 个人维护的个性化接口见群公告 +
- +交流群二维码 · 个人维护的个性化接口见群公告 + +
--- @@ -687,10 +663,16 @@ PORT=3018 # 服务端口 [MIT](./LICENSE) © tick-stock-panel contributors +> **分发与二次开发须知**:依据 MIT 协议,分发本软件或其任何部分(包括 fork 后发布、修改再发布、封装为自有产品或服务)时,必须完整保留原始版权声明与许可文本。未经版权所有者书面许可,移除、遮蔽或篡改版权署名、许可声明,或在再分发时隐匿本项目来源,均构成协议违约与著作权侵权,版权所有者保留依法追究责任的权利。 +> +> 协议全文与第三方组件许可清单亦可见应用内「设置 → 关于 → 开源许可」。 + 本项目依赖 [TickFlow](https://tickflow.org/auth/register?ref=V3KDKGXPEA) 提供数据服务,使用前请遵守其服务条款 内置数据源插件 [fuyao](https://fuyao.aicubes.cn/docs/api-reference/) 提供同花顺 REST 数据接口(行情 / 财务 / 龙虎榜 / 盘前风向标 / 交易日历等),需自备 API Key,使用前请遵守其服务条款 +内置数据源插件 [TdxQuant](https://help.tdx.com.cn/quant/) 提供通达信量化数据服务(实时行情 / 五档盘口 / 日K / 复权因子 / 分钟K),需自备 API Key,使用前请遵守其服务条款 + ## 社区 本开源项目已链接并认可 [LINUX DO 社区](https://linux.do)。 diff --git a/VERSION b/VERSION index 03f2afa1..d379b578 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v0.3.5 +v0.3.6 diff --git a/assets/support/wechat-appreciation.jpg b/assets/support/wechat-appreciation.jpg index ff950eb2..70cb1349 100644 Binary files a/assets/support/wechat-appreciation.jpg and b/assets/support/wechat-appreciation.jpg differ diff --git a/backend/app/__init__.py b/backend/app/__init__.py index cf72d223..e074083a 100644 --- a/backend/app/__init__.py +++ b/backend/app/__init__.py @@ -8,7 +8,7 @@ from pathlib import Path # ISCC)、macOS 包元数据 (tsp.spec 读它)、页面版本徽标与更新检查 (本模块) 全部同源, # 改版本只改那一份文件。dev 模式读仓库文件; PyInstaller 冻结后读 _MEIPASS/package.json # (tsp.spec 的 datas 会收集)。读取失败回退硬编码值 —— 版本号绝不阻塞应用启动。 -_FALLBACK_VERSION = "0.3.5" +_FALLBACK_VERSION = "0.3.6" def _load_version() -> str: diff --git a/backend/app/api/ext_data.py b/backend/app/api/ext_data.py index e276734c..90c1e960 100644 --- a/backend/app/api/ext_data.py +++ b/backend/app/api/ext_data.py @@ -108,6 +108,8 @@ class PullConfigReq(BaseModel): auth: PullAuthReq | None = None # 单次拉取请求超时 (秒), 默认 30 与历史行为一致; 大响应接口可调高 timeout_seconds: int = Field(30, ge=5, le=300) + # 两级拉取配方 (目录 → 逐条目明细); 缺省 (None) = 保留现有配置 + fanout: dict | None = None class ApiKeyReq(BaseModel): @@ -407,27 +409,6 @@ def list_configs(request: Request): return {"items": items} -@router.post("/presets/{config_id}/fetch") -async def fetch_preset_data(request: Request, config_id: str): - """手动触发内置预设 (概念/行业) 的数据拉取。 - - 注意: 必须在 /{config_id}/... 动态路由之前声明, 否则 'presets' 会被当成 config_id。 - 与通用 pull/run 不同: 走 ext_presets 的结构转换 (接口的 concepts/industries - 数组 → 拼接成字符串), 保证 schema 与现有数据一致。 - """ - from app.services.ext_presets import fetch_preset - - try: - n = await fetch_preset(config_id, _data_dir(request)) - except ValueError as e: - raise HTTPException(404, str(e)) from e - except Exception as e: - raise HTTPException(400, f"拉取失败: {e}") from e - - _refresh_views(request) - return {"status": "ok", "rows": n} - - @router.post("") def create_config(request: Request, body: CreateExtReq): """创建扩展数据配置。""" @@ -1016,6 +997,15 @@ def configure_pull(request: Request, config_id: str, body: PullConfigReq): # 保留历史状态字段; auth 缺省时沿用现有配置 (关闭鉴权需显式传 {"type":"none"}) old_pull = config.pull + # auth.key_ref / fanout: UI 未暴露的键在保存时沿用旧值, 避免一次普通保存 + # 就把出厂配方里的凭据引用与两级请求配方抹掉。 + old_auth = (old_pull.auth or {}) if old_pull else {} + if body.auth is not None: + new_auth: dict | None = body.auth.model_dump() + if new_auth and old_auth.get("key_ref") and not new_auth.get("key_ref"): + new_auth["key_ref"] = old_auth["key_ref"] + else: + new_auth = old_pull.auth if old_pull else None config.pull = PullConfig( url=body.url, method=body.method, @@ -1030,8 +1020,9 @@ def configure_pull(request: Request, config_id: str, body: PullConfigReq): date_param=body.date_param, date_format=body.date_format, time_field=body.time_field, - auth=body.auth.model_dump() if body.auth else (old_pull.auth if old_pull else None), + auth=new_auth, timeout_seconds=body.timeout_seconds, + fanout=body.fanout if body.fanout is not None else (old_pull.fanout if old_pull else None), last_run=old_pull.last_run if old_pull else None, last_status=old_pull.last_status if old_pull else None, last_message=old_pull.last_message if old_pull else None, diff --git a/backend/app/api/ext_pages.py b/backend/app/api/ext_pages.py new file mode 100644 index 00000000..c01c2185 --- /dev/null +++ b/backend/app/api/ext_pages.py @@ -0,0 +1,136 @@ +"""运行时页面包 API — 上传/清单回填/启停/删除/bundle 分发。 + +分发端点位于 /api 前缀下, 自动受集中鉴权中间件保护: 浏览器同源动态 +import 会携带登录 cookie, 与页面会话同一访问边界。 +""" +from __future__ import annotations + +import shutil +import tempfile +from pathlib import Path +from typing import Annotated + +from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile +from fastapi.responses import FileResponse +from pydantic import BaseModel + +from app.services.ext_pages import PagePackMeta, PagePackStore + +router = APIRouter(prefix="/api/ext-pages", tags=["ext-pages"]) + +# 单文件 ES 模块产物通常 < 1MB, 5MB 上限留足余量; 分块写防 OOM(同 ext_data) +_MAX_BUNDLE_BYTES = 5 * 1024 * 1024 +_UPLOAD_CHUNK_BYTES = 1024 * 1024 + + +async def _write_bundle_capped(file: UploadFile, dest: Path, max_bytes: int) -> None: + """分块写入, 累计超限立即 413 并停止读取(防内存放大)。""" + total = 0 + with dest.open("wb") as f: + while True: + chunk = await file.read(_UPLOAD_CHUNK_BYTES) + if not chunk: + break + total += len(chunk) + if total > max_bytes: + raise HTTPException(413, f"页面包过大(上限 {max_bytes // (1024 * 1024)}MB)") + f.write(chunk) + + +def _store(request: Request) -> PagePackStore: + return PagePackStore(request.app.state.repo.store.data_dir) + + +class ManifestBody(BaseModel): + label: str = "" + path: str = "" + error: str = "" + + +class EnabledBody(BaseModel): + enabled: bool + + +@router.post("/upload") +async def upload_page_pack( + request: Request, + id: Annotated[str, Form()], + file: Annotated[UploadFile, File()], +) -> PagePackMeta: + """上传页面包产物(.js/.mjs), 覆盖式: 同 ID 重新上传即更新。""" + suffix = Path(file.filename or "").suffix.lower() + if suffix not in (".js", ".mjs"): + raise HTTPException(400, "仅支持 .js / .mjs 页面包产物") + store = _store(request) + tmp_dir = Path(tempfile.mkdtemp()) + try: + tmp_path = tmp_dir / "bundle.js" + await _write_bundle_capped(file, tmp_path, _MAX_BUNDLE_BYTES) + if tmp_path.stat().st_size == 0: + raise HTTPException(400, "页面包为空文件") + try: + return store.save_upload(id, tmp_path) + except ValueError as e: + raise HTTPException(400, str(e)) from e + finally: + shutil.rmtree(tmp_dir, ignore_errors=True) + + +@router.get("") +def list_page_packs(request: Request) -> list[PagePackMeta]: + return _store(request).list() + + +@router.get("/{pack_id}/bundle.js") +def get_page_pack_bundle(pack_id: str, request: Request) -> FileResponse: + """分发产物。no-store + ETag(sha256): 热更新产物后浏览器立即取新, 未变时可用缓存协商。""" + store = _store(request) + try: + meta = store.get(pack_id) + except ValueError as e: + raise HTTPException(400, str(e)) from e + bundle = store.bundle_path(pack_id) if meta else None + if meta is None or bundle is None: + raise HTTPException(404, f"页面包 '{pack_id}' 不存在") + return FileResponse( + bundle, + media_type="text/javascript", + headers={ + "Cache-Control": "no-store", + "ETag": f'"{meta.sha256}"', + }, + ) + + +@router.put("/{pack_id}/manifest") +def put_page_pack_manifest(pack_id: str, body: ManifestBody, request: Request) -> PagePackMeta: + """前端试载结果回填: error 非空 → 停用并记错误, 否则启用并登记摘要。""" + store = _store(request) + try: + return store.update_manifest(pack_id, label=body.label, path=body.path, error=body.error) + except KeyError: + raise HTTPException(404, f"页面包 '{pack_id}' 不存在") from None + except ValueError as e: + raise HTTPException(400, str(e)) from e + + +@router.put("/{pack_id}/enabled") +def put_page_pack_enabled(pack_id: str, body: EnabledBody, request: Request) -> PagePackMeta: + store = _store(request) + try: + return store.set_enabled(pack_id, body.enabled) + except KeyError: + raise HTTPException(404, f"页面包 '{pack_id}' 不存在") from None + except ValueError as e: + raise HTTPException(400, str(e)) from e + + +@router.delete("/{pack_id}") +def delete_page_pack(pack_id: str, request: Request) -> dict: + store = _store(request) + try: + if not store.delete(pack_id): + raise HTTPException(404, f"页面包 '{pack_id}' 不存在") + except ValueError as e: + raise HTTPException(400, str(e)) from e + return {"ok": True} diff --git a/backend/app/api/kline.py b/backend/app/api/kline.py index 377a8343..6c1c62e9 100644 --- a/backend/app/api/kline.py +++ b/backend/app/api/kline.py @@ -1265,19 +1265,19 @@ async def trigger_minute_sync(repo, capset, *, override_days=None, extend_flag=N try: progress("sync_minute", 5, "解析标的池…") - universe = sorted(set(get_pool("watchlist")) | set(get_pool("CN_Equity_A"))) - # 补充 instruments 全量标的，覆盖北交所、新股等 + # 标的池 = instruments 维表 + 自选 (单一来源, 不拉 CN_Equity_A/SW1 池) + universe: set[str] = set(get_pool("watchlist")) inst_path = repo.store.data_dir / "instruments" / "instruments.parquet" if inst_path.exists(): try: import polars as pl inst = pl.read_parquet(inst_path, columns=["symbol"]) - universe = sorted(set(universe) | set(inst["symbol"].to_list())) + universe.update(inst["symbol"].to_list()) except Exception: # noqa: BLE001 pass # 剔除指数 symbol: 指数分钟K无本地存储, 落库会污染 kline_minute index_set = repo.get_index_symbol_set() - universe = [s for s in universe if s not in index_set] + universe = sorted({s for s in universe if s not in index_set}) progress("sync_minute", 10, f"标的池 {len(universe)} 只") days = override_days if override_days else get_minute_sync_days() diff --git a/backend/app/api/settings.py b/backend/app/api/settings.py index 96659ca7..fded9d46 100644 --- a/backend/app/api/settings.py +++ b/backend/app/api/settings.py @@ -60,6 +60,7 @@ def get_settings() -> dict: current_ai_context_window, current_ai_max_output_tokens, current_ai_model, + current_ai_protocol, current_ai_round_checkpoint, current_codex_command, current_codex_model, @@ -81,8 +82,11 @@ def get_settings() -> dict: "extras_caps": extras_caps(), # 首次使用引导 "onboarding_completed": preferences.get_onboarding_completed(), + # 配置文件位置 (AI 与数据源 Key 统一存放, 支持外部编辑后热加载) + "secrets_path": str(secrets_store.file_path()), # AI 配置 "ai_provider": ai_provider, + "ai_protocol": current_ai_protocol(), "ai_base_url": secrets_store.get_ai_config("ai_base_url", settings.ai_base_url), "ai_api_key_masked": secrets_store.mask(secrets_store.get_ai_key()), "has_ai_key": bool(secrets_store.get_ai_key()), @@ -254,6 +258,7 @@ class AiSettingsIn(BaseModel): base_url: str = "" api_key: str | None = None model: str = "" + protocol: str = "" # chat_completions | responses; 空=保持原值 reasoning_effort: str = Field(default="high", max_length=64) codex_command: str = "" codex_reasoning_effort: str = "" @@ -263,16 +268,63 @@ class AiSettingsIn(BaseModel): round_checkpoint: int | None = None # 助手工具轮次检查点; 0=不检查 +class AiTestIn(BaseModel): + """草稿连通性测试入参 — 只读表单值, 与保存完全解耦。""" + + provider: str = "openai_compat" + base_url: str = "" + api_key: str | None = None # 留空回退已保存 Key (改模型/协议不必重输) + model: str = "" + protocol: str = "" + + +@router.post("/ai/test") +async def test_ai_settings(req: AiTestIn) -> dict: + """用当前表单值发一次最小请求验证连通性, 不保存/不修改任何配置。 + + 与 POST /ai (保存) 解耦: 测试失败不影响已保存配置, 状态栏也不会把 + 未验证的草稿展示为已生效。错误保留 200 包体 (ok=false), 由前端内联 + 展示在测试结果徽章, 不触发全局 toast。 + """ + from app.services.ai_provider import ( + RESPONSES_PROTOCOL, + _format_openai_error, + _is_openai_transport_error, + probe_ai_config, + ) + + provider = req.provider or "openai_compat" + api_key = (req.api_key or "").strip() or (secrets_store.get_ai_key() or "") + protocol = req.protocol if req.protocol == RESPONSES_PROTOCOL else "" + try: + text = await probe_ai_config( + provider=provider, + api_key=api_key, + base_url=req.base_url.strip(), + model=req.model.strip(), + protocol=protocol, + ) + except Exception as exc: # noqa: BLE001 — 连通性测试: 任何异常都转为结果回报 + detail = str(exc) or repr(exc) + if _is_openai_transport_error(exc): + detail = _format_openai_error(exc) + return {"ok": False, "error": detail, "error_type": type(exc).__name__} + return {"ok": True, "model": req.model.strip() or provider, "response": text[:80]} + + @router.post("/ai") def save_ai_settings(req: AiSettingsIn) -> dict: """保存 AI 配置（全部持久化到 secrets.json）""" from app.config import settings from app.services.ai_provider import ( + CHAT_COMPLETIONS_PROTOCOL, OPENAI_PROVIDER, + RESPONSES_PROTOCOL, ai_configured, current_ai_context_window, current_ai_max_output_tokens, current_ai_model, + current_ai_protocol, current_ai_provider, current_ai_round_checkpoint, current_codex_command, @@ -304,6 +356,12 @@ def save_ai_settings(req: AiSettingsIn) -> dict: if req.base_url: updates["ai_base_url"] = req.base_url settings.ai_base_url = req.base_url + if req.protocol: + # 协议仅对 openai/openai_compat 有意义; 非法值 fail-loud, 避免静默落错协议。 + if req.protocol not in (CHAT_COMPLETIONS_PROTOCOL, RESPONSES_PROTOCOL): + raise HTTPException(status_code=400, detail=f"不支持的协议: {req.protocol}") + updates["ai_protocol"] = req.protocol + settings.ai_protocol = req.protocol if req.api_key is not None: if req.api_key: updates["ai_api_key"] = req.api_key @@ -345,6 +403,7 @@ def save_ai_settings(req: AiSettingsIn) -> dict: return { "ok": True, "ai_provider": provider, + "ai_protocol": current_ai_protocol(), "ai_model": current_ai_model(), "ai_openai_model": current_openai_model(), "ai_reasoning_effort": current_openai_reasoning_effort(), @@ -376,9 +435,10 @@ def clear_ai_settings() -> dict: "ai_codex_command", "ai_codex_reasoning_effort", ) - secrets_store.clear("ai_provider", "ai_base_url", "ai_api_key", "ai_model", "ai_codex_command", "ai_codex_reasoning_effort", "ai_max_output_tokens", "ai_context_window") + secrets_store.clear("ai_provider", "ai_protocol", "ai_base_url", "ai_api_key", "ai_model", "ai_codex_command", "ai_codex_reasoning_effort", "ai_max_output_tokens", "ai_context_window") # 同步重置运行时内存(provider 回默认值,其余置空) settings.ai_provider = "openai_compat" + settings.ai_protocol = "chat_completions" settings.ai_base_url = "" settings.ai_api_key = "" settings.ai_model = "" @@ -396,6 +456,10 @@ async def list_sponsor_models() -> dict: 其网关按 Origin 头过滤: 浏览器跨域请求只会拿到国产模型子集, 服务端请求无 Origin 头可取全量, 故由后端代理转发。 + + /v1/models 除 id 外还携带 pricing (discountRate/分层单价) 与 + context_length; 精简为 items 下发供前端展示折扣徽章与价格, + models 保持纯 id 列表向后兼容。 """ import httpx @@ -406,11 +470,30 @@ async def list_sponsor_models() -> dict: data = res.json() except Exception as exc: raise HTTPException(status_code=502, detail=f"获取模型列表失败: {exc}") from exc - models = sorted({ - item.get("id") for item in data.get("data", []) - if isinstance(item, dict) and isinstance(item.get("id"), str) and item["id"] - }) - return {"models": models} + + def _amount(pricing: dict, dim: str) -> float | None: + value = (pricing.get(dim) or {}).get("amount") + return float(value) if isinstance(value, (int, float)) else None + + items: list[dict] = [] + for raw in data.get("data", []): + if not isinstance(raw, dict): + continue + model_id = raw.get("id") + if not isinstance(model_id, str) or not model_id: + continue + pricing = raw.get("pricing") if isinstance(raw.get("pricing"), dict) else {} + rate = pricing.get("discountRate") + context_length = raw.get("context_length") + items.append({ + "id": model_id, + "context_length": context_length if isinstance(context_length, int) else None, + "discount_rate": float(rate) if isinstance(rate, (int, float)) else 1.0, + "input_price": _amount(pricing, "input"), + "output_price": _amount(pricing, "output"), + }) + items.sort(key=lambda x: x["id"]) + return {"models": [item["id"] for item in items], "items": items} # ===== 偏好设置 ===== @@ -1212,6 +1295,48 @@ class MainlineFilterIn(BaseModel): exclude_st: bool | None = None +class EmotionScoreConfigIn(BaseModel): + """情绪评分口径 (维度权重 / 子项权重与归一化区间 / 档位阈值)。 + + config 为整棵树或 null — 传 null 表示恢复出厂口径。不做逐字段白名单: + 结构由 emotion_score.merge_config 归一化 (未知 key 丢弃、缺项补默认、 + 阈值乱序自动重排), 用户改坏权重也不会让看板算不出分。 + """ + + config: dict | None = None + + +@router.get("/emotion-score/config") +def get_emotion_score_config() -> dict: + """情绪评分口径: 当前配置 + 出厂默认 + 维度顺序 + 档位名。 + + 看板「情绪雷达」卡片的配置对话框一次拉全 —— 前端不硬编码任何权重/区间/ + 阈值字面量, 表单全部由本接口返回的结构渲染。 + """ + from app.services import preferences + from app.services.emotion_score import public_config + + payload = public_config(preferences.get_emotion_score_config()) + payload["customized"] = preferences.get_emotion_score_config() is not None + return payload + + +@router.put("/emotion-score/config") +def update_emotion_score_config(req: EmotionScoreConfigIn) -> dict: + """保存情绪评分口径; config 传 null/{} = 恢复出厂。 + + 形状校验在 preferences 层 (经 emotion_score.merge_config), 值的合法性 + (负权重/区间倒置/阈值乱序) 由计算时归一化兜底, 故此处不做逐字段白名单。 + """ + from app.services import preferences + from app.services.emotion_score import public_config + + saved = preferences.set_emotion_score_config(req.config) + payload = public_config(saved) + payload["customized"] = saved is not None + return payload + + @router.put("/preferences/mainline-filter") def update_mainline_filter(req: MainlineFilterIn) -> dict: """更新市场主线过滤配置。部分更新; 修改后需重算主线(POST /api/regime/mainline/recompute)生效。 diff --git a/backend/app/api/task_registry.py b/backend/app/api/task_registry.py new file mode 100644 index 00000000..45844b4c --- /dev/null +++ b/backend/app/api/task_registry.py @@ -0,0 +1,63 @@ +"""进行中任务注册表 API — 前端 fire-and-forget 上报 + 页面挂载时查询。 + +PUT/DELETE 由任务启动/终态触发 (不阻塞 UI, 失败静默); GET 在任务页挂载时 +最多调用一次。全部是小 JSON 文件读写, 不触碰任务执行路径。 +""" +from __future__ import annotations + +import re + +from fastapi import APIRouter, HTTPException, Request +from pydantic import BaseModel, Field + +from app.services import task_registry + +router = APIRouter(prefix="/api/task-registry", tags=["task-registry"]) + +# id 由前端按 kind+参数指纹生成 (字母/数字/冒号/下划线/短横) +_VALID_ID = re.compile(r"^[A-Za-z0-9:_-]{1,128}$") + + +class TaskUpsertRequest(BaseModel): + kind: str = Field(max_length=32) + status: str = Field(max_length=16) + label: str = Field(default="", max_length=120) + payload: dict = Field(default_factory=dict) + + +def _data_dir(request: Request): + repo = getattr(request.app.state, "repo", None) + if repo is None: + raise HTTPException(status_code=503, detail="数据目录不可用") + return repo.store.data_dir + + +@router.get("") +def list_tasks(request: Request, kind: str | None = None) -> dict: + if kind is not None and kind not in task_registry.TASK_KINDS: + raise HTTPException(status_code=400, detail=f"未知的任务类型: {kind}") + return {"items": task_registry.list_items(_data_dir(request), kind)} + + +@router.put("/{task_id}") +def upsert_task(request: Request, task_id: str, req: TaskUpsertRequest) -> dict: + if not _VALID_ID.match(task_id): + raise HTTPException(status_code=400, detail="非法的任务 id") + try: + entry = task_registry.upsert( + _data_dir(request), + task_id, + kind=req.kind, + status=req.status, + label=req.label, + payload=req.payload, + ) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) from e + return {"ok": True, "item": entry} + + +@router.delete("/{task_id}") +def delete_task(request: Request, task_id: str) -> dict: + removed = task_registry.delete(_data_dir(request), task_id) + return {"ok": True, "removed": removed} diff --git a/backend/app/config.py b/backend/app/config.py index 0554ab5a..49a86a5f 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -84,6 +84,8 @@ class Settings(BaseSettings): # AI ai_provider: str = "openai_compat" + # OpenAI 系请求协议: chat_completions (兼容面最广) | responses (OpenAI 官方新协议)。 + ai_protocol: str = "chat_completions" ai_base_url: str = "https://llm.runninghub.ai/v1" ai_api_key: str = "" ai_model: str = "openai/gpt-6-astra-saver" diff --git a/backend/app/custom/assistant/routes.py b/backend/app/custom/assistant/routes.py index c3800d85..b455ab7b 100644 --- a/backend/app/custom/assistant/routes.py +++ b/backend/app/custom/assistant/routes.py @@ -3,11 +3,14 @@ POST /api/custom/assistant/chat NDJSON 事件流(协议见模块 __init__) GET /api/custom/assistant/status 供应商配置状态(前端门控展示) GET /api/custom/assistant/suggests 空会话快捷指令 +GET /api/custom/assistant/sessions 会话历史(服务端, 跨设备) +PUT /api/custom/assistant/sessions 会话历史整体保存(前端防抖写入) POST /api/custom/assistant/actions/{call_id}/decision 动作工具确认/拒绝(确认卡按钮回调) """ from __future__ import annotations +from pathlib import Path from typing import Literal from fastapi import APIRouter, HTTPException, Request @@ -15,6 +18,7 @@ from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field from app.custom.assistant import actions as assistant_actions +from app.custom.assistant import sessions as assistant_sessions from app.custom.assistant import tools as assistant_tools from app.custom.assistant.chat_service import chat_stream from app.services.ai_provider import ( @@ -45,6 +49,19 @@ class ActionDecisionRequest(BaseModel): approve: bool +class AssistantSessionsPayload(BaseModel): + """前端 store.ts 的持久化载荷: 会话数组 + 活跃会话 id。 + + sessions 用宽松的 list[dict] (消息形状由前端定义, 后端只裁剪条数), + 上限与前端一致, 防御异常客户端把文件写爆。字段名对齐前端 (activeId)。 + """ + + sessions: list[dict] = Field(max_length=assistant_sessions.MAX_SESSIONS) + active_id: str = Field(default="", max_length=64, alias="activeId") + + model_config = {"populate_by_name": True} + + def build_router() -> APIRouter: router = APIRouter(prefix="/api/custom/assistant", tags=["custom-assistant"]) @@ -89,6 +106,24 @@ def build_router() -> APIRouter: def suggests() -> dict: return {"suggests": assistant_tools.QUICK_SUGGESTS} + @router.get("/sessions") + def sessions(request: Request) -> dict: + repo = getattr(request.app.state, "repo", None) + if repo is None: + raise HTTPException(status_code=503, detail="数据目录不可用") + return assistant_sessions.load_sessions(Path(repo.store.data_dir)) + + @router.put("/sessions") + def put_sessions(request: Request, req: AssistantSessionsPayload) -> dict: + repo = getattr(request.app.state, "repo", None) + if repo is None: + raise HTTPException(status_code=503, detail="数据目录不可用") + n = assistant_sessions.save_sessions( + Path(repo.store.data_dir), + {"sessions": req.sessions, "activeId": req.active_id}, + ) + return {"ok": True, "sessions": n} + @router.post("/actions/{call_id}/decision") async def action_decision(call_id: str, req: ActionDecisionRequest) -> dict: """确认卡回调: 唤醒挂起中的动作工具(批准执行或拒绝)。""" diff --git a/backend/app/custom/assistant/sessions.py b/backend/app/custom/assistant/sessions.py new file mode 100644 index 00000000..cd11d959 --- /dev/null +++ b/backend/app/custom/assistant/sessions.py @@ -0,0 +1,104 @@ +"""AI 助手会话历史服务端存储 — 单用户, 前端为主写入方, 后端只存取与裁剪。 + +动机: 会话历史是用户产生的内容而非设备习惯, 原先只存浏览器 localStorage, +换设备即不可见; 2026-10 起同步到服务端, localStorage 降级为启动缓存/离线兜底。 + +文件: data/user_data/assistant_sessions.json + {"sessions": [...], "activeId": "..."} (字段名与前端 store.ts 载荷一致) + +容量纪律 (与前端 store.ts 相同): ≤20 会话 × 每会话 ≤200 条消息, 超限裁尾部。 +写走 fs_utils.atomic_write_text 原子替换; 读带 mtime 签名缓存 — GET 只在应用 +加载时调用一次、写经前端防抖, 磁盘 IO 可忽略, 不影响其他任何请求路径。 +""" +from __future__ import annotations + +import json +import logging +from pathlib import Path +from typing import Any + +from app.services.fs_utils import atomic_write_text + +logger = logging.getLogger(__name__) + +MAX_SESSIONS = 20 +MAX_MESSAGES_PER_SESSION = 200 +_FILENAME = "assistant_sessions.json" + +_cache: dict[str, Any] | None = None +_cache_sig: tuple[int, int] | None = None + + +def _path(data_dir: Path) -> Path: + p = data_dir / "user_data" / _FILENAME + p.parent.mkdir(parents=True, exist_ok=True) + return p + + +def _signature(p: Path) -> tuple[int, int] | None: + try: + st = p.stat() + return (st.st_mtime_ns, st.st_size) + except OSError: + return None + + +def _sanitize_sessions(data: Any) -> list[dict]: + """结构校验 + 容量裁剪: 只信 id 为字符串的会话, 消息截尾至 200 条。""" + if not isinstance(data, dict): + return [] + sessions = data.get("sessions") + if not isinstance(sessions, list): + return [] + out: list[dict] = [] + for s in sessions: + if not isinstance(s, dict) or not isinstance(s.get("id"), str) or not s["id"]: + continue + msgs = s.get("messages") + if not isinstance(msgs, list): + msgs = [] + out.append({**s, "messages": msgs[-MAX_MESSAGES_PER_SESSION:]}) + return out[:MAX_SESSIONS] + + +def _sanitize_active(data: Any) -> str: + if isinstance(data, dict): + active = data.get("activeId") + if isinstance(active, str): + return active + return "" + + +def load_sessions(data_dir: Path) -> dict[str, Any]: + """读取 {sessions, activeId}; 文件不存在/损坏返回空结构 (不抛错)。""" + global _cache, _cache_sig + p = _path(data_dir) + sig = _signature(p) + if sig is None: + return {"sessions": [], "activeId": ""} + if _cache is not None and sig == _cache_sig: + return _cache + try: + data = json.loads(p.read_text(encoding="utf-8")) + except Exception as e: # noqa: BLE001 — 损坏文件视为无历史, 不阻断 + logger.warning("assistant sessions 文件损坏, 视为空: %s", e) + return {"sessions": [], "activeId": ""} + result = { + "sessions": _sanitize_sessions(data), + "activeId": _sanitize_active(data), + } + _cache, _cache_sig = result, sig + return result + + +def save_sessions(data_dir: Path, payload: dict[str, Any]) -> int: + """校验 + 裁剪 + 原子写盘; 返回保留的会话数。""" + global _cache, _cache_sig + doc = { + "sessions": _sanitize_sessions(payload), + "activeId": _sanitize_active(payload), + } + p = _path(data_dir) + atomic_write_text(p, json.dumps(doc, ensure_ascii=False)) + _cache, _cache_sig = doc, _signature(p) + return len(doc["sessions"]) diff --git a/backend/app/custom/assistant/streaming.py b/backend/app/custom/assistant/streaming.py index 4f419831..04cb8109 100644 --- a/backend/app/custom/assistant/streaming.py +++ b/backend/app/custom/assistant/streaming.py @@ -17,6 +17,7 @@ tool_calls 分片, 流结束时统一返回本轮结果, 由 chat_service 决定 from __future__ import annotations import asyncio +import inspect from collections.abc import AsyncIterator, Sequence from typing import Any @@ -24,11 +25,13 @@ from openai import AsyncOpenAI from app import secrets_store from app.config import settings +from app.services import ai_responses from app.services.ai_provider import ( OPENAI_PROVIDER, current_ai_model, current_ai_provider, current_openai_reasoning_effort, + is_responses_protocol, normalize_openai_base_url, ) @@ -88,6 +91,14 @@ async def stream_openai_round( timeout: float = 240.0, ) -> AsyncIterator[dict[str, Any]]: """执行一轮流式补全: yield 文本 delta, 最后 yield 一个 round_end 事件。""" + if is_responses_protocol(): + # Responses 协议走平行实现, 事件输出与本函数同构, chat_service 无感。 + async for event in ai_responses.stream_round( + messages, tools, temperature=temperature, timeout=timeout, + ): + yield event + return + client = _client(timeout) model = current_ai_model() kwargs = _build_kwargs(temperature) @@ -159,10 +170,14 @@ async def stream_openai_round( if getattr(fn, "arguments", None): entry["arguments"] += str(fn.arguments) - # 让 SDK 流对象尽快释放(部分兼容网关要求显式 close)。 + # 让 SDK 流对象尽快释放(部分兼容网关要求显式 close); SDK 2.x 的 + # AsyncStream.close 是协程, 必须区分同步/异步两种签名。 close = getattr(stream, "close", None) if close is not None: - await asyncio.to_thread(close) + if inspect.iscoroutinefunction(close): + await close() + else: + await asyncio.to_thread(close) if calls: yield { diff --git a/backend/app/data_providers/capabilities.py b/backend/app/data_providers/capabilities.py index ccbb97ce..0a067662 100644 --- a/backend/app/data_providers/capabilities.py +++ b/backend/app/data_providers/capabilities.py @@ -198,3 +198,92 @@ def build_capability_matrix(current: dict[str, str], tickflow_tier: str = "none" "pending": pending, }) return {"tickflow_tier": tier_base, "capabilities": capabilities} + + +# --------------------------------------------------------------------------- +# 门禁视角 (面向各服务的启动/调度判断与日志) +# --------------------------------------------------------------------------- + +def _cap_specs_by_id() -> dict[str, dict]: + return {c["id"]: c for c in CAPABILITY_REGISTRY} + + +def _current_routing() -> dict[str, str]: + """偏好字段 → 当前生效源 (经 preferences getters, 非法值已回退默认)。 + + 与 api/settings.py 的 capability-matrix 端点同一组 getter; 显式列全, + 不用 getattr 拼函数名 (拼错会静默回退, 难排查)。 + """ + from app.services import preferences + + return { + "realtime_data_provider": preferences.get_realtime_data_provider(), + "daily_data_provider": preferences.get_daily_data_provider(), + "minute_data_provider": preferences.get_minute_data_provider(), + "full_minute_data_provider": preferences.get_full_minute_data_provider(), + "depth5_data_provider": preferences.get_depth5_data_provider(), + "adj_factor_provider": preferences.get_adj_factor_provider(), + "financial_data_provider": preferences.get_financial_provider(), + } + + +def resolve_capability_sources(tickflow_tier: str = "none") -> list[dict]: + """每个能力一行: 生效源 + 展示名 + 该源当前能否真正提供 (usable)。 + + 与 build_capability_matrix 同口径 (复用同一 candidates 组装), 供服务层 + 的门禁判断与日志标签使用 —— 门禁必须看「生效源能否提供」, 而不是只看 + TickFlow 套餐 (路由到插件时 TickFlow 档位无关; 同理路由到 TickFlow 而 + 档位不足时, 插件声明了也不该被算作可用)。 + + 返回 [{id, label, field, source, source_display, usable, tf_available}]。 + """ + tier_rank = _TIER_RANK.get(_tier_base(tickflow_tier), -1) + sources = _declared_sources() + current = _current_routing() + + rows: list[dict] = [] + for cap in CAPABILITY_REGISTRY: + field = cap.get("field") + if not field: + continue + effective = current.get(field, cap["default"]) + tf_available = tier_rank >= _TIER_RANK[cap["tf_tier"]] + provider_names = {"tickflow"} if tf_available else set() + provider_names |= { + s["name"] for s in sources + if cap["id"] in s["datasets"] and s["available"] + } + rows.append({ + "id": cap["id"], + "label": cap["label"], + "field": field, + "source": effective, + "source_display": _display_of(sources, effective), + "usable": effective in provider_names, + "tf_available": tf_available, + }) + return rows + + +def capability_gate(cap_id: str, tickflow_tier: str | None = None) -> tuple[bool, str]: + """单个能力的门禁: (是否可用, 日志标签)。 + + 标签形如 "日K ← fuyao" / "财务数据 ← TickFlow(档位不足)", 让「为什么没跑」 + 在日志里一眼可读, 而不必回设置页对照路由。 + + 可用性以生效源为准 (与设置页 usable 同口径): 路由到已声明的插件即可用, + 路由到 TickFlow 但档位不够则不可用。 + """ + if tickflow_tier is None: + from app.tickflow import policy + tickflow_tier = policy.base_tier_name() + + for row in resolve_capability_sources(tickflow_tier): + if row["id"] == cap_id: + label = f"{row['label']} ← {row['source_display']}" + if not row["usable"] and row["source"] == "tickflow" and not row["tf_available"]: + label += "(档位不足)" + elif not row["usable"]: + label += "(当前源不可用)" + return row["usable"], label + return False, f"{cap_id}(未知能力)" diff --git a/backend/app/data_providers/custom/loader.py b/backend/app/data_providers/custom/loader.py index 18a69aa2..ae183dde 100644 --- a/backend/app/data_providers/custom/loader.py +++ b/backend/app/data_providers/custom/loader.py @@ -578,6 +578,14 @@ def _register_one_plugin(manifest: dict) -> None: "homepage": manifest.get("homepage", ""), "api_key_env": manifest.get("api_key_env", ""), "api_key_masked": _plugin_key_masked(name, manifest.get("api_key_env", "")), + # 计费标识 (可选): 如 "points" → 设置页能力芯片/适配行标注「积分」 + "billing": manifest.get("billing", ""), + # 计费提示 (可选): 积分消耗与轮询频率/拉取范围的关系, 设置页按警示样式展示 + "billing_note": manifest.get("billing_note", ""), + # 使用要点 (可选): 各能力的规模/节奏说明, 设置页详情卡以列表展示 + "usage_notes": list(manifest.get("usage_notes", []) or []), + # Key/账务管理页 (可选): 设置页 Key 配置区展示入口 + "key_manage_url": manifest.get("key_manage_url", ""), } if not available: return # 依赖没装: 不注册, 但状态已记录供 UI 显示 diff --git a/backend/app/jobs/daily_pipeline.py b/backend/app/jobs/daily_pipeline.py index 100e8aef..7775ffb7 100644 --- a/backend/app/jobs/daily_pipeline.py +++ b/backend/app/jobs/daily_pipeline.py @@ -26,7 +26,7 @@ from app.market_time import cn_today from app.services import index_sync, instrument_sync, kline_sync from app.services import preferences as _prefs from app.tickflow.capabilities import Cap, CapabilitySet -from app.tickflow.pools import DEMO_SYMBOLS, get_pool +from app.tickflow.pools import get_pool from app.tickflow.repository import KlineRepository logger = logging.getLogger(__name__) @@ -139,35 +139,26 @@ def _invalidate(table: str | None = None) -> None: def _resolve_universe(capset: CapabilitySet, repo=None) -> list[str]: - """解析标的池 — 以 CN_Equity_A (沪深京A股 ~5522只) 为主。 + """解析标的池 — 本地 instruments 维表 + 自选 (单一来源, 无池兜底)。 - 有 batch 能力 → 直接拉 CN_Equity_A universe - 其他用户 → 用 instruments parquet + watchlist 兜底 + 不再拉 CN_Equity_A 池: 其 universe 匹配曾错配到 Expired 变体、触发 + SW1 行业聚合兜底, 产出 4224 只的残缺池。维表由盘后管道 Step 0 先同步, + 这里只读 parquet; 维表缺失时仅剩自选, 宁缺毋假 (不塞演示标的), + 由上游按空池结果如实呈现。 - repo 传入时过滤自选兜底里的指数 symbol (指数日K走独立 kline_index_* 存储, + repo 传入时过滤自选里的指数 symbol (指数日K走独立 kline_index_* 存储, 进股票池会污染 kline_daily/kline_minute)。ETF 刻意保留 (既有行为)。 """ - if capset.has(Cap.KLINE_DAILY_BATCH): - try: - all_a = get_pool("CN_Equity_A", refresh=True) - if all_a: - return sorted(all_a) - except Exception as e: # noqa: BLE001 - logger.warning("CN_Equity_A pool unavailable, fallback: %s", e) - - # Free 用户兜底: instruments parquet + watchlist + demo - base: set[str] = set(DEMO_SYMBOLS) - base.update(get_pool("watchlist")) - d = Path(settings.data_dir) - inst_path = d / "instruments" / "instruments.parquet" + base: set[str] = set(get_pool("watchlist")) + inst_path = Path(settings.data_dir) / "instruments" / "instruments.parquet" if inst_path.exists(): try: inst = pl.read_parquet(inst_path, columns=["symbol"]) base.update(inst["symbol"].to_list()) except Exception as e: # noqa: BLE001 - logger.warning("instruments supplement failed: %s", e) - # 过滤自选兜底里的指数 symbol (指数日K走独立 kline_index_* 存储, - # 进股票池会污染 kline_daily/kline_minute)。ETF 刻意保留 (既有行为)。 + logger.warning("instruments 维表读取失败, 标的池仅剩自选: %s", e) + else: + logger.warning("instruments 维表缺失, 标的池仅剩自选 %d 只", len(base)) if repo is not None: base -= set(repo.get_index_symbol_set()) return sorted(base) diff --git a/backend/app/main.py b/backend/app/main.py index 278245f6..3abd0d28 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -20,6 +20,7 @@ from app.api import ( data, events, ext_data, + ext_pages, factors, financials, indices, @@ -39,6 +40,7 @@ from app.api import ( signals, stock_analysis, strategy, + task_registry, watchlist, ) from app.api import auth as auth_api @@ -160,6 +162,19 @@ async def _application_lifespan(app: FastAPI): capset = detect_capabilities() app.state.capabilities = capset logger.info("ready; %d capabilities active", len(capset.all())) + # 各能力的生效源与可用性: 切换数据源后一眼看出路由落点, 无需回设置页对照 + try: + from app.data_providers.capabilities import resolve_capability_sources + + for row in resolve_capability_sources(): + logger.info( + " 能力路由 | %-8s ← %-12s %s", + row["label"], + row["source_display"], + "可用" if row["usable"] else "不可用", + ) + except Exception as e: # noqa: BLE001 — 仅日志, 失败不影响启动 + logger.warning("能力路由日志输出失败: %s", e) # 全局行情服务 qs = QuoteService() @@ -558,6 +573,8 @@ app.include_router(analysis.router) app.include_router(pipeline.router) app.include_router(data.router) app.include_router(ext_data.router) +app.include_router(ext_pages.router) +app.include_router(task_registry.router) app.include_router(financials.router) app.include_router(stock_analysis.router) app.include_router(market_recap.router) diff --git a/backend/app/plugins/fuyao/plugin.yaml b/backend/app/plugins/fuyao/plugin.yaml index cc063f71..5e8073f2 100644 --- a/backend/app/plugins/fuyao/plugin.yaml +++ b/backend/app/plugins/fuyao/plugin.yaml @@ -9,6 +9,6 @@ entry: app.plugins.fuyao.provider:FuyaoProvider check: app.plugins.fuyao.provider:availability datasets: [realtime, daily, adj_factor, financial] api_key_env: FUYAO_API_KEY # 声明后设置页提供 Key 输入框(先探后存, secrets.json 优先) -description: "同花顺官方 REST 数据 API。A 股实时快照、日K(原始价: 近端走 10d dump, 深窗口走 10 年全量 dump 一次下载秒级筛选, 兜底单标的接口)、除权因子(事件 dump + 本地日K dump 配价推导, 涨跌停自检, 秒级)、财务(利润表/资产负债表/现金流量表/指标, 字段映射至项目口径, bps 由估值反推; 股本无上游接口, 指标历史期建议切回 TickFlow 补齐);分钟未接入,自动回退 TickFlow。" +description: "同花顺官方金融数据 API。提供实时快照、日K(原始价)、除权因子、财务三表与指标;未接入分钟与股本,自动回退 TickFlow。" install_hint: "点击卡片中的输入框配置 API Key(https://fuyao.aicubes.cn 申请),或在 .env 中配置 FUYAO_API_KEY" homepage: "https://fuyao.aicubes.cn" diff --git a/backend/app/plugins/tdxquant/__init__.py b/backend/app/plugins/tdxquant/__init__.py new file mode 100644 index 00000000..710f8727 --- /dev/null +++ b/backend/app/plugins/tdxquant/__init__.py @@ -0,0 +1 @@ +"""TdxQuant(通达信量化数据服务)内置数据源插件。""" diff --git a/backend/app/plugins/tdxquant/client.py b/backend/app/plugins/tdxquant/client.py new file mode 100644 index 00000000..26417684 --- /dev/null +++ b/backend/app/plugins/tdxquant/client.py @@ -0,0 +1,260 @@ +"""TdxQuant — tdxaidata (通达信量化数据) 原生库封装。 + +tdxaidata 是 ctypes 包装的私有 TCP 协议客户端(aihs/aids.tdx.com.cn), +DLL 与运行时 ini 随 wheel 分发。两个实测约束决定本模块的形状: + - 原生层每进程加载一次、token 写在 DLL 旁的 ini 里 → Key 物化到私有 + lib 目录 + TDX_AI_DATA_LIB 指向, 不污染 site-packages、换 Key 只重写 ini; + - 空闲后首次调用会 "Invalid server response" → 统一重试包装。 +""" +from __future__ import annotations + +import hashlib +import logging +import os +import shutil +import sys +import time +from datetime import date +from pathlib import Path + +logger = logging.getLogger(__name__) + +# 全 A 股票池: 五个市场名实测(2026-10)合计约 5579 只; ETF/指数不在枚举内 +_MARKETS = ("上证主板", "科创板", "深证主板", "创业板", "北交所") + +# 快照批量字段(实测有效名; 注意昨收叫 LastClose、高低叫 Max/Min, 快照无 Name) +SNAPSHOT_FIELDS = ( + "Now", "Open", "Max", "Min", "LastClose", "Volume", "Amount", "RefreshTime", + "Buyp", "Buyv", "Sellp", "Sellv", +) + +_RETRIES = 3 +_NONE_RETRIES = 2 # 空返回(可能是永久性坏代码)只补试一次 +_RETRY_SLEEP_S = 0.6 +_UNIVERSE_TTL_S = 12 * 3600.0 + +_DLL_BY_PLATFORM = {"win32": "TdxAiData.dll", "darwin": "libTdxAiData.dylib"} +LIB_DLL_NAME = _DLL_BY_PLATFORM.get(sys.platform, "libTdxAiData.so") +LIB_ENV = "TDX_AI_DATA_LIB" # tdxaidata 读取: 指向 DLL 文件路径 + + +class TdxQuantError(RuntimeError): + """tdxaidata 调用失败(网络/参数/协议)。""" + + +def package_lib_dir() -> Path: + """tdxaidata 包自带的 lib 目录(平台 DLL + 运行时 ini)。""" + import tdxaidata + + return Path(tdxaidata.__file__).resolve().parent / "lib" + + +def write_token_ini(ini: Path, api_key: str) -> None: + """把 [Token] token= 写入 ini, 保留其余配置行。""" + lines = ( + ini.read_text(encoding="utf-8", errors="replace").splitlines() + if ini.exists() + else ["[Token]"] + ) + out: list[str] = [] + in_token = False + replaced = False + for line in lines: + stripped = line.strip() + if stripped.lower() == "[token]": + in_token = True + out.append(line) + continue + if stripped.startswith("[") and stripped.endswith("]"): + in_token = False + if in_token and "=" in line and stripped.lower().split("=", 1)[0].strip() == "token": + if not replaced: + out.append(f"token={api_key}") + replaced = True + continue + out.append(line) + if "[Token]" not in {line.strip() for line in out}: + out.extend(["[Token]", f"token={api_key}"]) + elif not replaced: + idx = next(i for i, line in enumerate(out) if line.strip().lower() == "[token]") + out.insert(idx + 1, f"token={api_key}") + ini.write_text("\n".join(out) + "\n", encoding="utf-8") + + +def materialize_lib(api_key: str, lib_root: Path, source: Path | None = None) -> Path: + """复制包内 lib 到私有目录并写入 token, 返回 DLL 路径。 + + marker 记 token 指纹 + DLL mtime/size, 未变化时直接复用(每轮 realtime + 轮询都会走到这里, 不能反复复制几十 MB 的 DLL)。 + """ + src = source or package_lib_dir() + dll = src / LIB_DLL_NAME + if not dll.exists(): + raise TdxQuantError(f"tdxaidata lib 缺少 {LIB_DLL_NAME}: {src}") + lib_root.mkdir(parents=True, exist_ok=True) + marker = lib_root / ".tdxquant_lib_marker" + stat = dll.stat() + want = f"{hashlib.sha256(api_key.encode()).hexdigest()}:{stat.st_mtime_ns}:{stat.st_size}" + copied = lib_root / LIB_DLL_NAME + if marker.exists() and marker.read_text(encoding="utf-8") == want and copied.exists(): + return copied + for item in src.iterdir(): + if item.is_file(): + shutil.copy2(item, lib_root / item.name) + for ini_name in ("TdxAiData.ini", "user.ini"): + write_token_ini(lib_root / ini_name, api_key) + marker.write_text(want, encoding="utf-8") + return copied + + +class TdxQuantClient: + """原生 tqs 的薄封装: 重试、股票池枚举、快照/日K/除权三个取数面。""" + + def __init__(self, api_key: str, lib_root: Path | None = None) -> None: + from app.config import settings + + self.api_key = api_key + self._lib_root = lib_root or (settings.data_dir / "user_data" / "tdxquant_lib") + dll = materialize_lib(api_key, self._lib_root) + os.environ[LIB_ENV] = str(dll) + from tdxaidata import tqs + from tdxaidata.tdxaidata import _NativeTdxAiData + + self._tqs = tqs + self._native_cls = _NativeTdxAiData + self._universe: list[str] = [] + self._universe_at = 0.0 + self._warmup() + + def _warmup(self, attempts: int = 5) -> None: + """DLL 冷启动后有一段"未就绪"窗口, 业务调用持续失败(实测首批整批丢弃)。 + + 用最小快照轮询到就绪; 一直失败不抛 — 后续调用可能自愈, 只留告警。 + """ + for _ in range(attempts): + try: + rows = self._tqs.get_market_snapshot_batch( + stock_list=["600519.SH"], field_list=["Now"] + ) + if isinstance(rows, dict) and rows: + return + except Exception: # noqa: BLE001 — 就绪探测, 任何异常都重试 + pass + time.sleep(1.2) + logger.warning("TdxQuant 原生层预热未成功(%d 次), 首批业务调用可能失败重试", attempts) + + def close(self) -> None: # 原生 DLL 常驻进程, 无可回收句柄; 保留接口对齐 loader + self._universe.clear() + + # ---- 原生调用(重试包装) ---- + def _raw(self, method: str, *args): + """直接调原生层并 _clean 解包; 异常重试 3 次、空返回补试 1 次。 + + 空闲后首连 "Invalid server response" 实测重试即恢复; 坏代码等 + 永久错误重试也只会快失败(返回 None), 不至于长阻塞。 + """ + last: Exception = TdxQuantError(f"{method} 未知失败") + for attempt in range(_RETRIES): + try: + out = self._tqs._clean(getattr(self._native_cls.get_instance(), method)(*args)) + if out is not None: + return out + last = TdxQuantError(f"{method} 返回空数据(第 {attempt + 1} 次)") + if attempt + 1 >= _NONE_RETRIES: + break + except Exception as exc: # noqa: BLE001 — 原生 ctypes 层异常类型不可枚举 + last = TdxQuantError(f"{method}: {exc}") + time.sleep(_RETRY_SLEEP_S) + raise last + + # ---- 股票池 ---- + def stock_universe(self) -> list[str]: + """五市场全 A 代码(带交易所后缀)。12h 缓存, 避免每轮快照重复枚举。""" + now = time.time() + if self._universe and now - self._universe_at < _UNIVERSE_TTL_S: + return list(self._universe) + last: TdxQuantError = TdxQuantError("股票列表为空(五个市场均无返回)") + for _attempt in range(_RETRIES): + out: list[str] = [] + try: + for market in _MARKETS: + lst = self._tqs.get_stock_list(market=market, list_type=0) + out.extend(code for code in (lst or []) if isinstance(code, str) and code) + except Exception as exc: # noqa: BLE001 — SDK 内部异常类型不可枚举 + last = TdxQuantError(f"get_stock_list: {exc}") + out = [] + if out: + self._universe, self._universe_at = out, now + return list(out) + time.sleep(_RETRY_SLEEP_S) + raise last + + # ---- 快照 ---- + def snapshot_batch(self, codes: list[str]) -> dict[str, dict]: + """批量快照 → {code: row(字段见 SNAPSHOT_FIELDS)}。SDK 内部已做批处理。 + + 空闲后首连易 "Invalid server response"(实测重试即恢复), 批级失败必须 + 重试 — provider 只做批间隔离, 这里不重试会整批丢标的。 + """ + if not codes: + return {} + last: TdxQuantError = TdxQuantError(f"snapshot_batch({len(codes)} 只): 未知失败") + for _attempt in range(_RETRIES): + try: + rows = self._tqs.get_market_snapshot_batch( + stock_list=list(codes), field_list=list(SNAPSHOT_FIELDS) + ) + except Exception as exc: # noqa: BLE001 — SDK 内部异常类型不可枚举 + last = TdxQuantError(f"snapshot_batch({len(codes)} 只): {exc}") + time.sleep(_RETRY_SLEEP_S) + continue + if isinstance(rows, dict) and rows: + return rows + last = TdxQuantError(f"snapshot_batch({len(codes)} 只): 空返回") + time.sleep(_RETRY_SLEEP_S) + raise last + + # ---- K 线(count 模式, 原始列) ---- + def his_dats(self, code: str, end_d: date, count: int, period: str = "1d") -> dict: + """单标的 K 线原生 dict: Date 'YYYYMMDD' / Open / High / Low / Close / + Volume(股) / Amount(万元) 等。count 模式下 start 恒空(服务端只用 end)。 + """ + if count <= 0: + return {} + end = f"{end_d:%Y-%m-%d} 00:00:00" + out = self._raw("get_his_dats", code, "", end, period, "none", int(count)) + return out if isinstance(out, dict) else {} + + # ---- 除权事件 ---- + def divid_factors( + self, code: str, start_d: date | None = None, end_d: date | None = None + ) -> list[dict]: + """单标的除权事件, 每行 {Date 'YYYYMMDD', Type, Bonus, ShareBonus, + AllotPrice, Allotment} — 金额/比例均为**每 10 股**口径(Bonus=10派X元)。 + """ + start = f"{start_d:%Y-%m-%d}" if start_d else "" + end = f"{end_d:%Y-%m-%d}" if end_d else "" + try: + out = self._tqs.get_divid_factors(code, start, end) + except Exception as exc: # SDK 内部异常类型不可枚举 + raise TdxQuantError(f"get_divid_factors({code}): {exc}") from exc + if isinstance(out, list): + return out + # 环境装有 pandas 时 SDK 返回 DataFrame, 归一为 list[dict]。 + # 注意 Date 会成为 DataFrame 索引(同 get_market_data 的日期索引惯例), + # reset_index 后若列名漂移, 把首列恢复为 'Date'。 + to_dict = getattr(out, "to_dict", None) + reset_index = getattr(out, "reset_index", None) + if callable(to_dict) and callable(reset_index): + try: + records = reset_index().to_dict("records") + if isinstance(records, list) and records: + if "Date" not in records[0]: + first_key = next(iter(records[0])) + if first_key not in ("Type", "Bonus", "AllotPrice", "ShareBonus", "Allotment"): + for row in records: + row["Date"] = row.pop(first_key) + return records + except Exception: # noqa: BLE001 — 转换失败按结构异常处理 + pass + raise TdxQuantError(f"get_divid_factors({code}) 返回异常结构: {type(out)!r}") diff --git a/backend/app/plugins/tdxquant/plugin.yaml b/backend/app/plugins/tdxquant/plugin.yaml new file mode 100644 index 00000000..69318651 --- /dev/null +++ b/backend/app/plugins/tdxquant/plugin.yaml @@ -0,0 +1,23 @@ +# TdxQuant — 通达信量化数据服务插件 (https://help.tdx.com.cn/quant/) +# 原生 TCP 私有协议 (tdxaidata 包, ctypes 包装跨平台 DLL), runtime: python。 +# tdxaidata 已是后端默认依赖; 只需配置 TDXQUANT_API_KEY, 未配置时灰显不可切换。 + +name: tdxquant +display_name: "TdxQuant (TdxAiData)" +runtime: python +entry: app.plugins.tdxquant.provider:TdxQuantProvider +check: app.plugins.tdxquant.provider:availability +datasets: [realtime, depth5, daily, adj_factor, minute, full_minute] +api_key_env: TDXQUANT_API_KEY # 声明后设置页提供 Key 输入框(先探后存, secrets.json 优先) +billing: points # 积分计费标识: 设置页每个能力芯片/适配行标注「积分」, 消耗以商城积分明细为准 +key_manage_url: "https://vip.tdx.com.cn/site/app/pc-mall/main.html#/aiKey" # Key/积分管理页(通达信商城), 设置页 Key 区提供入口 +description: "通达信 TdxAiData 数据服务(原生 TCP 协议, 依赖已内置): 全市场实时快照、五档盘口、日K、除权因子、分钟K、全量分钟六项能力; 按积分计费, 未接入财务, 自动回退 TickFlow。" +billing_note: "积分消耗与轮询频率、拉取范围直接相关: 间隔越短、范围越大消耗越高。建议实时轮询间隔 ≥60 秒, 全量分钟按需启用; 具体扣减以商城「积分明细」为准。" +usage_notes: + - "实时快照: 五市场全 A 股票池, 600 只/批, 全市场一轮约 10 批(30-60 秒)" + - "五档盘口: 随快照返回, 不产生额外请求" + - "日K / 除权因子: count 模式单标的拉取(原始价); 因子按交易所公式推导, 带涨跌停自检" + - "分钟K: 1m 逐标的, 240 根/日, 深度约 40 交易日" + - "全量分钟: 逐标的拉取, 全市场一轮约 10 分钟量级(上游无批量端点)" +install_hint: "依赖已内置; 在下方输入框填写数据服务 Key(通达信个人版商城-会员中心-积分和Key管理创建), 或配置环境变量 TDXQUANT_API_KEY" +homepage: "https://help.tdx.com.cn/quant/" diff --git a/backend/app/plugins/tdxquant/probe.py b/backend/app/plugins/tdxquant/probe.py new file mode 100644 index 00000000..16d2ae68 --- /dev/null +++ b/backend/app/plugins/tdxquant/probe.py @@ -0,0 +1,162 @@ +"""候选 Key 探测 — 独立子进程「格式 + 连通性」探测。 + +原生 DLL 每进程加载一次且绑定一个 token: 主进程可能已用旧 Key 加载, +候选 Key 无法在进程内替换, 因此探测必须另起进程(临时 lib 目录 + +TDX_AI_DATA_LIB 指向), 结论以 JSON 回传。 + +可靠性设计(实测教训): + - DLL 原生握手固定 ~30s(同 IP 并发会话疑似服务端节流), 超时必须 >45s; + - 子进程可能在原生层直接死亡(段错误级, 无任何输出) — 每阶段把进度写入 + 父进程指定的进度文件, 父进程据此定位死点并附退出码, 不再返回"无输出"; + - TDX 会话按 IP 复用: 同 IP 有活跃好 Key 会话时(常驻后端必然如此), + 错误 Key 的日K也能返回 ⇒ "无效 Key 拒绝"不可靠。本探测只验证 + 格式 + 连通性, Key 真伪由首次实拉(test_dataset / 日常轮询)暴露。 +""" +from __future__ import annotations + +import json +import logging +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +logger = logging.getLogger(__name__) + +_PROBE_ENV_KEY = "TDX_PROBE_CANDIDATE_KEY" +_PROBE_ENV_TMP = "TDX_PROBE_TMP_DIR" + +# 子进程入口: 复用 client.materialize_lib 保持与主进程同一套物化/写 token 逻辑 +_SCRIPT = """ +import json +import os +import time +from pathlib import Path + +key = os.environ.pop("TDX_PROBE_CANDIDATE_KEY", "") +tmp = Path(os.environ.pop("TDX_PROBE_TMP_DIR", "")) +progress = tmp / "progress.json" if str(tmp) else None + + +def stage(name, **extra): + if progress is None: + return + try: + progress.write_text(json.dumps({"stage": name, **extra}), encoding="utf-8") + except Exception: + pass + + +try: + stage("imports") + from app.plugins.tdxquant import client as tc + + stage("materialize") + dll = tc.materialize_lib(key, tmp) + os.environ[tc.LIB_ENV] = str(dll) + + stage("load_dll") + from tdxaidata.tdxaidata import _NativeTdxAiData + + native = _NativeTdxAiData.get_instance() + + # 连通性判定: 快照端点不依赖 token 鉴权, 拿到数据即证明 + # 服务器可达 + DLL 正常。轮询容忍 DLL 冷启动的未就绪窗口。 + stage("start") + reachable = False + for _ in range(3): + try: + if native.get_market_snapshot_batch("600519.SH", "Now"): + reachable = True + break + except Exception: + pass + time.sleep(1.2) + if not reachable: + stage("done", ok=False) + print(json.dumps({"ok": False, "reason": "行情服务器连接失败(网络不可达)"})) + raise SystemExit(0) + + # 日K仅作辅助信号(会话按 IP 复用时错误 Key 也能通过, 不能作拒绝依据) + stage("auth") + aux = "" + try: + raw = native.get_his_dats( + "600519.SH", "", "2026-01-01 00:00:00", "1d", "none", 3 + ) + from tdxaidata import tqs + + v = tqs._clean(raw) + aux = "his_ok" if isinstance(v, dict) and v.get("Date") else "his_empty" + except Exception as exc: + aux = "his_exc" + + stage("done", ok=True) + print(json.dumps({"ok": True, "reason": "", "aux": aux})) +except Exception as exc: + stage("exception", reason=str(exc)[:300]) + print(json.dumps({"ok": False, "reason": str(exc)[:300]})) +""" + + +def _read_stage(tmp_dir: str) -> str: + try: + data = json.loads((Path(tmp_dir) / "progress.json").read_text(encoding="utf-8")) + except Exception: # noqa: BLE001 — 进度文件缺失/损坏按未知处理 + return "未知(无进度文件)" + stage = str(data.get("stage", "?")) + if stage == "exception": + return f"异常: {str(data.get('reason', ''))[:150]}" + return stage + + +def _tail(text: str, limit: int = 150) -> str: + lines = (text or "").strip().splitlines() + return lines[-1][:limit] if lines else "(空)" + + +def run(api_key: str, timeout_s: float = 75.0) -> tuple[bool, str]: + """子进程探测候选 Key: 格式由调用方把关, 这里验证连通性(快照可达)。""" + if not api_key: + return False, "Key 为空" + backend_root = Path(__file__).resolve().parents[3] + tmp_dir = tempfile.mkdtemp(prefix="tdx_probe_") + env = { + **os.environ, + _PROBE_ENV_KEY: api_key, + _PROBE_ENV_TMP: tmp_dir, + "PYTHONPATH": str(backend_root), + } + try: + try: + proc = subprocess.run( + [sys.executable, "-c", _SCRIPT], + capture_output=True, + text=True, + timeout=timeout_s, + cwd=str(backend_root), + env=env, + ) + except subprocess.TimeoutExpired: + stage = _read_stage(tmp_dir) + return False, f"探测超时({timeout_s:.0f}s, 停在阶段[{stage}]) — 原生握手约需 30s, 请检查网络后重试" + for line in reversed((proc.stdout or "").splitlines()): + line = line.strip() + if line.startswith("{"): + try: + payload = json.loads(line) + except json.JSONDecodeError: + break + if payload.get("ok"): + return True, "ok(连通正常; Key 有效性以首次拉取为准)" + return False, f"探测失败: {payload.get('reason', '未知原因')}" + # 无 JSON 结论: 原生层直接死亡(无输出)或入口即崩 — 用进度文件定位死点 + stage = _read_stage(tmp_dir) + return False, ( + f"探测失败: 子进程死在阶段[{stage}], exit={proc.returncode}, " + f"stderr={_tail(proc.stderr)}" + ) + finally: + shutil.rmtree(tmp_dir, ignore_errors=True) diff --git a/backend/app/plugins/tdxquant/provider.py b/backend/app/plugins/tdxquant/provider.py new file mode 100644 index 00000000..2dd11fcf --- /dev/null +++ b/backend/app/plugins/tdxquant/provider.py @@ -0,0 +1,782 @@ +"""TdxQuant(通达信量化数据服务)内置数据源 provider。 + +方法签名对齐 custom.GenericHTTPProvider(service 分流点按这套签名调用), +插件被 loader 扫描注册后, 各 service 无需改动即可路由。 + +实现数据集: + - realtime A 股全市场快照(五市场股票池, 600 只/批) + - depth5 五档盘口(快照 Buyp/Buyv/Sellp/Sellv) + - daily A 股日K, 原始价(dividend=none), count 模式单标的拉取 + - adj_factor 除权因子: 事件接口 + 本地K线配价, 交易所公式推导, 涨跌停自检 + - minute 分钟K(period=1m count 模式, 单标的 240 根/日, 原始价) + - full_minute 全量分钟: 修复轮 get_intraday_batch(当日窗口) + + 稳态增量 get_intraday_latest(每股最新 count 根)。 +未声明 financial → provider_has_dataset 为 False, 回退 tickflow。 + +单位与口径 (CONTRIBUTING §3, 不可凭字段名推断; 2026-10 实测): + - 快照 Volume 已是手, 直接透传; Amount 为万元 → ×10000 转元。 + - 快照无涨跌幅字段 → change_pct = (Now-LastClose)/LastClose, 小数制。 + - 日K Volume 为股 → floor(/100) 转手; Amount 为万元 → ×10000 转元。 + - 分钟K与日K同源同口径 (实测茅台全天 240 根 sum(Volume)=3511100 ≈ 日K + 3511051 股): Volume 股 → floor(/100) 转手; Amount 万元 → ×10000 转元; + datetime = Date+Time 拼北京墙钟 naive (契约 §分钟K)。 + - 日K取数 dividend="none" 锁定原始价, 前复权由 indicators.pipeline 用本地因子计算。 + - 除权事件 Bonus/ShareBonus/Allotment 均为每 10 股口径 → ÷10 转每股再进公式。 + - ex_factor 为单事件比值(非累积), 累积链由 pipeline._apply_adj_factor 构建。 +""" + +from __future__ import annotations + +import contextlib +import logging +import math +import time +from collections.abc import Callable, Iterator +from dataclasses import dataclass, field +from datetime import date, datetime, timedelta +from zoneinfo import ZoneInfo + +import polars as pl + +from app.plugins.tdxquant.client import SNAPSHOT_FIELDS, TdxQuantClient, TdxQuantError + +logger = logging.getLogger(__name__) + +# 只声明真实提供的数据集; 其余数据集 provider_has_dataset 返回 False → 回退 tickflow +_DATASETS = ("realtime", "depth5", "daily", "adj_factor", "minute", "full_minute") + +API_KEY_ENV = "TDXQUANT_API_KEY" +SECRETS_FIELD = "tdxquant_api_key" # UI 配置的 Key 存 secrets.json, 优先级高于 .env + +_SNAPSHOT_BATCH = 600 # 实测 600 只/批正常返回 +_HIST_INTERVAL_S = 0.08 # 单标的 K 线/事件请求节流 +_HIST_MAX_COUNT = 2600 # count 模式单次上限(≈10.6 年日K, 项目深窗口即 10 年) +_ADJ_BACKDAYS = 30 # 推导因子时向前找"除权日前收盘"的回看天数(容忍长期停牌) +_ADJ_FUTURE_DAYS = 15 # 未来已公告事件窗口: 无前收盘自然跳过, 滚动增量轮补上 +_BEIJING = ZoneInfo("Asia/Shanghai") + +# 内部指数代码 → TDX 快照代码(实测 2026-10: 上证指数走 TDX 999999 别名) +_INDEX_CODE_MAP = {"000001.SH": "999999.SH"} + +_DAILY_SCHEMA = { + "symbol": pl.String, "date": pl.Date, + "open": pl.Float64, "high": pl.Float64, "low": pl.Float64, "close": pl.Float64, + "volume": pl.Int64, "amount": pl.Float64, +} +_ADJ_SCHEMA = { + "symbol": pl.String, "trade_date": pl.Date, "ex_factor": pl.Float64, + "dividend": pl.Float64, "bonus": pl.Float64, "allot": pl.Float64, + "allot_price": pl.Float64, "prev_close": pl.Float64, +} +_MINUTE_SCHEMA = { + "symbol": pl.String, "datetime": pl.Datetime("us"), + "open": pl.Float64, "high": pl.Float64, "low": pl.Float64, "close": pl.Float64, + "volume": pl.Float64, "amount": pl.Float64, +} +_MINUTE_BARS_PER_DAY = 240 # A 股 1m 一交易日根数(含收盘集合竞价分钟) +_MINUTE_MAX_COUNT = 9600 # 实测单次 count=9600(40 交易日)正常返回 +_MINUTE_INTERVAL_S = 0.08 # 单标的分钟请求节流(与日K同档) + + +def get_api_key() -> str: + from app import secrets_store + + return secrets_store.get_env_backed_secret(SECRETS_FIELD, API_KEY_ENV) + + +def _module_available() -> bool: + try: + import importlib.util + + return importlib.util.find_spec("tdxaidata") is not None + except Exception: # noqa: BLE001 — find_spec 对损坏安装也会抛 + return False + + +def availability() -> tuple[bool, str]: + """loader 启动自检: tdxaidata 已安装 + Key 已配置才注册。不抛异常。""" + if not _module_available(): + return False, "tdxaidata 未安装(默认依赖, 请检查 uv sync)" + if not get_api_key(): + return False, f"缺少 API Key(可在下方输入框直接填写,或配置环境变量 {API_KEY_ENV})" + return True, "ok" + + +def probe_api_key(api_key: str) -> tuple[bool, str]: + """候选 Key 探测: 格式校验 + 子进程连通性实探。 + + TDX 会话按 IP 复用(同 IP 活跃好 Key 会话时错误 Key 也能拉到日K, 实测), + "无效 Key 拒绝"不可靠 — 格式明显不对直接拒绝, 连通性通过即保存, + Key 真伪由首次实拉(test_dataset / 日常轮询)暴露, 卡片可一键试拉复核。 + """ + key = (api_key or "").strip() + if not key.startswith("TDX-") or len(key) < 12: + return False, "Key 格式应为 TDX- 开头(通达信商城创建的数据服务 Key)" + from app.plugins.tdxquant import probe + + return probe.run(key) + + +@dataclass +class _TdxQuantConfig: + """轻量 config shim, 让 custom loader 的 provider_has_dataset 能识别本 provider。""" + + name: str = "tdx" + display_name: str = "TdxQuant" + datasets: dict = field(default_factory=lambda: dict.fromkeys(_DATASETS)) + path: None = None + builtin: bool = True + + +def _to_float(value) -> float | None: + if value is None: + return None + try: + return float(value) + except (TypeError, ValueError): + return None + + +def _compact_date(value) -> date | None: + """原生 Date 'YYYYMMDD'(或 DataFrame 归一后的 ISO 'YYYY-MM-DD…')→ date。 + + 非法值返回 None, 不伪造。 + """ + try: + text = str(value).strip() + if len(text) >= 10 and text[4] == "-" and text[7] == "-": + text = text[:10].replace("-", "") + return date(int(text[:4]), int(text[4:6]), int(text[6:8])) + except (TypeError, ValueError): + return None + + +def _snapshot_ms(refresh_time, fallback_ms: int) -> int: + """RefreshTime 'HHMMSS'(服务端北京墙钟) + 今日北京日期 → epoch ms。 + + 解析失败退回本地时间; A 股时段无跨日歧义。 + """ + try: + text = str(refresh_time).strip().zfill(6) + hh, mm, ss = int(text[:2]), int(text[2:4]), int(text[4:6]) + if hh > 23 or mm > 59 or ss > 59: + raise ValueError(text) + now_bj = datetime.now(_BEIJING) + stamp = datetime(now_bj.year, now_bj.month, now_bj.day, hh, mm, ss, tzinfo=_BEIJING) + return int(stamp.timestamp() * 1000) + except (TypeError, ValueError): + return fallback_ms + + +def _map_snapshot_row(symbol: str, row: dict, fallback_ms: int) -> dict | None: + """TDX 快照行 → 内部 realtime record。字段缺失按口径推导, 不伪造数据。""" + last = _to_float(row.get("Now")) + prev = _to_float(row.get("LastClose")) + if last is None and prev is None: + return None + + change_amount = last - prev if (last is not None and prev is not None) else None + # 与 quote_service 推导同口径: 小数制, 不乘 100 + change_pct = change_amount / prev if (change_amount is not None and prev) else None + + volume = _to_float(row.get("Volume")) # 快照已为手, 零换算 + amount = _to_float(row.get("Amount")) # 万元 → 元 + + return { + "symbol": symbol, + "name": None, # 快照无名称, 由下游维表关联 + "last_price": last, + "prev_close": prev, + "open": _to_float(row.get("Open")), + "high": _to_float(row.get("Max")), + "low": _to_float(row.get("Min")), + "volume": math.floor(volume) if volume is not None else None, + "amount": amount * 10_000.0 if amount is not None else None, + "change_pct": change_pct, + "change_amount": change_amount, + "amplitude": None, # 快照未提供, 不启发式计算 + "turnover_rate": None, # 需股本口径 (§3.4), 交给 enriched 管道 + "timestamp": _snapshot_ms(row.get("RefreshTime"), fallback_ms), + "session": None, + } + + +def _ref_price( + prev_close: float, dividend: float, bonus: float, allot: float, allot_price: float +) -> float | None: + """交易所除权参考价: (P - D + AR·AP) / (1 + S + AR), 四舍五入(half-up)保留 2 位。""" + denom = 1.0 + bonus + allot + if denom <= 0: + return None + x = (prev_close - dividend + allot * allot_price) / denom + return math.floor(x * 100 + 0.5) / 100 + + +def _price_limit(symbol: str) -> float: + """按代码前缀给涨跌停幅度(自检容差用): 创业板/科创板 20%, 北交所 30%, 主板 10%。""" + code = symbol.split(".")[0] + if code.startswith(("300", "301", "688", "689")): + return 0.20 + if code.startswith(("8", "4", "92")): + return 0.30 + return 0.10 + + +def _kline_rows(symbol: str, raw: dict, start_d: date | None, end_d: date) -> list[dict]: + """原生 count 模式 K 线 dict → 内部行(窗口过滤, volume 股→手, amount 万元→元)。""" + + def _col(name: str, i: int) -> float | None: + values = raw.get(name) + return _to_float(values[i]) if values and i < len(values) else None + + dates = raw.get("Date") or [] + out: list[dict] = [] + for i, compact in enumerate(dates): + d = _compact_date(compact) + if d is None or d > end_d: + continue + if start_d is not None and d < start_d: + continue + volume = _col("Volume", i) + amount = _col("Amount", i) + out.append({ + "symbol": symbol, + "date": d, + "open": _col("Open", i), + "high": _col("High", i), + "low": _col("Low", i), + "close": _col("Close", i), + "volume": math.floor(volume / 100.0) if volume is not None else None, + "amount": amount * 10_000.0 if amount is not None else None, + }) + return out + + +def _bars_for_window(span_days: int) -> int: + """窗口自然日跨度 → count 模式请求根数(交易日≈自然日×5/7, 加停牌冗余)。""" + return max(30, min(_HIST_MAX_COUNT, int(span_days * 5 / 7) + 20)) + + +def _minute_dt(compact_date, hhmmss) -> datetime | None: + """Date 'YYYYMMDD' + Time 'HHMMSS' → 北京墙钟 naive datetime; 非法返回 None。""" + d = _compact_date(compact_date) + if d is None: + return None + t = str(hhmmss or "").strip() + if len(t) < 6 or not t[:6].isdigit(): + return None + try: + return datetime(d.year, d.month, d.day, int(t[:2]), int(t[2:4]), int(t[4:6])) + except ValueError: + return None + + +def _minute_rows( + symbol: str, raw: dict, start_dt: datetime | None, end_dt: datetime | None, + today_only: date | None = None, +) -> list[dict]: + """原生 1m K 线 dict → 内部分钟行(datetime 北京墙钟, volume 股→手, amount 万元→元)。 + + today_only 非空时仅保留该交易日的行(修复轮用: count 模式尾部会带前一日余量)。 + """ + dates = raw.get("Date") or [] + times = raw.get("Time") or [] + out: list[dict] = [] + for i, compact in enumerate(dates): + dt = _minute_dt(compact, times[i] if i < len(times) else None) + if dt is None: + continue + if today_only is not None and dt.date() != today_only: + continue + if start_dt is not None and dt < start_dt: + continue + if end_dt is not None and dt > end_dt: + continue + + def _col(name: str, idx: int = i) -> float | None: + values = raw.get(name) + return _to_float(values[idx]) if values and idx < len(values) else None + + close = _col("Close") + if close is None: + continue # 停牌/无成交分钟, 不伪造 + volume = _col("Volume") + amount = _col("Amount") + out.append({ + "symbol": symbol, "datetime": dt, + "open": _col("Open"), "high": _col("High"), + "low": _col("Low"), "close": close, + "volume": float(math.floor(volume / 100.0)) if volume is not None else None, + "amount": amount * 10_000.0 if amount is not None else None, + }) + return out + + +def _minute_count_for_window(start_dt: datetime | None, end_dt: datetime) -> int: + """窗口 → 单次 count 请求根数(交易日×240 + 两日余量, 封顶 _MINUTE_MAX_COUNT)。""" + span_days = (end_dt.date() - start_dt.date()).days if start_dt else 30 + return max(240, min(_MINUTE_MAX_COUNT, int(span_days * 5 / 7 * _MINUTE_BARS_PER_DAY) + 480)) + + +class TdxQuantProvider: + """tdx 数据源。realtime = 五市场全 A 快照批拉; depth5 复用同一快照的五档数组。""" + + name = "tdxquant" + builtin = True + + def __init__(self) -> None: + self.config = _TdxQuantConfig() + self._client: TdxQuantClient | None = None + + def close(self) -> None: # loader.load_all 重建注册表时会对每个 provider 调 close + if self._client is not None: + with contextlib.suppress(Exception): + self._client.close() + self._client = None + + def _get_client(self) -> TdxQuantClient: + if self._client is None: + self._client = TdxQuantClient(api_key=get_api_key()) + return self._client + + # ---- realtime ---- + def get_realtime(self) -> list[dict]: + """全市场实时快照 → 内部 realtime records。失败软返回空列表(不阻断轮询)。""" + client = self._get_client() + try: + universe = client.stock_universe() + except TdxQuantError as e: + logger.warning("TdxQuant 股票池枚举失败: %s", e) + return [] + + fallback_ms = int(time.time() * 1000) + records: list[dict] = [] + dropped = failed_batches = 0 + for i in range(0, len(universe), _SNAPSHOT_BATCH): + codes = universe[i : i + _SNAPSHOT_BATCH] + try: + rows = client.snapshot_batch(codes) + except TdxQuantError as e: + failed_batches += 1 + logger.warning("TdxQuant 快照批次失败(%d 只, 第 %d 批): %s", len(codes), i // _SNAPSHOT_BATCH + 1, e) + continue + for code, row in rows.items(): + rec = _map_snapshot_row(code, row, fallback_ms) + if rec is not None: + records.append(rec) + else: + dropped += 1 + if failed_batches: + logger.warning("TdxQuant 快照 %d/%d 批失败, 本轮仅返回 %d 条", + failed_batches, math.ceil(len(universe) / _SNAPSHOT_BATCH), len(records)) + if not records and universe: + logger.warning("TdxQuant 实时行情本轮 0 条(请求 %d 只), 疑似连接或协议异常", len(universe)) + elif dropped: + logger.info("TdxQuant 实时行情拉取完成: %d 条(丢弃 %d 行无价快照)", len(records), dropped) + return records + + def get_realtime_indices(self, symbols: list[str]) -> list[dict] | None: + """指数实时快照(可选插件协议, quote_service 鸭子类型调用)。 + + TDX 快照直接支持指数代码; 未映射的代码跳过(上层用本地日K兜底)。 + 失败返回 None 保留上轮有效缓存, 成功但无数据返回 []。 + """ + wanted: list[str] = [] + for s in symbols: + if not s: + continue + code = _INDEX_CODE_MAP.get(s, s) + if code not in wanted: + wanted.append(code) + if not wanted: + return [] + try: + rows = self._get_client().snapshot_batch(wanted) + except TdxQuantError as e: + logger.warning("TdxQuant 指数行情拉取失败: %s", e) + return None + fallback_ms = int(time.time() * 1000) + records = [] + for code, row in rows.items(): + rec = _map_snapshot_row(code, row, fallback_ms) + if rec is not None: + records.append(rec) + logger.info("TdxQuant 指数行情拉取完成: %d 条(请求 %d 只)", len(records), len(wanted)) + return records + + # ---- daily ---- + def get_daily( + self, + symbols: list[str], + start_time: datetime | None, + end_time: datetime | None, + asset_type: str = "stock", + on_chunk_done: Callable[[int, int], None] | None = None, + ) -> pl.DataFrame: + chunks = [ + df + for df in self.iter_daily( + symbols, start_time=start_time, end_time=end_time, + asset_type=asset_type, on_chunk_done=on_chunk_done, + ) + if not df.is_empty() + ] + return pl.concat(chunks, how="diagonal_relaxed") if chunks else pl.DataFrame(schema=_DAILY_SCHEMA) + + def iter_daily( + self, + symbols: list[str], + start_time: datetime | None, + end_time: datetime | None, + asset_type: str = "stock", + on_chunk_done: Callable[[int, int], None] | None = None, + ) -> Iterator[pl.DataFrame]: + """逐标的 count 模式拉日K → 内部契约(原始价, volume 手, amount 元)。 + + ETF/指数日K未实测覆盖, asset_type 仅接受 stock, 其余空手而归(上层回退)。 + """ + if not symbols or asset_type != "stock": + if on_chunk_done: + on_chunk_done(0, 0) + return + client = self._get_client() + start_d = start_time.date() if start_time else None + end_d = end_time.date() if end_time else date.today() + span = (end_d - start_d).days if start_d else _HIST_MAX_COUNT * 1.4 + count = _bars_for_window(int(span)) + for i, symbol in enumerate(symbols): + try: + raw = client.his_dats(symbol, end_d, count) + except TdxQuantError as e: + logger.warning("TdxQuant 日K %s 拉取失败: %s", symbol, e) + raw = {} + rows = _kline_rows(symbol, raw, start_d, end_d) + if rows and rows[0]["date"] is not None and start_d is not None and rows[0]["date"] > start_d: + logger.info("TdxQuant 日K %s: count=%d 仅覆盖到 %s(窗口起点 %s 之前的数据不在返回内)", + symbol, count, rows[0]["date"], start_d) + yield pl.DataFrame(rows, schema=_DAILY_SCHEMA) + if on_chunk_done: + on_chunk_done(i + 1, len(symbols)) + time.sleep(_HIST_INTERVAL_S) + + # ---- adj_factor ---- + def get_adj_factors( + self, + symbols: list[str], + start_time: datetime | None, + end_time: datetime | None, + asset_type: str = "stock", + on_chunk_done: Callable[[int, int], None] | None = None, + ) -> pl.DataFrame: + """除权因子 → 内部契约(symbol/trade_date/ex_factor, 单事件比值非累积) + + 事件明细列(dividend/bonus/allot/allot_price/prev_close, 每股口径)。 + + 数据链: divid_factors 事件接口(每 10 股口径 → ÷10 转每股) → 同一标的 + 一次 count 模式日K取齐窗口内收盘价 → 交易所公式推导 → 涨跌停自检剔除。 + 未来已公告事件无前收盘, 自然跳过, 滚动增量窗口补上。 + """ + if not symbols or asset_type != "stock": + return pl.DataFrame(schema=_ADJ_SCHEMA) + client = self._get_client() + start_d = start_time.date() if start_time else None + end_d = end_time.date() if end_time else date.today() + + out_rows: list[dict] = [] + for i, symbol in enumerate(symbols): + try: + events = client.divid_factors(symbol, start_d, end_d + timedelta(days=_ADJ_FUTURE_DAYS)) + except TdxQuantError as e: + logger.warning("TdxQuant 除权事件 %s 拉取失败: %s", symbol, e) + events = [] + events = [ev for ev in events if _compact_date(ev.get("Date")) is not None] + if events: + events.sort(key=lambda ev: str(ev.get("Date"))) + first_ex = _compact_date(events[0]["Date"]) + closes = self._close_map(client, symbol, first_ex - timedelta(days=_ADJ_BACKDAYS), end_d) + out_rows.extend(self._factor_rows(symbol, events, closes, start_d, end_d)) + if on_chunk_done: + on_chunk_done(i + 1, len(symbols)) + time.sleep(_HIST_INTERVAL_S) + if not out_rows: + return pl.DataFrame(schema=_ADJ_SCHEMA) + return ( + pl.DataFrame(out_rows, schema=_ADJ_SCHEMA) + .unique(subset=["symbol", "trade_date"], keep="last") + .sort(["symbol", "trade_date"]) + ) + + def _close_map(self, client: TdxQuantClient, symbol: str, start_d: date, end_d: date) -> dict[date, float]: + """一次 count 模式日K → {交易日: 原始收盘价}(推导前收盘与自检共用)。""" + try: + raw = client.his_dats(symbol, end_d, _bars_for_window((end_d - start_d).days)) + except TdxQuantError as e: + logger.warning("TdxQuant 除权配价 %s 日K失败: %s", symbol, e) + return {} + out: dict[date, float] = {} + dates = raw.get("Date") or [] + closes = raw.get("Close") or [] + for i, compact in enumerate(dates): + d = _compact_date(compact) + close = _to_float(closes[i]) if i < len(closes) else None + if d is not None and close is not None: + out[d] = close + return out + + def _factor_rows( + self, symbol: str, events: list[dict], closes: dict[date, float], + start_d: date | None, end_d: date, + ) -> list[dict]: + days = sorted(closes) + rows: list[dict] = [] + for ev in events: + exd = _compact_date(ev.get("Date")) + if exd is None or exd > end_d + timedelta(days=_ADJ_FUTURE_DAYS): + continue + if start_d is not None and exd < start_d: + continue + # 每 10 股口径 → 每股 (Bonus=10派X元, ShareBonus=10送Y股, Allotment=10配Z股) + dividend = (_to_float(ev.get("Bonus")) or 0.0) / 10.0 + bonus = (_to_float(ev.get("ShareBonus")) or 0.0) / 10.0 + allot = (_to_float(ev.get("Allotment")) or 0.0) / 10.0 + allot_price = _to_float(ev.get("AllotPrice")) + prev_days = [d for d in days if d < exd] + if not prev_days: + continue # 日K窗口未覆盖(远古事件或未来已公告事件) + p = closes[prev_days[-1]] + ref = _ref_price(p, dividend, bonus, allot, allot_price or 0.0) + if ref is None or ref <= 0: + logger.warning("TdxQuant 除权因子: %s %s 参考价无法计算(P=%s D=%s S=%s AR=%s AP=%s), 跳过", + symbol, exd, p, dividend, bonus, allot, allot_price) + continue + factor = p / ref + ex_days = [d for d in days if d >= exd] + if ex_days: # 涨跌停自检: 错误因子会令复权后除权日涨跌幅超出板限 + ret = closes[ex_days[0]] / (p / factor) - 1.0 + if abs(ret) > _price_limit(symbol) + 0.02: + logger.warning("TdxQuant 除权因子自检剔除 %s %s: 复权后除权日涨跌幅 %.1f%% 超出涨跌停", + symbol, exd, ret * 100) + continue + rows.append({ + "symbol": symbol, "trade_date": exd, "ex_factor": factor, + "dividend": dividend, "bonus": bonus, "allot": allot, + "allot_price": allot_price, "prev_close": p, + }) + return rows + + # ---- minute / full_minute ---- + def get_minute( + self, + symbols: list[str], + start_time: datetime | None, + end_time: datetime | None, + asset_type: str = "stock", + freq: str = "1m", + on_chunk_done: Callable[[int, int], None] | None = None, + ) -> pl.DataFrame: + """分钟K → 内部契约(symbol/datetime 北京墙钟 naive/ohlcv/volume 手/amount 元)。 + + period=1m count 模式单标的拉取(实测 240 根/日, count≤9600 正常); + freq 仅支持 1m(上游有 5m 等档位但项目契约按 1m 路由), 其余空手而归还上层回退; + ETF/指数分钟未实测覆盖, asset_type 仅接受 stock。 + """ + chunks = list(self.iter_minute( + symbols, start_time=start_time, end_time=end_time, + asset_type=asset_type, freq=freq, on_chunk_done=on_chunk_done, + )) + parts = [df for df in chunks if not df.is_empty()] + return pl.concat(parts, how="diagonal_relaxed") if parts else pl.DataFrame(schema=_MINUTE_SCHEMA) + + def iter_minute( + self, + symbols: list[str], + start_time: datetime | None, + end_time: datetime | None, + asset_type: str = "stock", + freq: str = "1m", + on_chunk_done: Callable[[int, int], None] | None = None, + ) -> Iterator[pl.DataFrame]: + if not symbols or asset_type != "stock" or freq != "1m": + if on_chunk_done: + on_chunk_done(0, 0) + return + client = self._get_client() + end_dt = end_time or datetime.now() + count = _minute_count_for_window(start_time, end_dt) + for i, symbol in enumerate(symbols): + try: + raw = client.his_dats(symbol, end_dt.date(), count, period="1m") + except TdxQuantError as e: + logger.warning("TdxQuant 分钟K %s 拉取失败: %s", symbol, e) + raw = {} + rows = _minute_rows(symbol, raw, start_time, end_dt) + yield pl.DataFrame(rows, schema=_MINUTE_SCHEMA) + if on_chunk_done: + on_chunk_done(i + 1, len(symbols)) + time.sleep(_MINUTE_INTERVAL_S) + + def get_intraday_batch( + self, symbols: list[str], count: int = 300, asset_type: str = "stock", + ) -> pl.DataFrame: + """全量分钟修复轮: 当日窗口逐标的拉取, 只保留今日 1m 行。 + + count 模式尾部天然带前一日余量, 按 today_only 过滤; 单标的请求 + 240+count 余量根。全市场一轮 5579 次调用(含节流约 10 分钟量级), + 由 minute_refresh 以修复轮节奏调度。 + """ + if not symbols or asset_type != "stock": + return pl.DataFrame(schema=_MINUTE_SCHEMA) + client = self._get_client() + today = datetime.now().date() + fetch = max(_MINUTE_BARS_PER_DAY, min(count, 480)) + _MINUTE_BARS_PER_DAY + out: list[pl.DataFrame] = [] + for symbol in symbols: + try: + raw = client.his_dats(symbol, today, fetch, period="1m") + except TdxQuantError as e: + logger.warning("TdxQuant 修复轮 %s 拉取失败: %s", symbol, e) + continue + rows = _minute_rows(symbol, raw, None, None, today_only=today) + if rows: + out.append(pl.DataFrame(rows, schema=_MINUTE_SCHEMA)) + time.sleep(_MINUTE_INTERVAL_S) + return pl.concat(out, how="diagonal_relaxed") if out else pl.DataFrame(schema=_MINUTE_SCHEMA) + + def get_intraday_latest(self, symbols: list[str] | None = None, count: int = 3) -> pl.DataFrame: + """全量分钟稳态增量轮: 每只标的最新 count 根 1m(symbols 缺省 = 全市场股票池)。 + + 与修复轮同为逐标的 count 调用(无批量端点), 差别在根数少、负载轻; + 失败标的跳过, 空轮由调用方按空处理。 + """ + client = self._get_client() + targets = list(symbols) if symbols else [] + if not targets: + try: + targets = client.stock_universe() + except TdxQuantError as e: + logger.warning("TdxQuant 增量轮股票池枚举失败: %s", e) + return pl.DataFrame(schema=_MINUTE_SCHEMA) + count = max(1, min(int(count), 30)) + today = datetime.now().date() + out: list[pl.DataFrame] = [] + for j, symbol in enumerate(targets): + try: + raw = client.his_dats(symbol, today, count, period="1m") + except TdxQuantError as e: + logger.warning("TdxQuant 增量轮 %s 拉取失败: %s", symbol, e) + continue + rows = _minute_rows(symbol, raw, None, None) + if rows: + out.append(pl.DataFrame(rows, schema=_MINUTE_SCHEMA)) + if j % 1000 == 999: + logger.info("TdxQuant 增量轮进度: %d/%d", j + 1, len(targets)) + time.sleep(_MINUTE_INTERVAL_S) + return pl.concat(out, how="diagonal_relaxed") if out else pl.DataFrame(schema=_MINUTE_SCHEMA) + + # ---- depth5 ---- + def get_depth_batch(self, symbols: list[str]) -> dict[str, dict]: + """五档盘口 → {symbol: 标准盘口字典}。数量单位手(透传), 价格转 float。""" + client = self._get_client() + out: dict[str, dict] = {} + fallback_ms = int(time.time() * 1000) + for i in range(0, len(symbols), _SNAPSHOT_BATCH): + batch = symbols[i : i + _SNAPSHOT_BATCH] + try: + rows = client.snapshot_batch(batch) + except TdxQuantError as e: + logger.warning("TdxQuant 五档批次失败(第 %d 批): %s", i // _SNAPSHOT_BATCH + 1, e) + continue + for code, row in rows.items(): + entry = self._depth_entry(row, fallback_ms) + if entry is not None: + out[code] = entry + return out + + def _depth_entry(self, row: dict, fallback_ms: int) -> dict | None: + def _prices(name: str) -> list[float] | None: + values = row.get(name) + if not values: + return None + return [_to_float(v) for v in values] + + def _volumes(name: str) -> list[int] | None: + values = row.get(name) + if not values: + return None + out: list[int] = [] + for v in values: + f = _to_float(v) + out.append(math.floor(f) if f is not None else 0) + return out + + bid_prices = _prices("Buyp") + ask_prices = _prices("Sellp") + if not bid_prices or not ask_prices: + return None + return { + "bid_prices": bid_prices, + "bid_volumes": _volumes("Buyv") or [], + "ask_prices": ask_prices, + "ask_volumes": _volumes("Sellv") or [], + "timestamp": _snapshot_ms(row.get("RefreshTime"), fallback_ms), + } + + # ---- 设置页试拉 ---- + def test_dataset(self, dataset: str, symbols: list[str] | None = None) -> dict: + if dataset in ("daily", "adj_factor"): + syms = [s for s in (symbols or [])][:3] or ["000001.SZ"] + try: + if dataset == "daily": + df = self.get_daily(syms, datetime.now() - timedelta(days=30), datetime.now()) + else: + df = self.get_adj_factors(syms, datetime.now() - timedelta(days=365), datetime.now()) + except TdxQuantError as e: + return {"provider": self.name, "dataset": dataset, "rows": 0, "error": str(e)} + head = df.head(5).to_dicts() + for row in head: # date/datetime → ISO 字符串, 保证 JSON 可序列化 + for k, v in list(row.items()): + if isinstance(v, (date, datetime)): + row[k] = v.isoformat() + return {"provider": self.name, "dataset": dataset, "rows": df.height, + "columns": df.columns, "preview": head} + if dataset in ("minute", "full_minute"): + syms = [s for s in (symbols or [])][:3] or ["600519.SH"] + try: + end = datetime.now() + df = (self.get_intraday_batch(syms) if dataset == "full_minute" + else self.get_minute(syms, end - timedelta(days=3), end)) + except TdxQuantError as e: + return {"provider": self.name, "dataset": dataset, "rows": 0, "error": str(e)} + head = df.head(5).to_dicts() + for row in head: + for k, v in list(row.items()): + if isinstance(v, (date, datetime)): + row[k] = v.isoformat() + return {"provider": self.name, "dataset": dataset, "rows": df.height, + "columns": df.columns, "preview": head} + if dataset in ("realtime", "depth5"): + syms = [s for s in (symbols or [])][:5] or ["600519.SH", "000001.SZ"] + try: + rows = self._get_client().snapshot_batch(syms) + except TdxQuantError as e: + return {"provider": self.name, "dataset": dataset, "rows": 0, "error": str(e)} + fallback_ms = int(time.time() * 1000) + if dataset == "realtime": + head = [r for r in (_map_snapshot_row(c, row, fallback_ms) for c, row in rows.items()) if r] + return {"provider": self.name, "dataset": dataset, "rows": len(rows), + "columns": list(head[0].keys()) if head else list(SNAPSHOT_FIELDS), + "preview": head[:5]} + depths = {c: d for c, d in ((c, self._depth_entry(row, fallback_ms)) for c, row in rows.items()) if d} + return {"provider": self.name, "dataset": dataset, "rows": len(depths), + "columns": ["symbol", "bid_prices", "bid_volumes", "ask_prices", "ask_volumes", "timestamp"], + "preview": [{"symbol": c, **d} for c, d in list(depths.items())[:5]]} + return {"provider": self.name, "dataset": dataset, "rows": 0, + "error": f"tdx 插件未接入 {dataset} 数据集(自动回退 TickFlow)"} + + +__all__ = [ + "API_KEY_ENV", + "TdxQuantProvider", + "availability", + "get_api_key", + "probe_api_key", +] diff --git a/backend/app/secrets_store.py b/backend/app/secrets_store.py index 5eee014c..bb81da97 100644 --- a/backend/app/secrets_store.py +++ b/backend/app/secrets_store.py @@ -30,6 +30,16 @@ def _path() -> Path: return p +def file_path() -> Path: + """secrets.json 绝对路径 (设置页展示用)。 + + 该文件是 AI 与数据源 Key 的统一配置源, 支持用户用编辑器直接修改; + load() 按 mtime 签名缓存失效, 外部保存后下一次请求即读取最新内容, + 无需重启。 + """ + return _path() + + def load() -> dict: """读取 secrets.json (带 mtime 签名缓存)。返回深拷贝, 调用方可自由修改。""" global _cache, _cache_sig diff --git a/backend/app/services/ai_provider.py b/backend/app/services/ai_provider.py index 6a5fdcb6..a8841b71 100644 --- a/backend/app/services/ai_provider.py +++ b/backend/app/services/ai_provider.py @@ -26,6 +26,10 @@ CODEX_CLI_PROVIDER = "codex_cli" CODEX_DEFAULT_COMMAND = "codex" CODEX_SUPPORTED_REASONING_EFFORTS = {"none", "minimal", "low", "medium", "high", "xhigh"} OPENAI_DEFAULT_REASONING_EFFORT = "high" +# OpenAI 系两种请求协议: chat.completions (兼容面最广) 与 Responses (官方新协议)。 +CHAT_COMPLETIONS_PROTOCOL = "chat_completions" +RESPONSES_PROTOCOL = "responses" +_SUPPORTED_PROTOCOLS = {CHAT_COMPLETIONS_PROTOCOL, RESPONSES_PROTOCOL} _CODEX_ENV_ALLOWLIST = ( "PATH", @@ -124,6 +128,24 @@ def current_ai_provider() -> str: return secrets_store.get_ai_config("ai_provider", settings.ai_provider) or OPENAI_COMPAT_PROVIDER +def current_ai_protocol() -> str: + """当前 OpenAI 系请求协议: chat_completions (默认) | responses。 + + 仅对 openai/openai_compat 两种 provider 有意义; Codex CLI 走本地子进程, + 恒为 chat_completions。非法存储值一律回退默认, 不抛错。 + """ + if is_codex_cli_provider(): + return CHAT_COMPLETIONS_PROTOCOL + value = str( + secrets_store.get_ai_config("ai_protocol", settings.ai_protocol) or "" + ).strip().lower() + return value if value in _SUPPORTED_PROTOCOLS else CHAT_COMPLETIONS_PROTOCOL + + +def is_responses_protocol() -> bool: + return current_ai_protocol() == RESPONSES_PROTOCOL + + def current_openai_model() -> str: return secrets_store.get_ai_config("ai_model", settings.ai_model) @@ -318,6 +340,15 @@ async def generate_ai_text( _check_input_budget(messages, max_tokens=max_tokens) if is_codex_cli_provider(): return await _run_codex_cli(messages, max_tokens=max_tokens, timeout=max(timeout, 600.0)) + if is_responses_protocol(): + from app.services import ai_responses + + return await ai_responses.run_text( + messages, + temperature=temperature, + max_tokens=max_tokens, + timeout=timeout, + ) return await _run_openai_once( messages, temperature=temperature, @@ -356,13 +387,24 @@ async def generate_ai_text_with_tools( for _ in range(max_rounds): _check_input_budget(req_messages, max_tokens=max_tokens) - message = await _run_openai_message_once( - req_messages, - temperature=temperature, - max_tokens=max_tokens, - timeout=timeout, - tools=tool_schemas, - ) + if is_responses_protocol(): + from app.services import ai_responses + + message = await ai_responses.run_message_once( + req_messages, + temperature=temperature, + max_tokens=max_tokens, + timeout=timeout, + tools=tool_schemas, + ) + else: + message = await _run_openai_message_once( + req_messages, + temperature=temperature, + max_tokens=max_tokens, + timeout=timeout, + tools=tool_schemas, + ) if message is None: return req_messages @@ -446,6 +488,18 @@ async def stream_ai_text( yield await _run_codex_cli(messages, max_tokens=max_tokens, timeout=max(timeout, 600.0)) return + if is_responses_protocol(): + from app.services import ai_responses + + async for chunk in ai_responses.stream_text( + messages, + temperature=temperature, + max_tokens=max_tokens, + timeout=timeout, + ): + yield chunk + return + async for chunk in _stream_openai( messages, temperature=temperature, @@ -631,6 +685,57 @@ def _openai_client(api_key: str, timeout: float): ) +async def probe_ai_config( + *, + provider: str, + api_key: str, + base_url: str, + model: str, + protocol: str = "", + timeout: float = 20.0, +) -> str: + """草稿配置一次性连通性探测 — 不读写任何配置状态。 + + 「测试」必须与「保存」解耦: 旧实现先保存草稿再测试, 测试失败时坏配置 + 已经落盘并被状态栏展示为已连接。本函数用表单值直接建临时客户端发一条 + 最小消息, 成功与否都不触碰 secrets。 + + 与正式调用共用 base URL 规范化与协议分支 (chat.completions / responses); + 不传输出上限/温度等可选参数, 规避 max_tokens 与 max_completion_tokens 等 + 模型差异。返回模型可见文本 (推理模型可能为空串 — 200 即证明鉴权/端点/ + 模型可用); 传输层异常原样抛出, 由调用方格式化后回报。 + """ + if provider == CODEX_CLI_PROVIDER: + if not codex_cli_available(): + raise RuntimeError("Codex CLI 不可用: 未找到可运行的 codex 命令") + return "" + if not api_key: + raise RuntimeError("未配置 API Key") + if not base_url: + raise ValueError("请先填写 API 地址") + if not model: + raise ValueError("请先填写模型名称") + + from openai import AsyncOpenAI + + user_agent = secrets_store.get_ai_config("ai_user_agent", "") or settings.ai_user_agent + client = AsyncOpenAI( + api_key=api_key, + base_url=normalize_openai_base_url(base_url), + timeout=timeout, + max_retries=0, + default_headers={"User-Agent": user_agent}, + ) + prompt = [{"role": "user", "content": "Reply exactly: OK"}] + if protocol == RESPONSES_PROTOCOL: + resp = await client.responses.create(model=model, input=prompt) + return str(getattr(resp, "output_text", "") or "") + resp = await client.chat.completions.create(model=model, messages=prompt) + if not getattr(resp, "choices", None): + return "" + return str(resp.choices[0].message.content or "") + + # 不同模型可能拒绝 temperature 或 reasoning_effort。这里不靠模型名猜测, # 只在 400 明确指出对应参数时移除该参数并重试; 每个参数最多移除一次。 _TEMP_REJECT_HINTS = ("temperature", "only 1 is allowed") diff --git a/backend/app/services/ai_responses.py b/backend/app/services/ai_responses.py new file mode 100644 index 00000000..0b28629f --- /dev/null +++ b/backend/app/services/ai_responses.py @@ -0,0 +1,516 @@ +"""OpenAI Responses API 协议适配 — 与 chat.completions 路径平行的第二协议。 + +设计约束(与 ai_provider._stream_openai / _iter_openai_text 对齐): +- kwargs 构建与 400 自适应重试同语义, 仅参数名不同: + max_tokens→max_output_tokens, reasoning_effort→reasoning={"effort"}。 +- 事件流归一化沿用 chat 版的防御: 长度截断 / 仅推理无正文 / 空响应都显式报错。 +- 工具调用走 function_call output item + function_call_output 回填, + 对外仍暴露 chat 形态 (role:tool / tool_calls), 上层 (generate_ai_text_with_tools, + assistant chat_service) 零改动; 转换在本模块进出两端完成。 +- 本模块不做 provider 判定, 由 ai_provider.is_responses_protocol() 分流进入。 + +配置读取全部经 ai_provider 模块限定调用(非 from-import 名字绑定), +保证测试可 monkeypatch ai_provider.current_ai_provider 等。 +""" +from __future__ import annotations + +import asyncio +import inspect +from collections.abc import AsyncIterator, Sequence +from types import SimpleNamespace +from typing import Any + +from app.services import ai_provider +from app.services.ai_provider import Message + +# Responses 的 incomplete_details.reason 与 status 语义对齐 chat 的 finish_reason。 +_LENGTH_INCOMPLETE_REASONS = {"length", "max_tokens", "max_output_tokens"} +# 推理事件前缀: reasoning_summary_text.delta / reasoning_text.delta 等。 +_REASONING_EVENT_PREFIXES = ( + "response.reasoning_summary", + "response.reasoning_text", + "response.reasoning.", +) + + +def responses_client(timeout: float): + """复用核心的 AsyncOpenAI 客户端 (Key/Base URL/UA/重试策略一处维护)。""" + ai_key = ai_provider.secrets_store.get_ai_key() + if not ai_key: + raise RuntimeError("AI API Key 未配置, 请在设置页配置") + return ai_provider._openai_client(ai_key, timeout) + + +# --------------------------------------------------------------------------- +# 消息与工具的双向转换 (chat 形态 ⇄ Responses input/tools) +# --------------------------------------------------------------------------- + + +def chat_input(messages: Sequence[Message]) -> list[dict[str, Any]]: + """chat messages → Responses input items。 + + role:tool → function_call_output; assistant.tool_calls → function_call items; + 其余 (system/user/assistant 纯文本) 走 easy-input message 形态。 + Gemini 兼容层的 extra_content 在 Responses 协议下无意义, 静默丢弃。 + """ + items: list[dict[str, Any]] = [] + for m in messages: + role = str(m.get("role") or "user") + if role == "tool": + items.append({ + "type": "function_call_output", + "call_id": str(m.get("tool_call_id") or ""), + "output": str(m.get("content") or ""), + }) + continue + tool_calls = m.get("tool_calls") + if role == "assistant" and tool_calls: + if m.get("content"): + items.append({"role": "assistant", "content": str(m["content"])}) + for tc in tool_calls: + fn = tc.get("function") or {} + items.append({ + "type": "function_call", + "call_id": str(tc.get("id") or ""), + "name": str(fn.get("name") or ""), + "arguments": str(fn.get("arguments") or ""), + }) + continue + items.append({"role": role, "content": str(m.get("content") or "")}) + return items + + +def responses_tools(tools: Sequence[dict]) -> list[dict[str, Any]]: + """chat 工具 schema → Responses 扁平工具 schema。 + + chat: {"type":"function","function":{name,description,parameters}} + responses: {"type":"function","name","description","parameters"} + 兼容已是扁平形态的输入(部分网关直接吐扁平 schema)。 + """ + out: list[dict[str, Any]] = [] + for t in tools: + if not isinstance(t, dict): + continue + fn = t.get("function") + if not isinstance(fn, dict): + fn = t if t.get("type") == "function" else None + if not fn: + continue + item: dict[str, Any] = { + "type": "function", + "name": str(fn.get("name") or ""), + } + if fn.get("description"): + item["description"] = str(fn["description"]) + if fn.get("parameters") is not None: + item["parameters"] = fn["parameters"] + if item["name"]: + out.append(item) + return out + + +def responses_kwargs( + *, + temperature: float | None, + max_tokens: int | None, + tools: Sequence[dict] | None = None, +) -> dict[str, Any]: + """构建 responses.create kwargs (与 _openai_kwargs 同语义, 参数名映射)。""" + kwargs: dict[str, Any] = {} + if max_tokens is not None: + kwargs["max_output_tokens"] = max_tokens + if temperature is not None: + kwargs["temperature"] = temperature + if ai_provider.current_ai_provider() == ai_provider.OPENAI_PROVIDER: + effort = ai_provider.current_openai_reasoning_effort() + if effort: + kwargs["reasoning"] = {"effort": effort} + if tools: + kwargs["tools"] = responses_tools(tools) + return kwargs + + +def _is_reasoning_rejected(exc: Exception) -> bool: + """True if the upstream 400 specifically rejects the reasoning param.""" + if getattr(exc, "status_code", None) != 400: + return False + text = (ai_provider._openai_error_detail(exc) or str(exc)).lower() + param = ai_provider._openai_error_param(exc) + return param in ("reasoning", "reasoning.effort", "effort") or "reasoning" in text + + +def responses_retry_kwargs(exc: Exception, kwargs: dict[str, Any]) -> dict[str, Any] | None: + """Remove one explicitly rejected optional argument for a bounded retry.""" + retry = dict(kwargs) + if "temperature" in retry and ai_provider._is_temperature_rejected(exc): + retry.pop("temperature") + return retry + if "reasoning" in retry and _is_reasoning_rejected(exc): + retry.pop("reasoning") + return retry + return None + + +# --------------------------------------------------------------------------- +# 终态检查 (completed/failed/incomplete → chat 版等价报错) +# --------------------------------------------------------------------------- + + +def _incomplete_reason(resp) -> str: + details = getattr(resp, "incomplete_details", None) + reason = getattr(details, "reason", None) if details is not None else None + if reason: + return str(reason) + # 网关兼容: incomplete_details 为 dict 的情况 + if isinstance(details, dict): + return str(details.get("reason") or "") + return "" + + +def _response_error_detail(resp) -> str: + err = getattr(resp, "error", None) + if err is None: + return "上游未返回错误详情" + message = getattr(err, "message", None) + if message is None and isinstance(err, dict): + message = err.get("message") + return ai_provider._compact_error_text(str(message or err)) or "上游未返回错误详情" + + +def _check_terminal_response(resp, *, content_seen: bool, reasoning_seen: bool) -> None: + """completed/failed/incomplete 终态的防御性检查, 报错文案与 chat 版一致。""" + status = str(getattr(resp, "status", "") or "") + if status == "failed": + raise RuntimeError(f"AI 服务请求失败: {_response_error_detail(resp)}") + if status != "incomplete": + return + reason = _incomplete_reason(resp) + if reason not in _LENGTH_INCOMPLETE_REASONS: + return + if reasoning_seen and not content_seen: + raise RuntimeError( + "AI 推理达到输出长度上限, 未生成正文; 请提高输出 Token 上限或改用非推理模型" + ) + raise RuntimeError("AI 输出达到长度上限, 内容不完整; 请提高输出 Token 上限后重试") + + +def _raise_no_content(reasoning_seen: bool) -> None: + if reasoning_seen: + raise RuntimeError("AI 仅返回推理内容, 未生成正文; 请检查模型配置或改用非推理模型") + raise RuntimeError("AI 服务未返回正文内容; 请检查模型配置或稍后重试") + + +# --------------------------------------------------------------------------- +# 非流式: 纯文本 / 工具循环单轮 +# --------------------------------------------------------------------------- + + +def _response_text(resp) -> str: + """聚合 Response 的全部 output_text (便利属性缺失时逐 item 提取)。""" + text = getattr(resp, "output_text", None) + if isinstance(text, str) and text: + return text + parts: list[str] = [] + for item in getattr(resp, "output", None) or []: + if str(getattr(item, "type", "") or "") != "message": + continue + for part in getattr(item, "content", None) or []: + if str(getattr(part, "type", "") or "") in ("output_text", "text"): + value = getattr(part, "text", None) + if value: + parts.append(str(value)) + return "".join(parts) + + +def _response_reasoning_seen(resp) -> bool: + for item in getattr(resp, "output", None) or []: + if str(getattr(item, "type", "") or "").startswith("reasoning"): + return True + return False + + +def _response_calls(resp) -> list[Any]: + """output 里的 function_call item → chat 形态 tool_calls (SimpleNamespace)。""" + calls = [] + for item in getattr(resp, "output", None) or []: + if str(getattr(item, "type", "") or "") != "function_call": + continue + calls.append(SimpleNamespace( + id=str(getattr(item, "call_id", None) or getattr(item, "id", "") or ""), + type="function", + function=SimpleNamespace( + name=str(getattr(item, "name", "") or ""), + arguments=str(getattr(item, "arguments", "") or ""), + ), + )) + return calls + + +async def _create_with_retry(client, *, payload: dict[str, Any], stream: bool): + kwargs = dict(payload) + while True: + try: + return await client.responses.create(**kwargs, stream=stream) + except Exception as exc: + retry = responses_retry_kwargs(exc, kwargs) + if retry is not None: + kwargs = retry + continue + if ai_provider._is_openai_transport_error(exc): + raise RuntimeError(ai_provider._format_openai_error(exc)) from exc + raise + + +async def run_text( + messages: Sequence[Message], + *, + temperature: float | None, + max_tokens: int | None, + timeout: float, +) -> str: + """非流式纯文本 (generate_ai_text 的 Responses 分支)。""" + client = responses_client(timeout) + payload = responses_kwargs(temperature=temperature, max_tokens=max_tokens) + payload["model"] = ai_provider.current_ai_model() + payload["input"] = chat_input(messages) + resp = await _create_with_retry(client, payload=payload, stream=False) + text = _response_text(resp).strip() + _check_terminal_response( + resp, content_seen=bool(text), reasoning_seen=_response_reasoning_seen(resp), + ) + return text + + +async def run_message_once( + messages: Sequence[Message], + *, + temperature: float | None, + max_tokens: int | None, + timeout: float, + tools: Sequence[dict] | None = None, +): + """非流式工具单轮 (generate_ai_text_with_tools 的 Responses 分支)。 + + 返回 chat 形态 message 对象 (content + tool_calls), 核心循环零改动。 + """ + client = responses_client(timeout) + payload = responses_kwargs(temperature=temperature, max_tokens=max_tokens, tools=tools) + payload["model"] = ai_provider.current_ai_model() + payload["input"] = chat_input(messages) + resp = await _create_with_retry(client, payload=payload, stream=False) + text = _response_text(resp) + calls = _response_calls(resp) + if not calls: + # 无工具调用时才做终态/空正文防御; 有调用时由上层执行工具进入下一轮。 + _check_terminal_response( + resp, content_seen=bool(text), reasoning_seen=_response_reasoning_seen(resp), + ) + return SimpleNamespace(content=text, tool_calls=calls) + + +# --------------------------------------------------------------------------- +# 流式: 纯文本 (stream_ai_text 分支) +# --------------------------------------------------------------------------- + + +async def stream_text( + messages: Sequence[Message], + *, + temperature: float | None, + max_tokens: int | None, + timeout: float, +) -> AsyncIterator[str]: + client = responses_client(timeout) + payload = responses_kwargs(temperature=temperature, max_tokens=max_tokens) + payload["model"] = ai_provider.current_ai_model() + payload["input"] = chat_input(messages) + stream = await _create_with_retry(client, payload=payload, stream=True) + try: + async for piece in iter_text(stream): + yield piece + except Exception as exc: + if ai_provider._is_openai_transport_error(exc): + raise RuntimeError(ai_provider._format_openai_error(exc)) from exc + raise + + +def _is_reasoning_event(event_type: str) -> bool: + return event_type.startswith(_REASONING_EVENT_PREFIXES) + + +async def iter_text(stream) -> AsyncIterator[str]: + """Responses 事件流 → 文本增量, 防御与 _iter_openai_text 等价。""" + content_seen = False + reasoning_seen = False + completed = None + + async for event in stream: + event_type = str(getattr(event, "type", "") or "") + if event_type == "response.output_text.delta": + delta = getattr(event, "delta", None) + if delta: + content_seen = True + yield str(delta) + elif event_type == "response.output_text.done" and not content_seen: + # 个别网关只发 done 不发 delta: done 携带全量文本, 兜底产出一次。 + text = getattr(event, "text", None) + if text: + content_seen = True + yield str(text) + elif _is_reasoning_event(event_type): + reasoning_seen = True + elif event_type in ("response.completed", "response.incomplete"): + completed = getattr(event, "response", None) + elif event_type in ("response.failed", "error"): + resp = getattr(event, "response", None) + if resp is not None and getattr(resp, "error", None) is not None: + raise RuntimeError(f"AI 服务请求失败: {_response_error_detail(resp)}") + err = getattr(event, "error", None) + if err is not None: + raise RuntimeError(f"AI 服务请求失败: {_response_error_detail(SimpleNamespace(error=err))}") + + if completed is not None: + _check_terminal_response(completed, content_seen=content_seen, reasoning_seen=reasoning_seen) + if not content_seen: + _raise_no_content(reasoning_seen) + + +# --------------------------------------------------------------------------- +# 流式: 工具轮 (assistant stream_openai_round 分支) +# --------------------------------------------------------------------------- + + +def _event_item_id(event) -> str: + """function_call 累积的 key: item_id 优先, 缺失时回退 output_index。""" + item_id = getattr(event, "item_id", None) + if item_id: + return str(item_id) + item = getattr(event, "item", None) + if item is not None: + item_id = getattr(item, "id", None) + if item_id: + return str(item_id) + return f"#{getattr(event, 'output_index', 0)}" + + +def _completed_calls(resp) -> list[dict[str, Any]]: + """completed.response.output 是权威工具调用列表 (顺序与完整性优于分片累积)。""" + calls = [] + for item in getattr(resp, "output", None) or []: + if str(getattr(item, "type", "") or "") != "function_call": + continue + calls.append({ + "id": str(getattr(item, "call_id", None) or getattr(item, "id", "") or ""), + "name": str(getattr(item, "name", "") or ""), + "arguments": str(getattr(item, "arguments", "") or ""), + }) + return calls + + +async def stream_round( + messages: Sequence[dict[str, Any]], + tools: Sequence[dict[str, Any]], + *, + temperature: float | None = 0.3, + timeout: float = 240.0, +) -> AsyncIterator[dict[str, Any]]: + """一轮 Responses 流式补全, 输出与 assistant streaming.stream_openai_round 同构。""" + client = responses_client(timeout) + payload = responses_kwargs(temperature=temperature, max_tokens=None, tools=tools) + payload["model"] = ai_provider.current_ai_model() + payload["input"] = chat_input(messages) + stream = await _create_with_retry(client, payload=payload, stream=True) + + content_seen = False + reasoning_seen = False + calls: dict[str, dict[str, Any]] = {} + ordered: list[dict[str, Any]] = [] + final_calls: list[dict[str, Any]] | None = None + completed = None + + async for event in stream: + event_type = str(getattr(event, "type", "") or "") + if event_type == "response.output_text.delta": + delta = getattr(event, "delta", None) + if delta: + content_seen = True + yield {"type": "text", "delta": str(delta)} + elif event_type == "response.output_text.done" and not content_seen: + text = getattr(event, "text", None) + if text: + content_seen = True + yield {"type": "text", "delta": str(text)} + elif _is_reasoning_event(event_type): + reasoning_seen = True + elif event_type == "response.output_item.added": + item = getattr(event, "item", None) + if item is not None and str(getattr(item, "type", "") or "") == "function_call": + key = _event_item_id(event) + entry = {"id": "", "name": "", "arguments": ""} + calls[key] = entry + ordered.append(entry) + call_id = getattr(item, "call_id", None) + if call_id: + entry["id"] = str(call_id) + if getattr(item, "name", None): + entry["name"] = str(item.name) + elif event_type == "response.function_call_arguments.delta": + key = _event_item_id(event) + entry = calls.get(key) + if entry is None: + entry = {"id": "", "name": "", "arguments": ""} + calls[key] = entry + ordered.append(entry) + delta = getattr(event, "delta", None) + if delta: + entry["arguments"] += str(delta) + elif event_type == "response.output_item.done": + item = getattr(event, "item", None) + if item is not None and str(getattr(item, "type", "") or "") == "function_call": + # done item 是权威值, 整条覆盖累积分片 (网关分片乱序/缺失时兜底)。 + key = _event_item_id(event) + entry = calls.get(key) + if entry is None: + entry = {"id": "", "name": "", "arguments": ""} + calls[key] = entry + ordered.append(entry) + call_id = getattr(item, "call_id", None) + if call_id: + entry["id"] = str(call_id) + if getattr(item, "name", None): + entry["name"] = str(item.name) + if getattr(item, "arguments", None): + entry["arguments"] = str(item.arguments) + elif event_type in ("response.completed", "response.incomplete"): + completed = getattr(event, "response", None) + if completed is not None: + final_calls = _completed_calls(completed) + elif event_type in ("response.failed", "error"): + resp = getattr(event, "response", None) + err = getattr(event, "error", None) + if err is None and resp is not None: + err = getattr(resp, "error", None) + raise RuntimeError( + f"AI 服务请求失败: {_response_error_detail(SimpleNamespace(error=err))}" + ) + + # 让 SDK 流对象尽快释放(部分兼容网关要求显式 close); SDK 2.x 的 + # AsyncStream.close 是协程, 必须区分同步/异步两种签名。 + close = getattr(stream, "close", None) + if close is not None: + if inspect.iscoroutinefunction(close): + await close() + else: + await asyncio.to_thread(close) + + if final_calls is None: + final_calls = ordered + if final_calls: + yield {"type": "round_end", "tool_calls": final_calls, "finish_reason": "tool_calls"} + return + + if completed is not None: + _check_terminal_response(completed, content_seen=content_seen, reasoning_seen=reasoning_seen) + if not content_seen: + _raise_no_content(reasoning_seen) + yield {"type": "round_end", "tool_calls": [], "finish_reason": "stop"} diff --git a/backend/app/services/depth_service.py b/backend/app/services/depth_service.py index a65e9d7e..a6ac606c 100644 --- a/backend/app/services/depth_service.py +++ b/backend/app/services/depth_service.py @@ -101,7 +101,7 @@ class DepthService: def boot_check(self) -> None: """启动补跑: 当天 depth5 文件不存在则 finalize 一次; 已存在则恢复内存缓存。""" if not self._has_capability(): - logger.info("depth sealed: 无 DEPTH5_BATCH 能力, 跳过启动补跑") + logger.info("depth sealed: %s 不可用, 跳过启动补跑", self._capability_label()) return today = cn_today() if self._persisted_for_date(today): @@ -190,7 +190,7 @@ class DepthService: 返回 {"ok": bool, "count": int, "msg": str} """ if not self._has_capability(): - return {"ok": False, "count": 0, "msg": "无五档盘口能力(需 Pro+)"} + return {"ok": False, "count": 0, "msg": f"{self._capability_label()} 不可用"} try: self._fetch_and_seal(persist=True) # 落盘, 刷新页面不丢 with self._lock: @@ -631,9 +631,21 @@ class DepthService: # ================================================================ def _has_capability(self) -> bool: - capset = self._get_capset() - from app.tickflow.capabilities import Cap - return capset.has(Cap.DEPTH5_BATCH) + """五档能力是否可用 — 以【生效源】为准, 与取数路径 (_call_depth_batch) 同口径。 + + 不能只看 TickFlow 档位: 五档路由到声明了 depth5 的插件 (如 tdxquant) 时, + TickFlow 档位无关; 反之路由到 TickFlow 但档位不足时, 插件声明不该算数。 + 本轮修复前此处只查 capset, 导致「五档已切到插件但启动补跑被跳过」。 + """ + from app.data_providers.capabilities import capability_gate + ok, _ = capability_gate("depth5") + return ok + + def _capability_label(self) -> str: + """门禁日志标签: "五档盘口 ← fuyao" / "…(档位不足)", 便于判断该切哪个源。""" + from app.data_providers.capabilities import capability_gate + _, label = capability_gate("depth5") + return label def _get_capset(self): """获取当前 capset(优先 app.state, 回退 detect)。""" diff --git a/backend/app/services/emotion_score.py b/backend/app/services/emotion_score.py new file mode 100644 index 00000000..0cd38d53 --- /dev/null +++ b/backend/app/services/emotion_score.py @@ -0,0 +1,243 @@ +"""情绪评分 — 维度 / 权重 / 区间的单一权威定义 + 可配置计算。 + +口径即配置: 6 个维度各自由若干「指标 → 归一化区间 → 加权」子项合成, 维度再按 +权重加权平均得到总分; 档位阈值把分数映射为 强势/偏暖/震荡/偏冷/冰点。 + +设计约束: + - DEFAULT_CONFIG 复现历史硬编码口径, 逐位一致 (所有子项权重和 = 1, 维度等权); + - 子项权重在维度内**按权重和归一化**, 因此用户改动权重不会因和不为 1 而畸变; + - 用户配置与默认配置做逐字段合并, 新增维度/子项时老配置自动补默认值, 不会 + 因缺字段而 KeyError; + - 指标值缺失 (None) 的维度取中性 50, 与历史实现「无主线数据取 50」一致。 + +消费方: app.services.market_overview_builder (看板总览 + 复盘情绪分)。 +""" +from __future__ import annotations + +import math + +# 档位: (阈值键, 展示名, 颜色语义) — 从高到低匹配 +LABELS: tuple[tuple[str, str], ...] = ( + ("strong", "强势"), + ("warm", "偏暖"), + ("neutral", "震荡"), + ("cold", "偏冷"), +) +FALLBACK_LABEL = "冰点" +NEUTRAL = 50 + + +def _default_config() -> dict: + """出厂口径 — 与改造前的硬编码计算逐位一致。""" + + def dim(label: str, terms: list[dict], **extra) -> dict: + return {"label": label, "weight": 1.0, "terms": terms, **extra} + + def term(key: str, label: str, weight: float, low: float, high: float, unit: str = "") -> dict: + return {"key": key, "label": label, "weight": weight, "low": low, "high": high, "unit": unit} + + return { + "dimensions": { + "index": dim("指数", [ + term("avg_index_pct", "指数平均涨跌", 1.0, -2.5, 2.5, "%"), + ]), + "profit": dim("赚钱", [ + term("up_pct", "上涨占比", 0.45, 20, 80, "%"), + term("avg_pct", "平均涨跌", 0.25, -0.02, 0.02), + term("median_pct", "中位涨跌", 0.20, -0.02, 0.02), + term("strong_diff_pct", "强涨-强跌占比差", 0.10, -8, 8, "%"), + ]), + "money": dim("量能", [ + term("avg_vol_ratio", "平均量比", 0.70, 0.6, 1.8), + term("high_vol_pct", "放量股占比", 0.30, 2, 12, "%"), + ]), + "speculation": dim("投机", [ + term("limit_up", "涨停家数", 0.25, 5, 90, "家"), + term("seal_rate", "封板率", 0.35, 30, 85, "%"), + term("max_boards", "最高连板", 0.25, 1, 8, "板"), + term("tier2_count", "2板及以上", 0.15, 0, 30, "家"), + ]), + # 抗跌为逆向维度: 下跌越多分越低, 故 invert + "resilience": dim("抗跌", [ + term("down_pct", "下跌占比", 0.55, 20, 80, "%"), + term("strong_down_pct", "强跌占比", 0.45, 1, 12, "%"), + ], invert=True), + "mainline": dim("主线", [ + term("mainline_avg", "主线平均涨跌", 0.65, -0.005, 0.03), + term("mainline_cover_pct", "主线覆盖率", 0.35, 1, 12, "%"), + ]), + }, + "thresholds": {"strong": 70, "warm": 55, "neutral": 45, "cold": 30}, + } + + +DEFAULT_CONFIG: dict = _default_config() + +# 维度展示顺序 = 雷达图顶点顺序 = 配置面板行序 (三者共用, 避免各处再排一遍) +DIMENSION_ORDER: tuple[str, ...] = tuple(DEFAULT_CONFIG["dimensions"]) + + +def _finite(value) -> float | None: + if value is None: + return None + try: + v = float(value) + except (TypeError, ValueError): + return None + return v if math.isfinite(v) else None + + +def score_of(value, low: float, high: float) -> int: + """指标 → 0..100 归一化分。与历史 `_score` 完全同语义 (含非有限输入取中性)。""" + if high <= low: + return NEUTRAL + v = _finite(value) + if v is None: + return NEUTRAL + return max(0, min(100, round((v - low) / (high - low) * 100))) + + +def merge_config(raw: dict | None) -> dict: + """用户配置 → 完整配置 (逐字段合并, 缺项补默认)。 + + 合并粒度到 term: 用户改过的权重/区间生效, 未提及的字段沿用出厂值。 + 维度/子项数量与 key 以出厂定义为准 (不支持增删指标 —— 指标值由计算方提供, + 凭空新增的 key 取不到数)。 + """ + merged = _default_config() + if not isinstance(raw, dict): + return merged + + raw_dims = raw.get("dimensions") + if isinstance(raw_dims, dict): + for key, dim in merged["dimensions"].items(): + user_dim = raw_dims.get(key) + if not isinstance(user_dim, dict): + continue + weight = _finite(user_dim.get("weight")) + if weight is not None and weight >= 0: + dim["weight"] = weight + user_terms = user_dim.get("terms") + if not isinstance(user_terms, list): + continue + by_key = {t["key"]: t for t in user_terms if isinstance(t, dict)} + for term in dim["terms"]: + user_term = by_key.get(term["key"]) + if not isinstance(user_term, dict): + continue + for field in ("weight", "low", "high"): + val = _finite(user_term.get(field)) + if val is not None: + term[field] = val + + raw_th = raw.get("thresholds") + if isinstance(raw_th, dict): + for key in merged["thresholds"]: + val = _finite(raw_th.get(key)) + if val is not None: + merged["thresholds"][key] = val + _normalize_thresholds(merged["thresholds"]) + return merged + + +def _normalize_thresholds(th: dict) -> None: + """阈值夹到 0..100 并保证降序 (强≥暖≥震荡≥冷)。 + + 用户可能把 强势 填得比 偏暖 还低, 那样档位区间为空、标签永远错。按数值重排 + 回填, 既保留用户意图的相对关系, 也保证映射单调。 + """ + for key in th: + val = _finite(th.get(key)) + th[key] = NEUTRAL if val is None else max(0.0, min(100.0, val)) + values = sorted((th[k] for k in th), reverse=True) + for (key, _), val in zip(LABELS, values): + th[key] = val + + +def label_of(score: float, thresholds: dict) -> str: + for key, name in LABELS: + if score >= thresholds.get(key, 0): + return name + return FALLBACK_LABEL + + +def dimension_value(dim: dict, metrics: dict) -> int: + """单个维度 → 0..100。子项权重和归一化; 全部子项缺数时取中性。""" + total_weight = 0.0 + acc = 0.0 + for term in dim["terms"]: + if term["key"] not in metrics: + continue # 该指标本次不可用 → 跳过, 不当作 0 + value = score_of(metrics[term["key"]], term["low"], term["high"]) + weight = max(0.0, float(term.get("weight") or 0)) + if weight <= 0: + continue + acc += value * weight + total_weight += weight + if total_weight <= 0: + return NEUTRAL + # 权重和已是 1 时不做除法: 出厂口径的 0.45+0.25+0.20+0.10 在浮点下是 + # 0.9999999999999999, 除一趟会把恰好落在 .5 的值推过舍入边界, 使默认结果 + # 与历史实现差 1 分。改过权重时才归一化 (此时结果本就该变)。 + weighted = acc if abs(total_weight - 1.0) < 1e-9 else acc / total_weight + if dim.get("invert"): + # 先取整再取反 (而非 round(100 - x)): 两者在 x 的小数部分恰为 .5 时 + # 结果可差 1, 出厂口径必须与历史实现逐位一致。 + return 100 - round(weighted) + return round(weighted) + + +def compute(metrics: dict, config: dict | None = None) -> dict: + """指标 + 配置 → {score, label, radar, dimensions}。 + + radar 保持历史形状 [{key,label,value}] (前端雷达图与复盘快照依赖), + dimensions 额外带权重与命中分, 供配置面板展示「各维度当前实际得分」。 + """ + cfg = merge_config(config) + radar: list[dict] = [] + dims: list[dict] = [] + for key in DIMENSION_ORDER: + dim = cfg["dimensions"][key] + value = dimension_value(dim, metrics) + radar.append({"key": key, "label": dim["label"], "value": value}) + dims.append({ + "key": key, + "label": dim["label"], + "value": value, + "weight": dim["weight"], + "invert": bool(dim.get("invert")), + }) + + # 等权时直接取整数均值: 权重在均值里本就约掉, 而 Python 3.12+ 的 sum() 对 + # 浮点走补偿求和、对整数是精确累加 —— 走 `int * 1.0` 会引入浮点误差, 让恰好 + # 落在 .5 的总分被舍到另一侧, 与历史实现差 1 分。改过权重时才走加权分支。 + weights = [d["weight"] for d in dims] + if weights and all(abs(w - weights[0]) < 1e-12 for w in weights): + score = round(sum(d["value"] for d in dims) / len(dims)) if dims else NEUTRAL + else: + total_weight = sum(weights) + if total_weight > 0: + score = round(sum(d["value"] * d["weight"] for d in dims) / total_weight) + elif dims: + score = round(sum(d["value"] for d in dims) / len(dims)) + else: + score = NEUTRAL + + return { + "score": score, + "label": label_of(score, cfg["thresholds"]), + "radar": radar, + "dimensions": dims, + "thresholds": cfg["thresholds"], + } + + +def public_config(config: dict | None = None) -> dict: + """完整配置 + 指标元数据 (供前端渲染配置面板)。""" + cfg = merge_config(config) + return { + "config": cfg, + "defaults": DEFAULT_CONFIG, + "dimension_order": list(DIMENSION_ORDER), + "labels": [{"key": k, "name": n} for k, n in LABELS] + [{"key": "fallback", "name": FALLBACK_LABEL}], + } diff --git a/backend/app/services/ext_data.py b/backend/app/services/ext_data.py index 86a41a1d..9c62e42b 100644 --- a/backend/app/services/ext_data.py +++ b/backend/app/services/ext_data.py @@ -53,6 +53,7 @@ class PullConfig: "next_run", "time_window_start", "time_window_end", "date_param", "date_format", "time_field", "auth", "timeout_seconds", "page_param", "page_size_param", "page_size", "page_start", "max_pages", + "fanout", ) def __init__( @@ -82,7 +83,9 @@ class PullConfig: page_size: int = 0, page_start: int = 1, max_pages: int = 20, + fanout: dict | None = None, ) -> None: + self.fanout = fanout or None # 两级拉取配方 (见 fanout.py), None=单请求 self.url = url self.method = method # GET | POST self.headers = headers or {} @@ -163,6 +166,7 @@ class PullConfig: "page_size": self.page_size, "page_start": self.page_start, "max_pages": self.max_pages, + "fanout": self.fanout, } @classmethod @@ -195,6 +199,7 @@ class PullConfig: page_size=d.get("page_size", 0), page_start=d.get("page_start", 1), max_pages=d.get("max_pages", 20), + fanout=d.get("fanout"), ) diff --git a/backend/app/services/ext_fanout.py b/backend/app/services/ext_fanout.py new file mode 100644 index 00000000..d7fda3bb --- /dev/null +++ b/backend/app/services/ext_fanout.py @@ -0,0 +1,155 @@ +"""两级拉取 (fanout) — 「目录 → 逐条目明细」接口的声明式配方。 + +适用面: 数据源只提供「类别清单」+「单类别成员」两个端点, 没有一次返回 +「成员 → 所属类别」全量映射的接口 (同花顺概念/行业、多数板块类 API)。 + +通用拉取是单请求 + 字段名映射, 表达不了这种形态; 与其按 config_id 硬编码 +特例, 不如把「第二级怎么发、结果怎么拼」写进 PullConfig.fanout —— 配置本身 +即配方, 在「数据」页展开扩展表 → 拉取 就能看到全貌, 也便于照抄接入自有源 +(见 docs/custom-data-source.md)。 + +fanout 配方字段: + url 第二级请求地址 (与 pull.url 同走标识头 + auth 鉴权) + param 第二级请求的查询参数名, 取值为第一级条目的 id_field + id_field 第一级条目中作为第二级参数的字段 + label_field 第一级条目中作展示名的字段 (可选, 缺省用 id_field; 仅用于告警) + items_path 第二级响应里成员数组的 dot-path (留空 = 响应本身即数组) + member_id_field 成员对象里取标的代码的字段 + member_label_field 成员对象里取标的名称的字段 (可选) + symbol_col 输出列名: 标的代码 + name_col 输出列名: 标的名称 + value_col 输出列名: 拼接后的类别 + separator 多类别拼接符 (概念用 ";", 行业层级用 "-") + concurrency 并发上限 (可选, 默认 8) + retries 单条目失败重试次数 (可选, 默认 1; 用尽则跳过该条目) + +产出: 行方向由「类别 → 成员」反转为「成员 → 类别清单」, 与单请求路径的 +行结构一致, 下游 rows_to_parquet / 分析页零改动。 +""" +from __future__ import annotations + +import asyncio +import logging +import math +from typing import Any + +from app.services.ext_data import ExtConfig + +logger = logging.getLogger(__name__) + +_DEFAULT_CONCURRENCY = 8 +_DEFAULT_RETRIES = 1 +_REQUIRED_KEYS = ( + "url", "param", "id_field", "items_path", "member_id_field", + "symbol_col", "name_col", "value_col", +) + + +def clean_label(value: Any) -> str: + """类别名归一: 剔除 None / NaN / 字符串 "nan" 等占位值。""" + if value is None or (isinstance(value, float) and not math.isfinite(value)): + return "" + text = str(value).strip() + return "" if text.casefold() in {"nan", "none", "null"} else text + + +def _raise_on_envelope_error(data: Any) -> None: + """常见信封 {code, message, data}: code 非 0 时抛出带服务端文案的错误。""" + if isinstance(data, dict): + code = data.get("code") + if code not in (0, None, "0"): + raise ValueError(f"接口返回错误 code={code}: {data.get('message', '')}") + + +async def fetch_fanout_rows(config: ExtConfig, target_date=None) -> list[dict]: + """执行两级拉取并反转为「成员 → 类别清单」行。""" + from app.services.ext_pull import _append_query_params, _extract_rows, _request_json + + pull = config.pull + if pull is None: + raise ValueError("拉取未配置") + fanout = pull.fanout or {} + missing = [k for k in _REQUIRED_KEYS if not fanout.get(k)] + if missing: + raise ValueError(f"fanout 配方缺少必填项: {', '.join(missing)}") + + # ---- 第一级: 类别清单 ---- + catalog = await _request_json(pull, config.id, target_date) + _raise_on_envelope_error(catalog) + items = _extract_rows(catalog, pull.response_path) + + id_field = fanout["id_field"] + label_field = fanout.get("label_field") or id_field + entries = [ + it for it in items + if isinstance(it, dict) and str(it.get(id_field) or "").strip() + ] + if not entries: + raise ValueError(f"第一级返回 0 个可用条目 (id_field={id_field})") + + # ---- 第二级: 逐条目成员 (并发 + 单条目失败容忍) ---- + concurrency = int(fanout.get("concurrency") or _DEFAULT_CONCURRENCY) + raw_retries = fanout.get("retries") + retries = int(_DEFAULT_RETRIES if raw_retries is None else raw_retries) + sem = asyncio.Semaphore(max(1, concurrency)) + + async def _members(entry: dict) -> list[dict] | None: + member_url = _append_query_params( + fanout["url"], {fanout["param"]: str(entry[id_field]).strip()} + ) + async with sem: + for attempt in range(retries + 1): + try: + data = await _request_json( + pull, config.id, None, url_override=member_url + ) + _raise_on_envelope_error(data) + return _extract_rows(data, fanout.get("items_path") or "") + except Exception: # noqa: BLE001 — 单条目失败: 重试后跳过 + if attempt == retries: + return None + await asyncio.sleep(0.5) + return None + + members_list = await asyncio.gather(*(_members(e) for e in entries)) + + # ---- 反转聚合: 成员 → 类别清单 (顺序跟随目录顺序, 保证拼接结果稳定) ---- + m_id = fanout["member_id_field"] + m_name = fanout.get("member_label_field") + agg: dict[str, dict] = {} + failed: list[str] = [] + for entry, members in zip(entries, members_list, strict=True): + if members is None: + failed.append(str(entry.get(label_field) or entry.get(id_field) or "")) + continue + label = clean_label(entry.get(label_field)) + for m in members: + if not isinstance(m, dict): + continue + sym = str(m.get(m_id) or "").strip() + if not sym: + continue + row = agg.get(sym) + if row is None: + row = { + fanout["symbol_col"]: sym, + fanout["name_col"]: (m.get(m_name) or "") if m_name else "", + fanout["value_col"]: [], + } + agg[sym] = row + if label and label not in row[fanout["value_col"]]: + row[fanout["value_col"]].append(label) + + if failed: + logger.warning( + "fanout 第二级失败 %d/%d (已跳过): %s", + len(failed), len(entries), "、".join(failed[:5]), + ) + if not agg: + raise ValueError("fanout 全部条目第二级拉取失败") + + separator = fanout.get("separator") or ";" + return [ + {**row, fanout["value_col"]: separator.join(row[fanout["value_col"]])} + for row in agg.values() + ] diff --git a/backend/app/services/ext_pages.py b/backend/app/services/ext_pages.py new file mode 100644 index 00000000..8c1ecb94 --- /dev/null +++ b/backend/app/services/ext_pages.py @@ -0,0 +1,133 @@ +"""运行时页面包存储 — data/ext_pages/{id}/ 目录式持久化。 + +页面包是前端扩展的打包产物(单文件 ES 模块): 上传后由前端在启动引导窗口 +动态 import 挂载(见 frontend/src/extensions/runtime.ts)。后端只负责存储、 +元数据与分发, 不解析 JS 内容 —— 清单契约(能否 import、apiVersion、路由 +冲突)由前端注册表校验, 校验结果经 manifest 接口回填到 page.json。 +""" +from __future__ import annotations + +import hashlib +import json +import re +import time +from dataclasses import asdict, dataclass +from pathlib import Path + +# 与 ExtConfigStore 同口径的 ID 白名单: 防 ../ 路径穿越写删 +_VALID_ID = re.compile(r"^[a-zA-Z0-9_]{1,64}$") + + +@dataclass +class PagePackMeta: + id: str + enabled: bool = False + # 前端试载成功后回填的清单摘要(展示与排障用, 真实注册以 bundle 为准) + label: str = "" + path: str = "" + size: int = 0 + sha256: str = "" + uploaded_at: float = 0.0 + # 前端试载/注册失败的错误信息; 空串表示未出错 + error: str = "" + + +def _sha256_file(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +class PagePackStore: + """每个页面包一个目录: page.json(元数据) + bundle.js(产物)。 + + 元数据写入走 临时文件+rename, 与 ext_data 的原子写同语义, 避免半写 + 状态被启动扫描读到。 + """ + + def __init__(self, data_dir: Path): + self.data_dir = Path(data_dir) / "ext_pages" + self.data_dir.mkdir(parents=True, exist_ok=True) + + def _dir(self, pack_id: str) -> Path: + if not _VALID_ID.match(pack_id or ""): + raise ValueError("页面包 ID 仅允许字母/数字/下划线, 长度 1-64") + return self.data_dir / pack_id + + def _write_meta(self, meta: PagePackMeta) -> None: + d = self._dir(meta.id) + d.mkdir(parents=True, exist_ok=True) + tmp = d / "page.json.tmp" + tmp.write_text(json.dumps(asdict(meta), ensure_ascii=False, indent=2), encoding="utf-8") + tmp.replace(d / "page.json") + + def save_upload(self, pack_id: str, bundle_src: Path) -> PagePackMeta: + """把已写好的临时 bundle 文件收编入库(同 ID 重新上传 = 覆盖更新)。""" + d = self._dir(pack_id) + d.mkdir(parents=True, exist_ok=True) + dest = d / "bundle.js" + # 临时目录与数据目录可能跨盘(Windows), 用 copy 而非 rename + dest.write_bytes(bundle_src.read_bytes()) + meta = PagePackMeta( + id=pack_id, + size=dest.stat().st_size, + sha256=_sha256_file(dest), + uploaded_at=time.time(), + ) + self._write_meta(meta) + return meta + + def get(self, pack_id: str) -> PagePackMeta | None: + p = self._dir(pack_id) / "page.json" + if not p.exists(): + return None + try: + return PagePackMeta(**json.loads(p.read_text(encoding="utf-8"))) + except (json.JSONDecodeError, TypeError): + # 半写/损坏的元数据按损坏页处理, 不阻断其他页面包 + return PagePackMeta(id=pack_id, error="page.json 损坏, 请重新上传") + + def bundle_path(self, pack_id: str) -> Path | None: + p = self._dir(pack_id) / "bundle.js" + return p if p.exists() else None + + def list(self) -> list[PagePackMeta]: + metas: list[PagePackMeta] = [] + for d in sorted(self.data_dir.iterdir()): + if not d.is_dir(): + continue + meta = self.get(d.name) + if meta is not None: + metas.append(meta) + return metas + + def update_manifest(self, pack_id: str, *, label: str, path: str, error: str) -> PagePackMeta: + """前端试载结果回填: error 非空 → 停用并记错误; 否则启用。""" + meta = self.get(pack_id) + if meta is None: + raise KeyError(pack_id) + meta.label = label + meta.path = path + meta.error = error + meta.enabled = not error + self._write_meta(meta) + return meta + + def set_enabled(self, pack_id: str, enabled: bool) -> PagePackMeta: + meta = self.get(pack_id) + if meta is None: + raise KeyError(pack_id) + meta.enabled = enabled + self._write_meta(meta) + return meta + + def delete(self, pack_id: str) -> bool: + d = self._dir(pack_id) + if not d.exists(): + return False + for f in d.iterdir(): + f.unlink(missing_ok=True) + d.rmdir() + return True diff --git a/backend/app/services/ext_presets.py b/backend/app/services/ext_presets.py index 527753bf..c649b825 100644 --- a/backend/app/services/ext_presets.py +++ b/backend/app/services/ext_presets.py @@ -1,22 +1,29 @@ -"""内置扩展数据预设 — 概念/行业启动时只创建配置, 等待用户手动获取 (#199)。 +"""内置扩展数据预设 — 概念/行业。 设计原则: - - 扩展数据通用逻辑零改动 (ExtConfig / fetch_and_ingest / API / 前端均不动) - - 仅在本模块做「接口结构 → 本地 schema」的转换 - - 「已存在则跳过」: 绝不覆盖用户已有数据, 老用户零影响 - - 拉取失败只记 warning, 不阻断启动 (保持「没数据也能跑」) + - 出厂即写【完整可用的拉取配置】(URL / 鉴权 / 两级请求配方), 不是空壳。 + 这样「数据」页展开表 → 拉取 就能看到一份真实可跑的配方, 照抄即可接入 + 自有源; 也免去"点获取按钮才拉"的隐藏路径。 + - 扩展数据通用逻辑零改动 (ExtConfig / fetch_and_ingest / API / 前端均不动): + 概念/行业所需的「目录 → 逐条目」形态由 PullConfig.fanout 声明式表达, + 通用引擎 (ext_fanout) 执行, 无按 id 的特例分支。 + - 「已存在则跳过」: 绝不覆盖用户已有配置与数据, 老用户零影响。 + - 出厂值变更时定向迁移: 仅当现有配置的 pull 精确命中历代出厂值时整体替换, + 用户自定义过的一律不碰 (见 _RETIRED_FACTORY_PULLS)。 + - 拉取失败只记 warning, 不阻断启动 (保持「没数据也能跑」)。 -种子数据来源 (概念/行业各自独立配置): - - 概念: https://shy313.com/api/plugins/market_flow/exports/ths-concepts - - 行业: https://shy313.com/api/plugins/market_flow/exports/ths-industries -作者更新数据只需改接口上的 JSON, 用户下次拉取自动同步, 无需发版。 +数据来源 (概念/行业共用 fuyao 同花顺指数接口): + - 板块清单: /api/a-share-index/catalog/ths-index-list (tag=cn_concept / industry) + - 板块成分: /api/a-share-index/constituents/ths-stock-list (单板块一次) + 接口方向为「板块→股票」, fanout 反转为「股票→板块名清单」。 + 鉴权复用 fuyao 行情插件同一凭据 (auth.key_ref = fuyao_api_key), 用户只需 + 在一处填 Key。全量约 390+320 次成分股调用, 8 并发可承受。 接入点: app.main.lifespan → ensure_builtin_presets(store.data_dir) """ from __future__ import annotations import logging -import math from pathlib import Path from app.services.ext_data import ( @@ -24,25 +31,39 @@ from app.services.ext_data import ( ExtConfigStore, ExtField, PullConfig, - rows_to_parquet, ) logger = logging.getLogger(__name__) -# 种子数据源 (概念/行业各自独立配置, 作者维护) -_CONCEPT_DATA_URL = "https://shy313.com/api/plugins/market_flow/exports/ths-concepts" -_INDUSTRY_DATA_URL = "https://shy313.com/api/plugins/market_flow/exports/ths-industries" +# fuyao 同花顺指数接口 (鉴权 X-api-key, 信封 {code, message, data:{item:[...]}}) +_FUYAO_BASE = "https://fuyao.aicubes.cn" +_FUYAO_CATALOG_PATH = "/api/a-share-index/catalog/ths-index-list" +_FUYAO_CONSTITUENTS_PATH = "/api/a-share-index/constituents/ths-stock-list" +# 成分股逐板块拉取: 全量 710 次调用, 8 并发 + 单板块失败重试 1 次 (实测无 429) +_FUYAO_CONCURRENCY = 8 +_FUYAO_TIMEOUT_S = 20.0 +_FUYAO_BOARD_RETRIES = 1 +# 复用 fuyao 行情插件的凭据 (secrets.json: fuyao_api_key / 环境变量 FUYAO_API_KEY) +_FUYAO_KEY_REF = "fuyao_api_key" + +_PRESET_IDS = ("ext_gn_ths", "ext_hy_ths") # --------------------------------------------------------------------------- -# 预设定义: 字段结构 + 拉取配方 +# 预设定义: 字段结构 + 完整拉取配方 # --------------------------------------------------------------------------- +def _fuyao_auth() -> dict: + """fuyao 鉴权: X-api-key 头, 凭据复用行情插件的 fuyao_api_key。""" + return {"type": "header", "header": "X-api-key", "key_ref": _FUYAO_KEY_REF} + + def _concept_preset() -> ExtConfig: """扩展概念 (ext_gn_ths)。 - 接口结构: [{symbol, name, concepts: [概念1, 概念2, ...]}] - 本地 schema: 股票代码 / 股票简称 / 所属概念(分号拼接) / symbol / code + 两级拉取: 板块目录 (tag=cn_concept) → 逐板块成分股, 反转为 + 「股票 → 所属概念」, 多个概念以分号拼接。 + 输出 schema: 股票代码 / 股票简称 / 所属概念 / symbol / code """ return ExtConfig( id="ext_gn_ths", @@ -55,16 +76,32 @@ def _concept_preset() -> ExtConfig: ExtField("股票简称", "string", "股票简称"), ExtField("所属概念", "string", "所属概念"), ], - description="同花顺概念分类 (启动仅创建配置, 在概念/行业页手动获取)", + description="同花顺概念分类 (fuyao 同花顺指数接口; 拉取配方见「拉取」页, 可照抄接自有源)", symbol_map={"type": "mapped", "col": "股票代码"}, code_map={"type": "computed", "from": "symbol", "method": "strip_exchange"}, pull=PullConfig( - url=_CONCEPT_DATA_URL, + url=f"{_FUYAO_BASE}{_FUYAO_CATALOG_PATH}?tag=cn_concept", method="GET", + response_path="data.item", + auth=_fuyao_auth(), + timeout_seconds=int(_FUYAO_TIMEOUT_S), schedule_minutes=1440, - # enabled=False: ensure_builtin_presets 承诺启动不拉取, PullScheduler - # 只调度 enabled 配置; 手动获取走 fetch_preset 独立路径不受影响 (#199) - enabled=False, + enabled=True, + fanout={ + "url": f"{_FUYAO_BASE}{_FUYAO_CONSTITUENTS_PATH}", + "param": "thscode", + "id_field": "thscode", + "label_field": "name", + "items_path": "data.item", + "member_id_field": "thscode", + "member_label_field": "name", + "symbol_col": "股票代码", + "name_col": "股票简称", + "value_col": "所属概念", + "separator": ";", + "concurrency": _FUYAO_CONCURRENCY, + "retries": _FUYAO_BOARD_RETRIES, + }, ), ) @@ -72,8 +109,8 @@ def _concept_preset() -> ExtConfig: def _industry_preset() -> ExtConfig: """扩展行业 (ext_hy_ths)。 - 接口结构: [{symbol, name, industries: [一级行业, 二级行业, 三级行业]}] - 本地 schema: 股票代码 / 股票简称 / 所属同花顺行业(横杠拼接) / symbol / code + fuyao 为扁平板块清单, 无一/二/三级层级路径, 单级粒度。 + 输出 schema: 股票代码 / 股票简称 / 所属同花顺行业 / symbol / code """ return ExtConfig( id="ext_hy_ths", @@ -86,15 +123,32 @@ def _industry_preset() -> ExtConfig: ExtField("股票简称", "string", "股票简称"), ExtField("所属同花顺行业", "string", "所属同花顺行业"), ], - description="同花顺行业分类 (启动仅创建配置, 在概念/行业页手动获取)", + description="同花顺行业分类 (fuyao 同花顺指数接口, 扁平板块无层级; 拉取配方见「拉取」页)", symbol_map={"type": "mapped", "col": "股票代码"}, code_map={"type": "computed", "from": "symbol", "method": "strip_exchange"}, pull=PullConfig( - url=_INDUSTRY_DATA_URL, + url=f"{_FUYAO_BASE}{_FUYAO_CATALOG_PATH}?tag=industry", method="GET", + response_path="data.item", + auth=_fuyao_auth(), + timeout_seconds=int(_FUYAO_TIMEOUT_S), schedule_minutes=1440, - # 同概念 preset: 出厂禁用, 避免启动即网络拉取 (#199) - enabled=False, + enabled=True, + fanout={ + "url": f"{_FUYAO_BASE}{_FUYAO_CONSTITUENTS_PATH}", + "param": "thscode", + "id_field": "thscode", + "label_field": "name", + "items_path": "data.item", + "member_id_field": "thscode", + "member_label_field": "name", + "symbol_col": "股票代码", + "name_col": "股票简称", + "value_col": "所属同花顺行业", + "separator": "-", + "concurrency": _FUYAO_CONCURRENCY, + "retries": _FUYAO_BOARD_RETRIES, + }, ), ) @@ -103,129 +157,6 @@ def _presets() -> list[ExtConfig]: return [_concept_preset(), _industry_preset()] -# --------------------------------------------------------------------------- -# 接口结构 → 本地 schema 转换 (仅预设使用) -# --------------------------------------------------------------------------- - -def _symbol_to_code(symbol: str) -> str: - """symbol (000001.SZ) → code (000001)。""" - return symbol.split(".", 1)[0] if "." in symbol else symbol - - -def _dimension_label(value: object) -> str: - if value is None or (isinstance(value, float) and not math.isfinite(value)): - return "" - text = str(value).strip() - return "" if text.casefold() in {"nan", "none", "null"} else text - - -def _flatten_concept_rows(raw_rows: list[dict]) -> list[dict]: - """概念: concepts 数组 → 分号拼接成「所属概念」字符串。 - - [{symbol, name, concepts:[...]}] → [{股票代码, 股票简称, 所属概念, symbol, code}] - 注: code 由 symbol 派生 (000001.SZ → 000001), 因 rows_to_parquet 不执行 code_map。 - """ - out: list[dict] = [] - for r in raw_rows: - sym = (r.get("symbol") or "").strip() - if not sym: - continue - concepts = r.get("concepts") or [] - labels = [label for c in concepts if (label := _dimension_label(c))] - out.append({ - "股票代码": sym, - "股票简称": r.get("name") or "", - "所属概念": ";".join(labels), - "symbol": sym, - "code": _symbol_to_code(sym), - }) - return out - - -def _flatten_industry_rows(raw_rows: list[dict]) -> list[dict]: - """行业: industries 数组 → 横杠拼接成「所属同花顺行业」字符串。 - - [{symbol, name, industries:[...]}] → [{股票代码, 股票简称, 所属同花顺行业, symbol, code}] - """ - out: list[dict] = [] - for r in raw_rows: - sym = (r.get("symbol") or "").strip() - if not sym: - continue - inds = r.get("industries") or [] - labels = [label for i in inds if (label := _dimension_label(i))] - out.append({ - "股票代码": sym, - "股票简称": r.get("name") or "", - "所属同花顺行业": "-".join(labels), - "symbol": sym, - "code": _symbol_to_code(sym), - }) - return out - - -# --------------------------------------------------------------------------- -# 拉取执行 (复用 httpx, 不依赖 fetch_and_ingest 的 PullConfig 路径) -# --------------------------------------------------------------------------- - -# 部分网络环境 (CDN/WAF/网关) 会把数组包成 {data: [...]}/{list: [...]}/{rows: [...]} 信封。 -# 这里做一次兼容解包, 避免误判为「接口返回不是数组」。 -_ENVELOPE_KEYS = ("data", "list", "rows", "result", "results") - - -async def _fetch_json(url: str) -> list[dict]: - """请求 JSON 接口, 返回行数组。超时 30s, 失败抛异常由调用方兜底。 - - 兼容两种上游返回形态: - - 直接是数组: [{...}, ...] → 原样返回 - - 信封包裹: {data: [{...}]} 等 → 自动解包 - """ - import httpx - - # 延迟导入避免与 ext_pull 循环依赖; 出站请求带 tsp 标识头 - from app.services.ext_pull import outbound_headers - - async with httpx.AsyncClient(timeout=30) as client: - resp = await client.get(url, headers=outbound_headers()) - resp.raise_for_status() - data = resp.json() - - if isinstance(data, list): - return data - - # 信封解包: 在常见键里找第一个值为数组的 - if isinstance(data, dict): - for key in _ENVELOPE_KEYS: - inner = data.get(key) - if isinstance(inner, list): - return inner - # 兜底: 遍历所有值, 取第一个数组 - for v in data.values(): - if isinstance(v, list): - return v - - raise ValueError( - f"接口返回不是数组 (type={type(data).__name__}), " - f"响应预览: {str(data)[:200]}" - ) - - -async def _seed_one(config: ExtConfig, flatten, data_dir: Path) -> int: - """拉取 + 转换 + 写入单个预设。返回写入行数。""" - from datetime import date - - raw = await _fetch_json(config.pull.url) - rows = flatten(raw) - if not rows: - raise ValueError(f"接口返回 0 行: {config.pull.url}") - n = rows_to_parquet(rows, config, data_dir, snapshot_date=date.today()) - return n - - -# --------------------------------------------------------------------------- -# 对外入口 -# --------------------------------------------------------------------------- - def get_preset(config_id: str) -> ExtConfig | None: """按 id 取预设定义 (供 API 层校验 id 合法性)。""" for c in _presets(): @@ -234,48 +165,77 @@ def get_preset(config_id: str) -> ExtConfig | None: return None -async def ensure_builtin_presets(data_dir: Path) -> None: - """启动时: 为缺失的预设创建 config.json (含 pull 配置), 但【不拉取数据】。 +# 历代出厂 pull 配方指纹 — 仅比对 url + fanout 是否存在, 命中即视为未改过的出厂值。 +# 换源/改配方后, 老用户的出厂配置需要整体替换为新配方; 用户自己填过 URL 的不动。 +_RETIRED_FACTORY_PULLS: dict[str, tuple[dict, ...]] = { + "ext_gn_ths": ( + # 换源前的自维护种子源 + {"url": "https://shy313.com/api/plugins/market_flow/exports/ths-concepts"}, + # 过渡期的 fuyao 目录标记 (无 fanout, 拉不到成分股) + {"url": f"{_FUYAO_BASE}{_FUYAO_CATALOG_PATH}?tag=cn_concept"}, + ), + "ext_hy_ths": ( + {"url": "https://shy313.com/api/plugins/market_flow/exports/ths-industries"}, + {"url": f"{_FUYAO_BASE}{_FUYAO_CATALOG_PATH}?tag=industry"}, + ), +} +# 出厂 description 前缀 (历代文案) — 命中则同步为当前文案, 用户自定义描述不动 +_FACTORY_DESC_PREFIXES = ( + "同花顺概念分类", + "同花顺行业分类", +) - 设计: 数据获取改为用户在概念/行业页手动点「获取数据」触发, 避免启动时 - 网络请求阻塞, 也避免「自动拉取」与「用户自主控制」的预期冲突。 + +def _is_retired_factory_pull(config_id: str, pull: PullConfig | None) -> bool: + """现有 pull 是否为历代出厂值 (不含当前配方), 需要补齐/替换为当前配方。 + + True 的三种情形: + - pull 为 None / 空 URL: 出厂空壳 (#199 之后、本次改动之前的形态) + - URL 命中历代出厂源, 且未带 fanout (换源前的种子源 / 只有目录的过渡配方) + 用户自定义过 URL 或已带 fanout 的配置一律 False (绝不覆盖)。 + """ + if pull is None: + return True + url = (pull.url or "").strip() + if not url: + return True # 空 URL 的空壳配置: 由出厂空壳演进而来, 需补齐 + if pull.fanout: + return False # 已带两级配方 (当前形态或用户自建), 不碰 + return any(url == retired["url"] for retired in _RETIRED_FACTORY_PULLS.get(config_id, ())) + + +async def ensure_builtin_presets(data_dir: Path) -> None: + """启动时: 为缺失的预设创建配置, 并把历代出厂空壳/旧配方补齐为当前配方。 安全保证: - - 已存在则完全跳过 (绝不覆盖用户数据) - - 只写 config.json, 失败只记 warning 不阻断启动 + - 已存在且是用户自定义配置 → 跳过, 绝不覆盖用户数据 + - 仅当 pull 为空 / 命中历代出厂配方指纹时, 用当前完整配方替换 + - description 仅当命中出厂前缀时同步, 用户自定义描述不动 + - 只写 config.json (不拉数据), 失败只记 warning 不阻断启动 + + 数据获取: 由拉取调度器按 schedule_minutes 执行, 或用户在数据页点「拉取 → 执行」。 """ store = ExtConfigStore(data_dir) - for config in _presets(): - existing = store.get(config.id) - if existing is not None: - # 用户已有此表 (老用户 / 自己重建过) → 一律不动 - continue try: - store.upsert(config) - logger.info("内置扩展表 %s 配置已就绪 (待用户手动获取数据)", config.id) - except Exception as e: + existing = store.get(config.id) + if existing is None: + store.upsert(config) + logger.info("内置扩展表 %s 配置已就绪 (含出厂拉取配方)", config.id) + continue + + changed = False + if _is_retired_factory_pull(config.id, existing.pull): + existing.pull = config.pull + changed = True + if not (existing.description or "").strip() or \ + (existing.description or "").startswith(_FACTORY_DESC_PREFIXES): + if existing.description != config.description: + existing.description = config.description + changed = True + + if changed: + store.upsert(existing) + logger.info("内置扩展表 %s 出厂配方已更新为当前版本", config.id) + except Exception as e: # noqa: BLE001 — 启动契约: 预设写入失败不阻断 logger.warning("内置扩展表 %s 配置写入失败 (不影响启动): %s", config.id, e) - - -async def fetch_preset(config_id: str, data_dir: Path) -> int: - """手动触发某个预设的数据拉取 (供 API 调用)。 - - Raises: - ValueError: config_id 不是内置预设 - Exception: 网络请求/解析/写入失败 (由 API 层转 HTTP 错误) - """ - config = get_preset(config_id) - if config is None: - raise ValueError(f"未知的内置预设: {config_id}") - - flatten = _flatten_concept_rows if config_id == "ext_gn_ths" else _flatten_industry_rows - - # 确保 config.json 存在 (用户可能从未启动过 ensure_builtin_presets) - store = ExtConfigStore(data_dir) - if store.get(config_id) is None: - store.upsert(config) - - n = await _seed_one(config, flatten, data_dir) - logger.info("内置扩展表 %s 手动拉取成功: %d 行", config_id, n) - return n diff --git a/backend/app/services/ext_pull.py b/backend/app/services/ext_pull.py index 6c08ddb4..a3f78373 100644 --- a/backend/app/services/ext_pull.py +++ b/backend/app/services/ext_pull.py @@ -117,19 +117,6 @@ def _apply_field_map(rows: list[dict], field_map: dict[str, str]) -> list[dict]: # 拉取执行 # --------------------------------------------------------------------------- -def _apply_preset_flatten(config_id: str, rows: list[dict]) -> list[dict]: - """对内置预设 (概念/行业) 应用结构转换, 与 fetch_preset 保持一致。 - - 延迟导入避免与 ext_presets 形成循环依赖。 - 非预设 id 原样返回。 - """ - if config_id not in ("ext_gn_ths", "ext_hy_ths"): - return rows - from app.services.ext_presets import _flatten_concept_rows, _flatten_industry_rows - flatten = _flatten_concept_rows if config_id == "ext_gn_ths" else _flatten_industry_rows - return flatten(rows) - - def _format_date_value(day: date, date_format: str) -> str: """按配置格式把交易日序列化为接口参数值。 @@ -160,17 +147,31 @@ def _apply_auth(config_id: str, auth: dict | None, url: str, headers: dict[str, header → {header: }, query → ?{param}=。Key 只存 secrets.json, 不落 config.json; 配置了鉴权但未设置 Key 时 fail-closed 直接报错, 避免不带凭据请求被服务端记成无效调用。返回 (可能追加了参数的) url。 + + auth.key_ref: 复用已有凭据而非本表专属 Key。值为 secrets.json 的字段名 + (如 "fuyao_api_key"), 环境变量兜底取该名的大写形式。同一数据源被行情插件 + 与扩展表共用时, 用户只需在一处填 Key。 """ from urllib.parse import quote + from app import secrets_store + from app.services.ext_data import get_ext_api_key auth_type = str((auth or {}).get("type") or "none").lower() if auth_type == "none": return url - key = get_ext_api_key(config_id) - if not key: - raise ValueError(f"已配置 {auth_type} 鉴权但未设置 API Key, 请在拉取设置中填写") + key_ref = str((auth or {}).get("key_ref") or "").strip() + if key_ref: + key = secrets_store.get_env_backed_secret(key_ref, key_ref.upper()) + if not key: + raise ValueError( + f"已配置鉴权但未找到凭据 {key_ref} — 请在 设置 → 数据源 配置对应 API Key" + ) + else: + key = get_ext_api_key(config_id) + if not key: + raise ValueError(f"已配置 {auth_type} 鉴权但未设置 API Key, 请在拉取设置中填写") if auth_type == "bearer": headers[str(auth.get("header") or "Authorization")] = f"Bearer {key}" elif auth_type == "header": @@ -208,14 +209,21 @@ def _assert_rows_date(rows: list[dict], day: date) -> None: async def _request_json( pull: PullConfig, config_id: str, day: date | None = None, extra_params: dict[str, str] | None = None, + *, + url_override: str | None = None, ) -> Any: """发起一次拉取请求并返回解析后的 JSON。 正式拉取 (带日期参数) 与设置页"测试" (不带) 共用同一实现, 保证 UA 标识头与 API Key 鉴权注入只有一套口径。 extra_params: 追加的 query 参数 (分页页码/每页条数), 值经 URL 编码。 + url_override: 直接指定请求地址 (两级拉取的第二级请求用), 不再套用 + date_param 与 pull.url。 """ - url = _with_date_param(pull.url, pull.date_param, day, pull.date_format) if day else pull.url + if url_override is not None: + url = url_override + else: + url = _with_date_param(pull.url, pull.date_param, day, pull.date_format) if day else pull.url if extra_params: url = _append_query_params(url, extra_params) # 每配置超时 (PullConfig.timeout_seconds, 默认 30 与历史行为一致); @@ -251,15 +259,12 @@ def _append_query_params(url: str, params: dict[str, str]) -> str: def _parse_rows_payload(config: ExtConfig, pull: PullConfig, data: Any) -> list[dict]: - """单次响应 → 行列表 (提取/预设转换/字段映射)。与写入链共用, 不含日期校验。""" + """单次响应 → 行列表 (提取/字段映射)。与写入链共用, 不含日期校验。 + + 两级拉取 (pull.fanout) 不走这里 —— 它的行由多个响应聚合而来, 见 + fetch_rows_for_date 的分支。 + """ rows = _extract_rows(data, pull.response_path) - - # 内置预设 (概念/行业): 应用结构转换, 让产出 schema 与分析页一致。 - # 否则 raw 接口列 (concepts/industries 数组、name) 会直接覆盖正确的 part.parquet, - # 导致分析页因找不到维度字段 (所属概念/所属同花顺行业) 而"数据消失"。 - # 见 ext_presets._flatten_* —— 手动拉取 / 定时拉取都必须走同一套转换。 - rows = _apply_preset_flatten(config.id, rows) - # 字段映射 return _apply_field_map(rows, pull.field_map) @@ -267,15 +272,20 @@ def _parse_rows_payload(config: ExtConfig, pull: PullConfig, data: Any) -> list[ async def fetch_rows_for_date(config: ExtConfig, target_date: date) -> list[dict]: """按日期请求外部 API 并解析为行 (不写盘)。空数据返回 []。 - 与 fetch_and_ingest 共用同一解析链 (response_path/预设转换/字段映射/ - 关联字段校验), 历史回补与当日拉取不产生第二套口径。 + 与 fetch_and_ingest 共用同一解析链 (response_path/字段映射/关联字段校验), + 历史回补与当日拉取不产生第二套口径。 配置了 pull.page_param 时按页循环拉取 (仅 GET), 空页/短页/达 max_pages 停。 + 配置了 pull.fanout 时走两级拉取 (目录 → 逐条目明细), 见 ext_fanout。 """ pull = config.pull if not pull or not pull.url: raise ValueError("拉取未配置或 URL 为空") - if pull.page_param and pull.method.upper() == "GET": + if pull.fanout: + from app.services.ext_fanout import fetch_fanout_rows + + rows = await fetch_fanout_rows(config, target_date) + elif pull.page_param and pull.method.upper() == "GET": collected: list[dict] = [] page = pull.page_start for _ in range(pull.max_pages): diff --git a/backend/app/services/extend_history.py b/backend/app/services/extend_history.py index 0249ad4e..9cab0edf 100644 --- a/backend/app/services/extend_history.py +++ b/backend/app/services/extend_history.py @@ -37,31 +37,29 @@ def _invalidate(table: str | None = None) -> None: invalidate_data_cache(table) -def _resolve_universe(capset: CapabilitySet) -> list[str]: - """解析标的池 — 与 daily_pipeline 独立的副本。""" - if capset.has(Cap.KLINE_DAILY_BATCH): - try: - from app.tickflow.pools import get_pool - all_a = get_pool("CN_Equity_A", refresh=True) - if all_a: - return sorted(all_a) - except Exception as e: - logger.warning("CN_Equity_A pool unavailable: %s", e) +def _resolve_universe(capset: CapabilitySet, repo=None) -> list[str]: + """解析标的池 — 与 daily_pipeline 同口径: instruments 维表 + 自选 (单一来源)。 - from app.tickflow.pools import DEMO_SYMBOLS, get_pool as _get_pool - from app.config import settings + 不再拉 CN_Equity_A 池 (含 SW1 聚合兜底); 维表由盘后管道同步维护。 + repo 传入时过滤自选里的指数 symbol (指数日K走独立存储)。 + """ from pathlib import Path + import polars as pl - base: set[str] = set(DEMO_SYMBOLS) - base.update(_get_pool("watchlist")) - d = Path(settings.data_dir) - inst_path = d / "instruments" / "instruments.parquet" + + from app.config import settings + from app.tickflow.pools import get_pool + + base: set[str] = set(get_pool("watchlist")) + inst_path = Path(settings.data_dir) / "instruments" / "instruments.parquet" if inst_path.exists(): try: inst = pl.read_parquet(inst_path, columns=["symbol"]) base.update(inst["symbol"].to_list()) - except Exception as e: - logger.warning("instruments supplement failed: %s", e) + except Exception as e: # noqa: BLE001 + logger.warning("instruments 维表读取失败, 标的池仅剩自选: %s", e) + if repo is not None: + base -= set(repo.get_index_symbol_set()) return sorted(base) @@ -131,7 +129,7 @@ def run_extend_history( # 2. 解析标的池 emit("extend_history", 5, "解析标的池…") - universe = _resolve_universe(capset) + universe = _resolve_universe(capset, repo) if not universe: return {"error": "标的池为空"} emit("extend_history", 8, f"标的池: {len(universe)} 只") diff --git a/backend/app/services/financial_sync.py b/backend/app/services/financial_sync.py index 556b619a..d5fe393b 100644 --- a/backend/app/services/financial_sync.py +++ b/backend/app/services/financial_sync.py @@ -53,6 +53,13 @@ def _financial_is_custom() -> bool: return custom_sources.provider_has_dataset(provider, "financial") +def _financial_capability_label() -> str: + """门禁日志标签: "财务数据 ← fuyao" / "…(档位不足)", 便于判断该切哪个源。""" + from app.data_providers.capabilities import capability_gate + _, label = capability_gate("financial") + return label + + def _fetch_table( table: str, symbols: list[str], @@ -62,7 +69,7 @@ def _fetch_table( """通过当前财务数据源拉取一张标准化财务表。""" is_custom = _financial_is_custom() if not is_custom and not capset.has(Cap.FINANCIAL): - logger.info("sync_%s skipped: no FINANCIAL capability", table) + logger.info("sync_%s skipped: %s 不可用", table, _financial_capability_label()) return pl.DataFrame() if not symbols: logger.warning("sync_%s skipped: no symbols", table) @@ -251,7 +258,7 @@ def sync_shares(data_dir: Path, capset: CapabilitySet) -> int: def sync_all(data_dir: Path, capset: CapabilitySet) -> dict[str, int]: """同步所有财务表。返回 {table: rows}。""" if not capset.has(Cap.FINANCIAL) and not _financial_is_custom(): - logger.info("sync_all financials skipped: no FINANCIAL capability") + logger.info("sync_all financials skipped: %s 不可用", _financial_capability_label()) return {} symbols = _get_symbols(data_dir) @@ -332,7 +339,7 @@ class FinancialScheduler: self._data_dir = data_dir self._capset = capset if not capset.has(Cap.FINANCIAL) and not _financial_is_custom(): - logger.info("FinancialScheduler skipped: no FINANCIAL capability") + logger.info("FinancialScheduler skipped: %s 不可用", _financial_capability_label()) return # 从持久化恢复上次同步时间: 重启后前端仍能显示真实最后同步时间,而非"尚未同步" try: diff --git a/backend/app/services/index_sync.py b/backend/app/services/index_sync.py index 00640efc..de1c3a1c 100644 --- a/backend/app/services/index_sync.py +++ b/backend/app/services/index_sync.py @@ -205,6 +205,7 @@ def sync_and_persist_index_daily( on_chunk_done(current, total) 每个批次完成后回调。 """ if not capset.has(Cap.KLINE_DAILY_BATCH): + kline_sync._log_cap_unavailable("daily", "指数/ETF 日K同步") return 0 if symbols_override: @@ -340,6 +341,7 @@ def sync_and_persist_etf_daily( on_chunk_done(current, total) 每个批次完成后回调。 """ if not capset.has(Cap.KLINE_DAILY_BATCH): + kline_sync._log_cap_unavailable("daily", "ETF 日K同步") return 0 if symbols_override: diff --git a/backend/app/services/kline_sync.py b/backend/app/services/kline_sync.py index c0a477f1..aa78ca72 100644 --- a/backend/app/services/kline_sync.py +++ b/backend/app/services/kline_sync.py @@ -31,6 +31,20 @@ from app.tickflow.repository import KlineRepository, replace_with_retry logger = logging.getLogger(__name__) + +def _log_cap_unavailable(cap_id: str, action: str) -> None: + """门禁未通过时记一条带生效源的日志。 + + 原实现是静默 return —— 切换数据源后「什么都没发生」且无迹可循。 + 带源标签后可一眼看出是路由到不可用源, 还是 TickFlow 档位不足。 + """ + try: + from app.data_providers.capabilities import capability_gate + _, label = capability_gate(cap_id) + except Exception: # noqa: BLE001 — 仅日志, 失败不应影响主流程 + label = cap_id + logger.info("%s 跳过: %s 不可用", action, label) + # ── SDK 批量调用守护 deadline ──────────────────────────────────────────── # TickFlow SDK 自带 timeout=30s×3 重试, 但只保护其覆盖的请求阶段; 真挂死在 # 不可中断的网络读里时调用永不返回 → 持有独占槽的重任务线程被连坐卡死, @@ -365,6 +379,7 @@ def sync_and_persist_daily_batch( # 自定义源未配置 daily → 回退 TickFlow if not capset.has(Cap.KLINE_DAILY_BATCH): + _log_cap_unavailable("daily", "日K同步") return _finalize(0) limit = resolve_limit(capset, Cap.KLINE_DAILY_BATCH, default_batch=100) @@ -653,6 +668,7 @@ def sync_adj_factor(symbols: list[str], repo: KlineRepository, # 自定义源未配置 adj_factor → 回退 TickFlow if not capset.has(Cap.ADJ_FACTOR): + _log_cap_unavailable("adj_factor", "除权因子同步") return 0, [] tf = get_client() diff --git a/backend/app/services/market_overview_builder.py b/backend/app/services/market_overview_builder.py index 4060d0bd..5f95ca56 100644 --- a/backend/app/services/market_overview_builder.py +++ b/backend/app/services/market_overview_builder.py @@ -70,14 +70,9 @@ def _board(symbol: str) -> str: return "其他" -def _score(value: float, low: float, high: float) -> int: - if high <= low: - return 50 - if not math.isfinite(value): - # 非有限输入不可映射: 取中性 50, 不得让 round(inf)/round(nan) 抛异常打断看板计算 - # (与 app/services/regime_builder.py 的同名实现保持一致) - return 50 - return max(0, min(100, round((value - low) / (high - low) * 100))) +# 注: 原 _score(value, low, high) 已迁至 app.services.emotion_score.score_of —— +# 情绪评分的区间与权重现由用户可配置, 归一化函数须与配置同处一个模块。 +# regime_builder 保留同名实现 (其区间是 A 股分位数校准的另一套口径, 不共用)。 # ================================================================ @@ -568,27 +563,33 @@ def build_market_overview( mainline_items = [*concept_rank["leading"][:3], *industry_rank["leading"][:3]] mainline_avg = max([_finite(item.get("avg_pct")) or 0 for item in mainline_items], default=0) mainline_cover_pct = max([(_finite(item.get("count")) or 0) / total * 100 for item in mainline_items], default=0) if total else 0 - mainline_score = round(_score(mainline_avg, -0.005, 0.03) * 0.65 + _score(mainline_cover_pct, 1, 12) * 0.35) if mainline_items else 50 - radar = [ - {"key": "index", "label": "指数", "value": _score(avg_index_pct, -2.5, 2.5)}, - {"key": "profit", "label": "赚钱", "value": round(_score(up_pct, 20, 80) * 0.45 + _score(avg_pct, -0.02, 0.02) * 0.25 + _score(median_pct, -0.02, 0.02) * 0.20 + _score(strong_diff_pct, -8, 8) * 0.10)}, - {"key": "money", "label": "量能", "value": round(_score(avg_vol_ratio, 0.6, 1.8) * 0.70 + _score(high_vol_pct, 2, 12) * 0.30)}, - {"key": "speculation", "label": "投机", "value": round(_score(limit_up, 5, 90) * 0.25 + _score(seal_rate, 30, 85) * 0.35 + _score(max_boards, 1, 8) * 0.25 + _score(tier2_count, 0, 30) * 0.15)}, - {"key": "resilience", "label": "抗跌", "value": 100 - round(_score(down_pct, 20, 80) * 0.55 + _score(strong_down_pct, 1, 12) * 0.45)}, - {"key": "mainline", "label": "主线", "value": mainline_score}, - ] - emotion_score = round(sum(r["value"] for r in radar) / len(radar)) if radar else 50 - if emotion_score >= 70: - emotion_label = "强势" - elif emotion_score >= 55: - emotion_label = "偏暖" - elif emotion_score >= 45: - emotion_label = "震荡" - elif emotion_score >= 30: - emotion_label = "偏冷" - else: - emotion_label = "冰点" + # 情绪评分 (维度/权重/区间/档位) 走可配置口径: 定义在 app.services.emotion_score, + # 用户可在看板「情绪雷达」卡片上改权重与区间。指标名即该模块的 term.key。 + from app.services import preferences + from app.services.emotion_score import compute as compute_emotion + + emotion = compute_emotion({ + "avg_index_pct": avg_index_pct, + "up_pct": up_pct, + "avg_pct": avg_pct, + "median_pct": median_pct, + "strong_diff_pct": strong_diff_pct, + "avg_vol_ratio": avg_vol_ratio, + "high_vol_pct": high_vol_pct, + "limit_up": limit_up, + "seal_rate": seal_rate, + "max_boards": max_boards, + "tier2_count": tier2_count, + "down_pct": down_pct, + "strong_down_pct": strong_down_pct, + # 主线仅在有条目时才有意义: 无条目时该维度取中性 50 (与历史实现一致) + **({"mainline_avg": mainline_avg, "mainline_cover_pct": mainline_cover_pct} if mainline_items else {}), + }, preferences.get_emotion_score_config()) + + radar = emotion["radar"] + emotion_score = emotion["score"] + emotion_label = emotion["label"] return _json_safe({ "as_of": str(as_of), diff --git a/backend/app/services/preferences.py b/backend/app/services/preferences.py index efd897ab..6e375e57 100644 --- a/backend/app/services/preferences.py +++ b/backend/app/services/preferences.py @@ -435,6 +435,38 @@ def get_mainline_filter_config() -> dict: } +def get_emotion_score_config() -> dict | None: + """情绪评分口径配置 (维度权重/子项权重与区间/档位阈值)。 + + 返回原始 dict 或 None (未自定义 = 用出厂口径)。**不在此处做结构校验与补全** —— + 合并与归一化统一由 emotion_score.merge_config 负责, 避免两处各写一套默认值而漂移。 + """ + v = load().get("emotion_score_config") + return v if isinstance(v, dict) else None + + +def set_emotion_score_config(cfg: dict | None) -> dict | None: + """保存情绪评分口径。传 None 或 {} = 恢复出厂口径 (清除该键)。 + + 只做形状校验, 值的合法性 (负数权重/区间倒置/阈值乱序) 由 emotion_score 层 + 在每次计算时归一化 —— 存进去什么由用户负责, 算出来一定安全。 + """ + from app.services.emotion_score import merge_config + + if not cfg: + with _SAVE_LOCK: + current = load() + current.pop("emotion_score_config", None) + atomic_write_text(_path(), json.dumps(current, indent=2, ensure_ascii=False)) + _invalidate_cache() + return None + + # 落盘前过一遍 merge_config: 校验形状并剔除未知字段, 避免脏配置长期驻留 + normalized = merge_config(cfg) + save({"emotion_score_config": normalized}) + return normalized + + def set_mainline_filter_config(cfg: dict) -> dict: """保存主线过滤配置(白名单字段, 部分更新)。修改后需重算主线生效。""" updates: dict = {} diff --git a/backend/app/services/quote_service.py b/backend/app/services/quote_service.py index ee7823da..948431f0 100644 --- a/backend/app/services/quote_service.py +++ b/backend/app/services/quote_service.py @@ -383,7 +383,10 @@ class QuoteService: if not self.is_realtime_allowed(): if preferences.get_realtime_quotes_enabled(): self._save_enabled(False) - logger.info("实时行情未启动:当前档位(none)无实时行情权限") + # 带生效源: 便于判断「该切哪个源」, 而不是只看到 TickFlow 档位 + from app.data_providers.capabilities import capability_gate + _, label = capability_gate("realtime") + logger.info("实时行情未启动: %s 不可用", label) return if preferences.get_realtime_quotes_enabled(): self.start() diff --git a/backend/app/services/task_registry.py b/backend/app/services/task_registry.py new file mode 100644 index 00000000..21126585 --- /dev/null +++ b/backend/app/services/task_registry.py @@ -0,0 +1,178 @@ +"""进行中任务注册表 — 长任务(回测/挖掘/优化/滚动验证)跨设备接回。 + +前端在启动任务时 fire-and-forget 上报 (status=running, id=kind+参数指纹), +终态时按同一 id 关闭; 其他设备打开对应页面时, 本地无 reconnect 键则从注册表 +取最近 running 项回填本地键, 复用既有 tryReconnect 机制 — 任务执行与重连 +逻辑零改动, 上报失败也只是退化为原来的单机行为。 + +文件: data/user_data/task_registry.json + {"items": [{id, kind, label, payload, status, started_at, updated_at}, ...]} + +容量与时效: ≤20 条 (超限裁 updated_at 最旧); running 超过 24h 视为孤儿 +(进程重启遗留), 读取时静默剔除并落盘回收。 +""" +from __future__ import annotations + +import json +import logging +import time +from pathlib import Path +from typing import Any + +from app.services.fs_utils import atomic_write_text + +logger = logging.getLogger(__name__) + +TASK_KINDS = frozenset({"backtest", "mining", "optimizer", "walkforward"}) +TASK_STATUSES = frozenset({"running", "done", "error"}) +MAX_ITEMS = 20 +ORPHAN_RUNNING_SECONDS = 24 * 3600 +_FILENAME = "task_registry.json" + +_cache: dict[str, Any] | None = None +_cache_sig: tuple[int, int] | None = None + + +def _path(data_dir: Path) -> Path: + p = data_dir / "user_data" / _FILENAME + p.parent.mkdir(parents=True, exist_ok=True) + return p + + +def _signature(p: Path) -> tuple[int, int] | None: + try: + st = p.stat() + return (st.st_mtime_ns, st.st_size) + except OSError: + return None + + +def _sanitize_items(data: Any) -> list[dict]: + if not isinstance(data, dict): + return [] + items = data.get("items") + if not isinstance(items, list): + return [] + out: list[dict] = [] + for it in items: + if not isinstance(it, dict): + continue + if it.get("kind") not in TASK_KINDS or it.get("status") not in TASK_STATUSES: + continue + if not isinstance(it.get("id"), str) or not it["id"]: + continue + out.append({ + "id": it["id"], + "kind": it["kind"], + "status": it["status"], + "label": it["label"] if isinstance(it.get("label"), str) else "", + "payload": it["payload"] if isinstance(it.get("payload"), dict) else {}, + "started_at": it.get("started_at") or 0, + "updated_at": it.get("updated_at") or 0, + }) + return out + + +def _sorted_trimmed(items: list[dict]) -> list[dict]: + ordered = sorted(items, key=lambda x: x.get("updated_at") or 0, reverse=True) + return ordered[:MAX_ITEMS] + + +def _prune_orphans(items: list[dict]) -> tuple[list[dict], int]: + now = time.time() + kept: list[dict] = [] + orphans = 0 + for it in items: + if ( + it["status"] == "running" + and it["updated_at"] + and now - it["updated_at"] > ORPHAN_RUNNING_SECONDS + ): + orphans += 1 + continue + kept.append(it) + return kept, orphans + + +def _load_raw(data_dir: Path) -> list[dict] | None: + global _cache, _cache_sig + p = _path(data_dir) + sig = _signature(p) + if sig is None: + return [] + if _cache is not None and sig == _cache_sig: + return _cache["items"] + try: + data = json.loads(p.read_text(encoding="utf-8")) + except Exception as e: # noqa: BLE001 — 损坏文件视为空注册表 + logger.warning("task registry 文件损坏, 视为空: %s", e) + return [] + items = _sorted_trimmed(_sanitize_items(data)) + _cache, _cache_sig = {"items": items}, sig + return items + + +def _save(data_dir: Path, items: list[dict]) -> None: + global _cache, _cache_sig + trimmed = _sorted_trimmed(items) + p = _path(data_dir) + atomic_write_text(p, json.dumps({"items": trimmed}, ensure_ascii=False)) + _cache, _cache_sig = {"items": trimmed}, _signature(p) + + +def list_items(data_dir: Path, kind: str | None = None) -> list[dict]: + """按 updated_at 倒序返回 (可选 kind 过滤); 顺带回收孤儿 running 项。""" + items = _load_raw(data_dir) + if items is None: + return [] + kept, orphans = _prune_orphans(items) + if orphans: + _save(data_dir, kept) + items = kept + if kind is None: + return items + return [it for it in items if it["kind"] == kind] + + +def upsert( + data_dir: Path, + task_id: str, + *, + kind: str, + status: str, + label: str = "", + payload: dict | None = None, +) -> dict: + if kind not in TASK_KINDS: + raise ValueError(f"未知的任务类型: {kind}") + if status not in TASK_STATUSES: + raise ValueError(f"未知的任务状态: {status}") + items = _load_raw(data_dir) or [] + now = time.time() + entry = { + "id": task_id, + "kind": kind, + "status": status, + "label": label, + "payload": payload or {}, + "started_at": now, + "updated_at": now, + } + for i, it in enumerate(items): + if it["id"] == task_id: + entry["started_at"] = it.get("started_at") or now + items[i] = entry + break + else: + items.append(entry) + _save(data_dir, items) + return entry + + +def delete(data_dir: Path, task_id: str) -> bool: + items = _load_raw(data_dir) or [] + remaining = [it for it in items if it["id"] != task_id] + if len(remaining) == len(items): + return False + _save(data_dir, remaining) + return True diff --git a/backend/app/strategy/prompts/strategy-guide.md b/backend/app/strategy/prompts/strategy-guide.md index 0c8800da..74d377d7 100644 --- a/backend/app/strategy/prompts/strategy-guide.md +++ b/backend/app/strategy/prompts/strategy-guide.md @@ -140,6 +140,7 @@ def filter_history(df: pl.DataFrame, params: dict) -> pl.DataFrame: - `LOOKBACK_DAYS` 决定引擎加载多少天的数据，设为策略逻辑需要的最大回看天数 - 优先使用 Polars 的 `with_columns`、`over("symbol")`、`group_by`、`join`、`filter` 实现历史逻辑，避免把数据转成 Python list/dict 循环 - 只有遇到表达式难以描述的复杂状态机时，才使用 `partition_by("symbol")` + `to_dicts()` 逐股票分析 +- **`to_dicts()` 逐行攒下的命中行重建 DataFrame 必须显式带 schema**：写 `pl.DataFrame(hits, schema=df.schema)`，**不能裸写 `pl.DataFrame(hits)`**。原因：从 dict 列表建表只扫描前 100 行推断列类型（默认 `infer_schema_length=100`），若某列在前 100 个命中行里全为 `None`，该列会被推断成 Null 类型，之后某行这一列出现数值时直接报 `ComputeError: could not append value ... to the builder`——是否触发取决于命中行的排列顺序，数据一变就可能复现，非常隐蔽；显式带 schema 同时保住 date/整型等原始列类型不被重新推断 - **返回所有匹配行，不要过滤 `latest`**；策略引擎会自动取最新日，回测引擎需要全区间命中 - 未声明 `filter_history()` 的策略走普通 `filter()` 路径，不受影响 - **date 类型参数必须先转换再与 `date` 列比较**：params 里的 `"type": "date"` 参数从 JSON 传来是字符串（如 `"2024-01-01"`），而数据中 `date` 列是 Polars Date 类型，**不能直接比较**，否则报错。必须先转换： @@ -288,7 +289,7 @@ REQUIRED_FEATURES = {"ma20", "momentum_20d", "vol_ratio_5d"} 6. Polars 策略允许 `import polars as pl` 和 `from datetime import date/datetime`；矩阵策略只允许 NumPy 和 `app.backtest.matrix` 协议/算子 7. 禁止使用 `open()`, `exec()`, `eval()`, `os`, `sys`, `subprocess` 8. **贴合用户需求优先**：第3/4节的指标列和信号列仅供参考，能用则用；如果用户需求需要自定义计算（如"前高""上次涨停价""N日内某个事件后X天"），直接在 `filter_history()` 中自行设计和计算，不需要局限于已有列 -9. `filter_history()` 中优先用 Polars 向量化语法；仅在复杂状态机无法清晰表达时，才用 `partition_by("symbol")` 逐股票分析 +9. `filter_history()` 中优先用 Polars 向量化语法；仅在复杂状态机无法清晰表达时，才用 `partition_by("symbol")` 逐股票分析；`to_dicts()` 逐行分析的命中行必须以 `pl.DataFrame(hits, schema=df.schema)` 重建，禁止裸 `pl.DataFrame(hits)`（dict 推断类型漂移，见第 2 节关键要点） ## 7. 策略示例 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 1229247c..fa35ea34 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -26,13 +26,15 @@ dependencies = [ "pypinyin>=0.50", # 股票名拼音首字母搜索 (同花顺式 payh → 平安银行) # TickFlow 官方 SDK "tickflow[all]>=0.1.23", + # TdxQuant 内置数据源插件的原生 TCP 客户端 (wheel 自带跨平台库, 无需系统依赖) + "tdxaidata>=1.2.2", # Scheduling "apscheduler>=3.10", # Config "pyyaml>=6.0", "python-dotenv>=1.0", # AI(可选,但默认装上) - "openai>=1.40", # OpenAI 兼容适配器复用 openai SDK + "openai>=1.66", # OpenAI 兼容适配器复用 openai SDK; 1.66+ 提供 Responses API "httpx>=0.27", "platformdirs>=4.0", # 桌面版用户数据目录 (跨平台持久可写) "winotify>=1.1; sys_platform == 'win32'", # Windows 系统通知 (进操作中心) diff --git a/backend/tests/test_ai_probe.py b/backend/tests/test_ai_probe.py new file mode 100644 index 00000000..ed07ad03 --- /dev/null +++ b/backend/tests/test_ai_probe.py @@ -0,0 +1,161 @@ +"""AI 草稿连通性测试端点 (POST /api/settings/ai/test) 的契约测试。 + +核心契约: 测试只读表单值, 与保存完全解耦 —— 测试失败不得写任何配置, +状态栏也不得把未验证的草稿展示为已生效。失败以 200 + ok=false 回报, +由前端内联展示, 不触发全局 toast。 +""" +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from app import secrets_store +from app.api import settings as settings_api +from app.services import ai_provider + + +def test_probe_requires_base_url_model_and_key(monkeypatch): + monkeypatch.setattr(secrets_store, "get_ai_key", lambda: "") + + with pytest.raises(ValueError, match="API 地址"): + import asyncio + asyncio.run(ai_provider.probe_ai_config( + provider="openai_compat", api_key="sk-x", base_url="", model="m", + )) + with pytest.raises(ValueError, match="模型"): + import asyncio + asyncio.run(ai_provider.probe_ai_config( + provider="openai_compat", api_key="sk-x", base_url="https://x.example/v1", model="", + )) + with pytest.raises(RuntimeError, match="API Key"): + import asyncio + asyncio.run(ai_provider.probe_ai_config( + provider="openai_compat", api_key="", base_url="https://x.example/v1", model="m", + )) + + +def test_probe_codex_unavailable_raises(monkeypatch): + import asyncio + + monkeypatch.setattr(ai_provider, "codex_cli_available", lambda: False) + with pytest.raises(RuntimeError, match="Codex CLI 不可用"): + asyncio.run(ai_provider.probe_ai_config( + provider="codex_cli", api_key="", base_url="", model="", + )) + + +@pytest.mark.asyncio +async def test_probe_chat_protocol_uses_given_key_and_base(monkeypatch): + calls: list[dict] = [] + + class _Completions: + async def create(self, **kwargs): + calls.append(kwargs) + message = SimpleNamespace(content="OK") + return SimpleNamespace(choices=[SimpleNamespace(message=message)]) + + class _Chat: + completions = _Completions() + + captured_client: dict = {} + + class _Client: + def __init__(self, **kwargs): + captured_client.update(kwargs) + self.chat = _Chat() + + import openai + monkeypatch.setattr(openai, "AsyncOpenAI", _Client) + + text = await ai_provider.probe_ai_config( + provider="openai_compat", + api_key="sk-draft", + base_url="https://draft.example/v1", + model="draft-model", + ) + + assert text == "OK" + assert captured_client["api_key"] == "sk-draft" + assert "draft.example" in captured_client["base_url"] + assert calls == [{ + "model": "draft-model", + "messages": [{"role": "user", "content": "Reply exactly: OK"}], + }] + + +@pytest.mark.asyncio +async def test_probe_responses_protocol_calls_responses(monkeypatch): + calls: list[dict] = [] + + class _Responses: + async def create(self, **kwargs): + calls.append(kwargs) + return SimpleNamespace(output_text="ok") + + class _Client: + def __init__(self, **kwargs): + self.responses = _Responses() + + import openai + monkeypatch.setattr(openai, "AsyncOpenAI", _Client) + + text = await ai_provider.probe_ai_config( + provider="openai_compat", + api_key="sk-draft", + base_url="https://draft.example/v1", + model="draft-model", + protocol="responses", + ) + + assert text == "ok" + assert calls[0]["model"] == "draft-model" + assert calls[0]["input"] == [{"role": "user", "content": "Reply exactly: OK"}] + + +@pytest.mark.asyncio +async def test_test_endpoint_failure_reports_error_without_saving(monkeypatch): + """关键回归: 测试失败 → ok=false 回报, 且不调用任何 secrets 写入。""" + saved: list = [] + monkeypatch.setattr(settings_api.secrets_store, "save", lambda updates: saved.append(updates)) + + async def boom(**kwargs): + raise RuntimeError("模型或接口地址不存在") + + monkeypatch.setattr(ai_provider, "probe_ai_config", boom) + + result = await settings_api.test_ai_settings(settings_api.AiTestIn( + provider="openai_compat", + base_url="https://bad.example/v1", + api_key="sk-bad", + model="no-such-model", + )) + + assert result["ok"] is False + assert "不存在" in result["error"] + assert saved == [] # 未写入任何配置 + + +@pytest.mark.asyncio +async def test_test_endpoint_falls_back_to_saved_key(monkeypatch): + """表单 Key 留空时回退已保存 Key (改模型/协议不必重输)。""" + seen: dict = {} + + async def fake_probe(**kwargs): + seen.update(kwargs) + return "OK" + + monkeypatch.setattr(ai_provider, "probe_ai_config", fake_probe) + monkeypatch.setattr(settings_api.secrets_store, "get_ai_key", lambda: "sk-saved") + + result = await settings_api.test_ai_settings(settings_api.AiTestIn( + provider="openai_compat", + base_url="https://api.example/v1", + api_key=None, + model="m1", + protocol="responses", + )) + + assert result["ok"] is True + assert seen["api_key"] == "sk-saved" + assert seen["protocol"] == "responses" diff --git a/backend/tests/test_ai_responses.py b/backend/tests/test_ai_responses.py new file mode 100644 index 00000000..32e798c2 --- /dev/null +++ b/backend/tests/test_ai_responses.py @@ -0,0 +1,613 @@ +"""Responses API 协议适配层 (app/services/ai_responses.py) 的契约测试。 + +伪事件流/伪响应全部用 SimpleNamespace 鸭子类型, 只对齐 SDK 暴露的字段名 +(event.type / event.delta / response.status / item.call_id 等) — 适配层 +本身也只依赖这些字段, 与 _iter_openai_text 的宽松风格一致。 +""" +from __future__ import annotations + +import json +from types import SimpleNamespace + +import pytest + +from app import secrets_store +from app.api import settings as settings_api +from app.config import settings as app_settings +from app.services import ai_provider, ai_responses + +# --------------------------------------------------------------------------- +# 事件 / 响应工厂 +# --------------------------------------------------------------------------- + + +def _evt(type_: str, **kw): + return SimpleNamespace(type=type_, **kw) + + +def _text_delta(s: str): + return _evt("response.output_text.delta", delta=s) + + +def _reasoning_delta(s: str = "内部推理"): + return _evt("response.reasoning_summary_text.delta", delta=s) + + +def _completed(output=None, status="completed", incomplete_reason=None): + resp = SimpleNamespace(status=status, output=list(output or []), error=None, output_text="") + if incomplete_reason is not None: + resp.incomplete_details = SimpleNamespace(reason=incomplete_reason) + return _evt("response.completed", response=resp) + + +async def _fake_stream(*events): + for e in events: + yield e + + +def _message_item(text: str): + part = SimpleNamespace(type="output_text", text=text) + return SimpleNamespace(type="message", content=[part]) + + +def _call_item(call_id: str, name: str, arguments: str): + return SimpleNamespace(type="function_call", call_id=call_id, name=name, arguments=arguments) + + +class _FakeApiError(Exception): + """带 status_code/body 的伪上游错误, 喂给重试判定逻辑。""" + + def __init__(self, status_code: int, body: dict): + super().__init__(json.dumps(body)) + self.status_code = status_code + self.body = body + + +def _fake_client(result=None, stream=None, error=None): + """伪 AsyncOpenAI: client.responses.create(**kwargs)。返回 (client, calls)。""" + calls: list[dict] = [] + + async def create(**kwargs): + calls.append(kwargs) + if error is not None: + raise error + if stream is not None: + return stream + return result + + return SimpleNamespace(responses=SimpleNamespace(create=create)), calls + + +# --------------------------------------------------------------------------- +# 消息 / 工具转换 +# --------------------------------------------------------------------------- + + +def test_chat_input_converts_tool_round_trip_shapes(): + messages = [ + {"role": "system", "content": "你是助手"}, + {"role": "user", "content": "查一下茅台"}, + { + "role": "assistant", + "content": "先查行情", + "tool_calls": [ + {"id": "call_1", "type": "function", + "function": {"name": "get_quote", "arguments": '{"symbol": "600519"}'}}, + ], + }, + {"role": "tool", "tool_call_id": "call_1", "content": '{"ok": true}'}, + ] + + items = ai_responses.chat_input(messages) + + assert items == [ + {"role": "system", "content": "你是助手"}, + {"role": "user", "content": "查一下茅台"}, + {"role": "assistant", "content": "先查行情"}, + {"type": "function_call", "call_id": "call_1", "name": "get_quote", + "arguments": '{"symbol": "600519"}'}, + {"type": "function_call_output", "call_id": "call_1", "output": '{"ok": true}'}, + ] + + +def test_chat_input_plain_assistant_and_missing_role(): + items = ai_responses.chat_input([ + {"role": "assistant", "content": "纯文本回复"}, + {"content": "无角色消息"}, + ]) + assert items == [ + {"role": "assistant", "content": "纯文本回复"}, + {"role": "user", "content": "无角色消息"}, + ] + + +def test_responses_tools_flattens_and_passes_through_flat(): + chat_schema = [{ + "type": "function", + "function": {"name": "get_quote", "description": "查行情", "parameters": {"type": "object"}}, + }] + flat_schema = [{ + "type": "function", + "name": "get_kline", + "parameters": {"type": "object"}, + }] + + assert ai_responses.responses_tools(chat_schema) == [{ + "type": "function", "name": "get_quote", + "description": "查行情", "parameters": {"type": "object"}, + }] + # 已扁平的 schema 原样接受(个别网关直接吐扁平形态); description 缺省可省 + assert ai_responses.responses_tools(flat_schema) == [{ + "type": "function", "name": "get_kline", "parameters": {"type": "object"}, + }] + # 非 dict / 无 function 体的条目静默跳过 + assert ai_responses.responses_tools([None, {"type": "other"}]) == [] + + +# --------------------------------------------------------------------------- +# kwargs 与自适应重试 +# --------------------------------------------------------------------------- + + +def test_responses_kwargs_maps_param_names(monkeypatch): + stored = {"ai_provider": "openai", "ai_reasoning_effort": "medium"} + monkeypatch.setattr(secrets_store, "load", lambda: stored) + + kwargs = ai_responses.responses_kwargs(temperature=0.3, max_tokens=1200) + assert kwargs == { + "max_output_tokens": 1200, + "temperature": 0.3, + "reasoning": {"effort": "medium"}, + } + + # 兼容 provider 不下发 reasoning (对齐 _openai_kwargs 的 provider 门控) + stored["ai_provider"] = "openai_compat" + kwargs = ai_responses.responses_kwargs(temperature=None, max_tokens=1200) + assert kwargs == {"max_output_tokens": 1200} + + +def test_responses_kwargs_omits_none_max_tokens(): + assert ai_responses.responses_kwargs(temperature=0.5, max_tokens=None) == {"temperature": 0.5} + + +def test_responses_kwargs_carries_tools(): + kwargs = ai_responses.responses_kwargs( + temperature=None, max_tokens=None, + tools=[{"type": "function", "function": {"name": "f", "parameters": {"type": "object"}}}], + ) + assert kwargs["tools"] == [{"type": "function", "name": "f", "parameters": {"type": "object"}}] + + +def test_responses_retry_kwargs_drops_rejected_params(): + temperature_reject = _FakeApiError(400, {"error": {"message": "temperature only 1 is allowed"}}) + reasoning_reject = _FakeApiError(400, {"error": {"message": "Unsupported parameter: 'reasoning'"}}) + + retry = ai_responses.responses_retry_kwargs( + temperature_reject, {"temperature": 0.3, "reasoning": {"effort": "high"}}, + ) + assert retry == {"reasoning": {"effort": "high"}} + + retry = ai_responses.responses_retry_kwargs( + reasoning_reject, {"temperature": 0.3, "reasoning": {"effort": "high"}}, + ) + assert retry == {"temperature": 0.3} + + # 非 400 / 无关 400 不重试 + assert ai_responses.responses_retry_kwargs( + _FakeApiError(500, {"error": {"message": "boom"}}), {"temperature": 0.3}, + ) is None + assert ai_responses.responses_retry_kwargs( + _FakeApiError(400, {"error": {"message": "model not found"}}), {"temperature": 0.3}, + ) is None + + +# --------------------------------------------------------------------------- +# iter_text: 事件流归一化 (与 _iter_openai_text 的防御等价) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_iter_text_yields_deltas_and_ignores_reasoning(): + stream = _fake_stream( + _reasoning_delta(), + _text_delta("完整"), + _text_delta("正文"), + _completed(), + ) + assert [p async for p in ai_responses.iter_text(stream)] == ["完整", "正文"] + + +@pytest.mark.asyncio +async def test_iter_text_done_event_fallback_without_deltas(): + """个别网关只发 output_text.done 不发 delta: done 全量兜底产出一次。""" + stream = _fake_stream( + _evt("response.output_text.done", text="一次成型"), + _completed(), + ) + assert [p async for p in ai_responses.iter_text(stream)] == ["一次成型"] + + +@pytest.mark.asyncio +async def test_iter_text_rejects_length_truncated_partial(): + stream = _fake_stream( + _text_delta("未完成正文"), + _completed(status="incomplete", incomplete_reason="max_output_tokens"), + ) + with pytest.raises(RuntimeError, match="输出达到长度上限"): + async for _ in ai_responses.iter_text(stream): + pass + + +@pytest.mark.asyncio +async def test_iter_text_rejects_reasoning_only_length_exhaustion(): + stream = _fake_stream( + _reasoning_delta(), + _completed(status="incomplete", incomplete_reason="max_output_tokens"), + ) + with pytest.raises(RuntimeError, match="推理达到输出长度上限"): + async for _ in ai_responses.iter_text(stream): + pass + + +@pytest.mark.asyncio +async def test_iter_text_rejects_stream_without_content(): + with pytest.raises(RuntimeError, match="未返回正文内容"): + async for _ in ai_responses.iter_text(_fake_stream(_completed())): + pass + + +@pytest.mark.asyncio +async def test_iter_text_failed_event_raises_with_detail(): + resp = SimpleNamespace( + status="failed", output=[], error=SimpleNamespace(message="quota exceeded"), + ) + stream = _fake_stream(_evt("response.failed", response=resp)) + with pytest.raises(RuntimeError, match="quota exceeded"): + async for _ in ai_responses.iter_text(stream): + pass + + +# --------------------------------------------------------------------------- +# 非流式: run_text / run_message_once +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_run_text_extracts_text_with_and_without_helper_attr(monkeypatch): + resp = SimpleNamespace( + status="completed", + output_text="聚合文本", + output=[_message_item("聚合文本")], + error=None, + ) + client, calls = _fake_client(result=resp) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + assert await ai_responses.run_text( + [{"role": "user", "content": "hi"}], temperature=0.3, max_tokens=100, timeout=5, + ) == "聚合文本" + assert calls[0]["input"] == [{"role": "user", "content": "hi"}] + assert calls[0]["max_output_tokens"] == 100 + + # output_text 便利属性缺失时逐 item 聚合 + resp2 = SimpleNamespace(status="completed", output=[_message_item("逐条"), _message_item("聚合")]) + client2, _ = _fake_client(result=resp2) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client2) + assert await ai_responses.run_text( + [{"role": "user", "content": "hi"}], temperature=None, max_tokens=None, timeout=5, + ) == "逐条聚合" + + +@pytest.mark.asyncio +async def test_run_text_incomplete_raises(monkeypatch): + resp = SimpleNamespace( + status="incomplete", + output_text="半截", + output=[], + incomplete_details=SimpleNamespace(reason="max_output_tokens"), + ) + client, _ = _fake_client(result=resp) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + with pytest.raises(RuntimeError, match="输出达到长度上限"): + await ai_responses.run_text( + [{"role": "user", "content": "hi"}], temperature=None, max_tokens=10, timeout=5, + ) + + +@pytest.mark.asyncio +async def test_run_message_once_returns_chat_shape_calls(monkeypatch): + resp = SimpleNamespace( + status="completed", + output_text="", + output=[_call_item("call_1", "get_quote", '{"symbol":"600519"}')], + ) + client, calls = _fake_client(result=resp) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + message = await ai_responses.run_message_once( + [{"role": "user", "content": "查"}], + temperature=0.3, max_tokens=None, timeout=5, + tools=[{"type": "function", "function": {"name": "get_quote"}}], + ) + tc = message.tool_calls[0] + assert (tc.id, tc.function.name, tc.function.arguments) == ( + "call_1", "get_quote", '{"symbol":"600519"}', + ) + # 工具 schema 以扁平形态下发 + assert calls[0]["tools"] == [{"type": "function", "name": "get_quote"}] + + +@pytest.mark.asyncio +async def test_run_message_once_retries_without_rejected_temperature(monkeypatch): + ok_resp = SimpleNamespace(status="completed", output_text="OK", output=[_message_item("OK")]) + error = _FakeApiError(400, {"error": {"message": "temperature only 1 is allowed"}}) + client, calls = _fake_client(result=ok_resp, error=error) + # 首次抛 400, 摘掉 temperature 后重试成功 + state = {"raised": False} + + async def create(**kwargs): + calls.append(kwargs) + if not state["raised"]: + state["raised"] = True + raise error + return ok_resp + + client = SimpleNamespace(responses=SimpleNamespace(create=create)) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + text = await ai_responses.run_message_once( + [{"role": "user", "content": "hi"}], temperature=0.3, max_tokens=None, timeout=5, + ) + assert text.content == "OK" + assert calls[0]["temperature"] == 0.3 + assert "temperature" not in calls[1] + + +# --------------------------------------------------------------------------- +# stream_round: 助手工具轮事件流 +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_stream_round_accumulates_tool_calls_from_fragments(monkeypatch): + stream = _fake_stream( + _evt("response.output_item.added", + item=SimpleNamespace(id="fc_1", type="function_call", call_id="call_1", + name="get_quote", arguments="")), + _text_delta("先查一下"), + _evt("response.function_call_arguments.delta", item_id="fc_1", delta='{"sym'), + _evt("response.function_call_arguments.delta", item_id="fc_1", delta='bol":"600519"}'), + _evt("response.output_item.done", + item=SimpleNamespace(id="fc_1", type="function_call", call_id="call_1", + name="get_quote", arguments='{"symbol":"600519"}')), + _completed(output=[_call_item("call_1", "get_quote", '{"symbol":"600519"}')]), + ) + client, _ = _fake_client(stream=stream) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + events = [e async for e in ai_responses.stream_round( + [{"role": "user", "content": "查茅台"}], + [{"type": "function", "function": {"name": "get_quote"}}], + temperature=0.3, timeout=5, + )] + + assert events[0] == {"type": "text", "delta": "先查一下"} + assert events[-1] == { + "type": "round_end", + "tool_calls": [{"id": "call_1", "name": "get_quote", "arguments": '{"symbol":"600519"}'}], + "finish_reason": "tool_calls", + } + + +@pytest.mark.asyncio +async def test_stream_round_completed_output_is_authoritative(monkeypatch): + """completed.response.output 优先于分片累积(网关乱序/缺 done 时兜底)。""" + stream = _fake_stream( + _evt("response.output_item.added", + item=SimpleNamespace(id="fc_1", type="function_call", call_id="call_1", + name="", arguments="")), + _evt("response.function_call_arguments.delta", item_id="fc_1", delta="半截参数"), + _completed(output=[_call_item("call_x", "get_kline", '{"days":5}')]), + ) + client, _ = _fake_client(stream=stream) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + events = [e async for e in ai_responses.stream_round( + [{"role": "user", "content": "查K线"}], [], temperature=None, timeout=5, + )] + assert events[-1]["tool_calls"] == [ + {"id": "call_x", "name": "get_kline", "arguments": '{"days":5}'}, + ] + + +@pytest.mark.asyncio +async def test_stream_round_text_only_round_ends_empty(monkeypatch): + stream = _fake_stream( + _text_delta("回答完毕"), + _completed(), + ) + client, _ = _fake_client(stream=stream) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + events = [e async for e in ai_responses.stream_round( + [{"role": "user", "content": "hi"}], [], temperature=None, timeout=5, + )] + assert events == [ + {"type": "text", "delta": "回答完毕"}, + {"type": "round_end", "tool_calls": [], "finish_reason": "stop"}, + ] + + +@pytest.mark.asyncio +async def test_stream_round_length_truncation_without_calls_raises(monkeypatch): + stream = _fake_stream( + _text_delta("半截"), + _completed(status="incomplete", incomplete_reason="max_output_tokens"), + ) + client, _ = _fake_client(stream=stream) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + with pytest.raises(RuntimeError, match="输出达到长度上限"): + async for _ in ai_responses.stream_round( + [{"role": "user", "content": "hi"}], [], temperature=None, timeout=5, + ): + pass + + +@pytest.mark.asyncio +async def test_stream_round_empty_response_raises(monkeypatch): + client, _ = _fake_client(stream=_fake_stream(_completed())) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + with pytest.raises(RuntimeError, match="未返回正文内容"): + async for _ in ai_responses.stream_round( + [{"role": "user", "content": "hi"}], [], temperature=None, timeout=5, + ): + pass + + +# --------------------------------------------------------------------------- +# 协议配置与入口接线 +# --------------------------------------------------------------------------- + + +def test_current_ai_protocol_config_and_fallback(monkeypatch): + stored: dict = {} + monkeypatch.setattr(secrets_store, "load", lambda: stored) + + assert ai_provider.current_ai_protocol() == "chat_completions" + + stored["ai_protocol"] = "responses" + assert ai_provider.current_ai_protocol() == "responses" + assert ai_provider.is_responses_protocol() is True + + stored["ai_protocol"] = "bogus" + assert ai_provider.current_ai_protocol() == "chat_completions" + + # Codex CLI 恒为 chat_completions, 存储值不生效 + stored["ai_provider"] = "codex_cli" + stored["ai_protocol"] = "responses" + assert ai_provider.current_ai_protocol() == "chat_completions" + + +@pytest.mark.asyncio +async def test_generate_ai_text_routes_to_responses(monkeypatch): + async def fake_run_text(messages, *, temperature, max_tokens, timeout): + return "responses 回复" + + monkeypatch.setattr(secrets_store, "load", lambda: {"ai_protocol": "responses"}) + monkeypatch.setattr(ai_responses, "run_text", fake_run_text) + + text = await ai_provider.generate_ai_text( + [{"role": "user", "content": "hi"}], temperature=0.3, max_tokens=100, timeout=5, + ) + assert text == "responses 回复" + + +@pytest.mark.asyncio +async def test_stream_ai_text_routes_to_responses(monkeypatch): + async def fake_stream_text(messages, *, temperature, max_tokens, timeout): + yield "流式" + yield "回复" + + monkeypatch.setattr(secrets_store, "load", lambda: {"ai_protocol": "responses"}) + monkeypatch.setattr(ai_responses, "stream_text", fake_stream_text) + + chunks = [c async for c in ai_provider.stream_ai_text( + [{"role": "user", "content": "hi"}], temperature=0.5, max_tokens=100, timeout=5, + )] + assert chunks == ["流式", "回复"] + + +@pytest.mark.asyncio +async def test_assistant_round_routes_to_responses(monkeypatch): + from app.custom.assistant import streaming + + async def fake_stream_round(messages, tools, *, temperature, timeout): + yield {"type": "text", "delta": "助手回复"} + yield {"type": "round_end", "tool_calls": [], "finish_reason": "stop"} + + monkeypatch.setattr(secrets_store, "load", lambda: {"ai_protocol": "responses"}) + monkeypatch.setattr(streaming, "is_responses_protocol", lambda: True) + monkeypatch.setattr(streaming.ai_responses, "stream_round", fake_stream_round) + + events = [e async for e in streaming.stream_openai_round( + [{"role": "user", "content": "hi"}], [], temperature=0.3, timeout=5, + )] + assert events == [ + {"type": "text", "delta": "助手回复"}, + {"type": "round_end", "tool_calls": [], "finish_reason": "stop"}, + ] + + +@pytest.mark.asyncio +async def test_stream_round_awaits_async_close(monkeypatch): + """SDK 2.x 的 AsyncStream.close 是协程: 必须真正 await, 而非 to_thread 丢弃。""" + closed: list[bool] = [] + + class _ClosableStream: + def __init__(self, events): + self._inner = _fake_stream(*events) + + def __aiter__(self): + return self + + async def __anext__(self): + return await self._inner.__anext__() + + async def close(self): + closed.append(True) + + stream = _ClosableStream([_text_delta("OK"), _completed()]) + client, _ = _fake_client(stream=stream) + monkeypatch.setattr(ai_responses, "responses_client", lambda timeout: client) + monkeypatch.setattr(secrets_store, "load", lambda: {}) + + events = [e async for e in ai_responses.stream_round( + [{"role": "user", "content": "hi"}], [], temperature=None, timeout=5, + )] + assert closed == [True] + assert events[-1]["finish_reason"] == "stop" + + +def test_save_ai_settings_persists_protocol(monkeypatch): + saved: dict = {} + monkeypatch.setattr(settings_api.secrets_store, "save", lambda updates: saved.update(updates)) + monkeypatch.setattr(settings_api.secrets_store, "load", lambda: saved) + original = app_settings.ai_protocol + try: + result = settings_api.save_ai_settings(settings_api.AiSettingsIn( + provider="openai_compat", + base_url="https://example.com/v1", + api_key="sk-test", + model="gpt-x", + protocol="responses", + )) + assert saved["ai_protocol"] == "responses" + assert result["ai_protocol"] == "responses" + assert ai_provider.current_ai_protocol() == "responses" + finally: + app_settings.ai_protocol = original + + +def test_save_ai_settings_rejects_unknown_protocol(monkeypatch): + saved: dict = {} + monkeypatch.setattr(settings_api.secrets_store, "save", lambda updates: saved.update(updates)) + monkeypatch.setattr(settings_api.secrets_store, "load", lambda: saved) + + from fastapi import HTTPException + with pytest.raises(HTTPException): + settings_api.save_ai_settings(settings_api.AiSettingsIn( + provider="openai_compat", protocol="grpc", + )) + assert "ai_protocol" not in saved diff --git a/backend/tests/test_custom_assistant_sessions.py b/backend/tests/test_custom_assistant_sessions.py new file mode 100644 index 00000000..58745351 --- /dev/null +++ b/backend/tests/test_custom_assistant_sessions.py @@ -0,0 +1,97 @@ +"""AI 助手会话历史服务端存储 (跨设备同步) 的服务与 API 测试。 + +契约: GET 空结构、PUT 保存往返、结构裁剪 (无 id 会话剔除 / 消息截尾 200 / +会话上限 20)、损坏文件按空处理、API 挂载在 assistant 扩展路由下。 +""" +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from app.custom.assistant import sessions as assistant_sessions +from app.custom.assistant.routes import build_router + + +@pytest.fixture +def client(tmp_path: Path) -> TestClient: + app = FastAPI() + app.include_router(build_router()) + app.state.repo = SimpleNamespace(store=SimpleNamespace(data_dir=tmp_path)) + return TestClient(app) + + +def _session(sid: str, n_msgs: int = 1) -> dict: + return { + "id": sid, + "title": f"会话{sid}", + "createdAt": 1700000000000, + "messages": [ + {"id": f"m{i}", "role": "user", "content": f"hi {i}", "ts": i} + for i in range(n_msgs) + ], + } + + +def test_api_roundtrip_and_empty_state(client: TestClient) -> None: + assert client.get("/api/custom/assistant/sessions").json() == { + "sessions": [], + "activeId": "", + } + + r = client.put( + "/api/custom/assistant/sessions", + json={"sessions": [_session("s1"), _session("s2")], "activeId": "s2"}, + ) + assert r.status_code == 200 and r.json() == {"ok": True, "sessions": 2} + + got = client.get("/api/custom/assistant/sessions").json() + assert [s["id"] for s in got["sessions"]] == ["s1", "s2"] + assert got["activeId"] == "s2" + + +def test_sanitize_trims_and_drops(client: TestClient, tmp_path: Path) -> None: + # 无 id/非字符串 id 的会话剔除 + bad = [{"title": "no id"}, {"id": 123, "messages": []}] + # 消息超 200 截尾 + long = _session("long", n_msgs=250) + # 会话超 20 截断 (服务层: API 层 max_length=20 会先拒绝更大的载荷) + many = [_session(f"s{i}") for i in range(25)] + + n = assistant_sessions.save_sessions( + tmp_path, {"sessions": [*bad, long, *many], "activeId": "s0"} + ) + assert n == assistant_sessions.MAX_SESSIONS + + got = assistant_sessions.load_sessions(tmp_path) + ids = [s["id"] for s in got["sessions"]] + assert len(ids) == 20 + assert "long" in ids # 输入顺序靠前, 保留 + assert len(next(s for s in got["sessions"] if s["id"] == "long")["messages"]) == 200 + + # API 层: 结构非法的会话静默剔除, 合法照常保存 + r = client.put( + "/api/custom/assistant/sessions", + json={"sessions": [*bad, _session("ok1")], "activeId": "ok1"}, + ) + assert r.status_code == 200 and r.json() == {"ok": True, "sessions": 1} + ids2 = [s["id"] for s in client.get("/api/custom/assistant/sessions").json()["sessions"]] + assert ids2 == ["ok1"] + + +def test_corrupted_file_treated_as_empty(tmp_path: Path) -> None: + p = tmp_path / "user_data" / "assistant_sessions.json" + p.parent.mkdir(parents=True) + p.write_text("{not json", encoding="utf-8") + assert assistant_sessions.load_sessions(tmp_path) == {"sessions": [], "activeId": ""} + + +def test_put_rejects_oversized_session_list(client: TestClient) -> None: + r = client.put( + "/api/custom/assistant/sessions", + json={"sessions": [_session(f"s{i}") for i in range(21)]}, + ) + assert r.status_code == 422 # 超过 max_length=20, 防御异常客户端 diff --git a/backend/tests/test_emotion_score.py b/backend/tests/test_emotion_score.py new file mode 100644 index 00000000..f61de9fc --- /dev/null +++ b/backend/tests/test_emotion_score.py @@ -0,0 +1,200 @@ +"""情绪评分口径 — 默认口径还原 + 配置归一化契约。 + +「默认口径必须与历史硬编码算法逐位一致」是这里的核心约束: 情绪分是看板与 +复盘快照的对外数字, 改动计算模块不能悄悄挪动它。原算法已从 +market_overview_builder 移除, 故本文件内联一份等价实现做对照。 +""" +from __future__ import annotations + +import math +import random + +from app.services.emotion_score import ( + DEFAULT_CONFIG, + DIMENSION_ORDER, + compute, + dimension_value, + label_of, + merge_config, + public_config, + score_of, +) + + +def _legacy_score(value: float, low: float, high: float) -> int: + """改造前 market_overview_builder._score 的逐字复制。""" + if high <= low: + return 50 + if not math.isfinite(value): + return 50 + return max(0, min(100, round((value - low) / (high - low) * 100))) + + +def _legacy_radar_and_score(m: dict) -> tuple[list[dict], int]: + """改造前看板情绪雷达的逐字复制 (含 mainline 预计算那一版)。""" + radar = [ + {"key": "index", "label": "指数", "value": _legacy_score(m["avg_index_pct"], -2.5, 2.5)}, + {"key": "profit", "label": "赚钱", "value": round( + _legacy_score(m["up_pct"], 20, 80) * 0.45 + + _legacy_score(m["avg_pct"], -0.02, 0.02) * 0.25 + + _legacy_score(m["median_pct"], -0.02, 0.02) * 0.20 + + _legacy_score(m["strong_diff_pct"], -8, 8) * 0.10)}, + {"key": "money", "label": "量能", "value": round( + _legacy_score(m["avg_vol_ratio"], 0.6, 1.8) * 0.70 + + _legacy_score(m["high_vol_pct"], 2, 12) * 0.30)}, + {"key": "speculation", "label": "投机", "value": round( + _legacy_score(m["limit_up"], 5, 90) * 0.25 + + _legacy_score(m["seal_rate"], 30, 85) * 0.35 + + _legacy_score(m["max_boards"], 1, 8) * 0.25 + + _legacy_score(m["tier2_count"], 0, 30) * 0.15)}, + {"key": "resilience", "label": "抗跌", "value": 100 - round( + _legacy_score(m["down_pct"], 20, 80) * 0.55 + + _legacy_score(m["strong_down_pct"], 1, 12) * 0.45)}, + {"key": "mainline", "label": "主线", "value": m["mainline_score"]}, + ] + return radar, round(sum(r["value"] for r in radar) / len(radar)) + + +def _sample(rng: random.Random) -> dict: + m = { + "avg_index_pct": rng.uniform(-6, 6), "up_pct": rng.uniform(0, 100), + "avg_pct": rng.uniform(-0.05, 0.05), "median_pct": rng.uniform(-0.05, 0.05), + "strong_diff_pct": rng.uniform(-20, 20), "avg_vol_ratio": rng.uniform(0.3, 3), + "high_vol_pct": rng.uniform(0, 40), "limit_up": rng.randint(0, 200), + "seal_rate": rng.uniform(0, 100), "max_boards": rng.randint(0, 12), + "tier2_count": rng.randint(0, 60), "down_pct": rng.uniform(0, 100), + "strong_down_pct": rng.uniform(0, 25), "mainline_avg": rng.uniform(-0.03, 0.08), + "mainline_cover_pct": rng.uniform(0, 30), + } + m["mainline_score"] = round( + _legacy_score(m["mainline_avg"], -0.005, 0.03) * 0.65 + + _legacy_score(m["mainline_cover_pct"], 1, 12) * 0.35) + return m + + +def test_default_matches_legacy_bit_for_bit() -> None: + """出厂口径与历史硬编码算法逐位一致 (随机 5000 组)。 + + 取整顺序在国内很敏感: 抗跌维度必须「先取整再取反」(100 - round(x)), + 写成 round(100 - x) 会在 x 的小数部分恰为 .5 时差 1 分。 + """ + rng = random.Random(7) + for _ in range(5000): + m = _sample(rng) + legacy_radar, legacy_score = _legacy_radar_and_score(m) + res = compute(m, None) + assert res["radar"] == legacy_radar, f"雷达不一致: {m}" + assert res["score"] == legacy_score, f"总分不一致: {m}" + + +def test_label_mapping_matches_legacy_over_full_range() -> None: + """档位映射覆盖 0..100 全整数区间, 与历史阈值一致。""" + def legacy_label(score: int) -> str: + if score >= 70: + return "强势" + if score >= 55: + return "偏暖" + if score >= 45: + return "震荡" + if score >= 30: + return "偏冷" + return "冰点" + + th = DEFAULT_CONFIG["thresholds"] + for score in range(101): + assert label_of(score, th) == legacy_label(score), f"{score} 档位不一致" + + +def test_score_of_handles_non_finite() -> None: + """非有限输入取中性 50, 不得抛异常打断看板计算。""" + for bad in (float("nan"), float("inf"), None, "abc"): + assert score_of(bad, 0, 1) == 50 + assert score_of(0.5, 0, 1) == 50 + assert score_of(5, 0, 1) == 100 # 超出上界夹到 100 + assert score_of(-5, 0, 1) == 0 # 低于下界夹到 0 + + +def test_merge_config_fills_defaults_and_ignores_unknown() -> None: + """部分配置 → 其余字段补默认; 未知 key 丢弃 (不因脏配置崩)。""" + merged = merge_config({ + "dimensions": { + "index": {"weight": 3.0, "terms": [{"key": "avg_index_pct", "low": -5}]}, + "no_such_dim": {"weight": 9}, + }, + "thresholds": {"strong": 80}, + "unknown_top": 1, + }) + assert merged["dimensions"]["index"]["weight"] == 3.0 + assert merged["dimensions"]["index"]["terms"][0]["low"] == -5 + assert merged["dimensions"]["index"]["terms"][0]["high"] == 2.5 # 未提及 → 默认 + assert "no_such_dim" not in merged["dimensions"] + assert "unknown_top" not in merged + assert merged["thresholds"]["strong"] == 80 + assert merged["thresholds"]["warm"] == DEFAULT_CONFIG["thresholds"]["warm"] + + +def test_thresholds_normalized_to_descending() -> None: + """用户把阈值填乱序 → 自动重排为降序, 保证档位区间非空。""" + merged = merge_config({"thresholds": {"strong": 20, "warm": 90, "neutral": 50, "cold": 10}}) + th = merged["thresholds"] + assert th["strong"] >= th["warm"] >= th["neutral"] >= th["cold"] + + +def test_thresholds_clamped_to_range() -> None: + """阈值越界 (负数/超 100) 夹回 0..100。""" + merged = merge_config({"thresholds": {"strong": 999, "cold": -50}}) + th = merged["thresholds"] + assert 0 <= th["cold"] <= 100 + assert 0 <= th["strong"] <= 100 + + +def test_dimension_weight_affects_score() -> None: + """改维度权重必须改变总分 (否则配置形同虚设)。""" + m = _sample(random.Random(1)) + base = compute(m, None)["score"] + heavy = merge_config({"dimensions": {"index": {"weight": 10.0}}}) + assert compute(m, heavy)["score"] != base + + +def test_term_weight_within_dimension_is_normalized() -> None: + """子项权重和不为 1 时按比例归一化, 不因总和变化而畸变。 + + 把「指数」维度 (单子项) 的权重从 1 改成 7, 该维度得分应不变 —— 单子项 + 归一化后权重的绝对大小不影响自身得分。 + """ + dim = merge_config(None)["dimensions"]["index"] + before = dimension_value(dim, {"avg_index_pct": 1.0}) + scaled = merge_config({"dimensions": {"index": {"terms": [ + {"key": "avg_index_pct", "weight": 7.0}]}}})["dimensions"]["index"] + assert dimension_value(scaled, {"avg_index_pct": 1.0}) == before + + +def test_missing_metric_takes_neutral() -> None: + """指标缺失 → 该维度取中性 50 (与历史「无主线数据取 50」一致)。""" + dim = merge_config(None)["dimensions"]["mainline"] + assert dimension_value(dim, {}) == 50 + + +def test_public_config_shape() -> None: + """配置接口返回结构稳定 (前端表单完全依赖它渲染)。""" + payload = public_config(None) + assert payload["dimension_order"] == list(DIMENSION_ORDER) + assert set(payload["config"]) == {"dimensions", "thresholds"} + assert payload["defaults"] == DEFAULT_CONFIG + names = [l["name"] for l in payload["labels"]] + assert "强势" in names and "冰点" in names + # 每个维度都带 label / weight / terms, 每个 term 带完整六字段 + for key in DIMENSION_ORDER: + dim = payload["config"]["dimensions"][key] + assert dim["label"] and "weight" in dim and dim["terms"] + for term in dim["terms"]: + assert set(term) == {"key", "label", "weight", "low", "high", "unit"} + + +def test_compute_returns_dimension_details() -> None: + """compute 额外返回 dimensions (带权重与 invert), 供配置面板展示。""" + res = compute(_sample(random.Random(3)), None) + assert len(res["dimensions"]) == len(DIMENSION_ORDER) + resilience = next(d for d in res["dimensions"] if d["key"] == "resilience") + assert resilience["invert"] is True + assert res["label"] in ("强势", "偏暖", "震荡", "偏冷", "冰点") diff --git a/backend/tests/test_ext_fanout.py b/backend/tests/test_ext_fanout.py new file mode 100644 index 00000000..d620e05c --- /dev/null +++ b/backend/tests/test_ext_fanout.py @@ -0,0 +1,293 @@ +"""通用两级拉取 (ext_fanout) 单测 — 目录 → 逐条目明细 → 反转为「成员 → 类别」。 + +全程 httpx.MockTransport, 不出网。覆盖: 信封解包报错、反转聚合与目录顺序、 +单条目失败容忍(重试后跳过)、全部条目失败报错、必填项校验、 +以及端到端落盘 (config.json + parquet, 走 fetch_and_ingest 通用路径)。 +""" +from __future__ import annotations + +import asyncio +from pathlib import Path + +import httpx +import polars as pl +import pytest + +from app.services.ext_data import ( + ExtConfig, + ExtConfigStore, + ExtField, + PullConfig, + rows_to_parquet, +) +from app.services.ext_fanout import clean_label, fetch_fanout_rows + +_CATALOG = "/api/a-share-index/catalog/ths-index-list" +_CONSTITUENTS = "/api/a-share-index/constituents/ths-stock-list" +_AUTH_HEADER = {"X-api-key": "test-key"} + + +def _ok(items: list[dict]) -> httpx.Response: + return httpx.Response( + 200, json={"code": 0, "message": "success", "data": {"item": items}} + ) + + +def _mock_transport( + catalog_by_tag: dict[str, list[dict]], + constituents_by_code: dict[str, list[dict]], + fail_codes: tuple[str, ...] = (), +) -> httpx.MockTransport: + def handler(request: httpx.Request) -> httpx.Response: + if request.url.path == _CATALOG: + tag = request.url.params.get("tag", "cn_concept") + return _ok(catalog_by_tag.get(tag, [])) + if request.url.path == _CONSTITUENTS: + code = request.url.params.get("thscode", "") + if code in fail_codes: + raise httpx.ConnectError(f"injected failure: {code}") + return _ok(constituents_by_code.get(code, [])) + return httpx.Response(404, text="not found") + + return httpx.MockTransport(handler) + + +_CATALOG_CN_CONCEPT = [ + {"thscode": "B1.TI", "name": "白酒概念"}, + {"thscode": "B2.TI", "name": "新能源汽车"}, +] +_CONSTITUENTS_BY_CODE = { + "B1.TI": [ + {"thscode": "600519.SH", "ticker": "600519", "name": "贵州茅台"}, + {"thscode": "000858.SZ", "ticker": "000858", "name": "五粮液"}, + ], + "B2.TI": [ + {"thscode": "600519.SH", "ticker": "600519", "name": "贵州茅台"}, + {"thscode": "300750.SZ", "ticker": "300750", "name": "宁德时代"}, + ], +} + + +def _concept_config(url_tag: str = "cn_concept", **fanout_overrides) -> ExtConfig: + """测试用概念表配置: 结构与出厂预设同构, 便于断言 schema。""" + fanout = { + "url": f"https://fuyao.aicubes.cn{_CONSTITUENTS}", + "param": "thscode", + "id_field": "thscode", + "label_field": "name", + "items_path": "data.item", + "member_id_field": "thscode", + "member_label_field": "name", + "symbol_col": "股票代码", + "name_col": "股票简称", + "value_col": "所属概念", + "separator": ";", + "concurrency": 8, + "retries": 1, + } + fanout.update(fanout_overrides) + return ExtConfig( + id="ext_gn_ths", + label="扩展概念", + mode="snapshot", + fields=[ + ExtField("symbol", "string", "标的代码"), + ExtField("code", "string", "代码"), + ExtField("股票代码", "string", "股票代码"), + ExtField("股票简称", "string", "股票简称"), + ExtField("所属概念", "string", "所属概念"), + ], + symbol_map={"type": "mapped", "col": "股票代码"}, + code_map={"type": "computed", "from": "symbol", "method": "strip_exchange"}, + pull=PullConfig( + url=f"https://fuyao.aicubes.cn{_CATALOG}?tag={url_tag}", + response_path="data.item", + auth={"type": "header", "header": "X-api-key"}, + fanout=fanout, + ), + ) + + +@pytest.fixture +def mock_http(monkeypatch: pytest.MonkeyPatch): + """把 ext_pull 的出站请求替换为 MockTransport (鉴权与标识头口径保持不变)。 + + 同时注入测试 Key: _apply_auth 在缺凭据时 fail-closed 报错, 这是生产要的 + 行为, 测 fanout 逻辑时需补上凭据。 + """ + import app.services.ext_pull as ext_pull + import app.services.ext_data as ext_data + + monkeypatch.setattr( + ext_data, "get_ext_api_key", lambda config_id: "test-key" + ) + + def install(transport: httpx.MockTransport): + real_request = ext_pull._request_json + + async def patched(pull, config_id, day=None, extra_params=None, *, url_override=None): + # 复用真实实现的 URL/鉴权拼装, 只把 transport 换成 Mock + from app.services.ext_pull import _apply_auth, _append_query_params, _with_date_param + url = url_override or ( + _with_date_param(pull.url, pull.date_param, day, pull.date_format) if day else pull.url + ) + if extra_params: + url = _append_query_params(url, extra_params) + headers = {"User-Agent": "test", "X-TSP-Client": "tick-stock-panel"} + url = _apply_auth(config_id, pull.auth, url, headers) + async with httpx.AsyncClient(timeout=30, transport=transport) as client: + resp = await client.request(pull.method.upper(), url, headers=headers) + resp.raise_for_status() + return resp.json() + + monkeypatch.setattr(ext_pull, "_request_json", patched) + return real_request + + return install + + +# ---- 值清洗 ---- + + +def test_clean_label_drops_missing_value_placeholders() -> None: + for placeholder in (None, float("nan"), "nan", "None", " null ", ""): + assert clean_label(placeholder) == "" + assert clean_label(" 银行 ") == "银行" + + +# ---- 必填项校验 ---- + + +def test_fanout_missing_required_keys_raises() -> None: + config = _concept_config() + config.pull.fanout.pop("member_id_field") + with pytest.raises(ValueError, match="缺少必填项"): + asyncio.run(fetch_fanout_rows(config)) + + +# ---- 信封解包 ---- + + +def test_fanout_envelope_error_raises(mock_http) -> None: + mock_http( + httpx.MockTransport( + lambda r: httpx.Response(200, json={"code": 2001, "message": "invalid key"}) + ) + ) + with pytest.raises(ValueError, match="code=2001"): + asyncio.run(fetch_fanout_rows(_concept_config())) + + +# ---- 反转聚合 ---- + + +def test_fanout_inverts_by_catalog_order(mock_http) -> None: + mock_http(_mock_transport({"cn_concept": _CATALOG_CN_CONCEPT}, _CONSTITUENTS_BY_CODE)) + + rows = asyncio.run(fetch_fanout_rows(_concept_config())) + by_symbol = {r["股票代码"]: r for r in rows} + assert len(by_symbol) == 3 + # 类别名跟随目录顺序拼接 (B1 在前 B2 在后), 与请求完成顺序无关 + assert by_symbol["600519.SH"]["所属概念"] == "白酒概念;新能源汽车" + assert by_symbol["600519.SH"]["股票简称"] == "贵州茅台" + assert by_symbol["000858.SZ"]["所属概念"] == "白酒概念" + assert by_symbol["300750.SZ"]["所属概念"] == "新能源汽车" + + +def test_fanout_tolerates_entry_failure(mock_http) -> None: + """单条目重试后仍失败 → 跳过该条目, 其余正常返回。""" + mock_http( + _mock_transport( + {"cn_concept": _CATALOG_CN_CONCEPT}, + _CONSTITUENTS_BY_CODE, + fail_codes=("B2.TI",), + ) + ) + rows = asyncio.run(fetch_fanout_rows(_concept_config())) + by_symbol = {r["股票代码"]: r for r in rows} + assert set(by_symbol) == {"600519.SH", "000858.SZ"} # B2 成员被跳过 + assert by_symbol["600519.SH"]["所属概念"] == "白酒概念" + + +def test_fanout_all_entries_failed_raises(mock_http) -> None: + mock_http( + _mock_transport( + {"cn_concept": _CATALOG_CN_CONCEPT}, + _CONSTITUENTS_BY_CODE, + fail_codes=("B1.TI", "B2.TI"), + ) + ) + with pytest.raises(ValueError, match="全部条目第二级拉取失败"): + asyncio.run(fetch_fanout_rows(_concept_config())) + + +def test_fanout_empty_catalog_raises(mock_http) -> None: + mock_http(_mock_transport({"cn_concept": []}, {})) + with pytest.raises(ValueError, match="第一级返回 0 个可用条目"): + asyncio.run(fetch_fanout_rows(_concept_config())) + + +def test_fanout_dedupes_repeated_labels(mock_http) -> None: + """同一标的在同一类别下重复出现 → 只记一次。""" + mock_http( + _mock_transport( + {"cn_concept": [{"thscode": "B1.TI", "name": "白酒概念"}]}, + { + "B1.TI": [ + {"thscode": "600519.SH", "name": "贵州茅台"}, + {"thscode": "600519.SH", "name": "贵州茅台"}, + ] + }, + ) + ) + rows = asyncio.run(fetch_fanout_rows(_concept_config())) + assert rows[0]["所属概念"] == "白酒概念" + + +# ---- 端到端落盘 (通用写入链) ---- + + +def test_fanout_rows_persist_via_generic_writer( + tmp_path: Path, mock_http +) -> None: + mock_http(_mock_transport({"cn_concept": _CATALOG_CN_CONCEPT}, _CONSTITUENTS_BY_CODE)) + config = _concept_config() + + rows = asyncio.run(fetch_fanout_rows(config)) + n = rows_to_parquet(rows, config, tmp_path) + assert n == 3 + + out = tmp_path / "ext_data" / "ext_gn_ths" / "part.parquet" + assert out.exists(), "快照 parquet 未落盘" + + df = pl.read_parquet(out) + row = df.filter(pl.col("symbol") == "600519.SH").to_dicts()[0] + assert row["所属概念"] == "白酒概念;新能源汽车" + assert row["股票简称"] == "贵州茅台" + assert row["code"] == "600519" + + +def test_fanout_industry_separator(mock_http) -> None: + """行业配方用 "-" 拼接层级, 概念用 ";": 同一引擎, 差别只在配方。""" + mock_http( + _mock_transport( + {"industry": [{"thscode": "H1.TI", "name": "金融"}]}, + {"H1.TI": [{"thscode": "600519.SH", "name": "贵州茅台"}]}, + ) + ) + config = _concept_config( + url_tag="industry", value_col="所属同花顺行业", separator="-" + ) + rows = asyncio.run(fetch_fanout_rows(config)) + assert rows[0]["所属同花顺行业"] == "金融" + + +def test_pull_config_roundtrip_preserves_fanout(tmp_path: Path) -> None: + """fanout 随 config.json 持久化与读回, 不丢字段。""" + config = _concept_config() + store = ExtConfigStore(tmp_path) + store.upsert(config) + + loaded = ExtConfigStore(tmp_path).get("ext_gn_ths") + assert loaded is not None and loaded.pull is not None + assert loaded.pull.fanout == config.pull.fanout diff --git a/backend/tests/test_ext_pages.py b/backend/tests/test_ext_pages.py new file mode 100644 index 00000000..6e0350bf --- /dev/null +++ b/backend/tests/test_ext_pages.py @@ -0,0 +1,138 @@ +"""运行时页面包 API 测试 — 上传校验/生命周期/bundle 分发。 + +TestClient 最小 app 风格(同 test_ext_market_level): 不起完整后端, +app.state.repo 只提供 store.data_dir。 +""" +from __future__ import annotations + +import io +from pathlib import Path +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from starlette.datastructures import UploadFile + +from app.api.ext_pages import router + + +@pytest.fixture +def client(tmp_path: Path) -> TestClient: + app = FastAPI() + app.include_router(router) + app.state.repo = SimpleNamespace(store=SimpleNamespace(data_dir=tmp_path)) + return TestClient(app) + + +def _bundle_js() -> bytes: + # 最小合法 ES 模块产物: default 导出 FrontendExtension 形状 + return ( + "const extension = { id: 'demo.page', apiVersion: 1, " + "routes: [{ id: 'demo-page', path: '/demo-page', component: () => null }], " + "navigation: [{ id: 'demo-page', routeId: 'demo-page', label: '演示页', icon: null }] };\n" + "export default extension;\n" + ).encode() + + +def _upload(client: TestClient, pack_id: str, data: bytes, filename: str = "extension.js"): + return client.post( + "/api/ext-pages/upload", + data={"id": pack_id}, + files={"file": (filename, io.BytesIO(data), "text/javascript")}, + ) + + +def test_upload_and_lifecycle(client: TestClient, tmp_path: Path): + r = _upload(client, "demo_page", _bundle_js()) + assert r.status_code == 200, r.text + meta = r.json() + assert meta["id"] == "demo_page" + assert meta["enabled"] is False # 上传后未验证, 默认停用 + assert meta["size"] == len(_bundle_js()) and len(meta["sha256"]) == 64 + assert (tmp_path / "ext_pages" / "demo_page" / "bundle.js").exists() + + # 清单回填 → 启用 + r = client.put( + "/api/ext-pages/demo_page/manifest", + json={"label": "演示页", "path": "/demo-page", "error": ""}, + ) + assert r.status_code == 200 and r.json()["enabled"] is True + + # 列表 + r = client.get("/api/ext-pages") + assert [m["id"] for m in r.json()] == ["demo_page"] + + # 分发: MIME + no-store + ETag + r = client.get("/api/ext-pages/demo_page/bundle.js") + assert r.status_code == 200 + assert r.headers["content-type"].startswith("text/javascript") + assert r.headers["cache-control"] == "no-store" + assert "etag" in r.headers + assert b"export default" in r.content + + # 停用 / 删除 + assert client.put("/api/ext-pages/demo_page/enabled", json={"enabled": False}).json()["enabled"] is False + assert client.delete("/api/ext-pages/demo_page").json() == {"ok": True} + assert client.get("/api/ext-pages/demo_page/bundle.js").status_code == 404 + + +def test_upload_rejects_bad_suffix_and_id(client: TestClient): + assert _upload(client, "x", b"a,b", "data.csv").status_code == 400 + # 路径穿越/非法 ID 一律拒绝(空 ID 由 FastAPI 表单校验 422), 不得落盘 + for bad in ("../evil", "a/b", "bad id", "x" * 0, ""): + assert _upload(client, bad, _bundle_js()).status_code in (400, 422) + + +def test_upload_size_cap_413(client: TestClient): + big = b"//" + b"x" * (5 * 1024 * 1024 + 10) + assert _upload(client, "big", big).status_code == 413 + + +def test_upload_empty_file_400(client: TestClient): + assert _upload(client, "empty", b"").status_code == 400 + + +def test_upload_overwrites_same_id(client: TestClient): + _upload(client, "same", _bundle_js()) + v2 = _bundle_js() + b"// v2\n" + _upload(client, "same", v2) + r = client.get("/api/ext-pages/same/bundle.js") + assert r.content == v2 # 覆盖式更新 + + +def test_manifest_error_disables(client: TestClient): + _upload(client, "bad", _bundle_js()) + r = client.put( + "/api/ext-pages/bad/manifest", + json={"label": "", "path": "", "error": "apiVersion 不匹配"}, + ) + assert r.json()["enabled"] is False and r.json()["error"] == "apiVersion 不匹配" + + +def test_missing_pack_404(client: TestClient): + assert client.get("/api/ext-pages/none/bundle.js").status_code == 404 + assert client.put("/api/ext-pages/none/enabled", json={"enabled": True}).status_code == 404 + + +def test_write_bundle_capped_stops_reading(tmp_path: Path): + """越限立即停止读取: 计数流在 413 后不应被继续消费。""" + import asyncio + + from app.api.ext_pages import _write_bundle_capped + + class CountingStream: + def __init__(self, data: bytes): + self._buf = io.BytesIO(data) + self.reads = 0 + + # UploadFile 经 run_in_threadpool 同步调用 file.read, 这里保持同步签名 + def read(self, n: int = -1) -> bytes: + self.reads += 1 + return self._buf.read(n) + + stream = CountingStream(b"x" * 200) + with pytest.raises(Exception) as exc: + asyncio.run(_write_bundle_capped(UploadFile(stream), tmp_path / "dummy.bin", max_bytes=64)) + assert getattr(exc.value, "status_code", None) == 413 + assert stream.reads == 1 # 超限块读出后不再继续读 diff --git a/backend/tests/test_ext_preset_dimension_values.py b/backend/tests/test_ext_preset_dimension_values.py index 82133a31..a18ccf20 100644 --- a/backend/tests/test_ext_preset_dimension_values.py +++ b/backend/tests/test_ext_preset_dimension_values.py @@ -1,28 +1,12 @@ +"""维度值清洗口径。 + +类别名里的 NaN/None 占位值清洗已随 fanout 迁移到 ext_fanout.clean_label +(见 test_ext_fanout.py); 此处保留消费侧的维度值清洗测试。 +""" from __future__ import annotations -from app.services.ext_presets import _flatten_concept_rows, _flatten_industry_rows from app.services.market_overview_builder import _dimension_values -def test_concept_flatten_drops_missing_value_placeholders(): - rows = _flatten_concept_rows([{ - "symbol": "600000.SH", - "name": "浦发银行", - "concepts": ["银行", "nan", None, float("nan"), " null ", "金融科技"], - }]) - - assert rows[0]["所属概念"] == "银行;金融科技" - - -def test_industry_flatten_drops_missing_value_placeholders(): - rows = _flatten_industry_rows([{ - "symbol": "600000.SH", - "name": "浦发银行", - "industries": ["金融", "None", "银行"], - }]) - - assert rows[0]["所属同花顺行业"] == "金融-银行" - - def test_overview_dimension_values_ignore_legacy_nan_group(): assert _dimension_values("人工智能;nan;芯片;NULL") == ["人工智能", "芯片"] diff --git a/backend/tests/test_ext_presets_startup.py b/backend/tests/test_ext_presets_startup.py index 49a142a5..7da5cc3f 100644 --- a/backend/tests/test_ext_presets_startup.py +++ b/backend/tests/test_ext_presets_startup.py @@ -1,20 +1,22 @@ -"""内置概念/行业 preset 启动不得自动拉取 (#199)。 +"""内置概念/行业 preset 启动契约。 -ensure_builtin_presets 的契约是「只创建 config.json, 不拉取数据, 等待用户手动获取」; -但 preset 出厂 PullConfig.enabled=True 会让 PullScheduler.refresh 在启动时立即调度 -_run_loop 并马上执行一次网络拉取 (启用后立即执行一次), 与契约矛盾。 +出厂即带【完整可用的拉取配方】(URL / 鉴权 / 两级请求 fanout), 不是空壳 — +「数据」页展开表 → 拉取 就能看到一份真实可跑的配方, 供接入自有源照抄。 -回归断言: 内置 preset 出厂 pull.enabled 必须为 False —— scheduler 的 enabled 过滤 -(ext_pull.refresh) 会因此跳过它们; 手动获取走 fetch_preset 独立路径, 不经过本开关 -(该路径行为由 test_ext_preset_dimension_values / test_ext_pull_refresh 覆盖)。 +ensure_builtin_presets 的契约: + - 只创建/补齐 config.json, 不拉取数据 (数据由调度器或用户手动触发) + - 空壳 / 历代出厂配方 → 补齐/替换为当前配方 + - 用户自定义过 pull (URL 非出厂值) 或 description → 一律不碰 + - 失败只记 warning, 不阻断启动 """ from __future__ import annotations import asyncio from pathlib import Path -from app.services.ext_data import ExtConfigStore +from app.services.ext_data import ExtConfigStore, PullConfig from app.services.ext_presets import ( + _RETIRED_FACTORY_PULLS, _concept_preset, _industry_preset, ensure_builtin_presets, @@ -23,37 +25,117 @@ from app.services.ext_presets import ( _PRESET_IDS = ("ext_gn_ths", "ext_hy_ths") -def test_builtin_presets_ship_disabled() -> None: - """出厂 preset 的 pull.enabled 必须为 False, 否则启动即网络拉取 (#199)。""" +def test_builtin_presets_ship_with_complete_pull() -> None: + """出厂 preset 自带完整拉取配方: URL / 鉴权 / 两级请求一应俱全。""" for preset in (_concept_preset(), _industry_preset()): - assert preset.pull is not None - assert preset.pull.url, "禁用归禁用, 手动获取仍需 url 配方" - assert preset.pull.enabled is False, f"{preset.id} 启动即自动拉取, 违反启动契约" + pull = preset.pull + assert pull is not None, f"{preset.id} 出厂未带拉取配方" + assert pull.url.startswith("https://"), f"{preset.id} 缺少第一级 URL" + assert pull.response_path, f"{preset.id} 缺少响应路径" + assert pull.auth and pull.auth.get("type") == "header", f"{preset.id} 缺少鉴权配置" + # 凭据复用 fuyao 行情插件, 用户只需在一处填 Key + assert pull.auth.get("key_ref") == "fuyao_api_key", f"{preset.id} 未复用 fuyao 凭据" + + fanout = pull.fanout + assert fanout, f"{preset.id} 缺少两级拉取配方" + assert fanout["url"].startswith("https://"), f"{preset.id} fanout 缺少第二级 URL" + for key in ("param", "id_field", "member_id_field", + "symbol_col", "name_col", "value_col"): + assert fanout.get(key), f"{preset.id} fanout 缺少 {key}" + assert fanout.get("separator"), f"{preset.id} fanout 缺少拼接符" -def test_ensure_builtin_presets_writes_disabled_configs(tmp_path: Path) -> None: - """全新数据目录: 启动只落禁用的 pull 配置, scheduler 扫描后无任务可建。""" +def test_ensure_builtin_presets_writes_complete_configs(tmp_path: Path) -> None: + """全新数据目录: 启动即落完整配方, 两张表都带 fanout。""" asyncio.run(ensure_builtin_presets(tmp_path)) store = ExtConfigStore(tmp_path) for cid in _PRESET_IDS: config = store.get(cid) assert config is not None, f"{cid} 配置未创建" - assert config.pull is not None - assert config.pull.enabled is False + assert config.pull is not None, f"{cid} 出厂配方未写入" + assert config.pull.url, f"{cid} 第一级 URL 为空" + assert config.pull.fanout, f"{cid} 两级配方未写入" -def test_ensure_builtin_presets_keeps_existing_user_config(tmp_path: Path) -> None: - """老用户/已存在的配置一律不动: 即使保留 enabled=True 也不被覆盖。""" +def test_ensure_builtin_presets_fills_empty_shell(tmp_path: Path) -> None: + """空壳配置 (无 pull, #199 之后的形态) → 补齐为完整配方。""" asyncio.run(ensure_builtin_presets(tmp_path)) store = ExtConfigStore(tmp_path) config = store.get("ext_gn_ths") - assert config is not None and config.pull is not None - config.pull.enabled = True + assert config is not None + config.pull = None store.upsert(config) asyncio.run(ensure_builtin_presets(tmp_path)) refreshed = ExtConfigStore(tmp_path).get("ext_gn_ths") assert refreshed is not None and refreshed.pull is not None - assert refreshed.pull.enabled is True, "已存在配置被静默改写, 违反「绝不覆盖」原则" + assert refreshed.pull.fanout, "空壳配置未被补齐为完整配方" + + +def test_ensure_builtin_presets_keeps_custom_user_pull(tmp_path: Path) -> None: + """用户自定义过 pull (URL 不等于历代出厂值) → 一律不动, 即使 enabled=True。""" + asyncio.run(ensure_builtin_presets(tmp_path)) + store = ExtConfigStore(tmp_path) + config = store.get("ext_gn_ths") + assert config is not None + + config.pull = PullConfig( + url="https://example.com/my-own-concepts", enabled=True + ) + store.upsert(config) + + asyncio.run(ensure_builtin_presets(tmp_path)) + + refreshed = ExtConfigStore(tmp_path).get("ext_gn_ths") + assert refreshed is not None and refreshed.pull is not None + assert refreshed.pull.url == "https://example.com/my-own-concepts" + assert refreshed.pull.enabled is True, "自定义拉取配置被静默改写, 违反「绝不覆盖」原则" + + +def test_ensure_builtin_presets_replaces_retired_factory_pulls(tmp_path: Path) -> None: + """pull.url 命中历代废弃出厂值 → 替换为带 fanout 的当前配方。""" + asyncio.run(ensure_builtin_presets(tmp_path)) + store = ExtConfigStore(tmp_path) + + for cid in _PRESET_IDS: + config = store.get(cid) + assert config is not None + for retired in _RETIRED_FACTORY_PULLS[cid]: + config.pull = PullConfig(url=retired["url"], enabled=True) + store.upsert(config) + + asyncio.run(ensure_builtin_presets(tmp_path)) + + refreshed = ExtConfigStore(tmp_path).get(cid) + assert refreshed is not None and refreshed.pull is not None + assert refreshed.pull.fanout, ( + f"{cid} 旧出厂配方 {retired['url']} 未替换为带 fanout 的当前配方" + ) + + +def test_ensure_builtin_presets_syncs_retired_factory_description(tmp_path: Path) -> None: + """description 命中出厂前缀 → 同步为当前文案; 用户自定义描述不动。""" + asyncio.run(ensure_builtin_presets(tmp_path)) + store = ExtConfigStore(tmp_path) + + config = store.get("ext_gn_ths") + assert config is not None + old_factory_desc = "同花顺概念分类 (首次启动自动拉取, 可在扩展数据页手动更新)" + config.description = old_factory_desc + store.upsert(config) + + asyncio.run(ensure_builtin_presets(tmp_path)) + + refreshed = ExtConfigStore(tmp_path).get("ext_gn_ths") + assert refreshed is not None + assert refreshed.description == _concept_preset().description, "旧出厂文案未同步" + + # 用户自定义描述 (非出厂前缀) → 不动 + refreshed.description = "我自己的概念表, 别动" + store.upsert(refreshed) + asyncio.run(ensure_builtin_presets(tmp_path)) + final = ExtConfigStore(tmp_path).get("ext_gn_ths") + assert final is not None + assert final.description == "我自己的概念表, 别动" diff --git a/backend/tests/test_packaging_spec.py b/backend/tests/test_packaging_spec.py index 2f867135..2f80a05f 100644 --- a/backend/tests/test_packaging_spec.py +++ b/backend/tests/test_packaging_spec.py @@ -29,3 +29,11 @@ def test_spec_declares_builtin_plugin_fuyao(): """fuyao 插件双声明不被回归 (同类动态发现问题, 见 spec 内注释)。""" assert "app.plugins.fuyao.provider" in SPEC_CODE assert "app/plugins/fuyao" in SPEC_CODE + + +def test_spec_declares_builtin_plugin_tdx(): + """tdx 插件双声明不被回归 (与 fuyao 同一处理方式; probe 子进程入口也要可导入)。""" + assert "app.plugins.tdxquant.provider" in SPEC_CODE + assert "app.plugins.tdxquant.client" in SPEC_CODE + assert "app.plugins.tdxquant.probe" in SPEC_CODE + assert '"plugins" / "tdxquant" / "plugin.yaml"' in SPEC_CODE # 源路径必须指向现存目录 diff --git a/backend/tests/test_resolve_universe_index.py b/backend/tests/test_resolve_universe_index.py index ef27d0e2..a64492e3 100644 --- a/backend/tests/test_resolve_universe_index.py +++ b/backend/tests/test_resolve_universe_index.py @@ -1,4 +1,5 @@ -"""_resolve_universe 指数过滤测试。""" +"""_resolve_universe 维表单一来源 + 指数过滤测试。""" +import polars as pl import pytest from app.jobs import daily_pipeline @@ -10,20 +11,59 @@ def repo(tmp_path): return KlineRepository(DataStore(tmp_path)) +class _Capset: + def __init__(self, has_batch: bool = False): + self._has_batch = has_batch + + def has(self, cap): + return self._has_batch + + +def _write_instruments(tmp_path, symbols): + out = tmp_path / "instruments" / "instruments.parquet" + out.parent.mkdir(parents=True, exist_ok=True) + pl.DataFrame({"symbol": symbols}).write_parquet(out) + + def test_resolve_universe_excludes_index_symbols(repo, monkeypatch, tmp_path): """自选里的指数不进入股票日K/分钟K同步池。""" - class _Capset: - def has(self, cap): - return False - + _write_instruments(tmp_path, ["600000.SH"]) monkeypatch.setattr( daily_pipeline, "get_pool", lambda name, refresh=False: ["600000.SH", "000001.SH"] if name == "watchlist" else [], ) - monkeypatch.setattr(daily_pipeline, "DEMO_SYMBOLS", []) monkeypatch.setattr(daily_pipeline.settings, "data_dir", tmp_path) monkeypatch.setattr(repo, "get_index_symbol_set", lambda: {"000001.SH"}) universe = daily_pipeline._resolve_universe(_Capset(), repo) assert "600000.SH" in universe assert "000001.SH" not in universe + + +def test_resolve_universe_never_fetches_pools(repo, monkeypatch, tmp_path): + """维表单一来源: 即使有 batch 能力也不拉 CN_Equity_A 池 (SW1 兜底随之消失)。""" + def _guard(name, refresh=False): + if name != "watchlist": + raise AssertionError(f"不得拉取标的池: {name}") + return ["600000.SH"] + + _write_instruments(tmp_path, ["000001.SZ"]) + monkeypatch.setattr(daily_pipeline, "get_pool", _guard) + monkeypatch.setattr(daily_pipeline.settings, "data_dir", tmp_path) + monkeypatch.setattr(repo, "get_index_symbol_set", lambda: set()) + + universe = daily_pipeline._resolve_universe(_Capset(has_batch=True), repo) + assert universe == ["000001.SZ", "600000.SH"] + + +def test_resolve_universe_watchlist_only_when_no_table(repo, monkeypatch, tmp_path): + """维表缺失时不伪造标的: 池=自选, 宁缺毋假。""" + monkeypatch.setattr( + daily_pipeline, "get_pool", + lambda name, refresh=False: ["600000.SH"] if name == "watchlist" else [], + ) + monkeypatch.setattr(daily_pipeline.settings, "data_dir", tmp_path) + monkeypatch.setattr(repo, "get_index_symbol_set", lambda: set()) + + universe = daily_pipeline._resolve_universe(_Capset(), repo) + assert universe == ["600000.SH"] diff --git a/backend/tests/test_secrets_external_edit.py b/backend/tests/test_secrets_external_edit.py new file mode 100644 index 00000000..2e3a13d8 --- /dev/null +++ b/backend/tests/test_secrets_external_edit.py @@ -0,0 +1,72 @@ +"""secrets.json 作为统一配置源的外部编辑契约。 + +用户可以直接用编辑器修改 secrets.json (AI + 数据源 Key 同文件), 保存后 +下一次请求即读取最新内容 —— 不需要重启服务。这里锁定三条: +① file_path() 指向 data/user_data/secrets.json; +② 外部改写文件后 load()/get_ai_config() 立刻反映新值 (mtime+size 签名缓存失效); +③ GET /api/settings 暴露 secrets_path 供设置页展示。 +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from app import secrets_store + + +@pytest.fixture() +def data_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + from app.config import settings + + monkeypatch.setattr(settings, "data_dir", tmp_path, raising=False) + yield tmp_path + + +def test_file_path_points_to_user_data_secrets(data_dir: Path) -> None: + assert secrets_store.file_path() == data_dir / "user_data" / "secrets.json" + + +def test_external_edit_is_picked_up_without_restart(data_dir: Path) -> None: + secrets_store.save({"ai_model": "old-model", "ai_base_url": "https://old.example/v1"}) + assert secrets_store.get_ai_config("ai_model") == "old-model" + + # 模拟用户编辑器直接覆写文件 (绕过 secrets_store.save)。 + # 内容长度与 mtime 都变化, 签名缓存必然失效。 + path = data_dir / "user_data" / "secrets.json" + path.write_text(json.dumps({ + "ai_model": "brand-new-model", + "ai_base_url": "https://new.example/v1", + "ai_api_key": "sk-new", + }, indent=2), encoding="utf-8") + + assert secrets_store.get_ai_config("ai_model") == "brand-new-model" + assert secrets_store.get_ai_config("ai_base_url") == "https://new.example/v1" + assert secrets_store.get_ai_key() == "sk-new" + + +def test_external_edit_with_same_size_still_reloads(data_dir: Path) -> None: + """同长度改写: 签名含 mtime_ns, 不能因 size 相同而吃到旧缓存。""" + import os + import time + + path = data_dir / "user_data" / "secrets.json" + secrets_store.save({"ai_model": "model-aaaa"}) + assert secrets_store.get_ai_config("ai_model") == "model-aaaa" + + path.write_text(json.dumps({"ai_model": "model-bbbb"}), encoding="utf-8") + # 强制 mtime 前进 (文件系统时间戳粒度可能吞掉快速连续写入) + future = time.time() + 5 + os.utime(path, (future, future)) + + assert secrets_store.get_ai_config("ai_model") == "model-bbbb" + + +def test_get_settings_exposes_secrets_path(data_dir: Path) -> None: + from app.api import settings as settings_api + + payload = settings_api.get_settings() + + assert payload["secrets_path"].endswith("secrets.json") + assert str(data_dir) in payload["secrets_path"] or Path(payload["secrets_path"]).name == "secrets.json" diff --git a/backend/tests/test_sponsor_models.py b/backend/tests/test_sponsor_models.py new file mode 100644 index 00000000..7c490db7 --- /dev/null +++ b/backend/tests/test_sponsor_models.py @@ -0,0 +1,134 @@ +"""sponsor-models 代理端点的精简字段契约测试。 + +上游 /v1/models 原始体量大 (144KB, 含分层 tiers/dimensions 等细粒度定价); +端点只透传前端展示需要的 id / context_length / discount_rate / 首档单价, +并以 models 纯 id 列表保持向后兼容。 +""" +from __future__ import annotations + +import httpx +import pytest + +from app.api import settings as settings_api + + +def _fake_upstream(payload: dict): + """把 httpx.AsyncClient 换成返回固定 /models 响应的伪实现。""" + + class _Resp: + status_code = 200 + + def raise_for_status(self): + return None + + def json(self): + return payload + + class _Client: + def __init__(self, *args, **kwargs): + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + async def get(self, url): + assert url == "https://llm.runninghub.ai/v1/models" + return _Resp() + + return _Client + + +@pytest.mark.asyncio +async def test_sponsor_models_returns_trimmed_pricing(monkeypatch): + payload = { + "object": "list", + "data": [ + { + "id": "deepseek/deepseek-v4.1-flash", + "object": "model", + "owned_by": "runninghub", + "context_length": 1048576, + "pricing": { + "billingType": "TOKEN_INPUT_TIERED", + "discountRate": 0.6, + "input": {"amount": 0.000295, "unit": "USD/1K", "tiers": []}, + "output": {"amount": 0.001177, "unit": "USD/1K", "tiers": []}, + "dimensions": {"cache_read": {"amount": 0.00001}}, + }, + }, + { + "id": "openai/gpt-6.1-sol", + "object": "model", + "context_length": 1050000, + "pricing": {"discountRate": 1.0, "input": {"amount": 0.002}, "output": {"amount": 0.01}}, + }, + {"id": "", "object": "model"}, # 空 id 剔除 + {"object": "model"}, # 无 id 剔除 + "garbage", # 非 dict 剔除 + ], + } + monkeypatch.setattr(httpx, "AsyncClient", _fake_upstream(payload)) + + result = await settings_api.list_sponsor_models() + + assert result["models"] == ["deepseek/deepseek-v4.1-flash", "openai/gpt-6.1-sol"] + assert result["items"] == [ + { + "id": "deepseek/deepseek-v4.1-flash", + "context_length": 1048576, + "discount_rate": 0.6, + "input_price": 0.000295, + "output_price": 0.001177, + }, + { + "id": "openai/gpt-6.1-sol", + "context_length": 1050000, + "discount_rate": 1.0, + "input_price": 0.002, + "output_price": 0.01, + }, + ] + + +@pytest.mark.asyncio +async def test_sponsor_models_tolerates_missing_pricing(monkeypatch): + """个别模型无 pricing 块: discount_rate 回退 1.0(无折扣), 价格为 None。""" + payload = {"data": [{"id": "vendor/plain-model", "context_length": None}]} + monkeypatch.setattr(httpx, "AsyncClient", _fake_upstream(payload)) + + result = await settings_api.list_sponsor_models() + + assert result["items"] == [{ + "id": "vendor/plain-model", + "context_length": None, + "discount_rate": 1.0, + "input_price": None, + "output_price": None, + }] + + +@pytest.mark.asyncio +async def test_sponsor_models_maps_upstream_error_to_502(monkeypatch): + from fastapi import HTTPException + + class _Broken: + def __init__(self, *args, **kwargs): + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + async def get(self, url): + raise httpx.ConnectError("boom") + + monkeypatch.setattr(httpx, "AsyncClient", _Broken) + + with pytest.raises(HTTPException) as exc_info: + await settings_api.list_sponsor_models() + assert exc_info.value.status_code == 502 diff --git a/backend/tests/test_task_registry.py b/backend/tests/test_task_registry.py new file mode 100644 index 00000000..e3e67754 --- /dev/null +++ b/backend/tests/test_task_registry.py @@ -0,0 +1,106 @@ +"""进行中任务注册表 (跨设备接回) 的服务与 API 测试。 + +契约: upsert 幂等 (同 id 更新不新增)、容量上限、孤儿 running 回收、 +API 生命周期 (PUT/GET/DELETE) 与非法输入拒绝。存取均为小文件原子写。 +""" +from __future__ import annotations + +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from app.api import task_registry as task_registry_api +from app.services import task_registry + + +@pytest.fixture +def client(tmp_path: Path) -> TestClient: + app = FastAPI() + app.include_router(task_registry_api.router) + app.state.repo = SimpleNamespace(store=SimpleNamespace(data_dir=tmp_path)) + return TestClient(app) + + +def test_upsert_idempotent_and_list_sorted(tmp_path: Path) -> None: + task_registry.upsert(tmp_path, "backtest:abc", kind="backtest", status="running", payload={"qs": "x=1"}) + task_registry.upsert( + tmp_path, "backtest:abc", kind="backtest", status="done", payload={"qs": "x=1"} + ) + task_registry.upsert(tmp_path, "mining:m1", kind="mining", status="running", payload={"run_id": "r1"}) + + items = task_registry.list_items(tmp_path) + assert [it["id"] for it in items] == ["mining:m1", "backtest:abc"] # updated_at 倒序 + assert items[1]["status"] == "done" # 同 id 更新而非新增 + assert items[1]["started_at"] > 0 + + only_bt = task_registry.list_items(tmp_path, kind="backtest") + assert [it["id"] for it in only_bt] == ["backtest:abc"] + + +def test_capacity_trim_keeps_newest(tmp_path: Path) -> None: + for i in range(task_registry.MAX_ITEMS + 5): + task_registry.upsert( + tmp_path, f"backtest:t{i}", kind="backtest", status="done" + ) + time.sleep(0.001) # 保证 updated_at 严格递增 + items = task_registry.list_items(tmp_path) + assert len(items) == task_registry.MAX_ITEMS + assert items[-1]["id"] == "backtest:t5" # t0..t4 最旧的被裁掉 + assert items[0]["id"] == f"backtest:t{task_registry.MAX_ITEMS + 4}" + + +def test_orphan_running_pruned_on_read(tmp_path: Path) -> None: + import json + + task_registry.upsert(tmp_path, "backtest:old", kind="backtest", status="running") + # 直接改文件把 updated_at 拨到 25h 前 (模拟进程重启遗留的孤儿 running) + p = tmp_path / "user_data" / task_registry._FILENAME + doc = json.loads(p.read_text(encoding="utf-8")) + doc["items"][0]["updated_at"] = time.time() - (task_registry.ORPHAN_RUNNING_SECONDS + 3600) + p.write_text(json.dumps(doc), encoding="utf-8") + + items = task_registry.list_items(tmp_path) + assert items == [], "孤儿 running 项应在读取时被回收" + # 回收已落盘: 重复读不再触发 + assert task_registry.list_items(tmp_path) == [] + + +def test_api_lifecycle(client: TestClient) -> None: + assert client.get("/api/task-registry").json() == {"items": []} + + r = client.put( + "/api/task-registry/backtest:k1", + json={"kind": "backtest", "status": "running", "label": "demo", "payload": {"qs": "a=1"}}, + ) + assert r.status_code == 200 and r.json()["item"]["id"] == "backtest:k1" + + listed = client.get("/api/task-registry", params={"kind": "backtest"}).json()["items"] + assert len(listed) == 1 and listed[0]["label"] == "demo" + + r = client.put( + "/api/task-registry/backtest:k1", + json={"kind": "backtest", "status": "done", "payload": {"qs": "a=1"}}, + ) + assert r.status_code == 200 + assert client.get("/api/task-registry").json()["items"][0]["status"] == "done" + + assert client.delete("/api/task-registry/backtest:k1").json()["removed"] is True + assert client.get("/api/task-registry").json()["items"] == [] + assert client.delete("/api/task-registry/backtest:k1").json()["removed"] is False + + +def test_api_rejects_bad_input(client: TestClient) -> None: + assert client.put( + "/api/task-registry/bad id!", json={"kind": "backtest", "status": "running"} + ).status_code == 400 # id 含非法字符 + assert client.put( + "/api/task-registry/backtest:x", json={"kind": "nope", "status": "running"} + ).status_code == 400 # 未知 kind + assert client.put( + "/api/task-registry/backtest:x", json={"kind": "backtest", "status": "weird"} + ).status_code == 400 # 未知 status + assert client.get("/api/task-registry", params={"kind": "nope"}).status_code == 400 diff --git a/backend/tests/test_tdxquant_provider.py b/backend/tests/test_tdxquant_provider.py new file mode 100644 index 00000000..744398a5 --- /dev/null +++ b/backend/tests/test_tdxquant_provider.py @@ -0,0 +1,625 @@ +"""TdxQuantProvider 契约与单位标准化测试。 + +不依赖真实网络与 tdxaidata 安装: 用假 TdxQuantClient 返回样例快照/K线/除权事件, +验证字段映射、单位口径 (CONTRIBUTING §3: 万元→元、股→手、每10股→每股、 +小数制涨跌幅)、lib 物化与 token 注入、重试语义、能力声明与设置页试拉。 +""" + +from __future__ import annotations + +import math +from datetime import date, datetime, timedelta +from pathlib import Path +from types import SimpleNamespace +from zoneinfo import ZoneInfo + +import pytest + +from app.plugins.tdxquant import client as tc +from app.plugins.tdxquant import provider as tp +from app.plugins.tdxquant.provider import TdxQuantProvider + +_BEIJING = ZoneInfo("Asia/Shanghai") + + +# ---------------------------------------------------------------- fakes +class _FakeClient: + """按测试需要预置返回值; 记录调用供批大小/参数断言。""" + + def __init__(self): + self.universe: list[str] = [] + self.universe_error: Exception | None = None + self.snapshot_results: list[object] = [] # 每次 snapshot_batch 弹出一个(dict 或 Exception) + self.snapshot_calls: list[list[str]] = [] + self.his: dict[str, object] = {} # code -> raw dict | Exception + self.his_default: object = {} + self.his_calls: list[dict] = [] + self.events: dict[str, object] = {} # code -> list[dict] | Exception + self.event_calls: list[dict] = [] + + def close(self) -> None: + pass + + def stock_universe(self) -> list[str]: + if self.universe_error: + raise self.universe_error + return list(self.universe) + + def snapshot_batch(self, codes): + self.snapshot_calls.append(list(codes)) + if not self.snapshot_results: + return {} + item = self.snapshot_results.pop(0) + if isinstance(item, Exception): + raise item + return item + + def his_dats(self, code, end_d, count, period="1d"): + self.his_calls.append({"code": code, "end": end_d, "count": count, "period": period}) + item = self.his.get(code, self.his_default) + if isinstance(item, Exception): + raise item + return item + + def divid_factors(self, code, start_d=None, end_d=None): + self.event_calls.append({"code": code, "start": start_d, "end": end_d}) + item = self.events.get(code, []) + if isinstance(item, Exception): + raise item + return item + + +def _provider(fake: _FakeClient) -> TdxQuantProvider: + p = TdxQuantProvider() + p._client = fake + return p + + +def _snap_row(**over) -> dict: + row = { + "Now": "1263.00", "LastClose": "1255.79", "Open": "1255.42", + "Max": "1282.00", "Min": "1255.42", "Volume": "35110", + "Amount": "445322.34", "RefreshTime": "153017", + "Buyp": ["1263.00", "1262.99"], "Buyv": ["73", "2"], + "Sellp": ["1263.06", "1263.11"], "Sellv": ["1", "3"], + } + row.update(over) + return row + + +# ---------------------------------------------------------------- client: lib 物化 +def _make_source_lib(root: Path) -> Path: + src = root / "src_lib" + src.mkdir() + (src / "TdxAiData.dll").write_bytes(b"FAKE_DLL_BYTES") + (src / "libTdxAiData.so").write_bytes(b"FAKE_SO_BYTES") + (src / "libTdxAiData.dylib").write_bytes(b"FAKE_DYLIB_BYTES") # macOS + (src / "TdxAiData.ini").write_text( + "[TcHqHost]\nHostNum=2\nPrimaryHost=0\n\n[Token]\ntoken=OLD\n", encoding="utf-8" + ) + (src / "user.ini").write_text("[Token]\ntoken=OLD\n", encoding="utf-8") + return src + + +def test_materialize_lib_writes_token_into_both_ini(tmp_path): + src = _make_source_lib(tmp_path) + lib_root = tmp_path / "tdxquant_lib" + dll = tc.materialize_lib("TDX-NEW-KEY", lib_root, source=src) + assert dll.name == tc.LIB_DLL_NAME + assert (lib_root / tc.LIB_DLL_NAME).exists() + for ini_name in ("TdxAiData.ini", "user.ini"): + text = (lib_root / ini_name).read_text(encoding="utf-8") + assert "token=TDX-NEW-KEY" in text + assert "token=OLD" not in text + # 其它配置段保留 + assert "[TcHqHost]" in (lib_root / "TdxAiData.ini").read_text(encoding="utf-8") + assert "HostNum=2" in (lib_root / "TdxAiData.ini").read_text(encoding="utf-8") + + +def test_materialize_lib_marker_skips_recopy(tmp_path, monkeypatch): + src = _make_source_lib(tmp_path) + lib_root = tmp_path / "tdxquant_lib" + tc.materialize_lib("TDX-KEY-A", lib_root, source=src) + + copies = [] + real_copy = tc.shutil.copy2 + + def _spy(src_p, dst_p): + copies.append(str(src_p)) + return real_copy(src_p, dst_p) + + monkeypatch.setattr(tc.shutil, "copy2", _spy) + tc.materialize_lib("TDX-KEY-A", lib_root, source=src) # 同 Key 同 DLL → 复用 + assert copies == [] + tc.materialize_lib("TDX-KEY-B", lib_root, source=src) # 换 Key → 重物化 + assert copies + assert "token=TDX-KEY-B" in (lib_root / "user.ini").read_text(encoding="utf-8") + + +def test_write_token_ini_adds_missing_section(tmp_path): + ini = tmp_path / "user.ini" + ini.write_text("[Other]\nfoo=1\n", encoding="utf-8") + tc.write_token_ini(ini, "K1") + text = ini.read_text(encoding="utf-8") + assert "token=K1" in text and "[Other]" in text and "foo=1" in text + + +# ---------------------------------------------------------------- client: 重试语义 +def _client_with_raw(monkeypatch, returns: list[object], sleep_log: list[float]): + c = object.__new__(tc.TdxQuantClient) + c.api_key = "k" + state = {"i": 0} + + def _raw_value(_method, *args): + item = returns[min(state["i"], len(returns) - 1)] + state["i"] += 1 + if isinstance(item, Exception): + raise item + return item + + c._tqs = SimpleNamespace(_clean=lambda raw: raw) + c._native_cls = SimpleNamespace(get_instance=lambda: SimpleNamespace(get_his_dats=_raw_value)) + c._universe, c._universe_at = [], 0.0 + monkeypatch.setattr(tc.time, "sleep", lambda s: sleep_log.append(s)) + return c + + +def test_raw_retries_transient_exception(monkeypatch): + sleeps: list[float] = [] + c = _client_with_raw( + monkeypatch, + [RuntimeError("Invalid server response"), {"Date": ["20260101"]}], + sleeps, + ) + out = c.his_dats("600519.SH", date(2026, 1, 10), 5) + assert out["Date"] == ["20260101"] + assert sleeps # 失败后有退避 + + +def test_raw_gives_up_on_persistent_none(monkeypatch): + sleeps: list[float] = [] + c = _client_with_raw(monkeypatch, [None, None], sleeps) + with pytest.raises(tc.TdxQuantError): + c.his_dats("600519.SH", date(2026, 1, 10), 5) + + +def test_snapshot_batch_retries_transient_failure(monkeypatch): + sleeps: list[float] = [] + c = _client_with_raw(monkeypatch, [], sleeps) + state = {"i": 0} + + def _snap(stock_list=None, field_list=None): + state["i"] += 1 + if state["i"] == 1: + raise RuntimeError("Invalid server response") + return {"600519.SH": {"Now": "10.0"}} + + c._tqs = type("T", (), {"get_market_snapshot_batch": staticmethod(_snap)})() + out = c.snapshot_batch(["600519.SH"]) + assert out == {"600519.SH": {"Now": "10.0"}} + assert state["i"] == 2 and sleeps + + +def test_raw_retries_none_then_data(monkeypatch): + sleeps: list[float] = [] + c = _client_with_raw(monkeypatch, [None, {"Date": ["20260101"]}], sleeps) + assert c.his_dats("600519.SH", date(2026, 1, 10), 5)["Date"] == ["20260101"] + + +# ---------------------------------------------------------------- provider: 快照映射 +class _FixedDT(datetime): + @classmethod + def now(cls, tz=None): + return cls(2026, 10, 11, 15, 0, 0, tzinfo=tz) + + +def _map_row(monkeypatch, **over) -> dict: + monkeypatch.setattr(tp, "datetime", _FixedDT) + return tp._map_snapshot_row("600519.SH", _snap_row(**over), 999) + + +def test_snapshot_row_units(monkeypatch): + rec = _map_row(monkeypatch) + assert rec["symbol"] == "600519.SH" + assert rec["last_price"] == 1263.00 + assert rec["prev_close"] == 1255.79 + assert rec["high"] == 1282.00 and rec["low"] == 1255.42 # Max/Min → high/low + assert rec["volume"] == 35110 # 快照已为手, 零换算 + assert rec["amount"] == pytest.approx(445322.34 * 10_000) # 万元 → 元 + # 涨跌幅小数制: (1263.00-1255.79)/1255.79 + assert rec["change_pct"] == pytest.approx((1263.00 - 1255.79) / 1255.79) + assert rec["change_amount"] == pytest.approx(1263.00 - 1255.79) + assert rec["name"] is None and rec["turnover_rate"] is None # 不伪造 + # RefreshTime 北京墙钟 → epoch ms + expect = datetime(2026, 10, 11, 15, 30, 17, tzinfo=_BEIJING) + assert rec["timestamp"] == int(expect.timestamp() * 1000) + + +def test_snapshot_row_bad_refresh_time_falls_back(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + rec = tp._map_snapshot_row("600519.SH", _snap_row(RefreshTime="99:99"), 123456) + assert rec["timestamp"] == 123456 + + +def test_compact_date_accepts_iso_and_compact(): + assert tp._compact_date("20260924") == date(2026, 9, 24) + assert tp._compact_date("2024-12-20 00:00:00") == date(2024, 12, 20) # DataFrame 归一形态 + assert tp._compact_date(None) is None + assert tp._compact_date("bad") is None + + +def test_snapshot_row_dropped_without_prices(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + assert tp._map_snapshot_row("600519.SH", _snap_row(Now=None, LastClose=None), 1) is None + + +# ---------------------------------------------------------------- provider: realtime +def test_realtime_batches_and_merges(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + fake = _FakeClient() + fake.universe = [f"60000{i}.SH" for i in range(tp._SNAPSHOT_BATCH)] + ["000001.SZ", "000002.SZ"] + fake.snapshot_results = [ + {"600000.SH": _snap_row()}, + {"000001.SZ": _snap_row(Now="11.59", LastClose="11.78")}, + ] + records = _provider(fake).get_realtime() + assert [len(c) for c in fake.snapshot_calls] == [tp._SNAPSHOT_BATCH, 2] # 600 只/批 + assert len(records) == 2 + assert {r["symbol"] for r in records} == {"600000.SH", "000001.SZ"} + + +def test_realtime_soft_fail_on_universe_error(): + fake = _FakeClient() + fake.universe_error = tc.TdxQuantError("get_stock_list(上证主板): boom") + assert _provider(fake).get_realtime() == [] + + +def test_realtime_isolates_failed_batch(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + fake = _FakeClient() + fake.universe = [f"6000{i:03d}.SH" for i in range(tp._SNAPSHOT_BATCH)] + ["000001.SZ"] + fake.snapshot_results = [ + tc.TdxQuantError("snapshot_batch(600 只): invalid"), # 首批失败被隔离 + {"000001.SZ": _snap_row(Now="11.59", LastClose="11.78")}, + ] + records = _provider(fake).get_realtime() + assert [r["symbol"] for r in records] == ["000001.SZ"] + + +def test_realtime_indices_maps_sh_index_code(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + fake = _FakeClient() + fake.snapshot_results = [{"999999.SH": _snap_row(Now="3813.79", LastClose="3811.90")}] + records = _provider(fake).get_realtime_indices(["000001.SH", "399001.SZ"]) + assert fake.snapshot_calls == [["999999.SH", "399001.SZ"]] # 000001.SH → 999999.SH + assert records and records[0]["last_price"] == 3813.79 + + +def test_realtime_indices_none_on_failure(): + fake = _FakeClient() + fake.snapshot_results = [tc.TdxQuantError("snapshot_batch(1 只): down")] + assert _provider(fake).get_realtime_indices(["000001.SH"]) is None + + +# ---------------------------------------------------------------- provider: daily +def _his_raw(*, dates=("20260102", "20260105"), volume="3123935.00", amount="386731.09") -> dict: + return { + "Date": list(dates), "Time": ["0"] * len(dates), + "Open": ["10.0"] * len(dates), "High": ["11.0"] * len(dates), + "Low": ["9.5"] * len(dates), "Close": ["10.5"] * len(dates), + "Volume": [volume] * len(dates), "Amount": [amount] * len(dates), + } + + +def test_daily_units_and_window_filter(): + fake = _FakeClient() + fake.his["600519.SH"] = _his_raw(dates=("20251231", "20260102", "20260105", "20260106")) + df = _provider(fake).get_daily( + ["600519.SH"], + datetime(2026, 1, 1), datetime(2026, 1, 5), + ) + assert df["date"].to_list() == [date(2026, 1, 2), date(2026, 1, 5)] # 窗口过滤 + row = df.row(0, named=True) + assert row["close"] == 10.5 + assert row["volume"] == math.floor(3123935.0 / 100) # 股 → 手 + assert row["amount"] == pytest.approx(386731.09 * 10_000) # 万元 → 元 + call = fake.his_calls[0] + assert call["period"] == "1d" + assert call["count"] >= 30 # count 模式根数下限 + + +def test_daily_etf_asset_type_unsupported_reports_empty(): + fake = _FakeClient() + chunks = list( + _provider(fake).iter_daily( + ["510300.SH"], datetime(2026, 1, 1), datetime(2026, 1, 31), asset_type="etf" + ) + ) + assert chunks == [] + + +def test_daily_symbol_error_isolated_but_iteration_continues(): + fake = _FakeClient() + fake.his["600000.SH"] = tc.TdxQuantError("get_his_dats: invalid") + fake.his["600001.SH"] = _his_raw() + done: list[tuple[int, int]] = [] + df = _provider(fake).get_daily( + ["600000.SH", "600001.SH"], datetime(2025, 12, 1), datetime(2026, 1, 31), + on_chunk_done=lambda cur, total: done.append((cur, total)), + ) + assert df.height == 2 # 只有 600001 的两根 + assert done == [(1, 2), (2, 2)] # 空批次也推进度 + + +# ---------------------------------------------------------------- provider: adj_factor +def _closes_raw(pairs: list[tuple[str, float]]) -> dict: + return { + "Date": [d for d, _ in pairs], "Close": [str(c) for _, c in pairs], + "Open": ["1"] * len(pairs), "High": ["1"] * len(pairs), "Low": ["1"] * len(pairs), + "Volume": ["100"] * len(pairs), "Amount": ["1"] * len(pairs), + } + + +def test_adj_factor_per10_to_per_share_and_formula(): + # 10派30.876元 → 每股 D=3.0876; P=30.876 → ref=(30.876-3.0876)/1=27.7884 → 27.79 + fake = _FakeClient() + fake.events["600519.SH"] = [ + {"Date": "20240619", "Type": "1", "Bonus": 30.876, "AllotPrice": 0.0, + "ShareBonus": 0.0, "Allotment": 0.0}, + ] + fake.his["600519.SH"] = _closes_raw([("20240618", 30.876), ("20240619", 28.0), ("20240620", 28.5)]) + df = _provider(fake).get_adj_factors( + ["600519.SH"], datetime(2024, 1, 1), datetime(2024, 12, 31) + ) + assert df.height == 1 + row = df.row(0, named=True) + assert row["trade_date"] == date(2024, 6, 19) + assert row["dividend"] == pytest.approx(3.0876) # 每 10 股 → 每股 + assert row["prev_close"] == 30.876 + ref = math.floor((30.876 - 3.0876) * 100 + 0.5) / 100 # half-up 两位 + assert row["ex_factor"] == pytest.approx(30.876 / ref) + # 涨跌停自检: 复权后除权日涨跌 28.0/(30.876/factor)-1 ≈ 0.1% 在板限内 → 保留 + + +def test_adj_factor_limit_check_drops_bad_factor(): + # 异常事件: 分红与价格严重不匹配 → 复权后除权日涨跌幅超板限 → 自检剔除 + fake = _FakeClient() + fake.events["600519.SH"] = [ + {"Date": "20240619", "Type": "1", "Bonus": 200.0, "AllotPrice": 0.0, + "ShareBonus": 0.0, "Allotment": 0.0}, + ] + fake.his["600519.SH"] = _closes_raw([("20240618", 30.876), ("20240619", 28.0)]) + df = _provider(fake).get_adj_factors( + ["600519.SH"], datetime(2024, 1, 1), datetime(2024, 12, 31) + ) + assert df.is_empty() + + +def test_adj_factor_future_event_without_prev_close_skipped(): + fake = _FakeClient() + fake.events["600519.SH"] = [ + {"Date": "20991231", "Type": "1", "Bonus": 30.876, "AllotPrice": 0.0, + "ShareBonus": 0.0, "Allotment": 0.0}, + ] + fake.his["600519.SH"] = _closes_raw([("20240618", 30.876)]) + df = _provider(fake).get_adj_factors( + ["600519.SH"], datetime(2024, 1, 1), datetime(2026, 12, 31) + ) + assert df.is_empty() # 无除权日前收盘 → 留给滚动增量 + + +def test_adj_factor_schema_columns(): + # 真实量级事件: P=10, 10派2 + 10送1 + 10配2(配股价5) → 参考价 8.31, 除权日收盘 8.35 + fake = _FakeClient() + fake.events["600000.SH"] = [ + {"Date": "20240619", "Type": "1", "Bonus": 2.0, "AllotPrice": 5.0, + "ShareBonus": 1.0, "Allotment": 2.0}, + ] + fake.his["600000.SH"] = _closes_raw([("20240618", 10.0), ("20240619", 8.35)]) + df = _provider(fake).get_adj_factors( + ["600000.SH"], datetime(2024, 1, 1), datetime(2024, 12, 31) + ) + assert df.columns == list(tp._ADJ_SCHEMA) + row = df.row(0, named=True) + assert row["dividend"] == pytest.approx(0.2) # 10派2 → 每股 0.2 + assert row["bonus"] == pytest.approx(0.1) # 10送1 → 每股 0.1 + assert row["allot"] == pytest.approx(0.2) # 10配2 → 每股 0.2 + assert row["allot_price"] == 5.0 + + +# ---------------------------------------------------------------- provider: depth5 +def test_depth_batch_maps_five_levels(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + fake = _FakeClient() + fake.snapshot_results = [{ + "600519.SH": _snap_row(), + "600000.SH": _snap_row(Buyp=None, Sellp=None), # 无五档 → 整标的省略 + }] + out = _provider(fake).get_depth_batch(["600519.SH", "600000.SH"]) + assert set(out) == {"600519.SH"} + entry = out["600519.SH"] + assert entry["bid_prices"] == [1263.00, 1262.99] + assert entry["bid_volumes"] == [73, 2] # 手, 透传为 int + assert entry["ask_prices"] == [1263.06, 1263.11] + assert entry["ask_volumes"] == [1, 3] + expect = datetime(2026, 10, 11, 15, 30, 17, tzinfo=_BEIJING) + assert entry["timestamp"] == int(expect.timestamp() * 1000) + + +# ---------------------------------------------------------------- availability / 能力 +def test_availability_missing_module(monkeypatch): + monkeypatch.setattr(tp, "_module_available", lambda: False) + ok, reason = tp.availability() + assert not ok and "tdxaidata" in reason + + +def test_availability_missing_key(monkeypatch): + monkeypatch.setattr(tp, "_module_available", lambda: True) + monkeypatch.setattr(tp, "get_api_key", lambda: "") + ok, reason = tp.availability() + assert not ok and tp.API_KEY_ENV in reason + + +def test_availability_ok(monkeypatch): + monkeypatch.setattr(tp, "_module_available", lambda: True) + monkeypatch.setattr(tp, "get_api_key", lambda: "TDX-xxx") + assert tp.availability() == (True, "ok") + + +def test_datasets_declaration(): + p = TdxQuantProvider() + assert set(p.config.datasets) == {"realtime", "depth5", "daily", "adj_factor", "minute", "full_minute"} + assert p.name == "tdxquant" and p.config.builtin is True + for absent in ("financial",): + assert absent not in p.config.datasets # 未声明 → provider_has_dataset False → 回退 tickflow + + +# ---------------------------------------------------------------- minute / full_minute +def _minute_raw(rows: list[tuple[str, str, float, float, float, float, float, float]]) -> dict: + """(Date, Time, O, H, L, C, Volume股, Amount万元) → 原生 1m 列式 dict。""" + return { + "Date": [r[0] for r in rows], "Time": [r[1] for r in rows], + "Open": [r[2] for r in rows], "High": [r[3] for r in rows], + "Low": [r[4] for r in rows], "Close": [r[5] for r in rows], + "Volume": [r[6] for r in rows], "Amount": [r[7] for r in rows], + } + + +def test_minute_units_and_wallclock_datetime(): + fake = _FakeClient() + fake.his = {"600519.SH": _minute_raw([ + ("20261009", "093100", 1255.0, 1256.0, 1254.9, 1255.5, 123_400.0, 15_500.0), + ("20261009", "150000", 1263.0, 1263.0, 1263.0, 1263.0, 38_600.00, 4869.62), + ])} + df = _provider(fake).get_minute( + ["600519.SH"], datetime(2026, 10, 8), datetime(2026, 10, 9, 23, 59) + ) + assert fake.his_calls[0]["period"] == "1m" + assert df.height == 2 + rows = df.sort("datetime").to_dicts() + assert rows[0]["datetime"] == datetime(2026, 10, 9, 9, 31, 0) # 北京墙钟 naive + assert rows[1]["datetime"] == datetime(2026, 10, 9, 15, 0, 0) + assert rows[0]["volume"] == 1234.0 # 股 → floor(/100) 手 + assert rows[1]["volume"] == 386.0 + assert rows[1]["amount"] == pytest.approx(48_696_200.0) # 万元 → 元 + + +def test_minute_window_filter_and_gap_skip(): + fake = _FakeClient() + fake.his = {"600519.SH": _minute_raw([ + ("20261008", "150000", 1250.0, 1250.0, 1250.0, 1250.0, 100_000.0, 12_500.0), # 窗口外 + ("20261009", "093100", 1255.0, 1256.0, 1254.9, 1255.5, 123_400.0, 15_500.0), + ("20261009", "093200", None, None, None, None, None, None), # 停牌缺口 + ])} + df = _provider(fake).get_minute( + ["600519.SH"], datetime(2026, 10, 9), datetime(2026, 10, 9, 23, 59) + ) + assert df.height == 1 + assert df.to_dicts()[0]["datetime"] == datetime(2026, 10, 9, 9, 31, 0) + + +def test_minute_freq_and_asset_guards(): + fake = _FakeClient() + p = _provider(fake) + assert p.get_minute(["600519.SH"], None, datetime.now(), freq="5m").is_empty() + assert p.get_minute(["510300.SH"], None, datetime.now(), asset_type="etf").is_empty() + assert fake.his_calls == [] # 未发起取数 + + +def test_minute_transport_error_soft(): + fake = _FakeClient() + fake.his = {"600519.SH": tc.TdxQuantError("boom")} + df = _provider(fake).get_minute(["600519.SH"], None, datetime.now()) + assert df.is_empty() # 软失败不阻断 + + +def test_get_intraday_batch_today_only(): + fake = _FakeClient() + today = date.today().strftime("%Y%m%d") + yday = (date.today() - timedelta(days=1)).strftime("%Y%m%d") + fake.his = {"600519.SH": _minute_raw([ + (yday, "150000", 1250.0, 1250.0, 1250.0, 1250.0, 100_000.0, 12_500.0), # 前日余量 → 过滤 + (today, "093100", 1255.0, 1256.0, 1254.9, 1255.5, 123_400.0, 15_500.0), + (today, "093200", 1255.5, 1257.0, 1255.0, 1256.0, 88_800.0, 11_160.0), + ])} + df = _provider(fake).get_intraday_batch(["600519.SH"]) + assert df.height == 2 # 只保留今日行 + assert fake.his_calls[0]["period"] == "1m" + + +def test_get_intraday_latest_universe_default(): + fake = _FakeClient() + fake.universe = ["600000.SH", "000001.SZ"] + today = date.today().strftime("%Y%m%d") + for sym in fake.universe: + fake.his[sym] = _minute_raw([ + (today, "093100", 10.0, 10.1, 9.9, 10.0, 50_000.0, 500.0), + (today, "093200", 10.0, 10.1, 9.9, 10.0, 50_000.0, 500.0), + ]) + df = _provider(fake).get_intraday_latest(count=3) + assert df.height == 4 # 每只 2 根 + assert set(df["symbol"].to_list()) == {"600000.SH", "000001.SZ"} + assert all(c["count"] <= 30 for c in fake.his_calls) + + +def test_test_dataset_minute(): + fake = _FakeClient() + today = date.today().strftime("%Y%m%d") + fake.his = {"600519.SH": _minute_raw([ + (today, "093100", 1255.0, 1256.0, 1254.9, 1255.5, 123_400.0, 15_500.0), + ])} + out = _provider(fake).test_dataset("minute", ["600519.SH"]) + assert out["rows"] == 1 + assert "datetime" in out["columns"] + + +# ---------------------------------------------------------------- provider: 试拉 +def test_test_dataset_realtime(monkeypatch): + monkeypatch.setattr(tp, "datetime", _FixedDT) + fake = _FakeClient() + fake.snapshot_results = [{"600519.SH": _snap_row()}] + out = _provider(fake).test_dataset("realtime") + assert out["rows"] == 1 and out["preview"][0]["symbol"] == "600519.SH" + + +def test_test_dataset_unsupported_dataset(): + out = _provider(_FakeClient()).test_dataset("financial") + assert "error" in out and "回退" in out["error"] + + +# ---------------------------------------------------------------- loader 集成 +def test_loader_manifest_parses_and_entry_importable(): + from importlib import import_module + + from app.data_providers.custom import loader + + manifest = loader.plugin_manifest("tdxquant") + assert manifest is not None + assert manifest["name"] == "tdxquant" + assert manifest["display_name"] == "TdxQuant (TdxAiData)" + assert set(manifest["datasets"]) == {"realtime", "depth5", "daily", "adj_factor", "minute", "full_minute"} + # 计费标识与 Key/积分管理页 (设置页积分角标 + 商城入口的契约来源) + assert manifest["billing"] == "points" + assert manifest["key_manage_url"].startswith("https://vip.tdx.com.cn/") + # 计费提示与使用要点 (设置页警示框 + 详情列表的契约来源) + assert "积分" in manifest["billing_note"] and "间隔" in manifest["billing_note"] + assert len(manifest["usage_notes"]) >= 4 + assert any("快照" in n for n in manifest["usage_notes"]) + module_path, _, class_name = manifest["entry"].partition(":") + provider_cls = getattr(import_module(module_path), class_name) + assert provider_cls.name == "tdxquant" + + +def test_loader_probe_plugin_key_dispatch(monkeypatch): + from app.data_providers.custom import loader + + monkeypatch.setattr(tp, "probe_api_key", lambda key: (key == "good", "ok" if key == "good" else "bad")) + assert loader.probe_plugin_key("tdxquant", "good") == (True, "ok") + assert loader.probe_plugin_key("tdxquant", "bad") == (False, "bad") + + +def test_close_resets_client(): + fake = _FakeClient() + p = _provider(fake) + p.close() + assert p._client is None diff --git a/backend/uv.lock b/backend/uv.lock index d28384b7..5acd6df2 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -2482,6 +2482,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/e1/b2df4bc09a1e51ff664c1e17018a4274b42e5e9352e4a478ea540512dc88/starlette-1.0.1-py3-none-any.whl", hash = "sha256:7c0e69b2ee1c848bd54669d908500117a3ee13de603a21427e5c6fc1adf98dcd", size = 72802, upload-time = "2026-05-21T21:58:56.551Z" }, ] +[[package]] +name = "tdxaidata" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0f/a8/3b1a4341769c081b3bdbc4e6c9991ac05c49585e256965cd9402f066daa6/tdxaidata-1.2.2.tar.gz", hash = "sha256:794011131704a7827746b89da490e43345ddd442644d6100890d39063485171e", size = 4430612, upload-time = "2026-09-30T10:08:12.158Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/c1/29b840db422aa32c87baedf2d402ccb8cbba8964d33caff3b20e5b3ddad4/tdxaidata-1.2.2-py3-none-any.whl", hash = "sha256:ceaf0be18cb54cd197e79cca5590d4ba7d7c3620b899e818d472d032f2c33849", size = 4443080, upload-time = "2026-09-30T10:08:07.807Z" }, +] + [[package]] name = "threadpoolctl" version = "3.6.0" @@ -2538,6 +2547,7 @@ dependencies = [ { name = "python-multipart" }, { name = "pyyaml" }, { name = "sse-starlette" }, + { name = "tdxaidata" }, { name = "tickflow", extra = ["all"] }, { name = "uvicorn", extra = ["standard"] }, { name = "winotify", marker = "sys_platform == 'win32'" }, @@ -2590,6 +2600,7 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.5" }, { name = "sse-starlette", specifier = ">=2.0" }, + { name = "tdxaidata", specifier = ">=1.2.2" }, { name = "tickflow", extras = ["all"], specifier = ">=0.1.23" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.30" }, { name = "vectorbt", marker = "extra == 'backtest'", specifier = ">=0.26" }, diff --git a/docs/configuration.md b/docs/configuration.md index 13f52708..d34b7715 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -64,6 +64,7 @@ AI_DAILY_TOKEN_BUDGET=500000 # 每日 token 预算上限 | 配置项 | 说明 | | :--- | :--- | | `AI_PROVIDER` | `openai_compat`(OpenAI 兼容,支持 DeepSeek / 通义 / OpenAI 等)或 `ollama`(本地模型) | +| `AI_PROTOCOL` | 请求协议: `chat_completions`(默认, 兼容面最广)或 `responses`(OpenAI 官方新协议, DeepSeek 官方亦已支持); 仅 openai 系 provider 生效, 也可在设置页切换 | | `AI_BASE_URL` | 接口地址,如 DeepSeek `https://api.deepseek.com/v1` | | `AI_API_KEY` | 留空则关闭 AI 功能 | | `AI_MODEL` | 模型名,如 `deepseek-chat` | diff --git a/docs/factor-system-design.md b/docs/factor-system-design.md index 1791df3a..10d16efe 100644 --- a/docs/factor-system-design.md +++ b/docs/factor-system-design.md @@ -804,7 +804,7 @@ researchUniverses() | price_limits.py:87-101 | numpy_limit_pct_vectors（当前名推断） | ✓ | | capabilities.py:44-45 | 复权口径一致性"不做路由耦合"注释 | ✓ | | index_const.py:12-15 | 核心四只代码（000001.SH/399001.SZ/399006.SZ/000680.SH） | ✓ | -| daily_pipeline.py:92-125 | _resolve_universe（CN_Equity_A 当前池） | ✓ | +| daily_pipeline.py | _resolve_universe（instruments 维表 + 自选，无池兜底） | ✓ | | kline_sync.py:358-367 | (symbol, trade_date) 去重 keep=last 原子合并 | ✓ | | scoring.py:13-51 / 53-66 / 91-104 / 108+ | VIRTUAL 依赖 35 项 / 预热表 / 依赖展开与预热推导 / scoring_value_expr | ✓（全文读取，附录A 由其逐字推导） | | monitor.py:1314-1322 | score_min/max 消费 result.scores | ✓ | diff --git a/docs/features.md b/docs/features.md index 3cdab61c..757f6400 100644 --- a/docs/features.md +++ b/docs/features.md @@ -161,6 +161,10 @@ ## 🧰 数据与扩展 +### 页面包(运行时扩展页面) + +页面可以打包成**单文件 ES 模块产物**(`frontend/page-pack-template` 一键构建),在「设置 → 页面包」上传即挂载为完整页面 — 路由、侧栏入口、菜单管理自动注册,**宿主无需重新构建**;本地 dev / 生产 / 桌面端一致生效。上传时即时试载校验契约(清单格式、路由冲突),坏包停用并显示原因;启停删除随时管理,刷新后生效。源码级扩展(`frontend/src/custom/`)与声明式分析页面仍是随构建的另外两条扩展路径。契约与安全边界见 [page-sdk.md](./page-sdk.md)。 + ### 数据源插件化 内置 [TickFlow](https://tickflow.org) 官方 SDK(日 K / 分钟 K / 指数 / 财务 / 实时行情),同时支持接入第三方数据源:YAML 声明自有 HTTP 接口,或开发插件源。多个数据集(日K / 除权因子 / 实时行情 / 分钟K / 五档盘口 / 财务,注册表持续扩展)按源声明自由组合,在「设置 → 数据源」切换。 @@ -195,6 +199,14 @@ timeseries 模式的表 (如人气排行) 默认只能从开启拉取之日起 数据量大的接口 (全市场排行、批量快照) 单次响应装不下时, 拉取配置可声明分页 (仅 GET): **页码参数名**(`page_param`, 如 `page`)、**每页条数参数名**(`page_size_param`, 如 `pageSize`)与**每页条数**(`page_size`)。配置后单次拉取按页循环请求并合并结果, 空页/不足一页自动判停; **最多页数**(`max_pages`, 默认 20, 上限 200)防止接口永远返回数据拖死循环。POST body 分页不支持 (请求体是固定模板)。 +##### 两级拉取协议 (目录 → 逐条目) + +有些数据源只提供「类别清单」+「单类别成员」两个端点 (同花顺概念/行业、多数板块类 API), 没有一次返回「标的 → 所属类别」全量映射的接口。这类源在拉取配置里声明 `fanout` 配方即可, 无需任何按表硬编码: 先请求 `pull.url` 拿第一级目录, 再按 `fanout.param` 并发拉每个条目的成员, 最后**反转为「标的 → 类别清单」**并与普通单请求路径产出同构的行。 + +配方字段 (除注明外均为字符串): `url` 第二级地址、`param` 第二级查询参数名、`id_field` 第一级条目中作为该参数值的字段、`label_field` 展示名字段 (可选, 仅用于告警)、`items_path` 第二级响应的成员数组 dot-path、`member_id_field` / `member_label_field` 成员里的代码与名称字段、`symbol_col` / `name_col` / `value_col` 输出列名、`separator` 多类别拼接符 (概念用 `;`, 行业层级用 `-`)、`concurrency` 并发上限 (默认 8)、`retries` 单条目失败重试次数 (默认 1, 用尽则跳过该条目并记 warning; 全部条目失败才报错)。鉴权与标识头沿用 `pull.auth` / 通用出站口径。 + +概念/行业两张内置表出厂即带完整配方, 在「数据」页展开本表 → **拉取** 可查看全部字段, 接入自有板块类源时照抄即可。 + ##### `/rows` 查询接口 (面向二次开发) 每张扩展表天然是标准 HTTP 接口 `GET /api/ext-data/{id}/rows`, 供页面之外的任意消费方调用: @@ -303,4 +315,4 @@ AI 客户端 ⇄ MCP stdio (薄桥) ⇄ HTTP + Bearer Token ⇄ 开放网关 ### 桌面客户端版本清单 -发布流水线会在每个 Release 附带 `latest.json`(版本号、三平台下载地址、sha256),设置 → 系统设置 → 检查更新即基于它(优先 GitHub API)提示新版本。 +发布流水线会在每个 Release 附带 `latest.json`(版本号、三平台下载地址、sha256),设置 → 关于 → 检查更新即基于它(优先 GitHub API)提示新版本。 diff --git a/docs/page-sdk.md b/docs/page-sdk.md new file mode 100644 index 00000000..66e494d4 --- /dev/null +++ b/docs/page-sdk.md @@ -0,0 +1,72 @@ +# 页面包开发指南 (Page Pack SDK v1) + +页面包是**运行时可挂载的页面产物**：把页面打包成单文件 ES 模块（`extension.js`），在「设置 → 页面包」上传，刷新后即成为完整页面——自动注册路由、侧栏入口，可在菜单设置里排序/隐藏。本地 dev、生产构建、桌面端三种运行形态一致生效，宿主无需重新构建。 + +## 快速开始 + +```bash +cp -r frontend/page-pack-template my-pack && cd my-pack +pnpm install && pnpm build # → dist/extension.js +``` + +打开宿主「设置 → 页面包」→ 填 ID（字母/数字/下划线）→ 选择 `dist/extension.js` → 上传并校验 → 刷新页面。 + +## 产物契约 (apiVersion 1) + +产物必须 `export default` 一个清单对象： + +```tsx +const extension = { + id: 'demo.hello', // 带命名空间且全局唯一; 冲突时整个包被拒 + apiVersion: 1 as const, + routes: [{ + id: 'hello', + path: '/demo-hello', // 非根绝对路径、静态(不支持 :param/通配), 不得覆盖核心或已有页面 + mount: (el, ctx) => { // 渲染进容器; 返回清理函数(可选) + const root = createRoot(el) + root.render() + return () => root.unmount() + }, + }], + navigation: [{ + id: 'hello', routeId: 'hello', label: '页面包演示', icon: 'Sparkles', + }], +} +export default extension +``` + +校验与冲突规则由宿主注册表统一执行（与源码级扩展 `frontend/src/custom/` 同一机制）；上传时即时试载校验，启动时再次校验，失败会停用该包并在面板显示原因。 + +## 能力桥 ctx + +页面包**自包含**（React 等依赖由模板打进产物，与宿主不共享模块实例），宿主能力经 `ctx` 暴露： + +| 能力 | 说明 | +| --- | --- | +| `ctx.api` | 宿主完整 API 封装（同源 fetch、自动携带登录态、统一错误处理） | +| `ctx.QK` | 集中查询键（若你自行使用 TanStack Query 可作 key 规范） | +| `ctx.cn` | 样式类合并工具（clsx + tailwind-merge） | +| `ctx.format` | 数值/日期格式化工具集 | + +**禁止** import 宿主内部模块（`@/lib/...` 等别名在产物构建中不可解析）；数据请求一律经 `ctx.api`，不得自行拼后端 URL 绕过封装。 + +## 图标白名单 + +`navigation[].icon` 按名称解析，白名单：`Activity BarChart3 Bell BookOpen Brain Calendar Database Globe LineChart PieChart Shield Sparkles Star Table TrendingUp Zap`（缺省 `Sparkles`）。需要扩充时在 `frontend/src/extensions/runtime.tsx` 的 `ICON_ALLOWLIST` 增补。 + +## 约束与边界 + +- **样式**：宿主 Tailwind 按需编译，任意类名不保证存在——布局关键样式用内联样式；主题色用宿主 CSS 变量（如 `var(--accent)`）。 +- **路由集成**：页面挂在宿主路由容器内（`/你的路径` 可直达、可刷新），但包内不共享宿主 Router 上下文——需要跳转宿主页面时用 `window.location` 或自行包 `MemoryRouter`。 +- **生效方式**：上传/启停后**刷新页面生效**（挂载发生在启动引导窗口），不做热插拔。 +- **大小**：上限 5MB/包；自包含 React 后典型产物 60–150KB（minify + 可自行加 brotli/gzip 由后端协商）。 +- **安全**：页面包是任意代码，与登录会话同权运行——只上传自己或可信来源的产物；这是单用户产品的既定边界，第三方插件沙箱不在 v1 范围。 +- **版本**：宿主升级破坏契约时提升 `apiVersion` 并保留旧版兼容期；产物内 `apiVersion` 不匹配会被明确拒绝。 + +## 与其他扩展方式的取舍 + +| 方式 | 适用 | 见 | +| --- | --- | --- | +| 页面包（运行时产物） | 免重建挂载、独立分发 | 本文档 | +| 源码级扩展 `frontend/src/custom//extension.tsx` | 随宿主一起构建维护、可用全部核心模块 | `docs/secondary-development.md` §3 | +| 声明式分析页面（零代码） | 扩展数据 + 模板化列表/分组页 | 设置 → 扩展页面 | diff --git a/docs/plugin-development.md b/docs/plugin-development.md index ce53134f..e394b012 100644 --- a/docs/plugin-development.md +++ b/docs/plugin-development.md @@ -34,6 +34,11 @@ hidden: false # (可选)true = 已加载但对设置 description: "数据源描述" install_hint: "pip install xxx" # 未装依赖时显示的安装提示 homepage: "https://example.com" # (可选)官网/申请地址, 显示在设置页 Key 配置说明中 +billing: points # (可选)计费标识: "points" → 设置页能力芯片/适配行标注「积分」 +billing_note: "消耗与频率/范围相关,间隔越短越高" # (可选)计费提示: 设置页按警示框展示 +usage_notes: # (可选)使用要点: 各能力规模/节奏, 详情卡以列表展示 + - "实时快照: 600 只/批, 全市场一轮约 10 批" +key_manage_url: "https://example.com/me" # (可选)Key/账务管理页, 显示在设置页 Key 配置区(如积分明细入口) ``` 只声明真实提供的数据集;未声明的数据集 `provider_has_dataset` 返回 False,自动回退 @@ -313,6 +318,11 @@ uv run --extra dev python -m ruff check app/plugins// tests/test_= 0.8 && z <= 1.5) { - document.documentElement.style.zoom = String(z) - } - } catch (e) { /* ignore */ } + var raw = localStorage.getItem('tf-page-zoom') + var z = raw == null ? 0.9 : JSON.parse(raw) + if (typeof z !== 'number' || z < 0.8 || z > 1.5) z = 0.9 + document.documentElement.style.zoom = String(z) + } catch (e) { + document.documentElement.style.zoom = '0.9' + } })() diff --git a/frontend/package.json b/frontend/package.json index 490915e3..add8f574 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "tickflow-stock-panel-frontend", "private": true, - "version": "0.3.5", + "version": "0.3.6", "type": "module", "scripts": { "dev": "vite", diff --git a/frontend/page-pack-template/README.md b/frontend/page-pack-template/README.md new file mode 100644 index 00000000..d5cfa723 --- /dev/null +++ b/frontend/page-pack-template/README.md @@ -0,0 +1,25 @@ +# 页面包模板 (Page Pack Template) + +把一个页面打包成**单文件 ES 模块产物**（`dist/extension.js`），上传到运行中的面板（设置 → 页面包），刷新后即成为完整页面——自动注册路由与侧栏入口，可在菜单设置里排序/隐藏。本地 dev、Docker、桌面端一致生效，**宿主无需重新构建**。 + +## 打包 + +```bash +pnpm install +pnpm build # → dist/extension.js +``` + +产物自包含（React 等依赖由本模板打进 bundle，不与宿主共享模块实例），典型体积 60–150KB，上限 5MB。 + +## 上传 + +面板 **设置 → 页面包**：填 ID（字母/数字/下划线）→ 选择 `dist/extension.js` → 上传并校验（即时试载，清单不合法或路由冲突会被拒绝并显示原因）→ 启用 → 刷新页面。 + +## 开发要点 + +- `src/extension.tsx` — 产物清单（id / routes / navigation）与示例页面；`mount(el, ctx)` 渲染进容器，返回清理函数。 +- `src/tsp.ts` — 从 `window.__TSP_EXT__` 取宿主能力桥 `ctx`：`ctx.api`（宿主完整 API 封装）、`ctx.QK`、`ctx.cn`、`ctx.format`。 +- **禁止** import 宿主内部模块（`@/lib/...` 在产物构建中不可解析）；数据请求一律经 `ctx.api`。 +- 样式：宿主 Tailwind 按需编译，任意类名不保证存在——布局关键样式用内联样式，主题色用宿主 CSS 变量（如 `var(--accent)`）。 + +完整契约（apiVersion、图标白名单、路由规则、安全边界）见 [`docs/page-sdk.md`](../../docs/page-sdk.md)。 diff --git a/frontend/page-pack-template/package.json b/frontend/page-pack-template/package.json new file mode 100644 index 00000000..08008568 --- /dev/null +++ b/frontend/page-pack-template/package.json @@ -0,0 +1,20 @@ +{ + "name": "tsp-page-pack", + "private": true, + "version": "0.1.0", + "type": "module", + "scripts": { + "build": "vite build" + }, + "dependencies": { + "react": "^18.3.1", + "react-dom": "^18.3.1" + }, + "devDependencies": { + "@types/react": "^18.3.12", + "@types/react-dom": "^18.3.1", + "@vitejs/plugin-react": "^4.3.4", + "typescript": "^5.6.3", + "vite": "^5.4.11" + } +} diff --git a/frontend/page-pack-template/src/extension.tsx b/frontend/page-pack-template/src/extension.tsx new file mode 100644 index 00000000..2cb67f01 --- /dev/null +++ b/frontend/page-pack-template/src/extension.tsx @@ -0,0 +1,86 @@ +/** + * 页面包示例 — 复制本模板改造你的页面。 + * + * 产物契约(v1): + * - default 导出清单对象; apiVersion 固定 1 + * - id 带命名空间 (如 demo.hello), 全局唯一, 与核心/其他扩展冲突会被拒 + * - routes[].path 非根绝对路径且静态 (不支持 :param / 通配) + * - mount(el, ctx): 把页面渲染进容器, 返回清理函数(可选) + * - navigation[].icon 用白名单图标名 (Sparkles/Zap/Star/... 见 docs/page-sdk.md) + * + * 样式约束: 宿主 Tailwind 按需编译, 不保证任意类名存在 — 布局关键样式 + * 用内联样式; 主题色可用宿主 CSS 变量 (如 var(--accent))。 + */ +import { useEffect, useRef, useState } from 'react' +import { createRoot } from 'react-dom/client' +import { ctx } from './tsp' + +function HelloPage() { + const [count, setCount] = useState(0) + const [serverTime, setServerTime] = useState('') + const alive = useRef(true) + + useEffect(() => { + // 经能力桥调用宿主 API (同源、自带登录态); 这里取服务端时间做演示 + ctx.api?.health?.().then((h: { status?: string } | undefined) => { + if (alive.current) setServerTime(h?.status ? `服务状态: ${h.status}` : '') + }).catch(() => {}) + return () => { alive.current = false } + }, []) + + const card: React.CSSProperties = { + maxWidth: 640, + margin: '3rem auto', + padding: '2rem', + borderRadius: 16, + border: '1px solid rgba(125,125,125,0.25)', + background: 'rgba(125,125,125,0.06)', + fontFamily: 'inherit', + } + const button: React.CSSProperties = { + height: 36, + padding: '0 14px', + borderRadius: 10, + border: 'none', + background: 'var(--accent, #3b82f6)', + color: '#fff', + fontSize: 13, + cursor: 'pointer', + } + + return ( +
+
+

页面包演示页

+

+ 这是一个自包含的运行时页面: 打包成单文件 ES 模块后上传到「设置 → 页面包」, + 刷新即出现在侧栏。{serverTime} +

+ +
+
+ ) +} + +const extension = { + id: 'demo.hello', + apiVersion: 1 as const, + routes: [ + { + id: 'hello', + path: '/demo-hello', + mount: (el: HTMLElement) => { + const root = createRoot(el) + root.render() + return () => root.unmount() + }, + }, + ], + navigation: [ + { id: 'hello', routeId: 'hello', label: '页面包演示', icon: 'Sparkles' }, + ], +} + +export default extension diff --git a/frontend/page-pack-template/src/tsp.ts b/frontend/page-pack-template/src/tsp.ts new file mode 100644 index 00000000..d9ba8903 --- /dev/null +++ b/frontend/page-pack-template/src/tsp.ts @@ -0,0 +1,18 @@ +/** + * tsp.ts — 宿主能力桥读取。 + * + * window.__TSP_EXT__ 由宿主 main.tsx 在引导时安装; 页面包经 mount(el, ctx) + * 收到的 ctx 即本对象。api 为宿主完整 API 封装(同源 fetch, 自带登录态), + * QK 为集中查询键, cn 为样式合并工具, format 为格式化工具集。 + * + * 自包含说明: 页面包产物自带 React/ReactDOM (由本模板的 vite 构建打入), + * 与宿主不共享模块实例 —— 请勿 import 宿主内部模块, 只经由 ctx 取能力。 + */ +type Bridge = { + api: any + QK: any + cn: (...inputs: any[]) => string + format: any +} + +export const ctx = (window as unknown as { __TSP_EXT__: Bridge }).__TSP_EXT__ diff --git a/frontend/page-pack-template/vite.config.ts b/frontend/page-pack-template/vite.config.ts new file mode 100644 index 00000000..949c907c --- /dev/null +++ b/frontend/page-pack-template/vite.config.ts @@ -0,0 +1,26 @@ +import { defineConfig } from 'vite' +import react from '@vitejs/plugin-react' + +// 页面包构建: 单文件自包含 ES 模块 (react 等全部打入), 产物 dist/extension.js +// 直接上传到宿主「设置 → 页面包」。不做代码分割 — 宿主按整个文件加载挂载。 +export default defineConfig({ + plugins: [react()], + // lib 模式不会自动替换依赖里的 process.env.NODE_ENV, 浏览器无 process 全局 + define: { + 'process.env.NODE_ENV': JSON.stringify('production'), + }, + build: { + outDir: 'dist', + lib: { + entry: 'src/extension.tsx', + formats: ['es'], + }, + rollupOptions: { + output: { + entryFileNames: 'extension.js', + chunkFileNames: '[name].js', + assetFileNames: '[name][extname]', + }, + }, + }, +}) diff --git a/frontend/public/donate-wechat.jpg b/frontend/public/donate-wechat.jpg new file mode 100644 index 00000000..70cb1349 Binary files /dev/null and b/frontend/public/donate-wechat.jpg differ diff --git a/frontend/src/components/Layout.tsx b/frontend/src/components/Layout.tsx index d1b689ae..c910ad8c 100644 --- a/frontend/src/components/Layout.tsx +++ b/frontend/src/components/Layout.tsx @@ -21,6 +21,7 @@ import { useToggleRealtimeQuotes, } from '@/lib/useSharedMutations' import { QK } from '@/lib/queryKeys' +import { compareByDefaultNav, defaultNavIndex, seedHiddenForNewUser } from '@/lib/navOrder' import { Settings, DatabaseZap, @@ -40,6 +41,7 @@ import { PanelLeftClose, PanelLeftOpen, Download, + HardDriveDownload, } from 'lucide-react' import { IconDashboard, @@ -47,18 +49,7 @@ import { IconStrategy, IconFactors, IconBacktest, - IconStockFocus, - IconLadder, - IconConcept, - IconIndustry, - IconFinancials, - IconMonitor, IconRegime, - IconAlert, - IconLots, - IconPaper, - IconSignals, - IconReview, IconIndices, IconData, type BrandIconProps, @@ -96,18 +87,7 @@ const nav = [ { to: '/screener', label: '策略', icon: IconStrategy }, { to: '/factors', label: '因子', icon: IconFactors }, { to: '/backtest', label: '回测', icon: IconBacktest }, - { to: '/stock-analysis', label: '个股分析', icon: IconStockFocus }, - { to: '/limit-ladder', label: '连板梯队', icon: IconLadder }, - { to: '/concept-analysis', label: '概念分析', icon: IconConcept }, - { to: '/industry-analysis', label: '行业分析', icon: IconIndustry }, - { to: '/financials', label: '财务分析', icon: IconFinancials }, - { to: '/monitor', label: '监控中心', icon: IconMonitor }, { to: '/regime', label: '市场环境', icon: IconRegime }, - { to: '/abnormal', label: '异动监控', icon: IconAlert }, - { to: '/lots', label: '持仓提醒', icon: IconLots }, - { to: '/paper', label: '模拟盘', icon: IconPaper }, - { to: '/signals', label: '信号库', icon: IconSignals }, - { to: '/review', label: '复盘', icon: IconReview }, { to: '/indices', label: '指数', icon: IconIndices }, { to: '/data', label: '数据', icon: IconData }, ] as const @@ -172,12 +152,15 @@ function SidebarIndexQuotes({ rows, items }: { rows: IndexQuote[] | undefined; i + {/* 名称不换行 + 涨跌幅不压缩: 两列布局下单卡内容区仅 85px, + 「上证指数」41px + 涨跌幅 36px 已接近占满, 任其收缩或换行会把 + 指数名折成两行 (曾出现「上证指/数」)。px-1.5 是为此留出的余量。 */}
- {item.name} - {fmtIndexPct(pct)} + {item.name} + {fmtIndexPct(pct)}
{fmtIndexValue(value)} @@ -189,6 +172,30 @@ function SidebarIndexQuotes({ rows, items }: { rows: IndexQuote[] | undefined; i ) } +// ===== 头部快捷行: 数据同步入口 (描边分段按钮, 与扩展快捷方式并排成组) ===== +// 点击直达 /data; 同步中转圈 / 刚完成绿勾, 空闲无角标保持安静。 +function QuickDataEntry({ syncing, justDone }: { syncing: boolean; justDone: boolean }) { + return ( + + {syncing + ? + : } + 数据 + {!syncing && justDone && } + + ) +} + // ===== 数据源能力健康卡 ===== // 能力路由架构下的侧栏状态: 不再展示「主数据源 + TickFlow 档位」(单源时代遗留 — // 五个能力各自路由, 拿日K的源代表全局是随意的), 改为回答「各能力当前是否都有源在供」。 @@ -250,8 +257,7 @@ function DataSourceHealthBadge({ matrix }: { matrix: CapabilityMatrix | undefine onKeyDown={e => { if (e.key === 'Escape') setPopPos(null) }} className="group relative flex items-center gap-2 overflow-hidden rounded-md py-1.5 pl-2.5 pr-2 transition-colors duration-150 hover:bg-elevated/70" > - - + {/* 能力方格 (按注册顺序: 实时/日K/分钟/除权/财务), 与悬浮卡逐格同色对应 */} {loading @@ -334,8 +340,7 @@ function AIConfigBadge({ configured, model }: { configured?: boolean; model?: st className="group relative flex items-center gap-2 overflow-hidden rounded-md py-1.5 pl-2.5 pr-2 transition-colors duration-150 hover:bg-elevated/70" title={`AI 配置 — ${descText}`} > - - + {configured ? ( {model || '已接入模型'} @@ -395,8 +400,9 @@ export function Layout() { // 持久化到 localStorage; 迁移旧两态键 tf-nav-collapsed (收起 → 图标条)。 const [navState, setNavState] = useState(() => { try { + // 两态导航: expanded(14rem) / rail(3.5rem 图标条)。历史三态版本的 'hidden' 归一为展开 const v = localStorage.getItem('tf-nav-state') - if (v === 'expanded' || v === 'rail' || v === 'hidden') return v + if (v === 'expanded' || v === 'rail') return v return localStorage.getItem('tf-nav-collapsed') === '1' ? 'rail' : 'expanded' } catch { return 'expanded' } }) @@ -478,9 +484,9 @@ export function Layout() { // 开启实时行情时若存在排队中的挖掘任务 → 确认弹窗 (实时落盘会让排队任务开跑即失败) const [miningQueuedWarning, setMiningQueuedWarning] = useState(null) const miningWarnBackdrop = useDialogBackdrop(() => setMiningQueuedWarning(null)) - // 三态循环切换 (仅桌面): 展开 → 图标条 → 隐藏 → 展开 + // 两态切换 (仅桌面): 展开 <-> 图标条 const toggleNavCollapsed = () => { - setNavStatePersist(navState === 'expanded' ? 'rail' : navState === 'rail' ? 'hidden' : 'expanded') + setNavStatePersist(navState === 'expanded' ? 'rail' : 'expanded') } // 指数条: 固定核心四只 (产品契约, 不再可配置), 常驻显示 const sidebarIndexes = CORE_INDEXES @@ -556,15 +562,18 @@ export function Layout() { // 合并内置页面 + 可见的扩展分析菜单 type NavIcon = (props: BrandIconProps) => ReactNode - type NavItem = { to: string; label: string; icon: NavIcon; badge?: string } + // arch: 'ext' = 扩展域页面 (官方扩展/自定义分析), 核心域 nav 常量不带标。 + // 与菜单设置 ARCH_CLASS 同口径: nav 常量 8 项全部核心, 其余两个来源全部扩展。 + type NavItem = { to: string; label: string; icon: NavIcon; badge?: string; arch?: 'core' | 'ext' } const analysisNav: NavItem[] = (analysisMenus?.items ?? []) .filter(m => m.visible) - .map(m => ({ to: `/analysis/${m.id}`, label: m.label, icon: m.icon === 'tags' ? Tags : BarChart3 })) + .map(m => ({ to: `/analysis/${m.id}`, label: m.label, icon: m.icon === 'tags' ? Tags : BarChart3, arch: 'ext' as const })) const extensionNav: NavItem[] = getFrontendExtensionNavigation().map(item => ({ to: item.route.path, label: item.label, icon: item.icon, badge: item.badge, + arch: 'ext' as const, })) const allNav: NavItem[] = [...nav, ...analysisNav, ...extensionNav] @@ -580,26 +589,51 @@ export function Layout() { const merged = [...ordered] for (const item of allNav) { if (seen.has(item.to)) continue - // 未保存过排序的新条目: 内置页插回默认位置(排在已保存的默认前驱之后), - // 分析/扩展菜单仍追加到末尾 - const defaultIndex = nav.findIndex(n => n.to === item.to) + // 未保存过排序的新条目: 默认基准内的路径插回默认位置(排在已出现的 + // 最近默认前驱之后), 分析菜单/未知扩展追加到末尾。基准含迁移到 + // custom/ 的扩展域页面, 保证老用户升级无感 (lib/navOrder)。 + const di = defaultNavIndex(item.to) let anchor = -1 - if (defaultIndex > 0) { - for (let i = defaultIndex - 1; i >= 0 && anchor < 0; i -= 1) { - anchor = merged.findIndex(n => n.to === nav[i].to) + if (Number.isFinite(di) && di > 0) { + for (let i = di - 1; i >= 0 && anchor < 0; i -= 1) { + anchor = merged.findIndex(n => defaultNavIndex(n.to) === i) } } if (anchor >= 0) merged.splice(anchor + 1, 0, item) - else if (defaultIndex >= 0) merged.unshift(item) + else if (di === 0) merged.unshift(item) else merged.push(item) } return merged })() - : allNav + // 无保存排序 (新装/从未调整过菜单): 按迁移前默认基准排序, 扩展域页面 + // 不沉底 — 老用户升级侧栏顺序与迁移前一致; 未知路径稳定追加在后。 + : [...allNav].sort((a, b) => compareByDefaultNav(a.to, b.to)) const hiddenIds = new Set(prefs?.nav_hidden ?? []) const visibleNavItems = navItems.filter(n => !hiddenIds.has(n.to) && !hiddenIds.has(n.to.replace(/^\/analysis\//, ''))) + // 新装用户一次性种子: 扩展域页面默认隐藏, 菜单设置里可逐个开启。 + // 老用户 (升级前已完成 onboarding) 只写标记不动偏好, 升级前后侧栏一致。 + // 种子标记存 localStorage (每设备一次, 保存成功后才写); 隐藏偏好本体持久化在后端 nav_hidden。 + const extSeedRef = useRef(false) + useEffect(() => { + if (!prefs || !settingsState || extSeedRef.current) return + let seeded = false + try { seeded = localStorage.getItem('tf-ext-pages-seed') === '1' } catch { return } + if (seeded) { extSeedRef.current = true; return } + const next = seedHiddenForNewUser(settingsState.onboarding_completed, prefs.nav_hidden) + if (!next) { + extSeedRef.current = true + try { localStorage.setItem('tf-ext-pages-seed', '1') } catch {} + return + } + extSeedRef.current = true + api.saveNavHidden(next) + .then(() => { try { localStorage.setItem('tf-ext-pages-seed', '1') } catch {} }) + .then(() => qc.invalidateQueries({ queryKey: QK.preferences })) + .catch(() => { /* 保存失败不写标记, 下次启动重试; 结果集幂等, 重复触发无害 */ }) + }, [prefs, settingsState, qc]) + const doEnableRealtime = async () => { await toggleQuote.mutateAsync(true) // 仅在交易时段立即获取一次行情 @@ -704,10 +738,7 @@ export function Layout() { style={{ color: BRAND }} /> {!railMode && ( -
+
Tick Stock Panel
)} @@ -719,7 +750,7 @@ export function Layout() { 'flex items-center rounded-btn text-muted hover:text-foreground hover:bg-elevated/60 transition-colors duration-150 ease-smooth', railMode ? 'justify-center p-1.5' : 'ml-auto p-1.5', )} - title={railMode ? '隐藏菜单 (再点击左缘按钮可唤出)' : '收起菜单'} + title={railMode ? '展开菜单' : '收起菜单'} > {railMode ? @@ -741,17 +772,22 @@ export function Layout() { {!railMode && (
- -