mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 15:37:44 +00:00
feat: add LLM-finetune scenario (#1314)
* refine prompt * small update * fix a small bug * remove debug config after execution * fix: only remove <think> at start * feat: support creating dataset & multi-eval frame (#1302) * feat: add iterative evolve and evaluation support with partial chain stop * feat: add FTDataEvaluator and support multiple implement functions in finetune * feat: data implement for pre-proposal and proposal and add datasets (#1303) * feat:(1) support for multi layer dataset extraction (2) add category.json for dataset in datasets/ * fix: fix bug for generate category.json * feat: add get_dataset_folder_desc * init data proposal and merge qzli/ft * update data proposal prompts and add max_position_embeddings and resolve confilcts * remove sample counts in data proposal * turn data and train to unified hypo_gen * refine prompts * remove category.json and add it to dataset_info * fix jinja problem and proposal done * lint * add ai-generated description and raw readme into dataset_info.json * update prompt for description * add datasets * initial fix for proposal of data * final version for data proposal * lint * feat: add stats in dataset_info, and enable data coder (#1306) * refactor(dataset): add stats into dataset_info.json, and remove dataset from gitignore_folder * feat: enable data coder and run data process * feat: Merge data coder (#1307) * feat: implement finetune data coding, evaluation, and config improvements * fix: deepspeed config path * fix: dataset info columns --------- Co-authored-by: Young <afe.young@gmail.com> * replace str length with token_limit * add readme to dataset_info and remove useless blank lines in scenario description * feat: dataset prepare * fix: extract prams script name * feat: add loss&predictions samples to feedback * remove duplicate envs and and add llm_api_preferences and enhance reasoning token limits * feat: network for ft_env * fix: remove gpt-4o, which has low quota * feat: a simple ui * feat: merge data and train task type (#1309) * feat: filter redundant prams of lf * fix: ui bug caused by removing task_type * fix: force agent to use high concurrency, and remove redundant prompt * feat: extract info from llama factory log, and check data exists before download * fix: add compatibility rules * feat: llm evaluator for data coder * feat: openai package in ft docker, and refine prompt * feat: refine ft ui, add more info * feat: add raw logs * refine data coder prompt(for feedback debug) * feat: select dataset in scen init * fix: ui for docker log seperately * feat: sync log through blob * improve ui, and add llm feedback in Runner&Exp2FB (#1312) * fix: ui bug to visualize docker log, and lint * feat: unified docker log for ft env, and some refactor * fix bugs and improve ui * feat: save log of evaluator(single feedback) * feat: add evaluator, set cleanup docker log * feat: call llm in RunnerEvaluator and Feedback * fix: extract structured error message in RunnerEvaluator * feat: feedback improve, and fix some bugs * feat: feedback improve when runner fails * small update * feat(UI): add running info and benchmark metric in loop expander * feat(UI): add render markdown toggle * feat: refine prompts and add error type in exp2fb * feat: add filterd params reason, set default benchmark timeout to infinite, and refine train loss express * recover dataset deepscaler * feat: set timeout in .env * refactor: unifiied ft_env timeout * feat: debug mode for data coder * feat: deliver data_stats after generate debug_data * feat: use gpt-5.1 as judge model, set judge_retry, and refine debug mode prompt * refine prompt * refactor: llama factory manager logic, and refine data processing prompt * feat(DockerEnv): support GPU selection via CUDA_VISIBLE_DEVICES * feat: set api concurrency via .env * fix: ft env timeout bug * feat: enable CondaEnv run * fix: can't update bin path in first run, and path bug in lf manager * feat(ui): set log path through .env * refactor(ui): wrap_lines, remove css * feat(coder): retry when parse code-block fail * fix: refine single-fb in ui, and fix path bug(not allow proposal to decide path) * fix: opencompass CondaEnv torch compatible with vllm * fix: refine error text in coding * feat: deepspeed config for CondaEnv * feat: memory estimator * fix: deepspeed package for condaenv * fix: use `client.chat.completions.create()` only * feat: flash attention for condaenv * feat: strong and weak models interface * fix: condaenv package dependency * use multi round conversation in llm finetune proposal * refine prompt for data processing * enable evolving in data coder * maximize output token size * fix: refine ui * fix: optional packages for llama factory * fix: torch denpendency for b200 * fix: opencompass dependency * update cot prompts * skip the sub implement * skip conda preparation if env exists * update chemcot datasets * fix: unify docker to use litellm * update readme and instructions * fix: set CUDA_VISIBLE_DEVICES for CondaEnv * feat: add panorama dataset, refactor dataset interface * feat: calculate token using tiktoken, and ndarray bug * fix: download subtasks of chemcotdataset seperately * feat: customized prepare func for datasets * feat: update new benchmarks * add datasets package * docs: readme for llm finetune * feat: download raw data directly, with post-process function * feat: analyze raw dataset * suppress litellm debug info * feat(ui): summary page * feat: run multi-jobs * feat: improve ui * feat: add path and checkout options to LLM finetune loop entrypoint * feat: add FinanceIQ_ppl benchmark with auto-download and dataset desc rendering * refactor: remove unused imports and dead code, fix session folder logging * feat: enable tablebench and tableInstruct dataset * refine dataset readme, and coder prompt * refine proposal and coder prompt * fix: ui path (default log path) * feat: add automatic LoRA model merging for benchmarking with vLLM * refactor: reorganize finetune benchmark and merge modules under benchmark dir * refactor: modularize benchmark config and error extraction for finetune scenario * fix: update benchmark import paths and disable env cache for device info * refactor docke&conda env and fix import bugs * modify init python file * feat: add FinanceIQ dataset split utility and integrate with pipeline * feat: set weak and strong model by env, distribute workload across models * feat: sample dataset and rm params for tensorboard, wandb * update script to run jobs * refine proposal prompt, remove specific dataset name * fix(ui): auto switch log folder * fix: estimate the processed full data after sample * feat: filter raw data more aggressively, and lower data_eval standard * feat: sync workspace to blob * feat: rdkit for chemcotbench * update qwen2.5&llama3.1 context * fix: force failure on validation error and remove try/except in validator * feat: unified error sample extraction (with test scripts) * feat: set conda cache with .env * feat: skip data eval if data pass in last evo * fix: rm redundant param * fix ui bug * refactor: centralize assign_code_list_to_evo in MultiProcessEvolvingStrategy * feat: add test_params.yaml generation and workspace cleanup improvements for finetune * refactor: replace get_clear_ws_cmd with clear_workspace and update prompts for hard check criteria * add bioprobench dataset * fix: handle commas in training config extraction and refactor prompt includes * bioprobench description * add bioprobench readme * feat: merge lora adapter for blackwell gpu * feat: support for multi benchmarks in one job * change dfficult aware content for training * update difficulty-aware and logging principles * fix: resolve variable name conflict in FTRunnerEvaluator * set job id accuracy to minute * feat(ui): display one selected metric per benchmark * feat: store sota exp, and fix ws_ckp bug * fix: truncate data.json in feedback * fix: opencompass data for conda env * fix: save only the last model * feat: set log path and ws path * fix: set overwrite_cache to avoid lock contention(through injecting params) * feat: redirect stdout to file in localenv * add pickle cache to dataset desc * fix CI * fix: remove redundant wrapper * feat: set python_unbuffered * move redirect stdout to env run * fix a small bug * move model folder * feat(ui): display benchmark baseline * fix: enrich scenario and benchmark description * fix: rewrite runner eval to accept easier * feat: compare with baseline when no SOTA * update tablebench readme * fix: switch back to single benchmark (for baseline) * feat(ui): add ws path in ui * refactor: update SOTA tracking to use DAG traversal and parent selection * fix: prioritize local_selection in trace and refactor sibling retrieval logic * refactor: unify error handling in feedback generation and update workspace injection * feat: add skip_loop_error_stepname to control error skip step in LoopBase * fix: set local_selection to NEW_ROOT for experiments without parent * feat: set different ports for jobs * feat: set different ports for jobs * feat: add upper data size limit for LLM fine-tuning and update related prompts * fix: replace get_truncated_stdout() with stdout for consistent output handling * refactor: remove data.json from cache and workspace logic, focus on script-based reuse * fix: rm target_scenario * feat: add selective cache extraction and custom cache key for data processing * fix(ui): bug when displaying tablebench * fix: filter config in dataset_info.json * feat: add test set, set valid set * feat(ui): update test score, and set color for final decision * feat: add test score for baseline and update ui * fix: use [-100:] as test range * feat: update data_stats in runner * feat: wait for opencompass init when run multi jobs * fix: adjust test&valid split * feat: force to generate COT(with <think> token), and add answer format in scenarios.json * feat: improve ui * fix: unify benchmark volume mounts and set extra_volumes for conda env * fix(ui): number color * fix: update GPU memory handling to use total memory in GB and streamline code * fix: set use_cot_postprocessor * feat: add env_dict to config classes and merge env vars in Env run * fix: let coder obey proposal * fix(ui): direction bug and update chemcot core metirc * fix: set consistent benchmark mount points and env vars for docker and conda * fix: addintional target for LoRA * feat: workspace dir log for benchmark running * fix: tableInstruct path bug and update benchmark description * feat: timeout for whole job * fix: align FinanceIQ import to opencompass * feat: use llm_judge for FinanceIQ * feat: switch to turn on <think> or not * feat: using scripts to redirect stdout, and run in different windows * feat: sync litellm log * fix: gpu memory format * fix: escape special characters in benchmark desc * fix: set data processing timeout to 1h * feat: set valid_loss and save_best_model * fix: inject timeout and stage * fix: loss history extract logic * feat: inject output dir * feat: inject eval batch size * feat: inject save_total_limit * feat: update data prompt * fix: escape shell special characters * fix: tablebench visualization UI * fix: move implementation validation to coder, and ignore injected params * feat: README for FinanceIQ dataset * fix: bioprobench desc error * fix: remove task alignment when coder eval * fix: FinanceIQ now extracts last capital as answer * fix: stdout contains binary data * feat: recover estimate full output and set eval setting automatically * fix(ui): precision for summary table * fix(ui): import error * feat: try to use lora * fix(api): fix litellm bug for code block * fix: refine prompts to give agent more decision space * chore(ci): fix mypy typing issues * chore(ci): format code with black * chore(ci): fix ruff lint violations * chore(ci): sort imports with isort * chore(ci): format code with black * test: temporarily skip extract_parameters imports due to numpy pin * fix: compatibility issues for qlib scenarios on finetune branch * fix(fin_factor): skip to fb for coder error * fix(loop): default skip to feedback step on skip_loop_error When skip_loop_error exception happens and skip_loop_error_stepname is not explicitly set, default to jumping to 'feedback' step if it exists, otherwise fall back to the last step (record). This prevents KeyError when record step tries to access feedback data that doesn't exist because we skipped the feedback phase. Also removed redundant skip_loop_error_stepname from finetune loop since it's now the default behavior. * add 'skip to record' to DS scenario like other scenarios * fix 2 scenarios bug about rd_loop class * fix: lint(mypy, ruff, black) error * fix: mypy lint error * fix data science scenario bug --------- Co-authored-by: Xu Yang <peteryang@vip.qq.com> Co-authored-by: Qizheng Li <jenssenlee@163.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: amstrongzyf <201840057@smail.nju.edu.cn> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: amstrongzyf <amstrongzyf@126.com> Co-authored-by: chelsea97 <zhuowbrown@gmail.com> Co-authored-by: SunsetWolf <Lv.Linlang@hotmail.com>
This commit is contained in:
XianBW
@@ -55,5 +55,7 @@ EMBEDDING_MODEL="litellm_proxy/BAAI/bge-large-en-v1.5"
|
||||
# Cache Setting (Optional):
|
||||
# USE_CHAT_CACHE=True
|
||||
# USE_EMBEDDING_CACHE=True
|
||||
# FT_DOCKER_ENABLE_CACHE=True
|
||||
# DS_DOCKER_ENABLE_CACHE=True
|
||||
# Senario Configs:
|
||||
# ==========================================
|
||||
+3
-1
@@ -182,4 +182,6 @@ static/
|
||||
# AI assistant
|
||||
.cursor/
|
||||
.claude/
|
||||
AGENTS.md
|
||||
AGENTS.md
|
||||
|
||||
scripts/
|
||||
@@ -99,7 +99,7 @@ mypy:
|
||||
# First deal with the core folder, and then gradually increase the scope of detection,
|
||||
# and eventually realize the detection of the complete project.
|
||||
ruff:
|
||||
$(PIPRUN) ruff check rdagent/core --ignore FBT001,FBT002,I001 # --exclude rdagent/scripts,git_ignore_folder
|
||||
$(PIPRUN) ruff check rdagent/core --ignore FBT001,FBT002,I001,E501 # --exclude rdagent/scripts,git_ignore_folder
|
||||
|
||||
# Check lint with toml-sort.
|
||||
toml-sort:
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
# LLM Fine-tuning (FT) 场景运行指南
|
||||
|
||||
本文档介绍如何运行 RD-Agent 的 LLM Fine-tuning 场景。
|
||||
|
||||
## 简介
|
||||
|
||||
FT 场景用于自动化优化大语言模型在特定 benchmark 上的表现。系统会自动：
|
||||
1. 生成数据处理和训练代码
|
||||
2. 执行模型微调
|
||||
3. 在目标 benchmark 上评估模型性能
|
||||
4. 根据反馈迭代改进
|
||||
|
||||
## 支持的 Benchmark
|
||||
|
||||
| 类别 | Benchmark | 数据集 | 描述 |
|
||||
|------|-----------|--------|------|
|
||||
| Math | `aime24`, `aime25` | `deepscaler` | AIME 数学竞赛 |
|
||||
| Patent | `panorama_par4pc` | `panorama-par4pc` | 专利现有技术检索 |
|
||||
| Patent | `panorama_pi4pc` | `panorama-pi4pc` | 专利段落识别 |
|
||||
| Patent | `panorama_noc4pc` | `panorama-noc4pc` | 专利新颖性分类 |
|
||||
| Chemistry | `chemcotbench_mol_und` | `chemcot-mol_und` | 分子理解 |
|
||||
| Chemistry | `chemcotbench_mol_edit` | `chemcot-mol_edit` | 分子编辑 |
|
||||
| Chemistry | `chemcotbench_mol_opt` | `chemcot-mol_opt` | 分子优化 |
|
||||
| Chemistry | `chemcotbench_reaction` | `chemcot-rxn` | 化学反应预测 |
|
||||
|
||||
> 数据集配置位于 `rdagent/scenarios/finetune/datasets/__init__.py` 的 `DATASETS` 字典中。
|
||||
|
||||
>运行时agent会查看所有数据集，根据target benchmark和scenario选出与之相关的。
|
||||
|
||||
## 环境配置
|
||||
|
||||
### 1. 运行环境
|
||||
|
||||
确保已安装 `rdagent` 主运行环境，其他需要的运行环境会自动创建
|
||||
|
||||
> 在 `.env` 配置文件中通过设置 `FT_Coder_CoSTEER_env_type = conda/docker` 来配置
|
||||
|
||||
### 2. .env 配置文件
|
||||
|
||||
在项目根目录创建 `.env` 文件，参考以下模板：
|
||||
|
||||
```bash
|
||||
# ========== API Configuration ==========
|
||||
BACKEND=rdagent.oai.backend.LiteLLMAPIBackend
|
||||
CHAT_MODEL=gpt-5.2
|
||||
CHAT_TEMPERATURE=1
|
||||
CHAT_STREAM=True
|
||||
OPENAI_API_KEY=sk-xxx
|
||||
OPENAI_API_BASE=http://your-api-endpoint
|
||||
|
||||
EMBEDDING_MODEL=text-embedding-ada-002
|
||||
EMBEDDING_USE_AZURE=True
|
||||
|
||||
# ========== Global Configs ==========
|
||||
MAX_RETRY=12000
|
||||
RETRY_WAIT_SECONDS=5
|
||||
MULTI_PROC_N=16
|
||||
STEP_SEMAPHORE=1
|
||||
|
||||
# ========== Cache Settings ==========
|
||||
DUMP_CHAT_CACHE=False
|
||||
USE_CHAT_CACHE=False
|
||||
DUMP_EMBEDDING_CACHE=True
|
||||
USE_EMBEDDING_CACHE=True
|
||||
LOG_LLM_CHAT_CONTENT=True
|
||||
|
||||
CHAT_FREQUENCY_PENALTY=0.1
|
||||
CHAT_PRESENCE_PENALTY=0.0
|
||||
|
||||
# ========== FT Scenario Specific ==========
|
||||
FT_FILE_PATH=/path/to/your/finetune/workspace
|
||||
|
||||
# Environment type: docker or conda
|
||||
# Set to "conda" when Docker is unavailable
|
||||
FT_Coder_CoSTEER_env_type=conda
|
||||
|
||||
# Docker settings (only used when env_type=docker)
|
||||
FT_DOCKER_ENABLE_CACHE=True
|
||||
FT_UPDATE_LLAMA_FACTORY=False
|
||||
|
||||
# Data processing API concurrency (adjust based on target API capacity)
|
||||
FT_API_MAX_WORKERS=1000
|
||||
|
||||
# Data processing Model
|
||||
FT_STRONG_MODELS='["gpt-5", "gpt-5.1"]'
|
||||
FT_WEAK_MODELS='["gpt-4o-mini"]'
|
||||
|
||||
# Benchmark and target (can be overridden in script)
|
||||
FT_TARGET_BENCHMARK=aime25
|
||||
FT_USER_TARGET_SCENARIO="I need to enhance the model's performance on math reasoning tasks."
|
||||
|
||||
# Timeout settings
|
||||
FT_DATA_PROCESSING_TIMEOUT=28800
|
||||
|
||||
# Judge settings (optional)
|
||||
# FT_JUDGE_MODEL=gpt-5.1
|
||||
# FT_JUDGE_RETRY=10
|
||||
|
||||
REASONING_THINK_RM=True
|
||||
|
||||
# ========== Logging ==========
|
||||
LOG_FORMAT_CONSOLE="{time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | <cyan>{process}</cyan> | {name}:{function}:{line} - {message}"
|
||||
|
||||
# ========== HuggingFace ==========
|
||||
HF_TOKEN=hf_xxx
|
||||
```
|
||||
|
||||
## 运行方法
|
||||
|
||||
### 基本命令
|
||||
|
||||
```bash
|
||||
# 激活 conda 环境
|
||||
conda activate rdagent
|
||||
|
||||
# 运行 FT 场景
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py --base-model <MODEL>
|
||||
```
|
||||
|
||||
### 命令行参数
|
||||
|
||||
| 参数 | 说明 | 示例 |
|
||||
|------|------|------|
|
||||
| `--base-model` | 基础模型名称（必需，其他都可以不填） | `Qwen/Qwen2.5-7B-Instruct` |
|
||||
| `--benchmark` | 目标 benchmark | `aime25` |
|
||||
| `--benchmark-description` | Benchmark 描述 | - |
|
||||
| `--dataset` | 指定数据集 | - |
|
||||
| `--step-n` | 步数限制 | `10` |
|
||||
| `--loop-n` | 循环次数限制 | `5` |
|
||||
| `--timeout` | 总时间限制 | - |
|
||||
|
||||
### 运行示例
|
||||
|
||||
```bash
|
||||
# 在 AIME25 上微调 Qwen2.5-7B
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py \
|
||||
--base-model Qwen/Qwen2.5-7B-Instruct
|
||||
|
||||
# 指定 GPU 运行
|
||||
CUDA_VISIBLE_DEVICES=0,1 dotenv run -- python rdagent/app/finetune/llm/loop.py \
|
||||
--base-model Qwen/Qwen2.5-7B-Instruct
|
||||
|
||||
# 限制循环次数
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py \
|
||||
--base-model Qwen/Qwen2.5-7B-Instruct \
|
||||
--loop-n 3
|
||||
```
|
||||
|
||||
### 多任务并行运行
|
||||
|
||||
创建 `tasks.json` 配置文件：
|
||||
```json
|
||||
{
|
||||
"tasks": [
|
||||
{"model": "Qwen/Qwen2.5-7B-Instruct", "benchmark": "aime25", "gpus": "0,1"},
|
||||
{"model": "Qwen/Qwen2.5-7B-Instruct", "benchmark": "gsm8k", "gpus": "2,3"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
使用 `run_ft_deploy.sh` 脚本运行：
|
||||
```bash
|
||||
./run_ft_deploy.sh tasks.json # 正常运行
|
||||
./run_ft_deploy.sh tasks.json --dry-run # 仅预览配置
|
||||
./run_ft_deploy.sh tasks.json --no-sync # 禁用 blob 同步
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>run_ft_deploy.sh 脚本参考</summary>
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# 多任务并行部署脚本（简化版）
|
||||
|
||||
RDAGENT_DIR="$HOME/RD-Agent"
|
||||
ENV_TEMPLATE=".env.ft"
|
||||
STAGGER_DELAY=60
|
||||
|
||||
cd "$RDAGENT_DIR"
|
||||
source ~/miniconda3/etc/profile.d/conda.sh
|
||||
conda activate rdagent
|
||||
|
||||
CONFIG_FILE="${1:-tasks.json}"
|
||||
NUM_TASKS=$(jq '.tasks | length' "$CONFIG_FILE")
|
||||
|
||||
for ((i=0; i<NUM_TASKS; i++)); do
|
||||
model=$(jq -r ".tasks[$i].model" "$CONFIG_FILE")
|
||||
benchmark=$(jq -r ".tasks[$i].benchmark" "$CONFIG_FILE")
|
||||
gpus=$(jq -r ".tasks[$i].gpus" "$CONFIG_FILE")
|
||||
|
||||
# 更新 .env 中的 benchmark
|
||||
cp "$ENV_TEMPLATE" .env
|
||||
sed -i "s|^FT_TARGET_BENCHMARK=.*|FT_TARGET_BENCHMARK=$benchmark|" .env
|
||||
|
||||
CUDA_VISIBLE_DEVICES=$gpus \
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py --base-model "$model" &
|
||||
|
||||
# 首个任务等待环境创建，后续任务错开启动
|
||||
[[ $i -eq 0 ]] && sleep 120 || sleep $STAGGER_DELAY
|
||||
done
|
||||
|
||||
wait
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Blob 日志同步
|
||||
|
||||
使用 Azure Blob 在多台机器间同步日志文件。
|
||||
|
||||
### 1. 生成 SAS Token
|
||||
|
||||
```bash
|
||||
# 首先登录 Azure CLI
|
||||
az login
|
||||
|
||||
# 生成 Token（默认有效期 7 天）
|
||||
bash rdagent/utils/blob/gen_token.sh
|
||||
|
||||
# 或指定过期时间
|
||||
bash rdagent/utils/blob/gen_token.sh 2025-01-31T00:00Z
|
||||
```
|
||||
|
||||
Token 会保存到 `git_ignore_folder/.az_sas_token`。
|
||||
|
||||
### 2. 同步日志
|
||||
|
||||
同步路径：`log/` ↔ `blob://epeastus/rdagent/FinetuneAgenticLLM/FT_qizheng/logs`
|
||||
|
||||
```bash
|
||||
# 上传本地日志到 Blob
|
||||
bash rdagent/utils/blob/azsync.sh up
|
||||
|
||||
# 从 Blob 下载日志到本地
|
||||
bash rdagent/utils/blob/azsync.sh down
|
||||
```
|
||||
|
||||
> 如需修改远程路径，编辑 `rdagent/utils/blob/azsync.sh` 中的 `REMOTE_PATH` 变量。
|
||||
|
||||
## 日志查看
|
||||
|
||||
运行日志保存在 `log/` 目录下：
|
||||
|
||||
```
|
||||
log/
|
||||
└── 2025-01-01_12-00-00-123456/
|
||||
├── Loop_0/
|
||||
│ ├── direct_exp_gen/ # 假设生成
|
||||
│ ├── coding/ # 代码生成
|
||||
│ ├── running/ # 训练执行
|
||||
│ └── feedback/ # 反馈总结
|
||||
└── Loop_1/
|
||||
└── ...
|
||||
```
|
||||
|
||||
|
||||
@@ -1,43 +1,125 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import SettingsConfigDict
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS, ExtendedBaseSettings
|
||||
from rdagent.core.conf import ExtendedBaseSettings
|
||||
|
||||
|
||||
class LLMFinetuneScen(ExtendedBaseSettings):
|
||||
class LLMFinetunePropSetting(ExtendedBaseSettings):
|
||||
"""LLM Fine-tune dedicated property settings.
|
||||
|
||||
- Adjust timeouts and template
|
||||
- Use FT_ env prefix for overrides
|
||||
"""
|
||||
|
||||
model_config = SettingsConfigDict(env_prefix="FT_", protected_namespaces=())
|
||||
scen: str = "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
|
||||
"""
|
||||
Scenario class for data science tasks.
|
||||
- For Kaggle competitions, use: "rdagent.scenarios.data_science.scen.KaggleScen"
|
||||
- For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
|
||||
- For LLM finetune scenarios, use: "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
|
||||
- For Data science finetune scenarios, use: "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
|
||||
|
||||
# Main Components
|
||||
scen: str = "rdagent.scenarios.finetune.scen.scenario.LLMFinetuneScen"
|
||||
"""Scenario class for LLM fine-tuning tasks."""
|
||||
|
||||
hypothesis_gen: str = "rdagent.scenarios.finetune.proposal.proposal.LLMFinetuneExpGen"
|
||||
"""Hypothesis generation class for LLM fine-tuning tasks."""
|
||||
|
||||
coder: str = "rdagent.components.coder.finetune.LLMFinetuneCoSTEER"
|
||||
"""Code generator.
|
||||
Function: Generate LLM fine-tuning code based on experiment design.
|
||||
"""
|
||||
|
||||
hypothesis_gen: str = "rdagent.app.finetune.llm.proposal.FinetuneExpGen"
|
||||
"""Hypothesis generation class"""
|
||||
runner: str = "rdagent.scenarios.finetune.train.runner.LLMFinetuneRunner" # TODO
|
||||
"""Code runner.
|
||||
Function: Execute LLM fine-tuning code in a Docker environment.
|
||||
"""
|
||||
|
||||
debug_timeout: int = 36000
|
||||
"""The timeout limit for running on debugging data"""
|
||||
summarizer: str = "rdagent.scenarios.finetune.dev.feedback.FTExperiment2Feedback"
|
||||
"""Result summarizer - To be implemented.
|
||||
Function: Analyze fine-tuning results and generate feedback, including performance metrics and error analysis.
|
||||
"""
|
||||
|
||||
# Timeouts (longer for LLM training, all for Docker container timeout)
|
||||
full_timeout: int = 360000
|
||||
"""The timeout limit for running on full data"""
|
||||
"""Full training timeout in seconds (default 100 hours, env: FT_FULL_TIMEOUT). Used in running stage for complete model training."""
|
||||
data_processing_timeout: int = 3600
|
||||
"""Data processing script timeout in seconds (default 1 hour, env: FT_DATA_PROCESSING_TIMEOUT). Used for full data processing in running stage."""
|
||||
debug_data_processing_timeout: int = 1200
|
||||
"""Debug data processing timeout in seconds (default 20 minutes, env: FT_DEBUG_DATA_PROCESSING_TIMEOUT). Used for --debug mode in coding stage."""
|
||||
micro_batch_timeout: int = 1800
|
||||
"""Micro-batch test timeout in seconds (default 30 minutes, env: FT_MICRO_BATCH_TIMEOUT)."""
|
||||
|
||||
# Pipeline behavior
|
||||
coder_on_whole_pipeline: bool = True
|
||||
enable_model_dump: bool = True
|
||||
app_tpl: str = "app/finetune/llm/tpl"
|
||||
app_tpl: str = "scenarios/finetune"
|
||||
|
||||
# Benchmark evaluation (always enabled as part of evaluation pipeline)
|
||||
|
||||
benchmark_timeout: int = 0
|
||||
"""Benchmark evaluation timeout in seconds. 0 means no timeout."""
|
||||
|
||||
# Judge API configuration (for llmjudge benchmarks like AIME)
|
||||
judge_model: str = "gpt-5.1"
|
||||
"""LLM judge model name for evaluation"""
|
||||
|
||||
judge_api_key: str | None = None
|
||||
"""API key for judge model (if None, will try to use from environment)"""
|
||||
|
||||
judge_api_base: str | None = None
|
||||
"""API base URL for judge model (if None, will use default)"""
|
||||
|
||||
judge_retry: int = 10
|
||||
"""Number of retries for LLM judge API calls (env: FT_JUDGE_RETRY)"""
|
||||
|
||||
benchmark_limit: int | None = None
|
||||
"""Limit number of samples for benchmark evaluation (None for full evaluation). Use for quick testing and debugging."""
|
||||
|
||||
benchmark_num_runs: int = 1
|
||||
"""Number of times to run each sample (for computing average or pass@k). Set >1 for multiple runs."""
|
||||
|
||||
benchmark_pass_k: list[int] | None = None
|
||||
"""Pass@k parameter list for code generation tasks (e.g., [1, 5, 10]). None to disable."""
|
||||
|
||||
# Data paths and processing
|
||||
file_path: Path = Path.cwd() / "git_ignore_folder" / "finetune_files"
|
||||
show_nan_columns: bool = False
|
||||
sample_data_by_LLM: bool = True
|
||||
|
||||
# LLM-specific fields
|
||||
user_target_scenario: str | None = None
|
||||
target_benchmark: str | None = None
|
||||
"""Benchmark dataset to evaluate on. Supported: aime25, aime24, mmlu, gsm8k, math, etc."""
|
||||
benchmark_description: str | None = None
|
||||
base_model: str | None = None
|
||||
dataset: str | None = None
|
||||
upper_data_size_limit: int = 2000
|
||||
|
||||
# Data processing LLM models (for API calls in data processing scripts)
|
||||
strong_models: list[str] = ["gpt-5", "gpt-5.1"]
|
||||
"""Strong models for complex tasks (CoT generation, reasoning) - supports list (env: FT_STRONG_MODELS)"""
|
||||
|
||||
weak_models: list[str] = ["gpt-4o-mini", "o4-mini", "gpt-5-mini"]
|
||||
"""Weak models for simple tasks (filtering, format conversion) - supports list (env: FT_WEAK_MODELS)"""
|
||||
|
||||
embedding_models: list[str] = ["text-embedding-3-small", "text-embedding-3-large"]
|
||||
|
||||
# Docker settings
|
||||
docker_enable_cache: bool = False
|
||||
"""Enable Docker cache for training (set via FT_DOCKER_ENABLE_CACHE)"""
|
||||
|
||||
# data sample count
|
||||
data_sample_count: int = 3
|
||||
|
||||
# API concurrency for data processing
|
||||
api_max_workers: int = 1000
|
||||
"""Max concurrent workers for LLM API calls in data processing scripts (env: FT_API_MAX_WORKERS)"""
|
||||
|
||||
# Coder settings
|
||||
coder_max_loop: int = 10
|
||||
|
||||
# CoT format settings
|
||||
force_think_token: bool = False
|
||||
"""Force <think> token wrapping for CoT training data (env: FT_FORCE_THINK_TOKEN).
|
||||
When True: Data must be wrapped in <think>...</think> format, benchmark uses extract-non-reasoning-content postprocessor.
|
||||
When False: CoT reasoning required but format is flexible, no postprocessor needed."""
|
||||
|
||||
|
||||
def update_settings(competition: str):
|
||||
"""
|
||||
Update the RD_AGENT_SETTINGS with the values from LLM_FINETUNE_SETTINGS.
|
||||
"""
|
||||
LLM_FINETUNE_SETTINGS = LLMFinetuneScen()
|
||||
RD_AGENT_SETTINGS.app_tpl = LLM_FINETUNE_SETTINGS.app_tpl
|
||||
os.environ["DS_CODER_COSTEER_EXTRA_EVALUATOR"] = '["rdagent.app.finetune.share.eval.PrevModelLoadEvaluator"]'
|
||||
for field_name, new_value in LLM_FINETUNE_SETTINGS.model_dump().items():
|
||||
if hasattr(DS_RD_SETTING, field_name):
|
||||
setattr(DS_RD_SETTING, field_name, new_value)
|
||||
DS_RD_SETTING.competition = competition
|
||||
# Global setting instance for LLM finetuning scenario
|
||||
FT_RD_SETTING = LLMFinetunePropSetting()
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# FT Job Runner
|
||||
|
||||
批量并行运行多个 LLM 微调任务的脚本。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
# 1. 准备环境配置
|
||||
cp .env.template .env
|
||||
# 编辑 .env，填入 API key 等配置
|
||||
|
||||
# 2. 准备任务配置
|
||||
cp tasks.json.example tasks.json
|
||||
# 编辑 tasks.json，定义要运行的任务
|
||||
|
||||
# 3. 运行
|
||||
./run_ft_job.sh
|
||||
```
|
||||
|
||||
## 用法
|
||||
|
||||
```bash
|
||||
./run_ft_job.sh [tasks.json]
|
||||
```
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `tasks.json` | 任务配置文件路径（可选，默认使用同目录下的 `tasks.json`） |
|
||||
| `-h, --help` | 显示帮助信息 |
|
||||
|
||||
### 示例
|
||||
|
||||
```bash
|
||||
# 使用默认配置
|
||||
./run_ft_job.sh
|
||||
|
||||
# 指定自定义配置文件
|
||||
./run_ft_job.sh /path/to/my_tasks.json
|
||||
```
|
||||
|
||||
## 配置文件
|
||||
|
||||
### tasks.json
|
||||
|
||||
定义要并行运行的任务列表：
|
||||
|
||||
```json
|
||||
{
|
||||
"tasks": [
|
||||
{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"benchmark": "aime25",
|
||||
"gpus": "0,1"
|
||||
},
|
||||
{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"benchmark": "gsm8k",
|
||||
"gpus": "2,3",
|
||||
"scenario": "自定义优化目标"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 必填 | 默认值 | 说明 |
|
||||
|------|:----:|--------|------|
|
||||
| `model` | ✅ | - | HuggingFace 模型路径 |
|
||||
| `benchmark` | ✅ | - | 评估基准（如 `aime25`, `gsm8k`） |
|
||||
| `gpus` | ❌ | `"0"` | 使用的 GPU 编号 |
|
||||
| `scenario` | ❌ | `"Improve model performance on {benchmark}"` | 优化目标描述 |
|
||||
|
||||
### .env
|
||||
|
||||
环境配置文件，包含 API 密钥、模型设置等。从 `.env.template` 复制并修改：
|
||||
|
||||
```bash
|
||||
cp .env.template .env
|
||||
```
|
||||
|
||||
主要配置项：
|
||||
|
||||
| 配置 | 说明 |
|
||||
|------|------|
|
||||
| `OPENAI_API_KEY` | OpenAI API 密钥 |
|
||||
| `OPENAI_API_BASE` | API 地址 |
|
||||
| `FT_Coder_CoSTEER_env_type` | 环境类型：`docker` 或 `conda` |
|
||||
| `HF_TOKEN` | HuggingFace Token |
|
||||
|
||||
## 输出
|
||||
|
||||
运行后会在 `log/` 目录下创建 job 文件夹：
|
||||
|
||||
```
|
||||
log/2025-12-23/
|
||||
├── aime25_Qwen3-8B.log # 任务日志
|
||||
├── gsm8k_Qwen3-8B.log
|
||||
└── aime25_Qwen3-8B/ # 任务 trace（Loop 数据）
|
||||
├── Loop_0/
|
||||
└── ...
|
||||
```
|
||||
|
||||
## 监控
|
||||
|
||||
### 命令行
|
||||
|
||||
```bash
|
||||
# 查看所有任务日志
|
||||
tail -f log/2025-12-23/*.log
|
||||
|
||||
# 查看特定任务
|
||||
tail -f log/2025-12-23/aime25_Qwen3-8B.log
|
||||
```
|
||||
|
||||
### Web UI
|
||||
|
||||
```bash
|
||||
streamlit run rdagent/app/finetune/llm/ui/app.py
|
||||
```
|
||||
|
||||
在 UI 中选择 Job Folder 为对应的日志目录即可查看运行状态。
|
||||
|
||||
## 依赖
|
||||
|
||||
- `jq`：JSON 解析工具
|
||||
- `conda` 环境：`rdagent`
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. 任务启动间隔默认为 60 秒（`STAGGER_DELAY`），避免同时启动造成资源竞争
|
||||
2. 确保指定的 GPU 编号不冲突
|
||||
3. 如果同一天多次运行，会自动创建 `log/2025-12-23_1/`、`log/2025-12-23_2/` 等目录
|
||||
Executable
+186
@@ -0,0 +1,186 @@
|
||||
#!/bin/bash
|
||||
# Run multiple FT tasks in parallel under a single job directory
|
||||
#
|
||||
# Usage: ./run_ft_job.sh [tasks.json]
|
||||
#
|
||||
# Config format (tasks.json):
|
||||
# {
|
||||
# "tasks": [
|
||||
# {"model": "Qwen/Qwen3-8B", "benchmark": "aime25", "gpus": "0,1"},
|
||||
# {"model": "Qwen/Qwen3-8B", "benchmark": "gsm8k", "gpus": "2,3"}
|
||||
# ]
|
||||
# }
|
||||
|
||||
set -e
|
||||
|
||||
# ========== CONFIG ==========
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
RDAGENT_DIR="$(cd "$SCRIPT_DIR/../../../../.." && pwd)"
|
||||
ENV_FILE="$SCRIPT_DIR/.env"
|
||||
SCENARIOS_FILE="$SCRIPT_DIR/scenarios.json"
|
||||
STAGGER_DELAY=60
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 [tasks.json]"
|
||||
echo "Run multiple FT tasks under a single job directory."
|
||||
echo "UI: streamlit run rdagent/app/finetune/llm/ui/app.py"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ========== PARSE ARGS ==========
|
||||
CONFIG_FILE=""
|
||||
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
-h|--help) usage ;;
|
||||
*) [[ -z "$CONFIG_FILE" ]] && CONFIG_FILE="$arg" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -z "$CONFIG_FILE" ]] && CONFIG_FILE="$SCRIPT_DIR/tasks.json"
|
||||
[[ ! -f "$CONFIG_FILE" ]] && echo "Error: Config not found: $CONFIG_FILE" && exit 1
|
||||
|
||||
# Check .env file
|
||||
if [[ ! -f "$ENV_FILE" ]]; then
|
||||
echo "Error: .env not found at $ENV_FILE"
|
||||
echo "Please create it from template: cp $SCRIPT_DIR/.env.template $ENV_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check jq
|
||||
command -v jq &>/dev/null || { echo "Error: jq required"; exit 1; }
|
||||
|
||||
# ========== SETUP ==========
|
||||
# Get log and workspace base paths from environment or use defaults
|
||||
# Default to project-relative paths; can be overridden by environment variables
|
||||
FT_LOG_BASE="${FT_LOG_BASE:-$RDAGENT_DIR/log}"
|
||||
FT_WORKSPACE_BASE="${FT_WORKSPACE_BASE:-$RDAGENT_DIR/git_ignore_folder/RD-Agent_workspace}"
|
||||
|
||||
JOB_ID=$(date +%Y-%m-%d_%H-%M)
|
||||
JOB_DIR="$FT_LOG_BASE/$JOB_ID"
|
||||
if [[ -d "$JOB_DIR" ]]; then
|
||||
i=1; while [[ -d "${JOB_DIR}_$i" ]]; do ((i++)); done
|
||||
JOB_ID="${JOB_ID}_$i"; JOB_DIR="${JOB_DIR}_$i"
|
||||
fi
|
||||
mkdir -p "$JOB_DIR"
|
||||
|
||||
cd "$RDAGENT_DIR"
|
||||
|
||||
NUM_TASKS=$(jq '.tasks | length' "$CONFIG_FILE")
|
||||
|
||||
echo "=============================================="
|
||||
echo "FT Job: $JOB_ID"
|
||||
echo "=============================================="
|
||||
echo "Config: $CONFIG_FILE"
|
||||
echo "Tasks: $NUM_TASKS"
|
||||
echo "Log: $JOB_DIR"
|
||||
echo "Workspace: $FT_WORKSPACE_BASE/$JOB_ID"
|
||||
echo ""
|
||||
|
||||
# Setup tmux session
|
||||
TMUX_SESSION="rdagent"
|
||||
tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true
|
||||
tmux new-session -d -s "$TMUX_SESSION" -n "main"
|
||||
echo "Tmux session created: $TMUX_SESSION"
|
||||
echo ""
|
||||
|
||||
for ((i=0; i<NUM_TASKS; i++)); do
|
||||
model=$(jq -r ".tasks[$i].model" "$CONFIG_FILE")
|
||||
benchmark=$(jq -r ".tasks[$i].benchmark" "$CONFIG_FILE")
|
||||
gpus=$(jq -r ".tasks[$i].gpus // \"0\"" "$CONFIG_FILE")
|
||||
port=$(jq -r ".tasks[$i].port // empty" "$CONFIG_FILE")
|
||||
task_timeout=$(jq -r ".tasks[$i].timeout // \"12h\"" "$CONFIG_FILE")
|
||||
|
||||
# Load benchmark_description: tasks.json -> scenarios.json
|
||||
benchmark_desc=$(jq -r ".tasks[$i].benchmark_description // empty" "$CONFIG_FILE")
|
||||
if [[ -z "$benchmark_desc" ]]; then
|
||||
benchmark_desc=$(jq -r ".[\"$benchmark\"].benchmark_description // empty" "$SCENARIOS_FILE")
|
||||
fi
|
||||
# Note: Special characters in benchmark_desc are handled by writing to env file
|
||||
model_name=$(basename "$model")
|
||||
task_name="${benchmark}_${model_name}"
|
||||
trace_path="$JOB_DIR/$task_name"
|
||||
|
||||
port_info=""
|
||||
[[ -n "$port" ]] && port_info=", port=$port"
|
||||
echo "Task $i: $task_name (model=$model, benchmark=$benchmark, gpus=$gpus$port_info)"
|
||||
|
||||
# Run task in tmux window with script -c for output capture
|
||||
task_workspace="$FT_WORKSPACE_BASE/$JOB_ID/$task_name"
|
||||
mkdir -p "$task_workspace"
|
||||
LOG_FILE="$JOB_DIR/${task_name}.log"
|
||||
|
||||
# Write task-specific env file (avoids command-line escaping issues with special chars)
|
||||
TASK_ENV_FILE="$task_workspace/.task_env"
|
||||
cat > "$TASK_ENV_FILE" << EOF
|
||||
CUDA_VISIBLE_DEVICES='$gpus'
|
||||
LOG_TRACE_PATH='$trace_path'
|
||||
WORKSPACE_PATH='$task_workspace'
|
||||
FT_TARGET_BENCHMARK='$benchmark'
|
||||
EOF
|
||||
# Escape shell special characters for double-quoted string: \ " ` $
|
||||
if [[ -n "$benchmark_desc" ]]; then
|
||||
escaped_desc="$benchmark_desc"
|
||||
escaped_desc="${escaped_desc//\\/\\\\}" # \ -> \\
|
||||
escaped_desc="${escaped_desc//\"/\\\"}" # " -> \"
|
||||
escaped_desc="${escaped_desc//\`/\\\`}" # ` -> \`
|
||||
escaped_desc="${escaped_desc//\$/\\\$}" # $ -> \$
|
||||
echo "FT_BENCHMARK_DESCRIPTION=\"$escaped_desc\"" >> "$TASK_ENV_FILE"
|
||||
fi
|
||||
[[ -n "$port" ]] && echo "OPENAI_API_BASE='http://localhost:$port'" >> "$TASK_ENV_FILE"
|
||||
|
||||
# Create tmux window for this task and get its full target (e.g., rdagent:1.0)
|
||||
# Use "session:" format to ensure window is created in the correct session
|
||||
WIN_TARGET=$(tmux new-window -t "$TMUX_SESSION:" -n "$benchmark" -P)
|
||||
|
||||
# Build the command with environment setup (env vars loaded from file)
|
||||
timeout_arg=""
|
||||
[[ -n "$task_timeout" ]] && timeout_arg="--timeout $task_timeout"
|
||||
|
||||
TASK_CMD="source ~/miniconda3/etc/profile.d/conda.sh && conda activate qz_rdagent"
|
||||
TASK_CMD="$TASK_CMD && set -a && source '$ENV_FILE' && source '$TASK_ENV_FILE' && set +a"
|
||||
TASK_CMD="$TASK_CMD && cd '$RDAGENT_DIR'"
|
||||
TASK_CMD="$TASK_CMD && python rdagent/app/finetune/llm/loop.py --base-model '$model' $timeout_arg"
|
||||
|
||||
# Run with script -c to capture terminal output (using full target for reliability)
|
||||
tmux send-keys -t "$WIN_TARGET" "script -q '$LOG_FILE' -c \"$TASK_CMD\"" Enter
|
||||
|
||||
echo " Window: $benchmark"
|
||||
echo ""
|
||||
|
||||
# Stagger starts
|
||||
if [[ $i -eq 0 ]]; then
|
||||
# First task: wait for initialization
|
||||
# Get FT_FILE_PATH from .env or use default
|
||||
FT_FILE_PATH=$(grep -E "^FT_FILE_PATH=" "$ENV_FILE" | cut -d= -f2 | tr -d '"' || echo "")
|
||||
[[ -z "$FT_FILE_PATH" ]] && FT_FILE_PATH="$RDAGENT_DIR/git_ignore_folder/finetune"
|
||||
DATASET_INFO="$FT_FILE_PATH/datasets/dataset_info.json"
|
||||
|
||||
echo " Waiting for scenario initialization (dataset_info.json)..."
|
||||
while [[ ! -f "$DATASET_INFO" ]]; do
|
||||
sleep 5
|
||||
done
|
||||
echo " Scenario initialized!"
|
||||
|
||||
echo " Waiting for llm_finetune conda env..."
|
||||
while ! conda run -n llm_finetune python -c "import requests" 2>/dev/null; do
|
||||
sleep 10
|
||||
done
|
||||
|
||||
echo " Waiting for opencompass conda env..."
|
||||
while ! conda run -n opencompass python -c "import opencompass" 2>/dev/null; do
|
||||
sleep 10
|
||||
done
|
||||
echo " Environment ready!"
|
||||
elif [[ $i -lt $((NUM_TASKS - 1)) ]]; then
|
||||
sleep $STAGGER_DELAY
|
||||
fi
|
||||
done
|
||||
|
||||
echo "=============================================="
|
||||
echo "All tasks started in tmux session: $TMUX_SESSION"
|
||||
echo " - Attach: tmux attach -t $TMUX_SESSION"
|
||||
echo " - List: tmux list-windows -t $TMUX_SESSION"
|
||||
echo " - Select: tmux select-window -t $TMUX_SESSION:{window_name}"
|
||||
echo "Monitor: tail -f $JOB_DIR/*.log"
|
||||
echo "UI: streamlit run rdagent/app/finetune/llm/ui/app.py (Job Folder: $JOB_DIR)"
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"tasks": [
|
||||
{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"benchmark": "aime25",
|
||||
"gpus": "0,1"
|
||||
},
|
||||
{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"benchmark": "gsm8k",
|
||||
"gpus": "2,3"
|
||||
},
|
||||
{
|
||||
"model": "meta-llama/Llama-3-8B",
|
||||
"benchmark": "aime25",
|
||||
"gpus": "4,5",
|
||||
"scenario": "Improve AIME 2025 math reasoning with custom approach"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,40 +1,101 @@
|
||||
"""
|
||||
LLM Fine-tuning Entry Point
|
||||
|
||||
Standard RDLoop entry point for LLM fine-tuning, consistent with data science implementation.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Optional, cast
|
||||
|
||||
import fire
|
||||
import typer
|
||||
from typing_extensions import Annotated
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.app.finetune.llm.conf import update_settings
|
||||
from rdagent.core.utils import import_class
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.data_science.loop import DataScienceRDLoop
|
||||
from rdagent.scenarios.finetune.loop import LLMFinetuneRDLoop
|
||||
|
||||
|
||||
def main(
|
||||
model: str | None = None,
|
||||
dataset: str | None = None,
|
||||
path: Optional[str] = None,
|
||||
checkout: Annotated[bool, typer.Option("--checkout/--no-checkout", "-c/-C")] = True,
|
||||
user_target_scenario: Optional[str] = None,
|
||||
benchmark: Optional[str] = None,
|
||||
benchmark_description: Optional[str] = None,
|
||||
dataset: Optional[str] = None,
|
||||
base_model: Optional[str] = None,
|
||||
upper_data_size_limit: Optional[int] = None,
|
||||
step_n: Optional[int] = None,
|
||||
loop_n: Optional[int] = None,
|
||||
timeout: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
LLM fine-tuning entry point
|
||||
|
||||
Parameters
|
||||
----------
|
||||
dataset :
|
||||
Dateset name, used for finetune.
|
||||
path :
|
||||
A path like `$LOG_PATH/__session__/1/0_propose`. This indicates that we restore the state after finishing step 0 in loop 1.
|
||||
checkout :
|
||||
Used to control the log session path. Boolean type, default is True.
|
||||
- If True, the new loop will use the existing folder and clear logs for sessions after the one corresponding to the given path.
|
||||
- If False, the new loop will use the existing folder but keep the logs for sessions after the one corresponding to the given path.
|
||||
dataset : str
|
||||
Dataset name for fine-tuning (e.g., 'shibing624/alpaca-zh')
|
||||
base_model : str, optional
|
||||
Model name for fine-tuning (e.g., 'Qwen/Qwen2.5-1.5B-Instruct').
|
||||
If not provided, auto-selects optimal model based on hardware and dataset.
|
||||
step_n : int, optional
|
||||
Number of steps to run; if None, runs indefinitely until completion or error
|
||||
loop_n : int, optional
|
||||
Number of loops to run; if None, runs indefinitely until completion or error
|
||||
timeout : str, optional
|
||||
Maximum duration for the entire process
|
||||
|
||||
Auto R&D Evolving loop for models finetune.
|
||||
You can continue running a session by using the command:
|
||||
Examples:
|
||||
.. code-block:: bash
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh --base-model Qwen/Qwen2.5-1.5B-Instruct
|
||||
dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh # TODO: not enabled yet
|
||||
"""
|
||||
if not dataset:
|
||||
raise Exception("Please specify dataset name.")
|
||||
|
||||
model_folder = Path(DS_RD_SETTING.local_data_path) / dataset / "prev_model"
|
||||
if not model_folder.exists():
|
||||
raise Exception(f"Please put the model path to {model_folder}.")
|
||||
update_settings(dataset)
|
||||
rd_loop: DataScienceRDLoop = DataScienceRDLoop(DS_RD_SETTING)
|
||||
asyncio.run(rd_loop.run())
|
||||
if user_target_scenario:
|
||||
FT_RD_SETTING.user_target_scenario = user_target_scenario
|
||||
assert (
|
||||
FT_RD_SETTING.user_target_scenario is None
|
||||
), "user_target_scenario is not yet supported, please specify via benchmark and benchmark_description"
|
||||
if upper_data_size_limit:
|
||||
FT_RD_SETTING.upper_data_size_limit = upper_data_size_limit
|
||||
logger.info(f"Set upper_data_size_limit to {FT_RD_SETTING.upper_data_size_limit}")
|
||||
if benchmark and benchmark_description:
|
||||
FT_RD_SETTING.target_benchmark = benchmark
|
||||
FT_RD_SETTING.benchmark_description = benchmark_description
|
||||
assert FT_RD_SETTING.user_target_scenario or (
|
||||
FT_RD_SETTING.target_benchmark and FT_RD_SETTING.benchmark_description
|
||||
), "Either user_target_scenario or target_benchmark must be specified for LLM fine-tuning."
|
||||
|
||||
# Update configuration with provided parameters
|
||||
if dataset:
|
||||
FT_RD_SETTING.dataset = dataset
|
||||
if base_model:
|
||||
FT_RD_SETTING.base_model = base_model
|
||||
|
||||
# Create and run LLM fine-tuning loop
|
||||
data_set_target = FT_RD_SETTING.dataset if FT_RD_SETTING.dataset else "auto generated dataset"
|
||||
model_target = FT_RD_SETTING.base_model if FT_RD_SETTING.base_model else "auto selected model"
|
||||
|
||||
# Temporary assertion until auto-selection is implemented
|
||||
assert (
|
||||
FT_RD_SETTING.base_model is not None
|
||||
), "Base model auto selection not yet supported, please specify via --base-model"
|
||||
|
||||
logger.info(f"Starting LLM fine-tuning on dataset='{data_set_target}' with model='{model_target}'")
|
||||
|
||||
if path is None:
|
||||
loop = LLMFinetuneRDLoop(FT_RD_SETTING)
|
||||
else:
|
||||
loop = cast(LLMFinetuneRDLoop, LLMFinetuneRDLoop.load(str(path), checkout=checkout))
|
||||
|
||||
asyncio.run(loop.run(step_n=step_n, loop_n=loop_n, all_duration=timeout))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
fire.Fire(main)
|
||||
typer.run(main)
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
scenario_description: |-
|
||||
------Background of the scenario------
|
||||
You are a world-class machine learning engineer. Your task is to finetune a model on the given dataset using QLoRA method.
|
||||
------Dataset Description------
|
||||
{{ raw_description }}
|
||||
|
||||
competition_background: |-
|
||||
## QLoRA Fine-Tuning
|
||||
You are a world-class machine learning engineer and prompt engineer specializing in parameter-efficient fine-tuning of large language models using **QLoRA**. Your expertise includes 4-bit quantization, low-rank adaptation, and maximizing performance on GPU clusters. You are committed to building accurate, resource-efficient, and robust LLMs.
|
||||
|
||||
- **Fine-Tuning Method**: QLoRA (4-bit quantized LoRA)
|
||||
- **Training Dataset**:
|
||||
> {{ raw_description }}
|
||||
@@ -1,46 +0,0 @@
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.core.proposal import ExpGen
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.proposal.exp_gen.base import DSHypothesis, DSTrace
|
||||
from rdagent.scenarios.data_science.proposal.exp_gen.proposal import DSProposalV2ExpGen
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class FinetuneExpGen(DSProposalV2ExpGen):
|
||||
def gen(
|
||||
self,
|
||||
trace: DSTrace,
|
||||
) -> DSExperiment:
|
||||
component_desc = T("scenarios.data_science.share:component_description_in_pipeline").r()
|
||||
|
||||
if (sota_exp_fb := trace.sota_experiment_fb()) is None:
|
||||
sota_exp, fb_to_sota_exp = None, None
|
||||
else:
|
||||
sota_exp, fb_to_sota_exp = sota_exp_fb
|
||||
|
||||
if not isinstance(sota_exp, DSExperiment):
|
||||
eda_output = None
|
||||
else:
|
||||
eda_output = sota_exp.experiment_workspace.file_dict.get("EDA.md", None)
|
||||
scenario_desc = self.scen.get_scenario_all_desc(eda_output=eda_output)
|
||||
|
||||
# TODO: this is a over simplified version. More features will be added after more survey
|
||||
sota_exp_desc = "No previous SOTA experiments available."
|
||||
failed_exp_feedback_list_desc = "No previous experiments available."
|
||||
|
||||
return self.task_gen(
|
||||
component_desc=component_desc,
|
||||
scenario_desc=scenario_desc,
|
||||
sota_exp_desc=sota_exp_desc,
|
||||
sota_exp=sota_exp,
|
||||
hypotheses=[
|
||||
DSHypothesis(
|
||||
component="Model",
|
||||
)
|
||||
],
|
||||
pipeline=True,
|
||||
failed_exp_feedback_list_desc=failed_exp_feedback_list_desc,
|
||||
fb_to_sota_exp=fb_to_sota_exp,
|
||||
)
|
||||
@@ -1,87 +0,0 @@
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.data_science.scen import DataScienceScen
|
||||
from rdagent.scenarios.data_science.scen.utils import describe_data_folder_v2
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class LLMFinetuneScen(DataScienceScen):
|
||||
"""LLMFinetuneScen Scenario"""
|
||||
|
||||
def __init__(self, competition: str) -> None:
|
||||
self._download_data(competition=competition)
|
||||
super().__init__(competition)
|
||||
self._analysis_competition_description()
|
||||
|
||||
def _get_data_folder_description(self) -> str:
|
||||
folder_desc = describe_data_folder_v2(
|
||||
Path(DS_RD_SETTING.local_data_path) / self.competition, show_nan_columns=DS_RD_SETTING.show_nan_columns
|
||||
)
|
||||
return folder_desc
|
||||
|
||||
def _download_data(self, competition: str):
|
||||
"""
|
||||
Download dateset from Hugging Face Hub
|
||||
|
||||
Parameters
|
||||
----------
|
||||
- competition (str): Dateset ID, like "shibing624/alpaca-zh".
|
||||
"""
|
||||
save_path = f"{DS_RD_SETTING.local_data_path}/{competition}"
|
||||
if Path(save_path).exists():
|
||||
logger.info(f"{save_path} already exists.")
|
||||
else:
|
||||
logger.info(f"Downloading {competition} to {save_path}")
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
snapshot_download(
|
||||
repo_id=competition,
|
||||
repo_type="dataset",
|
||||
local_dir=save_path,
|
||||
local_dir_use_symlinks=False,
|
||||
)
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"Please install huggingface_hub first. "
|
||||
'You can install it with `pip install -U "huggingface_hub[cli]"`.'
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error when downloading {competition}: {e}")
|
||||
raise e
|
||||
|
||||
def _get_description(self):
|
||||
if (fp := Path(f"{DS_RD_SETTING.local_data_path}/{self.competition}/README.md")).exists():
|
||||
logger.info(f"{self.competition}/Found README.md, loading from local file.")
|
||||
return fp.read_text()
|
||||
|
||||
def _get_direction(self):
|
||||
return True
|
||||
|
||||
@property
|
||||
def rich_style_description(self) -> str:
|
||||
raise NotImplementedError
|
||||
|
||||
@property
|
||||
def background(self) -> str:
|
||||
background_template = T(".prompts:competition_background")
|
||||
background_prompt = background_template.r(
|
||||
raw_description=self.raw_description,
|
||||
)
|
||||
return background_prompt
|
||||
|
||||
def get_competition_full_desc(self) -> str:
|
||||
return T(".prompts:scenario_description").r(
|
||||
raw_description=self.raw_description,
|
||||
)
|
||||
|
||||
def get_scenario_all_desc(self, eda_output=None) -> str:
|
||||
"""
|
||||
eda_output depends on dynamic .md files from current workspace, not fixed.
|
||||
"""
|
||||
return T(".prompts:scenario_description").r(
|
||||
raw_description=self.raw_description,
|
||||
)
|
||||
@@ -1,71 +0,0 @@
|
||||
pipeline_coder:
|
||||
system: |-
|
||||
You are a world-class ML engineer specializing in parameter-efficient LLM fine-tuning with QLoRA.
|
||||
Design a single-file `main.py` that:
|
||||
• Loads a pretrained model from `./workspace_input/prev_model`.
|
||||
• Attaches 4-bit LoRA adapters, runs fine-tuning, evaluates on the validation set.
|
||||
• Uses `print()` for progress and debug output (no `logging` or progress bars).
|
||||
• Wraps file reads in `try/except` only to catch missing files—do not suppress other errors.
|
||||
• Hardcodes all paths and hyperparameters—no CLI parsing.
|
||||
• Is directly executable via `python main.py`.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## The runtime environment your code will running on
|
||||
{{ runtime_environment }}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. Ensure that the dataset is loaded strictly from `{% include "scenarios.data_science.share:scen.input_path" %}`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).
|
||||
2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
3. You should be very careful about the try catch block in your code. You may use it to handle missing files in data reading, but you should not use it to handle the errors in your code. Especially use it to bypass the errors in your code. Directly solve the errors in your code instead of using try catch block to bypass them.
|
||||
4. Initialize random seeds and specify device (`cpu`/`cuda`) for reproducibility.
|
||||
5. Ensure `main.py` runs end-to-end: training → validation → save `./scores.csv`.
|
||||
6. Save finetuned adapter to `./models/` directory.
|
||||
7. When run the code again, the code will skip finetune process and directly load the finetuned adapter from `./models/` directory.
|
||||
|
||||
{% if enable_debug_mode %}
|
||||
Your code will be executed in a debug mode with following command:
|
||||
```bash
|
||||
python main.py --debug
|
||||
```
|
||||
In debug mode, you should only sample smallest possible subset from the training data and run the minimum epochs to quickly test the correctness of the code.
|
||||
In debug mode, you should implement a timer to measure the time taken for your debug configuration and estimate the time required for the full run.
|
||||
For example, you can sample smallest possible subset from the training data and run for one epoch, then the full run with ten epochs will take one hundred times the time taken for the debug run. The scale is calculated by yourself depending on the data sampling and epoch number you choose. If your full run enables early stopping, the scale should be smaller considering the early stopping will stop the training earlier than the full epochs.
|
||||
You should sample the data after train valid split. When you split the data after sampling, you might get a class with only one sample which might cause the split strategy to fail.
|
||||
Your debug code should run exactly the same as the full run, except for the data sampling and epoch number, to ensure the correctness of the code.
|
||||
You should print total time and estimated time in standard output using print function in the following schema:
|
||||
=== Start of Debug Information ===
|
||||
debug_time: time_taken_for_debug_run_in_seconds (e.g., 'debug_time: 10.0')
|
||||
estimated_time: estimated_time_for_full_run_in_seconds (e.g., 'estimated_time: 100.0')
|
||||
=== End of Debug Information ===
|
||||
User will use the following code to match: re.search(r"(.*?)=== Start of Debug Information ===(.*)=== End of Debug Information ===", stdout, re.DOTALL).groups()[1]
|
||||
Notice, data sampling should only be applied in debug mode. Always use the full data in the full run!
|
||||
Example code:
|
||||
```python
|
||||
if args.debug:
|
||||
sample_size = int(0.01 * len(train_dataset)) # 1% for debug
|
||||
else:
|
||||
sample_size = len(train_dataset)
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the full runable code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
@@ -1,10 +0,0 @@
|
||||
system: |-
|
||||
You are a world-class ML engineer specializing in parameter-efficient LLM fine-tuning with QLoRA.
|
||||
Design a single-file `main.py` that:
|
||||
• Loads a pretrained model from `./workspace_input/prev_model`.
|
||||
• Attaches 4-bit LoRA adapters, runs fine-tuning, evaluates on the validation set.
|
||||
• Uses `print()` for progress and debug output (no `logging` or progress bars).
|
||||
• Wraps file reads in `try/except` only to catch missing files—do not suppress other errors.
|
||||
• Hardcodes all paths and hyperparameters—no CLI parsing.
|
||||
• Is directly executable via `python main.py`.
|
||||
|
||||
@@ -1,82 +0,0 @@
|
||||
scenario_problem:
|
||||
system: |-
|
||||
You are a world-class machine learning and prompt engineer specializing in parameter-efficient fine-tuning of large language models using QLoRA (4-bit quantized LoRA adapters).
|
||||
Each iteration (trace) represents one training run or adapter update. If an iteration’s validation metric exceeds the current best, it becomes the new SOTA adapter; otherwise it is a failed experiment.
|
||||
Your task is to analyze the scenario (and SOTA, if given) and identify a concise list of **2–3 Key Challenges** that most critically limit fine-tuning performance.
|
||||
|
||||
### Core Analysis Dimensions
|
||||
1. **Adapter-Model Alignment**
|
||||
Compare current LoRA adapter configuration against model capacity and task complexity.
|
||||
2. **Optimization Dynamics**
|
||||
Identify where training diverges or plateaus (e.g. LR too high/low, quantization noise).
|
||||
3. **Data-Model Coherence**
|
||||
Spot mismatches between the dataset’s characteristics and model input preprocessing or sequence length.
|
||||
|
||||
## Key Challenges / Core Problems
|
||||
Categorize each challenge as one of:
|
||||
- **Data-Driven Challenge**
|
||||
Issues in dataset size, domain mismatch, label noise, sequence length distribution, etc.
|
||||
- **Model-Optimization Challenge**
|
||||
LoRA rank selection, quantization artifacts, learning rate schedule, gradient accumulation, etc.
|
||||
|
||||
### For Each Challenge
|
||||
1. Be **specific** and **actionable**.
|
||||
2. Focus on **methodological** aspects, not trivial bugs.
|
||||
3. Directly tie to improving the **target metric**.
|
||||
4. If no SOTA exists, include at least one challenge that guides building a minimal baseline adapter.
|
||||
|
||||
{% if task_output_format is not none %}
|
||||
{% endif %}
|
||||
|
||||
task_gen:
|
||||
system: |-
|
||||
You are an expert in LLM fine-tuning with QLoRA. Each iteration applies a specific hypothesis to improve the current adapter (SOTA) or establish an initial adapter.
|
||||
|
||||
**Inputs**:
|
||||
- Scenario: base model, task, data path, evaluation metric
|
||||
- Current SOTA adapter & feedback (if any)
|
||||
- Proposed Hypothesis
|
||||
- Failed runs feedback (if any)
|
||||
|
||||
**Your task**: Outline a conceptual plan for `main.py` that implements the Proposed Hypothesis.
|
||||
|
||||
**Standards**:
|
||||
- Run via `python main.py` with no CLI args; configs are hard-coded.
|
||||
- No code or pseudo-code—describe each step in plain language.
|
||||
- Do **not** use progress bars.
|
||||
- Do **not** infer test indices from sample files.
|
||||
|
||||
**Sketch**:
|
||||
1. **Load Data**
|
||||
- Read train/validation files from the given data path.
|
||||
- Tokenize or preprocess inputs for the model.
|
||||
|
||||
2. **Initialize Model & Adapter**
|
||||
- Load the base LLM.
|
||||
- Attach a QLoRA adapter.
|
||||
|
||||
3. **Train with Hypothesis**
|
||||
- Apply the hypothesis change (e.g., modify learning schedule, adapter config).
|
||||
- Train and validate iteratively.
|
||||
|
||||
4. **Validate & Record**
|
||||
- Compute the metric on validation set.
|
||||
- Save results to `scores.csv` (with adapter name and “ensemble”).
|
||||
|
||||
5. **Generate Submission**
|
||||
- Write `submission.jsonl` or `.csv` matching the competition format exactly.
|
||||
|
||||
**Key Reminders for Developer**:
|
||||
- Hard-code all paths; do not rely on sample files for indices.
|
||||
- Ensure tokenizer and model names match.
|
||||
- Validate output formats for `scores.csv` and `submission`.
|
||||
- Handle file I/O robustly (e.g., zipped data).
|
||||
|
||||
{% if task_output_format is not none %}
|
||||
## [Partial Response Format 1] Task Output Format:
|
||||
{{ task_output_format }}
|
||||
Your final output should strictly adhere to the following JSON format.
|
||||
{
|
||||
"task_design": ---The dict corresponding to task output format---,
|
||||
}
|
||||
{% endif %}
|
||||
@@ -1,18 +0,0 @@
|
||||
competition_description_template:
|
||||
system: |-
|
||||
You are a data science assistant that extracts structured information from unstructured text.
|
||||
The user will provide you a description of an LLM fine-tuning project, and you need to extract specific details from it.
|
||||
For the dataset, the user has already reviewed and provided any additional context—include that information in your response.
|
||||
Please answer in JSON format with the following schema:
|
||||
{
|
||||
"Task Type": "The type of fine-tuning task, e.g., 'Question Answering', 'Text Classification', 'Summarization', 'Translation', 'Code Generation'",
|
||||
"Data Type": "The type of data used for fine-tuning, e.g., 'Text (Natural Language)', 'Code', 'Multimodal', 'Dialogue'",
|
||||
"Brief Description": "A concise summary of the fine-tuning project and its objectives",
|
||||
"Dataset Description": "A description of the dataset as organized in the Processed Data folder: list files, formats, sizes, and any pre-processing steps applied, reconciled with contextual details from the project description",
|
||||
"Training Specifications": "Details of the fine-tuning setup, including base model name, number of epochs, batch size, learning rate, optimizer, and any scheduler or early-stopping rules",
|
||||
"Output Format": "The expected model output format per sample (e.g., single label, probability distribution over N classes, generated text sequence)",
|
||||
"Channels per Sample": "An integer indicating output dimensionality per example (e.g., 1 for single regression value, N for N-class probabilities, variable for generated text)",
|
||||
"Evaluation Metric Description": "A precise explanation of how model performance is measured, including the formula or procedure used",
|
||||
"Metric Name": "The name of the evaluation metric (e.g., 'Accuracy', 'ROUGE-L', 'BLEU', 'F1'), please only choose one metric name",
|
||||
"Metric Direction": true or false // true if higher is better, false if lower is better
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
# FT (Fine-tune) scenario UI
|
||||
@@ -0,0 +1,207 @@
|
||||
"""
|
||||
FT (Fine-tune) Timeline Viewer
|
||||
Hierarchical view: Session > Loop > Stage > EvoLoop > Events
|
||||
|
||||
Run:
|
||||
streamlit run rdagent/app/finetune/llm/ui/app.py
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import streamlit as st
|
||||
from streamlit import session_state as state
|
||||
|
||||
from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
|
||||
from rdagent.app.finetune.llm.ui.components import render_session, render_summary
|
||||
from rdagent.app.finetune.llm.ui.config import ALWAYS_VISIBLE_TYPES, OPTIONAL_TYPES
|
||||
from rdagent.app.finetune.llm.ui.data_loader import (
|
||||
get_summary,
|
||||
get_valid_sessions,
|
||||
load_ft_session,
|
||||
)
|
||||
from rdagent.app.finetune.llm.ui.ft_summary import render_job_summary
|
||||
|
||||
DEFAULT_LOG_BASE = "log/"
|
||||
|
||||
|
||||
def get_job_options(base_path: Path) -> list[str]:
|
||||
"""
|
||||
Scan directory and return job options list.
|
||||
- "." means standalone tasks in root directory
|
||||
- Others are job directory names
|
||||
"""
|
||||
options = []
|
||||
has_root_tasks = False
|
||||
job_dirs = []
|
||||
|
||||
if not base_path.exists():
|
||||
return options
|
||||
|
||||
for d in base_path.iterdir():
|
||||
if not d.is_dir():
|
||||
continue
|
||||
# Check if standalone task (has __session__ directly)
|
||||
if (d / "__session__").exists():
|
||||
has_root_tasks = True
|
||||
# Check if job directory (subdirs have __session__)
|
||||
else:
|
||||
try:
|
||||
if any((sub / "__session__").exists() for sub in d.iterdir() if sub.is_dir()):
|
||||
job_dirs.append(d.name)
|
||||
except PermissionError:
|
||||
pass
|
||||
|
||||
# Sort job dirs by name descending (newest first, since names are date-based)
|
||||
job_dirs.sort(reverse=True)
|
||||
|
||||
# Add job dirs first, then root tasks at the end
|
||||
options.extend(job_dirs)
|
||||
if has_root_tasks:
|
||||
options.append(". (Current)")
|
||||
|
||||
return options
|
||||
|
||||
|
||||
def main():
|
||||
st.set_page_config(layout="wide", page_title="FT Timeline", page_icon="🔬")
|
||||
|
||||
# ========== Sidebar ==========
|
||||
with st.sidebar:
|
||||
# View mode selection
|
||||
view_mode = st.radio("View Mode", ["Job Summary", "Single Task"], horizontal=True)
|
||||
|
||||
st.divider()
|
||||
|
||||
default_log = os.environ.get("FT_LOG_PATH", DEFAULT_LOG_BASE)
|
||||
job_folder = default_log # Initialize for both modes
|
||||
selected_types = ALWAYS_VISIBLE_TYPES.copy() # Initialize for both modes
|
||||
is_root_job = False # Track if viewing root tasks
|
||||
|
||||
if view_mode == "Job Summary":
|
||||
# Job Summary mode
|
||||
st.header("Job")
|
||||
base_folder = st.text_input("Base Folder", value=default_log, key="base_folder_input")
|
||||
base_path = Path(base_folder)
|
||||
|
||||
job_options = get_job_options(base_path)
|
||||
if job_options:
|
||||
selected_job = st.selectbox("Select Job", job_options, key="job_select")
|
||||
if selected_job.startswith("."):
|
||||
job_folder = base_folder
|
||||
is_root_job = True
|
||||
else:
|
||||
job_folder = str(base_path / selected_job)
|
||||
# Save to session_state for Single Task mode
|
||||
state.selected_job_folder = job_folder
|
||||
else:
|
||||
st.warning("No jobs found in this directory")
|
||||
job_folder = base_folder
|
||||
|
||||
if st.button("Refresh", type="primary", key="refresh_job"):
|
||||
st.rerun()
|
||||
else:
|
||||
# Single Task mode
|
||||
st.header("Session")
|
||||
# Use job_folder from Job Summary mode if available
|
||||
default_path = getattr(state, "selected_job_folder", default_log)
|
||||
log_folder = st.text_input("Log Folder", value=default_path)
|
||||
log_path = Path(log_folder)
|
||||
|
||||
sessions = get_valid_sessions(log_path)
|
||||
if not sessions:
|
||||
st.warning("No valid sessions found")
|
||||
return
|
||||
|
||||
selected_session = st.selectbox("Session", sessions)
|
||||
|
||||
if st.button("Load", type="primary") or "session" not in state:
|
||||
with st.spinner("Loading..."):
|
||||
state.session = load_ft_session(log_path / selected_session)
|
||||
state.session_name = selected_session
|
||||
|
||||
st.divider()
|
||||
|
||||
# Optional type toggles
|
||||
st.subheader("Show More")
|
||||
selected_types = ALWAYS_VISIBLE_TYPES.copy()
|
||||
for event_type, (label, default) in OPTIONAL_TYPES.items():
|
||||
if st.toggle(label, value=default, key=f"toggle_{event_type}"):
|
||||
selected_types.append(event_type)
|
||||
|
||||
st.divider()
|
||||
|
||||
# Display options
|
||||
st.subheader("Display Options")
|
||||
state.render_markdown = st.toggle("Render Prompts", value=False, key="render_markdown_toggle")
|
||||
|
||||
st.divider()
|
||||
|
||||
# Summary in sidebar
|
||||
if "session" in state:
|
||||
summary = get_summary(state.session)
|
||||
st.subheader("Summary")
|
||||
st.metric("Loops", summary.get("loop_count", 0))
|
||||
st.metric("LLM Calls", summary.get("llm_call_count", 0))
|
||||
success = summary.get("docker_success", 0)
|
||||
fail = summary.get("docker_fail", 0)
|
||||
st.metric("Docker", f"{success}✓ / {fail}✗")
|
||||
|
||||
# ========== Main Content ==========
|
||||
if view_mode == "Job Summary":
|
||||
st.title("📊 FT Job Summary")
|
||||
job_path = Path(job_folder)
|
||||
if job_path.exists():
|
||||
render_job_summary(job_path, is_root=is_root_job)
|
||||
else:
|
||||
st.warning(f"Job folder not found: {job_folder}")
|
||||
return
|
||||
|
||||
# Single Task mode
|
||||
st.title("🔬 FT Timeline Viewer")
|
||||
|
||||
if "session" not in state:
|
||||
st.info("Select a session and click **Load** to view")
|
||||
return
|
||||
|
||||
session = state.session
|
||||
summary = get_summary(session)
|
||||
|
||||
# Global info header (Base Model, Datasets, Benchmark) - compact style
|
||||
scenario_event = next((e for e in session.init_events if e.type == "scenario"), None)
|
||||
dataset_event = next((e for e in session.init_events if e.type == "dataset_selection"), None)
|
||||
|
||||
if scenario_event or dataset_event:
|
||||
if scenario_event and hasattr(scenario_event.content, "base_model"):
|
||||
st.markdown(f"🧠 **Model:** `{scenario_event.content.base_model}`")
|
||||
if dataset_event:
|
||||
selected = (
|
||||
dataset_event.content.get("selected_datasets", []) if isinstance(dataset_event.content, dict) else []
|
||||
)
|
||||
if selected:
|
||||
st.markdown(f"📂 **Datasets:** `{', '.join(selected)}`")
|
||||
if scenario_event and hasattr(scenario_event.content, "target_benchmark"):
|
||||
st.markdown(f"🎯 **Benchmark:** `{scenario_event.content.target_benchmark}`")
|
||||
# Display baseline benchmark score
|
||||
if scenario_event and hasattr(scenario_event.content, "baseline_benchmark_score"):
|
||||
baseline = scenario_event.content.baseline_benchmark_score
|
||||
if baseline and isinstance(baseline, dict):
|
||||
benchmark_name = getattr(scenario_event.content, "target_benchmark", "")
|
||||
accuracy_summary = baseline.get("accuracy_summary", {})
|
||||
if accuracy_summary:
|
||||
result = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if result:
|
||||
metric_name, score, _ = result
|
||||
st.markdown(f"📊 **Baseline:** `{metric_name} = {score:.1f}`")
|
||||
|
||||
# Summary bar
|
||||
render_summary(summary)
|
||||
|
||||
st.divider()
|
||||
|
||||
# Hierarchical view
|
||||
render_session(session, selected_types)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Benchmark processors for core metric extraction.
|
||||
|
||||
Each benchmark has its own processor that knows how to extract
|
||||
the core metric name and value from accuracy_summary data.
|
||||
"""
|
||||
|
||||
from .bioprobench import BioProBenchProcessor
|
||||
from .chemcotbench import ChemCotBenchProcessor
|
||||
from .financeiq import FinanceIQProcessor
|
||||
from .panorama import PanoramaProcessor
|
||||
from .tablebench import TableBenchProcessor
|
||||
|
||||
PROCESSORS = [
|
||||
FinanceIQProcessor,
|
||||
PanoramaProcessor,
|
||||
ChemCotBenchProcessor,
|
||||
TableBenchProcessor,
|
||||
BioProBenchProcessor,
|
||||
]
|
||||
|
||||
|
||||
def get_core_metric_score(benchmark_name: str, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
"""Get core metric name, score, and direction for a benchmark.
|
||||
|
||||
Args:
|
||||
benchmark_name: The benchmark name (e.g., "FinanceIQ", "panorama_par4pc")
|
||||
accuracy_summary: {dataset_name: {metric: value, ...}, ...}
|
||||
|
||||
Returns:
|
||||
(metric_name, value, higher_is_better) or None
|
||||
- metric_name: includes "(average)" suffix if multiple datasets are averaged
|
||||
- value: the score
|
||||
- higher_is_better: True if higher values are better (use ↑), False otherwise (use ↓)
|
||||
"""
|
||||
for processor in PROCESSORS:
|
||||
if processor.match(benchmark_name):
|
||||
return processor.get_core_metric(accuracy_summary)
|
||||
|
||||
# Default fallback: use first numeric value with "accuracy" label
|
||||
scores = []
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
if "accuracy" in metrics:
|
||||
scores.append(float(metrics["accuracy"]))
|
||||
else:
|
||||
for v in metrics.values():
|
||||
if isinstance(v, (int, float)):
|
||||
scores.append(float(v))
|
||||
break
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
if len(scores) == 1:
|
||||
return ("accuracy", avg, True) # higher is better
|
||||
else:
|
||||
return ("accuracy (average)", avg, True) # higher is better
|
||||
|
||||
|
||||
__all__ = [
|
||||
"get_core_metric_score",
|
||||
"PROCESSORS",
|
||||
"FinanceIQProcessor",
|
||||
"PanoramaProcessor",
|
||||
"ChemCotBenchProcessor",
|
||||
"TableBenchProcessor",
|
||||
"BioProBenchProcessor",
|
||||
]
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Base class for benchmark core metric extraction."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class BenchmarkProcessor(ABC):
|
||||
"""Base class for benchmark core metric extraction."""
|
||||
|
||||
# Metrics where higher values are better (default assumption)
|
||||
# Override in subclass if needed
|
||||
HIGHER_IS_BETTER: set[str] = {
|
||||
"accuracy",
|
||||
"exact_match",
|
||||
"f1",
|
||||
"f1_score",
|
||||
"macro_f1",
|
||||
"correct_rate",
|
||||
"success_rate",
|
||||
"gold_hit_rate",
|
||||
"score",
|
||||
"scaffold_hard",
|
||||
"kendall_tau",
|
||||
"ROUGE-L",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
"""Check if this processor handles the given benchmark."""
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
"""Extract core metric name, value, and direction from accuracy_summary.
|
||||
|
||||
Args:
|
||||
accuracy_summary: {dataset_name: {metric: value, ...}, ...}
|
||||
|
||||
Returns:
|
||||
(metric_name, value, higher_is_better) or None
|
||||
- metric_name: includes "(average)" suffix if multiple datasets
|
||||
- value: the score
|
||||
- higher_is_better: True if higher values are better, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
def is_higher_better(cls, metric_name: str) -> bool:
|
||||
"""Check if higher values are better for this metric."""
|
||||
# Remove (average) suffix for checking
|
||||
base_metric = metric_name.replace(" (average)", "").strip()
|
||||
return base_metric.lower() in {m.lower() for m in cls.HIGHER_IS_BETTER}
|
||||
@@ -0,0 +1,60 @@
|
||||
"""BioProBench benchmark processor."""
|
||||
|
||||
from .base import BenchmarkProcessor
|
||||
|
||||
|
||||
class BioProBenchProcessor(BenchmarkProcessor):
|
||||
"""BioProBench: Biology protocol benchmark with different task types."""
|
||||
|
||||
CORE_METRICS = {
|
||||
"pqa": "accuracy",
|
||||
"ord": "kendall_tau",
|
||||
"err": "f1",
|
||||
"gen": "ROUGE-L",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
return "bioprobench" in benchmark_name.lower()
|
||||
|
||||
@classmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
scores = []
|
||||
metrics_used = []
|
||||
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
ds_lower = ds.lower()
|
||||
# Find matching core metric
|
||||
core_metric = "accuracy" # fallback
|
||||
for pattern, metric in cls.CORE_METRICS.items():
|
||||
if pattern in ds_lower:
|
||||
core_metric = metric
|
||||
break
|
||||
|
||||
if core_metric in metrics:
|
||||
scores.append(float(metrics[core_metric]))
|
||||
metrics_used.append(core_metric)
|
||||
elif core_metric.lower() in [k.lower() for k in metrics.keys()]:
|
||||
# Case-insensitive fallback for metrics like "ROUGE-L"
|
||||
for k, v in metrics.items():
|
||||
if k.lower() == core_metric.lower():
|
||||
scores.append(float(v))
|
||||
metrics_used.append(core_metric)
|
||||
break
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
unique = list(set(metrics_used))
|
||||
|
||||
if len(scores) == 1:
|
||||
metric_name = unique[0]
|
||||
elif len(unique) == 1:
|
||||
metric_name = f"{unique[0]} (average)"
|
||||
else:
|
||||
metric_name = "mixed (average)"
|
||||
|
||||
return (metric_name, avg, cls.is_higher_better(metric_name))
|
||||
@@ -0,0 +1,105 @@
|
||||
"""ChemCotBench benchmark processor."""
|
||||
|
||||
from .base import BenchmarkProcessor
|
||||
|
||||
|
||||
class ChemCotBenchProcessor(BenchmarkProcessor):
|
||||
"""ChemCotBench: Chemistry reasoning with various subtasks.
|
||||
|
||||
All metrics are 0-100 percentages, enabling unified averaging within each subset.
|
||||
"""
|
||||
|
||||
# Define core metric field names for each task
|
||||
CORE_METRICS = {
|
||||
# Molecular understanding
|
||||
"mol_und_fg_count": "accuracy",
|
||||
"mol_und_ring_count": "accuracy",
|
||||
"mol_und_murcko_scaffold": "scaffold_hard", # Exact match rate (0-100%)
|
||||
"mol_und_ring_system_scaffold": "score", # "Yes" ratio (0-100%)
|
||||
"mol_und_equivalence": "accuracy",
|
||||
# Molecular editing
|
||||
"mol_edit_add": "correct_rate",
|
||||
"mol_edit_delete": "correct_rate",
|
||||
"mol_edit_sub": "correct_rate",
|
||||
# Molecular optimization (prefix match)
|
||||
"mol_opt_": "success_rate",
|
||||
# Reaction tasks - unified to exact_match
|
||||
"reaction_fs": "exact_match",
|
||||
"reaction_retro": "exact_match",
|
||||
"reaction_nepp": "exact_match",
|
||||
"reaction_rcr": "exact_match",
|
||||
"reaction_mechsel": "exact_match", # Will fallback to accuracy if exact_match not found
|
||||
}
|
||||
|
||||
# Metric groups: unified display names for each subset
|
||||
METRIC_GROUPS = {
|
||||
"mol_und": "accuracy", # mol_und subset displays as accuracy
|
||||
"mol_edit": "correct_rate",
|
||||
"mol_opt": "success_rate",
|
||||
"reaction": "exact_match", # reaction subset displays as exact_match
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
return "chemcot" in benchmark_name.lower()
|
||||
|
||||
@classmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
scores = []
|
||||
group_detected = None
|
||||
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
ds_lower = ds.lower()
|
||||
|
||||
# Detect subset type
|
||||
for group in cls.METRIC_GROUPS:
|
||||
if group in ds_lower:
|
||||
group_detected = group
|
||||
break
|
||||
|
||||
# Find matching core metric
|
||||
core_metric = "accuracy" # fallback
|
||||
for pattern, metric in cls.CORE_METRICS.items():
|
||||
# Prefix match for patterns ending with _
|
||||
if pattern.endswith("_"):
|
||||
if pattern in ds_lower:
|
||||
core_metric = metric
|
||||
break
|
||||
else:
|
||||
if pattern in ds_lower:
|
||||
core_metric = metric
|
||||
break
|
||||
|
||||
# Try to get metric value with fallback support
|
||||
value = None
|
||||
if core_metric in metrics:
|
||||
value = float(metrics[core_metric])
|
||||
elif core_metric == "exact_match" and "accuracy" in metrics:
|
||||
# reaction_mechsel fallback: exact_match -> accuracy
|
||||
value = float(metrics["accuracy"])
|
||||
|
||||
if value is not None:
|
||||
scores.append(value)
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
|
||||
# Use unified metric name for the detected subset
|
||||
if group_detected and group_detected in cls.METRIC_GROUPS:
|
||||
unified_name = cls.METRIC_GROUPS[group_detected]
|
||||
if len(scores) == 1:
|
||||
metric_name = unified_name
|
||||
else:
|
||||
metric_name = f"{unified_name} (average)"
|
||||
else:
|
||||
# Fallback for unknown subsets
|
||||
if len(scores) == 1:
|
||||
metric_name = "accuracy"
|
||||
else:
|
||||
metric_name = "accuracy (average)"
|
||||
|
||||
return (metric_name, avg, cls.is_higher_better(metric_name))
|
||||
@@ -0,0 +1,29 @@
|
||||
"""FinanceIQ benchmark processor."""
|
||||
|
||||
from .base import BenchmarkProcessor
|
||||
|
||||
|
||||
class FinanceIQProcessor(BenchmarkProcessor):
|
||||
"""FinanceIQ: 10 exam subjects, all use accuracy."""
|
||||
|
||||
@classmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
return "financeiq" in benchmark_name.lower()
|
||||
|
||||
@classmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
scores = []
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
if "accuracy" in metrics:
|
||||
scores.append(float(metrics["accuracy"]))
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
if len(scores) == 1:
|
||||
return ("accuracy", avg, True) # higher is better
|
||||
else:
|
||||
return ("accuracy (average)", avg, True) # higher is better
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Panorama benchmark processor."""
|
||||
|
||||
from .base import BenchmarkProcessor
|
||||
|
||||
|
||||
class PanoramaProcessor(BenchmarkProcessor):
|
||||
"""Panorama: Different sub-datasets use different metrics."""
|
||||
|
||||
CORE_METRICS = {
|
||||
"par4pc": "macro_f1",
|
||||
"pi4pc": "gold_hit_rate",
|
||||
"noc4pc": "macro_f1",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
return "panorama" in benchmark_name.lower()
|
||||
|
||||
@classmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
scores = []
|
||||
metrics_used = []
|
||||
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
ds_lower = ds.lower()
|
||||
# Find matching core metric
|
||||
core_metric = "accuracy" # fallback
|
||||
for pattern, metric in cls.CORE_METRICS.items():
|
||||
if pattern in ds_lower:
|
||||
core_metric = metric
|
||||
break
|
||||
|
||||
if core_metric in metrics:
|
||||
scores.append(float(metrics[core_metric]))
|
||||
metrics_used.append(core_metric)
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
unique = list(set(metrics_used))
|
||||
|
||||
if len(scores) == 1:
|
||||
metric_name = unique[0]
|
||||
elif len(unique) == 1:
|
||||
metric_name = f"{unique[0]} (average)"
|
||||
else:
|
||||
metric_name = "mixed (average)"
|
||||
|
||||
return (metric_name, avg, cls.is_higher_better(metric_name))
|
||||
@@ -0,0 +1,60 @@
|
||||
"""TableBench benchmark processor."""
|
||||
|
||||
from .base import BenchmarkProcessor
|
||||
|
||||
|
||||
class TableBenchProcessor(BenchmarkProcessor):
|
||||
"""TableBench: Table QA with different subtasks."""
|
||||
|
||||
CORE_METRICS = {
|
||||
"fact": "accuracy",
|
||||
"numerical": "accuracy",
|
||||
"analysis": "accuracy",
|
||||
"visualization": "Pass@1", # TableBench visualization uses Pass@1 as core metric
|
||||
}
|
||||
|
||||
# TableBench-specific metrics where higher is better
|
||||
HIGHER_IS_BETTER = BenchmarkProcessor.HIGHER_IS_BETTER | {
|
||||
"Pass@1",
|
||||
"ECR@1",
|
||||
"Parse@1",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def match(cls, benchmark_name: str) -> bool:
|
||||
return "tablebench" in benchmark_name.lower()
|
||||
|
||||
@classmethod
|
||||
def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
|
||||
scores = []
|
||||
metrics_used = []
|
||||
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
if not isinstance(metrics, dict):
|
||||
continue
|
||||
ds_lower = ds.lower()
|
||||
# Find matching core metric
|
||||
core_metric = "accuracy" # fallback
|
||||
for pattern, metric in cls.CORE_METRICS.items():
|
||||
if pattern in ds_lower:
|
||||
core_metric = metric
|
||||
break
|
||||
|
||||
if core_metric in metrics:
|
||||
scores.append(float(metrics[core_metric]))
|
||||
metrics_used.append(core_metric)
|
||||
|
||||
if not scores:
|
||||
return None
|
||||
|
||||
avg = sum(scores) / len(scores)
|
||||
unique = list(set(metrics_used))
|
||||
|
||||
if len(scores) == 1:
|
||||
metric_name = unique[0]
|
||||
elif len(unique) == 1:
|
||||
metric_name = f"{unique[0]} (average)"
|
||||
else:
|
||||
metric_name = "mixed (average)"
|
||||
|
||||
return (metric_name, avg, cls.is_higher_better(metric_name))
|
||||
@@ -0,0 +1,721 @@
|
||||
"""
|
||||
FT UI Components - Hierarchical Event Renderers
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import plotly.graph_objects as go
|
||||
import streamlit as st
|
||||
|
||||
from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
|
||||
from rdagent.app.finetune.llm.ui.config import ICONS
|
||||
from rdagent.app.finetune.llm.ui.data_loader import Event, EvoLoop, Loop, Session
|
||||
|
||||
|
||||
def convert_latex_for_streamlit(text: str) -> str:
|
||||
"""Convert LaTeX syntax to Streamlit-compatible format.
|
||||
|
||||
Streamlit uses $...$ and $$...$$ for LaTeX rendering.
|
||||
This converts \(...\) and \[...\] to the Streamlit format.
|
||||
"""
|
||||
if not text:
|
||||
return text
|
||||
# Convert \(...\) to $...$
|
||||
text = text.replace(r"\(", "$").replace(r"\)", "$")
|
||||
# Convert \[...\] to $$...$$
|
||||
text = text.replace(r"\[", "$$").replace(r"\]", "$$")
|
||||
return text
|
||||
|
||||
|
||||
def format_duration(seconds: float | None) -> str:
|
||||
if seconds is None:
|
||||
return ""
|
||||
if seconds < 60:
|
||||
return f"{seconds:.1f}s"
|
||||
minutes = int(seconds // 60)
|
||||
secs = seconds % 60
|
||||
return f"{minutes}m {secs:.0f}s"
|
||||
|
||||
|
||||
def render_session(session: Session, show_types: list[str]) -> None:
|
||||
"""Render full session with hierarchy"""
|
||||
# Init events (before any loop)
|
||||
if session.init_events:
|
||||
filtered = [e for e in session.init_events if e.type in show_types]
|
||||
if filtered:
|
||||
with st.expander("🚀 **Initialization**", expanded=False):
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Loops
|
||||
for loop_id in sorted(session.loops.keys()):
|
||||
loop = session.loops[loop_id]
|
||||
render_loop(loop, show_types)
|
||||
|
||||
|
||||
def render_loop(loop: Loop, show_types: list[str]) -> None:
|
||||
"""Render a single loop with lazy loading"""
|
||||
# 1. Coding stage results
|
||||
evo_results = []
|
||||
for evo in loop.coding.values():
|
||||
if evo.success is True:
|
||||
evo_results.append("✓")
|
||||
elif evo.success is False:
|
||||
evo_results.append("✗")
|
||||
coding_str = f"💻{''.join(evo_results)}" if evo_results else ""
|
||||
|
||||
# 2. Running stage results
|
||||
runner_success = None
|
||||
benchmark_score = None
|
||||
for event in loop.runner:
|
||||
# Docker (Full Train) result - check exit_code, not LLM evaluation
|
||||
if event.type == "docker_exec" and "Full Train" in event.title and event.success is not None:
|
||||
runner_success = event.success
|
||||
# Benchmark score - use core metric from processor
|
||||
if event.type == "feedback" and "Benchmark Result" in event.title:
|
||||
content = event.content
|
||||
if isinstance(content, dict):
|
||||
benchmark_name = content.get("benchmark_name", "")
|
||||
accuracy_summary = content.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
result = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if result is not None:
|
||||
_, benchmark_score, _ = result
|
||||
|
||||
# 3. Get feedback decision for benchmark score coloring
|
||||
feedback_decision = None
|
||||
for event in loop.feedback:
|
||||
if event.type == "feedback" and "Feedback:" in event.title:
|
||||
feedback_decision = event.success
|
||||
break
|
||||
|
||||
# 4. Build title string (only show existing stages)
|
||||
parts = []
|
||||
if coding_str:
|
||||
parts.append(coding_str)
|
||||
if runner_success is not None:
|
||||
runner_str = "🏃✓" if runner_success else "🏃✗"
|
||||
parts.append(runner_str)
|
||||
# Show benchmark score with emoji based on feedback decision
|
||||
if benchmark_score is not None:
|
||||
if feedback_decision is True:
|
||||
parts.append(f"✅📊{benchmark_score:.2f}")
|
||||
elif feedback_decision is False:
|
||||
parts.append(f"❌📊{benchmark_score:.2f}")
|
||||
else:
|
||||
parts.append(f"📊{benchmark_score:.2f}")
|
||||
|
||||
result_str = " ".join(parts) if parts else ""
|
||||
|
||||
loop_key = f"loop_{loop.loop_id}_loaded"
|
||||
with st.expander(f"🔄 **Loop {loop.loop_id}** {result_str}", expanded=False):
|
||||
if not st.session_state.get(loop_key, False):
|
||||
# Lazy load: show button first
|
||||
if st.button("📥 Load Content", key=f"load_{loop.loop_id}"):
|
||||
st.session_state[loop_key] = True
|
||||
st.rerun()
|
||||
else:
|
||||
# Render actual content
|
||||
_render_loop_content(loop, show_types)
|
||||
|
||||
|
||||
def _render_loop_content(loop: Loop, show_types: list[str]) -> None:
|
||||
"""Render loop content (called after lazy load)"""
|
||||
# Exp Gen
|
||||
if loop.exp_gen:
|
||||
filtered = [e for e in loop.exp_gen if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 🧪 Experiment Generation")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Coding (Evo Loops)
|
||||
if loop.coding:
|
||||
st.markdown("#### 💻 Coding")
|
||||
for evo_id in sorted(loop.coding.keys()):
|
||||
evo = loop.coding[evo_id]
|
||||
render_evo_loop(evo, show_types)
|
||||
|
||||
# Runner
|
||||
if loop.runner:
|
||||
filtered = [e for e in loop.runner if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 🏃 Running(Full Train)")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Feedback
|
||||
if loop.feedback:
|
||||
filtered = [e for e in loop.feedback if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 📊 Feedback")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
|
||||
def render_evo_loop(evo: EvoLoop, show_types: list[str]) -> None:
|
||||
"""Render evolution loop"""
|
||||
filtered = [e for e in evo.events if e.type in show_types]
|
||||
if not filtered:
|
||||
return
|
||||
|
||||
status = "🟢" if evo.success else "🔴" if evo.success is False else "⚪"
|
||||
with st.expander(f"{status} Evo {evo.evo_id}", expanded=False):
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
|
||||
def render_event(event: Event) -> None:
|
||||
"""Render a single event"""
|
||||
icon = ICONS.get(event.type, "📌")
|
||||
duration_str = f" ({format_duration(event.duration)})" if event.duration else ""
|
||||
|
||||
status = ""
|
||||
if event.success is True:
|
||||
status = "🟢 "
|
||||
elif event.success is False:
|
||||
status = "🔴 "
|
||||
|
||||
title = f"{event.time_str} {icon} {status}{event.title}{duration_str}"
|
||||
|
||||
renderers = {
|
||||
"scenario": render_scenario,
|
||||
"llm_call": render_llm_call,
|
||||
"template": render_template,
|
||||
"experiment": render_experiment,
|
||||
"code": render_code,
|
||||
"docker_exec": render_docker_exec,
|
||||
"evaluator": render_docker_exec, # Reuse docker_exec renderer for evaluator feedback
|
||||
"feedback": render_feedback,
|
||||
"token": render_token,
|
||||
"time": render_time_info,
|
||||
"settings": render_settings,
|
||||
"hypothesis": render_hypothesis,
|
||||
"dataset_selection": render_dataset_selection,
|
||||
}
|
||||
|
||||
renderer = renderers.get(event.type, render_generic)
|
||||
with st.expander(title, expanded=False):
|
||||
# Pass event.title to docker_exec/evaluator renderers for context-aware labels
|
||||
if event.type in ("docker_exec", "evaluator"):
|
||||
renderer(event.content, event.title)
|
||||
else:
|
||||
renderer(event.content)
|
||||
|
||||
|
||||
def render_scenario(content: Any) -> None:
|
||||
"""Render scenario details (main info shown in page header, this shows extras)."""
|
||||
import json
|
||||
|
||||
# 1. User target scenario
|
||||
if hasattr(content, "user_target_scenario") and content.user_target_scenario:
|
||||
st.markdown(f"**Target Scenario:** {content.user_target_scenario}")
|
||||
|
||||
# 2. Benchmark description
|
||||
if hasattr(content, "benchmark_description") and content.benchmark_description:
|
||||
st.markdown(f"**Benchmark Description:** {content.benchmark_description}")
|
||||
|
||||
# 3. Full timeout
|
||||
if hasattr(content, "real_full_timeout"):
|
||||
try:
|
||||
timeout_hours = content.real_full_timeout() / 60 / 60
|
||||
st.markdown(f"**Full Train Timeout:** {timeout_hours:.2f} hours")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 4. Device info - formatted nicely
|
||||
if hasattr(content, "device_info") and content.device_info:
|
||||
device = content.device_info
|
||||
# Parse string to dict if needed
|
||||
if isinstance(device, str):
|
||||
try:
|
||||
device = json.loads(device)
|
||||
except json.JSONDecodeError:
|
||||
st.markdown(f"**Device:** `{device}`")
|
||||
device = None
|
||||
if isinstance(device, dict):
|
||||
parts = []
|
||||
# Runtime info
|
||||
runtime = device.get("runtime", {})
|
||||
if runtime.get("python_version"):
|
||||
parts.append(f"🐍 Python `{runtime['python_version'].split()[0]}`")
|
||||
if runtime.get("os"):
|
||||
parts.append(f"💻 {runtime['os']}")
|
||||
# GPU info
|
||||
gpu_info = device.get("gpu", {})
|
||||
gpus = gpu_info.get("gpus", [])
|
||||
if gpus:
|
||||
gpu_name = gpus[0].get("name", "Unknown")
|
||||
gpu_mem_gb = gpus[0].get("memory_total_gb", 0)
|
||||
if len(gpus) > 1:
|
||||
parts.append(f"🎮 {len(gpus)}x {gpu_name} ({gpu_mem_gb}GB)")
|
||||
else:
|
||||
parts.append(f"🎮 {gpu_name} ({gpu_mem_gb}GB)")
|
||||
if parts:
|
||||
st.markdown(" · ".join(parts))
|
||||
|
||||
# 5. Model info (detailed specs)
|
||||
if hasattr(content, "model_info") and content.model_info:
|
||||
model_info = content.model_info
|
||||
if isinstance(model_info, dict) and model_info:
|
||||
with st.expander("Model Info", expanded=False):
|
||||
# Show key specs in a readable format
|
||||
if "specs" in model_info and model_info["specs"]:
|
||||
st.markdown("**Specs:**")
|
||||
st.code(model_info["specs"], language="text", wrap_lines=True)
|
||||
# Show other fields
|
||||
other_info = {k: v for k, v in model_info.items() if k != "specs" and v}
|
||||
if other_info:
|
||||
st.json(other_info)
|
||||
|
||||
# 6. Memory report (estimation based on hardware and model)
|
||||
if hasattr(content, "memory_report") and content.memory_report:
|
||||
with st.expander("Memory Estimation", expanded=False):
|
||||
st.code(content.memory_report, language="text", wrap_lines=True)
|
||||
|
||||
|
||||
def render_dataset_selection(content: Any) -> None:
|
||||
if not isinstance(content, dict):
|
||||
st.json(content) if content else st.info("No content")
|
||||
return
|
||||
|
||||
selected = content.get("selected_datasets", [])
|
||||
total = content.get("total_datasets", 0)
|
||||
reasoning = content.get("reasoning", "")
|
||||
|
||||
if selected:
|
||||
st.markdown(f"**Selected ({len(selected)}/{total}):** " + ", ".join(f"`{ds}`" for ds in selected))
|
||||
|
||||
if reasoning:
|
||||
with st.expander("Selection Reasoning", expanded=True):
|
||||
st.markdown(reasoning)
|
||||
|
||||
|
||||
def render_hypothesis(content: Any) -> None:
|
||||
"""Render hypothesis content (Base Model shown in page header, not here)."""
|
||||
if hasattr(content, "hypothesis") and content.hypothesis:
|
||||
st.markdown("**Hypothesis:**")
|
||||
st.markdown(content.hypothesis)
|
||||
if hasattr(content, "reason") and content.reason:
|
||||
with st.expander("Reason", expanded=False):
|
||||
st.markdown(content.reason)
|
||||
|
||||
|
||||
def render_settings(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
st.json(content)
|
||||
else:
|
||||
st.code(str(content), wrap_lines=True)
|
||||
|
||||
|
||||
def render_llm_call(content: Any) -> None:
|
||||
if not isinstance(content, dict):
|
||||
st.json(content) if content else st.info("No content")
|
||||
return
|
||||
|
||||
if content.get("start") and content.get("end"):
|
||||
duration = (content["end"] - content["start"]).total_seconds()
|
||||
st.caption(f"Duration: {format_duration(duration)}")
|
||||
|
||||
# Check if markdown rendering is enabled
|
||||
render_md = st.session_state.get("render_markdown_toggle", False)
|
||||
|
||||
system = content.get("system", "")
|
||||
if system:
|
||||
with st.expander("System Prompt", expanded=False):
|
||||
if render_md:
|
||||
st.markdown(system)
|
||||
else:
|
||||
st.code(system, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
user = content.get("user", "")
|
||||
if user:
|
||||
with st.expander("User Prompt", expanded=False):
|
||||
if render_md:
|
||||
st.markdown(user)
|
||||
else:
|
||||
st.code(user, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
resp = content.get("resp", "")
|
||||
if resp:
|
||||
st.markdown("**Response:**")
|
||||
if render_md:
|
||||
st.markdown(resp)
|
||||
elif resp.strip().startswith("{") or resp.strip().startswith("["):
|
||||
st.code(resp, language="json", line_numbers=True, wrap_lines=True)
|
||||
elif resp.strip().startswith("```"):
|
||||
st.markdown(resp)
|
||||
else:
|
||||
st.code(resp, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def render_template(content: Any) -> None:
|
||||
if not isinstance(content, dict):
|
||||
st.json(content) if content else st.info("No content")
|
||||
return
|
||||
|
||||
uri = content.get("uri", "")
|
||||
st.caption(f"URI: `{uri}`")
|
||||
|
||||
context = content.get("context", {})
|
||||
if context:
|
||||
with st.expander("Context Variables", expanded=False):
|
||||
st.json(context)
|
||||
|
||||
template = content.get("template", "")
|
||||
if template:
|
||||
with st.expander("Template", expanded=False):
|
||||
st.code(template, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
rendered = content.get("rendered", "")
|
||||
if rendered:
|
||||
with st.expander("Rendered", expanded=True):
|
||||
st.code(rendered, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def render_experiment(content: Any) -> None:
|
||||
"""Render experiment tasks (Base Model and Datasets shown in page header, not here)."""
|
||||
if isinstance(content, list):
|
||||
for i, task in enumerate(content):
|
||||
if len(content) > 1:
|
||||
st.markdown(f"**Task {i}**")
|
||||
|
||||
if hasattr(task, "description") and task.description:
|
||||
st.markdown("**Description:**")
|
||||
st.markdown(task.description)
|
||||
else:
|
||||
st.json(content) if content else st.info("No content")
|
||||
|
||||
|
||||
def render_code(content: Any) -> None:
|
||||
if not isinstance(content, list):
|
||||
st.info("No code available")
|
||||
return
|
||||
|
||||
for i, ws in enumerate(content):
|
||||
if not hasattr(ws, "file_dict") or not ws.file_dict:
|
||||
continue
|
||||
|
||||
if len(content) > 1:
|
||||
st.markdown(f"**Workspace {i}**")
|
||||
|
||||
for filename, code in ws.file_dict.items():
|
||||
lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
|
||||
with st.expander(filename, expanded=False):
|
||||
st.code(code, language=lang, line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def _extract_evaluator_name(title: str) -> str:
|
||||
"""Extract evaluator name from event title like 'Eval (Data Processing) ✓'."""
|
||||
match = re.search(r"\(([^)]+)\)", title)
|
||||
return match.group(1) if match else ""
|
||||
|
||||
|
||||
def _render_single_feedback(fb: Any, evaluator_name: str = "") -> None:
|
||||
"""Render a single CoSTEERSingleFeedback object.
|
||||
|
||||
Structure:
|
||||
- execution: LLM-generated execution summary (what happened, success/failure reason)
|
||||
- raw_execution: Raw script stdout/stderr output
|
||||
- return_checking: LLM-generated data quality assessment
|
||||
- code: LLM-generated code improvement suggestions
|
||||
"""
|
||||
decision = getattr(fb, "final_decision", None)
|
||||
if decision is True:
|
||||
st.success("Execution: PASS")
|
||||
elif decision is False:
|
||||
st.error("Execution: FAIL")
|
||||
|
||||
# 1. Execution Summary (LLM-generated)
|
||||
execution = getattr(fb, "execution", "")
|
||||
if execution:
|
||||
label = f"{evaluator_name} Summary" if evaluator_name else "Execution Summary"
|
||||
with st.expander(label, expanded=True):
|
||||
st.code(execution, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
# 2. Raw Execution Log (script stdout)
|
||||
raw_execution = getattr(fb, "raw_execution", "")
|
||||
if raw_execution:
|
||||
with st.expander("Raw Output (stdout)", expanded=False):
|
||||
st.code(raw_execution, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
# 3. Data Quality Check (LLM-generated)
|
||||
return_checking = getattr(fb, "return_checking", "")
|
||||
if return_checking:
|
||||
with st.expander("Data Quality Check", expanded=False):
|
||||
st.code(return_checking, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
# 4. Code Improvement Suggestions (LLM-generated, often very long)
|
||||
code_fb = getattr(fb, "code", "")
|
||||
if code_fb:
|
||||
with st.expander("Code Improvement Suggestions", expanded=False):
|
||||
# Use markdown rendering if content contains markdown formatting
|
||||
if "**" in code_fb or "```" in code_fb or "- " in code_fb:
|
||||
st.markdown(code_fb)
|
||||
else:
|
||||
st.code(code_fb, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def render_docker_exec(content: Any, event_title: str = "") -> None:
|
||||
# Extract evaluator name from event title for context-aware labels
|
||||
evaluator_name = _extract_evaluator_name(event_title)
|
||||
|
||||
# Docker run raw output (dict with exit_code/stdout)
|
||||
if isinstance(content, dict) and ("exit_code" in content or "stdout" in content or "success" in content):
|
||||
# Show workspace ID if available (only the UUID part)
|
||||
workspace_path = content.get("workspace_path")
|
||||
if workspace_path:
|
||||
workspace_id = Path(workspace_path).name
|
||||
st.caption(f"📁 `{workspace_id}`")
|
||||
|
||||
exit_code = content.get("exit_code")
|
||||
success = content.get("success")
|
||||
if exit_code is not None:
|
||||
if exit_code == 0:
|
||||
st.success(f"Exit code: {exit_code}")
|
||||
else:
|
||||
st.error(f"Exit code: {exit_code}")
|
||||
elif success is not None:
|
||||
if success:
|
||||
st.success("Execution: PASS")
|
||||
else:
|
||||
st.error("Execution: FAIL")
|
||||
|
||||
stdout = content.get("stdout", "")
|
||||
if stdout:
|
||||
label = f"{evaluator_name} Output" if evaluator_name else "Execution Output"
|
||||
with st.expander(label, expanded=True):
|
||||
st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
|
||||
return
|
||||
|
||||
# CoSTEERMultiFeedback (has feedback_list)
|
||||
if hasattr(content, "feedback_list"):
|
||||
for i, fb in enumerate(content.feedback_list):
|
||||
if len(content.feedback_list) > 1:
|
||||
st.markdown(f"**Feedback {i}**")
|
||||
_render_single_feedback(fb, evaluator_name)
|
||||
return
|
||||
|
||||
# Single CoSTEERSingleFeedback (has final_decision)
|
||||
if hasattr(content, "final_decision"):
|
||||
_render_single_feedback(content, evaluator_name)
|
||||
return
|
||||
|
||||
# FTExperiment (runner result)
|
||||
if hasattr(content, "sub_workspace_list"):
|
||||
for ws in content.sub_workspace_list:
|
||||
if not hasattr(ws, "running_info") or ws.running_info is None:
|
||||
continue
|
||||
|
||||
info = ws.running_info
|
||||
running_time = getattr(info, "running_time", None)
|
||||
if running_time:
|
||||
st.metric("Running Time", f"{running_time:.1f}s")
|
||||
|
||||
stdout = getattr(info, "stdout", "")
|
||||
if stdout:
|
||||
with st.expander("Full Train Log", expanded=True):
|
||||
st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
result = getattr(info, "result", {})
|
||||
if result:
|
||||
render_training_result(result)
|
||||
return
|
||||
|
||||
st.json(content) if content else st.info("No content")
|
||||
|
||||
|
||||
def render_feedback(content: Any) -> None:
|
||||
# Handle benchmark result (dict with accuracy_summary)
|
||||
if isinstance(content, dict) and "accuracy_summary" in content:
|
||||
render_benchmark_result(content)
|
||||
return
|
||||
|
||||
col1, col2, col3 = st.columns(3)
|
||||
with col1:
|
||||
decision = getattr(content, "decision", None)
|
||||
if decision is not None:
|
||||
st.metric("Decision", "Accept" if decision else "Reject")
|
||||
with col2:
|
||||
acceptable = getattr(content, "acceptable", None)
|
||||
if acceptable is not None:
|
||||
st.metric("Acceptable", "Yes" if acceptable else "No")
|
||||
with col3:
|
||||
error_type = getattr(content, "observations", None)
|
||||
if error_type:
|
||||
st.metric("Error Type", error_type)
|
||||
|
||||
# FT scenario only uses code_change_summary (observations, hypothesis_evaluation,
|
||||
# new_hypothesis, eda_improvement are DS scenario specific)
|
||||
fields = [
|
||||
("code_change_summary", "Code Change Summary"),
|
||||
]
|
||||
|
||||
for attr, label in fields:
|
||||
value = getattr(content, attr, None)
|
||||
if value:
|
||||
with st.expander(label, expanded=False):
|
||||
st.markdown(value)
|
||||
|
||||
reason = getattr(content, "reason", None)
|
||||
if reason:
|
||||
with st.expander("Reason (Full Details)", expanded=True):
|
||||
st.code(reason, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
exception = getattr(content, "exception", None)
|
||||
if exception:
|
||||
st.error(f"Exception: {exception}")
|
||||
|
||||
|
||||
def render_token(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
col1, col2, col3 = st.columns(3)
|
||||
with col1:
|
||||
st.metric("Prompt", content.get("prompt_tokens", 0))
|
||||
with col2:
|
||||
st.metric("Completion", content.get("completion_tokens", 0))
|
||||
with col3:
|
||||
st.metric("Total", content.get("total_tokens", 0))
|
||||
else:
|
||||
st.json(content) if content else st.info("No content")
|
||||
|
||||
|
||||
def render_time_info(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
for k, v in content.items():
|
||||
st.metric(k, f"{v:.1f}s" if isinstance(v, (int, float)) else str(v))
|
||||
else:
|
||||
st.json(content) if content else st.info("No content")
|
||||
|
||||
|
||||
def render_generic(content: Any) -> None:
|
||||
if hasattr(content, "__dict__"):
|
||||
st.json(vars(content))
|
||||
elif content:
|
||||
st.json(content)
|
||||
else:
|
||||
st.info("No content")
|
||||
|
||||
|
||||
def render_training_result(result: dict) -> None:
|
||||
training_metrics = result.get("training_metrics", {})
|
||||
loss_history = training_metrics.get("loss_history", {})
|
||||
|
||||
# loss_history is Dict[str, List[Dict]] with "train" and "eval" keys
|
||||
train_history = loss_history.get("train", []) if isinstance(loss_history, dict) else []
|
||||
if train_history:
|
||||
fig = go.Figure()
|
||||
steps = [entry.get("step", i) for i, entry in enumerate(train_history)]
|
||||
losses = [entry.get("loss", 0) for entry in train_history]
|
||||
fig.add_trace(go.Scatter(x=steps, y=losses, mode="lines+markers", name="Loss"))
|
||||
fig.update_layout(title="Training Loss", xaxis_title="Step", yaxis_title="Loss", height=300)
|
||||
st.plotly_chart(fig, use_container_width=True)
|
||||
|
||||
col1, col2 = st.columns(2)
|
||||
initial_loss = training_metrics.get("initial_loss")
|
||||
final_loss = training_metrics.get("final_loss")
|
||||
if initial_loss:
|
||||
col1.metric("Initial Loss", f"{initial_loss:.4f}")
|
||||
if final_loss:
|
||||
col2.metric("Final Loss", f"{final_loss:.4f}")
|
||||
|
||||
# Validation benchmark ([:100]) - used for SOTA judgment
|
||||
benchmark = result.get("benchmark", {})
|
||||
if benchmark:
|
||||
st.markdown("**Validation Benchmark**")
|
||||
# Detect format: old format has "accuracy_summary" at top level,
|
||||
# new format has benchmark names as keys with nested accuracy_summary
|
||||
if "accuracy_summary" in benchmark:
|
||||
# Old format: {accuracy_summary: {...}, error_samples: [...]}
|
||||
accuracy_summary = benchmark.get("accuracy_summary", {})
|
||||
if accuracy_summary:
|
||||
rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
|
||||
st.dataframe(rows)
|
||||
else:
|
||||
# New format: {bm_name: {accuracy_summary: {...}}, ...}
|
||||
for bm_name, bm_result in benchmark.items():
|
||||
if isinstance(bm_result, dict) and "accuracy_summary" in bm_result:
|
||||
st.markdown(f"*{bm_name}:*")
|
||||
accuracy_summary = bm_result.get("accuracy_summary", {})
|
||||
if accuracy_summary:
|
||||
rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
|
||||
st.dataframe(rows)
|
||||
|
||||
# Test benchmark ([100:200]) - frontend display only, not visible to agent
|
||||
benchmark_test = result.get("benchmark_test", {})
|
||||
if benchmark_test and benchmark_test != benchmark: # Avoid duplicate display for small datasets
|
||||
st.markdown("**Test Benchmark**")
|
||||
if "accuracy_summary" in benchmark_test:
|
||||
accuracy_summary = benchmark_test.get("accuracy_summary", {})
|
||||
if accuracy_summary:
|
||||
rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
|
||||
st.dataframe(rows)
|
||||
else:
|
||||
for bm_name, bm_result in benchmark_test.items():
|
||||
if isinstance(bm_result, dict) and "accuracy_summary" in bm_result:
|
||||
st.markdown(f"*{bm_name}:*")
|
||||
accuracy_summary = bm_result.get("accuracy_summary", {})
|
||||
if accuracy_summary:
|
||||
rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
|
||||
st.dataframe(rows)
|
||||
|
||||
|
||||
def render_benchmark_result(content: dict) -> None:
|
||||
"""Render benchmark evaluation result"""
|
||||
import pandas as pd
|
||||
|
||||
benchmark_name = content.get("benchmark_name", "Unknown")
|
||||
st.markdown(f"**Benchmark: {benchmark_name}**")
|
||||
|
||||
# Accuracy summary table
|
||||
# accuracy_summary is a dict: {dataset_name: {metric: value, ...}, ...}
|
||||
accuracy_summary = content.get("accuracy_summary", {})
|
||||
if accuracy_summary and isinstance(accuracy_summary, dict):
|
||||
st.markdown("**Accuracy Summary:**")
|
||||
# Convert dict {dataset: {metric: value}} to list of dicts for dataframe
|
||||
rows = []
|
||||
for ds, metrics in accuracy_summary.items():
|
||||
row = {"dataset": ds, **metrics}
|
||||
rows.append(row)
|
||||
|
||||
# Create DataFrame and reorder columns
|
||||
df = pd.DataFrame(rows)
|
||||
cols = ["dataset"] + [c for c in df.columns if c != "dataset"]
|
||||
df = df[cols]
|
||||
st.dataframe(df)
|
||||
|
||||
# Error samples
|
||||
error_samples = content.get("error_samples", [])
|
||||
if error_samples:
|
||||
with st.expander(f"Error Samples ({len(error_samples)})", expanded=False):
|
||||
for i, sample in enumerate(error_samples):
|
||||
with st.expander(f"Sample {i+1} (Gold: {sample.get('gold', 'N/A')})", expanded=False):
|
||||
st.markdown(
|
||||
'<div style="font-size: 0.85em;">',
|
||||
unsafe_allow_html=True,
|
||||
)
|
||||
st.markdown("**Question:**")
|
||||
st.markdown(convert_latex_for_streamlit(sample.get("question", "N/A")))
|
||||
st.markdown("---")
|
||||
st.markdown(f"**Gold:** `{sample.get('gold', 'N/A')}`")
|
||||
st.markdown("---")
|
||||
st.markdown("**Model Output:**")
|
||||
st.markdown(convert_latex_for_streamlit(sample.get("model_output", "N/A")))
|
||||
st.markdown("</div>", unsafe_allow_html=True)
|
||||
|
||||
|
||||
def render_summary(summary: dict) -> None:
|
||||
col1, col2, col3, col4 = st.columns(4)
|
||||
with col1:
|
||||
st.metric("Loops", summary.get("loop_count", 0))
|
||||
with col2:
|
||||
st.metric("LLM Calls", summary.get("llm_call_count", 0))
|
||||
with col3:
|
||||
llm_time = summary.get("llm_total_time", 0)
|
||||
st.metric("LLM Time", format_duration(llm_time))
|
||||
with col4:
|
||||
success = summary.get("docker_success", 0)
|
||||
fail = summary.get("docker_fail", 0)
|
||||
st.metric("Executions", f"{success}✓ / {fail}✗")
|
||||
@@ -0,0 +1,69 @@
|
||||
"""
|
||||
FT UI Configuration Constants
|
||||
|
||||
Centralized configuration for FT Timeline Viewer.
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
# Event type definition
|
||||
EventType = Literal[
|
||||
"scenario",
|
||||
"llm_call",
|
||||
"template",
|
||||
"experiment",
|
||||
"code",
|
||||
"docker_exec",
|
||||
"evaluator", # Evaluator feedback (separate from docker_exec)
|
||||
"feedback",
|
||||
"token",
|
||||
"time",
|
||||
"settings",
|
||||
"hypothesis",
|
||||
"dataset_selection",
|
||||
]
|
||||
|
||||
# Event type icons
|
||||
ICONS = {
|
||||
"scenario": "🎯",
|
||||
"llm_call": "💬",
|
||||
"template": "📋",
|
||||
"experiment": "🧪",
|
||||
"code": "📄",
|
||||
"docker_exec": "🐳",
|
||||
"evaluator": "📝", # Evaluator feedback icon
|
||||
"feedback": "📊",
|
||||
"token": "🔢",
|
||||
"time": "⏱️",
|
||||
"settings": "⚙️",
|
||||
"hypothesis": "💡",
|
||||
"dataset_selection": "📂",
|
||||
}
|
||||
|
||||
# Evaluator configuration mapping (name, default_stage)
|
||||
EVALUATOR_CONFIG = {
|
||||
"FTDataEvaluator": ("Data Processing", "coding"),
|
||||
"FTCoderEvaluator": ("Micro-batch Test", "coding"),
|
||||
"FTRunnerEvaluator": ("Full Train", "runner"),
|
||||
}
|
||||
|
||||
# Always visible event types
|
||||
ALWAYS_VISIBLE_TYPES = [
|
||||
"scenario",
|
||||
"dataset_selection",
|
||||
"hypothesis",
|
||||
"llm_call",
|
||||
"experiment",
|
||||
"code",
|
||||
"docker_exec",
|
||||
"evaluator",
|
||||
"feedback",
|
||||
]
|
||||
|
||||
# Optional event types with toggle config (label, default_enabled)
|
||||
OPTIONAL_TYPES = {
|
||||
"template": ("📋 Template", False),
|
||||
"token": ("🔢 Token", False),
|
||||
"time": ("⏱️ Time", False),
|
||||
"settings": ("⚙️ Settings", False),
|
||||
}
|
||||
@@ -0,0 +1,455 @@
|
||||
"""
|
||||
FT UI Data Loader
|
||||
Load pkl logs and convert to hierarchical timeline structure
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import streamlit as st
|
||||
|
||||
from rdagent.app.finetune.llm.ui.config import EVALUATOR_CONFIG, EventType
|
||||
from rdagent.log.storage import FileStorage
|
||||
|
||||
|
||||
@dataclass
|
||||
class Event:
|
||||
"""Timeline event"""
|
||||
|
||||
type: EventType
|
||||
timestamp: datetime
|
||||
tag: str
|
||||
title: str
|
||||
content: Any
|
||||
loop_id: int | None = None
|
||||
evo_id: int | None = None
|
||||
stage: str = ""
|
||||
duration: float | None = None
|
||||
success: bool | None = None
|
||||
|
||||
@property
|
||||
def time_str(self) -> str:
|
||||
return self.timestamp.strftime("%H:%M:%S")
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvoLoop:
|
||||
"""Evolution loop containing events"""
|
||||
|
||||
evo_id: int
|
||||
events: list[Event] = field(default_factory=list)
|
||||
success: bool | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Loop:
|
||||
"""Main loop containing stages"""
|
||||
|
||||
loop_id: int
|
||||
exp_gen: list[Event] = field(default_factory=list)
|
||||
coding: dict[int, EvoLoop] = field(default_factory=dict) # evo_id -> EvoLoop
|
||||
runner: list[Event] = field(default_factory=list)
|
||||
feedback: list[Event] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Session:
|
||||
"""Session containing init events and loops"""
|
||||
|
||||
init_events: list[Event] = field(default_factory=list)
|
||||
loops: dict[int, Loop] = field(default_factory=dict) # loop_id -> Loop
|
||||
|
||||
|
||||
def extract_loop_id(tag: str) -> int | None:
|
||||
match = re.search(r"Loop_(\d+)", tag)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def extract_evo_id(tag: str) -> int | None:
|
||||
match = re.search(r"evo_loop_(\d+)", tag)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def extract_stage(tag: str) -> str:
|
||||
if "direct_exp_gen" in tag:
|
||||
return "exp_gen"
|
||||
if "coding" in tag:
|
||||
return "coding"
|
||||
if "running" in tag: # Note: tag uses "running", not "runner"
|
||||
return "runner"
|
||||
if "feedback" in tag:
|
||||
return "feedback"
|
||||
return ""
|
||||
|
||||
|
||||
def get_valid_sessions(log_folder: Path) -> list[str]:
|
||||
if not log_folder.exists():
|
||||
return []
|
||||
sessions = []
|
||||
for d in log_folder.iterdir():
|
||||
if d.is_dir() and d.joinpath("__session__").exists():
|
||||
sessions.append(d.name)
|
||||
return sorted(sessions, reverse=True)
|
||||
|
||||
|
||||
def parse_event(tag: str, content: Any, timestamp: datetime) -> Event | None:
|
||||
loop_id = extract_loop_id(tag)
|
||||
evo_id = extract_evo_id(tag)
|
||||
stage = extract_stage(tag)
|
||||
|
||||
# Scenario
|
||||
if tag == "scenario":
|
||||
model = getattr(content, "base_model", "Unknown")
|
||||
return Event(type="scenario", timestamp=timestamp, tag=tag, title=f"Scenario: {model}", content=content)
|
||||
|
||||
# Dataset selection
|
||||
if "dataset_selection" in tag:
|
||||
selected = content.get("selected_datasets", []) if isinstance(content, dict) else []
|
||||
total = content.get("total_datasets", 0) if isinstance(content, dict) else 0
|
||||
return Event(
|
||||
type="dataset_selection",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Dataset Selection: {len(selected)}/{total}",
|
||||
content=content,
|
||||
)
|
||||
|
||||
# Settings
|
||||
if "SETTINGS" in tag:
|
||||
name = tag.replace("_SETTINGS", "").replace("SETTINGS", "")
|
||||
return Event(type="settings", timestamp=timestamp, tag=tag, title=f"Settings: {name}", content=content)
|
||||
|
||||
# Hypothesis
|
||||
if tag == "hypothesis" or (loop_id is not None and "hypothesis" in tag):
|
||||
return Event(
|
||||
type="hypothesis",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Hypothesis",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="exp_gen",
|
||||
)
|
||||
|
||||
# LLM Call
|
||||
if "debug_llm" in tag:
|
||||
if isinstance(content, dict) and ("user" in content or "system" in content):
|
||||
duration = None
|
||||
if content.get("start") and content.get("end"):
|
||||
duration = (content["end"] - content["start"]).total_seconds()
|
||||
return Event(
|
||||
type="llm_call",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="LLM Call",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage,
|
||||
duration=duration,
|
||||
)
|
||||
|
||||
# Template
|
||||
if "debug_tpl" in tag:
|
||||
if isinstance(content, dict) and "uri" in content:
|
||||
uri = content.get("uri", "")
|
||||
tpl_name = uri.split(":")[-1] if ":" in uri else uri
|
||||
return Event(
|
||||
type="template",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Template: {tpl_name}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
# Experiment generation
|
||||
if "experiment generation" in tag:
|
||||
task_count = len(content) if isinstance(content, list) else 1
|
||||
return Event(
|
||||
type="experiment",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Experiment ({task_count} task)",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
# Evolving code
|
||||
if "evolving code" in tag:
|
||||
file_count = 0
|
||||
if isinstance(content, list):
|
||||
for ws in content:
|
||||
if hasattr(ws, "file_dict"):
|
||||
file_count += len(ws.file_dict)
|
||||
return Event(
|
||||
type="code",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Code ({file_count} files)",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage or "coding",
|
||||
)
|
||||
|
||||
# Benchmark execution (Docker or Conda) - must check before generic docker_run/conda_run
|
||||
if "docker_run.Benchmark" in tag or "conda_run.Benchmark" in tag:
|
||||
benchmark_name = content.get("benchmark_name", "Unknown") if isinstance(content, dict) else "Unknown"
|
||||
exit_code = content.get("exit_code") if isinstance(content, dict) else None
|
||||
success = exit_code == 0 if exit_code is not None else None
|
||||
env_type = "Docker" if "docker_run" in tag else "Conda"
|
||||
return Event(
|
||||
type="docker_exec",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Benchmark ({benchmark_name}) [{env_type}] {'✓' if success else '✗' if success is False else ''}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="runner",
|
||||
success=success,
|
||||
)
|
||||
|
||||
# Environment run (Docker or Conda, raw execution logged before LLM evaluation)
|
||||
if "docker_run." in tag or "conda_run." in tag:
|
||||
is_docker = "docker_run." in tag
|
||||
tag_prefix = "docker_run." if is_docker else "conda_run."
|
||||
class_name = tag.split(tag_prefix)[-1].split(".")[0]
|
||||
|
||||
# FTWorkspace unified logging - determine type from entry command
|
||||
if class_name == "FTWorkspace":
|
||||
entry = content.get("entry", "") if isinstance(content, dict) else ""
|
||||
if "llamafactory-cli train" in entry:
|
||||
# Distinguish by yaml file name: debug_train.yaml for micro-batch, train.yaml for full training
|
||||
if "debug_train.yaml" in entry:
|
||||
evaluator_name, default_stage = "Micro-batch Test", "coding"
|
||||
else:
|
||||
evaluator_name, default_stage = "Full Train", "runner"
|
||||
elif "process_data" in entry.lower():
|
||||
evaluator_name, default_stage = "Data Processing", "coding"
|
||||
elif entry.startswith("rm "):
|
||||
evaluator_name, default_stage = "Cleanup", "runner"
|
||||
else:
|
||||
evaluator_name, default_stage = "Env Run", "coding"
|
||||
else:
|
||||
evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
|
||||
|
||||
exit_code = content.get("exit_code") if isinstance(content, dict) else None
|
||||
success = exit_code == 0 if exit_code is not None else content.get("success")
|
||||
env_label = "Docker" if is_docker else "Conda"
|
||||
title = f"{env_label} ({evaluator_name}) {'✓' if success else '✗' if success is False else ''}"
|
||||
return Event(
|
||||
type="docker_exec",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=title,
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage or default_stage,
|
||||
success=success,
|
||||
)
|
||||
|
||||
# Docker execution (individual evaluator feedback, logged after LLM evaluation)
|
||||
if "docker_exec." in tag:
|
||||
class_name = tag.split("docker_exec.")[-1].split(".")[0]
|
||||
evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
|
||||
success = getattr(content, "final_decision", None)
|
||||
title = f"Eval ({evaluator_name}) {'✓' if success else '✗' if success is False else '?'}"
|
||||
return Event(
|
||||
type="docker_exec",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=title,
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage or default_stage,
|
||||
success=success,
|
||||
)
|
||||
|
||||
# Evaluator feedback (logged from FT evaluators with final_decision)
|
||||
if "evaluator_feedback." in tag:
|
||||
class_name = tag.split("evaluator_feedback.")[-1].split(".")[0]
|
||||
evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
|
||||
success = getattr(content, "final_decision", None)
|
||||
title = f"Eval ({evaluator_name}) {'✓' if success else '✗' if success is False else '?'}"
|
||||
return Event(
|
||||
type="evaluator", # Use dedicated evaluator type with 📝 icon
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=title,
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage or default_stage,
|
||||
success=success,
|
||||
)
|
||||
|
||||
# Final feedback
|
||||
if "feedback.feedback" in tag or (tag.endswith(".feedback") and "evo_loop" not in tag):
|
||||
decision = getattr(content, "decision", None)
|
||||
return Event(
|
||||
type="feedback",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Feedback: {'Accept' if decision else 'Reject'}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="feedback",
|
||||
success=decision,
|
||||
)
|
||||
|
||||
# Benchmark result (supports benchmark_result, benchmark_result.validation, benchmark_result.test)
|
||||
if "benchmark_result" in tag:
|
||||
benchmark_name = content.get("benchmark_name", "Unknown") if isinstance(content, dict) else "Unknown"
|
||||
accuracy = content.get("accuracy_summary", {}) if isinstance(content, dict) else {}
|
||||
# Extract split from tag or content
|
||||
split = content.get("split", "") if isinstance(content, dict) else ""
|
||||
if not split and "." in tag:
|
||||
split = tag.split(".")[-1] # e.g., "validation" or "test" from "benchmark_result.validation"
|
||||
split_label = f" [{split.title()}]" if split and split != "default" else ""
|
||||
return Event(
|
||||
type="feedback",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Benchmark Result{split_label} ({benchmark_name}: {len(accuracy)} datasets)",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="runner",
|
||||
)
|
||||
|
||||
# Runner result
|
||||
if "runner result" in tag:
|
||||
return Event(
|
||||
type="docker_exec",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Full Train",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="runner",
|
||||
)
|
||||
|
||||
# Token cost
|
||||
if "token_cost" in tag:
|
||||
if isinstance(content, dict):
|
||||
total = content.get("total_tokens", 0)
|
||||
return Event(
|
||||
type="token",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Token: {total}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
evo_id=evo_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
# Time info
|
||||
if "time_info" in tag:
|
||||
return Event(
|
||||
type="time", timestamp=timestamp, tag=tag, title="Time Info", content=content, loop_id=loop_id, stage=stage
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
@st.cache_data(ttl=300, hash_funcs={Path: str})
|
||||
def load_ft_session(log_path: Path) -> Session:
|
||||
"""Load events into hierarchical session structure"""
|
||||
session = Session()
|
||||
storage = FileStorage(log_path)
|
||||
|
||||
events = []
|
||||
for msg in storage.iter_msg():
|
||||
if not msg.tag:
|
||||
continue
|
||||
event = parse_event(msg.tag, msg.content, msg.timestamp)
|
||||
if event:
|
||||
events.append(event)
|
||||
|
||||
# Sort by timestamp
|
||||
events.sort(key=lambda e: e.timestamp)
|
||||
|
||||
# Organize into hierarchy
|
||||
for event in events:
|
||||
if event.loop_id is None:
|
||||
session.init_events.append(event)
|
||||
continue
|
||||
|
||||
# Ensure loop exists
|
||||
if event.loop_id not in session.loops:
|
||||
session.loops[event.loop_id] = Loop(loop_id=event.loop_id)
|
||||
loop = session.loops[event.loop_id]
|
||||
|
||||
# Place event in appropriate stage
|
||||
if event.stage == "exp_gen":
|
||||
loop.exp_gen.append(event)
|
||||
elif event.stage == "coding":
|
||||
if event.evo_id is not None:
|
||||
if event.evo_id not in loop.coding:
|
||||
loop.coding[event.evo_id] = EvoLoop(evo_id=event.evo_id)
|
||||
evo = loop.coding[event.evo_id]
|
||||
evo.events.append(event)
|
||||
# Use evaluator feedback (final_decision) for evo success, fallback to docker_exec
|
||||
if event.type in ("evaluator", "docker_exec") and event.success is not None:
|
||||
if evo.success is None:
|
||||
evo.success = event.success
|
||||
else:
|
||||
evo.success = evo.success and event.success # AND logic: all evaluators must pass
|
||||
else:
|
||||
# Coding events without evo_id go to evo 0
|
||||
if 0 not in loop.coding:
|
||||
loop.coding[0] = EvoLoop(evo_id=0)
|
||||
loop.coding[0].events.append(event)
|
||||
elif event.stage == "runner":
|
||||
loop.runner.append(event)
|
||||
elif event.stage == "feedback":
|
||||
loop.feedback.append(event)
|
||||
else:
|
||||
# Unknown stage - put in exp_gen
|
||||
loop.exp_gen.append(event)
|
||||
|
||||
return session
|
||||
|
||||
|
||||
def get_summary(session: Session) -> dict:
|
||||
"""Get summary statistics"""
|
||||
llm_calls = []
|
||||
docker_execs = []
|
||||
|
||||
# Collect from init
|
||||
for e in session.init_events:
|
||||
if e.type == "llm_call":
|
||||
llm_calls.append(e)
|
||||
elif e.type == "docker_exec":
|
||||
docker_execs.append(e)
|
||||
|
||||
# Collect from loops
|
||||
for loop in session.loops.values():
|
||||
for e in loop.exp_gen + loop.runner + loop.feedback:
|
||||
if e.type == "llm_call":
|
||||
llm_calls.append(e)
|
||||
elif e.type == "docker_exec":
|
||||
docker_execs.append(e)
|
||||
for evo in loop.coding.values():
|
||||
for e in evo.events:
|
||||
if e.type == "llm_call":
|
||||
llm_calls.append(e)
|
||||
elif e.type == "docker_exec":
|
||||
docker_execs.append(e)
|
||||
|
||||
return {
|
||||
"loop_count": len(session.loops),
|
||||
"llm_call_count": len(llm_calls),
|
||||
"llm_total_time": sum(e.duration or 0 for e in llm_calls),
|
||||
"docker_success": sum(1 for e in docker_execs if e.success is True),
|
||||
"docker_fail": sum(1 for e in docker_execs if e.success is False),
|
||||
}
|
||||
@@ -0,0 +1,587 @@
|
||||
"""
|
||||
FT Job Summary View
|
||||
Display summary table for all tasks in a job directory
|
||||
"""
|
||||
|
||||
import pickle
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import streamlit as st
|
||||
from pandas.io.formats.style import Styler
|
||||
|
||||
from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
|
||||
|
||||
|
||||
def is_valid_task(task_path: Path) -> bool:
|
||||
"""Check if directory is a valid FT task (has __session__ subdirectory)"""
|
||||
return task_path.is_dir() and (task_path / "__session__").exists()
|
||||
|
||||
|
||||
def get_loop_dirs(task_path: Path) -> list[Path]:
|
||||
"""Get sorted list of Loop directories"""
|
||||
loops = [d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")]
|
||||
return sorted(loops, key=lambda d: int(d.name.split("_")[1]))
|
||||
|
||||
|
||||
def extract_benchmark_score(loop_path: Path, split: str = "") -> tuple[str, float, bool] | None:
|
||||
"""Extract benchmark score, metric name, and direction from loop directory.
|
||||
|
||||
Args:
|
||||
loop_path: Path to loop directory
|
||||
split: Filter by split type ("validation", "test", or "" for any)
|
||||
|
||||
Returns:
|
||||
(metric_name, score, higher_is_better) or None
|
||||
- metric_name includes "(average)" suffix if multiple datasets are averaged
|
||||
- higher_is_better: True if higher values are better
|
||||
"""
|
||||
for pkl_file in loop_path.rglob("**/benchmark_result*/**/*.pkl"):
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
content = pickle.load(f)
|
||||
if isinstance(content, dict):
|
||||
# Check split filter
|
||||
content_split = content.get("split", "")
|
||||
if split and content_split != split:
|
||||
continue
|
||||
|
||||
benchmark_name = content.get("benchmark_name", "")
|
||||
accuracy_summary = content.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
result = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if result is not None:
|
||||
return result
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def extract_benchmark_scores(loop_path: Path) -> dict[str, tuple[str, float, bool] | None]:
|
||||
"""Extract both validation and test benchmark scores from loop directory.
|
||||
|
||||
Returns:
|
||||
Dict with keys "validation" and "test", each containing
|
||||
(metric_name, score, higher_is_better) or None
|
||||
"""
|
||||
return {
|
||||
"validation": extract_benchmark_score(loop_path, split="validation"),
|
||||
"test": extract_benchmark_score(loop_path, split="test"),
|
||||
}
|
||||
|
||||
|
||||
def extract_baseline_score(task_path: Path) -> tuple[str, float] | None:
|
||||
"""Extract baseline benchmark score from scenario object (legacy, validation only).
|
||||
|
||||
Returns:
|
||||
(metric_name, score) or None
|
||||
"""
|
||||
scenario_dir = task_path / "scenario"
|
||||
if not scenario_dir.exists():
|
||||
return None
|
||||
|
||||
for pkl_file in scenario_dir.rglob("*.pkl"):
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
scenario = pickle.load(f)
|
||||
baseline_score = getattr(scenario, "baseline_benchmark_score", None)
|
||||
if baseline_score and isinstance(baseline_score, dict):
|
||||
benchmark_name = getattr(scenario, "target_benchmark", "")
|
||||
accuracy_summary = baseline_score.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
result = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if result is not None:
|
||||
metric_name, score, _ = result
|
||||
return metric_name, score
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def extract_baseline_scores(task_path: Path) -> dict[str, tuple[str, float, bool] | None]:
|
||||
"""Extract both validation and test baseline benchmark scores from scenario.
|
||||
|
||||
Returns:
|
||||
{"validation": (metric_name, score, higher_is_better) or None,
|
||||
"test": (metric_name, score, higher_is_better) or None}
|
||||
"""
|
||||
scenario_dir = task_path / "scenario"
|
||||
if not scenario_dir.exists():
|
||||
return {"validation": None, "test": None}
|
||||
|
||||
for pkl_file in scenario_dir.rglob("*.pkl"):
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
scenario = pickle.load(f)
|
||||
|
||||
benchmark_name = getattr(scenario, "target_benchmark", "")
|
||||
result = {"validation": None, "test": None}
|
||||
|
||||
# Validation score
|
||||
baseline_val = getattr(scenario, "baseline_benchmark_score", None)
|
||||
if baseline_val and isinstance(baseline_val, dict):
|
||||
accuracy_summary = baseline_val.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
core = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if core:
|
||||
result["validation"] = core
|
||||
|
||||
# Test score (new format only)
|
||||
baseline_test = getattr(scenario, "baseline_benchmark_score_test", None)
|
||||
if baseline_test and isinstance(baseline_test, dict):
|
||||
accuracy_summary = baseline_test.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
core = get_core_metric_score(benchmark_name, accuracy_summary)
|
||||
if core:
|
||||
result["test"] = core
|
||||
|
||||
return result
|
||||
except Exception:
|
||||
pass
|
||||
return {"validation": None, "test": None}
|
||||
|
||||
|
||||
def get_loop_status(
|
||||
task_path: Path, loop_id: int
|
||||
) -> tuple[str, float | None, float | None, str | None, bool | None, bool]:
|
||||
"""
|
||||
Get loop status, validation score, test score, metric name with direction arrow, feedback decision, and direction.
|
||||
Returns: (status_str, val_score_or_none, test_score_or_none, metric_display_or_none, feedback_decision, higher_is_better)
|
||||
Status: 'C'=Coding, 'R'=Running, 'X'=Failed, score_str=Success
|
||||
metric_display: metric name with direction arrow (e.g., "accuracy ↑")
|
||||
feedback_decision: True=accepted, False=rejected, None=no feedback
|
||||
higher_is_better: True if higher values are better for this metric
|
||||
"""
|
||||
loop_path = task_path / f"Loop_{loop_id}"
|
||||
if not loop_path.exists():
|
||||
return "-", None, None, None, None, True
|
||||
|
||||
# Check for benchmark results first (highest priority - means completed)
|
||||
scores = extract_benchmark_scores(loop_path)
|
||||
val_result = scores.get("validation")
|
||||
test_result = scores.get("test")
|
||||
|
||||
# Fallback to old format (no split) if no validation/test found
|
||||
if val_result is None and test_result is None:
|
||||
legacy_result = extract_benchmark_score(loop_path, split="")
|
||||
if legacy_result is not None:
|
||||
val_result = legacy_result # Treat legacy as validation
|
||||
|
||||
# Get feedback decision (used for both score coloring and fallback status)
|
||||
feedback_decision = None
|
||||
feedback_files = list(loop_path.rglob("**/feedback/**/*.pkl"))
|
||||
for f in feedback_files:
|
||||
try:
|
||||
with open(f, "rb") as fp:
|
||||
content = pickle.load(fp)
|
||||
decision = getattr(content, "decision", None)
|
||||
if decision is not None:
|
||||
feedback_decision = decision
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if val_result is not None:
|
||||
metric_name, val_score, higher_is_better = val_result
|
||||
test_score = test_result[1] if test_result else None
|
||||
arrow = "↑" if higher_is_better else "↓"
|
||||
metric_display = f"{metric_name} {arrow}"
|
||||
# Format: "val/test" or just "val" if no test
|
||||
if test_score is not None:
|
||||
status_str = f"{val_score:.2f}/{test_score:.2f}"
|
||||
else:
|
||||
status_str = f"{val_score:.2f}"
|
||||
return status_str, val_score, test_score, metric_display, feedback_decision, higher_is_better
|
||||
|
||||
# Check feedback stage (no benchmark result, use feedback decision directly)
|
||||
if feedback_decision is not None:
|
||||
return ("OK" if feedback_decision else "X"), None, None, None, feedback_decision, True
|
||||
|
||||
# Check running stage
|
||||
running_files = list(loop_path.rglob("**/running/**/*.pkl"))
|
||||
if running_files:
|
||||
return "R", None, None, None, None, True
|
||||
|
||||
# Check coding stage
|
||||
coding_files = list(loop_path.rglob("**/coding/**/*.pkl"))
|
||||
if coding_files:
|
||||
return "C", None, None, None, None, True
|
||||
|
||||
# Has directory but no recognized files
|
||||
return "?", None, None, None, None, True
|
||||
|
||||
|
||||
def get_max_loops(job_path: Path) -> int:
|
||||
"""Get maximum number of loops across all tasks"""
|
||||
max_loops = 0
|
||||
for task_dir in job_path.iterdir():
|
||||
if is_valid_task(task_dir):
|
||||
loops = get_loop_dirs(task_dir)
|
||||
max_loops = max(max_loops, len(loops))
|
||||
return max_loops
|
||||
|
||||
|
||||
def get_job_summary_df(job_path: Path) -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||
"""Generate summary DataFrame and decision DataFrame for all tasks in job
|
||||
|
||||
Each loop column shows "val/test" format when both scores are available.
|
||||
Best columns show the best validation and test scores separately.
|
||||
|
||||
Returns:
|
||||
(df, decisions_df): df is display data, decisions_df has same structure
|
||||
but values are True/False/None for feedback decision
|
||||
"""
|
||||
if not job_path.exists():
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
|
||||
if not tasks:
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
max_loops = get_max_loops(job_path)
|
||||
if max_loops == 0:
|
||||
max_loops = 10 # Default display columns
|
||||
|
||||
data = []
|
||||
decisions_data = []
|
||||
for task_path in tasks:
|
||||
row = {"Task": task_path.name}
|
||||
decision_row = {"Task": task_path.name}
|
||||
best_val_score = None
|
||||
best_test_score = None
|
||||
best_metric = None
|
||||
best_higher_is_better = True # Default to higher is better
|
||||
|
||||
# Extract baseline scores (validation and test) from scenario
|
||||
baseline_scores = extract_baseline_scores(task_path)
|
||||
val_baseline = baseline_scores.get("validation")
|
||||
test_baseline = baseline_scores.get("test")
|
||||
if val_baseline and test_baseline:
|
||||
row["Baseline"] = f"{val_baseline[1]:.2f}/{test_baseline[1]:.2f}"
|
||||
elif val_baseline:
|
||||
row["Baseline"] = f"{val_baseline[1]:.2f}"
|
||||
else:
|
||||
row["Baseline"] = "-"
|
||||
decision_row["Baseline"] = None
|
||||
|
||||
for i in range(max_loops):
|
||||
status, val_score, test_score, metric_name, feedback_decision, higher_is_better = get_loop_status(
|
||||
task_path, i
|
||||
)
|
||||
row[f"L{i}"] = status
|
||||
decision_row[f"L{i}"] = feedback_decision
|
||||
if val_score is not None:
|
||||
# Use higher_is_better to determine if this score is better
|
||||
if best_val_score is None:
|
||||
best_val_score = val_score
|
||||
best_higher_is_better = higher_is_better
|
||||
best_metric = metric_name
|
||||
elif (higher_is_better and val_score > best_val_score) or (
|
||||
not higher_is_better and val_score < best_val_score
|
||||
):
|
||||
best_val_score = val_score
|
||||
best_higher_is_better = higher_is_better
|
||||
best_metric = metric_name
|
||||
if test_score is not None:
|
||||
# Use same direction as validation score for consistency
|
||||
if best_test_score is None:
|
||||
best_test_score = test_score
|
||||
elif (best_higher_is_better and test_score > best_test_score) or (
|
||||
not best_higher_is_better and test_score < best_test_score
|
||||
):
|
||||
best_test_score = test_score
|
||||
|
||||
# Show best validation and test scores
|
||||
if best_val_score is not None and best_test_score is not None:
|
||||
row["Best"] = f"{best_val_score:.2f}/{best_test_score:.2f}"
|
||||
elif best_val_score is not None:
|
||||
row["Best"] = f"{best_val_score:.2f}"
|
||||
else:
|
||||
row["Best"] = "-"
|
||||
row["Metric"] = best_metric if best_metric else "-"
|
||||
decision_row["Metric"] = None
|
||||
decision_row["Best"] = None
|
||||
data.append(row)
|
||||
decisions_data.append(decision_row)
|
||||
|
||||
# Ensure column order: Task, Metric, Baseline, L0, L1, ..., Best
|
||||
df = pd.DataFrame(data)
|
||||
decisions_df = pd.DataFrame(decisions_data)
|
||||
if not df.empty:
|
||||
loop_cols = [c for c in df.columns if c.startswith("L")]
|
||||
cols = ["Task", "Metric", "Baseline"] + sorted(loop_cols, key=lambda x: int(x[1:])) + ["Best"]
|
||||
df = df[cols]
|
||||
decisions_df = decisions_df[cols]
|
||||
return df, decisions_df
|
||||
|
||||
|
||||
def style_status_cell(val: str, decision: bool | None = None) -> str:
|
||||
"""Style cell based on status value and feedback decision
|
||||
|
||||
Args:
|
||||
val: The cell value
|
||||
decision: True=accepted (green), False=rejected (red), None=no feedback (gray)
|
||||
"""
|
||||
if val == "-":
|
||||
return "color: #888"
|
||||
if val == "C":
|
||||
return "color: #f0ad4e; font-weight: bold" # Orange for coding
|
||||
if val == "R":
|
||||
return "color: #5bc0de; font-weight: bold" # Blue for running
|
||||
if val == "X":
|
||||
return "color: #d9534f; font-weight: bold" # Red for failed
|
||||
if val == "OK":
|
||||
return "color: #5cb85c; font-weight: bold" # Green for success
|
||||
if val == "?":
|
||||
return "color: #888"
|
||||
|
||||
# Check if it's a numeric score (with optional "/" separator)
|
||||
is_numeric = False
|
||||
try:
|
||||
float(val)
|
||||
is_numeric = True
|
||||
except ValueError:
|
||||
if "/" in val:
|
||||
parts = val.split("/")
|
||||
try:
|
||||
float(parts[0])
|
||||
is_numeric = True
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
if is_numeric:
|
||||
# Use decision for coloring (use == instead of is for numpy.bool_ compatibility)
|
||||
if decision == True:
|
||||
return "color: #5cb85c; font-weight: bold" # Green for accepted
|
||||
elif decision == False:
|
||||
return "color: #d9534f; font-weight: bold" # Red for rejected
|
||||
else:
|
||||
return "color: #888" # Gray for no feedback
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
def style_df_with_decisions(df: pd.DataFrame, decisions_df: pd.DataFrame) -> Styler:
|
||||
"""Apply styling to dataframe based on decision data
|
||||
|
||||
Args:
|
||||
df: Display dataframe
|
||||
decisions_df: DataFrame with same shape, containing True/False/None values
|
||||
"""
|
||||
|
||||
def apply_styles(row_idx: int, col: str) -> str:
|
||||
val = df.iloc[row_idx][col]
|
||||
decision = decisions_df.iloc[row_idx][col] if col in decisions_df.columns else None
|
||||
return style_status_cell(str(val), decision)
|
||||
|
||||
# Build style matrix
|
||||
styles = pd.DataFrame("", index=df.index, columns=df.columns)
|
||||
for row_idx in range(len(df)):
|
||||
for col in df.columns:
|
||||
styles.iloc[row_idx][col] = apply_styles(row_idx, col)
|
||||
|
||||
return df.style.apply(lambda _: styles, axis=None)
|
||||
|
||||
|
||||
def render_job_summary(job_path: Path, is_root: bool = False) -> None:
|
||||
"""Render job summary UI"""
|
||||
title = "Standalone Tasks" if is_root else f"Job: {job_path.name}"
|
||||
st.subheader(title)
|
||||
|
||||
df, decisions_df = get_job_summary_df(job_path)
|
||||
if df.empty:
|
||||
st.warning("No valid tasks found in this job directory")
|
||||
return
|
||||
|
||||
# Display legend
|
||||
st.markdown(
|
||||
"**Legend:** "
|
||||
"<span style='color:#f0ad4e'>C</span>=Coding, "
|
||||
"<span style='color:#5bc0de'>R</span>=Running, "
|
||||
"<span style='color:#5cb85c'>Score</span>=Accepted, "
|
||||
"<span style='color:#d9534f'>Score/X</span>=Rejected/Failed, "
|
||||
"<span style='color:#888'>Score</span>=No feedback",
|
||||
unsafe_allow_html=True,
|
||||
)
|
||||
|
||||
# Style and display dataframe
|
||||
styled_df = style_df_with_decisions(df, decisions_df)
|
||||
st.dataframe(styled_df, use_container_width=True, hide_index=True)
|
||||
|
||||
# Summary stats
|
||||
col1, col2, col3 = st.columns(3)
|
||||
with col1:
|
||||
st.metric("Tasks", len(df))
|
||||
with col2:
|
||||
# Count tasks with any score
|
||||
tasks_with_score = df["Best"].apply(lambda x: x != "-").sum()
|
||||
st.metric("With Score", tasks_with_score)
|
||||
with col3:
|
||||
# Count tasks with at least one improved loop (decision=True)
|
||||
loop_cols = [c for c in decisions_df.columns if c.startswith("L")]
|
||||
tasks_improved = decisions_df[loop_cols].apply(lambda row: any(v is True for v in row), axis=1).sum()
|
||||
st.metric("Improved", tasks_improved)
|
||||
|
||||
# Detailed scores table
|
||||
render_task_detail_selector(job_path)
|
||||
|
||||
|
||||
def extract_full_benchmark(loop_path: Path, split: str = "") -> dict | None:
|
||||
"""Extract full accuracy_summary from loop directory.
|
||||
|
||||
Args:
|
||||
loop_path: Path to loop directory
|
||||
split: Filter by split type ("validation", "test", or "" for any)
|
||||
|
||||
Returns:
|
||||
accuracy_summary dict {dataset: {metric: value, ...}, ...} or None
|
||||
"""
|
||||
for pkl_file in loop_path.rglob("**/benchmark_result*/**/*.pkl"):
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
content = pickle.load(f)
|
||||
if isinstance(content, dict):
|
||||
# Check split filter
|
||||
content_split = content.get("split", "")
|
||||
if split and content_split != split:
|
||||
continue
|
||||
|
||||
accuracy_summary = content.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
return accuracy_summary
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def extract_baseline_full_benchmark(task_path: Path, split: str = "validation") -> dict | None:
|
||||
"""Extract full accuracy_summary from baseline scenario.
|
||||
|
||||
Args:
|
||||
task_path: Path to task directory
|
||||
split: "validation" or "test"
|
||||
|
||||
Returns:
|
||||
accuracy_summary dict or None
|
||||
"""
|
||||
scenario_dir = task_path / "scenario"
|
||||
if not scenario_dir.exists():
|
||||
return None
|
||||
|
||||
for pkl_file in scenario_dir.rglob("*.pkl"):
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
scenario = pickle.load(f)
|
||||
|
||||
if split == "validation":
|
||||
baseline = getattr(scenario, "baseline_benchmark_score", None)
|
||||
else:
|
||||
baseline = getattr(scenario, "baseline_benchmark_score_test", None)
|
||||
|
||||
if baseline and isinstance(baseline, dict):
|
||||
accuracy_summary = baseline.get("accuracy_summary", {})
|
||||
if isinstance(accuracy_summary, dict) and accuracy_summary:
|
||||
return accuracy_summary
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def get_task_full_benchmark_df(task_path: Path, split: str) -> pd.DataFrame:
|
||||
"""Generate full benchmark table for a single task and split.
|
||||
|
||||
Returns DataFrame with columns: Dataset, Metric, Baseline, Loop_0, Loop_1, ...
|
||||
Each row is a dataset-metric combination.
|
||||
"""
|
||||
# Collect all sources (Baseline + Loops)
|
||||
sources = ["Baseline"]
|
||||
loop_dirs = sorted(
|
||||
[d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")],
|
||||
key=lambda x: int(x.name.split("_")[1]),
|
||||
)
|
||||
sources.extend([d.name for d in loop_dirs])
|
||||
|
||||
# Collect all accuracy_summaries
|
||||
all_summaries = {}
|
||||
|
||||
# Baseline
|
||||
baseline_summary = extract_baseline_full_benchmark(task_path, split)
|
||||
if baseline_summary:
|
||||
all_summaries["Baseline"] = baseline_summary
|
||||
|
||||
# Loops
|
||||
for loop_dir in loop_dirs:
|
||||
loop_summary = extract_full_benchmark(loop_dir, split)
|
||||
if loop_summary:
|
||||
all_summaries[loop_dir.name] = loop_summary
|
||||
|
||||
if not all_summaries:
|
||||
return pd.DataFrame()
|
||||
|
||||
# Collect all dataset-metric combinations
|
||||
all_keys = set()
|
||||
for summary in all_summaries.values():
|
||||
for dataset, metrics in summary.items():
|
||||
if isinstance(metrics, dict):
|
||||
for metric in metrics.keys():
|
||||
all_keys.add((dataset, metric))
|
||||
|
||||
# Sort keys for consistent display
|
||||
all_keys = sorted(all_keys)
|
||||
|
||||
# Build table data
|
||||
data = []
|
||||
for dataset, metric in all_keys:
|
||||
row = {"Dataset": dataset, "Metric": metric}
|
||||
for source in sources:
|
||||
summary = all_summaries.get(source, {})
|
||||
metrics_dict = summary.get(dataset, {})
|
||||
value = metrics_dict.get(metric) if isinstance(metrics_dict, dict) else None
|
||||
if value is not None:
|
||||
row[source] = f"{value:.2f}" if isinstance(value, float) else str(value)
|
||||
else:
|
||||
row[source] = "-"
|
||||
data.append(row)
|
||||
|
||||
df = pd.DataFrame(data)
|
||||
# Ensure column order
|
||||
if not df.empty:
|
||||
cols = ["Dataset", "Metric"] + [s for s in sources if s in df.columns]
|
||||
df = df[cols]
|
||||
return df
|
||||
|
||||
|
||||
def render_task_detail_selector(job_path: Path) -> None:
|
||||
"""Render task selector dropdown and full benchmark tables."""
|
||||
tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
|
||||
if not tasks:
|
||||
return
|
||||
|
||||
st.markdown("---")
|
||||
st.subheader("Detailed Benchmark Scores")
|
||||
|
||||
# Task selector dropdown
|
||||
task_names = [t.name for t in tasks]
|
||||
selected_task = st.selectbox("Select Task", options=task_names, index=0, key="task_detail_selector")
|
||||
|
||||
if selected_task:
|
||||
task_path = job_path / selected_task
|
||||
|
||||
# Display Validation and Test tables side by side
|
||||
col1, col2 = st.columns(2)
|
||||
|
||||
with col1:
|
||||
st.markdown("**Validation**")
|
||||
df_val = get_task_full_benchmark_df(task_path, "validation")
|
||||
if not df_val.empty:
|
||||
st.dataframe(df_val, use_container_width=True, hide_index=True)
|
||||
else:
|
||||
st.info("No validation scores")
|
||||
|
||||
with col2:
|
||||
st.markdown("**Test**")
|
||||
df_test = get_task_full_benchmark_df(task_path, "test")
|
||||
if not df_test.empty:
|
||||
st.dataframe(df_test, use_container_width=True, hide_index=True)
|
||||
else:
|
||||
st.info("No test scores")
|
||||
@@ -12,12 +12,13 @@ from typing_extensions import Annotated
|
||||
|
||||
from rdagent.app.qlib_rd_loop.conf import FACTOR_PROP_SETTING
|
||||
from rdagent.components.workflow.rd_loop import RDLoop
|
||||
from rdagent.core.exception import FactorEmptyError
|
||||
from rdagent.core.exception import CoderError, FactorEmptyError
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
|
||||
|
||||
class FactorRDLoop(RDLoop):
|
||||
skip_loop_error = (FactorEmptyError,)
|
||||
skip_loop_error = (FactorEmptyError, CoderError)
|
||||
skip_loop_error_stepname = "feedback"
|
||||
|
||||
def running(self, prev_out: dict[str, Any]):
|
||||
exp = self.runner.develop(prev_out["coding"])
|
||||
|
||||
@@ -136,23 +136,6 @@ class FactorReportLoop(FactorRDLoop, metaclass=LoopMeta):
|
||||
logger.log_object(exp.sub_workspace_list, tag="coder result")
|
||||
return exp
|
||||
|
||||
def feedback(self, prev_out: dict[str, Any]):
|
||||
e = prev_out.get(self.EXCEPTION_KEY, None)
|
||||
if e is not None:
|
||||
feedback = HypothesisFeedback(
|
||||
observations=str(e),
|
||||
hypothesis_evaluation="",
|
||||
new_hypothesis="",
|
||||
reason="",
|
||||
decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["direct_exp_gen"]["exp_gen"], feedback))
|
||||
else:
|
||||
feedback = self.summarizer.generate_feedback(prev_out["running"], self.trace)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["running"], feedback))
|
||||
|
||||
|
||||
def main(report_folder=None, path=None, all_duration=None, checkout=True):
|
||||
"""
|
||||
|
||||
@@ -107,15 +107,13 @@ class QuantRDLoop(RDLoop):
|
||||
reason="",
|
||||
decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["direct_exp_gen"]["exp_gen"], feedback))
|
||||
else:
|
||||
if prev_out["direct_exp_gen"]["propose"].action == "factor":
|
||||
feedback = self.factor_summarizer.generate_feedback(prev_out["running"], self.trace)
|
||||
elif prev_out["direct_exp_gen"]["propose"].action == "model":
|
||||
feedback = self.model_summarizer.generate_feedback(prev_out["running"], self.trace)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["running"], feedback))
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
return feedback
|
||||
|
||||
|
||||
def main(
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.finetune.conf import get_ft_env
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
app = typer.Typer(help="Run LLM fine-tuning environment commands.")
|
||||
|
||||
|
||||
@app.command()
|
||||
def run(
|
||||
dataset: str,
|
||||
model: str,
|
||||
cmd: str,
|
||||
local_path: str = "./",
|
||||
mount_path: str | None = None,
|
||||
):
|
||||
"""
|
||||
Launch the LLM fine-tuning environment for a specific dataset and model, then run the
|
||||
provided command.
|
||||
|
||||
Example:
|
||||
1) start the container:
|
||||
dotenv run -- python -m rdagent.app.utils.ws_ft alpaca_gpt4_zh qwen2-7b "sleep 3600" --local-path your_workspace
|
||||
|
||||
2) then run the following command to enter the latest container:
|
||||
- docker exec -it `docker ps --filter 'status=running' -l --format '{{.Names}}'` bash
|
||||
Or you can attach to the container by specifying the container name (find it in the run info)
|
||||
- docker exec -it sweet_robinson bash
|
||||
|
||||
Arguments:
|
||||
dataset: The dataset name for fine-tuning.
|
||||
model: The base model name for fine-tuning.
|
||||
cmd: The shell command or script entry point to execute inside
|
||||
the environment.
|
||||
"""
|
||||
# Don't set time limitation and always disable cache
|
||||
env = get_ft_env(
|
||||
running_timeout_period=None,
|
||||
enable_cache=False,
|
||||
)
|
||||
|
||||
if mount_path is not None:
|
||||
env.conf.mount_path = mount_path
|
||||
|
||||
env.run(entry=cmd, local_path=local_path)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
app()
|
||||
@@ -28,6 +28,7 @@ class CoSTEER(Developer[Experiment]):
|
||||
with_knowledge: bool = True,
|
||||
knowledge_self_gen: bool = True,
|
||||
max_loop: int | None = None,
|
||||
stop_eval_chain_on_fail: bool = False,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
@@ -46,6 +47,7 @@ class CoSTEER(Developer[Experiment]):
|
||||
self.evolving_strategy = es
|
||||
self.evaluator = eva
|
||||
self.evolving_version = evolving_version
|
||||
self.stop_eval_chain_on_fail = stop_eval_chain_on_fail
|
||||
|
||||
# init rag method
|
||||
self.rag = (
|
||||
@@ -99,10 +101,10 @@ class CoSTEER(Developer[Experiment]):
|
||||
evolving_strategy=self.evolving_strategy,
|
||||
rag=self.rag,
|
||||
with_knowledge=self.with_knowledge,
|
||||
with_feedback=True,
|
||||
knowledge_self_gen=self.knowledge_self_gen,
|
||||
enable_filelock=self.settings.enable_filelock,
|
||||
filelock_path=self.settings.filelock_path,
|
||||
stop_eval_chain_on_fail=self.stop_eval_chain_on_fail,
|
||||
)
|
||||
|
||||
# Evolving the solution
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import json
|
||||
from abc import abstractmethod
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, List
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Dict, Generator, List
|
||||
|
||||
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.evaluation import Evaluator, Feedback
|
||||
from rdagent.core.evolving_agent import RAGEvaluator
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import Task, Workspace
|
||||
from rdagent.core.utils import multiprocessing_wrapper
|
||||
@@ -37,12 +39,16 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
It is design align the phases of the implemented code
|
||||
- Execution -> Return Value -> Code -> Final Decision
|
||||
"""
|
||||
execution: str
|
||||
execution: str # Summarized execution feedback
|
||||
# execution_feedback
|
||||
return_checking: str | None # including every check in the testing (constraints about the generated value)
|
||||
# value_feedback, shape_feedback, value_generated_flag
|
||||
code: str
|
||||
final_decision: bool | None = None
|
||||
raw_execution: str = "" # Full raw stdout for UI display
|
||||
source_feedback: Dict[str, bool] = field(
|
||||
default_factory=dict
|
||||
) # Record the source of the feedback since it might be merged from multiple feedbacks, stores the mapping from source tag to its final_decision, this dict also includes the feedback source of itself
|
||||
|
||||
@staticmethod
|
||||
def val_and_update_init_dict(data: dict) -> dict:
|
||||
@@ -72,8 +78,8 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
raise ValueError(f"'final_decision' must be a boolean, not {type(data['final_decision'])}")
|
||||
|
||||
for attr in "execution", "return_checking", "code":
|
||||
if data[attr] is not None and not isinstance(data[attr], str):
|
||||
raise ValueError(f"'{attr}' must be a string, not {type(data[attr])}")
|
||||
if data.get(attr) is not None and not isinstance(data[attr], str):
|
||||
data[attr] = json.dumps(data[attr], indent=2, ensure_ascii=False)
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
@@ -95,6 +101,10 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
attr,
|
||||
"\n\n".join([getattr(_fb, attr) for _fb in feedback_li if getattr(_fb, attr) is not None]),
|
||||
)
|
||||
fb.source_feedback = {}
|
||||
for _fb in feedback_li:
|
||||
for tag, decision in _fb.source_feedback.items():
|
||||
fb.source_feedback[tag] = decision
|
||||
return fb
|
||||
|
||||
def __str__(self) -> str:
|
||||
@@ -125,6 +135,7 @@ class CoSTEERSingleFeedbackDeprecated(CoSTEERSingleFeedback):
|
||||
final_feedback: str = None,
|
||||
value_generated_flag: bool = None,
|
||||
final_decision_based_on_gt: bool = None,
|
||||
source_feedback: dict = None,
|
||||
) -> None:
|
||||
self.execution_feedback = execution_feedback
|
||||
self.code_feedback = code_feedback
|
||||
@@ -133,6 +144,7 @@ class CoSTEERSingleFeedbackDeprecated(CoSTEERSingleFeedback):
|
||||
self.final_feedback = final_feedback
|
||||
self.value_generated_flag = value_generated_flag
|
||||
self.final_decision_based_on_gt = final_decision_based_on_gt
|
||||
self.source_feedback = source_feedback if source_feedback is not None else {}
|
||||
|
||||
# TODO:
|
||||
# Not general enough. So we should not put them in the general costeer feedback
|
||||
@@ -226,6 +238,7 @@ class CoSTEEREvaluator(Evaluator):
|
||||
# TODO:
|
||||
# I think we should have unified interface for all evaluates, for examples.
|
||||
# So we should adjust the interface of other factors
|
||||
# Based on the implementation, I think a better name is some name like task-implement evaluator
|
||||
@abstractmethod
|
||||
def evaluate(
|
||||
self,
|
||||
@@ -237,19 +250,23 @@ class CoSTEEREvaluator(Evaluator):
|
||||
raise NotImplementedError("Please implement the `evaluator` method")
|
||||
|
||||
|
||||
class CoSTEERMultiEvaluator(CoSTEEREvaluator):
|
||||
class CoSTEERMultiEvaluator(RAGEvaluator):
|
||||
"""This is for evaluation of experiment. Due to we have multiple tasks, so we will return a list of evaluation feebacks"""
|
||||
|
||||
def __init__(self, single_evaluator: CoSTEEREvaluator | list[CoSTEEREvaluator], *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
def __init__(self, single_evaluator: CoSTEEREvaluator | list[CoSTEEREvaluator], scen: "Scenario") -> None:
|
||||
super().__init__()
|
||||
self.scen = scen
|
||||
self.single_evaluator = single_evaluator
|
||||
|
||||
def evaluate(
|
||||
def evaluate_iter(
|
||||
self,
|
||||
evo: EvolvingItem,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERMultiFeedback:
|
||||
) -> Generator[CoSTEERMultiFeedback, EvolvingItem | None, CoSTEERMultiFeedback]:
|
||||
evo = yield CoSTEERMultiFeedback(
|
||||
[]
|
||||
) # it will receive the evo first, so the first yield is for get the sent evo instead of generate useful feedback
|
||||
|
||||
eval_l = self.single_evaluator if isinstance(self.single_evaluator, list) else [self.single_evaluator]
|
||||
|
||||
# 1) Evaluate each sub_task
|
||||
@@ -279,7 +296,12 @@ class CoSTEERMultiEvaluator(CoSTEEREvaluator):
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
# None received, we skip the rest and return the overall feedback directly
|
||||
evo_next_iter = yield CoSTEERMultiFeedback(multi_implementation_feedback)
|
||||
task_li_feedback_li.append(multi_implementation_feedback)
|
||||
if evo_next_iter is None:
|
||||
break
|
||||
evo = evo_next_iter
|
||||
|
||||
# 2) merge the feedbacks along the sub_tasks to aggregate the multiple evaluation feedbacks
|
||||
merged_task_feedback = []
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import abstractmethod
|
||||
from typing import Callable, Generator
|
||||
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
@@ -26,7 +27,6 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
self.settings = settings
|
||||
self.improve_mode = improve_mode # improve mode means we only implement the task which has failed before. The main diff is the first loop will not implement all tasks.
|
||||
|
||||
@abstractmethod
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: Task,
|
||||
@@ -58,6 +58,15 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def implement_func_list(self) -> list[Callable]:
|
||||
"""
|
||||
One evolve solution will be divided into multiple implement functions.
|
||||
The functions will be called sequentially.
|
||||
|
||||
`implement_one_task` is the default implementation. Please refer to its signature for more details.
|
||||
"""
|
||||
return [self.implement_one_task]
|
||||
|
||||
@abstractmethod
|
||||
def assign_code_list_to_evo(self, code_list: list[dict], evo: EvolvingItem) -> None:
|
||||
"""
|
||||
@@ -66,19 +75,48 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
Due to the implement_one_task take `workspace` as input and output the `modification`.
|
||||
We should apply implementation to evo
|
||||
|
||||
Assumptions:
|
||||
- The modidication on evo should happen in-place!!
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def evolve(
|
||||
def assign_code_list_to_evo(self, code_list: list[dict | None], evo) -> None:
|
||||
"""Assign code modifications to evolving item.
|
||||
|
||||
For runner, coder already generated full training config, so typically no modifications.
|
||||
But this method is required by the abstract base class.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
|
||||
# If there are any modifications (usually empty for runner)
|
||||
if code_list[index]:
|
||||
# Handle change summary if present
|
||||
if self.KEY_CHANGE_SUMMARY in code_list[index]:
|
||||
evo.sub_workspace_list[index].change_summary = code_list[index].pop(self.KEY_CHANGE_SUMMARY)
|
||||
# Inject any modified files
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
|
||||
return evo
|
||||
|
||||
def evolve_iter(
|
||||
self,
|
||||
*,
|
||||
evo: EvolvingItem,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
evolving_trace: list[EvoStep] = [],
|
||||
**kwargs,
|
||||
) -> EvolvingItem:
|
||||
) -> Generator[EvolvingItem, EvolvingItem, None]:
|
||||
if queried_knowledge is None:
|
||||
raise ValueError(
|
||||
"MultiProcessEvolvingStrategy requires queried_knowledge for efficient implementation. Please set with_knowledge=True in CoSTEER constructor."
|
||||
)
|
||||
code_list = [None for _ in range(len(evo.sub_tasks))]
|
||||
|
||||
last_feedback = None
|
||||
@@ -111,24 +149,24 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
{}
|
||||
) # empty implementation for skipped task, but assign_code_list_to_evo will still assign it
|
||||
|
||||
result = multiprocessing_wrapper(
|
||||
[
|
||||
(
|
||||
self.implement_one_task,
|
||||
for implement_func in self.implement_func_list():
|
||||
result = multiprocessing_wrapper(
|
||||
[
|
||||
(
|
||||
evo.sub_tasks[target_index],
|
||||
queried_knowledge,
|
||||
evo.experiment_workspace,
|
||||
None if last_feedback is None else last_feedback[target_index],
|
||||
),
|
||||
)
|
||||
for target_index in to_be_finished_task_index
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
for index, target_index in enumerate(to_be_finished_task_index):
|
||||
code_list[target_index] = result[index]
|
||||
implement_func,
|
||||
(
|
||||
evo.sub_tasks[target_index],
|
||||
queried_knowledge,
|
||||
evo.experiment_workspace,
|
||||
None if last_feedback is None else last_feedback[target_index],
|
||||
),
|
||||
)
|
||||
for target_index in to_be_finished_task_index
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
for index, target_index in enumerate(to_be_finished_task_index):
|
||||
code_list[target_index] = result[index]
|
||||
|
||||
evo = self.assign_code_list_to_evo(code_list, evo)
|
||||
|
||||
return evo
|
||||
self.assign_code_list_to_evo(code_list, evo)
|
||||
yield evo
|
||||
|
||||
@@ -100,6 +100,38 @@ class CoSTEERRAGStrategy(RAGStrategy):
|
||||
|
||||
|
||||
class CoSTEERQueriedKnowledge(QueriedKnowledge):
|
||||
"""
|
||||
Data container for knowledge retrieved from the CoSTEER knowledge base during a query operation.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
success_task_to_knowledge_dict : dict, optional
|
||||
A mapping between task information strings and their corresponding `CoSTEERKnowledge` objects
|
||||
for tasks that were successfully completed.
|
||||
Type: dict[str, CoSTEERKnowledge]
|
||||
Example:
|
||||
{
|
||||
"task_info_1": CoSTEERKnowledge(target_task=Task(...),
|
||||
implementation=FBWorkspace(...),
|
||||
feedback=CoSTEERSingleFeedback(...)),
|
||||
"task_info_2": CoSTEERKnowledge(...)
|
||||
}
|
||||
failed_task_info_set : set, optional
|
||||
A set containing task information strings that were attempted but failed repeatedly beyond
|
||||
the allowed trial limit.
|
||||
Type: set[str]
|
||||
Example:
|
||||
{
|
||||
"failed_task_info_1",
|
||||
"failed_task_info_2"
|
||||
}
|
||||
|
||||
Returns
|
||||
-------
|
||||
None
|
||||
This class is a data holder, initialization does not return any value.
|
||||
"""
|
||||
|
||||
def __init__(self, success_task_to_knowledge_dict: dict = {}, failed_task_info_set: set = set()) -> None:
|
||||
self.success_task_to_knowledge_dict = success_task_to_knowledge_dict
|
||||
self.failed_task_info_set = failed_task_info_set
|
||||
@@ -134,6 +166,8 @@ class CoSTEERQueriedKnowledgeV1(CoSTEERQueriedKnowledge):
|
||||
|
||||
|
||||
class CoSTEERRAGStrategyV1(CoSTEERRAGStrategy):
|
||||
"""it is deprecated"""
|
||||
|
||||
def __init__(self, settings: CoSTEERSettings, *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
self.current_generated_trace_count = 0
|
||||
@@ -245,6 +279,62 @@ class CoSTEERRAGStrategyV1(CoSTEERRAGStrategy):
|
||||
|
||||
|
||||
class CoSTEERQueriedKnowledgeV2(CoSTEERQueriedKnowledgeV1):
|
||||
"""
|
||||
Aggregation subclass of `CoSTEERQueriedKnowledgeV1` that extends the queried knowledge to also
|
||||
include mappings between tasks and knowledge related to similar errors from successful executions.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
task_to_former_failed_traces : dict, optional
|
||||
Mapping from task information strings to a tuple containing:
|
||||
- A list of `CoSTEERKnowledge` objects representing the most recent failed attempts for that task.
|
||||
- An optional `CoSTEERKnowledge` object of the latest failed attempt after a successful execution,
|
||||
or `None` if not applicable.
|
||||
Type: dict[str, tuple[list[CoSTEERKnowledge], CoSTEERKnowledge | None]]
|
||||
Example:
|
||||
{
|
||||
"task_info_A": ([CoSTEERKnowledge(...), CoSTEERKnowledge(...)], None),
|
||||
"task_info_B": ([CoSTEERKnowledge(...), CoSTEERKnowledge(...)], CoSTEERKnowledge(...))
|
||||
}
|
||||
|
||||
task_to_similar_task_successful_knowledge : dict, optional
|
||||
Mapping from task information strings to a list of `CoSTEERKnowledge` objects representing
|
||||
knowledge from similar tasks that have been successfully completed.
|
||||
Type: dict[str, list[CoSTEERKnowledge]]
|
||||
Example:
|
||||
{
|
||||
"task_info_A": [CoSTEERKnowledge(...), CoSTEERKnowledge(...)],
|
||||
"task_info_C": []
|
||||
}
|
||||
|
||||
task_to_similar_error_successful_knowledge : dict, optional
|
||||
Mapping from task information strings to a list of tuples, each containing:
|
||||
- A string describing the error(s) encountered.
|
||||
- A tuple of two `CoSTEERKnowledge` objects:
|
||||
* The first corresponds to the trace where that error was encountered.
|
||||
* The second is related to a successful implementation that had the same error in a prior attempt.
|
||||
Type: dict[str, list[tuple[str, tuple[CoSTEERKnowledge, CoSTEERKnowledge]]]]
|
||||
Example:
|
||||
{
|
||||
"task_info_B": [
|
||||
(
|
||||
"1. ErrorType: ValueError; Error line: some_function_call()",
|
||||
(CoSTEERKnowledge(...), CoSTEERKnowledge(...))
|
||||
)
|
||||
]
|
||||
}
|
||||
|
||||
**kwargs : dict
|
||||
Additional keyword arguments passed to the parent constructor, such as:
|
||||
- success_task_to_knowledge_dict: dict[str, CoSTEERKnowledge]
|
||||
- failed_task_info_set: set[str]
|
||||
|
||||
Returns
|
||||
-------
|
||||
None
|
||||
This class is purely a data container and does not return a value upon initialization.
|
||||
"""
|
||||
|
||||
# Aggregation of knowledge
|
||||
def __init__(
|
||||
self,
|
||||
@@ -338,7 +428,7 @@ class CoSTEERRAGStrategyV2(CoSTEERRAGStrategy):
|
||||
self.current_generated_trace_count = len(evolving_trace)
|
||||
return None
|
||||
|
||||
def query(self, evo: EvolvableSubjects, evolving_trace: list[EvoStep]) -> CoSTEERQueriedKnowledge | None:
|
||||
def query(self, evo: EvolvableSubjects, evolving_trace: list[EvoStep]) -> CoSTEERQueriedKnowledge:
|
||||
conf_knowledge_sampler = self.settings.v2_knowledge_sampler
|
||||
queried_knowledge_v2 = CoSTEERQueriedKnowledgeV2(
|
||||
success_task_to_knowledge_dict=self.knowledgebase.success_task_to_knowledge_dict,
|
||||
|
||||
@@ -67,7 +67,7 @@ class EnsembleCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
|
||||
stdout += f"\nNOTE: the above scripts run with return code {ret_code}"
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
@@ -13,7 +10,6 @@ from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
|
||||
from rdagent.utils.fmt import shrink_text
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
@@ -69,7 +65,7 @@ class FeatureCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
workflow_code=implementation.all_codes,
|
||||
)
|
||||
user_prompt = T(".prompts:feature_eval.user").r(
|
||||
stdout=result.get_truncated_stdout(),
|
||||
stdout=result.stdout,
|
||||
workflow_stdout=workflow_stdout,
|
||||
)
|
||||
|
||||
|
||||
@@ -71,7 +71,7 @@ class ModelGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
) # only check the model changed this time
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
|
||||
if stdout is None:
|
||||
|
||||
@@ -163,7 +163,7 @@ class PipelineCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
result = implementation.run(
|
||||
env=env, entry=f"strace -e trace=file -f -o trace.log python -m coverage run main.py"
|
||||
)
|
||||
result_stdout = result.get_truncated_stdout()
|
||||
result_stdout = result.stdout
|
||||
|
||||
nb_conversion_ret_code = 0
|
||||
nb_conversion_check_text = ""
|
||||
@@ -261,7 +261,7 @@ class PipelineCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
implementation.inject_files(**{"test/submission_format_test.py": base_check_code})
|
||||
# stdout += "----Submission Check 1-----\n"
|
||||
submission_result = implementation.run(env=env, entry="python test/submission_format_test.py")
|
||||
submission_check_out = submission_result.get_truncated_stdout()
|
||||
submission_check_out = submission_result.stdout
|
||||
submission_ret_code = submission_result.exit_code
|
||||
stdout += "\n" + submission_check_out
|
||||
|
||||
|
||||
@@ -56,7 +56,7 @@ class DataLoaderCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
test_code = (DIRNAME / "eval_tests" / "data_loader_test.txt").read_text()
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
match = re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===(.*)", stdout, re.DOTALL)
|
||||
stdout_part_1, eda_output, stdout_part_2 = match.groups() if match else (stdout, None, "")
|
||||
|
||||
@@ -125,7 +125,7 @@ class WorkflowGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
implementation.inject_files(**{"test/submission_format_test.py": base_check_code})
|
||||
# stdout += "----Submission Check 1-----\n"
|
||||
submission_result = implementation.run(env=env, entry="python test/submission_format_test.py")
|
||||
submission_check_out = submission_result.get_truncated_stdout()
|
||||
submission_check_out = submission_result.stdout
|
||||
submission_ret_code = submission_result.exit_code
|
||||
stdout += "\n" + submission_check_out
|
||||
|
||||
|
||||
@@ -0,0 +1,391 @@
|
||||
"""
|
||||
LLM Fine-tuning CoSTEER Implementation
|
||||
|
||||
This module provides fine-tuning specific components for the CoSTEER framework,
|
||||
including evaluators and evolving strategies.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
import yaml
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEERMultiEvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DATA_SCRIPT_NAME,
|
||||
FT_PATHS,
|
||||
FT_TEST_PARAMS_FILE_NAME,
|
||||
FT_YAML_FILE_NAME,
|
||||
FTCoderCoSTEERSettings,
|
||||
)
|
||||
from rdagent.components.coder.finetune.eval import FTCoderEvaluator, FTDataEvaluator
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.scenarios.finetune.scen.llama_factory_manager import LLaMAFactory_manager
|
||||
from rdagent.scenarios.finetune.scen.utils import FinetuneDatasetDescriptor
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
|
||||
class LLMFinetuneEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
"""LLM Fine-tuning specific evolving strategy"""
|
||||
|
||||
def __init__(self, scen: Scenario, settings, *args, **kwargs):
|
||||
super().__init__(scen, settings)
|
||||
self.llama_factory_manager = LLaMAFactory_manager
|
||||
|
||||
def implement_func_list(self) -> list[Callable]:
|
||||
return [self.implement_data, self.implement_lf_config]
|
||||
|
||||
def implement_data(
|
||||
self,
|
||||
target_task: Task,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
prev_task_feedback: CoSTEERSingleFeedback | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Generate data processing script based on task.
|
||||
|
||||
This method generates a Python script that processes seed datasets
|
||||
and outputs a data.json file in Alpaca format.
|
||||
|
||||
Returns:
|
||||
dict with "process_data.py" key containing the script code,
|
||||
or empty dict if data already exists.
|
||||
"""
|
||||
# Check if proposal decided to skip data processing (reuse SOTA's data processing script)
|
||||
if getattr(target_task, "skip_data_processing", False):
|
||||
# Defensive check: ensure data script actually exists before skipping
|
||||
script_exists = False
|
||||
if workspace is not None:
|
||||
script_exists = FT_DATA_SCRIPT_NAME in workspace.file_dict
|
||||
|
||||
if script_exists:
|
||||
logger.info("Proposal decided to skip data processing, reusing SOTA's data script")
|
||||
return {}
|
||||
else:
|
||||
logger.warning(
|
||||
"skip_data_processing=True but process_data.py not found in workspace, "
|
||||
"this indicates SOTA injection failed - system design issue"
|
||||
)
|
||||
# Don't fallback silently, let it fail early to expose the issue
|
||||
|
||||
# check whether the current code passes evaluation
|
||||
if (
|
||||
prev_task_feedback is not None
|
||||
and "FTDataEvaluator" in prev_task_feedback.source_feedback
|
||||
and prev_task_feedback.source_feedback["FTDataEvaluator"]
|
||||
):
|
||||
logger.info("Previous data processing code passed evaluation, skipping regeneration")
|
||||
return {}
|
||||
|
||||
# build former failed trace
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[target_task.get_task_information()]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(FT_YAML_FILE_NAME)
|
||||
!= workspace.file_dict.get(FT_YAML_FILE_NAME)
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# Get dataset information for the task
|
||||
involving_datasets = getattr(target_task, "involving_datasets", [])
|
||||
dataset_info = self._get_dataset_info(involving_datasets, datasets_path=FT_PATHS.datasets)
|
||||
|
||||
# Generate data processing script using LLM
|
||||
system_prompt = T(".prompts:data_coder.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=target_task.get_task_information(),
|
||||
dataset_info=dataset_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
api_max_workers=FT_RD_SETTING.api_max_workers,
|
||||
datasets_path=FT_PATHS.datasets,
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
|
||||
user_prompt = T(".prompts:data_coder.user").r(
|
||||
datasets_path=FT_PATHS.datasets,
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
latest_code=workspace.file_dict.get(FT_DATA_SCRIPT_NAME, "") if workspace else "",
|
||||
latest_feedback=prev_task_feedback,
|
||||
involved_dataset_folder_desc={
|
||||
ds_name: FinetuneDatasetDescriptor().describe_dataset_folder(
|
||||
Path(FT_RD_SETTING.file_path) / "datasets" / ds_name, include_dataset_readme=True
|
||||
)
|
||||
for ds_name in involving_datasets
|
||||
},
|
||||
)
|
||||
|
||||
script_code = APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="python",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
logger.info(f"Generated data processing script ({len(script_code)} chars)")
|
||||
|
||||
return {FT_DATA_SCRIPT_NAME: script_code}
|
||||
|
||||
def _get_dataset_info(self, involving_datasets: list[str], datasets_path: str = None) -> str:
|
||||
"""Read dataset_info.json and return information for specified datasets.
|
||||
|
||||
Handles unified tasks structure:
|
||||
- readme: Dataset README content
|
||||
- file_tree: Directory structure
|
||||
- total_samples: Total sample count
|
||||
- tasks: Dict of task info (use "_root" for root-level data files)
|
||||
|
||||
Args:
|
||||
involving_datasets: List of dataset names to include
|
||||
datasets_path: Base path for datasets (e.g., "/assets/datasets/")
|
||||
"""
|
||||
datasets_dir = Path(FT_RD_SETTING.file_path) / "datasets"
|
||||
dataset_info_path = datasets_dir / "dataset_info.json"
|
||||
|
||||
# Use provided path or get from config
|
||||
if datasets_path is None:
|
||||
datasets_path = FT_PATHS.datasets
|
||||
|
||||
if not dataset_info_path.exists():
|
||||
logger.warning(f"dataset_info.json not found at {dataset_info_path}")
|
||||
return "No dataset information available."
|
||||
|
||||
try:
|
||||
with open(dataset_info_path, "r", encoding="utf-8") as f:
|
||||
all_dataset_info = json.load(f)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read dataset_info.json: {e}")
|
||||
return f"Error reading dataset info: {e}"
|
||||
|
||||
# Filter to only involved datasets, or use all if none specified
|
||||
if involving_datasets:
|
||||
filtered_info = {name: info for name, info in all_dataset_info.items() if name in involving_datasets}
|
||||
else:
|
||||
filtered_info = all_dataset_info
|
||||
|
||||
if not filtered_info:
|
||||
return "No matching datasets found in dataset_info.json."
|
||||
|
||||
# Format dataset info for the prompt
|
||||
info_parts = []
|
||||
for name, info in filtered_info.items():
|
||||
info_text = f"### Dataset: {name}\n"
|
||||
# IMPORTANT: Tell LLM the full path to dataset directory
|
||||
dataset_full_path = f"{datasets_path}{name}/"
|
||||
info_text += f"- **Dataset path**: `{dataset_full_path}` (each dataset has its own subdirectory)\n"
|
||||
info_text += f"- Total samples: {info.get('total_samples', 'N/A')}\n"
|
||||
info_text += f"- Size: {info.get('total_size_mb', 'N/A')} MB\n"
|
||||
|
||||
# File tree for understanding directory structure
|
||||
if info.get("file_tree"):
|
||||
file_tree = info["file_tree"]
|
||||
# Truncate if too long
|
||||
if len(file_tree) > 1000:
|
||||
file_tree = file_tree[:1000] + "\n..."
|
||||
info_text += f"\n**File Structure** (relative to `{dataset_full_path}`):\n```\n{file_tree}\n```\n"
|
||||
|
||||
# Handle unified tasks structure
|
||||
tasks = info.get("tasks", {})
|
||||
if tasks:
|
||||
info_text += "\n**Tasks:**\n"
|
||||
for task_name, task_info in tasks.items():
|
||||
# "_root" indicates data files are in root directory
|
||||
display_name = "(root)" if task_name == "_root" else task_name
|
||||
info_text += f"\n#### {display_name}\n"
|
||||
# Show full paths for data files
|
||||
files = task_info.get("files", [])
|
||||
info_text += f"- Files: {files}\n"
|
||||
if files:
|
||||
info_text += f" - Full path example: `{dataset_full_path}{files[0]}`\n"
|
||||
info_text += f"- Sample count: {task_info.get('sample_count', 'N/A')}\n"
|
||||
if task_info.get("column_stats"):
|
||||
# Show key token stats
|
||||
stats_summary = []
|
||||
for col, stats in task_info["column_stats"].items():
|
||||
if stats.get("p50_tokens", 0) > 0:
|
||||
stats_summary.append(f"{col}: p50={stats['p50_tokens']}, p99={stats['p99_tokens']}")
|
||||
if stats_summary:
|
||||
info_text += f"- Token stats: {'; '.join(stats_summary[:5])}\n"
|
||||
|
||||
# README excerpt
|
||||
if info.get("readme"):
|
||||
readme = info["readme"]
|
||||
if len(readme) > 500:
|
||||
readme = readme[:500] + "..."
|
||||
info_text += f"\n**README:**\n{readme}\n"
|
||||
|
||||
info_parts.append(info_text)
|
||||
|
||||
return "\n".join(info_parts)
|
||||
|
||||
def implement_lf_config(
|
||||
self,
|
||||
target_task: Task,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
prev_task_feedback: CoSTEERSingleFeedback | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Implement a single fine-tuning task by generating LlamaFactory config"""
|
||||
if prev_task_feedback is not None and prev_task_feedback.source_feedback.get("FTCoderEvaluator", False):
|
||||
logger.info("Previous training code passed evaluation, skipping regeneration")
|
||||
return {}
|
||||
|
||||
task_info = target_task.get_task_information()
|
||||
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[task_info] if queried_knowledge is not None else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(FT_YAML_FILE_NAME)
|
||||
!= workspace.file_dict.get(FT_YAML_FILE_NAME)
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# Get task parameters from the task object
|
||||
base_model = getattr(target_task, "base_model")
|
||||
|
||||
# Use LLM to generate LlamaFactory config YAML
|
||||
# Coder will decide method based on hypothesis and available parameters
|
||||
config_files = self._generate_llamafactory_config_with_llm(
|
||||
base_model=base_model,
|
||||
task_info=task_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge,
|
||||
prev_feedback=prev_task_feedback,
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
# Return generated config files directly - validation happens in evaluator
|
||||
return config_files
|
||||
|
||||
def _generate_llamafactory_config_with_llm(
|
||||
self,
|
||||
base_model: str,
|
||||
task_info: str = "",
|
||||
queried_former_failed_knowledge: tuple = None,
|
||||
prev_feedback=None,
|
||||
workspace=None,
|
||||
) -> dict[str, str]:
|
||||
"""Generate LlamaFactory configuration YAML using LLM"""
|
||||
|
||||
# Query LLaMA Factory parameters: shared params once + method-specific params
|
||||
available_methods = self.llama_factory_manager.methods
|
||||
shared_params = self.llama_factory_manager.format_shared_params()
|
||||
|
||||
# Format method-specific parameters only (no duplication of shared params)
|
||||
methods_specific_params = {}
|
||||
for method in available_methods:
|
||||
methods_specific_params[method] = self.llama_factory_manager.format_method_specific_params(method)
|
||||
|
||||
# Use environment-aware paths (Docker vs Conda)
|
||||
# Note: datasets_path in finetune_coder uses workspace path where processed
|
||||
# data.json and dataset_info.json are located (generated by FTDataEvaluator)
|
||||
|
||||
# Generate prompts using templates with all required parameters
|
||||
system_prompt = T(".prompts:finetune_coder.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=task_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
available_methods=", ".join(available_methods),
|
||||
shared_params=shared_params,
|
||||
methods_specific_params=methods_specific_params,
|
||||
)
|
||||
|
||||
# Read data_stats.json from workspace (injected by FTDataEvaluator)
|
||||
data_stats = workspace.file_dict.get("data_stats.json", "")
|
||||
|
||||
user_prompt = T(".prompts:finetune_coder.user").r(
|
||||
latest_code=workspace.file_dict.get(FT_YAML_FILE_NAME, ""),
|
||||
latest_feedback=prev_feedback,
|
||||
base_model=base_model,
|
||||
models_path=FT_PATHS.models,
|
||||
datasets_path=FT_PATHS.workspace, # Training config uses workspace path for processed data
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
deepspeed_path=FT_PATHS.deepspeed,
|
||||
data_stats=data_stats,
|
||||
has_think_token=self.scen.model_info.get("has_think_token", False),
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
|
||||
# Call LLM to generate config (multi-turn)
|
||||
session = APIBackend().build_chat_session(session_system_prompt=system_prompt)
|
||||
|
||||
# Turn 1: Generate main training config
|
||||
train_config_yaml = session.build_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="yaml",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
|
||||
# Validate main config YAML syntax
|
||||
yaml.safe_load(train_config_yaml)
|
||||
logger.info("Extracted main YAML config successfully")
|
||||
|
||||
# Turn 2: Generate test parameters (test_params.yaml)
|
||||
test_params_prompt = T(".prompts:finetune_coder.user_test_params").r(workspace_path=FT_PATHS.workspace)
|
||||
test_params_yaml = session.build_chat_completion(
|
||||
user_prompt=test_params_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="yaml",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
|
||||
# Validate test params YAML syntax
|
||||
yaml.safe_load(test_params_yaml)
|
||||
logger.info("Extracted test params YAML successfully")
|
||||
|
||||
return {FT_YAML_FILE_NAME: train_config_yaml, FT_TEST_PARAMS_FILE_NAME: test_params_yaml}
|
||||
|
||||
|
||||
class LLMFinetuneCoSTEER(CoSTEER):
|
||||
"""LLM Fine-tuning CoSTEER implementation"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
settings = FTCoderCoSTEERSettings()
|
||||
eva = CoSTEERMultiEvaluator([FTDataEvaluator(scen=scen), FTCoderEvaluator(scen=scen)], scen=scen)
|
||||
es = LLMFinetuneEvolvingStrategy(scen=scen, settings=settings)
|
||||
|
||||
super().__init__(
|
||||
*args,
|
||||
settings=settings,
|
||||
eva=eva,
|
||||
es=es,
|
||||
evolving_version=2,
|
||||
scen=scen,
|
||||
max_loop=FT_RD_SETTING.coder_max_loop if hasattr(FT_RD_SETTING, "coder_max_loop") else 5,
|
||||
stop_eval_chain_on_fail=True, # finetune involve partial implementation.
|
||||
**kwargs,
|
||||
)
|
||||
@@ -0,0 +1,387 @@
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.scen.utils import _compute_column_stats
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import (
|
||||
BenchmarkCondaConf,
|
||||
BenchmarkCondaEnv,
|
||||
BenchmarkDockerConf,
|
||||
BenchmarkDockerEnv,
|
||||
DockerEnv,
|
||||
Env,
|
||||
FTCondaConf,
|
||||
FTCondaEnv,
|
||||
FTDockerEnv,
|
||||
)
|
||||
|
||||
|
||||
def is_docker_env(env: Env) -> bool:
|
||||
"""Check if the environment is Docker-based."""
|
||||
return isinstance(env, DockerEnv)
|
||||
|
||||
|
||||
def get_workspace_prefix(env: Env) -> str:
|
||||
"""Return workspace path prefix based on env type.
|
||||
|
||||
Docker uses /workspace as mount point, conda uses current directory.
|
||||
"""
|
||||
return "/workspace" if is_docker_env(env) else "."
|
||||
|
||||
|
||||
FT_YAML_FILE_NAME = "train.yaml"
|
||||
FT_DATA_PROC_FILE_NAME = "data_process.py"
|
||||
FT_DEBUG_YAML_FILE_NAME = "debug_train.yaml"
|
||||
FT_TEST_PARAMS_FILE_NAME = "test_params.yaml"
|
||||
FT_DATA_FILE_NAME = "data.json"
|
||||
FT_DATA_SCRIPT_NAME = "process_data.py"
|
||||
|
||||
# ENV Info: the path of the model and dataset in the container/environment
|
||||
FT_MODEL_PATH = "/assets/models"
|
||||
FT_DATASET_PATH = "/assets/datasets"
|
||||
|
||||
|
||||
def get_data_processing_cache_key(local_path: str | Path) -> list[list[str]]:
|
||||
"""Generate cache key based only on data processing script and dataset info.
|
||||
|
||||
This ensures that data processing results are reused as long as the script
|
||||
and dataset configuration remain unchanged, even if other files in the
|
||||
workspace (like training config) have been modified.
|
||||
"""
|
||||
content = []
|
||||
local_path = Path(local_path)
|
||||
# We only care about the script that generates data and the dataset configuration
|
||||
for filename in [FT_DATA_SCRIPT_NAME, "dataset_info.json"]:
|
||||
file_path = local_path / filename
|
||||
if file_path.exists():
|
||||
content.append([filename, file_path.read_text()])
|
||||
content.sort(key=lambda x: x[0])
|
||||
return content
|
||||
|
||||
|
||||
class FTPathConfig:
|
||||
"""Centralized path configuration for FT scenario.
|
||||
|
||||
Provides environment-aware paths for Docker vs Conda modes.
|
||||
Uses lazy evaluation (properties) to avoid import-time errors.
|
||||
|
||||
Usage:
|
||||
from rdagent.components.coder.finetune.conf import FT_PATHS
|
||||
|
||||
models_path = FT_PATHS.models # e.g., "/assets/models/" or "/path/to/finetune/models/"
|
||||
datasets_path = FT_PATHS.datasets # e.g., "/assets/datasets/" or "/path/to/finetune/datasets/"
|
||||
workspace_path = FT_PATHS.workspace # e.g., "/workspace/" or "./"
|
||||
"""
|
||||
|
||||
@property
|
||||
def is_docker(self) -> bool:
|
||||
"""Check if current environment is Docker-based."""
|
||||
# FIXME: the env should work in same way for docker and conda env.
|
||||
# We should not expose the env type everywhere.
|
||||
return FTCoderCoSTEERSettings().env_type == "docker"
|
||||
|
||||
@property
|
||||
def models(self) -> str:
|
||||
"""Model directory path (with trailing slash)."""
|
||||
if self.is_docker:
|
||||
return FT_MODEL_PATH + "/"
|
||||
return str(FT_RD_SETTING.file_path / "models") + "/"
|
||||
|
||||
@property
|
||||
def datasets(self) -> str:
|
||||
"""Dataset directory path for raw datasets (with trailing slash)."""
|
||||
if self.is_docker:
|
||||
return FT_DATASET_PATH + "/"
|
||||
return str(FT_RD_SETTING.file_path / "datasets") + "/"
|
||||
|
||||
@property
|
||||
def workspace(self) -> str:
|
||||
"""Workspace path prefix for prompts (with trailing slash)."""
|
||||
return "/workspace/" if self.is_docker else "./"
|
||||
|
||||
@property
|
||||
def deepspeed(self) -> str:
|
||||
"""DeepSpeed config directory."""
|
||||
if self.is_docker:
|
||||
return "/app/examples/deepspeed/"
|
||||
# Conda mode: use bundled deepspeed configs in project
|
||||
# Path: conf.py -> finetune -> coder -> components -> rdagent -> scenarios/finetune/env/conda/deepspeed
|
||||
rdagent_root = Path(__file__).parent.parent.parent.parent
|
||||
deepspeed_path = rdagent_root / "scenarios" / "finetune" / "env" / "conda" / "deepspeed"
|
||||
return str(deepspeed_path) + "/" if deepspeed_path.exists() else ""
|
||||
|
||||
|
||||
# Singleton instance for path configuration
|
||||
FT_PATHS = FTPathConfig()
|
||||
|
||||
|
||||
class FTCoderCoSTEERSettings(CoSTEERSettings):
|
||||
"""LLM Fine-tuning CoSTEER settings"""
|
||||
|
||||
class Config:
|
||||
env_prefix = "FT_Coder_CoSTEER_"
|
||||
|
||||
max_seconds_multiplier: int = 8
|
||||
"""LLM training takes longer, use higher multiplier"""
|
||||
|
||||
env_type: str = "docker"
|
||||
"""Environment type for LLM fine-tuning (docker/conda)"""
|
||||
|
||||
extra_eval: list[str] = []
|
||||
"""Extra evaluators"""
|
||||
|
||||
|
||||
def _get_standard_ft_volumes() -> dict:
|
||||
"""Get standard mount volume configuration for LLM finetune environments.
|
||||
|
||||
Creates standard directory mappings:
|
||||
- models -> /assets/models (ro)
|
||||
- datasets -> /assets/datasets (ro)
|
||||
|
||||
Returns:
|
||||
Dictionary of local_path -> docker_mount_config mappings
|
||||
"""
|
||||
base_path = Path(FT_RD_SETTING.file_path)
|
||||
volumes = {}
|
||||
|
||||
# Read-only mounts for data and models
|
||||
readonly_mounts = [
|
||||
("models", FT_MODEL_PATH),
|
||||
("datasets", FT_DATASET_PATH),
|
||||
]
|
||||
|
||||
for local_dir, docker_path in readonly_mounts:
|
||||
local_path = base_path / local_dir
|
||||
volumes[str(local_path)] = {"bind": docker_path, "mode": "ro"}
|
||||
|
||||
return volumes
|
||||
|
||||
|
||||
def get_ft_env(
|
||||
extra_volumes: dict = {},
|
||||
operation: str = "full_training",
|
||||
enable_cache: bool | None = None,
|
||||
) -> Env:
|
||||
"""LLM finetune dedicated environment construction function.
|
||||
|
||||
Automatically includes standard finetune volume mounts:
|
||||
- models -> /assets/models (ro)
|
||||
- datasets -> /assets/datasets (ro)
|
||||
- output -> /workspace/output (rw, auto-created)
|
||||
|
||||
Note: .llama_factory_info is no longer automatically mounted.
|
||||
Pass llama_factory_info volume via extra_volumes when needed.
|
||||
|
||||
Args:
|
||||
extra_volumes: Additional volume mounts beyond standard ones
|
||||
operation: Operation type for timeout selection.
|
||||
- "data_processing": Data processing (data_processing_timeout)
|
||||
- "micro_batch": Micro-batch test (micro_batch_timeout)
|
||||
- "full_training": Full training (full_timeout)
|
||||
enable_cache: Whether to enable caching (None means use config value)
|
||||
|
||||
Returns:
|
||||
Configured environment ready for use
|
||||
"""
|
||||
|
||||
conf = FTCoderCoSTEERSettings()
|
||||
|
||||
# Select timeout based on operation type
|
||||
timeout_map = {
|
||||
"data_processing": FT_RD_SETTING.data_processing_timeout,
|
||||
"debug_data_processing": FT_RD_SETTING.debug_data_processing_timeout,
|
||||
"micro_batch": FT_RD_SETTING.micro_batch_timeout,
|
||||
"full_training": FT_RD_SETTING.full_timeout,
|
||||
}
|
||||
running_timeout_period = timeout_map.get(operation, FT_RD_SETTING.full_timeout)
|
||||
|
||||
# Use config value if enable_cache is not explicitly provided
|
||||
if enable_cache is None:
|
||||
enable_cache = FT_RD_SETTING.docker_enable_cache
|
||||
|
||||
# Use dedicated LLM docker or conda env based on config
|
||||
if conf.env_type == "docker":
|
||||
env = FTDockerEnv()
|
||||
# Docker mode: setup volume mounts for models/datasets
|
||||
standard_volumes = _get_standard_ft_volumes()
|
||||
combined_volumes = standard_volumes.copy()
|
||||
combined_volumes.update(extra_volumes)
|
||||
env.conf.extra_volumes = combined_volumes
|
||||
elif conf.env_type == "conda":
|
||||
env = FTCondaEnv(conf=FTCondaConf()) # Auto-installs dependencies if env doesn't exist
|
||||
# Conda mode: no volume mounts needed, use local paths directly
|
||||
# extra_volumes are ignored in conda mode
|
||||
else:
|
||||
raise ValueError(f"Unknown env type: {conf.env_type}")
|
||||
|
||||
env.conf.running_timeout_period = running_timeout_period
|
||||
env.conf.enable_cache = enable_cache
|
||||
env.prepare()
|
||||
return env
|
||||
|
||||
|
||||
def get_data_processing_env(
|
||||
enable_cache: bool | None = None,
|
||||
is_debug: bool = False,
|
||||
) -> tuple[Env, dict]:
|
||||
"""Get environment for data processing scripts with LLM API access.
|
||||
|
||||
This environment is configured for running data processing scripts that may
|
||||
need to call LLM APIs. It includes:
|
||||
- Standard finetune volume mounts (datasets, models)
|
||||
- LLM API environment variables (OPENAI_API_KEY, OPENAI_BASE_URL, etc.)
|
||||
|
||||
Args:
|
||||
enable_cache: Whether to enable Docker caching
|
||||
is_debug: Whether running in debug mode (shorter timeout, default 20 min vs 1 hour)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_vars) where env_vars contains LLM API keys
|
||||
to be passed to env.run() as the env parameter
|
||||
"""
|
||||
env = get_ft_env(
|
||||
operation="debug_data_processing" if is_debug else "data_processing",
|
||||
enable_cache=enable_cache,
|
||||
)
|
||||
|
||||
# Collect LLM API environment variables to pass to env.run()
|
||||
llm_env_vars = {"PYTHONPATH": "./"} # Base env var
|
||||
|
||||
# Pass OPENAI_API_KEY directly
|
||||
if api_key := os.getenv("OPENAI_API_KEY"):
|
||||
llm_env_vars["OPENAI_API_KEY"] = api_key
|
||||
|
||||
# Read OPENAI_API_BASE from env, but pass as OPENAI_BASE_URL (OpenAI SDK expects this name)
|
||||
if api_base := os.getenv("OPENAI_API_BASE"):
|
||||
llm_env_vars["OPENAI_BASE_URL"] = api_base
|
||||
|
||||
# Pass model pools as JSON environment variables for load balancing
|
||||
llm_env_vars["STRONG_MODEL_POOL"] = json.dumps(FT_RD_SETTING.strong_models)
|
||||
llm_env_vars["WEAK_MODEL_POOL"] = json.dumps(FT_RD_SETTING.weak_models)
|
||||
|
||||
return env, llm_env_vars
|
||||
|
||||
|
||||
def clear_workspace(workspace: FBWorkspace, env: Env) -> None:
|
||||
"""
|
||||
Clean the files in LLM finetune workspace.
|
||||
Only keeps the files that are injected by the coder (in workspace.file_dict) and `logs`.
|
||||
|
||||
Args:
|
||||
workspace: The workspace object containing the file dictionary.
|
||||
env: The environment to execute the clean command in.
|
||||
"""
|
||||
target_path = workspace.workspace_path
|
||||
if not target_path.exists():
|
||||
return
|
||||
|
||||
# The cache_path is created when mounting, so the permissions changes does not work.
|
||||
keep_items = {"logs", T("scenarios.data_science.share:scen.cache_path").r()}
|
||||
|
||||
for file_path in workspace.file_dict.keys():
|
||||
top_level = Path(file_path).parts[0]
|
||||
keep_items.add(top_level)
|
||||
|
||||
remove_items = []
|
||||
for item in target_path.iterdir():
|
||||
if item.name in keep_items:
|
||||
continue
|
||||
remove_items.append(item.name)
|
||||
|
||||
if remove_items:
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
# Construct rm command with all items to remove
|
||||
# Items are relative to workspace root inside the env
|
||||
items_str = " ".join([f"'{ws_prefix}/{item}'" for item in remove_items])
|
||||
cmd = f"rm -rf {items_str}"
|
||||
workspace.execute(env=env, entry=cmd)
|
||||
|
||||
|
||||
def get_benchmark_env(
|
||||
extra_volumes: dict = {},
|
||||
timeout: int | None = None,
|
||||
) -> Env:
|
||||
"""OpenCompass benchmark environment construction function.
|
||||
|
||||
Supports both Docker and conda environments based on FT_Coder_CoSTEER_env_type.
|
||||
|
||||
Args:
|
||||
extra_volumes: Additional volume mounts (only used in Docker mode)
|
||||
timeout: Running timeout in seconds (None uses config default)
|
||||
|
||||
Returns:
|
||||
Configured environment ready for benchmark evaluation
|
||||
"""
|
||||
conf = FTCoderCoSTEERSettings()
|
||||
|
||||
# Use benchmark-specific timeout or config default
|
||||
if timeout is None:
|
||||
# 0 means no timeout, use 7 days as practical "infinite"
|
||||
timeout = FT_RD_SETTING.benchmark_timeout if FT_RD_SETTING.benchmark_timeout > 0 else 86400 * 7
|
||||
|
||||
benchmark_volumes = {}
|
||||
# Setup finetune share folder mount for models
|
||||
(FT_RD_SETTING.file_path / "benchmarks").mkdir(parents=True, exist_ok=True)
|
||||
# NOTE: we choose a folder in the workspace as the mount point due to we may run multiple instances in same
|
||||
# host machine. If conda env is used, the mount point will conflict with each other.
|
||||
benchmark_volumes[str((FT_RD_SETTING.file_path / "benchmarks").resolve())] = {
|
||||
"bind": "./benchmarks",
|
||||
"mode": "rw",
|
||||
}
|
||||
env_dict = {"COMPASS_DATA_CACHE": "./benchmarks/opencompass_data"}
|
||||
# Mount models directory for LoRA base model access (vLLM needs base model config)
|
||||
models_path = FT_RD_SETTING.file_path / "models"
|
||||
if models_path.exists():
|
||||
benchmark_volumes[str(models_path.resolve())] = {"bind": FT_MODEL_PATH, "mode": "ro"}
|
||||
benchmark_volumes.update(extra_volumes)
|
||||
|
||||
if conf.env_type == "docker":
|
||||
docker_conf = BenchmarkDockerConf()
|
||||
docker_conf.running_timeout_period = timeout
|
||||
docker_conf.extra_volumes = benchmark_volumes
|
||||
docker_conf.env_dict = env_dict
|
||||
env = BenchmarkDockerEnv(conf=docker_conf)
|
||||
elif conf.env_type == "conda":
|
||||
# NOTE:
|
||||
# We assume user has the permissions to create the softlink in the target directory.
|
||||
# If we have requirements in the future, we suggest make the target directory configurable in BenchmarkCondaConf.
|
||||
conda_conf = BenchmarkCondaConf()
|
||||
conda_conf.running_timeout_period = timeout
|
||||
conda_conf.extra_volumes = benchmark_volumes
|
||||
conda_conf.env_dict = env_dict
|
||||
env = BenchmarkCondaEnv(conf=conda_conf) # Auto-installs dependencies if env doesn't exist
|
||||
else:
|
||||
raise ValueError(f"Unknown env type: {conf.env_type}")
|
||||
|
||||
env.prepare()
|
||||
return env
|
||||
|
||||
|
||||
def inject_data_stats(implementation: FBWorkspace, data: list, stdout: str) -> None:
|
||||
"""Compute token statistics and inject data_stats.json.
|
||||
|
||||
Used by both FTDataEvaluator (coding stage) and FTRunnerEvaluator (running stage).
|
||||
|
||||
Args:
|
||||
implementation: The workspace to inject data_stats.json into
|
||||
data: The data list from data.json
|
||||
stdout: The stdout from process_data.py execution
|
||||
"""
|
||||
token_stats = _compute_column_stats(data)
|
||||
|
||||
data_stats = {
|
||||
"total_samples": len(data),
|
||||
"token_stats": token_stats,
|
||||
"stdout_summary": stdout,
|
||||
}
|
||||
|
||||
implementation.inject_files(**{"data_stats.json": json.dumps(data_stats, indent=2)})
|
||||
logger.info(f"Injected data_stats.json with {len(data)} samples")
|
||||
@@ -0,0 +1,399 @@
|
||||
"""
|
||||
LLM Fine-tuning Evaluation Components
|
||||
|
||||
Provides simplified evaluation: parameter filtering + micro-batch testing.
|
||||
No redundant LLM feedback generation - test results speak for themselves.
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DATA_FILE_NAME,
|
||||
FT_DATA_SCRIPT_NAME,
|
||||
FT_YAML_FILE_NAME,
|
||||
clear_workspace,
|
||||
get_data_processing_cache_key,
|
||||
get_data_processing_env,
|
||||
get_ft_env,
|
||||
get_workspace_prefix,
|
||||
inject_data_stats,
|
||||
)
|
||||
from rdagent.components.coder.finetune.unified_validator import (
|
||||
SYSTEM_MANAGED_PARAMS,
|
||||
LLMConfigValidator,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
|
||||
class FTDataEvaluator(CoSTEEREvaluator):
|
||||
"""Evaluator for data processing results.
|
||||
|
||||
This evaluator:
|
||||
1. Executes the process_data.py script in Docker
|
||||
2. Validates the output data.json file
|
||||
3. Generates dataset_info.json for LlamaFactory
|
||||
"""
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: Optional[QueriedKnowledge] = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Evaluate data processing implementation with LLM feedback."""
|
||||
|
||||
script_code = implementation.file_dict.get(FT_DATA_SCRIPT_NAME, "")
|
||||
data_json_path = implementation.workspace_path / FT_DATA_FILE_NAME
|
||||
execution_output = ""
|
||||
exit_code = 0
|
||||
data = None
|
||||
error_msg = None
|
||||
|
||||
# Step 1: Check script exists
|
||||
if not script_code:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution=f"No {FT_DATA_SCRIPT_NAME} found",
|
||||
return_checking="Data processing script missing",
|
||||
code="Please generate a data processing script first.",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTDataEvaluator")
|
||||
return feedback
|
||||
|
||||
# NOTE: we depends cache for speeding up the process of data generation.
|
||||
# So we clear the workspace every time.
|
||||
|
||||
# Step 3: Execute script in DEBUG mode (generates ~10 samples for fast validation)
|
||||
env, env_vars = get_data_processing_env(is_debug=True)
|
||||
|
||||
# Clear workspace (except logs and file_dict items) before data processing
|
||||
clear_workspace(implementation, env=env)
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
|
||||
# Use FTWorkspace.run() for unified Docker logging
|
||||
# --debug flag tells the script to generate only ~10 samples
|
||||
result = implementation.run(
|
||||
env=env,
|
||||
entry=f"python {ws_prefix}/{FT_DATA_SCRIPT_NAME} --debug",
|
||||
env_vars=env_vars,
|
||||
cache_key_extra_func=get_data_processing_cache_key,
|
||||
cache_files_to_extract=[FT_DATA_FILE_NAME],
|
||||
)
|
||||
execution_output = result.stdout if hasattr(result, "stdout") else str(result)
|
||||
exit_code = result.exit_code if hasattr(result, "exit_code") else -1
|
||||
|
||||
# Step 4: Validate output
|
||||
if not data_json_path.exists():
|
||||
error_msg = f"{FT_DATA_FILE_NAME} not generated"
|
||||
else:
|
||||
validation_result = self._validate_data_json(data_json_path)
|
||||
if not validation_result["valid"]:
|
||||
error_msg = validation_result["error"]
|
||||
else:
|
||||
self._update_dataset_info(implementation, validation_result["sample_count"])
|
||||
|
||||
# Step 5: Load data if valid
|
||||
if error_msg is None and data_json_path.exists():
|
||||
with open(data_json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Step 5.5: Compute token stats and inject data_stats for yaml coder
|
||||
if data is not None and error_msg is None:
|
||||
inject_data_stats(implementation, data, execution_output)
|
||||
|
||||
# Step 6: Generate LLM feedback
|
||||
# Truncate stdout from end for LLM (summary at the end is more useful)
|
||||
stdout_summary = execution_output[-1500:] if execution_output else ""
|
||||
return self._generate_llm_feedback(
|
||||
target_task=target_task,
|
||||
script_code=script_code if error_msg else "", # Only show script on error
|
||||
stdout=stdout_summary, # Always show summary (truncated from end)
|
||||
exit_code=exit_code,
|
||||
data=data,
|
||||
error_msg=error_msg,
|
||||
queried_knowledge=queried_knowledge,
|
||||
raw_stdout=execution_output, # Full log for UI
|
||||
)
|
||||
|
||||
def _generate_llm_feedback(
|
||||
self,
|
||||
target_task: Task,
|
||||
script_code: str,
|
||||
stdout: str,
|
||||
exit_code: int,
|
||||
data: Optional[list],
|
||||
error_msg: Optional[str],
|
||||
queried_knowledge: Optional[QueriedKnowledge],
|
||||
raw_stdout: str = "",
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Generate LLM-based feedback for data processing evaluation."""
|
||||
|
||||
# Prepare data statistics and samples
|
||||
if data:
|
||||
stats = self._analyze_data_quality(data)
|
||||
data_stats = json.dumps(stats, indent=2)
|
||||
sampled_data = self._sample_data(data)
|
||||
data_samples = json.dumps(sampled_data, indent=2, ensure_ascii=False)
|
||||
sample_count = len(sampled_data)
|
||||
total_samples = len(data)
|
||||
else:
|
||||
data_stats = json.dumps({"error": error_msg or "No data generated"})
|
||||
data_samples = "[]"
|
||||
sample_count = 0
|
||||
total_samples = 0
|
||||
|
||||
# Extract similar successful knowledge
|
||||
queried_similar_successful_knowledge = []
|
||||
if queried_knowledge is not None:
|
||||
task_info = target_task.get_task_information()
|
||||
queried_similar_successful_knowledge = queried_knowledge.task_to_similar_task_successful_knowledge.get(
|
||||
task_info, []
|
||||
)
|
||||
|
||||
# Build prompts
|
||||
system_prompt = T(".prompts:data_eval.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
upper_data_size_limit=FT_RD_SETTING.upper_data_size_limit,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
user_prompt = T(".prompts:data_eval.user").r(
|
||||
task_desc=target_task.get_task_information(),
|
||||
script_code=script_code,
|
||||
exit_code=exit_code,
|
||||
stdout=stdout[:3000] if stdout else "", # Empty string triggers {% if stdout %} = false
|
||||
data_stats=data_stats,
|
||||
sample_count=sample_count,
|
||||
total_samples=total_samples,
|
||||
data_samples=data_samples,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Generating LLM feedback for data evaluation (samples: {total_samples}, has_error: {bool(error_msg)})"
|
||||
)
|
||||
|
||||
feedback = build_cls_from_json_with_retry(
|
||||
CoSTEERSingleFeedback,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
init_kwargs_update_func=CoSTEERSingleFeedback.val_and_update_init_dict,
|
||||
)
|
||||
|
||||
# NOTE: 0 exit code is a hard criteria for success
|
||||
if exit_code != 0:
|
||||
feedback.final_decision = False
|
||||
|
||||
feedback.raw_execution = raw_stdout
|
||||
feedback.source_feedback[self.__class__.__name__] = feedback.final_decision
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTDataEvaluator")
|
||||
return feedback
|
||||
|
||||
def _validate_data_json(self, data_json_path: Path) -> dict:
|
||||
"""Validate data.json file format and content."""
|
||||
try:
|
||||
with open(data_json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Must be a non-empty list
|
||||
if not isinstance(data, list):
|
||||
return {"valid": False, "error": "data.json must be a JSON array", "sample_count": 0}
|
||||
|
||||
if len(data) == 0:
|
||||
return {"valid": False, "error": "data.json is empty", "sample_count": 0}
|
||||
|
||||
# Check required fields in samples
|
||||
required_fields = ["instruction", "output"]
|
||||
for i, sample in enumerate(data[:10]): # Check first 10 samples
|
||||
if not isinstance(sample, dict):
|
||||
return {"valid": False, "error": f"Sample {i} is not a dict", "sample_count": 0}
|
||||
|
||||
missing = [f for f in required_fields if f not in sample]
|
||||
if missing:
|
||||
return {"valid": False, "error": f"Sample {i} missing fields: {missing}", "sample_count": 0}
|
||||
|
||||
# Check for empty required fields
|
||||
for field in required_fields:
|
||||
if not sample.get(field):
|
||||
return {
|
||||
"valid": False,
|
||||
"error": f"Sample {i} has empty '{field}' field",
|
||||
"sample_count": 0,
|
||||
}
|
||||
|
||||
return {"valid": True, "error": None, "sample_count": len(data)}
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
return {"valid": False, "error": f"Invalid JSON: {e}", "sample_count": 0}
|
||||
except Exception as e:
|
||||
return {"valid": False, "error": f"Error reading file: {e}", "sample_count": 0}
|
||||
|
||||
def _update_dataset_info(self, implementation: FBWorkspace, sample_count: int):
|
||||
"""Generate dataset_info.json for LlamaFactory to use the processed data.
|
||||
|
||||
Note: LlamaFactory's columns mapping uses internal names (prompt, query, response)
|
||||
that map to the actual column names in the data file (instruction, input, output).
|
||||
See: https://github.com/hiyouga/LLaMA-Factory/blob/main/src/llamafactory/data/parser.py
|
||||
"""
|
||||
dataset_info = {
|
||||
"processed_data": {
|
||||
"file_name": FT_DATA_FILE_NAME,
|
||||
"formatting": "alpaca",
|
||||
"columns": {
|
||||
"prompt": "instruction",
|
||||
"query": "input",
|
||||
"response": "output",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
implementation.inject_files(**{"dataset_info.json": json.dumps(dataset_info, indent=2)})
|
||||
logger.info(f"Updated dataset_info.json with processed_data ({sample_count} samples)")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to update dataset_info.json: {e}")
|
||||
|
||||
def _sample_data(self, data: list, n: int = 5) -> list:
|
||||
"""Random sampling for LLM evaluation."""
|
||||
if len(data) <= n:
|
||||
return data
|
||||
return random.sample(data, n)
|
||||
|
||||
def _analyze_data_quality(self, data: list) -> dict:
|
||||
"""Analyze data quality statistics for all fields."""
|
||||
if not data:
|
||||
return {"total_samples": 0, "error": "Empty data"}
|
||||
|
||||
# Analyze length stats for all standard fields
|
||||
fields = ["instruction", "input", "output"]
|
||||
stats = {"total_samples": len(data)}
|
||||
|
||||
for field in fields:
|
||||
lens = [len(str(d.get(field, ""))) for d in data]
|
||||
empty_count = sum(1 for d in data if not d.get(field))
|
||||
stats[f"{field}_len"] = {
|
||||
"min": min(lens),
|
||||
"max": max(lens),
|
||||
"avg": round(sum(lens) / len(lens), 1),
|
||||
}
|
||||
stats[f"{field}_empty_ratio"] = round(empty_count / len(data) * 100, 1)
|
||||
|
||||
# Detect duplicates by full record (instruction + input + output)
|
||||
record_set = set(
|
||||
(str(d.get("instruction", "")), str(d.get("input", "")), str(d.get("output", ""))) for d in data
|
||||
)
|
||||
duplicate_count = len(data) - len(record_set)
|
||||
stats["duplicate_count"] = duplicate_count
|
||||
stats["duplicate_ratio"] = round(duplicate_count / len(data) * 100, 1)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
class FTCoderEvaluator(CoSTEEREvaluator):
|
||||
"""Evaluator for LLM fine-tuning implementations with simplified validation"""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Evaluate LLM fine-tuning implementation with two-step validation"""
|
||||
|
||||
task_info = target_task.get_task_information()
|
||||
|
||||
# Check task history
|
||||
if queried_knowledge is not None:
|
||||
if task_info in queried_knowledge.success_task_to_knowledge_dict:
|
||||
return queried_knowledge.success_task_to_knowledge_dict[task_info].feedback
|
||||
elif task_info in queried_knowledge.failed_task_info_set:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution="Task failed too many times, skipping.",
|
||||
return_checking="Task failed too many times, skipping.",
|
||||
code="Task failed too many times, skipping.",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
|
||||
env = get_ft_env(operation="micro_batch")
|
||||
config_yaml = implementation.file_dict.get(FT_YAML_FILE_NAME, "")
|
||||
if not config_yaml:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution=f"No {FT_YAML_FILE_NAME} found",
|
||||
return_checking="Configuration file missing",
|
||||
code="No valid configuration file",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
|
||||
# Two-step validation: parameter filtering + micro-batch test
|
||||
validation_result = LLMConfigValidator().validate_and_test(
|
||||
config_yaml=config_yaml, workspace=implementation, env=env
|
||||
)
|
||||
# NOTE: Docker execution is logged by FTWorkspace.run() automatically
|
||||
|
||||
# Update config with filtered version
|
||||
if validation_result.filtered_config != config_yaml:
|
||||
implementation.inject_files(**{FT_YAML_FILE_NAME: validation_result.filtered_config})
|
||||
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[target_task.get_task_information()]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
|
||||
system_prompt = T(".prompts:finetune_eval.system").r(
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
system_managed_params=SYSTEM_MANAGED_PARAMS,
|
||||
)
|
||||
user_prompt = T(".prompts:finetune_eval.user").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=target_task.get_task_information(),
|
||||
stdout=validation_result.execution_output or "No output",
|
||||
code_yaml=implementation.file_dict[FT_YAML_FILE_NAME],
|
||||
workspace_files="\n".join(
|
||||
[
|
||||
f"- {file.name} ({file.stat().st_size} bytes)"
|
||||
for file in implementation.workspace_path.rglob("*")
|
||||
if file.is_file() and "checkpoint" not in file.absolute().as_posix()
|
||||
]
|
||||
),
|
||||
)
|
||||
feedback = build_cls_from_json_with_retry(
|
||||
CoSTEERSingleFeedback,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
init_kwargs_update_func=CoSTEERSingleFeedback.val_and_update_init_dict,
|
||||
)
|
||||
|
||||
# Force failure if validation failed programmatically
|
||||
if not validation_result.success:
|
||||
feedback.final_decision = False
|
||||
logger.warning("FTCoderEvaluator: Forced final_decision=False due to validation failure")
|
||||
|
||||
feedback.raw_execution = validation_result.raw_stdout or ""
|
||||
feedback.source_feedback[self.__class__.__name__] = feedback.final_decision
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
@@ -0,0 +1,39 @@
|
||||
"""
|
||||
LLM Fine-tuning Experiment Components
|
||||
|
||||
Defines tasks for LLM fine-tuning following data science pattern.
|
||||
"""
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
|
||||
|
||||
class FTTask(CoSTEERTask):
|
||||
"""Training task class for LLM fine-tuning operations - follows data science pattern"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_model: str,
|
||||
description: str,
|
||||
benchmark: str,
|
||||
involving_datasets: Optional[List[str]] = None,
|
||||
skip_data_processing: bool = False,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(name="LLM-Fine-Tuning", description=description, *args, **kwargs)
|
||||
self.base_model = base_model
|
||||
self.benchmark = benchmark
|
||||
self.involving_datasets = involving_datasets or []
|
||||
self.skip_data_processing = skip_data_processing # If True, reuse SOTA's data processing script
|
||||
|
||||
def get_task_information(self) -> str:
|
||||
"""Get task information for coder prompt generation"""
|
||||
task_desc = f"""name: {self.name}
|
||||
description: {self.description}
|
||||
base_model: {self.base_model}
|
||||
"""
|
||||
if self.involving_datasets:
|
||||
task_desc += f"involving_datasets: {self.involving_datasets}\n"
|
||||
return task_desc
|
||||
@@ -0,0 +1,742 @@
|
||||
data_coder:
|
||||
system: |-
|
||||
You are a world-class data engineer specializing in preparing training data for large language model fine-tuning.
|
||||
Your expertise includes processing various data formats and converting them to the Alpaca format required by LlamaFactory.
|
||||
|
||||
# Part 1: Context
|
||||
|
||||
## 1.1 Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
## 1.2 Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## 1.3 Available Datasets
|
||||
The following datasets are available for processing:
|
||||
{{ dataset_info }}
|
||||
|
||||
## 1.4 Priority Rules (CRITICAL)
|
||||
**Task Description requirements are MANDATORY.** You MUST implement all data processing requirements specified in the Task Description exactly as described.
|
||||
|
||||
# Part 2: Output Specification
|
||||
|
||||
## 2.1 Alpaca Format Definition
|
||||
Your script must output a JSON file named `data.json` in the current working directory (`{{ workspace_path }}`).
|
||||
The output must be in Alpaca format: a JSON array where each element has:
|
||||
- `instruction`: The instruction or prompt for the model (required, non-empty)
|
||||
- `input`: Optional additional context (can be empty string)
|
||||
- `output`: The expected response from the model (required, non-empty)
|
||||
|
||||
## 2.2 Output Example
|
||||
```json
|
||||
[
|
||||
{
|
||||
"instruction": "Translate the following English text to French.",
|
||||
"input": "Hello, how are you?",
|
||||
"output": "Bonjour, comment allez-vous?"
|
||||
},
|
||||
{
|
||||
"instruction": "Summarize the following article.",
|
||||
"input": "Article content here...",
|
||||
"output": "Summary of the article..."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## 2.3 Data Quality Awareness (IMPORTANT)
|
||||
- Raw datasets may contain low-quality, noisy, or incorrect samples
|
||||
- It is better to DISCARD questionable samples than to include them in training data
|
||||
- When encountering samples that are ambiguous, malformed, or have inconsistent answers, prefer filtering them out
|
||||
- A smaller but high-quality dataset is more valuable than a larger noisy one
|
||||
- High filtering rate is acceptable and expected - it means the script is doing quality control properly
|
||||
|
||||
## 2.4 Data Validation Rules
|
||||
Before writing the final data.json, implement these validations:
|
||||
|
||||
### 2.4.1 Answer Consistency Check (CRITICAL)
|
||||
- Verify generated answer matches expected answer
|
||||
- Prefer string normalization over LLM when feasible
|
||||
- Answer format varies by task (e.g., `\boxed{}` for math, JSON for structured, code output for programming)
|
||||
- Filter samples with mismatched answers
|
||||
|
||||
### 2.4.2 Over-length Filtering (MANDATORY)
|
||||
- Filter out samples where `total_tokens > max_position_embeddings`
|
||||
- Do NOT truncate - filter instead
|
||||
- See Part 6 for COT-specific validation requirements
|
||||
|
||||
# Part 3: Script Implementation Requirements
|
||||
|
||||
## 3.1 Basic Conventions
|
||||
1. Read data from `{{ datasets_path }}` directory (mounted read-only)
|
||||
2. Use standard Python libraries (json, csv, os, pathlib) when possible
|
||||
3. Handle file encoding properly (use utf-8)
|
||||
4. Include error handling for file operations
|
||||
5. Print progress information to stdout for debugging
|
||||
6. **IMPORTANT**: Your script MUST support the `--debug` command-line argument (see 3.2). Other than `--debug`, do NOT expect any other command-line arguments.
|
||||
|
||||
## 3.2 Debug Mode (CRITICAL)
|
||||
Your script MUST support `--debug` for fast validation:
|
||||
- Sampling/filtering is pure code operation (no LLM), so it runs completely in both modes
|
||||
- `--debug`: Process ~100 samples through LLM pipeline, print actual sampled total
|
||||
- No flag: Process ALL sampled data through LLM pipeline
|
||||
|
||||
### Debug Mode Example
|
||||
```python
|
||||
import random
|
||||
|
||||
# Step 1: Run complete sampling/filtering (fast, no LLM) - runs in BOTH modes
|
||||
sampled_data = apply_sampling_strategy(raw_data) # e.g., 50000 → 2000
|
||||
|
||||
# Step 2: Limit LLM processing in debug mode only
|
||||
if args.debug:
|
||||
samples_to_process = random.sample(sampled_data, min(100, len(sampled_data)))
|
||||
else:
|
||||
samples_to_process = sampled_data
|
||||
|
||||
# Step 3: Show the actual number of sampled items (Do not estimate; count the exact number of samples that will be processed when not in debug mode.)
|
||||
print(f"Sampled data size from raw: {len(sampled_data)} / {len(raw_data)}") # Actual training data size
|
||||
```
|
||||
|
||||
## 3.3 Logging Convention
|
||||
Only print progress at 20%, 40%, 60%, 80%, 100%. No per-item logs.
|
||||
|
||||
## 3.4 Output Statistics Format
|
||||
Your script should print statistics at the end of execution:
|
||||
|
||||
### Script Execution Summary (REQUIRED)
|
||||
```
|
||||
# Debug mode (--debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Debug samples processed: {debug_processed_count}
|
||||
Estimated full output: ~{int(actual_output / debug_processed_count * sampled_count)}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
|
||||
# Full mode (no --debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
```
|
||||
|
||||
### CoT Quality Statistics (REQUIRED for COT tasks)
|
||||
```
|
||||
========== COT QUALITY STATS ==========
|
||||
COT format check: {with_think_tags}/{total} have <think> tags
|
||||
Over-length filtered: {count} ({percentage}%)
|
||||
Answer consistency check: {passed}/{total} passed
|
||||
Length distribution: p25={}, p50={}, p75={}, p99={}
|
||||
=======================================
|
||||
```
|
||||
|
||||
# Part 4: Scope Clarification (IMPORTANT)
|
||||
**Your script should ONLY handle data processing and output data.json.**
|
||||
- DO NOT generate training configuration files (e.g., train.yaml, training_config.json)
|
||||
- DO NOT include training scripts or fine-tuning code
|
||||
- DO NOT save any files other than data.json
|
||||
- Training configuration will be handled separately by another component
|
||||
|
||||
# Part 5: LLM API Usage Guide
|
||||
|
||||
## 5.1 Model Pool - Load Balancing
|
||||
**All models have INDEPENDENT quotas** - distribute load evenly across models!
|
||||
|
||||
```python
|
||||
import os, json
|
||||
import litellm; litellm.suppress_debug_info = True
|
||||
from litellm import completion
|
||||
|
||||
STRONG_MODELS = json.loads(os.getenv("STRONG_MODEL_POOL", "[]")) # CoT generation
|
||||
WEAK_MODELS = json.loads(os.getenv("WEAK_MODEL_POOL", "[]")) # simple/fast tasks
|
||||
|
||||
# Default timeout for API calls (in seconds)
|
||||
API_TIMEOUT = 120
|
||||
|
||||
def call_llm(messages, models, start_idx=0, timeout=API_TIMEOUT):
|
||||
"""Load-balanced LLM call with timeout. Use start_idx to distribute across models."""
|
||||
if not models:
|
||||
raise RuntimeError("Model pool is empty. Set STRONG_MODEL_POOL/WEAK_MODEL_POOL env vars.")
|
||||
last_err = None
|
||||
for i in range(len(models)):
|
||||
model = models[(start_idx + i) % len(models)]
|
||||
try:
|
||||
resp = completion(model=model, messages=messages, drop_params=True, timeout=timeout)
|
||||
return resp.choices[0].message.content
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
continue
|
||||
raise RuntimeError(f"All models failed. Last error: {last_err}")
|
||||
```
|
||||
|
||||
## 5.2 Timeout & Efficiency (CRITICAL)
|
||||
- Set `timeout=120` for API calls to prevent blocking on complex problems
|
||||
- If timeout after retries, skip sample and continue
|
||||
- Prefer string/regex over LLM for validation (answer check, structure check) when possible
|
||||
|
||||
## 5.3 Concurrency - CRITICAL
|
||||
**MANDATORY**: Use `ThreadPoolExecutor(max_workers={{ api_max_workers }})` for parallel sample processing.
|
||||
- DO NOT use `os.cpu_count()` - it limits parallelism unnecessarily
|
||||
- The value {{ api_max_workers }} is intentional for maximizing API throughput
|
||||
- Pass `start_idx=sample_index % len(models)` to distribute load evenly
|
||||
|
||||
```python
|
||||
with ThreadPoolExecutor(max_workers={{ api_max_workers }}) as executor: # NOT os.cpu_count()!
|
||||
futures = {executor.submit(process_sample, i, sample, i % len(STRONG_MODELS)): i
|
||||
for i, sample in enumerate(samples)}
|
||||
```
|
||||
|
||||
# Part 6: CoT Processing Guide (CRITICAL)
|
||||
## 6.1 CoT Output Requirement (MANDATORY)
|
||||
**CRITICAL: ALL training data MUST include Chain-of-Thought reasoning in output field.**
|
||||
|
||||
### Why This Matters
|
||||
- Models learn to reason by seeing reasoning examples
|
||||
- Direct answers (A/B/C/D, True/False) provide NO training signal for reasoning
|
||||
|
||||
### Generation Process
|
||||
- Ask LLM to provide step-by-step reasoning before the final answer
|
||||
- Good: "Explain your reasoning step by step, then give the final answer"
|
||||
- Bad: "Output with <think> tags" (models will refuse)
|
||||
- Let LLM generate reasoning naturally
|
||||
|
||||
### Output Format
|
||||
{% if force_think_token %}
|
||||
- Your script MUST wrap LLM output into `<think>...</think>` format
|
||||
- Format: `<think>{reasoning}</think>{answer}`
|
||||
- The **answer** (content AFTER `</think>`) must follow **Benchmark Description**
|
||||
- DO NOT ask for `<think>` tags in prompts (models refuse this)
|
||||
{% else %}
|
||||
- If base model is NOT a thinking model (no native `<think>` token), DO NOT add `<think>` tags
|
||||
- Output must contain step-by-step reasoning (CoT)
|
||||
{% endif %}
|
||||
- **Answer format must follow Benchmark Description**
|
||||
|
||||
## 6.2 Post-Processing Validation
|
||||
{% if force_think_token %}
|
||||
- **Structure check**: `"<think>" in output and "</think>" in output`
|
||||
{% endif %}
|
||||
- **Content check**: Output must contain reasoning (not just direct answer)
|
||||
- **Answer check**: Answer format must match Benchmark Description
|
||||
|
||||
# Part 7: Previous Failed Attempts
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Part 8: Response Format
|
||||
Provide ONLY the Python script in a markdown code block:
|
||||
```python
|
||||
# Your complete Python script here
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the code block.
|
||||
|
||||
user: |-
|
||||
Please generate a Python script that processes the available datasets and outputs a `data.json` file in Alpaca format.
|
||||
|
||||
The script will be executed in two modes:
|
||||
1. **Debug mode (coding phase):** `python {{ workspace_path }}process_data.py --debug` - process 100 samples for fast validation
|
||||
2. **Full mode (running phase):** `python {{ workspace_path }}process_data.py` - generates all samples for training
|
||||
|
||||
Dataset files are located at: {{ datasets_path }}
|
||||
|
||||
## Detailed Dataset Descriptions
|
||||
{% for ds_name, ds_desc in involved_dataset_folder_desc.items() %}
|
||||
### Dataset: {{ ds_name }}
|
||||
(Note: All file paths for this dataset are relative to `{{ datasets_path }}{{ ds_name }}/`)
|
||||
{{ ds_desc }}
|
||||
{% endfor %}
|
||||
|
||||
Output file should be: {{ workspace_path }}data.json
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Data Processing Script
|
||||
```python
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Script
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the 'Previous Data Processing Script' based on the feedback above. Do not create a new script. Consider the feedback carefully and make necessary corrections. If the feedback asks for more information or logging, make sure to include that in your revised script to help the evaluator to better assess your implementation.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new Data Processing Script based on the task description.
|
||||
{% endif %}
|
||||
|
||||
**IMPORTANT**: Make sure your script supports the `--debug` argument as described in the system prompt.
|
||||
|
||||
finetune_coder:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in large language model fine-tuning using LlamaFactory.
|
||||
Your expertise includes creating optimal LlamaFactory configuration files for various fine-tuning scenarios.
|
||||
|
||||
# Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
## Previous Failed Attempts
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Available Fine-tuning Methods
|
||||
{{ available_methods }}
|
||||
|
||||
## Shared Parameters
|
||||
These parameters apply to all fine-tuning methods:
|
||||
{{ shared_params }}
|
||||
|
||||
## Method-Specific Parameters
|
||||
{% for method, params_desc in methods_specific_params.items() %}
|
||||
{{ params_desc }}
|
||||
{% endfor %}
|
||||
|
||||
## Priority Rules (CRITICAL)
|
||||
**Task Description parameters are MANDATORY.** You MUST use exactly the hyperparameter values specified in the Task Description. Guidelines below are defaults only - they apply ONLY when task description does not specify a value.
|
||||
|
||||
## Requirements
|
||||
1. Create a LlamaFactory configuration file named `train.yaml`
|
||||
2. Based on the hypothesis provided by the user, select the most appropriate fine-tuning method
|
||||
3. Generate full training configuration (no sample limit)
|
||||
4. Ensure all parameters are valid for LlamaFactory
|
||||
5. **Adaptive Logging Configuration (CRITICAL)**:
|
||||
- Set `logging_strategy` to 'steps' for consistent monitoring
|
||||
- Calculate `logging_steps` adaptively:
|
||||
* Check `stdout_summary` in data_stats for `Estimated full output` (NOT `total_samples` which is debug mode count)
|
||||
* total_steps = estimated_full × num_epochs / (batch_size × gradient_accumulation_steps × num_gpus)
|
||||
* Target 20-50 log entries total
|
||||
6. **Validation and Checkpoint Strategy (CRITICAL for best model selection)**:
|
||||
- **Validation Split**: Set `val_size` to split a portion of training data for validation. Choose ratio based on dataset size and task needs.
|
||||
- **Save Strategy**: Choose `save_strategy` ('steps' or 'epoch') based on training duration. MUST ensure `eval_strategy` == `save_strategy`.
|
||||
- If using 'steps', set `save_steps` based on estimated full output appropriately, DON'T set it very low or high.
|
||||
- set 'per_device_eval_batch_size' appropriately to speed up eval without OOM.
|
||||
- **Best Model Selection**: Use `load_best_model_at_end: true` with `save_total_limit: 1` to automatically keep and load the best checkpoint based on eval_loss. Note: `save_total_limit` will be force-injected to 1.
|
||||
7. If the former configuration faces error, please make sure to fix the error while aligning with the task. If these two goals conflict, please prioritize fixing the error.
|
||||
|
||||
## Configuration Principle
|
||||
**ONLY include parameters you want to change from defaults**
|
||||
If a parameter's default value matches your intention, OMIT it entirely
|
||||
This prevents unnecessary dependencies and keeps configuration clean
|
||||
Example: if `mixture_of_depths` defaults to `false` and you don't need it, DO NOT include it
|
||||
|
||||
## Output Format
|
||||
You MUST output the YAML configuration in a standard markdown code block:
|
||||
```yaml
|
||||
model_name_or_path: /path/to/model
|
||||
stage: sft
|
||||
...
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the YAML block.
|
||||
|
||||
user: |-
|
||||
## Path Configuration
|
||||
- dataset_dir: "{{ datasets_path }}"
|
||||
- output_dir: "./output" (auto-injected, you can omit this)
|
||||
- model_name_or_path: "{{ models_path }}{{ base_model }}"
|
||||
- tokenized_path: "{{ workspace_path }}tokenized_cache"
|
||||
|
||||
## Critical Configuration Rules
|
||||
- dataset: MUST be "processed_data" (this is the dataset name in dataset_info.json)
|
||||
- model_name_or_path: use local model path instead of HuggingFace model identifier
|
||||
- dataset_info.json is located at: "{{ datasets_path }}dataset_info.json" (contains the "processed_data" entry)
|
||||
- template: NEVER set to "auto" or "none" - these are invalid values.
|
||||
- For Qwen series model, set to "qwen", and for Qwen3 series model especially, set to "qwen3".
|
||||
- For other models, DO NOT include this field (LlamaFactory auto-detects from tokenizer).
|
||||
- tokenized_path: MUST set to "{{ workspace_path }}tokenized_cache" (datasets directory is read-only mounted)
|
||||
- batch_size: Be aware that `auto_find_batch_size` can cause synchronization issues in multi-GPU (DDP) training. Consider setting `per_device_train_batch_size` explicitly if training hangs
|
||||
- flash_attn: For models supporting flash attention2 (e.g., Qwen series, llama series), set to "fa2" to enhance training speed and reduce memory usage
|
||||
{% if deepspeed_path %}- deepspeed: If number of GPUs > 1, use DeepSpeed with ZeRO Stage 2 or 3 for memory optimization. specifically, set to "{{ deepspeed_path }}ds_z3_config.json" for ZeRO Stage 3, otherwise use "{{ deepspeed_path }}ds_z2_config.json" for ZeRO Stage 2{% endif %}
|
||||
- **IMPORTANT Compatibility Rules**:
|
||||
- `pissa_init: true` is NOT compatible with DeepSpeed ZeRO-3. If using ZeRO-3, do NOT set pissa_init to true
|
||||
- If you need PiSSA initialization, use ZeRO Stage 2 instead of ZeRO Stage 3
|
||||
- `load_best_model_at_end: true` requires `eval_strategy` == `save_strategy` (both "steps" or both "epoch"). Always set both to the same value.
|
||||
|
||||
{% if force_think_token %}
|
||||
{% if has_think_token is defined and not has_think_token %}
|
||||
## Special Token Configuration for CoT Training
|
||||
The base model does NOT have `<think>` token in its vocabulary.
|
||||
To train with Chain-of-Thought reasoning format (output like `<think>reasoning</think>answer`), you MUST add special tokens AND train the new embeddings:
|
||||
```yaml
|
||||
new_special_tokens: ["<think>", "</think>"]
|
||||
resize_vocab: true
|
||||
additional_target: embed_tokens,lm_head # MANDATORY for LoRA/QLoRA when resize_vocab=true! And Full Training does not need this field.
|
||||
```
|
||||
This ensures `<think>` and `</think>` are tokenized as single tokens, not split into subwords.
|
||||
{% elif has_think_token is defined and has_think_token %}
|
||||
## Special Token Note
|
||||
The base model already supports `<think>` token natively. No need to add special tokens for CoT training.
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{# When force_think_token=false, no special token configuration needed #}
|
||||
|
||||
{% if data_stats %}
|
||||
## Processed Data Statistics (from debug mode)
|
||||
{{ data_stats }}
|
||||
|
||||
**Your Task**: Implement the training configuration specified in the task description.
|
||||
|
||||
- Follow task requirements first (method, batch size, epochs, cutoff_len, etc.)
|
||||
- Apply technical constraints only when task doesn't specify:
|
||||
- `cutoff_len`: ≤ min(max_position_embeddings, memory limit, data p99)
|
||||
- `per_device_train_batch_size`: Choose based on Memory Estimates table
|
||||
- `gradient_accumulation_steps`: Adjust for stable training (effective_batch = batch × accum × gpus)
|
||||
- Validation setup: `val_size`, `eval_strategy` == `save_strategy`, `load_best_model_at_end: true`
|
||||
{% endif %}
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Configuration
|
||||
```yaml
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Configuration
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the configuration based on the feedback above and the hypothesis.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new configuration for the model {{ base_model }} based on the hypothesis above.
|
||||
|
||||
**Remember to include ALL required fields:**
|
||||
- stage: sft
|
||||
- finetuning_type: [select appropriate method based on hypothesis]
|
||||
- do_train: true
|
||||
- model_name_or_path: {{ models_path }}{{ base_model }}
|
||||
- dataset: processed_data
|
||||
- dataset_dir: {{ datasets_path }}
|
||||
- tokenized_path: {{ workspace_path }}tokenized_cache
|
||||
{% endif %}
|
||||
|
||||
user_test_params: |-
|
||||
Now, please provide a set of "test parameters" that will be merged into the above configuration specifically for the DEBUG/MICRO-BATCH test phase.
|
||||
|
||||
The debug phase runs on a very small subset (~10 samples).
|
||||
You need to override parameters that adapt to the dataset for quick debugging the yaml config.
|
||||
|
||||
**Example for Test Parameters:**
|
||||
- Set `num_train_epochs` to 1.
|
||||
- Set `max_samples` to a very small number.
|
||||
|
||||
**Output Format:**
|
||||
Output ONLY the YAML block for these test parameters:
|
||||
```yaml
|
||||
num_train_epochs: 1
|
||||
...
|
||||
```
|
||||
|
||||
finetune_eval:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in evaluating fine-tuning configurations for large language models using LlamaFactory.
|
||||
Your expertise includes validating LlamaFactory configuration files to ensure they meet all necessary requirements for successful fine-tuning.
|
||||
|
||||
You will be provided with:
|
||||
1. A detailed scenario description which requires a fine-tuning LLM.
|
||||
2. A yaml configuration file named `train.yaml` created for LlamaFactory fine-tuning.
|
||||
3. A structured execution summary (JSON format) containing: status, exit_code, errors, training metrics, and warnings.
|
||||
4. The files generated during the execution.
|
||||
5. Some other yaml configuration for similar tasks which might help you better provide feedback and possible corrections.
|
||||
|
||||
Your task is to:
|
||||
1. Check the execution summary to determine if the run succeeded.
|
||||
2. validate the provided `train.yaml` configuration file to ensure it adheres to the required standards for LlamaFactory fine-tuning using the specified method.
|
||||
3. Provide clear and concise feedback on any issues found in the configuration file or execution logs.
|
||||
4. Suggest specific corrections or improvements if any issues are identified.
|
||||
|
||||
You must give a false final decision only if:
|
||||
- The execution fails with non-zero exit code.
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
### Similar Successful Implementations to help training config Improvement
|
||||
The user has done several similar tasks and get some successful implementations. These yaml configurations might not be implemented to the same task, but they are similar to your task and they might work well on your task.
|
||||
Please refer to these successful implementation and provide your suggestions in your response on how to correct your current code based on these successful implementations.
|
||||
## Successful Implementations for Similar Tasks
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Similar Task {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Yaml configurations:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Important Notice
|
||||
- You may find that the execution is short with limited data and iterations. This is expected as we are only validating the configuration file's correctness and not performing full-scale training. Don't treat this as a failure. Also do not put this information in your feedback.
|
||||
|
||||
## Output Format
|
||||
Please respond with your feedback in the following JSON format without anything else.
|
||||
```json
|
||||
{
|
||||
"execution": "State if run succeeded. If errors, include all messages verbatim. Classify cause: algorithm, implementation, or environment."
|
||||
"return_checking": "Plain text. Examine the generated files from the user input. Does the output contains a fine-tuned model or expected artifacts? If not, specify what is missing or incorrect.",
|
||||
"code": "Plain text. Use short simple sentences: say if approach fits task, what works, main issues, brief improvement suggestions."
|
||||
"final_decision": <true/false>, # Final decision on whether the configuration is acceptable for full data fine-tuning
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
# Scenario Information
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Yaml Configuration File
|
||||
```yaml
|
||||
{{ code_yaml }}
|
||||
|
||||
## Execution Summary (Structured)
|
||||
```json
|
||||
{{ stdout }}
|
||||
```
|
||||
|
||||
## Workspace Files
|
||||
{{ workspace_files }}
|
||||
|
||||
data_eval:
|
||||
system: |-
|
||||
You are a data quality expert for LLM fine-tuning using LlamaFactory.
|
||||
Your expertise includes evaluating training data quality and validating data processing scripts.
|
||||
|
||||
You will evaluate:
|
||||
1. **Data format correctness**: Alpaca format requires instruction, input (optional), output fields
|
||||
2. **Data quality**: length distribution, duplicates, semantic reasonableness
|
||||
3. **Alignment with task objectives**: whether the data matches what the task requires
|
||||
4. **Code logic correctness**: whether the processing script is well-designed
|
||||
|
||||
## The Main Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
## Similar Successful Data Processing Examples
|
||||
The following are successful data processing implementations for similar tasks:
|
||||
{% for knowledge in queried_similar_successful_knowledge %}
|
||||
### Example {{ loop.index }}:
|
||||
**Task:** {{ knowledge.target_task.get_task_information() }}
|
||||
**Code:**
|
||||
```python
|
||||
{{ knowledge.implementation.file_dict.get("process_data.py", "N/A") }}
|
||||
```
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Debug Mode Context (IMPORTANT)
|
||||
This evaluation runs during the CODING phase in DEBUG MODE.
|
||||
- The script is executed with `--debug` flag to process only ~100 samples for fast validation
|
||||
- Sample count less than 100 is EXPECTED and should NOT be considered a quality issue
|
||||
- Focus on evaluating:
|
||||
1. Data format correctness (Alpaca format)
|
||||
2. Data quality of the generated samples
|
||||
3. Script logic correctness (will it work in full mode?)
|
||||
- Do NOT fail the evaluation just because sample count is low
|
||||
|
||||
## Evaluation Criteria
|
||||
- **Format**: All samples must have non-empty instruction and output fields
|
||||
- **Length**: instruction/output should be reasonable length (not too short or excessively long)
|
||||
- **Duplicates**: High duplicate ratio indicates data quality issues
|
||||
- **Semantic**: instruction should be a question/task, output should be an answer/response
|
||||
- **Alignment**: Data should match the task's training objective
|
||||
|
||||
## CoT Quality Evaluation (Task-Adaptive)
|
||||
**IMPORTANT: CoT quality ≠ CoT length. Adapt criteria based on task type from README metadata.**
|
||||
|
||||
**Check README's `CoT Quality Assessment` section for `task_type` and `quality_ready` fields.**
|
||||
|
||||
1. **Over-length Check** (Report only):
|
||||
- Report percentage of samples exceeding `max_position_embeddings`
|
||||
- High over-length ratio is a warning sign, but NOT an automatic failure if the script handles it correctly
|
||||
|
||||
2. **Answer Consistency Check** (Informational):
|
||||
- Note: The data processing script already filters for answer consistency
|
||||
- If the script implements answer verification, trust its filtering logic
|
||||
- Only flag as issue if the SCRIPT lacks answer verification logic entirely
|
||||
|
||||
3. **Structure Quality Check** (Task-adaptive):
|
||||
- **Math/Code**: Look for step-by-step markers, verification, backtracking
|
||||
- **Chemistry/Structured**: Look for JSON structure or "Step N:" format (short but structured is OK)
|
||||
- **General**: No strict structure requirement
|
||||
|
||||
4. **Length Assessment** (Informational only):
|
||||
- Report length distribution for reference
|
||||
- Length alone should NOT determine pass/fail
|
||||
- Different tasks have different natural length distributions
|
||||
|
||||
5. **Polish Quality Assessment**:
|
||||
- All data must be polished before use
|
||||
- If README shows `baseline_quality: high`: verify enrichment was applied
|
||||
- If README shows `baseline_quality: low`: verify full generation/rewrite was done
|
||||
- Check polish met the requirements in `polish_strategy`
|
||||
|
||||
**Include in return_checking:**
|
||||
- "Task type: {type}, Quality ready: {ready}"
|
||||
- "CoT stats: p50={}, over-length={X}%, structure quality={Y}%"
|
||||
- Assessment based on task-appropriate criteria
|
||||
|
||||
## Hard Check Criteria (AUTOMATIC FAIL if not met)
|
||||
{% if force_think_token %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- EVERY sample MUST contain `<think>` and `</think>` tags
|
||||
- Content AFTER `</think>` must be non-empty
|
||||
|
||||
**Rejection:** "FAIL: {X} samples missing <think> tags."
|
||||
{% else %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- Output must contain reasoning content (not just a direct answer)
|
||||
- Answer format must match **Benchmark Description**
|
||||
- Do NOT reject for reasoning quality or answer correctness
|
||||
|
||||
**Rejection:** "FAIL: {X}% of samples are direct answers without reasoning."
|
||||
{% endif %}
|
||||
|
||||
### 2. Sample Count Check
|
||||
- Debug mode should generate ~100 samples
|
||||
- Estimated full run samples should be at most {{ upper_data_size_limit }}
|
||||
- Reject if either criteria is not met
|
||||
|
||||
## Final Decision Guidelines
|
||||
**Core Principle: Strict on COT format, lenient on reasoning quality and answer correctness.**
|
||||
|
||||
- **Approve (true)** if:
|
||||
- Script runs successfully (exit_code == 0)
|
||||
- At least 1 sample is generated
|
||||
{% if force_think_token %}- ALL samples have `<think>` and `</think>` tags (MANDATORY){% else %}- ALL samples contain reasoning content (not just direct answers){% endif %}
|
||||
- Data format is correct (Alpaca format with instruction/output)
|
||||
|
||||
- **Reject (false)** if ANY of these:
|
||||
- Script fails to run (exit_code != 0)
|
||||
- Zero samples are generated
|
||||
{% if force_think_token %}- **ANY sample missing `<think>` or `</think>` tags (HARD FAIL)**{% else %}- **ANY sample missing reasoning content (just direct answer)**{% endif %}
|
||||
- Data format is fundamentally broken
|
||||
- **Data does NOT match task description requirements**
|
||||
|
||||
- **Do NOT reject** for:
|
||||
- Low sample count in debug mode (expected)
|
||||
- Moderate quality variations in individual samples
|
||||
- Length distribution not matching ideal patterns
|
||||
- High filtering rate (script doing its job)
|
||||
|
||||
## Important Note
|
||||
- Do not summarize the code into your feedback and DO NOT copy the task description also. Only provide new insights based on your evaluation.
|
||||
- If you think the current logging information is not sufficient to find out the issues, please specify what additional logging information is needed in your feedback and put this information in 'code' block. The user will add further provide you the additional logging information in the next iteration.
|
||||
- Do not write any code in your response, use plain text only.
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only (no markdown code block):
|
||||
{
|
||||
"execution": "Script execution status and data generation result. Include exit code and any errors.",
|
||||
"return_checking": "Data quality analysis: format validation, length distribution assessment, duplicate ratio, semantic issues found; Hard check criteria: does the solution meet the hard check criteria",
|
||||
"code": "Code issues and specific improvement suggestions. What works well, what needs fixing.",
|
||||
"final_decision": true/false
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
{% if script_code %}
|
||||
|
||||
# Data Processing Script (for debugging)
|
||||
```python
|
||||
{{ script_code }}
|
||||
```
|
||||
{% endif %}
|
||||
{% if stdout %}
|
||||
|
||||
# Execution Output ({% if exit_code != 0 %}error logs{% else %}summary{% endif %})
|
||||
```
|
||||
Exit code: {{ exit_code }}
|
||||
{{ stdout }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
# Data Statistics
|
||||
```json
|
||||
{{ data_stats }}
|
||||
```
|
||||
|
||||
# Sample Data ({{ sample_count }} samples from total {{ total_samples }}) [DEBUG MODE]
|
||||
```json
|
||||
{{ data_samples }}
|
||||
```
|
||||
|
||||
runner_eval:
|
||||
system: |-
|
||||
You are a world-class ML engineer evaluating LLM fine-tuning results.
|
||||
|
||||
## Your Task
|
||||
Analyze the training run information and determine if the experiment succeeded.
|
||||
|
||||
## Evaluation Criteria (for final_decision)
|
||||
1. **Execution Success**: Did training complete without errors? Check exit_code and model outputs.
|
||||
2. **Benchmark Execution**: Did benchmark run successfully? Check benchmark results availability.
|
||||
|
||||
## Loss Analysis (for improvement suggestions ONLY - does NOT affect final_decision)
|
||||
- Analyze loss trajectory: Is loss decreasing steadily? Any signs of overfitting?
|
||||
- Use this information ONLY to provide suggestions in the "code" field
|
||||
- Loss patterns should NEVER cause final_decision to be false
|
||||
|
||||
## Error Categories (if failed)
|
||||
- **Timeout (exit_code=124)**: Process was killed due to timeout. Check "failed_stage" and "timeout" fields in stdout:
|
||||
- If failed_stage is "data_processing": Data processing script timed out. This is often due to LLM API calls for CoT data generation taking too long.
|
||||
- If failed_stage is "training": Training timed out.
|
||||
- **OOM**: GPU memory exhaustion - suggest batch size/model changes
|
||||
- **CUDA**: Driver/device issues - suggest environment checks
|
||||
- **Config**: Invalid parameters - suggest specific fixes
|
||||
- **Data**: Dataset issues - suggest data pipeline fixes
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only:
|
||||
{
|
||||
"execution": "Execution status: SUCCESS or FAILED with category [OOM/CUDA/Config/Data]. Include key metrics or error details.",
|
||||
"return_checking": "If success: benchmark analysis. If failed: what failed and expected behavior.",
|
||||
"code": "Configuration assessment and improvement suggestions",
|
||||
"final_decision": true/false // Set to true as long as training succeeded (exit_code=0) and benchmark ran successfully
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Training Configuration
|
||||
```yaml
|
||||
{{ config_yaml }}
|
||||
```
|
||||
|
||||
# Execution Info
|
||||
- Exit Code: {{ exit_code }}
|
||||
- Model Output Files: {{ model_files_status }}
|
||||
{% if failed_stage %}- Failed Stage: {{ failed_stage }}
|
||||
- Stage Timeout Config: {{ timeout_seconds }} seconds
|
||||
{% endif %}
|
||||
|
||||
# Benchmark Results
|
||||
```json
|
||||
{{ benchmark_result }}
|
||||
```
|
||||
|
||||
# Loss History (train loss and eval_loss if validation enabled)
|
||||
```json
|
||||
{{ loss_history }}
|
||||
```
|
||||
{% include "components.coder.finetune.prompts:runner_eval.train_output" %}
|
||||
|
||||
train_output: |-
|
||||
# Training Output (key information extracted from stdout)
|
||||
```
|
||||
{{ stdout }}
|
||||
```
|
||||
@@ -0,0 +1,304 @@
|
||||
"""
|
||||
Simplified LLM Fine-tuning Configuration Validator
|
||||
|
||||
Two-step validation:
|
||||
1. Parameter filtering - Remove unsupported parameters
|
||||
2. Micro-batch testing - Runtime validation with small dataset
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Set
|
||||
|
||||
import yaml
|
||||
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DEBUG_YAML_FILE_NAME,
|
||||
FT_TEST_PARAMS_FILE_NAME,
|
||||
get_ft_env,
|
||||
get_workspace_prefix,
|
||||
)
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.scen.llama_factory_manager import LLaMAFactory_manager
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
# System-managed parameters that are automatically injected during validation.
|
||||
# These should NOT be checked for alignment in eval prompts.
|
||||
# Single source of truth: modify here to change injected parameters.
|
||||
SYSTEM_MANAGED_PARAMS = {
|
||||
"overwrite_cache": True, # Avoid HF datasets cache lock contention
|
||||
"save_only_model": True, # Save disk space
|
||||
# "save_total_limit": 1, # Limit checkpoint count to save disk space
|
||||
"output_dir": "./output", # Standardize model output location
|
||||
"per_device_eval_batch_size": 1, # Prevent OOM during evaluation
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
"""Configuration validation result"""
|
||||
|
||||
success: bool
|
||||
filtered_config: str
|
||||
execution_output: str = "" # Parsed/summarized output for LLM
|
||||
raw_stdout: str = "" # Full raw stdout for UI display
|
||||
errors: List[str] = field(default_factory=list)
|
||||
execution_time: float = 0.0
|
||||
|
||||
|
||||
class LLMConfigValidator:
|
||||
"""LLM configuration validator with two-step validation:
|
||||
|
||||
1. Parameter filtering - Remove unsupported parameters
|
||||
2. Micro-batch test - Runtime validation with small dataset
|
||||
|
||||
The micro-batch test inherently validates completeness, so no separate completeness check is needed.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._supported_params_cache: Optional[Set[str]] = None
|
||||
|
||||
def validate_and_test(self, config_yaml: str, workspace: FBWorkspace, env) -> ValidationResult:
|
||||
"""Three-step validation: parameter filtering + injection + micro-batch testing"""
|
||||
start_time = time.time()
|
||||
|
||||
# Step 1: Parameter filtering
|
||||
filtered_config, removed_params = self._filter_parameters(config_yaml)
|
||||
|
||||
# Step 2: Inject required parameters for multi-task environments
|
||||
injected_config = self._inject_required_parameters(filtered_config)
|
||||
|
||||
# Step 3: Micro-batch testing (validates everything at runtime)
|
||||
result = self._run_micro_batch_test(injected_config, workspace, env)
|
||||
result.execution_time = time.time() - start_time
|
||||
|
||||
# Add filtered params info to execution_output for agent learning
|
||||
if removed_params:
|
||||
filter_info = (
|
||||
f"\n\n[Filtered Parameters] {len(removed_params)} unsupported params removed: {removed_params}"
|
||||
)
|
||||
result.execution_output += filter_info
|
||||
|
||||
return result
|
||||
|
||||
def _filter_parameters(self, config_yaml: str) -> tuple[str, List[str]]:
|
||||
"""Filter configuration parameters to only include supported ones.
|
||||
|
||||
Returns:
|
||||
tuple: (filtered_yaml, removed_params_list)
|
||||
"""
|
||||
config_dict = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config_dict, dict):
|
||||
return config_yaml, []
|
||||
|
||||
supported_params = self._get_supported_parameters()
|
||||
|
||||
filtered_config = {}
|
||||
removed_params = []
|
||||
for k, v in config_dict.items():
|
||||
if k in supported_params:
|
||||
filtered_config[k] = v
|
||||
else:
|
||||
removed_params.append(k)
|
||||
|
||||
if removed_params:
|
||||
logger.info(f"Filtered out {len(removed_params)} unsupported parameters: {removed_params}")
|
||||
|
||||
return yaml.dump(filtered_config, default_flow_style=False, sort_keys=False), removed_params
|
||||
|
||||
def _inject_required_parameters(self, config_yaml: str) -> str:
|
||||
"""Inject required parameters for multi-task environments.
|
||||
|
||||
Uses SYSTEM_MANAGED_PARAMS as the single source of truth.
|
||||
"""
|
||||
config = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config, dict):
|
||||
return config_yaml
|
||||
|
||||
config.update(SYSTEM_MANAGED_PARAMS)
|
||||
|
||||
logger.info(f"Injected required parameters: {SYSTEM_MANAGED_PARAMS}")
|
||||
return yaml.dump(config, default_flow_style=False, sort_keys=False)
|
||||
|
||||
def _get_supported_parameters(self) -> Set[str]:
|
||||
"""Get supported parameters from LlamaFactory Manager"""
|
||||
if self._supported_params_cache is not None:
|
||||
return self._supported_params_cache
|
||||
|
||||
all_params = LLaMAFactory_manager.get_parameters()
|
||||
|
||||
# Extract all parameter names from all parameter types (including nested structures)
|
||||
supported_params = set()
|
||||
for param_type, params_dict in all_params.items():
|
||||
if isinstance(params_dict, dict):
|
||||
# Recursively extract parameter names from nested dictionaries
|
||||
for key, value in params_dict.items():
|
||||
if isinstance(value, dict) and "name" in value:
|
||||
# This is a parameter definition with metadata
|
||||
supported_params.add(key)
|
||||
elif isinstance(value, dict):
|
||||
# This is a nested category (e.g., BaseModelArguments, LoraArguments)
|
||||
# Extract parameter names from the nested structure
|
||||
for nested_key, nested_value in value.items():
|
||||
if isinstance(nested_value, dict) and "name" in nested_value:
|
||||
supported_params.add(nested_key)
|
||||
|
||||
if not supported_params:
|
||||
raise RuntimeError("No parameters found in LlamaFactory Manager")
|
||||
|
||||
logger.info(f"Loaded {len(supported_params)} parameters from LlamaFactory Manager")
|
||||
self._supported_params_cache = supported_params
|
||||
return supported_params
|
||||
|
||||
def _parse_execution_log(self, stdout: str, exit_code: int, failed_stage: str = None) -> str:
|
||||
"""Parse execution log and extract key information for LLM evaluation.
|
||||
|
||||
Reduces log from ~36k tokens to ~500 tokens by extracting only:
|
||||
- Status and exit code
|
||||
- Error messages (if any)
|
||||
- Training metrics (if successful)
|
||||
- Warnings (limited)
|
||||
- Timeout and stage information (if applicable)
|
||||
|
||||
Args:
|
||||
stdout: The execution output
|
||||
exit_code: The process exit code
|
||||
failed_stage: Which stage failed - "data_processing" or "training"
|
||||
"""
|
||||
result = {
|
||||
"status": "success" if exit_code == 0 else "failed",
|
||||
"exit_code": exit_code,
|
||||
}
|
||||
|
||||
# Handle timeout (exit_code 124)
|
||||
if exit_code == 124:
|
||||
result["timeout"] = True
|
||||
if failed_stage:
|
||||
result["failed_stage"] = failed_stage
|
||||
|
||||
# 1. Extract error information (highest priority)
|
||||
# Strategy: extract rank0's error block (each line prefixed with [rank0]:)
|
||||
error_text = None
|
||||
|
||||
# Method A: Extract [rank0]: prefixed lines and reconstruct traceback
|
||||
rank0_lines = re.findall(r"\[rank0\]:[^\n]+", stdout)
|
||||
if rank0_lines:
|
||||
rank0_block = "\n".join(line.replace("[rank0]: ", "").replace("[rank0]:", "") for line in rank0_lines)
|
||||
# Find traceback in rank0 block
|
||||
tb_match = re.search(
|
||||
r"Traceback \(most recent call last\):.*?(?:Error|Exception):[^\n]+", rank0_block, re.DOTALL
|
||||
)
|
||||
if tb_match:
|
||||
error_text = tb_match.group(0)
|
||||
|
||||
# Method B: Fallback to generic traceback (no rank prefix)
|
||||
# Use findall to get ALL tracebacks, then keep the first one (root cause)
|
||||
if not error_text:
|
||||
all_tracebacks = re.findall(
|
||||
r"Traceback \(most recent call last\):.*?(?:Error|Exception):[^\n]+", stdout, re.DOTALL
|
||||
)
|
||||
if all_tracebacks:
|
||||
# First traceback is usually the root cause
|
||||
error_text = all_tracebacks[0]
|
||||
if len(all_tracebacks) > 1:
|
||||
error_text += f"\n\n[Note: {len(all_tracebacks)} total errors, showing root cause]"
|
||||
|
||||
if error_text:
|
||||
# Limit length but keep from the END (actual error type/message is at the end of traceback)
|
||||
result["error"] = error_text[-4000:] if len(error_text) > 4000 else error_text
|
||||
|
||||
# 2. Extract training information
|
||||
if "Running training" in stdout:
|
||||
result["training_started"] = True
|
||||
|
||||
# Extract training config
|
||||
# NOTE: we may have log like "Num examples = 1,000,000" and "Num Epochs = 1,000"; So we need to handle ","
|
||||
num_examples = re.search(r"Num examples\s*=\s*([\d,]+)", stdout)
|
||||
num_epochs = re.search(r"Num Epochs\s*=\s*([\d,]+)", stdout)
|
||||
if num_examples:
|
||||
result["num_examples"] = int(num_examples.group(1).replace(",", ""))
|
||||
if num_epochs:
|
||||
result["num_epochs"] = int(num_epochs.group(1).replace(",", ""))
|
||||
|
||||
# Extract final metrics (JSON format from trainer output)
|
||||
final_metrics = re.search(r"\{'train_runtime':[^}]+\}", stdout)
|
||||
if final_metrics:
|
||||
try:
|
||||
metrics = eval(final_metrics.group(0)) # Safe: only numbers and strings
|
||||
result["final_metrics"] = {
|
||||
"train_loss": metrics.get("train_loss"),
|
||||
"train_runtime": metrics.get("train_runtime"),
|
||||
"train_samples_per_second": metrics.get("train_samples_per_second"),
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Check completion
|
||||
if "Training completed" in stdout:
|
||||
result["completed"] = True
|
||||
|
||||
# 3. Extract warnings (limit to 20)
|
||||
warnings = re.findall(r"\[WARNING[^\]]*\][^\n]+", stdout)
|
||||
if warnings:
|
||||
result["warnings"] = list(set(warnings))[:20]
|
||||
|
||||
# 4. Fallback: if parsing failed, include truncated raw log
|
||||
if not result.get("error") and not result.get("training_started"):
|
||||
result["raw_log_tail"] = stdout[-2000:] if len(stdout) > 2000 else stdout
|
||||
|
||||
return json.dumps(result, indent=2, ensure_ascii=False)
|
||||
|
||||
def _run_micro_batch_test(self, config_yaml: str, workspace: FBWorkspace, env) -> ValidationResult:
|
||||
"""Run micro-batch training test for runtime validation"""
|
||||
result = ValidationResult(success=True, filtered_config=config_yaml)
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
|
||||
# Create micro-batch test configuration
|
||||
config = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config, dict):
|
||||
result.success = False
|
||||
result.execution_output = "Invalid YAML configuration"
|
||||
result.errors.append("Invalid configuration for micro-batch test")
|
||||
return result
|
||||
|
||||
test_config = config.copy()
|
||||
|
||||
# Load extra test parameters from workspace (generated by coder in 2nd turn)
|
||||
extra_test_params = yaml.safe_load(workspace.file_dict[FT_TEST_PARAMS_FILE_NAME])
|
||||
|
||||
# Merge extra test parameters (overrides previous settings)
|
||||
if extra_test_params:
|
||||
test_config.update(extra_test_params)
|
||||
|
||||
# Run micro-batch training
|
||||
workspace.inject_files(**{FT_DEBUG_YAML_FILE_NAME: yaml.dump(test_config, default_flow_style=False)})
|
||||
training_result = workspace.run(
|
||||
env=env,
|
||||
entry=f"llamafactory-cli train {FT_DEBUG_YAML_FILE_NAME}",
|
||||
)
|
||||
|
||||
# Remove micro-batch test files
|
||||
workspace.remove_files([FT_DEBUG_YAML_FILE_NAME, FT_TEST_PARAMS_FILE_NAME])
|
||||
|
||||
# Parse and store structured execution output (reduces ~36k tokens to ~500)
|
||||
raw_stdout = training_result.stdout if training_result.stdout else ""
|
||||
result.raw_stdout = raw_stdout # Keep full log for UI
|
||||
result.execution_output = self._parse_execution_log(raw_stdout, training_result.exit_code)
|
||||
|
||||
# Check results
|
||||
progress_indicators = ["train_loss", "Training:", "Epoch", "loss:", "step"]
|
||||
has_progress = any(ind.lower() in training_result.stdout.lower() for ind in progress_indicators)
|
||||
|
||||
if training_result.exit_code == 0 and has_progress:
|
||||
logger.info("Micro-batch test passed")
|
||||
result.success = True
|
||||
else:
|
||||
result.success = False
|
||||
result.errors.append(f"Micro-batch test failed (exit_code={training_result.exit_code})")
|
||||
|
||||
return result
|
||||
@@ -7,14 +7,14 @@ class BasePropSetting(ExtendedBaseSettings):
|
||||
You can add following config in the subclass to distinguish the environment variables.
|
||||
"""
|
||||
|
||||
scen: str = ""
|
||||
knowledge_base: str = ""
|
||||
knowledge_base_path: str = ""
|
||||
hypothesis_gen: str = ""
|
||||
interactor: str = ""
|
||||
hypothesis2experiment: str = ""
|
||||
coder: str = ""
|
||||
runner: str = ""
|
||||
summarizer: str = ""
|
||||
scen: str | None = None
|
||||
knowledge_base: str | None = None
|
||||
knowledge_base_path: str | None = None
|
||||
hypothesis_gen: str | None = None
|
||||
interactor: str | None = None
|
||||
hypothesis2experiment: str | None = None
|
||||
coder: str | None = None
|
||||
runner: str | None = None
|
||||
summarizer: str | None = None
|
||||
|
||||
evolving_n: int = 10
|
||||
|
||||
@@ -30,14 +30,30 @@ class RDLoop(LoopBase, metaclass=LoopMeta):
|
||||
logger.log_object(scen, tag="scenario")
|
||||
logger.log_object(PROP_SETTING.model_dump(), tag="RDLOOP_SETTINGS")
|
||||
logger.log_object(RD_AGENT_SETTINGS.model_dump(), tag="RD_AGENT_SETTINGS")
|
||||
self.hypothesis_gen: HypothesisGen = import_class(PROP_SETTING.hypothesis_gen)(scen)
|
||||
self.hypothesis_gen: HypothesisGen = (
|
||||
import_class(PROP_SETTING.hypothesis_gen)(scen)
|
||||
if hasattr(PROP_SETTING, "hypothesis_gen") and PROP_SETTING.hypothesis_gen
|
||||
else None
|
||||
)
|
||||
|
||||
self.hypothesis2experiment: Hypothesis2Experiment = import_class(PROP_SETTING.hypothesis2experiment)()
|
||||
self.hypothesis2experiment: Hypothesis2Experiment = (
|
||||
import_class(PROP_SETTING.hypothesis2experiment)()
|
||||
if hasattr(PROP_SETTING, "hypothesis2experiment") and PROP_SETTING.hypothesis2experiment
|
||||
else None
|
||||
)
|
||||
|
||||
self.coder: Developer = import_class(PROP_SETTING.coder)(scen)
|
||||
self.runner: Developer = import_class(PROP_SETTING.runner)(scen)
|
||||
self.coder: Developer = (
|
||||
import_class(PROP_SETTING.coder)(scen) if hasattr(PROP_SETTING, "coder") and PROP_SETTING.coder else None
|
||||
)
|
||||
self.runner: Developer = (
|
||||
import_class(PROP_SETTING.runner)(scen) if hasattr(PROP_SETTING, "runner") and PROP_SETTING.runner else None
|
||||
)
|
||||
|
||||
self.summarizer: Experiment2Feedback = import_class(PROP_SETTING.summarizer)(scen)
|
||||
self.summarizer: Experiment2Feedback = (
|
||||
import_class(PROP_SETTING.summarizer)(scen)
|
||||
if hasattr(PROP_SETTING, "summarizer") and PROP_SETTING.summarizer
|
||||
else None
|
||||
)
|
||||
self.trace = Trace(scen=scen)
|
||||
super().__init__()
|
||||
|
||||
@@ -72,21 +88,21 @@ class RDLoop(LoopBase, metaclass=LoopMeta):
|
||||
return exp
|
||||
|
||||
def feedback(self, prev_out: dict[str, Any]):
|
||||
# TODO: the logic branch of exception should be moved to summarizer
|
||||
e = prev_out.get(self.EXCEPTION_KEY, None)
|
||||
if e is not None:
|
||||
feedback = HypothesisFeedback(
|
||||
observations=str(e),
|
||||
hypothesis_evaluation="",
|
||||
new_hypothesis="",
|
||||
reason="",
|
||||
reason=str(e),
|
||||
decision=False,
|
||||
code_change_summary="",
|
||||
acceptable=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["direct_exp_gen"]["exp_gen"], feedback))
|
||||
else:
|
||||
feedback = self.summarizer.generate_feedback(prev_out["running"], self.trace)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["running"], feedback))
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
return feedback
|
||||
|
||||
# TODO: `def record(self, prev_out: dict[str, Any]):` has already been hard coded into LoopBase
|
||||
# So we should add it into RDLoop class to make sure every RDLoop Sub Class be aware of it.
|
||||
def record(self, prev_out: dict[str, Any]):
|
||||
feedback = prev_out["feedback"]
|
||||
exp = prev_out.get("running") or prev_out.get("coding") or prev_out.get("direct_exp_gen", {}).get("exp_gen")
|
||||
self.trace.sync_dag_parent_and_hist((exp, feedback), prev_out[self.LOOP_IDX_KEY])
|
||||
|
||||
+105
-22
@@ -3,13 +3,20 @@ from __future__ import annotations
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Generator
|
||||
from contextlib import nullcontext
|
||||
from typing import Any, Generic, TypeVar
|
||||
from typing import Generic, TypeVar, cast
|
||||
|
||||
from filelock import FileLock
|
||||
from tqdm import tqdm
|
||||
|
||||
from rdagent.core.evaluation import EvaluableObj, Evaluator, Feedback
|
||||
from rdagent.core.evolving_framework import EvolvableSubjects, EvolvingStrategy, EvoStep
|
||||
from rdagent.core.evolving_framework import (
|
||||
EvolvableSubjects,
|
||||
EvolvingStrategy,
|
||||
EvoStep,
|
||||
IterEvaluator,
|
||||
RAGStrategy,
|
||||
)
|
||||
from rdagent.core.exception import EvaluatorDidNotTerminateError
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
|
||||
ASpecificEvaluator = TypeVar("ASpecificEvaluator", bound=Evaluator)
|
||||
@@ -26,22 +33,46 @@ class EvoAgent(ABC, Generic[ASpecificEvaluator, ASpecificEvolvableSubjects]):
|
||||
def multistep_evolve(
|
||||
self,
|
||||
evo: ASpecificEvolvableSubjects,
|
||||
eva: ASpecificEvaluator | Feedback,
|
||||
eva: ASpecificEvaluator,
|
||||
) -> Generator[ASpecificEvolvableSubjects, None, None]:
|
||||
"""
|
||||
yield EvolvableSubjects for caller for easier process control and logging.
|
||||
"""
|
||||
|
||||
|
||||
class RAGEvaluator(Evaluator):
|
||||
class RAGEvaluator(IterEvaluator):
|
||||
|
||||
@abstractmethod
|
||||
def evaluate(
|
||||
def evaluate_iter(
|
||||
self,
|
||||
eo: EvaluableObj,
|
||||
queried_knowledge: object = None,
|
||||
) -> Feedback:
|
||||
raise NotImplementedError
|
||||
queried_knowledge: object | None = None,
|
||||
evolving_trace: list[EvoStep] | None = None,
|
||||
) -> Generator[Feedback, EvaluableObj | None, Feedback]:
|
||||
"""
|
||||
|
||||
1) It will yield a evaluation for each implement part and yield the feedback for that part.
|
||||
2) And finally, it will get the summarize all the feedback and return a overall feedback.
|
||||
|
||||
Sending a None feedback will stop the evaluation chain and just return the overall feedback.
|
||||
|
||||
Assumptions:
|
||||
- The evaluation process will make modifications on evo in-place.
|
||||
|
||||
A typical implementation of this method is:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
evo = yield Feedback() # it will receive the evo first, so the first yield is for get the sent evo instead of generate useful feedback
|
||||
assert evo is not None
|
||||
for partial_eval_func in self.evaluate_func_iter():
|
||||
partial_fb = partial_eval_func(evo, queried_knowledge, evolving_trace)
|
||||
# return the partial feedback and receive the evolved solution for next iteration
|
||||
yield partial_fb
|
||||
|
||||
final_fb = get_final_fb(...)
|
||||
return final_fb
|
||||
|
||||
"""
|
||||
|
||||
|
||||
class RAGEvoAgent(EvoAgent[RAGEvaluator, ASpecificEvolvableSubjects], Generic[ASpecificEvolvableSubjects]):
|
||||
@@ -50,27 +81,66 @@ class RAGEvoAgent(EvoAgent[RAGEvaluator, ASpecificEvolvableSubjects], Generic[AS
|
||||
self,
|
||||
max_loop: int,
|
||||
evolving_strategy: EvolvingStrategy,
|
||||
rag: Any,
|
||||
rag: RAGStrategy,
|
||||
*,
|
||||
with_knowledge: bool = False,
|
||||
with_feedback: bool = True,
|
||||
knowledge_self_gen: bool = False,
|
||||
enable_filelock: bool = False,
|
||||
filelock_path: str | None = None,
|
||||
stop_eval_chain_on_fail: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Initialize a Retrieval-Augmented Generation (RAG) based evolutionary agent.
|
||||
|
||||
Args:
|
||||
max_loop (int): Maximum number of evolution loops to execute.
|
||||
evolving_strategy (EvolvingStrategy): Strategy defining how the subjects evolve each step.
|
||||
rag (RAGStrategy): Retrieval-Augmented Generation strategy instance used for knowledge querying and/or creation.
|
||||
with_knowledge (bool, optional): If True, retrieves knowledge from RAG for each evolution step. Defaults to False.
|
||||
knowledge_self_gen (bool, optional): If True, enable RAG to load, generate, dump new knowledge from evolving trace. Defaults to False.
|
||||
enable_filelock (bool, optional): If True, enables file-based lock when accessing/modifying the RAG knowledge base. Defaults to False.
|
||||
filelock_path (str | None, optional): Path to the lock file when enable_filelock is True. Defaults to None.
|
||||
|
||||
This class coordinates the multi-step evolution process with optional:
|
||||
- Knowledge retrieval before evolving.
|
||||
- Feedback collection after evolving.
|
||||
- Self-generation and persisting of knowledge base updates.
|
||||
|
||||
Evolving trace is maintained across steps for adaptive strategies and knowledge generation.
|
||||
"""
|
||||
super().__init__(max_loop, evolving_strategy)
|
||||
self.rag = rag
|
||||
self.evolving_trace: list[EvoStep[ASpecificEvolvableSubjects]] = []
|
||||
self.with_knowledge = with_knowledge
|
||||
self.with_feedback = with_feedback
|
||||
self.knowledge_self_gen = knowledge_self_gen
|
||||
self.enable_filelock = enable_filelock
|
||||
self.filelock_path = filelock_path
|
||||
self.stop_eval_chain_on_fail = stop_eval_chain_on_fail
|
||||
|
||||
def _get_overall_feedback(
|
||||
self,
|
||||
eva_iter: Generator[Feedback, EvaluableObj | None, Feedback],
|
||||
evo: EvolvableSubjects,
|
||||
eval_failed_happened: bool,
|
||||
) -> Feedback:
|
||||
"""get overall feedback from eva_iter"""
|
||||
try:
|
||||
if self.stop_eval_chain_on_fail and eval_failed_happened:
|
||||
fb = eva_iter.send(
|
||||
None,
|
||||
) # send the signal to skip the rest partial evaluation and return the overall feedback directly
|
||||
else:
|
||||
fb = eva_iter.send(evo)
|
||||
if not fb:
|
||||
eval_failed_happened = True
|
||||
raise EvaluatorDidNotTerminateError
|
||||
except StopIteration as e:
|
||||
return cast("Feedback", e.value)
|
||||
|
||||
def multistep_evolve(
|
||||
self,
|
||||
evo: ASpecificEvolvableSubjects,
|
||||
eva: RAGEvaluator | Feedback,
|
||||
eva: RAGEvaluator,
|
||||
) -> Generator[ASpecificEvolvableSubjects, None, None]:
|
||||
for evo_loop_id in tqdm(range(self.max_loop), "Implementing"):
|
||||
with logger.tag(f"evo_loop_{evo_loop_id}"):
|
||||
@@ -80,22 +150,35 @@ class RAGEvoAgent(EvoAgent[RAGEvaluator, ASpecificEvolvableSubjects], Generic[AS
|
||||
# TODO: Putting the evolving trace in here doesn't actually work
|
||||
queried_knowledge = self.rag.query(evo, self.evolving_trace)
|
||||
|
||||
# 2. evolve
|
||||
evo = self.evolving_strategy.evolve(
|
||||
# 2. evolve:
|
||||
# A compelete solution of an evo can be break down into multiple evolving steps.
|
||||
# Each evolving step can be evaluated separately.
|
||||
# Assumptions:
|
||||
# - if we want to stop on some point of the implementation, we must have a according evaluator (Otherwise, It is meaningless to stop)
|
||||
evo_iter = self.evolving_strategy.evolve_iter(
|
||||
evo=evo,
|
||||
evolving_trace=self.evolving_trace,
|
||||
queried_knowledge=queried_knowledge,
|
||||
)
|
||||
eva_iter = eva.evaluate_iter(
|
||||
evolving_trace=self.evolving_trace,
|
||||
queried_knowledge=queried_knowledge,
|
||||
)
|
||||
next(eva_iter) # kick off the first iteration
|
||||
eval_failed_happened = False
|
||||
for evolved_evo in evo_iter:
|
||||
step_feedback = eva_iter.send(evolved_evo)
|
||||
if not step_feedback:
|
||||
eval_failed_happened = True
|
||||
if self.stop_eval_chain_on_fail:
|
||||
break
|
||||
overall_feedback = self._get_overall_feedback(eva_iter, evolved_evo, eval_failed_happened)
|
||||
|
||||
# 3. Pack evolve results
|
||||
es = EvoStep[ASpecificEvolvableSubjects](evo, queried_knowledge)
|
||||
es = EvoStep[ASpecificEvolvableSubjects](evolved_evo, queried_knowledge, overall_feedback)
|
||||
|
||||
# 4. Evaluation
|
||||
if self.with_feedback:
|
||||
es.feedback = (
|
||||
eva if isinstance(eva, Feedback) else eva.evaluate(evo, queried_knowledge=queried_knowledge)
|
||||
)
|
||||
logger.log_object(es.feedback, tag="evolving feedback")
|
||||
logger.log_object(es.feedback, tag="evolving feedback")
|
||||
|
||||
# 5. update trace
|
||||
self.evolving_trace.append(es)
|
||||
@@ -110,6 +193,6 @@ class RAGEvoAgent(EvoAgent[RAGEvaluator, ASpecificEvolvableSubjects], Generic[AS
|
||||
yield evo # yield the control to caller for process control and logging.
|
||||
|
||||
# 7. check if all tasks are completed
|
||||
if self.with_feedback and es.feedback is not None and es.feedback.finished():
|
||||
if es.feedback is not None and es.feedback.finished():
|
||||
logger.info("All tasks in evolving subject have been completed.")
|
||||
break
|
||||
|
||||
@@ -2,10 +2,11 @@ from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Generator
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, Generic, TypeVar
|
||||
|
||||
from rdagent.core.evaluation import EvaluableObj
|
||||
from rdagent.core.evaluation import EvaluableObj, Evaluator
|
||||
from rdagent.core.knowledge_base import KnowledgeBase
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -62,19 +63,78 @@ class EvolvingStrategy(ABC, Generic[ASpecificEvolvableSubjects]):
|
||||
self.scen = scen
|
||||
|
||||
@abstractmethod
|
||||
def evolve(
|
||||
def evolve_iter(
|
||||
self,
|
||||
*evo: ASpecificEvolvableSubjects,
|
||||
evolving_trace: list[EvoStep[ASpecificEvolvableSubjects]] | None = None,
|
||||
evo: ASpecificEvolvableSubjects,
|
||||
queried_knowledge: QueriedKnowledge | None = None,
|
||||
**kwargs: Any,
|
||||
) -> ASpecificEvolvableSubjects:
|
||||
"""The evolving trace is a list of (evolvable_subjects, feedback) ordered
|
||||
evolving_trace: list[EvoStep] | None = None,
|
||||
) -> Generator[ASpecificEvolvableSubjects, None, None]:
|
||||
"""
|
||||
The evolving trace is a list of (evolvable_subjects, feedback) ordered
|
||||
according to the time.
|
||||
|
||||
The reason why the parameter is important for the evolving.
|
||||
- evolving_trace: the historical feedback is important.
|
||||
- queried_knowledge: queried knowledge
|
||||
|
||||
Assumptions:
|
||||
- The evolving process will make modifications in-place. So the yield evo and the parameter evo are the same object!!!!
|
||||
|
||||
|
||||
Typical implementation of this method is:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
for evolve_function in self.evolve_func_iter():
|
||||
yield evolve_function(evo=evo, queried_knowledge=queried_knowledge, evolving_trace=evolving_trace)
|
||||
# evolve_function will return a partial evolved solution.
|
||||
"""
|
||||
|
||||
|
||||
class IterEvaluator(Evaluator):
|
||||
"""
|
||||
Some evolving implementation (i.e. evolve_iter) will iteratively implement partial solutions before a complete final solution.
|
||||
|
||||
According to that strategy, we have iterative evaluation
|
||||
"""
|
||||
|
||||
def evaluate(self, eo: EvaluableObj) -> Feedback:
|
||||
"""
|
||||
Default implementation that runs evaluate_iter to completion.
|
||||
Iterative evaluators can override this for custom behavior,
|
||||
or just implement evaluate_iter for standard iteration.
|
||||
"""
|
||||
gen = self.evaluate_iter()
|
||||
next(gen) # Kick off the generator
|
||||
try:
|
||||
return gen.send(eo)
|
||||
except StopIteration as e:
|
||||
return e.value # type: ignore[no-any-return]
|
||||
|
||||
@abstractmethod
|
||||
def evaluate_iter(self) -> Generator[Feedback, EvaluableObj | None, Feedback]:
|
||||
"""
|
||||
|
||||
1) It will yield a evaluation for each implement part and yield the feedback for that part.
|
||||
2) And finally, it will get the summarize all the feedback and return a overall feedback.
|
||||
|
||||
Sending a None feedback will stop the evaluation chain and just return the overall feedback.
|
||||
|
||||
A typical implementation of this method is:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
evo = yield Feedback() # it will receive the evo first, so the first yield is for get the sent evo instead of generate useful feedback
|
||||
assert evo is not None
|
||||
for partial_eval_func in self.evaluate_func_iter():
|
||||
partial_fb = partial_eval_func(evo)
|
||||
# return the partial feedback and receive the evolved solution for next iteration
|
||||
evo_next_iter = yield partial_fb
|
||||
evo = evo_next_iter
|
||||
|
||||
final_fb = get_final_fb(...)
|
||||
return final_fb
|
||||
|
||||
"""
|
||||
|
||||
|
||||
@@ -98,7 +158,7 @@ class RAGStrategy(ABC, Generic[ASpecificEvolvableSubjects]):
|
||||
evo: ASpecificEvolvableSubjects,
|
||||
evolving_trace: list[EvoStep],
|
||||
**kwargs: Any,
|
||||
) -> QueriedKnowledge | None:
|
||||
) -> QueriedKnowledge:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
|
||||
@@ -10,6 +10,16 @@ class FormatError(WorkflowError):
|
||||
"""
|
||||
|
||||
|
||||
class CodeBlockParseError(FormatError):
|
||||
"""Raised when code block extraction fails after all strategies."""
|
||||
|
||||
def __init__(self, message: str, content: str, language: str) -> None:
|
||||
self.message = message
|
||||
self.content = content
|
||||
self.language = language
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class CoderError(WorkflowError):
|
||||
"""
|
||||
Exceptions raised when Implementing and running code.
|
||||
@@ -64,3 +74,9 @@ class PolicyError(Exception):
|
||||
"""
|
||||
Exceptions raised due to content management policy
|
||||
"""
|
||||
|
||||
|
||||
class EvaluatorDidNotTerminateError(RuntimeError):
|
||||
"""
|
||||
Evaluator generator did not terminate with a final Feedback.
|
||||
"""
|
||||
|
||||
@@ -242,6 +242,18 @@ class FBWorkspace(Workspace):
|
||||
target_file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
target_file_path.write_text(v)
|
||||
|
||||
def remove_files(self, file_names: str | list[str]) -> None:
|
||||
"""
|
||||
Remove specified files from the workspace.
|
||||
"""
|
||||
if isinstance(file_names, str):
|
||||
file_names = [file_names]
|
||||
for file_name in file_names:
|
||||
target_file_path = self.workspace_path / file_name
|
||||
if target_file_path.exists():
|
||||
target_file_path.unlink() # Unlink the file if it exists
|
||||
self.file_dict.pop(file_name, None) # Safely remove the key from file_dict
|
||||
|
||||
def get_files(self) -> list[Path]:
|
||||
"""
|
||||
Get the environment description.
|
||||
@@ -264,6 +276,11 @@ class FBWorkspace(Workspace):
|
||||
"""
|
||||
Load the workspace from the file_dict
|
||||
"""
|
||||
# NOTE: this is a deprecated method, use inject_from_workspace instead
|
||||
# TODO: remove this method; it is only for compatibility with old codes
|
||||
self.inject_from_workspace(workspace)
|
||||
|
||||
def inject_from_workspace(self, workspace: FBWorkspace) -> None:
|
||||
for name, code in workspace.file_dict.items():
|
||||
self.inject_files(**{name: code})
|
||||
|
||||
@@ -292,7 +309,7 @@ class FBWorkspace(Workspace):
|
||||
Before each execution, make sure to prepare and inject code.
|
||||
"""
|
||||
result = self.run(env, entry)
|
||||
return result.get_truncated_stdout() # NOTE: truncating just for aligning with the old code.
|
||||
return result.stdout # NOTE: truncating just for aligning with the old code.
|
||||
|
||||
def run(self, env: Env, entry: str) -> EnvResult:
|
||||
"""
|
||||
|
||||
+92
-12
@@ -96,13 +96,13 @@ class ExperimentFeedback(Feedback):
|
||||
class HypothesisFeedback(ExperimentFeedback):
|
||||
def __init__(
|
||||
self,
|
||||
observations: str,
|
||||
hypothesis_evaluation: str,
|
||||
new_hypothesis: str,
|
||||
reason: str,
|
||||
*,
|
||||
code_change_summary: str | None = None,
|
||||
decision: bool,
|
||||
code_change_summary: str = "",
|
||||
*,
|
||||
observations: str | None = None,
|
||||
hypothesis_evaluation: str | None = None,
|
||||
new_hypothesis: str | None = None,
|
||||
eda_improvement: str | None = None,
|
||||
acceptable: bool | None = None,
|
||||
) -> None:
|
||||
@@ -118,10 +118,18 @@ class HypothesisFeedback(ExperimentFeedback):
|
||||
self.acceptable = acceptable
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"""{super().__str__()}
|
||||
Observations: {self.observations}
|
||||
Hypothesis Evaluation: {self.hypothesis_evaluation}
|
||||
New Hypothesis: {self.new_hypothesis}"""
|
||||
upper_str = f"""{super().__str__()}"""
|
||||
if self.observations is not None:
|
||||
upper_str += f"\nObservations: {self.observations}"
|
||||
if self.hypothesis_evaluation is not None:
|
||||
upper_str += f"\nHypothesis Evaluation: {self.hypothesis_evaluation}"
|
||||
if self.new_hypothesis is not None:
|
||||
upper_str += f"\nNew Hypothesis: {self.new_hypothesis}"
|
||||
if self.eda_improvement is not None:
|
||||
upper_str += f"\nEDA Improvement: {self.eda_improvement}"
|
||||
if self.acceptable is not None:
|
||||
upper_str += f"\nOverall Acceptable: {self.acceptable}"
|
||||
return upper_str
|
||||
|
||||
|
||||
ASpecificScen = TypeVar("ASpecificScen", bound=Scenario)
|
||||
@@ -131,6 +139,7 @@ ASpecificKB = TypeVar("ASpecificKB", bound=KnowledgeBase)
|
||||
class Trace(Generic[ASpecificScen, ASpecificKB]):
|
||||
NodeType = tuple[Experiment, ExperimentFeedback] # Define NodeType as a new type representing the tuple
|
||||
NEW_ROOT: tuple = ()
|
||||
SEL_LATEST_SOTA: tuple = (-1,) # select the SOTA experiment in latest node
|
||||
|
||||
def __init__(self, scen: ASpecificScen, knowledge_base: ASpecificKB | None = None) -> None:
|
||||
self.scen: ASpecificScen = scen
|
||||
@@ -160,7 +169,9 @@ class Trace(Generic[ASpecificScen, ASpecificKB]):
|
||||
|
||||
# TODO: self.hist is 2-tuple now, remove hypothesis from it, change old code for this later.
|
||||
self.knowledge_base: ASpecificKB | None = knowledge_base
|
||||
self.current_selection: tuple[int, ...] = (-1,)
|
||||
|
||||
# The next expending point of the selection. Set it as a state of the trace will make
|
||||
self.current_selection: tuple[int, ...] = self.SEL_LATEST_SOTA
|
||||
|
||||
def get_sota_hypothesis_and_experiment(self) -> tuple[Hypothesis | None, Experiment | None]:
|
||||
"""Access the last experiment result, sub-task, and the corresponding hypothesis."""
|
||||
@@ -240,6 +251,70 @@ class Trace(Generic[ASpecificScen, ASpecificKB]):
|
||||
|
||||
return ancestors
|
||||
|
||||
def sync_dag_parent_and_hist(
|
||||
self,
|
||||
exp_and_fb: NodeType,
|
||||
cur_loop_id: int,
|
||||
) -> None:
|
||||
"""
|
||||
Adding corresponding parent index to the dag_parent when the hist is going to be changed.
|
||||
Should be called when the hist is changed.
|
||||
"""
|
||||
# Prioritize local_selection from the experiment if available
|
||||
exp = exp_and_fb[0]
|
||||
selection = getattr(exp, "local_selection", None)
|
||||
if selection is None:
|
||||
selection = self.get_current_selection()
|
||||
|
||||
if len(self.hist) == 0 or len(selection) == 0:
|
||||
# the node we are going to add is the first node of hist / root node of a new sub-trace
|
||||
self.dag_parent.append(self.NEW_ROOT)
|
||||
|
||||
else:
|
||||
current_node_idx = selection[0]
|
||||
|
||||
if current_node_idx == -1:
|
||||
# the current selection is the latest one
|
||||
current_node_idx = len(self.hist) - 1
|
||||
|
||||
self.dag_parent.append((current_node_idx,))
|
||||
self.hist.append(exp_and_fb)
|
||||
self.idx2loop_id[len(self.hist) - 1] = cur_loop_id
|
||||
|
||||
def get_children(self, parent_idx: int | None = None) -> list[NodeType]:
|
||||
"""
|
||||
Get all children nodes for a given parent index.
|
||||
If parent_idx is None, returns the root nodes (experiments starting from scratch).
|
||||
"""
|
||||
target_parents = (parent_idx,) if parent_idx is not None else self.NEW_ROOT
|
||||
children = []
|
||||
for i, parents in enumerate(self.dag_parent):
|
||||
if parents == target_parents and i < len(self.hist):
|
||||
children.append(self.hist[i])
|
||||
return children
|
||||
|
||||
def get_sota_experiment(self, node_id: int | None = None) -> Experiment | None:
|
||||
"""
|
||||
Get the SOTA experiment from the trace by traversing ancestors backwards from node_id.
|
||||
"""
|
||||
# NOTE: it is first used in the finetune scenario.
|
||||
if node_id is None:
|
||||
selection = self.get_current_selection()
|
||||
if self.is_selection_new_tree(selection):
|
||||
return None
|
||||
node_id = selection[0]
|
||||
|
||||
if node_id == -1:
|
||||
if not self.hist:
|
||||
return None
|
||||
node_id = len(self.hist) - 1
|
||||
|
||||
ancestors = self.get_parents(node_id)
|
||||
for i in reversed(ancestors):
|
||||
if self.hist[i][1].decision:
|
||||
return self.hist[i][0]
|
||||
return None
|
||||
|
||||
|
||||
class CheckpointSelector:
|
||||
"""
|
||||
@@ -298,7 +373,7 @@ class ExpGen(ABC):
|
||||
self.scen = scen
|
||||
|
||||
@abstractmethod
|
||||
def gen(self, trace: Trace, plan: ExperimentPlan | None = None) -> Experiment:
|
||||
def gen(self, trace: Trace) -> Experiment:
|
||||
"""
|
||||
Generate the experiment based on the trace.
|
||||
Planning is part of gen, but since we may support multi-stage planning,
|
||||
@@ -379,7 +454,12 @@ class Experiment2Feedback(ABC):
|
||||
self.scen = scen
|
||||
|
||||
@abstractmethod
|
||||
def generate_feedback(self, exp: Experiment, trace: Trace) -> ExperimentFeedback:
|
||||
def generate_feedback(
|
||||
self,
|
||||
exp: Experiment,
|
||||
trace: Trace,
|
||||
exception: Exception | None = None,
|
||||
) -> ExperimentFeedback:
|
||||
"""
|
||||
The `exp` should be executed and the results should be included, as well as the comparison
|
||||
between previous results (done by LLM).
|
||||
|
||||
@@ -16,7 +16,7 @@ from typing import Any, Callable, List, Optional, Tuple, Type, Union, cast
|
||||
import pytz
|
||||
from pydantic import BaseModel, TypeAdapter
|
||||
|
||||
from rdagent.core.exception import PolicyError
|
||||
from rdagent.core.exception import CodeBlockParseError, PolicyError
|
||||
from rdagent.core.utils import LLM_CACHE_SEED_GEN, SingletonBaseClass
|
||||
from rdagent.log import LogColors
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
@@ -136,6 +136,62 @@ class JSONParser:
|
||||
return json.dumps(obj)
|
||||
|
||||
|
||||
class CodeBlockParser:
|
||||
"""
|
||||
Generic code block extractor supporting multiple languages.
|
||||
Raises CodeBlockParseError on extraction failure to trigger retry.
|
||||
"""
|
||||
|
||||
SUPPORTED_LANGUAGES = {
|
||||
"python": ["python", "py", "python3", "Python", "Py"],
|
||||
"yaml": ["yaml", "yml"],
|
||||
}
|
||||
|
||||
def __init__(self, language: str = "python", fallback_to_raw: bool = False) -> None:
|
||||
"""
|
||||
Args:
|
||||
language: Target language type (python, yaml, etc.)
|
||||
fallback_to_raw: If True, return raw content when extraction fails.
|
||||
If False (default), raise CodeBlockParseError to trigger retry.
|
||||
"""
|
||||
self.language = language.lower()
|
||||
self.fallback_to_raw = fallback_to_raw
|
||||
self._lang_aliases = self._get_language_aliases(self.language)
|
||||
|
||||
def _get_language_aliases(self, language: str) -> List[str]:
|
||||
"""Get all possible aliases for the language."""
|
||||
for lang, aliases in self.SUPPORTED_LANGUAGES.items():
|
||||
if language in [lang] + aliases:
|
||||
return [lang] + aliases
|
||||
return [language]
|
||||
|
||||
def parse(self, content: str) -> str:
|
||||
"""
|
||||
Parse content and extract code block with exact language tag.
|
||||
|
||||
Returns:
|
||||
Extracted code string.
|
||||
|
||||
Raises:
|
||||
CodeBlockParseError: When extraction fails and fallback_to_raw=False.
|
||||
"""
|
||||
# Match code block with exact language tag (```python, ```yaml, etc.)
|
||||
for alias in self._lang_aliases:
|
||||
pattern = rf"```{alias}\s*\n(.*?)\n```"
|
||||
match = re.search(pattern, content, re.DOTALL | re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
|
||||
if self.fallback_to_raw:
|
||||
return content.strip()
|
||||
|
||||
raise CodeBlockParseError(
|
||||
message=f"Failed to extract {self.language} code block",
|
||||
content=content,
|
||||
language=self.language,
|
||||
)
|
||||
|
||||
|
||||
class SQliteLazyCache(SingletonBaseClass):
|
||||
def __init__(self, cache_location: str) -> None:
|
||||
super().__init__()
|
||||
@@ -267,7 +323,14 @@ class ChatSession:
|
||||
)
|
||||
end_time = datetime.now(pytz.timezone("Asia/Shanghai"))
|
||||
logger.log_object(
|
||||
{"user": user_prompt, "resp": response, "start": start_time, "end": end_time}, tag="debug_llm"
|
||||
{
|
||||
"system": self.system_prompt,
|
||||
"user": user_prompt,
|
||||
"resp": response,
|
||||
"start": start_time,
|
||||
"end": end_time,
|
||||
},
|
||||
tag="debug_llm",
|
||||
)
|
||||
|
||||
messages.append(
|
||||
@@ -568,6 +631,8 @@ class APIBackend(ABC):
|
||||
json_target_type: Optional[str] = None,
|
||||
add_json_in_prompt: bool = False,
|
||||
response_format: Optional[Union[dict, Type[BaseModel]]] = None,
|
||||
code_block_language: Optional[str] = None,
|
||||
code_block_fallback: bool = False,
|
||||
**kwargs: Any,
|
||||
) -> str:
|
||||
"""
|
||||
@@ -609,6 +674,14 @@ class APIBackend(ABC):
|
||||
**kwargs,
|
||||
)
|
||||
all_response += response
|
||||
|
||||
# Handle litellm bug: finish_reason='stop' but code block not closed
|
||||
# TODO: this is a temporary solution, and should be removed when litellm is fixed.
|
||||
if finish_reason == "stop" and code_block_language:
|
||||
if all_response.count("```") % 2 == 1: # Odd count = unclosed code block
|
||||
logger.warning("Detected unclosed code block with finish_reason='stop', treating as truncated")
|
||||
finish_reason = "length"
|
||||
|
||||
if finish_reason is None or finish_reason != "length":
|
||||
break # we get a full response now.
|
||||
new_messages.append({"role": "assistant", "content": response})
|
||||
@@ -617,13 +690,14 @@ class APIBackend(ABC):
|
||||
|
||||
# 2) refine the response and return
|
||||
if LLM_SETTINGS.reasoning_think_rm:
|
||||
# Strategy 1: Try to match complete <think>...</think> pattern
|
||||
match = re.search(r"<think>(.*?)</think>(.*)", all_response, re.DOTALL)
|
||||
# Only remove <think>...</think> if it appears at the beginning of the response
|
||||
# Strategy 1: Try to match complete <think>...</think> pattern at the start
|
||||
match = re.match(r"\s*<think>(.*?)</think>(.*)", all_response, re.DOTALL)
|
||||
if match:
|
||||
_, all_response = match.groups()
|
||||
else:
|
||||
# Strategy 2: If no complete match, try to match only </think>
|
||||
match = re.search(r"</think>(.*)", all_response, re.DOTALL)
|
||||
# Strategy 2: If no complete match, try to match only </think> at the start
|
||||
match = re.match(r"\s*</think>(.*)", all_response, re.DOTALL)
|
||||
if match:
|
||||
all_response = match.group(1)
|
||||
# If no match at all, keep original content
|
||||
@@ -636,6 +710,14 @@ class APIBackend(ABC):
|
||||
# deepseek will enter this branch
|
||||
TypeAdapter(json_target_type).validate_json(all_response)
|
||||
|
||||
# 4) code block extraction
|
||||
if code_block_language:
|
||||
code_parser = CodeBlockParser(
|
||||
language=code_block_language,
|
||||
fallback_to_raw=code_block_fallback,
|
||||
)
|
||||
all_response = code_parser.parse(all_response)
|
||||
|
||||
if response_format is not None:
|
||||
if not isinstance(response_format, dict) and issubclass(response_format, BaseModel):
|
||||
# It may raise TypeError if initialization fails
|
||||
|
||||
@@ -204,9 +204,13 @@ class LiteLLMAPIBackend(APIBackend):
|
||||
logger.info(
|
||||
f"Current Cost: ${float(cost):.10f}; Accumulated Cost: ${float(ACC_COST):.10f}; {finish_reason=}",
|
||||
)
|
||||
|
||||
prompt_tokens = token_counter(model=model, messages=messages)
|
||||
completion_tokens = token_counter(model=model, text=content)
|
||||
try:
|
||||
prompt_tokens = token_counter(model=model, messages=messages)
|
||||
completion_tokens = token_counter(model=model, text=content)
|
||||
except ValueError as e:
|
||||
logger.warning(f"Token counting failed for model {model}: {e}. Skip token statistics.")
|
||||
prompt_tokens = 0
|
||||
completion_tokens = 0
|
||||
logger.log_object(
|
||||
{
|
||||
"model": model,
|
||||
|
||||
@@ -61,6 +61,7 @@ class LLMSettings(ExtendedBaseSettings):
|
||||
|
||||
# Chat configs
|
||||
openai_api_key: str = "" # TODO: simplify the key design.
|
||||
openai_api_base: str = ""
|
||||
chat_openai_api_key: str | None = None
|
||||
chat_openai_base_url: str | None = None #
|
||||
chat_azure_api_base: str = ""
|
||||
|
||||
@@ -100,7 +100,7 @@ class DSRunnerEvaluator(CoSTEEREvaluator):
|
||||
|
||||
# execute workflow
|
||||
result = implementation.run(env=env, entry="python -m coverage run main.py")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
execute_ret_code = result.exit_code
|
||||
implementation.running_info.running_time = result.running_time
|
||||
|
||||
|
||||
@@ -88,6 +88,7 @@ class DataScienceRDLoop(RDLoop):
|
||||
# NOTE: we move the DataScienceRDLoop here to be easier to be imported
|
||||
skip_loop_error = (CoderError, RunnerError)
|
||||
withdraw_loop_error = (PolicyError,)
|
||||
skip_loop_error_stepname = "record"
|
||||
|
||||
# when using more advanced proposals(merged, parallel, etc.), we provide a default exp_gen for convinience.
|
||||
default_exp_gen: type[ExpGen] = DSProposalV2ExpGen
|
||||
|
||||
@@ -477,7 +477,7 @@ class ValidationSelector(SOTAexpSelector):
|
||||
result = ws.run(
|
||||
env=env, entry=f"python {script_type}.py --cache-buster={time.time()}"
|
||||
) # Do not cache the result
|
||||
stdout = re.sub(r"^chmod:.*\n?", "", result.get_truncated_stdout(), flags=re.MULTILINE)
|
||||
stdout = re.sub(r"^chmod:.*\n?", "", result.stdout, flags=re.MULTILINE)
|
||||
|
||||
if result.exit_code == 0:
|
||||
logger.info(f"Successfully generated and ran {script_type}.py.")
|
||||
@@ -487,7 +487,7 @@ class ValidationSelector(SOTAexpSelector):
|
||||
running_timeout_period=DS_RD_SETTING.full_timeout,
|
||||
)
|
||||
result = ws.run(env=env, entry=f"python reference_code.py")
|
||||
stdout = re.sub(r"^chmod:.*\n?", "", result.get_truncated_stdout(), flags=re.MULTILINE)
|
||||
stdout = re.sub(r"^chmod:.*\n?", "", result.stdout, flags=re.MULTILINE)
|
||||
if result.exit_code == 0:
|
||||
# move submission.csv to mock_folder
|
||||
if Path(ws.workspace_path / "submission.csv").exists():
|
||||
@@ -559,7 +559,7 @@ def process_experiment(
|
||||
env.conf.running_timeout_period = DS_RD_SETTING.debug_timeout
|
||||
result = ws.run(env=env, entry="python grade.py")
|
||||
if result.exit_code == 0:
|
||||
grade_stdout = re.sub(r"^chmod:.*\n?", "", result.get_truncated_stdout(), flags=re.MULTILINE)
|
||||
grade_stdout = re.sub(r"^chmod:.*\n?", "", result.stdout, flags=re.MULTILINE)
|
||||
logger.info(f"Ran grade.py for {competition}/{loop_id}; exit_code: {result.exit_code}")
|
||||
else:
|
||||
logger.warning(f"Skipping grading for {competition}/{loop_id} due to main.py execution failure.")
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
from .benchmark import get_benchmark_ranges, run_benchmark
|
||||
|
||||
__all__ = ["get_benchmark_ranges", "run_benchmark"]
|
||||
@@ -0,0 +1,397 @@
|
||||
"""
|
||||
Benchmark Evaluation using OpenCompass
|
||||
|
||||
Evaluator that runs OpenCompass in Docker to evaluate fine-tuned models on standard benchmarks.
|
||||
|
||||
Configure benchmark behavior via editting .env to cover default settings in conf.py:
|
||||
```
|
||||
FT_BENCHMARK_DATASETS='["aime25", "gsm8k"]'
|
||||
FT_BENCHMARK_NUM_RUNS=4
|
||||
FT_JUDGE_MODEL="gpt-4"
|
||||
FT_JUDGE_API_KEY="sk-xxx"
|
||||
FT_JUDGE_API_BASE="https://api.openai.com/v1"
|
||||
```
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_MODEL_PATH,
|
||||
get_benchmark_env,
|
||||
get_ft_env,
|
||||
get_workspace_prefix,
|
||||
is_docker_env,
|
||||
)
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_conf import LLM_SETTINGS
|
||||
from rdagent.scenarios.finetune.benchmark.data.adaptor import (
|
||||
BENCHMARK_CONFIG_DICT,
|
||||
BenchmarkConfig,
|
||||
)
|
||||
from rdagent.scenarios.finetune.benchmark.data.default import extract_error_samples
|
||||
from rdagent.scenarios.finetune.benchmark.merge.merge import (
|
||||
check_if_merging_needed,
|
||||
merge_model,
|
||||
)
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
def get_model_inference_config(base_model_name: str, gpu_count: int) -> dict:
|
||||
"""
|
||||
Load model inference configuration from YAML file.
|
||||
|
||||
Args:
|
||||
base_model_name: HuggingFace model name (e.g., "Qwen/Qwen3-8B")
|
||||
gpu_count: GPU count for tensor_parallel_size (from scenario.device_info)
|
||||
|
||||
Returns:
|
||||
dict: Merged configuration (model-specific overrides default)
|
||||
Uses exact match first, then longest prefix match, finally default only.
|
||||
"""
|
||||
config_data = yaml.safe_load(open(Path(__file__).parent / "configs" / "models.yaml", "r"))
|
||||
|
||||
default_config = config_data.get("default", {})
|
||||
models_config = config_data.get("models", {})
|
||||
|
||||
# 1. Exact match
|
||||
if base_model_name in models_config:
|
||||
model_specific = models_config[base_model_name]
|
||||
else:
|
||||
# 2. Prefix match - find longest matching prefix
|
||||
model_specific = {}
|
||||
best_match_len = 5
|
||||
for configured_model in models_config:
|
||||
if base_model_name.startswith(configured_model) and len(configured_model) > best_match_len:
|
||||
model_specific = models_config[configured_model]
|
||||
best_match_len = len(configured_model)
|
||||
|
||||
final_config = {**default_config, **model_specific}
|
||||
|
||||
# Handle auto tensor_parallel_size
|
||||
if final_config.get("tensor_parallel_size") == "auto":
|
||||
if gpu_count <= 0:
|
||||
final_config["tensor_parallel_size"] = 1
|
||||
else:
|
||||
# Round down to nearest power of 2
|
||||
power = 0
|
||||
while (1 << (power + 1)) <= gpu_count:
|
||||
power += 1
|
||||
final_config["tensor_parallel_size"] = 1 << power
|
||||
|
||||
return final_config
|
||||
|
||||
|
||||
def detect_model_type(model_path: str) -> bool:
|
||||
"""
|
||||
Detect whether the given model path corresponds to a LoRA adapter.
|
||||
|
||||
Returns:
|
||||
True if LoRA adapter, False otherwise.
|
||||
"""
|
||||
model_dir = Path(model_path)
|
||||
|
||||
# LoRA (llama-factory style)
|
||||
if (model_dir / "adapter_config.json").exists():
|
||||
return True
|
||||
|
||||
# Alternate LoRA file indicators
|
||||
for fname in ("adapter_model.bin", "adapter_model.safetensors"):
|
||||
if (model_dir / fname).exists():
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def run_benchmark(
|
||||
workspace_path: str,
|
||||
model_path: str,
|
||||
model_name: str,
|
||||
benchmark_name: str,
|
||||
gpu_count: int,
|
||||
test_range: Optional[str] = "[:100]",
|
||||
num_runs: int = 1,
|
||||
pass_k: Optional[List[int]] = None,
|
||||
max_error_samples: int = 10,
|
||||
result_subdir: str = "",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Run benchmark evaluation on a fine-tuned model.
|
||||
|
||||
Args:
|
||||
workspace_path: Path to workspace directory
|
||||
model_path: Path to fine-tuned model (supports full/LoRA auto-detection)
|
||||
model_name: HuggingFace model name
|
||||
benchmark_name: Benchmark dataset name (e.g., "aime25", "gsm8k")
|
||||
gpu_count: GPU count for tensor_parallel_size (from scenario.device_info)
|
||||
test_range: Python slice string for dataset sampling (e.g., "[:100]", "[-100:]").
|
||||
Negative indexing allows automatic adaptation to varying subset sizes.
|
||||
num_runs: Number of times to run each sample (default: 1)
|
||||
pass_k: Optional list of k values for pass@k evaluation (e.g., [1, 5, 10])
|
||||
max_error_samples: Maximum number of error samples to extract for feedback
|
||||
result_subdir: Subdirectory for results (e.g., "validation", "test")
|
||||
|
||||
Returns:
|
||||
Dict containing:
|
||||
- accuracy_summary: Dict mapping dataset -> {metric: value}, grouped by dataset
|
||||
- error_samples: List of error samples for feedback analysis
|
||||
"""
|
||||
# Load configurations
|
||||
benchmark_cfg: BenchmarkConfig = BENCHMARK_CONFIG_DICT[benchmark_name]
|
||||
dataset_imports = benchmark_cfg.dataset
|
||||
|
||||
# Auto download dependent data if configured on this benchmark
|
||||
if benchmark_cfg.download is not None:
|
||||
benchmark_cfg.download()
|
||||
|
||||
model_is_lora = detect_model_type(model_path)
|
||||
inference_config = get_model_inference_config(model_name, gpu_count)
|
||||
workspace_path = Path(workspace_path)
|
||||
|
||||
# Get environment first to determine path prefix
|
||||
env = get_benchmark_env()
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
is_docker = is_docker_env(env)
|
||||
|
||||
# Determine model paths based on environment type
|
||||
model_rel_path = Path(model_path).relative_to(workspace_path)
|
||||
adapter_path_in_env = Path(ws_prefix) / model_rel_path
|
||||
|
||||
if model_is_lora:
|
||||
if is_docker:
|
||||
# Docker: use /assets/models mount
|
||||
model_path_in_env = Path(FT_MODEL_PATH) / model_name
|
||||
else:
|
||||
# Conda: use actual file path
|
||||
model_path_in_env = Path(FT_RD_SETTING.file_path) / "models" / model_name
|
||||
lora_path_in_env = adapter_path_in_env
|
||||
|
||||
# Check if we need to merge the model (e.g. vLLM doesn't support LoRA with modules_to_save)
|
||||
if check_if_merging_needed(model_path):
|
||||
merged_model_dir_inside_env = Path(ws_prefix) / "merged_model"
|
||||
|
||||
# Create a temporary environment for merging (use FT env as it has peft/transformers)
|
||||
merge_env = get_ft_env()
|
||||
|
||||
merge_model(
|
||||
env=merge_env,
|
||||
workspace_path=workspace_path,
|
||||
base_model_path=str(model_path_in_env),
|
||||
adapter_path=str(lora_path_in_env),
|
||||
output_path=str(merged_model_dir_inside_env),
|
||||
)
|
||||
|
||||
# Switch to using the merged model
|
||||
model_path_in_env = merged_model_dir_inside_env
|
||||
model_is_lora = False
|
||||
lora_path_in_env = ""
|
||||
adapter_path_in_env = merged_model_dir_inside_env
|
||||
else:
|
||||
model_path_in_env = adapter_path_in_env
|
||||
lora_path_in_env = ""
|
||||
|
||||
# Prepare template variables (merge inference config from models.yaml)
|
||||
template_vars = {
|
||||
# Model configuration
|
||||
"model_abbr": f"ft-{benchmark_name}",
|
||||
"model_path": model_path_in_env,
|
||||
"is_lora": model_is_lora,
|
||||
"lora_path": lora_path_in_env,
|
||||
# Dataset configuration
|
||||
"dataset_imports": [dataset_imports],
|
||||
"test_range": test_range,
|
||||
"num_runs": num_runs,
|
||||
"pass_k": pass_k,
|
||||
"work_dir": adapter_path_in_env,
|
||||
# Merge all inference parameters from models.yaml (default + model-specific)
|
||||
**inference_config,
|
||||
}
|
||||
|
||||
# Override use_cot_postprocessor based on force_think_token setting
|
||||
# When force_think_token=false, we don't need the CoT postprocessor to extract answers
|
||||
if not FT_RD_SETTING.force_think_token:
|
||||
template_vars["use_cot_postprocessor"] = False
|
||||
|
||||
# Render Jinja2 template
|
||||
config_content = T("rdagent.scenarios.finetune.benchmark.configs.opencompass_template:template").r(**template_vars)
|
||||
|
||||
# Note: env was already created above via get_benchmark_env()
|
||||
|
||||
(workspace_path / "config.py").write_text(config_content)
|
||||
# Use result_subdir for validation/test separation
|
||||
if result_subdir:
|
||||
benchmark_work_dir = f"{ws_prefix}/benchmark_results/{result_subdir}"
|
||||
else:
|
||||
benchmark_work_dir = f"{ws_prefix}/benchmark_results"
|
||||
|
||||
# Logging
|
||||
logger.info(f"Running benchmark '{benchmark_name}' on model: {model_path}")
|
||||
logger.info(f"Base model: {model_name}, LoRA?: {model_is_lora}")
|
||||
logger.info(f"Workspace: {workspace_path}")
|
||||
logger.info(f"Benchmark work_dir: {benchmark_work_dir}")
|
||||
if test_range:
|
||||
logger.info(f"Dataset range: {test_range}")
|
||||
|
||||
# Environment variables
|
||||
env_vars = {
|
||||
"OC_JUDGE_MODEL": FT_RD_SETTING.judge_model or LLM_SETTINGS.chat_model,
|
||||
"OC_JUDGE_API_KEY": FT_RD_SETTING.judge_api_key or LLM_SETTINGS.openai_api_key,
|
||||
"OC_JUDGE_API_BASE": FT_RD_SETTING.judge_api_base or LLM_SETTINGS.openai_api_base,
|
||||
"OC_JUDGE_RETRY": str(FT_RD_SETTING.judge_retry),
|
||||
}
|
||||
|
||||
# Check if results already exist (skip re-running if cached)
|
||||
results_base = workspace_path / "benchmark_results"
|
||||
if result_subdir:
|
||||
results_base = results_base / result_subdir
|
||||
timestamped_dirs = sorted([d for d in results_base.glob("202*_*") if d.is_dir()], reverse=True)
|
||||
|
||||
if timestamped_dirs:
|
||||
logger.info(f"Found existing results in {timestamped_dirs[0].name}, skipping benchmark execution")
|
||||
else:
|
||||
# Run OpenCompass
|
||||
entry_cmd = f"opencompass {ws_prefix}/config.py --work-dir {benchmark_work_dir}"
|
||||
|
||||
result = env.run(
|
||||
entry=entry_cmd,
|
||||
local_path=str(workspace_path),
|
||||
env=env_vars,
|
||||
)
|
||||
|
||||
# Log execution immediately (for UI display)
|
||||
tag_prefix = "docker_run" if is_docker else "conda_run"
|
||||
logger.log_object(
|
||||
{
|
||||
"exit_code": result.exit_code,
|
||||
"stdout": (result.stdout or ""),
|
||||
"benchmark_name": benchmark_name,
|
||||
"model_path": str(model_path),
|
||||
"workspace_path": str(workspace_path),
|
||||
},
|
||||
tag=f"{tag_prefix}.Benchmark",
|
||||
)
|
||||
|
||||
# Check execution status
|
||||
if result.exit_code != 0:
|
||||
error_msg = result.stdout[-2000:] if result.stdout else "No output"
|
||||
raise RuntimeError(f"Benchmark execution failed (exit_code={result.exit_code})\n{error_msg}")
|
||||
|
||||
# Re-scan for timestamped directories after execution
|
||||
timestamped_dirs = sorted([d for d in results_base.glob("202*_*") if d.is_dir()], reverse=True)
|
||||
|
||||
# OpenCompass stores results in results/<model_name>/<dataset>.json
|
||||
results_subdir = timestamped_dirs[0] / "summary"
|
||||
|
||||
results_csv_path = sorted([f for f in results_subdir.rglob("*.csv")], reverse=True)[0]
|
||||
logger.info(f"Detailed results CSV: {results_csv_path.relative_to(results_base)}")
|
||||
|
||||
# Read CSV content for accuracy summary (grouped by dataset)
|
||||
df = pd.read_csv(results_csv_path)
|
||||
# Get score column (the model name column, e.g., 'api-chemcotbench')
|
||||
score_col = [c for c in df.columns if c not in ["dataset", "version", "metric", "mode"]][0]
|
||||
# Pivot to group by dataset, with metrics as columns (use pivot_table to handle duplicates)
|
||||
pivoted = df.pivot_table(index="dataset", columns="metric", values=score_col, aggfunc="first").to_dict("index")
|
||||
# Filter out NaN values (different datasets have different metrics)
|
||||
accuracy_summary = {ds: {k: v for k, v in metrics.items() if pd.notna(v)} for ds, metrics in pivoted.items()}
|
||||
|
||||
# Extract error samples for feedback
|
||||
error_samples = extract_error_samples(
|
||||
timestamped_dirs[0],
|
||||
max_samples=max_error_samples,
|
||||
)
|
||||
|
||||
# Log benchmark result for UI display
|
||||
# Use result_subdir to distinguish validation vs test in tag
|
||||
log_tag = f"benchmark_result.{result_subdir}" if result_subdir else "benchmark_result"
|
||||
logger.log_object(
|
||||
{
|
||||
"accuracy_summary": accuracy_summary,
|
||||
"error_samples": error_samples,
|
||||
"benchmark_name": benchmark_name,
|
||||
"split": result_subdir or "default", # validation, test, or default
|
||||
},
|
||||
tag=log_tag,
|
||||
)
|
||||
|
||||
return {
|
||||
"accuracy_summary": accuracy_summary,
|
||||
"error_samples": error_samples,
|
||||
}
|
||||
|
||||
|
||||
def get_benchmark_ranges() -> tuple[str, str]:
|
||||
"""Get validation and test range strings for benchmark evaluation.
|
||||
|
||||
Uses dynamic expressions that adapt to any dataset size:
|
||||
- For small datasets (<200): splits 50/50 to avoid overlap
|
||||
- For large datasets (>=200): takes 100 samples each
|
||||
|
||||
The expressions use OpenCompass's eval mechanism with index_list variable.
|
||||
|
||||
Returns:
|
||||
Tuple of (validation_range, test_range) - guaranteed non-overlapping:
|
||||
- validation: first min(100, 50%) samples
|
||||
- test: last min(100, 50%) samples
|
||||
"""
|
||||
return "[:min(100, len(index_list)//2)]", "[-min(100, len(index_list)//2):]"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
"""Test benchmark evaluation on Qwen3-1.7B with LoRA adapter."""
|
||||
# Configuration - Fill in your LoRA adapter path and model name
|
||||
LORA_ADAPTER_PATH = "/home/v-qizhengli/workspace/FT_workspace/gitignore_folder/B200/B200_FT_workspace/limo/train/b200_sweep_yamls/saves/qwen3-1.7b/lora_b200_lr1e-4_acc4/checkpoint-100"
|
||||
MODEL_NAME = "Qwen/Qwen3-1.7B"
|
||||
BENCHMARK = "aime25"
|
||||
GPU_COUNT = 1
|
||||
|
||||
print("=" * 80)
|
||||
print("Benchmark Evaluation Test")
|
||||
print("=" * 80)
|
||||
print(f"\nEnvironment: FT_JUDGE_API_KEY={'Set' if FT_RD_SETTING.judge_api_key else 'Not Set'}")
|
||||
print(f"Judge API Base: {FT_RD_SETTING.judge_api_base or 'Not Set'}")
|
||||
|
||||
if not Path(LORA_ADAPTER_PATH).exists():
|
||||
print(f"\nPlease set LORA_ADAPTER_PATH to a valid checkpoint directory")
|
||||
print(f"Current path does not exist: {LORA_ADAPTER_PATH}")
|
||||
exit(1)
|
||||
|
||||
print(f"\nModel: {MODEL_NAME}")
|
||||
print(f"Adapter: {LORA_ADAPTER_PATH}")
|
||||
print(f"Benchmark: {BENCHMARK}")
|
||||
print("-" * 80)
|
||||
|
||||
try:
|
||||
# Create FBWorkspace for test (auto-generates UUID workspace)
|
||||
test_task = Task(name=f"benchmark_test_{BENCHMARK}")
|
||||
test_workspace = FBWorkspace(target_task=test_task)
|
||||
test_workspace.prepare()
|
||||
|
||||
print(f"\nWorkspace: {test_workspace.workspace_path}")
|
||||
|
||||
result = run_benchmark(
|
||||
workspace_path=str(test_workspace.workspace_path),
|
||||
model_path=LORA_ADAPTER_PATH,
|
||||
model_name=MODEL_NAME,
|
||||
benchmark_name=BENCHMARK,
|
||||
gpu_count=GPU_COUNT,
|
||||
)
|
||||
|
||||
print("\nEvaluation completed!")
|
||||
print(f"Accuracy Summary: {result['accuracy_summary']}")
|
||||
print(f"Error Samples: {len(result['error_samples'])} samples")
|
||||
print(f"\nResults saved to: {test_workspace.workspace_path / 'benchmark_results'}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nEvaluation failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
@@ -0,0 +1,86 @@
|
||||
# Model Inference Parameters Configuration
|
||||
# Used by benchmark.py to determine inference settings for different models
|
||||
|
||||
# Default configuration (used when model is not explicitly listed)
|
||||
default:
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
batch_size: 16
|
||||
tensor_parallel_size: auto # Will be auto-determined based on GPU count
|
||||
gpu_memory_utilization: 0.9
|
||||
repetition_penalty: 1.0
|
||||
dtype: bfloat16
|
||||
enable_thinking: false
|
||||
use_cot_postprocessor: true # Enable CoT postprocessor to extract answer from <think>...</think>answer format
|
||||
|
||||
# Model-specific configurations (override default values)
|
||||
models:
|
||||
# Qwen3 series - support thinking mode and longer sequences
|
||||
"Qwen/Qwen3-8B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true # Qwen3-specific feature
|
||||
|
||||
"Qwen/Qwen3-32B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true
|
||||
|
||||
"Qwen/Qwen3-1.7B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true
|
||||
gpu_memory_utilization: 0.7 # It does not use too much GPU memory. But it is worth
|
||||
|
||||
# Qwen2.5 series - standard configuration with CoT postprocessor for fine-tuned models
|
||||
"Qwen/Qwen2.5-7B-Instruct":
|
||||
temperature: 0.0 # Greedy decoding for consistency
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
use_cot_postprocessor: true # Extract answer from CoT format after fine-tuning
|
||||
|
||||
"Qwen/Qwen2.5-32B-Instruct":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
|
||||
# Llama 3.1 series (128K context, 4K max output)
|
||||
"meta-llama/Llama-3.1-8B-Instruct":
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: 40
|
||||
max_seq_len: 32768 # 131072
|
||||
max_out_len: 4096
|
||||
|
||||
|
||||
# Mistral series
|
||||
"mistralai/Mistral-7B-Instruct-v0.3":
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: 50
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
|
||||
# DeepSeek series
|
||||
"deepseek-ai/deepseek-coder-33b-instruct":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 16384
|
||||
max_out_len: 4096
|
||||
@@ -0,0 +1,118 @@
|
||||
# Auto-generated OpenCompass Config for RD-Agent Benchmark
|
||||
# DO NOT EDIT MANUALLY - Generated by benchmark.py
|
||||
|
||||
template: |-
|
||||
from mmengine.config import read_base
|
||||
from opencompass.models import VLLMwithChatTemplate
|
||||
|
||||
# ==================== Dataset Import ====================
|
||||
with read_base():
|
||||
{% for dataset_module in dataset_imports %}
|
||||
from {{ dataset_module }} import *
|
||||
{% endfor %}
|
||||
|
||||
# Aggregate all dataset variables
|
||||
datasets = sum([v for k, v in locals().items() if (k == 'datasets' or k.endswith('_datasets')) and isinstance(v, list)], [])
|
||||
|
||||
# Apply dataset modifications
|
||||
for ds in datasets:
|
||||
{% if test_range %}
|
||||
# Apply dataset range (e.g., "[:100]" for validation, "[-100:]" for test)
|
||||
if 'reader_cfg' not in ds:
|
||||
ds['reader_cfg'] = {}
|
||||
ds['reader_cfg']['test_range'] = '{{ test_range }}'
|
||||
|
||||
# Sync to evaluator's dataset_cfg
|
||||
if 'eval_cfg' in ds and 'evaluator' in ds['eval_cfg']:
|
||||
evaluator = ds['eval_cfg']['evaluator']
|
||||
if isinstance(evaluator, dict) and 'dataset_cfg' in evaluator:
|
||||
if 'reader_cfg' not in evaluator['dataset_cfg']:
|
||||
evaluator['dataset_cfg']['reader_cfg'] = {}
|
||||
evaluator['dataset_cfg']['reader_cfg']['test_range'] = '{{ test_range }}'
|
||||
{% endif %}
|
||||
{% if num_runs and num_runs > 1 %}
|
||||
# Multiple runs (repeat each sample n times for averaging or pass@k)
|
||||
ds['n'] = {{ num_runs }}
|
||||
{% endif %}
|
||||
{% if pass_k %}
|
||||
# Pass@k evaluation
|
||||
ds['k'] = {{ pass_k }}
|
||||
{% endif %}
|
||||
pass
|
||||
|
||||
# ==================== Model Configuration ====================
|
||||
models = [
|
||||
dict(
|
||||
type=VLLMwithChatTemplate,
|
||||
abbr='{{ model_abbr }}',
|
||||
path='{{ model_path }}',
|
||||
model_kwargs=dict(
|
||||
tensor_parallel_size={{ tensor_parallel_size }},
|
||||
gpu_memory_utilization={{ gpu_memory_utilization }},
|
||||
trust_remote_code=True,
|
||||
dtype='{{ dtype }}',
|
||||
max_model_len={{ max_seq_len }},
|
||||
{% if is_lora %}
|
||||
enable_lora=True,
|
||||
max_lora_rank=64,
|
||||
max_cpu_loras=1,
|
||||
{% endif %}
|
||||
),
|
||||
{% if is_lora %}
|
||||
lora_path='{{ lora_path }}',
|
||||
{% endif %}
|
||||
max_seq_len={{ max_seq_len }},
|
||||
max_out_len={{ max_out_len }},
|
||||
batch_size={{ batch_size }},
|
||||
generation_kwargs=dict(
|
||||
temperature={{ temperature }},
|
||||
top_p={{ top_p }},
|
||||
top_k={{ top_k }},
|
||||
{% if repetition_penalty != 1.0 %}
|
||||
repetition_penalty={{ repetition_penalty }},
|
||||
{% endif %}
|
||||
),
|
||||
{% if enable_thinking %}
|
||||
chat_template_kwargs=dict(enable_thinking=True),
|
||||
{% endif %}
|
||||
{% if enable_thinking or use_cot_postprocessor %}
|
||||
pred_postprocessor=dict(type='extract-non-reasoning-content'),
|
||||
{% endif %}
|
||||
run_cfg=dict(
|
||||
num_gpus={{ tensor_parallel_size }},
|
||||
num_procs=1,
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
# ==================== Inference Configuration ====================
|
||||
infer = dict(
|
||||
partitioner=dict(
|
||||
type='NaivePartitioner',
|
||||
),
|
||||
runner=dict(
|
||||
type='LocalRunner',
|
||||
max_num_workers=16,
|
||||
task=dict(
|
||||
type='OpenICLInferTask',
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# ==================== Evaluation Configuration ====================
|
||||
eval = dict(
|
||||
partitioner=dict(
|
||||
type='NaivePartitioner',
|
||||
),
|
||||
runner=dict(
|
||||
type='LocalRunner',
|
||||
max_num_workers=16,
|
||||
task=dict(
|
||||
type='OpenICLEvalTask',
|
||||
dump_details=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# ==================== Work Directory ====================
|
||||
work_dir = '{{ work_dir }}'
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
Benchmark dataset configuration and data preparation adaptor for finetune benchmarks.
|
||||
|
||||
This module centralizes:
|
||||
- Mapping of benchmark names to OpenCompass dataset config import paths.
|
||||
- Optional dataset download / preparation hooks for benchmarks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Dict, Optional
|
||||
|
||||
from rdagent.scenarios.finetune.benchmark.data import financeiq_gen
|
||||
|
||||
DownloadFunc = Callable[[], None]
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkConfig:
|
||||
"""
|
||||
Configuration for a single benchmark.
|
||||
|
||||
Attributes:
|
||||
dataset: Import path for the dataset config in OpenCompass.
|
||||
download: Optional function to ensure the dataset is available (e.g. download from HF).
|
||||
"""
|
||||
|
||||
dataset: str
|
||||
download: Optional[DownloadFunc] = None
|
||||
|
||||
|
||||
# Mapping from benchmark_name -> benchmark configuration.
|
||||
BENCHMARK_CONFIG_DICT: Dict[str, BenchmarkConfig] = {
|
||||
# Math Reasoning Benchmarks
|
||||
"aime24": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.aime2024.aime2024_gen_17d799",
|
||||
),
|
||||
"aime25": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.aime2025.aime2025_cascade_eval_gen_5e9f4f",
|
||||
),
|
||||
"math": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.math.math_0shot_gen_393424",
|
||||
),
|
||||
# General Knowledge Benchmarks
|
||||
"mmlu": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.mmlu.mmlu_gen",
|
||||
),
|
||||
# Code Generation Benchmarks
|
||||
"humaneval": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.humaneval.humaneval_gen",
|
||||
),
|
||||
"mbpp": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.mbpp.mbpp_gen",
|
||||
),
|
||||
# PANORAMA - Patent Analysis Benchmarks (zero-shot)
|
||||
"panorama": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_gen",
|
||||
),
|
||||
"panorama_par4pc": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_par4pc_gen",
|
||||
),
|
||||
"panorama_pi4pc": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_pi4pc_gen",
|
||||
),
|
||||
"panorama_noc4pc": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_noc4pc_gen",
|
||||
),
|
||||
# PANORAMA - Patent Analysis Benchmarks (CoT)
|
||||
"panorama_par4pc_cot": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_par4pc_cot_gen",
|
||||
),
|
||||
"panorama_pi4pc_cot": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_pi4pc_cot_gen",
|
||||
),
|
||||
"panorama_noc4pc_cot": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.panorama.panorama_noc4pc_cot_gen",
|
||||
),
|
||||
# ChemCoTBench - Chemistry Reasoning Benchmarks
|
||||
"chemcotbench": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.chemcotbench.chemcotbench_gen",
|
||||
),
|
||||
"chemcotbench_mol_und": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.chemcotbench.chemcotbench_mol_und_gen",
|
||||
),
|
||||
"chemcotbench_mol_edit": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.chemcotbench.chemcotbench_mol_edit_gen",
|
||||
),
|
||||
"chemcotbench_mol_opt": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.chemcotbench.chemcotbench_mol_opt_gen",
|
||||
),
|
||||
"chemcotbench_reaction": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.chemcotbench.chemcotbench_reaction_gen",
|
||||
),
|
||||
# TableBench - Table Question Answering Benchmarks
|
||||
"tablebench_data_analysis": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.tablebench.tablebench_data_analysis_gen",
|
||||
),
|
||||
"tablebench_fact_checking": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.tablebench.tablebench_fact_checking_gen",
|
||||
),
|
||||
"tablebench_numerical_reasoning": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.tablebench.tablebench_numerical_reasoning_gen",
|
||||
),
|
||||
"tablebench_visualization": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.tablebench.tablebench_visualization_gen",
|
||||
),
|
||||
"tablebench_gen": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.tablebench.tablebench_gen",
|
||||
),
|
||||
# BioProBench
|
||||
"bioprobench_gen": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.bioprobench.bioprobench_gen",
|
||||
),
|
||||
"bioprobench_ord": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.bioprobench.bioprobench_ord",
|
||||
),
|
||||
"bioprobench_err": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.bioprobench.bioprobench_err",
|
||||
),
|
||||
"bioprobench_pqa": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.bioprobench.bioprobench_pqa",
|
||||
),
|
||||
# Native OpenCompass benchmarks
|
||||
"FinanceIQ_gen": BenchmarkConfig(
|
||||
dataset="opencompass.configs.datasets.FinanceIQ.FinanceIQ_gen_e0e6b5",
|
||||
download=financeiq_gen.download_financeiq_dataset,
|
||||
),
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
"""
|
||||
Error sample extraction from OpenCompass benchmark results.
|
||||
|
||||
This module provides a unified approach to extract error samples from various
|
||||
OpenCompass evaluator formats using both results and predictions directories.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
|
||||
# ============================================================================
|
||||
# Helper Functions
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _to_bool(value: Any) -> bool:
|
||||
"""
|
||||
Unified boolean conversion supporting multiple types.
|
||||
|
||||
Handles: list, str, bool, None, and other types.
|
||||
Key: [False] -> False, [True] -> True
|
||||
"""
|
||||
if value is None:
|
||||
return False
|
||||
if isinstance(value, list):
|
||||
return all(_to_bool(v) for v in value) if value else False
|
||||
if isinstance(value, str):
|
||||
return value.strip().upper() in ("A", "CORRECT", "TRUE", "YES", "1")
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _is_correct(sample: Dict) -> bool:
|
||||
"""
|
||||
Unified correctness check - returns True if sample is correct (should be skipped).
|
||||
|
||||
Checks fields in priority order from results directory.
|
||||
"""
|
||||
# Direct fields
|
||||
for field in ["cascade_correct", "correct", "is_correct", "exact_match"]:
|
||||
if field in sample:
|
||||
return _to_bool(sample[field])
|
||||
|
||||
# Nested llm_evaluation
|
||||
llm_eval = sample.get("llm_evaluation")
|
||||
if llm_eval and isinstance(llm_eval, list) and llm_eval:
|
||||
return _to_bool(llm_eval[0].get("llm_correct"))
|
||||
|
||||
# Nested rule_evaluation
|
||||
rule_eval = sample.get("rule_evaluation")
|
||||
if rule_eval and isinstance(rule_eval, list) and rule_eval:
|
||||
return _to_bool(rule_eval[0].get("correct"))
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _format_value(value: Any) -> str:
|
||||
"""Format value to string, handling list/dict/None."""
|
||||
if value is None:
|
||||
return "N/A"
|
||||
if isinstance(value, list):
|
||||
return str(value[0]) if value else "N/A"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _format_prompt(prompt: Any) -> str:
|
||||
"""
|
||||
Format prompt to readable string (matches model input format).
|
||||
|
||||
Handles:
|
||||
- Simple string: return as-is
|
||||
- Single message dict: extract prompt field
|
||||
- Single-turn list [{'role': 'HUMAN', 'prompt': '...'}]: return prompt directly (no prefix)
|
||||
- Multi-turn few-shot: format with ChatML-style role markers
|
||||
"""
|
||||
if isinstance(prompt, str):
|
||||
return prompt
|
||||
if isinstance(prompt, dict):
|
||||
return prompt.get("prompt", str(prompt))
|
||||
if isinstance(prompt, list) and prompt:
|
||||
first = prompt[0]
|
||||
# Check if it's conversation format
|
||||
if isinstance(first, dict) and "role" in first:
|
||||
# Single-turn: return prompt directly without prefix
|
||||
if len(prompt) == 1:
|
||||
return first.get("prompt", str(first))
|
||||
# Multi-turn few-shot: format with ChatML-style markers
|
||||
parts = []
|
||||
for msg in prompt:
|
||||
if isinstance(msg, dict):
|
||||
role = msg.get("role", "UNKNOWN")
|
||||
content = msg.get("prompt", str(msg))
|
||||
# Map HUMAN/BOT to user/assistant
|
||||
role_name = "user" if role == "HUMAN" else "assistant"
|
||||
parts.append(f"<|im_start|>{role_name}\n{content}<|im_end|>")
|
||||
else:
|
||||
parts.append(str(msg))
|
||||
return "\n".join(parts)
|
||||
# Single item list (not conversation format)
|
||||
if isinstance(first, dict):
|
||||
return first.get("prompt", str(first))
|
||||
return str(first)
|
||||
return "N/A"
|
||||
|
||||
|
||||
def _extract_tag_content(prompt: Any, tag_name: str) -> str:
|
||||
"""
|
||||
Extract content from <tag_name Begin>...<tag_name End> in prompt.
|
||||
|
||||
Used for extracting Original Question and Predicted Answer from LLM Judge prompts.
|
||||
"""
|
||||
if isinstance(prompt, list):
|
||||
prompt = str(prompt)
|
||||
prompt_str = str(prompt)
|
||||
|
||||
start_tag = f"<{tag_name} Begin>"
|
||||
end_tag = f"<{tag_name} End>"
|
||||
|
||||
start = prompt_str.find(start_tag)
|
||||
end = prompt_str.find(end_tag)
|
||||
|
||||
if start != -1 and end > start:
|
||||
content = prompt_str[start + len(start_tag) : end].strip()
|
||||
# Clean up formatting artifacts
|
||||
if content.startswith(": \\n"):
|
||||
content = content[4:]
|
||||
return content.strip()
|
||||
|
||||
return "N/A"
|
||||
|
||||
|
||||
def _get_question(sample: Dict, pred_entry: Dict) -> str:
|
||||
"""Extract question - prioritize predictions for complete content."""
|
||||
# 1. Priority: predictions directory origin_prompt
|
||||
if pred_entry.get("origin_prompt"):
|
||||
return _format_prompt(pred_entry["origin_prompt"])
|
||||
|
||||
# 2. Results directory direct fields
|
||||
for field in ["origin_prompt", "prompt", "source"]:
|
||||
if field in sample and sample[field]:
|
||||
return _format_prompt(sample[field])
|
||||
|
||||
# 3. Nested llm_evaluation (extract from <Original Question> tag)
|
||||
llm_eval = sample.get("llm_evaluation")
|
||||
if llm_eval and isinstance(llm_eval, list) and llm_eval:
|
||||
prompt = llm_eval[0].get("origin_prompt")
|
||||
if prompt:
|
||||
content = _extract_tag_content(prompt, "Original Question")
|
||||
if content != "N/A":
|
||||
return content
|
||||
|
||||
return sample.get("example_abbr", "N/A")
|
||||
|
||||
|
||||
def _get_gold(sample: Dict, pred_entry: Dict) -> str:
|
||||
"""Extract gold/reference answer - prioritize predictions."""
|
||||
# 1. Priority: predictions directory
|
||||
if pred_entry.get("gold") is not None:
|
||||
return _format_value(pred_entry["gold"])
|
||||
|
||||
# 2. Results directory direct fields
|
||||
for field in ["gold", "answer", "reference", "references"]:
|
||||
if field in sample and sample[field] is not None:
|
||||
return _format_value(sample[field])
|
||||
|
||||
# 3. Nested structures
|
||||
for nested in ["llm_evaluation", "rule_evaluation"]:
|
||||
eval_data = sample.get(nested)
|
||||
if eval_data and isinstance(eval_data, list) and eval_data:
|
||||
gold = eval_data[0].get("gold") or eval_data[0].get("answer")
|
||||
if gold is not None:
|
||||
return _format_value(gold)
|
||||
|
||||
return "N/A"
|
||||
|
||||
|
||||
def _get_prediction(sample: Dict, pred_entry: Dict) -> str:
|
||||
"""Extract model prediction/output - prioritize predictions."""
|
||||
# 1. Priority: predictions directory
|
||||
if pred_entry.get("prediction") is not None:
|
||||
return _format_value(pred_entry["prediction"])
|
||||
|
||||
# 2. Results directory direct fields (PANORAMA and similar formats)
|
||||
for field in ["pred_raw", "pred", "origin_prediction"]:
|
||||
if field in sample:
|
||||
return _format_value(sample[field])
|
||||
|
||||
# 3. Nested rule_evaluation.pred (CascadeEvaluator extracted answer)
|
||||
rule_eval = sample.get("rule_evaluation")
|
||||
if rule_eval and isinstance(rule_eval, list) and rule_eval:
|
||||
pred = rule_eval[0].get("pred")
|
||||
if pred is not None:
|
||||
return _format_value(pred)
|
||||
|
||||
return "N/A"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Main Entry Point
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def extract_error_samples(
|
||||
results_base: Path,
|
||||
max_samples: int = 10,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Extract error samples from OpenCompass benchmark results.
|
||||
|
||||
Uses both results and predictions directories:
|
||||
- results: correctness judgment
|
||||
- predictions: complete question/gold/prediction content
|
||||
|
||||
Args:
|
||||
results_base: Path to benchmark_results/{timestamp} directory
|
||||
max_samples: Maximum number of error samples to return
|
||||
|
||||
Returns:
|
||||
List of error samples, each containing:
|
||||
- question: The original prompt/question
|
||||
- gold: The expected/ground truth answer
|
||||
- model_output: The model's actual output
|
||||
- silver_answers (optional): For PANORAMA evaluator
|
||||
- custom_score (optional): For PANORAMA evaluator
|
||||
"""
|
||||
errors: List[Dict[str, Any]] = []
|
||||
results_dir = results_base / "results"
|
||||
predictions_dir = results_base / "predictions"
|
||||
|
||||
if not results_dir.exists():
|
||||
logger.warning(f"Results directory not found: {results_dir}")
|
||||
return errors
|
||||
|
||||
for result_file in results_dir.rglob("*.json"):
|
||||
with open(result_file) as f:
|
||||
results_data = json.load(f)
|
||||
|
||||
# Load corresponding predictions file
|
||||
rel_path = result_file.relative_to(results_dir)
|
||||
pred_file = predictions_dir / rel_path
|
||||
predictions: Dict[str, Any] = {}
|
||||
if pred_file.exists():
|
||||
with open(pred_file) as f:
|
||||
predictions = json.load(f)
|
||||
|
||||
details = results_data.get("details", [])
|
||||
if not details:
|
||||
continue
|
||||
|
||||
# Handle both list and dict formats
|
||||
if isinstance(details, list):
|
||||
iterator = enumerate(details)
|
||||
else:
|
||||
iterator = details.items()
|
||||
|
||||
for idx, sample in iterator:
|
||||
if not isinstance(sample, dict):
|
||||
continue
|
||||
|
||||
# Skip correct samples (from results)
|
||||
if _is_correct(sample):
|
||||
continue
|
||||
|
||||
# Get predictions entry (complete content)
|
||||
pred_entry = predictions.get(str(idx), {})
|
||||
|
||||
# Build error sample with core fields
|
||||
error = {
|
||||
"question": _get_question(sample, pred_entry),
|
||||
"gold": _get_gold(sample, pred_entry),
|
||||
"model_output": _get_prediction(sample, pred_entry),
|
||||
}
|
||||
|
||||
# Add PANORAMA extra fields if present
|
||||
if "silver" in sample:
|
||||
error["silver_answers"] = sample.get("silver", [])
|
||||
if "custom_score" in sample:
|
||||
error["custom_score"] = sample.get("custom_score", 0.0)
|
||||
|
||||
errors.append(error)
|
||||
|
||||
# Random sample if we have more than max_samples
|
||||
if len(errors) > max_samples:
|
||||
errors = random.sample(errors, max_samples)
|
||||
|
||||
logger.info(f"Extracted {len(errors)} error samples from benchmark results")
|
||||
return errors
|
||||
@@ -0,0 +1,148 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.datasets.financeiq.split import split_financeiq_dataset
|
||||
|
||||
|
||||
def download_financeiq_dataset() -> None:
|
||||
"""
|
||||
Download and arrange the FinanceIQ dataset for OpenCompass.
|
||||
|
||||
This downloads from `Duxiaoman-DI/FinanceIQ` into:
|
||||
<FT_RD_SETTING.file_path>/benchmarks/opencompass_data/data/FinanceIQ
|
||||
|
||||
The repo structure includes a `data` subdirectory; we move `dev` and `test`
|
||||
up one level to match the expected OpenCompass layout.
|
||||
"""
|
||||
target_dir = FT_RD_SETTING.file_path / "benchmarks" / "opencompass_data" / "data" / "FinanceIQ"
|
||||
if target_dir.exists():
|
||||
logger.info(f"FinanceIQ dataset already exists at {target_dir}")
|
||||
return
|
||||
|
||||
logger.info(f"Downloading FinanceIQ dataset to {target_dir}")
|
||||
target_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
subprocess.check_call(
|
||||
[
|
||||
"git",
|
||||
"clone",
|
||||
"https://huggingface.co/datasets/Duxiaoman-DI/FinanceIQ",
|
||||
str(target_dir),
|
||||
]
|
||||
)
|
||||
|
||||
# Move dev and test folders to upper level (opencompass_data/data/FinanceIQ)
|
||||
data_subdir = target_dir / "data"
|
||||
if data_subdir.exists():
|
||||
for folder in ("dev", "test"):
|
||||
src = data_subdir / folder
|
||||
if src.exists():
|
||||
shutil.move(str(src), str(target_dir / folder))
|
||||
shutil.rmtree(data_subdir)
|
||||
|
||||
# Apply split for benchmark (keep test set only)
|
||||
split_financeiq_dataset(str(target_dir), split="test")
|
||||
|
||||
|
||||
def extract_error_samples(results_base: Path, max_samples: int = 10) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
(Deprecated, processed by unified logic now)
|
||||
Extract error samples specifically for FinanceIQ_gen benchmark.
|
||||
|
||||
FinanceIQ_gen result files (per subject) look like:
|
||||
|
||||
{
|
||||
"accuracy": 60.0,
|
||||
"details": {
|
||||
"type": "GEN",
|
||||
"0": {
|
||||
"prompt": [...],
|
||||
"origin_prediction": "...",
|
||||
"predictions": "D",
|
||||
"references": "B"
|
||||
},
|
||||
"1": { ... },
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
We treat a sample as error when predictions != references.
|
||||
The question text is taken from the last HUMAN prompt in the prompt list.
|
||||
|
||||
Args:
|
||||
results_base: Path to benchmark_results/{timestamp} directory
|
||||
max_samples: Maximum number of error samples to return
|
||||
|
||||
Returns:
|
||||
List of error samples, each containing:
|
||||
- question: The original prompt/question
|
||||
- gold: The expected/ground truth answer (references)
|
||||
- model_output: The model's actual output (predictions)
|
||||
"""
|
||||
error_samples: List[Dict[str, Any]] = []
|
||||
results_dir = results_base / "results" / "ft-FinanceIQ_gen"
|
||||
|
||||
if not results_dir.exists():
|
||||
logger.warning(f"FinanceIQ_gen results directory not found: {results_dir}")
|
||||
return error_samples
|
||||
|
||||
# Iterate through all FinanceIQ subject JSON files
|
||||
for result_file in sorted(results_dir.glob("*.json")):
|
||||
with open(result_file) as f:
|
||||
data = json.load(f)
|
||||
|
||||
details = data.get("details", {})
|
||||
if not isinstance(details, dict):
|
||||
continue
|
||||
|
||||
# Each key in details except "type" is a sample index
|
||||
for key, sample in details.items():
|
||||
if key == "type" or not isinstance(sample, dict):
|
||||
continue
|
||||
|
||||
pred = sample.get("predictions")
|
||||
gold = sample.get("references")
|
||||
|
||||
# Skip if either is missing
|
||||
if pred is None or gold is None:
|
||||
continue
|
||||
|
||||
# Only keep incorrect predictions
|
||||
if str(pred) == str(gold):
|
||||
continue
|
||||
|
||||
prompt_list = sample.get("prompt", [])
|
||||
question = "N/A"
|
||||
if isinstance(prompt_list, list) and prompt_list:
|
||||
# Take the last HUMAN message as the question
|
||||
for msg in reversed(prompt_list):
|
||||
if isinstance(msg, dict) and msg.get("role") == "HUMAN":
|
||||
question = msg.get("prompt", "N/A")
|
||||
break
|
||||
|
||||
error_samples.append(
|
||||
{
|
||||
"question": question,
|
||||
"gold": str(gold),
|
||||
"model_output": str(pred),
|
||||
}
|
||||
)
|
||||
|
||||
if not error_samples:
|
||||
logger.info("No FinanceIQ_gen error samples found")
|
||||
return error_samples
|
||||
|
||||
# Random sampling if too many error samples
|
||||
if len(error_samples) > max_samples:
|
||||
error_samples = random.sample(error_samples, max_samples)
|
||||
|
||||
logger.info(f"Extracted {len(error_samples)} FinanceIQ_gen error samples from {results_dir}")
|
||||
return error_samples
|
||||
@@ -0,0 +1,75 @@
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.components.coder.finetune.conf import get_workspace_prefix
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
BLACKWELL_GPU_KEYWORDS = ["b100", "b200", "b300"]
|
||||
|
||||
|
||||
def is_blackwell_gpu() -> bool:
|
||||
"""Check if the current GPU is NVIDIA Blackwell architecture (B100, B200, B300)."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
gpu_names = result.stdout.strip().lower()
|
||||
return any(kw in gpu_names for kw in BLACKWELL_GPU_KEYWORDS)
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def check_if_merging_needed(model_path: str | Path) -> bool:
|
||||
"""
|
||||
Check if the model needs to be merged before benchmarking.
|
||||
Usually required when LoRA adapter has modules_to_save which vLLM doesn't support.
|
||||
"""
|
||||
config_path = Path(model_path) / "adapter_config.json"
|
||||
if not config_path.exists():
|
||||
return False
|
||||
with open(config_path, "r") as f:
|
||||
config = json.load(f)
|
||||
# Check for modules_to_save which requires merging for vLLM
|
||||
# The logic is based in https://github.com/vllm-project/vllm/issues/9280
|
||||
if config.get("modules_to_save") is not None:
|
||||
logger.info(f"Model merging required due to modules_to_save: {config.get('modules_to_save')}")
|
||||
return True
|
||||
if is_blackwell_gpu():
|
||||
logger.info("Model merging required due to Blackwell GPU (B100/B200/B300)")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def merge_model(env, workspace_path: Path, base_model_path: str, adapter_path: str, output_path: str):
|
||||
"""
|
||||
Merge LoRA adapter into base model using a template-generated script.
|
||||
"""
|
||||
# Prepare template variables
|
||||
template_vars = {
|
||||
"base_model_path": base_model_path,
|
||||
"adapter_path": adapter_path,
|
||||
"output_path": output_path,
|
||||
}
|
||||
|
||||
# Render Jinja2 template
|
||||
merge_script = T("rdagent.scenarios.finetune.benchmark.merge.merge_model_template:template").r(**template_vars)
|
||||
|
||||
script_path = workspace_path / "merge_model.py"
|
||||
script_path.write_text(merge_script)
|
||||
|
||||
logger.info(f"Starting model merging from {adapter_path}...")
|
||||
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
cmd = f"python {ws_prefix}/merge_model.py"
|
||||
|
||||
result = env.run(cmd, local_path=str(workspace_path))
|
||||
if result.exit_code != 0:
|
||||
raise RuntimeError(f"Model merging failed (exit_code={result.exit_code}):\n{result.stdout}")
|
||||
logger.info("Model merging completed.")
|
||||
@@ -0,0 +1,44 @@
|
||||
# Jinja2 template for merging LoRA models
|
||||
# Used by benchmark.py to generate a merging script
|
||||
|
||||
template: |-
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from peft import PeftModel
|
||||
import os
|
||||
import shutil
|
||||
|
||||
base_model_path = "{{ base_model_path }}"
|
||||
adapter_path = "{{ adapter_path }}"
|
||||
output_path = "{{ output_path }}"
|
||||
|
||||
print(f"Loading base model from {base_model_path}...")
|
||||
base_model = AutoModelForCausalLM.from_pretrained(
|
||||
base_model_path,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
local_files_only=True
|
||||
)
|
||||
|
||||
print(f"Loading LoRA adapter from {adapter_path}...")
|
||||
model = PeftModel.from_pretrained(base_model, adapter_path, local_files_only=True)
|
||||
|
||||
print(f"Loading tokenizer from {adapter_path}...")
|
||||
try:
|
||||
tokenizer = AutoTokenizer.from_pretrained(adapter_path, trust_remote_code=True, local_files_only=True)
|
||||
except:
|
||||
print("Tokenizer not found in adapter, loading from base model...")
|
||||
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True, local_files_only=True)
|
||||
|
||||
print("Merging model...")
|
||||
model = model.merge_and_unload()
|
||||
|
||||
if os.path.exists(output_path):
|
||||
print(f"Removing existing output path: {output_path}")
|
||||
shutil.rmtree(output_path)
|
||||
|
||||
print(f"Saving merged model to {output_path}...")
|
||||
model.save_pretrained(output_path)
|
||||
tokenizer.save_pretrained(output_path)
|
||||
print("Merge Done.")
|
||||
@@ -0,0 +1,190 @@
|
||||
# 数据集管理模块
|
||||
|
||||
本模块管理 LLM Finetune 场景的数据集，通过 `snapshot_download` 下载完整的 HuggingFace 仓库。
|
||||
|
||||
## 设计目标
|
||||
|
||||
1. **简洁性**: 下载完整的 HF 仓库，保留原始文件结构
|
||||
2. **可扩展性**: 支持可选的 `post_download_fn` 进行自定义处理（如删除测试集）
|
||||
|
||||
## 使用方法
|
||||
|
||||
```python
|
||||
from rdagent.scenarios.finetune.datasets import prepare, prepare_all, DATASETS
|
||||
|
||||
# 1. 查看已注册的数据集
|
||||
print(DATASETS.keys())
|
||||
# ['chemcot', 'panorama', 'deepscaler', 'financeiq']
|
||||
|
||||
# 2. 准备单个数据集（下载到本地）
|
||||
path = prepare("chemcot")
|
||||
# 下载至: datasets/chemcot/
|
||||
|
||||
# 3. 准备所有数据集
|
||||
prepare_all()
|
||||
```
|
||||
|
||||
## 数据集配置
|
||||
|
||||
每个数据集通过 `DatasetConfig` 配置：
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class DatasetConfig:
|
||||
repo_id: str # HuggingFace 仓库 ID
|
||||
post_download_fn: Optional[Callable[[str], None]] # 下载后处理函数
|
||||
```
|
||||
|
||||
## 已注册数据集
|
||||
|
||||
| 名称 | 仓库 | 描述 |
|
||||
|------|------|------|
|
||||
| `chemcot` | OpenMol/ChemCoTDataset | 化学推理 + CoT |
|
||||
| `panorama` | LG-AI-Research/PANORAMA | 专利审查基准 |
|
||||
| `deepscaler` | agentica-org/DeepScaleR-Preview-Dataset | 数学推理 |
|
||||
| `financeiq` | Duxiaoman-DI/FinanceIQ | 金融问答 |
|
||||
|
||||
## 添加新数据集
|
||||
|
||||
在 `__init__.py` 的 `DATASETS` 字典中添加配置：
|
||||
|
||||
```python
|
||||
DATASETS["my-dataset"] = DatasetConfig(
|
||||
repo_id="organization/dataset-name",
|
||||
post_download_fn=my_cleanup_function, # 可选
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## README 替换机制
|
||||
|
||||
**重要**: 下载数据集时，本地 README 会覆盖 HuggingFace 原始 README。
|
||||
|
||||
### 工作原理
|
||||
|
||||
```python
|
||||
# __init__.py 中的逻辑
|
||||
custom_readme = Path(__file__).parent / name / "README.md"
|
||||
if custom_readme.exists():
|
||||
shutil.copy(custom_readme, out_dir / "README.md")
|
||||
```
|
||||
|
||||
1. 数据集下载完成后，检查 `datasets/{name}/README.md` 是否存在
|
||||
2. 如果存在，用本地版本覆盖下载目录中的 README
|
||||
3. 这样可以为每个数据集提供**定制化的文档**
|
||||
|
||||
### 目录结构
|
||||
|
||||
```
|
||||
rdagent/scenarios/finetune/datasets/
|
||||
├── __init__.py # 主模块: prepare(), prepare_all(), DATASETS
|
||||
├── README.md # 本文档
|
||||
├── chemcot/
|
||||
│ └── README.md # ChemCoT 数据集文档（会覆盖 HF 原版）
|
||||
├── panorama/
|
||||
│ └── README.md # PANORAMA 数据集文档（会覆盖 HF 原版）
|
||||
├── deepscaler/
|
||||
│ └── README.md # DeepScaleR 数据集文档（会覆盖 HF 原版）
|
||||
└── financeiq/
|
||||
└── README.md # FinanceIQ 数据集文档（会覆盖 HF 原版）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## README 编写规范
|
||||
|
||||
为每个数据集编写 README 时，建议包含以下内容：
|
||||
|
||||
### 1. 基础信息（必需）
|
||||
|
||||
```markdown
|
||||
# 数据集名称
|
||||
|
||||
简要描述 + 论文链接
|
||||
|
||||
**Repository**: [HuggingFace 链接]
|
||||
|
||||
## Overview
|
||||
|
||||
数据集规模、来源、用途的概述
|
||||
```
|
||||
|
||||
### 2. 数据集规模（必需）
|
||||
|
||||
```markdown
|
||||
## Dataset Scale
|
||||
|
||||
| 类别 | 子任务 | 样本数 |
|
||||
|------|--------|--------|
|
||||
| xxx | xxx | 1,234 |
|
||||
| **Total** | **N subtasks** | **总数** |
|
||||
```
|
||||
|
||||
### 3. 数据字段说明（必需）
|
||||
|
||||
```markdown
|
||||
## Data Fields
|
||||
|
||||
| 字段 | 类型 | 描述 |
|
||||
|------|------|------|
|
||||
| `id` | string | 唯一标识符 |
|
||||
| `query` | string | 问题/指令 |
|
||||
| `answer` | string | 答案 |
|
||||
| ... | ... | ... |
|
||||
```
|
||||
|
||||
### 4. CoT 质量评估（关键）
|
||||
|
||||
这是最重要的部分，直接告诉使用者数据是否可用、如何处理：
|
||||
|
||||
```markdown
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: [数据质量的核心警告]
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | low / medium / high / N/A |
|
||||
| task_type | math / chemistry / legal / ... |
|
||||
| polish_difficulty | low / medium / high |
|
||||
|
||||
**Baseline**: [详细说明]
|
||||
- 如果有 CoT: 说明来源、验证方式、质量问题
|
||||
- 如果没有 CoT: 明确标注 "NO CoT"，说明必须生成
|
||||
```
|
||||
|
||||
### 5. Baseline 性能（推荐）
|
||||
|
||||
```markdown
|
||||
## Baseline Performance
|
||||
|
||||
| Task | Best Model | Score |
|
||||
|------|-----------|-------|
|
||||
| xxx | GPT-4o | 85.2% |
|
||||
```
|
||||
|
||||
### 6. 许可证（必需）
|
||||
|
||||
```markdown
|
||||
## License
|
||||
|
||||
MIT / CC-BY-NC-4.0 / ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 示例参考
|
||||
|
||||
- **DeepScaleR**: [deepscaler/README.md](deepscaler/README.md) - 标杆示例，CoT Quality Assessment 写得最清晰
|
||||
- **ChemCoT**: [chemcot/README.md](chemcot/README.md) - 有 CoT 但需要精化的情况
|
||||
- **PANORAMA**: [panorama/README.md](panorama/README.md) - 没有 CoT 的情况
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **Token**: 私有数据集需要设置 `HF_TOKEN` 环境变量
|
||||
2. **缓存**: HuggingFace hub 会自动缓存下载内容
|
||||
3. **强制刷新**: 使用 `prepare(name, force=True)` 重新下载
|
||||
4. **README 优先级**: 本地 README 会覆盖 HuggingFace 原版，确保文档一致性
|
||||
@@ -0,0 +1,133 @@
|
||||
"""Dataset preparation module for finetune scenarios.
|
||||
|
||||
Usage:
|
||||
from rdagent.scenarios.finetune.datasets import prepare, prepare_all
|
||||
|
||||
prepare("chemcot") # Download ChemCoT dataset
|
||||
prepare("panorama") # Download PANORAMA dataset
|
||||
prepare_all() # Prepare all registered datasets
|
||||
"""
|
||||
|
||||
import shutil
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.scenarios.finetune.datasets.chemcot import normalize_rcr
|
||||
from rdagent.scenarios.finetune.datasets.financeiq.split import split_financeiq_dataset
|
||||
from rdagent.scenarios.finetune.download.hf import download_dataset
|
||||
|
||||
|
||||
@dataclass
|
||||
class DatasetConfig:
|
||||
"""Configuration for a registered dataset.
|
||||
|
||||
Attributes:
|
||||
repo_id: HuggingFace dataset repository ID
|
||||
post_download_fn: Optional function to run after download (e.g., remove test split)
|
||||
"""
|
||||
|
||||
repo_id: str
|
||||
post_download_fn: Optional[Callable[[str], None]] = field(default=None)
|
||||
|
||||
|
||||
def _remove_eval_splits(out_dir: str) -> None:
|
||||
"""Remove validation and test split files to prevent data leakage."""
|
||||
for pattern in ["*validation*", "*test*"]:
|
||||
for f in Path(out_dir).rglob(pattern):
|
||||
if f.is_file():
|
||||
f.unlink()
|
||||
elif f.is_dir():
|
||||
shutil.rmtree(f)
|
||||
|
||||
|
||||
# Dataset registry: name -> DatasetConfig
|
||||
DATASETS: dict[str, DatasetConfig] = {
|
||||
"chemcot": DatasetConfig(
|
||||
repo_id="OpenMol/ChemCoTDataset",
|
||||
post_download_fn=normalize_rcr,
|
||||
),
|
||||
"panorama": DatasetConfig(
|
||||
repo_id="LG-AI-Research/PANORAMA",
|
||||
post_download_fn=_remove_eval_splits,
|
||||
),
|
||||
"deepscaler": DatasetConfig(
|
||||
repo_id="agentica-org/DeepScaleR-Preview-Dataset",
|
||||
),
|
||||
"financeiq": DatasetConfig(
|
||||
repo_id="Duxiaoman-DI/FinanceIQ",
|
||||
post_download_fn=lambda out_dir: split_financeiq_dataset(out_dir, split="train"),
|
||||
),
|
||||
"tableinstruct": DatasetConfig(
|
||||
repo_id="Multilingual-Multimodal-NLP/TableInstruct",
|
||||
),
|
||||
"bioprobench": DatasetConfig(
|
||||
repo_id="bowenxian/BioProBench",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def prepare(name: str, force: bool = False) -> str:
|
||||
"""Download dataset to local directory using snapshot_download.
|
||||
|
||||
Downloads the entire HuggingFace dataset repository, preserving the original
|
||||
file structure.
|
||||
|
||||
Args:
|
||||
name: Dataset name (must be registered in DATASETS)
|
||||
force: If True, re-download even if exists
|
||||
|
||||
Returns:
|
||||
Path to the dataset directory
|
||||
"""
|
||||
if name not in DATASETS:
|
||||
raise ValueError(f"Unknown dataset: {name}. Available: {list(DATASETS.keys())}")
|
||||
|
||||
config = DATASETS[name]
|
||||
out_dir = Path(FT_RD_SETTING.file_path) / "datasets" / name
|
||||
|
||||
# Skip if already exists and not forcing
|
||||
if not force and out_dir.exists():
|
||||
return str(out_dir)
|
||||
|
||||
# Download using snapshot_download
|
||||
download_dataset(
|
||||
repo_id=config.repo_id,
|
||||
out_dir=str(out_dir),
|
||||
force=force,
|
||||
)
|
||||
|
||||
# Run post-download processing if defined
|
||||
if config.post_download_fn:
|
||||
config.post_download_fn(str(out_dir))
|
||||
|
||||
# Copy custom README if exists in source code
|
||||
custom_readme = Path(__file__).parent / name / "README.md"
|
||||
if custom_readme.exists():
|
||||
shutil.copy(custom_readme, out_dir / "README.md")
|
||||
|
||||
return str(out_dir)
|
||||
|
||||
|
||||
def prepare_all(force: bool = False) -> dict[str, str]:
|
||||
"""Prepare all registered datasets.
|
||||
|
||||
Args:
|
||||
force: If True, re-download even if exists
|
||||
|
||||
Returns:
|
||||
Dict mapping dataset name to download path
|
||||
"""
|
||||
return {name: prepare(name, force=force) for name in DATASETS}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
dataset_name = sys.argv[1]
|
||||
path = prepare(dataset_name)
|
||||
print(f"Dataset prepared at: {path}")
|
||||
else:
|
||||
print(f"Available datasets: {list(DATASETS.keys())}")
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
license: cc-by-4.0
|
||||
configs:
|
||||
- config_name: PQA
|
||||
data_files:
|
||||
- split: train
|
||||
path: PQA.json
|
||||
- split: test
|
||||
path: PQA_test.json
|
||||
- config_name: ERR
|
||||
data_files:
|
||||
- split: train
|
||||
path: ERR.json
|
||||
- split: test
|
||||
path: ERR_test.json
|
||||
- config_name: ORD
|
||||
data_files:
|
||||
- split: train
|
||||
path: ORD.json
|
||||
- split: test
|
||||
path: ORD_test.json
|
||||
- config_name: GEN
|
||||
data_files:
|
||||
- split: train
|
||||
path: GEN.json
|
||||
- split: test
|
||||
path: GEN_test.json
|
||||
---
|
||||
|
||||
# BioProBench Dataset for LLM Fine-Tuning
|
||||
|
||||
BioProBench is a large-scale, multi-task benchmark focused on biological protocol understanding and reasoning for large language models (LLMs). It spans four fine-tuning tasks provided here: Protocol Question Answering (PQA), Step Ordering (ORD), Error Correction (ERR), and Protocol Generation (GEN).
|
||||
|
||||
This dataset is built on a raw corpus of ~27K biological protocols and provides over 550K structured instances across tasks, with a held-out test set of 1,000 examples per task. See the original benchmark for full details:
|
||||
- Code: https://github.com/YuyangSunshine/bioprotocolbench/
|
||||
- Dataset hub: https://huggingface.co/BioProBench
|
||||
- License: CC BY 4.0
|
||||
|
||||
## Data Files
|
||||
|
||||
The JSON files for each task (train/test) are organized per task. If your fine-tuning pipeline expects local files, place them alongside this README or update paths accordingly.
|
||||
|
||||
- PQA: [bioprobench/PQA.json](bioprobench/PQA.json), [bioprobench/PQA_test.json](bioprobench/PQA_test.json)
|
||||
- ERR: [bioprobench/ERR.json](bioprobench/ERR.json), [bioprobench/ERR_test.json](bioprobench/ERR_test.json)
|
||||
- ORD: [bioprobench/ORD.json](bioprobench/ORD.json), [bioprobench/ORD_test.json](bioprobench/ORD_test.json)
|
||||
- GEN: [bioprobench/GEN.json](bioprobench/GEN.json), [bioprobench/GEN_test.json](bioprobench/GEN_test.json)
|
||||
|
||||
## Task Definitions and Fields
|
||||
|
||||
### PQA — Protocol Question Answering
|
||||
Multiple-choice QA over protocol content.
|
||||
- Fields:
|
||||
- `question`: the question string
|
||||
- `choices`: list of candidate answers
|
||||
- `answer`: the correct answer
|
||||
- `type`: category of the question (e.g., parameter, reagent, operation)
|
||||
- `id`: unique identifier
|
||||
|
||||
### ORD — Step Ordering
|
||||
Order protocol steps correctly (top-level or sub-step sequences).
|
||||
- Fields:
|
||||
- `question`: prompt describing the step list and context/title
|
||||
- `wrong_steps`: list of steps in a shuffled or incorrect order
|
||||
- `correct_steps`: steps in the correct chronological order
|
||||
- `type`: sequence granularity (e.g., `top`, `child`)
|
||||
- `id`: unique identifier
|
||||
|
||||
### ERR — Error Correction
|
||||
Detect and correct errors in protocol text with local context.
|
||||
- Fields:
|
||||
- `context`: object with `purpose`, `prior_step`, `next_step`
|
||||
- `corrupted_text`: the erroneous text (may be `null` for correct cases)
|
||||
- `corrected_text`: corrected version of the text
|
||||
- `is_correct`: boolean indicating whether the provided text was already correct
|
||||
- `type`: category (e.g., parameter, reagent, operation, or `correct`)
|
||||
- `error_description`: brief rationale for the correction
|
||||
- `id`: unique identifier
|
||||
|
||||
### GEN — Protocol Generation
|
||||
Generate concise, single-level, numbered protocol steps from prompts.
|
||||
- Fields:
|
||||
- `system_prompt`: role/system instruction
|
||||
- `instruction`: formatting and style constraints
|
||||
- `input`: task description or query
|
||||
- `output`: list of numbered steps (flat 1., 2., 3. ...)
|
||||
- `id`: unique identifier
|
||||
- `type`: difficulty tag (e.g., `easy`)
|
||||
|
||||
## Splits
|
||||
- Train: use the non-`_test.json` files per task.
|
||||
- Test: each task provides a held-out set of 1,000 examples.
|
||||
|
||||
## Training Data Guidelines (CRITICAL for Fine-tuning)
|
||||
|
||||
### ERR — Error Correction
|
||||
|
||||
**CRITICAL: Answer Semantics**
|
||||
|
||||
The benchmark prompt says: "If you find anything wrong, answer False."
|
||||
|
||||
| Condition | `is_correct` field | Correct training output |
|
||||
|-----------|-------------------|------------------------|
|
||||
| Protocol step is CORRECT | `True` | `[ANSWER_START]True[ANSWER_END]` |
|
||||
| Protocol step HAS ERRORS | `False` | `[ANSWER_START]False[ANSWER_END]` |
|
||||
|
||||
**Important**: The training data generation script MUST use this logic:
|
||||
```python
|
||||
def gold_answer_from_is_correct(is_correct: bool) -> str:
|
||||
# True = step is correct, False = step has errors
|
||||
return ANSWER_TRUE if is_correct else ANSWER_FALSE
|
||||
```
|
||||
|
||||
Do NOT invert this logic - the benchmark evaluator compares model output directly with `is_correct` field.
|
||||
|
||||
### ORD — Step Ordering
|
||||
|
||||
**Output Format (CRITICAL)**
|
||||
- Answer MUST be a valid Python list: `[0, 2, 1, 3]`
|
||||
- NOT space-separated: `0 2 1 3` (WRONG)
|
||||
- NOT comma-separated without brackets: `0, 2, 1, 3` (WRONG)
|
||||
|
||||
**Training Data Format**
|
||||
- Can include brief reasoning (1-2 sentences)
|
||||
- Final answer MUST be in format: `[ANSWER_START][list][ANSWER_END]`
|
||||
- Example: `[ANSWER_START][2, 0, 1, 3][ANSWER_END]`
|
||||
|
||||
### GEN — Protocol Generation
|
||||
|
||||
**Output Format**
|
||||
- Step-by-step protocol wrapped in `[ANSWER_START]...[ANSWER_END]`
|
||||
- CoT (Chain-of-Thought) format is acceptable for this task
|
||||
|
||||
### Common Notes
|
||||
- All tasks support `<think>...</think>` tags for CoT reasoning (evaluator will strip them)
|
||||
- Answer MUST be wrapped in `[ANSWER_START]` and `[ANSWER_END]` tags
|
||||
|
||||
## License
|
||||
- CC BY 4.0 — see https://creativecommons.org/licenses/by/4.0/
|
||||
|
||||
## Notes
|
||||
- Tasks cover protocol QA, ordering, correction, and generation (REA is part of the broader benchmark but not included in the files above).
|
||||
- Data spans diverse biological domains and repositories; see the original benchmark for details.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: mit
|
||||
tags:
|
||||
- chemistry
|
||||
- chain-of-thought
|
||||
- molecular-reasoning
|
||||
size_categories:
|
||||
- 10K<n<100K
|
||||
task_categories:
|
||||
- text-generation
|
||||
- question-answering
|
||||
---
|
||||
|
||||
# ChemCoT Dataset
|
||||
|
||||
Chemical reasoning dataset with Chain-of-Thought annotations from [ChemCoTBench](https://arxiv.org/abs/2505.21318).
|
||||
|
||||
**Repository**: [OpenMol/ChemCoTDataset](https://huggingface.co/datasets/OpenMol/ChemCoTDataset)
|
||||
|
||||
## Overview
|
||||
|
||||
The **ChemCoTDataset** provides ~23K high-quality chain-of-thought samples for training chemical reasoning models. CoT annotations were distilled from state-of-the-art reasoning models (Gemini-2.5-pro, DeepSeek-R1, Claude-3.7-sonnet-thinking) and validated by 13 chemistry PhD candidates with >90% accuracy.
|
||||
|
||||
### Dataset Scale
|
||||
|
||||
| Category | Subtasks | Samples |
|
||||
|----------|----------|---------|
|
||||
| mol_und | fg_count, ring_count, ring_system_scaffold, Murcko_scaffold | 6,319 |
|
||||
| mol_edit | add, delete, sub | 4,497 |
|
||||
| mol_opt | drd, gsk, jnk, qed, solubility, logp | 5,587 |
|
||||
| rxn | fs_by_product, fs_major_product, rcr | 6,820 |
|
||||
| **Total** | **16 subtasks** | **23,223** |
|
||||
|
||||
## Tasks
|
||||
|
||||
### 1. Molecular Understanding (mol_und)
|
||||
|
||||
| Subtask | Description |
|
||||
|---------|-------------|
|
||||
| `fg_count` | Functional group counting |
|
||||
| `ring_count` | Ring counting |
|
||||
| `Murcko_scaffold` | Murcko scaffold extraction |
|
||||
| `ring_system_scaffold` | Ring system scaffold extraction |
|
||||
|
||||
**Metrics**: MAE for counting, Tanimoto similarity for scaffold extraction
|
||||
|
||||
### 2. Molecular Editing (mol_edit)
|
||||
|
||||
| Subtask | Description |
|
||||
|---------|-------------|
|
||||
| `add` | Add functional groups to molecules |
|
||||
| `delete` | Delete functional groups from molecules |
|
||||
| `sub` | Substitute functional groups in molecules |
|
||||
|
||||
**Metrics**: Pass@1 (validity and instruction matching)
|
||||
|
||||
### 3. Molecular Optimization (mol_opt)
|
||||
|
||||
| Subtask | Description |
|
||||
|---------|-------------|
|
||||
| `logp` | LogP (lipophilicity) optimization |
|
||||
| `solubility` | Aqueous solubility optimization |
|
||||
| `qed` | QED (drug-likeness) optimization |
|
||||
| `drd` | DRD2 binding affinity optimization |
|
||||
| `gsk` | GSK3-beta binding affinity optimization |
|
||||
| `jnk` | JNK3 binding affinity optimization |
|
||||
|
||||
**Metrics**: Mean improvement rate, Success rate
|
||||
|
||||
### 4. Reaction Prediction (rxn)
|
||||
|
||||
| Subtask | Description |
|
||||
|---------|-------------|
|
||||
| `fs` | Forward synthesis (major product + by-product prediction) |
|
||||
| `rcr` | Reaction Condition Recommendation (catalyst prediction) |
|
||||
|
||||
**Metrics**: Top-1 accuracy, Fingerprint similarity
|
||||
|
||||
## Data Format
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `id` | string | Unique sample identifier |
|
||||
| `query` | string | The chemical problem/question |
|
||||
| `task` | string | Task category (mol_und, mol_edit, mol_opt, rxn) |
|
||||
| `subtask` | string | Specific subtask name |
|
||||
| `struct_cot` | string | Structured chain-of-thought reasoning |
|
||||
| `raw_cot` | string | Raw chain-of-thought annotation |
|
||||
| `meta` | object | Additional metadata |
|
||||
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: Distilled CoT may require domain refinement.
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | medium-high |
|
||||
| task_type | chemistry |
|
||||
| polish_difficulty | medium |
|
||||
|
||||
**Baseline**: CoT distilled from Gemini-2.5-pro/DeepSeek-R1/Claude, validated by 13 chemistry PhD candidates (>90% accuracy). Paper notes: *"distillation strategy falters in chemistry"* - consider expert refinement for optimal results.
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
@@ -0,0 +1,38 @@
|
||||
"""ChemCoT dataset preparation utilities."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def normalize_rcr(out_dir: str) -> None:
|
||||
"""Normalize rcr.json to match standard data format.
|
||||
|
||||
Fixes:
|
||||
1. Move `gt` from top-level into `meta`
|
||||
2. Rename `cot_result` to `struct_cot` and strip markdown wrapper
|
||||
"""
|
||||
rcr_path = Path(out_dir) / "chemcotbench-cot" / "rxn" / "rcr.json"
|
||||
if not rcr_path.exists():
|
||||
return
|
||||
|
||||
with open(rcr_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
for item in data:
|
||||
# 1. Move gt from top-level into meta
|
||||
if "gt" in item:
|
||||
meta = json.loads(item["meta"]) if isinstance(item["meta"], str) else item["meta"]
|
||||
meta["gt"] = item.pop("gt")
|
||||
item["meta"] = json.dumps(meta)
|
||||
|
||||
# 2. Rename cot_result -> struct_cot, strip markdown wrapper
|
||||
if "cot_result" in item:
|
||||
cot = item.pop("cot_result").strip()
|
||||
if cot.startswith("```json"):
|
||||
cot = cot[7:]
|
||||
if cot.endswith("```"):
|
||||
cot = cot[:-3]
|
||||
item["struct_cot"] = cot.strip()
|
||||
|
||||
with open(rcr_path, "w") as f:
|
||||
json.dump(data, f, indent=4)
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
size_categories:
|
||||
- 10K<n<100K
|
||||
license: mit
|
||||
configs:
|
||||
- config_name: default
|
||||
data_files:
|
||||
- split: train
|
||||
path: data/train-*
|
||||
splits:
|
||||
- name: train
|
||||
num_examples: 40315
|
||||
---
|
||||
|
||||
# DeepScaleR Mathematical Reasoning Dataset
|
||||
|
||||
Dataset for DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL.
|
||||
|
||||
> DeepScaleR-1.5B-Preview achieves **43.1% Pass@1 accuracy on AIME 2024**, representing a **15% improvement** over the base model (28.8%) and **surpassing OpenAI's O1-Preview performance** with just 1.5B parameters through distributed reinforcement learning.
|
||||
|
||||
## Overview
|
||||
|
||||
The **DeepScaleR dataset** is a carefully curated collection of approximately **40,000 unique mathematics problem-answer pairs** designed for training mathematical reasoning models through reinforcement learning. This dataset covers a wide range of competition-level mathematics problems from high school to olympiad level, providing a robust foundation for scaling RL algorithms on reasoning tasks.
|
||||
|
||||
DeepScaleR demonstrates that sophisticated mathematical reasoning can be achieved through strategic data curation combined with iterative context length scaling (8K→16K→24K) using Group Relative Policy Optimization (GRPO).
|
||||
|
||||
|
||||
### Data Sources
|
||||
|
||||
Our training dataset consists of problems compiled from prestigious mathematics competitions and curated datasets:
|
||||
|
||||
- **AIME** (American Invitational Mathematics Examination) problems (1984-2023)
|
||||
- **AMC** (American Mathematics Competition) problems (prior to 2023)
|
||||
- **Omni-MATH** dataset
|
||||
- **Still** dataset
|
||||
|
||||
### Data Fields
|
||||
|
||||
The dataset contains three key fields:
|
||||
|
||||
- `problem`: The mathematical problem statement, formatted with LaTeX notation
|
||||
- `solution`: Official solution to the problem, including LaTeX formatting and boxed final answers. If there is no solution, the `solution` field is an empty string
|
||||
- `answer`: The final mathematical result/answer, usually extracted from the solution
|
||||
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: Raw data must be polished before training.
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | low |
|
||||
| task_type | math |
|
||||
| polish_difficulty | high |
|
||||
|
||||
**Baseline**: 82% empty `solution`, 18% too short (p50=373 tokens, summary-style). Need to generate exploratory CoT (For your reference, the length of a well-structured CoT is usually longer than 1/4 * the model max_position_embeddings tokens) for all samples.
|
||||
|
||||
## License
|
||||
|
||||
This dataset is released under the MIT License.
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
language:
|
||||
- zh
|
||||
license: cc-by-nc-sa-4.0
|
||||
tags:
|
||||
- finance
|
||||
- chinese
|
||||
- multiple-choice
|
||||
- professional-certification
|
||||
size_categories:
|
||||
- 1K<n<10K
|
||||
task_categories:
|
||||
- question-answering
|
||||
- text-generation
|
||||
---
|
||||
|
||||
# FinanceIQ Dataset
|
||||
|
||||
Chinese financial professional certification exam questions covering 10 major financial domains.
|
||||
|
||||
**Repository**: [LlamaFactory/FinanceIQ](https://huggingface.co/datasets/LlamaFactory/FinanceIQ)
|
||||
|
||||
## Overview
|
||||
|
||||
The **FinanceIQ dataset** is a comprehensive collection of approximately **6,179 multiple-choice questions** from Chinese financial professional certification exams. It covers 10 distinct financial domains, providing a robust benchmark for evaluating financial reasoning capabilities in Chinese language models.
|
||||
|
||||
### Dataset Scale
|
||||
|
||||
| Category | Chinese Name | Samples |
|
||||
|----------|--------------|---------|
|
||||
| Insurance (CICE) | 保险从业资格CICE | 596 |
|
||||
| Fund Practitioner | 基金从业资格 | 772 |
|
||||
| Futures Practitioner | 期货从业资格 | 333 |
|
||||
| CPA | 注册会计师（CPA） | 1,211 |
|
||||
| Financial Planner | 理财规划师 | 195 |
|
||||
| Tax Advisor | 税务师 | 388 |
|
||||
| Actuary (Financial Math) | 精算师-金融数学 | 44 |
|
||||
| Economist | 经济师 | 420 |
|
||||
| Securities Practitioner | 证券从业资格 | 1,076 |
|
||||
| Banking Practitioner | 银行从业资格 | 1,144 |
|
||||
| **Total** | **10 categories** | **6,179** |
|
||||
|
||||
## Tasks
|
||||
|
||||
### Single-Choice Question Answering
|
||||
|
||||
**Task**: Select the correct answer (A/B/C/D) from four options for each financial question.
|
||||
|
||||
**Evaluation**: LLM Judge (comparing model's answer selection with ground truth)
|
||||
|
||||
**Metrics**: Accuracy per category, Average accuracy
|
||||
|
||||
## Data Format (CSV Fields)
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `Question` | string | The question text in Chinese |
|
||||
| `A` | string | Option A text |
|
||||
| `B` | string | Option B text |
|
||||
| `C` | string | Option C text |
|
||||
| `D` | string | Option D text |
|
||||
| `Answer` | string | Correct answer (A, B, C, or D) |
|
||||
|
||||
### Example Data
|
||||
|
||||
```csv
|
||||
Question,A,B,C,D,Answer
|
||||
关于生命价值理论的理解，以下哪一项表述是不正确的？,补偿生命经济价值可能受到的损失...,个人预期收入的货币价值...,任何触及个人收入能力的事件...,早逝、残疾、退休或失业可能导致...,B
|
||||
```
|
||||
|
||||
## Data Split Strategy
|
||||
|
||||
The dataset uses an end-based split strategy:
|
||||
|
||||
- **Test set**: Takes from the END of each category (up to 50 samples per category, or 50% if fewer)
|
||||
- **Train set**: Takes the remaining samples from the START
|
||||
|
||||
This ensures consistent train/test separation across all categories.
|
||||
|
||||
## Category Distribution Analysis
|
||||
|
||||
**Important**: Sample distribution is highly imbalanced:
|
||||
|
||||
| Category | Samples | % of Total | Note |
|
||||
|----------|---------|------------|------|
|
||||
| CPA | 1,211 | 19.6% | Largest |
|
||||
| Banking | 1,144 | 18.5% | |
|
||||
| Securities | 1,076 | 17.4% | |
|
||||
| Fund | 772 | 12.5% | |
|
||||
| Insurance | 596 | 9.6% | |
|
||||
| Economist | 420 | 6.8% | |
|
||||
| Tax | 388 | 6.3% | |
|
||||
| Futures | 333 | 5.4% | |
|
||||
| Financial Planner | 195 | 3.2% | Small |
|
||||
| **Actuary** | **44** | **0.7%** | **Critically small** |
|
||||
|
||||
**Recommendation**: When generating training data, ensure balanced sampling across categories, especially for Actuary (精算师) which has only 44 samples.
|
||||
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: Raw data contains only Q&A pairs, no reasoning chains.
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | N/A (no CoT) |
|
||||
| task_type | finance reasoning |
|
||||
| polish_difficulty | medium |
|
||||
|
||||
**Baseline**: Questions are multiple-choice format without explanations. **You MUST generate CoT** (chain-of-thought reasoning) for training samples to achieve good results.
|
||||
|
||||
## Baseline Performance
|
||||
|
||||
| Model | Accuracy |
|
||||
|-------|----------|
|
||||
| Qwen2.5-7B-Instruct (zero-shot) | ~65% |
|
||||
|
||||
**Note**: The Actuary (精算师-金融数学) category is particularly challenging, with baseline accuracy around 27-36%.
|
||||
|
||||
## License
|
||||
|
||||
CC-BY-NC-SA-4.0 License
|
||||
@@ -0,0 +1 @@
|
||||
from .split import get_split_indices, split_financeiq_dataset
|
||||
@@ -0,0 +1,64 @@
|
||||
import csv
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
|
||||
def get_split_indices(
|
||||
total_count: int, split: Literal["train", "test"], test_limit: int = 100, test_ratio: float = 0.5
|
||||
) -> slice:
|
||||
"""
|
||||
Calculate the slice for train/test split.
|
||||
|
||||
Rule:
|
||||
- Test set size = min(total_count * test_ratio, test_limit)
|
||||
- Test set takes from the END of the data.
|
||||
- Train set takes the rest (from the START).
|
||||
"""
|
||||
test_count = min(int(math.ceil(total_count * test_ratio)), test_limit)
|
||||
|
||||
if split == "test":
|
||||
return slice(total_count - test_count, total_count)
|
||||
else:
|
||||
return slice(0, total_count - test_count)
|
||||
|
||||
|
||||
def split_financeiq_dataset(data_dir: str, split: Literal["train", "test"]) -> None:
|
||||
"""
|
||||
Iterate over CSV files in the directory and apply the split in-place.
|
||||
"""
|
||||
path = Path(data_dir)
|
||||
|
||||
# Process CSV files
|
||||
for f in list(path.rglob("*.csv")):
|
||||
# HACK:
|
||||
# FinanceIQ specific: 'dev' folder is small and used for few-shot.
|
||||
# We preserve it for benchmarking (split='test') but remove for training (split='train') to avoid leakage.
|
||||
# Some times, the training in debug mode of llama factory will only check few samples. Which may results in failures
|
||||
rel_parts = f.relative_to(path).parts
|
||||
if "dev" in rel_parts:
|
||||
if split == "train":
|
||||
f.unlink()
|
||||
continue
|
||||
|
||||
rows = []
|
||||
header = None
|
||||
# Use 'utf-8-sig' to handle potential BOM in Excel-saved CSVs, or just 'utf-8'
|
||||
# Assuming 'utf-8' for now as it's standard for HF datasets
|
||||
with open(f, "r", encoding="utf-8", newline="") as fp:
|
||||
reader = csv.reader(fp)
|
||||
try:
|
||||
header = next(reader)
|
||||
rows = list(reader)
|
||||
except StopIteration:
|
||||
# Empty file
|
||||
continue
|
||||
|
||||
indices = get_split_indices(len(rows), split)
|
||||
new_rows = rows[indices]
|
||||
|
||||
with open(f, "w", encoding="utf-8", newline="") as fp:
|
||||
writer = csv.writer(fp)
|
||||
if header:
|
||||
writer.writerow(header)
|
||||
writer.writerows(new_rows)
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: cc-by-nc-4.0
|
||||
tags:
|
||||
- patent
|
||||
- legal
|
||||
- retrieval
|
||||
- classification
|
||||
size_categories:
|
||||
- 100K<n<1M
|
||||
task_categories:
|
||||
- text-classification
|
||||
- question-answering
|
||||
---
|
||||
|
||||
# PANORAMA Dataset
|
||||
|
||||
Patent examination benchmark capturing decision trails and rationales from [PANORAMA](https://huggingface.co/datasets/LG-AI-Research/PANORAMA).
|
||||
|
||||
**Repository**: [LG-AI-Research/PANORAMA](https://huggingface.co/datasets/LG-AI-Research/PANORAMA)
|
||||
|
||||
## Tasks
|
||||
|
||||
### 1. PAR4PC: Prior-Art Retrieval for Patent Claims
|
||||
|
||||
**Task**: Multi-label classification - select relevant prior-art documents from 8 candidates.
|
||||
|
||||
**Train samples**: 54,028
|
||||
|
||||
**Metrics**: Exact Match Accuracy, Custom Score (partial credit)
|
||||
|
||||
### 2. PI4PC: Paragraph Identification for Patent Claims
|
||||
|
||||
**Task**: Single-choice - identify the most relevant paragraph in a prior-art document.
|
||||
|
||||
**Train samples**: 64,210
|
||||
|
||||
**Metrics**: Exact Match Accuracy
|
||||
|
||||
### 3. NOC4PC: Novelty and Non-Obviousness Classification
|
||||
|
||||
**Task**: Ternary classification - determine if a claim should be ALLOW, 102 rejection, or 103 rejection.
|
||||
|
||||
**Train samples**: 136,211
|
||||
|
||||
**Metrics**: Macro F1-score, Per-class Accuracy
|
||||
|
||||
## Legal Background
|
||||
|
||||
- **35 U.S.C. §102 (Novelty)**: Claim rejected if anticipated by a single prior art reference
|
||||
- **35 U.S.C. §103 (Non-Obviousness)**: Claim rejected if obvious from combining prior art
|
||||
|
||||
## Data Format (Parquet Fields)
|
||||
|
||||
### PAR4PC / PI4PC Format
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `application_number` | str | Patent application identifier |
|
||||
| `claim_number` | int64 | Specific claim number being evaluated |
|
||||
| `context` | dict | Patent context: `{abstract: str, claims: list[str], title: str}` |
|
||||
| `options` | dict | 8 candidate documents: `{A: {abstract, claims, patent_id, title}, B: {...}, ...}` |
|
||||
| `gold_answers` | ndarray | Correct answer labels, e.g. `array(['G'])` or `array(['A', 'C'])` |
|
||||
| `silver_answers` | ndarray | Partially correct answers |
|
||||
| `negative_answers` | ndarray | Incorrect options |
|
||||
|
||||
**Note**: PI4PC has an additional `prior_art_specification` field containing the relevant prior-art document text.
|
||||
|
||||
### NOC4PC Format
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `application_number` | str | Patent application identifier |
|
||||
| `claim_number` | int64 | Specific claim number being evaluated |
|
||||
| `context` | dict | Patent context: `{abstract: str, claims: list[str], title: str}` |
|
||||
| `prior_art_specifications` | list | Prior art document specifications |
|
||||
| `answer` | str | Classification label: `ALLOW`, `102`, or `103` |
|
||||
|
||||
**Important**: Array fields (`gold_answers`, `silver_answers`, `negative_answers`) are `numpy.ndarray` type.
|
||||
Use `.tolist()` to convert to Python list before processing.
|
||||
|
||||
### Example Data
|
||||
|
||||
```python
|
||||
{
|
||||
"application_number": 14281639,
|
||||
"claim_number": 1,
|
||||
"context": {
|
||||
"abstract": "In an endodontic procedure...",
|
||||
"claims": ["claim 1 text", "claim 2 text", ...],
|
||||
"title": "Method for irrigating root canals"
|
||||
},
|
||||
"options": {
|
||||
"A": {"abstract": "...", "claims": [...], "patent_id": "US1234567", "title": "..."},
|
||||
"B": {"abstract": "...", "claims": [...], "patent_id": "US2345678", "title": "..."},
|
||||
# ... G, H
|
||||
},
|
||||
"gold_answers": array(['G'], dtype=object), # numpy.ndarray, use .tolist() -> ['G']
|
||||
"negative_answers": array(['A', 'B', 'C', 'D', 'E', 'F', 'H'], dtype=object)
|
||||
}
|
||||
```
|
||||
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: This dataset does NOT contain CoT annotations.
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | N/A (no CoT) |
|
||||
| task_type | legal reasoning |
|
||||
| polish_difficulty | high |
|
||||
|
||||
**Baseline**: Raw data contains rejection reasons but NO step-by-step reasoning chains. Paper explicitly states *"lacked ground-truth CoTs"*. **You MUST generate CoT** for all samples before training.
|
||||
|
||||
## Baseline Performance (CoT Prompting)
|
||||
|
||||
| Task | Best Model | Score |
|
||||
|------|-----------|-------|
|
||||
| PAR4PC | Gemma-3-12B | 77.30% |
|
||||
| PI4PC | GPT-4o | 62.62% |
|
||||
| NOC4PC | Claude-3.7-Sonnet | 45.40% |
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@article{panorama2024,
|
||||
title={PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination},
|
||||
author={LG AI Research and KAIST},
|
||||
year={2024},
|
||||
url={https://huggingface.co/datasets/LG-AI-Research/PANORAMA}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
CC-BY-NC-4.0 License
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
size_categories:
|
||||
- 1K<n<10K
|
||||
license: mit
|
||||
configs:
|
||||
- config_name: test
|
||||
data_files:
|
||||
- split: test
|
||||
path: data/test-*
|
||||
splits:
|
||||
- name: test
|
||||
num_examples: 886
|
||||
- config_name: train
|
||||
data_files:
|
||||
- split: train
|
||||
path: data/train-*
|
||||
splits:
|
||||
- name: train
|
||||
num_examples: ~10K
|
||||
---
|
||||
|
||||
# TableBench: Table Question Answering Dataset
|
||||
|
||||
Dataset for TableBench: A Comprehensive and Complex Benchmark for Table Question Answering.
|
||||
|
||||
> TableBench is a **comprehensive** and **complex** benchmark designed to evaluate Table Question Answering (TableQA) capabilities, covering **18 question categories** across **4 major categories** with **886** carefully curated test cases.
|
||||
|
||||
## Overview
|
||||
|
||||
The **TableBench dataset** consists of two main components:
|
||||
|
||||
1. **TableBench (Test)**: 886 high-quality test cases for evaluation across 4 major reasoning categories
|
||||
2. **TableInstruct (Train)**: Large-scale training dataset with diverse table QA examples
|
||||
|
||||
TableBench substantially pushes the boundaries of large language models in complex TableQA scenarios, aligning closely with the "Reasoning Complexity of Questions" dimension in real-world Table QA applications.
|
||||
|
||||
### Task Categories
|
||||
|
||||
The benchmark covers **4 major categories** with **18 sub-tasks**:
|
||||
|
||||
1. **Fact Checking**: Verify factual statements against table data
|
||||
- Simple fact verification, cross-table validation, temporal consistency
|
||||
|
||||
2. **Numerical Reasoning**: Mathematical computations and comparisons
|
||||
- Arithmetic operations, aggregations, comparative analysis
|
||||
|
||||
3. **Data Analysis**: Complex analytical reasoning
|
||||
- Impact analysis, correlation analysis, trend forecasting, statistical analysis
|
||||
|
||||
4. **Visualization**: Chart generation and interpretation
|
||||
- Bar charts, line charts, pie charts, scatter plots
|
||||
|
||||
### Data Sources
|
||||
|
||||
**Test Data (TableBench)**:
|
||||
- Repository: [Multilingual-Multimodal-NLP/TableBench](https://huggingface.co/datasets/Multilingual-Multimodal-NLP/TableBench)
|
||||
- 886 carefully curated and verified test cases
|
||||
- Enhanced version released April 2025 with error corrections
|
||||
|
||||
**Train Data (TableInstruct)**:
|
||||
- Repository: [Multilingual-Multimodal-NLP/TableInstruct](https://huggingface.co/datasets/Multilingual-Multimodal-NLP/TableInstruct)
|
||||
- Large-scale instruction tuning dataset for table QA
|
||||
- Diverse question types and reasoning patterns
|
||||
|
||||
### Data Fields
|
||||
|
||||
The dataset contains the following key fields:
|
||||
|
||||
The TableInstruct dataset contains the following fields:
|
||||
|
||||
- `id` (string): Unique identifier for each sample
|
||||
- `qtype` (string): Major task category (4 values)
|
||||
- `FactChecking`, `NumericalReasoning`, `DataAnalysis`, `Visualization`
|
||||
- `qsubtype` (string): Specific sub-task type (18 values)
|
||||
- Examples: `Counting`, `Aggregation`, `Comparison`, `CorrelationAnalysis`, etc.
|
||||
- `instruction` (string): Complete instruction template with task guidelines
|
||||
- Contains the full prompt template defining how to approach the task
|
||||
- Includes role definition, guidelines, code format requirements
|
||||
- Typically 800-15,000 characters depending on instruction type
|
||||
- `instruction_type` (string): Reasoning strategy type (4 values)
|
||||
- `DP` (Direct Prompting), `TCoT` (Textual Chain-of-Thought)
|
||||
- `PoT` (Program-of-Thought), `SCoT` (Structured Chain-of-Thought)
|
||||
- `table` (string): Table data in JSON format
|
||||
- Structure: `{"columns": [...], "data": [[...], [...], ...]}`
|
||||
- `question` (string): Specific question about the table
|
||||
- `response` (string): Model's answer including reasoning process
|
||||
|
||||
**TableBench Test Dataset Fields**:
|
||||
|
||||
- `question`: The table question or task description
|
||||
- `table`: The table data (JSON format)
|
||||
- `answer`: The ground truth answer
|
||||
- `category`: Major category
|
||||
- `subcategory`: Specific sub-task type
|
||||
|
||||
<!-- - `question`: The table question or task description
|
||||
- `table`: The table data (various formats: CSV, JSON, markdown)
|
||||
- `answer`: The ground truth answer or expected output
|
||||
- `category`: Major category (Fact Checking, Numerical Reasoning, Data Analysis, Visualization)
|
||||
- `subcategory`: Specific sub-task type
|
||||
- `reasoning_steps`: Optional chain-of-thought reasoning (for training data) -->
|
||||
|
||||
### Instruction Types and Reasoning Strategies
|
||||
Tablebench training data (TableInstruct) supports multiple instruction types content that define how the model approaches reasoning and generates answers. Understanding these types is crucial for dataset filtering and fine-tuning strategy selection.
|
||||
|
||||
### Available Instruction Type
|
||||
**1. Direct Prompting(DP)**
|
||||
**Characteristics**:
|
||||
- Provides solutions directly without intermediate reasoning steps
|
||||
- Simplest instruction format focused on immediate answer generation
|
||||
- Best for straightforward fact-checking and simple queries
|
||||
**Instruction Template Pattern**：
|
||||
You are a table analyst. Your task is to answer questions based on the table content.
|
||||
Read the table below in JSON format: [TABLE]
|
||||
Question: [QUESTION]
|
||||
Answer directly.
|
||||
**Response Format**:
|
||||
[Direct Answer]
|
||||
|
||||
**2. Textual Chain-of-Thought (TCoT)**
|
||||
**Characteristics**:
|
||||
- LLMs incrementally derive intermediate steps through textual reasoning
|
||||
- Natural language explanations for each reasoning step
|
||||
- Suitable for complex reasoning requiring logical deduction
|
||||
|
||||
**Instruction Template Pattern**:
|
||||
You are a table analyst. Your task is to answer questions based on the table content.
|
||||
[Guidelines for step-by-step reasoning]
|
||||
Think step by step
|
||||
Show your reasoning process
|
||||
Provide the final answer
|
||||
***Response Format**:
|
||||
Let's analyze this step by step:
|
||||
[First reasoning step]
|
||||
[Second reasoning step]
|
||||
...
|
||||
Final Answer: [Answer]
|
||||
|
||||
|
||||
#### 3. Program-of-Thought (PoT)
|
||||
|
||||
**Characteristics**:
|
||||
- Decomposes problems into executable Python code
|
||||
- Separates computation from reasoning using programming
|
||||
- Ideal for numerical reasoning and computational tasks
|
||||
- Most common type in TableInstruct for analytical tasks
|
||||
|
||||
**Instruction Template Pattern** (actual from dataset):
|
||||
You are a data analyst proficient in Python. Your task is to write executable Python
|
||||
code to analyze the table and then answer questions.
|
||||
[Guidelines]
|
||||
1. Based on the question, write out your analytical approach, then write Python code
|
||||
2. The code needs to be concise and easy to understand
|
||||
3. Code blocks need to strictly start with
|
||||
'''
|
||||
import pandas as pd
|
||||
df = pd.read_csv('table.csv')
|
||||
...
|
||||
print(f'Final Answer: {answer}')
|
||||
'''
|
||||
4.Your analysis must be based entirely on the above data
|
||||
5.Generate executable code with results using print function
|
||||
6.Ensure to load the table with: df = pd.read_csv('table.csv')
|
||||
|
||||
|
||||
#### 4. Symbolic Chain-of-Thought (SCoT)
|
||||
|
||||
**Characteristics**:
|
||||
- A methodology that utilizes Python-based instructions to facilitate logical reasoning
|
||||
- Combines symbolic reasoning with executable code verification
|
||||
- Three primary steps repeated until a definitive conclusion is derived
|
||||
- Distinguishes itself from PoT by emphasizing iterative analysis-generation-simulation cycles
|
||||
|
||||
**Three-Step Process**:
|
||||
- **STEP-1**: Analyzing the available information to determine the next move
|
||||
- **STEP-2**: Generating instructions using Python programming language commands
|
||||
- **STEP-3**: Simulating the outcomes by executing the instructions and analyzing the results
|
||||
|
||||
**Instruction Template Pattern**:
|
||||
You are a table analyst. Use symbolic reasoning with iterative Python commands.
|
||||
Process:
|
||||
STEP-1: Analyze available information to determine the next move
|
||||
STEP-2: Generate Python programming language commands
|
||||
STEP-3: Simulate outcomes by executing instructions and analyzing results
|
||||
Repeat these three steps until reaching a definitive conclusion
|
||||
|
||||
|
||||
|
||||
|
||||
### Evaluation Metrics
|
||||
|
||||
Different metrics are used based on task type:
|
||||
|
||||
| Task Type | Metric | Description |
|
||||
|-----------|--------|-------------|
|
||||
| Fact Checking | Exact Match (EM) | Exact match of predicted statement |
|
||||
| Numerical Reasoning | Exact Match (EM) | Correctness of numerical outputs |
|
||||
| Impact Analysis | Exact Match (EM) | Precise match of influential factors |
|
||||
| Correlation/Trend/Stats | EM_with_error_10 | ±10% numerical margin of error |
|
||||
| Other Data Analysis | ROUGE-L | For open-ended textual responses |
|
||||
| Visualization | Pass@1 | Correct chart generated on first attempt |
|
||||
|
||||
## CoT Quality Assessment
|
||||
|
||||
**IMPORTANT**: Consider enhancing reasoning chains during training preparation.
|
||||
|
||||
| Dimension | Value |
|
||||
|-----------|-------|
|
||||
| baseline_quality | medium-high |
|
||||
| task_type | table_qa |
|
||||
| polish_difficulty | medium |
|
||||
|
||||
**Baseline**: Training data (TableInstruct) contains reasoning examples, but test data focuses on final answers. For complex reasoning tasks (Data Analysis, Numerical Reasoning), generating detailed step-by-step CoT can significantly improve model performance.
|
||||
|
||||
**Recommendation**: For Data Analysis and Numerical Reasoning categories, expand reasoning chains to include:
|
||||
- Table understanding and schema identification
|
||||
- Step-by-step computation or logical reasoning
|
||||
- Intermediate results and verification
|
||||
- Final answer with confidence indicators
|
||||
|
||||
## Example
|
||||
|
||||
### Fact Checking
|
||||
```json
|
||||
{
|
||||
"question": "Based on the table, verify if the statement is true: 'Company A had higher revenue than Company B in Q4 2023'",
|
||||
"table": "| Company | Q4 2023 Revenue |\n|---------|----------------|\n| A | $2.5M |\n| B | $3.1M |",
|
||||
"answer": "False",
|
||||
"category": "Fact Checking",
|
||||
"subcategory": "simple_fact_verification"
|
||||
}
|
||||
```
|
||||
|
||||
### Numerical Reasoning
|
||||
```json
|
||||
{
|
||||
"question": "What is the total revenue across all quarters for Product X?",
|
||||
"table": "| Quarter | Product X Revenue |\n|---------|------------------|\n| Q1 | 150 |\n| Q2 | 200 |\n| Q3 | 175 |\n| Q4 | 225 |",
|
||||
"answer": "750",
|
||||
"category": "Numerical Reasoning",
|
||||
"subcategory": "aggregation"
|
||||
}
|
||||
```
|
||||
|
||||
### Data Analysis
|
||||
```json
|
||||
{
|
||||
"question": "Analyze the correlation between marketing spend and sales growth. What is the correlation coefficient?",
|
||||
"table": "| Month | Marketing ($K) | Sales Growth (%) |\n|-------|----------------|------------------|\n| Jan | 50 | 12 |\n| Feb | 75 | 18 |\n| Mar | 60 | 15 |",
|
||||
"answer": "0.95",
|
||||
"category": "Data Analysis",
|
||||
"subcategory": "correlation_analysis"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## License
|
||||
|
||||
This dataset is released under the MIT License.
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
"""
|
||||
LLM Fine-tuning Experiment Feedback Generation
|
||||
|
||||
Provides feedback analysis for LLM fine-tuning experiments, including
|
||||
model performance evaluation, training metrics analysis, and improvement suggestions.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.core.proposal import (
|
||||
Experiment2Feedback,
|
||||
ExperimentFeedback,
|
||||
HypothesisFeedback,
|
||||
)
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.log.utils import dict_get_with_warning
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.scenarios.finetune.experiment.experiment import FTExperiment
|
||||
from rdagent.scenarios.finetune.proposal.proposal import FTHypothesis
|
||||
from rdagent.scenarios.finetune.proposal.trace import FTTrace
|
||||
from rdagent.utils import convert2bool
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class FTExperiment2Feedback(Experiment2Feedback):
|
||||
"""Generate feedback for LLM fine-tuning experiments"""
|
||||
|
||||
def __init__(self, scen: Scenario, version: str = "exp_feedback") -> None:
|
||||
super().__init__(scen)
|
||||
self.version = version
|
||||
|
||||
def generate_feedback(
|
||||
self, exp: FTExperiment, trace: FTTrace | None = None, exception: Exception | None = None
|
||||
) -> ExperimentFeedback:
|
||||
"""
|
||||
Generate comprehensive feedback for LLM fine-tuning experiment.
|
||||
|
||||
Args:
|
||||
exp: The experiment to analyze
|
||||
trace: Experiment trace (optional)
|
||||
exception: If provided, indicates experiment failed and contains error details
|
||||
|
||||
Note: If exception is None, it means training succeeded and we evaluate quality/effectiveness.
|
||||
If exception is provided, we analyze the failure cause.
|
||||
"""
|
||||
# Get task information
|
||||
task_desc = exp.sub_tasks[0].get_task_information()
|
||||
|
||||
# Initialize for SOTA update logic later
|
||||
sota_benchmark = None
|
||||
|
||||
if exception is not None:
|
||||
# Error case: use error analysis prompt
|
||||
version = "exp_feedback_error"
|
||||
error_info = str(exception)
|
||||
|
||||
# Try to get FTRunnerEvaluator's analysis result from workspace
|
||||
# This contains structured feedback (execution, return_checking, code) instead of raw error string
|
||||
runner_feedback = None
|
||||
if exp.sub_workspace_list:
|
||||
for ws in exp.sub_workspace_list:
|
||||
if ws and hasattr(ws, "feedback") and ws.feedback:
|
||||
runner_feedback = ws.feedback
|
||||
break
|
||||
|
||||
if runner_feedback:
|
||||
# Use FTRunnerEvaluator's structured analysis result
|
||||
error_info = f"""## Execution Analysis
|
||||
{runner_feedback.execution}
|
||||
|
||||
## Return Checking
|
||||
{runner_feedback.return_checking}
|
||||
|
||||
## Code Analysis
|
||||
{runner_feedback.code}"""
|
||||
|
||||
system_prompt = T(f".prompts:{version}.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
)
|
||||
# Get workspace files safely
|
||||
workspace_files = {}
|
||||
if hasattr(exp, "experiment_workspace") and exp.experiment_workspace is not None:
|
||||
workspace_files = exp.experiment_workspace.file_dict
|
||||
user_prompt = T(f".prompts:{version}.user").r(
|
||||
hypothesis=exp.hypothesis,
|
||||
task_desc=task_desc,
|
||||
workspace_files=workspace_files,
|
||||
error_info=error_info,
|
||||
)
|
||||
else:
|
||||
# Success case: use normal feedback prompt
|
||||
version = self.version
|
||||
# Process experiment result - handle both new and legacy formats
|
||||
exp_result = exp.experiment_workspace.running_info.result
|
||||
if isinstance(exp_result, dict) and "benchmark" in exp_result:
|
||||
# New format: contains benchmark and training_metrics
|
||||
benchmark = exp_result.get("benchmark", {})
|
||||
raw_metrics = exp_result.get("training_metrics", {})
|
||||
# Pass loss_history directly (simpler and preserves full information)
|
||||
loss_history = raw_metrics.get("loss_history", {"train": [], "eval": []})
|
||||
# Sample train entries if too many to avoid token bloat
|
||||
if len(loss_history.get("train", [])) > 60:
|
||||
loss_history["train"] = loss_history["train"][:30] + loss_history["train"][-30:]
|
||||
training_metrics = (
|
||||
{"loss_history": loss_history} if (loss_history.get("train") or loss_history.get("eval")) else {}
|
||||
)
|
||||
else:
|
||||
# Legacy format: exp_result is directly the benchmark result (list of dicts)
|
||||
benchmark = {"accuracy_summary": exp_result, "error_samples": []}
|
||||
training_metrics = {}
|
||||
|
||||
# Get SOTA experiment's benchmark results for comparison
|
||||
sota_benchmark = trace.sota_benchmark() if trace else None
|
||||
|
||||
# Get baseline benchmark (always exists, computed at scenario init)
|
||||
baseline_benchmark = getattr(self.scen, "baseline_benchmark_score", None)
|
||||
|
||||
system_prompt = T(f".prompts:{version}.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
has_sota=sota_benchmark is not None,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
user_prompt = T(f".prompts:{version}.user").r(
|
||||
hypothesis=exp.hypothesis,
|
||||
task_desc=task_desc,
|
||||
workspace_files=exp.experiment_workspace.file_dict,
|
||||
execution_time=exp.experiment_workspace.running_info.running_time,
|
||||
benchmark=benchmark,
|
||||
training_metrics=training_metrics,
|
||||
sota_benchmark=sota_benchmark,
|
||||
baseline_benchmark=baseline_benchmark,
|
||||
)
|
||||
|
||||
resp_dict = json.loads(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=True,
|
||||
json_target_type=Dict[str, str | bool | int],
|
||||
)
|
||||
)
|
||||
|
||||
# Extract feedback components
|
||||
error_type = resp_dict.get("Error Type") if exception is not None else None
|
||||
hypothesis_feedback = HypothesisFeedback(
|
||||
code_change_summary=dict_get_with_warning(resp_dict, "Code Summary", "No code summary provided"),
|
||||
reason=dict_get_with_warning(resp_dict, "Reason", "No reasoning provided"),
|
||||
decision=convert2bool(dict_get_with_warning(resp_dict, "Decision", "no")),
|
||||
acceptable=exception is None, # Only acceptable if no error
|
||||
observations=error_type, # Store error type for history display
|
||||
)
|
||||
|
||||
return hypothesis_feedback
|
||||
@@ -0,0 +1,234 @@
|
||||
exp_feedback:
|
||||
system: |-
|
||||
You are an expert AI assistant specializing in analyzing LLM fine-tuning experiments.
|
||||
|
||||
Below is the scenario context for the current LLM fine-tuning task:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze the LLM fine-tuning experiment's hypothesis, implementation, and execution results to provide comprehensive feedback.
|
||||
Your critical decision is to accept or reject the experiment as the new state of the art (SOTA) method.
|
||||
|
||||
# Decision Making Framework:
|
||||
## Step 0: Pre-definition
|
||||
- The user has proposed a hypothesis for fine-tuning a specific base model. Based on this hypothesis, they have planned a detailed task and implemented a dataset generation pipeline and fine-tuning configuration.
|
||||
- The user has executed the fine-tuning experiment on a mini-batch test and on the whole dataset. The execution was successful.
|
||||
- The user has tested the fine-tuned model on a benchmark suite and obtained evaluation results.
|
||||
|
||||
## Step 1: Benchmark Metrics Evaluation (HIGHEST PRIORITY)
|
||||
**This is the most critical step. Benchmark performance is the primary decision factor.**
|
||||
- The user will provide you the benchmark evaluation results after executing the fine-tuned model on a benchmark suite.
|
||||
{% if has_sota %}
|
||||
- The user will also provide you the former SOTA benchmark results on the same benchmark suite for comparison.
|
||||
- If the current experiment **exceeds SOTA on the primary metrics**, this is a strong signal to ACCEPT.
|
||||
- If the results are significantly worse than SOTA, reject with [Benchmark Performance Issue].
|
||||
{% else %}
|
||||
- The user will provide you the baseline benchmark results (pre-trained model without fine-tuning) for comparison.
|
||||
- If the current experiment **exceeds baseline**, this is a strong signal to ACCEPT.
|
||||
- If the results are worse than or equal to baseline, reject with [Benchmark Performance Issue].
|
||||
{% endif %}
|
||||
|
||||
## Step 2: Code Quality Assessment
|
||||
- Evaluate the implementation quality and best practices
|
||||
- Compare the implementation against sota methods. If the implementation is significantly worse than sota methods, reject the experiment and start your reason by: [Implementation Quality Issue].
|
||||
|
||||
## Step 3: Final Decision (Acceptance as SOTA)
|
||||
You MUST determine the "Decision" (yes/no) based on the following:
|
||||
|
||||
{% if has_sota %}
|
||||
**Compare with SOTA**
|
||||
- **Primary rule**: If benchmark results exceed SOTA → Decision: "yes"
|
||||
- Consider metrics comprehensively, but prioritize actual performance over hypothesis alignment
|
||||
- Set "Decision": "no" only if SOTA is still better on the primary metrics
|
||||
{% else %}
|
||||
**Compare with BASELINE (no SOTA yet)**
|
||||
- **Primary rule**: If benchmark results exceed baseline → Decision: "yes"
|
||||
- The baseline results will be provided in the user prompt
|
||||
- Set "Decision": "no" only if results are worse than or equal to baseline
|
||||
{% endif %}
|
||||
- A config that "doesn't match hypothesis" but produces better results is still a valid finding worth accepting.
|
||||
|
||||
# Core improvement identification
|
||||
## Failure identification (On rejection)
|
||||
- The user has provided you the hypothesis, task description, implementation code, execution logs, and benchmark results. You should analyze them and provide an explaination in depth.
|
||||
- Identify the main cause of failure. Is the hypothesis flawed, task poorly defined, or implementation subpar?
|
||||
- Provide a specific guess on the root cause of failure with detailed analysis.
|
||||
- Put your analysis in the "reason" field of your final response.
|
||||
|
||||
## Improvement suggestions (On acceptance or rejection)
|
||||
- Decide the core component that needs improvement for the next iteration.
|
||||
- Suggest specific improvements or alternative approaches.
|
||||
- Put your suggestions in the "reason" field of your final response.
|
||||
|
||||
# Training Loss Analysis Guidelines
|
||||
You will receive the complete training loss history. Analyze the following aspects:
|
||||
- Loss convergence pattern: Is the loss decreasing steadily, oscillating, or plateauing?
|
||||
- Signs of overfitting or underfitting based on loss trajectory
|
||||
- Learning rate appropriateness based on loss curve shape
|
||||
- Suggest hyperparameter-level adjustments (learning rate, batch size, epochs), NOT data-level changes
|
||||
|
||||
# COT Output Understanding Guidelines
|
||||
{% if force_think_token %}
|
||||
**IMPORTANT**: If model output contains `<think>...</think>` tags, this is NORMAL and EXPECTED.
|
||||
|
||||
- During benchmark evaluation, a postprocessor REMOVES `<think>...</think>` content
|
||||
- The evaluator ONLY sees content AFTER `</think>`
|
||||
- Having `<think>` tags is correct CoT training behavior, NOT an error
|
||||
{% endif %}
|
||||
{# When force_think_token=false, model output won't have <think> tags, no special explanation needed #}
|
||||
|
||||
# Error Sample Analysis Guidelines (CRITICAL - Avoid Benchmark Leakage)
|
||||
You will receive model outputs for incorrectly answered questions.
|
||||
**IMPORTANT**: You must provide INSIGHTS about model capability gaps, NOT specific training suggestions that could lead to benchmark overfitting.
|
||||
|
||||
**DO:**
|
||||
- Identify error patterns (e.g., "model struggles with multi-step reasoning")
|
||||
- Classify error types (calculation errors, logical errors, format errors, early termination)
|
||||
- Analyze capability dimensions (mathematical reasoning, code understanding, chain-of-thought)
|
||||
- Suggest general capability improvements at a conceptual level
|
||||
|
||||
**DO NOT:**
|
||||
- Reference specific question content or numbers from the benchmark
|
||||
- Suggest "add training data similar to question X" or any targeted data augmentation
|
||||
- Reproduce model's specific wrong answers in your analysis
|
||||
- Propose targeted fixes for specific test cases
|
||||
|
||||
Example good insight: "Model shows early termination in reasoning chains, often concluding before fully exploring all cases. This suggests insufficient training on long-form reasoning tasks."
|
||||
Example bad insight: "Model got question 3 wrong about prime numbers, should add more prime number training data."
|
||||
|
||||
# Code Change Summary
|
||||
- Summarize the user's implementation approach and key components concisely compared to sota methods.
|
||||
|
||||
Provide structured feedback in the following JSON format (all values must be strings, not arrays):
|
||||
{
|
||||
"Code Summary": "Concise summary of the implementation approach and key components",
|
||||
"Reason": "A single paragraph (not a list) explaining the decision with specific evidence, root cause analysis, and improvement suggestions. Limit to 3-5 sentences.",
|
||||
"Decision": "yes or no - whether this experiment should be accepted as the new SOTA (see Step 3)"
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Current LLM Fine-tuning Experiment Analysis
|
||||
|
||||
## Hypothesis
|
||||
{{ hypothesis }}
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Workspace Files
|
||||
{% for file_name, file_content in workspace_files.items() %}
|
||||
- {{ file_name }}: {{ file_content }}
|
||||
{% endfor %}
|
||||
|
||||
**Execution Time**: {{ execution_time }} seconds
|
||||
|
||||
## Training Metrics
|
||||
{% if training_metrics %}
|
||||
```json
|
||||
{{ training_metrics | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No training metrics available.
|
||||
{% endif %}
|
||||
|
||||
## Benchmark Results
|
||||
### Accuracy Summary
|
||||
{% if benchmark.accuracy_summary %}
|
||||
```json
|
||||
{{ benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No accuracy summary available.
|
||||
{% endif %}
|
||||
|
||||
### Error Sample Analysis ({{ benchmark.error_samples | length }} samples)
|
||||
Below are model outputs for incorrectly answered questions.
|
||||
Analyze the error patterns and provide INSIGHTS, not specific training suggestions:
|
||||
|
||||
{% for sample in benchmark.error_samples %}
|
||||
**Error {{ loop.index }}:**
|
||||
- Question: {{ sample.question[:1000] }}{% if sample.question | length > 1000 %}... (truncated){% endif %}
|
||||
- Expected Answer: {{ sample.gold }}
|
||||
- Model Output: {{ sample.model_output[:500] }}{% if sample.model_output | length > 500 %}... (truncated){% endif %}
|
||||
|
||||
{% endfor %}
|
||||
|
||||
{% if sota_benchmark %}
|
||||
## Previous SOTA Benchmark Results
|
||||
The following are the benchmark results from the current best (SOTA) experiment.
|
||||
Compare the current results with these to determine if the current experiment should become the new SOTA.
|
||||
|
||||
### SOTA Accuracy Summary
|
||||
{% if sota_benchmark.accuracy_summary %}
|
||||
```json
|
||||
{{ sota_benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No SOTA accuracy summary available.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
## Baseline Benchmark Results (Pre-trained Model)
|
||||
**No SOTA exists yet.** Compare against the BASELINE (model performance before fine-tuning).
|
||||
**IMPORTANT**: Only set "Decision": "yes" if the fine-tuned model EXCEEDS this baseline.
|
||||
|
||||
### Baseline Accuracy Summary
|
||||
```json
|
||||
{{ baseline_benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
exp_feedback_error:
|
||||
system: |-
|
||||
You are an expert LLM fine-tuning debugger specializing in analyzing experiment failures.
|
||||
|
||||
Below is the scenario context:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze why the LLM fine-tuning experiment failed and provide actionable feedback.
|
||||
|
||||
# Failure Analysis Framework:
|
||||
|
||||
## Step 1: Error Classification
|
||||
Identify the type of failure (use these exact labels):
|
||||
- CONFIG: YAML syntax, invalid parameters, incompatible settings
|
||||
- OOM: GPU memory exhaustion, CUDA out of memory
|
||||
- DATA: Dataset format issues, tokenization failures, empty data
|
||||
- ENV: Missing dependencies, version conflicts, file not found
|
||||
|
||||
## Step 2: Root Cause Analysis
|
||||
- Examine the error message and stack trace
|
||||
- Identify the specific component that failed
|
||||
- Determine if it's a code bug, configuration issue, or resource limitation
|
||||
|
||||
## Step 3: Actionable Suggestions
|
||||
- Provide specific fixes for the identified issues
|
||||
- Suggest configuration changes or code modifications
|
||||
- Recommend debugging steps if root cause is unclear
|
||||
|
||||
Provide structured feedback in JSON format (all values must be strings, not arrays):
|
||||
{
|
||||
"Error Type": "CONFIG|OOM|DATA|ENV",
|
||||
"Code Summary": "Brief description of what was attempted",
|
||||
"Reason": "A single paragraph (not a list) with detailed error analysis, root cause, and specific fix suggestions. Limit to 3-5 sentences.",
|
||||
"Decision": "no"
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Failed LLM Fine-tuning Experiment Analysis
|
||||
|
||||
## Hypothesis
|
||||
{{ hypothesis }}
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Workspace Files
|
||||
{% for file_name, file_content in workspace_files.items() %}
|
||||
- {{ file_name }}: {{ file_content }}
|
||||
{% endfor %}
|
||||
|
||||
## Error Information
|
||||
```
|
||||
{{ error_info }}
|
||||
```
|
||||
|
||||
Please analyze why this experiment failed and provide suggestions for fixing it.
|
||||
@@ -0,0 +1,25 @@
|
||||
"""
|
||||
Hugging Face download utility module
|
||||
|
||||
Provides functions to download models and datasets from the Hugging Face Hub.
|
||||
|
||||
Main functions:
|
||||
- download_dataset: Download entire dataset repo using snapshot_download
|
||||
- download_model: Download model repo using snapshot_download
|
||||
|
||||
For high-level dataset management (with registered datasets), use:
|
||||
from rdagent.scenarios.finetune.datasets import prepare, prepare_all
|
||||
|
||||
Environment variable configuration:
|
||||
- HF_TOKEN / HUGGINGFACE_TOKEN / HUGGING_FACE_HUB_TOKEN: Hugging Face access token
|
||||
- FT_FILE_PATH: Root directory for finetuning files (managed by FT_RD_SETTING)
|
||||
|
||||
Usage example:
|
||||
from rdagent.scenarios.finetune.download.hf import download_dataset, download_model
|
||||
|
||||
# Download dataset
|
||||
dataset_path = download_dataset("OpenMol/ChemCoTDataset", "/path/to/chemcot")
|
||||
|
||||
# Download model
|
||||
model_path = download_model("Qwen/Qwen2.5-7B")
|
||||
"""
|
||||
@@ -0,0 +1,108 @@
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def _ensure_parent(path: Path) -> None:
|
||||
os.makedirs(path.parent, mode=0o777, exist_ok=True)
|
||||
|
||||
|
||||
def _get_hf_token(token: Optional[str] = None) -> Optional[str]:
|
||||
"""Get HuggingFace token from parameter or environment variables."""
|
||||
return (
|
||||
token
|
||||
or os.environ.get("HF_TOKEN")
|
||||
or os.environ.get("HUGGINGFACE_TOKEN")
|
||||
or os.environ.get("HUGGING_FACE_HUB_TOKEN")
|
||||
)
|
||||
|
||||
|
||||
def download_dataset(
|
||||
repo_id: str,
|
||||
out_dir: str,
|
||||
token: Optional[str] = None,
|
||||
revision: Optional[str] = None,
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""
|
||||
Download HuggingFace dataset to a specified directory using snapshot_download.
|
||||
Preserves the original file structure from HuggingFace.
|
||||
|
||||
Args:
|
||||
repo_id: HuggingFace dataset repository ID
|
||||
out_dir: Directory to save the dataset
|
||||
token: HuggingFace token for private datasets
|
||||
revision: Specific revision to download
|
||||
force: If True, re-download even if exists
|
||||
|
||||
Returns:
|
||||
Path to the downloaded dataset directory
|
||||
"""
|
||||
save_path = Path(out_dir)
|
||||
_ensure_parent(save_path)
|
||||
|
||||
if force and save_path.exists():
|
||||
shutil.rmtree(save_path)
|
||||
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
except Exception as e:
|
||||
raise ImportError(
|
||||
"huggingface_hub is missing. Please install it first: pip install -U 'huggingface_hub[cli]'"
|
||||
) from e
|
||||
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
repo_type="dataset",
|
||||
local_dir=str(save_path),
|
||||
local_dir_use_symlinks=False,
|
||||
token=_get_hf_token(token),
|
||||
revision=revision,
|
||||
)
|
||||
return str(save_path)
|
||||
|
||||
|
||||
def download_model(
|
||||
repo_id: str,
|
||||
out_dir_root: Optional[str] = None,
|
||||
token: Optional[str] = None,
|
||||
revision: Optional[str] = None,
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""
|
||||
Download Hugging Face model to a subdirectory under the specified root: <out_dir_root>/<repo_id>
|
||||
Returns the actual download directory path as a string.
|
||||
"""
|
||||
if out_dir_root:
|
||||
save_root = Path(out_dir_root)
|
||||
else:
|
||||
# Use FT_RD_SETTING for default root directory
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
|
||||
if not FT_RD_SETTING.file_path:
|
||||
raise ValueError("No out_dir_root specified and FT_FILE_PATH not set")
|
||||
save_root = Path(FT_RD_SETTING.file_path) / "model"
|
||||
|
||||
save_path = save_root / repo_id
|
||||
_ensure_parent(save_path)
|
||||
|
||||
if force and save_path.exists():
|
||||
shutil.rmtree(save_path)
|
||||
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
except Exception as e:
|
||||
raise ImportError(
|
||||
"huggingface_hub is missing. Please install it first: pip install -U 'huggingface_hub[cli]'"
|
||||
) from e
|
||||
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
repo_type="model",
|
||||
local_dir=str(save_path),
|
||||
local_dir_use_symlinks=False,
|
||||
token=_get_hf_token(token),
|
||||
revision=revision,
|
||||
)
|
||||
return str(save_path)
|
||||
@@ -0,0 +1,58 @@
|
||||
# LLaMA Factory Environment Requirements
|
||||
# Equivalent to: rdagent/scenarios/finetune/docker/llm_finetune_docker/Dockerfile
|
||||
# Docker base: hiyouga/llamafactory:0.9.4 uses PyTorch 2.6.0 + CUDA 12.4 + flash-attn 2.7.4
|
||||
|
||||
# PyTorch 2.9.0 with CUDA 12.8 (for B200 GPUs with sm_100 architecture)
|
||||
# Note: PyTorch 2.6.0 only supports up to sm_90, B200 requires 2.8.0+
|
||||
# For non-B200 machines with CUDA 12.4, change to cu124 and torch==2.6.0
|
||||
--index-url https://download.pytorch.org/whl/cu128
|
||||
torch==2.9.0
|
||||
torchvision==0.24.0
|
||||
|
||||
# Reset to default index for other packages
|
||||
--index-url https://pypi.org/simple
|
||||
|
||||
# Core LlamaFactory package (PyPI latest is 0.9.3, Docker uses 0.9.4 from GitHub)
|
||||
llamafactory==0.9.3
|
||||
|
||||
# FlashAttention-2: installed separately via llm_finetune_flash_attn.txt
|
||||
# (requires torch installed first, and --no-build-isolation flag)
|
||||
|
||||
# Transformers library (for tokenizer)
|
||||
transformers
|
||||
|
||||
# Additional dependencies (matches Dockerfile line 17)
|
||||
bitsandbytes>=0.39.0
|
||||
mixture-of-depth>=1.1.6
|
||||
litellm
|
||||
|
||||
# Common utilities for data processing scripts
|
||||
requests
|
||||
|
||||
# DeepSpeed for memory optimization
|
||||
# Note: LlamaFactory 0.9.3 requires deepspeed<=0.16.9 (hardcoded check in parser.py)
|
||||
deepspeed>=0.10.0,<=0.16.9
|
||||
|
||||
# LlamaFactory optional dependencies (commonly used)
|
||||
# Liger Kernel - fused triton kernels for training acceleration
|
||||
liger-kernel>=0.5.5
|
||||
|
||||
# Metrics for evaluation
|
||||
nltk
|
||||
jieba
|
||||
rouge-chinese
|
||||
|
||||
# Advanced optimizers
|
||||
galore-torch
|
||||
apollo-torch
|
||||
badam>=1.2.1
|
||||
adam-mini
|
||||
|
||||
# Quantization
|
||||
hqq
|
||||
|
||||
# FP8 training support
|
||||
torchao>=0.8.0
|
||||
|
||||
# Chemistry support
|
||||
rdkit
|
||||
@@ -0,0 +1,22 @@
|
||||
# OpenCompass Benchmark Environment Requirements
|
||||
# Equivalent to: rdagent/scenarios/finetune/docker/opencompass/Dockerfile
|
||||
|
||||
# PyTorch 2.9.0 with CUDA 12.8 (for B200 GPUs with sm_100 architecture)
|
||||
# Note: PyTorch 2.1.0 only supports up to sm_90, B200 requires 2.8.0+
|
||||
# For non-B200 machines with CUDA 12.4, change to cu124 and torch==2.6.0
|
||||
--index-url https://download.pytorch.org/whl/cu128
|
||||
torch==2.9.0
|
||||
torchvision==0.24.0
|
||||
|
||||
# Reset to default index for other packages
|
||||
--index-url https://pypi.org/simple
|
||||
|
||||
# vLLM for model inference (latest version supports PyTorch 2.9.0)
|
||||
vllm>=0.12.0
|
||||
|
||||
# OpenCompass benchmark framework (custom fork with cascade eval support)
|
||||
opencompass @ git+https://github.com/Jensen246/opencompass.git
|
||||
|
||||
# Math evaluation dependencies (matches Dockerfile line 22)
|
||||
math_verify
|
||||
latex2sympy2_extended
|
||||
@@ -0,0 +1,20 @@
|
||||
FROM hiyouga/llamafactory:0.9.4
|
||||
|
||||
# Set CUDA environment variables for DeepSpeed compilation
|
||||
ENV CUDA_HOME=/usr/local/cuda
|
||||
ENV PATH=$CUDA_HOME/bin:$PATH
|
||||
ENV LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
|
||||
|
||||
RUN apt-get clean && apt-get update && apt-get install -y \
|
||||
curl \
|
||||
vim \
|
||||
git \
|
||||
build-essential \
|
||||
git-lfs \
|
||||
unzip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install "bitsandbytes>=0.39.0" "mixture-of-depth>=1.1.6" "litellm"
|
||||
|
||||
# Set working directory for experiments
|
||||
WORKDIR /workspace
|
||||
@@ -0,0 +1,40 @@
|
||||
FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get clean && apt-get update && apt-get install -y \
|
||||
curl \
|
||||
vim \
|
||||
git \
|
||||
build-essential \
|
||||
git-lfs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Upgrade pip
|
||||
RUN pip install --upgrade pip setuptools wheel --no-cache-dir
|
||||
|
||||
# Install OpenCompass with vLLM backend support
|
||||
RUN git clone https://github.com/Jensen246/opencompass.git /opencompass
|
||||
WORKDIR /opencompass
|
||||
|
||||
RUN pip install ".[vllm]" --no-cache-dir
|
||||
|
||||
# Install math evaluation dependencies for AIME/MATH benchmarks
|
||||
RUN pip install math_verify latex2sympy2_extended --no-cache-dir
|
||||
|
||||
# Install peft and transformers for model merging
|
||||
RUN pip install peft transformers --no-cache-dir
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /workspace
|
||||
|
||||
# Set environment variables for cache directories
|
||||
ENV HF_HOME=/benchmarks/hf_cache
|
||||
ENV HF_HUB_CACHE=/benchmarks/hf_cache/hub
|
||||
ENV TRANSFORMERS_CACHE=/benchmarks/hf_cache/transformers
|
||||
ENV HF_DATASETS_CACHE=/benchmarks/datasets
|
||||
ENV COMPASS_DATA_CACHE=/benchmarks/opencompass_data
|
||||
|
||||
# Fix MKL threading layer compatibility issue with vLLM
|
||||
ENV MKL_SERVICE_FORCE_INTEL=1
|
||||
ENV MKL_THREADING_LAYER=GNU
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import re
|
||||
from typing import Literal
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from rdagent.components.coder.finetune.conf import FT_YAML_FILE_NAME
|
||||
from rdagent.core.experiment import Experiment, Task
|
||||
from rdagent.scenarios.finetune.experiment.workspace import FTWorkspace
|
||||
|
||||
COMPONENT = Literal["Training"]
|
||||
|
||||
|
||||
class FTExperiment(Experiment[Task, FTWorkspace, FTWorkspace]):
|
||||
def __init__(self, sub_tasks: list[Task], *args, **kwargs) -> None:
|
||||
super().__init__(sub_tasks=sub_tasks, *args, **kwargs)
|
||||
# Status
|
||||
# - Initial: blank;
|
||||
# - Injecting from SOTA code;
|
||||
# - New version no matter successful or not
|
||||
# the initial workspace or the successful new version after coding
|
||||
self.experiment_workspace = FTWorkspace()
|
||||
|
||||
self.format_check_result = None
|
||||
# this field is optional. It is not none only when we have a format checker. Currently, only following cases are supported.
|
||||
# - mle-bench
|
||||
|
||||
def is_ready_to_run(self) -> bool:
|
||||
"""
|
||||
ready to run does not indicate the experiment is runnable
|
||||
(so it is different from `trace.next_incomplete_component`.)
|
||||
"""
|
||||
return self.experiment_workspace is not None and FT_YAML_FILE_NAME in self.experiment_workspace.file_dict
|
||||
@@ -0,0 +1,104 @@
|
||||
"""
|
||||
FT-specific Workspace implementation with minimal checkpoint strategy.
|
||||
|
||||
This module provides FTWorkspace, which configures checkpoint to only save
|
||||
configuration files (train.yaml), excluding all training outputs.
|
||||
|
||||
Design Philosophy:
|
||||
- Checkpoint is for code version control during CoSTEER evolution
|
||||
- Model persistence is handled separately by Runner's save_model()
|
||||
- This separation keeps concerns clear and checkpoints lightweight
|
||||
"""
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from rdagent.components.coder.finetune.conf import FT_YAML_FILE_NAME
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.utils.env import CacheKeyFunc, DockerEnv, LocalEnv
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from rdagent.utils.env import Env
|
||||
|
||||
from rdagent.utils.env import EnvResult
|
||||
|
||||
|
||||
class FTWorkspace(FBWorkspace):
|
||||
"""
|
||||
Fine-tuning workspace with minimal checkpoint strategy and unified Docker logging.
|
||||
|
||||
Checkpoint Strategy:
|
||||
- Only saves configuration files (train.yaml) for version control
|
||||
- Training outputs (models, checkpoints) are excluded by design
|
||||
- Final model persistence is Runner's responsibility, not checkpoint's
|
||||
"""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
# Configure checkpoint to save essential files for training
|
||||
# Training outputs (models, checkpoints) are managed separately by save_final_model()
|
||||
RD_AGENT_SETTINGS.workspace_ckp_white_list_names = [
|
||||
FT_YAML_FILE_NAME, # train.yaml - training config
|
||||
"dataset_info.json", # LlamaFactory dataset config
|
||||
]
|
||||
RD_AGENT_SETTINGS.workspace_ckp_size_limit = 100 * 1024
|
||||
|
||||
def run(
|
||||
self,
|
||||
env: "Env",
|
||||
entry: str,
|
||||
env_vars: dict | None = None,
|
||||
cache_key_extra_func: CacheKeyFunc | None = None,
|
||||
cache_files_to_extract: list[str] | None = None,
|
||||
) -> "EnvResult":
|
||||
"""Execute the code in the environment with unified Docker logging.
|
||||
|
||||
Args:
|
||||
env: The environment to run in (DockerEnv, LocalEnv, etc.)
|
||||
entry: The command to execute
|
||||
env_vars: Optional additional environment variables (e.g., LLM API keys)
|
||||
Will be merged with default {"PYTHONPATH": "./"}
|
||||
cache_key_extra_func: Optional extra function for cache key calculation
|
||||
cache_files_to_extract: Optional list of files to extract from cache
|
||||
|
||||
Returns:
|
||||
EnvResult with stdout, exit_code, running_time
|
||||
"""
|
||||
self.prepare()
|
||||
self.inject_files(**self.file_dict)
|
||||
|
||||
# Merge default env with custom env_vars
|
||||
run_env = {"PYTHONPATH": "./"}
|
||||
if env_vars:
|
||||
run_env.update(env_vars)
|
||||
|
||||
result = env.run(
|
||||
entry,
|
||||
str(self.workspace_path),
|
||||
env=run_env,
|
||||
cache_key_extra_func=cache_key_extra_func,
|
||||
cache_files_to_extract=cache_files_to_extract,
|
||||
)
|
||||
|
||||
# Unified execution logging for FT scenario (supports both Docker and Conda)
|
||||
if isinstance(env, DockerEnv):
|
||||
tag_prefix = "docker_run"
|
||||
elif isinstance(env, LocalEnv):
|
||||
tag_prefix = "conda_run"
|
||||
else:
|
||||
tag_prefix = "env_run"
|
||||
|
||||
logger.log_object(
|
||||
{
|
||||
"exit_code": result.exit_code,
|
||||
"stdout": result.stdout or "",
|
||||
"running_time": result.running_time,
|
||||
"entry": entry,
|
||||
"workspace_path": str(self.workspace_path),
|
||||
},
|
||||
tag=f"{tag_prefix}.FTWorkspace",
|
||||
)
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,61 @@
|
||||
import asyncio
|
||||
from typing import Any
|
||||
|
||||
from rdagent.app.finetune.llm.conf import LLMFinetunePropSetting
|
||||
from rdagent.components.coder.finetune.conf import get_ft_env
|
||||
from rdagent.components.workflow.rd_loop import RDLoop
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.proposal import HypothesisFeedback
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.proposal.trace import FTTrace
|
||||
|
||||
|
||||
class LLMFinetuneRDLoop(RDLoop):
|
||||
"""LLM fine-tuning loop using standard RDLoop workflow"""
|
||||
|
||||
skip_loop_error = (CoderError,)
|
||||
withdraw_loop_error = ()
|
||||
|
||||
def __init__(self, PROP_SETTING: LLMFinetunePropSetting):
|
||||
# Store finetune-specific settings
|
||||
self.ft_rd_setting = PROP_SETTING
|
||||
self.dataset = PROP_SETTING.dataset
|
||||
self.model = PROP_SETTING.base_model
|
||||
|
||||
# Initialize using base class
|
||||
super().__init__(PROP_SETTING)
|
||||
|
||||
# Replace generic Trace with FTTrace for SOTA tracking
|
||||
self.trace = FTTrace(scen=self.trace.scen)
|
||||
|
||||
async def direct_exp_gen(self, prev_out: dict[str, Any]):
|
||||
"""Generate LLM fine-tuning experiment"""
|
||||
exp = await self.hypothesis_gen.async_gen(self.trace, self)
|
||||
logger.log_object(exp.hypothesis, tag="hypothesis")
|
||||
logger.log_object(exp.sub_tasks, tag="experiment generation")
|
||||
return exp
|
||||
|
||||
def coding(self, prev_out: dict[str, Any]):
|
||||
"""Generate fine-tuning code"""
|
||||
exp = prev_out["direct_exp_gen"]
|
||||
exp = self.coder.develop(exp)
|
||||
logger.log_object(exp.sub_workspace_list, tag="coder result")
|
||||
return exp
|
||||
|
||||
def feedback(self, prev_out: dict[str, Any]):
|
||||
"""Generate feedback for LLM fine-tuning experiment - always call LLM"""
|
||||
|
||||
# Get experiment from available sources
|
||||
exp = prev_out.get("running") or prev_out.get("coding") or prev_out.get("direct_exp_gen")
|
||||
e = prev_out.get(self.EXCEPTION_KEY, None)
|
||||
feedback = self.summarizer.generate_feedback(exp, self.trace, exception=e)
|
||||
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
return feedback
|
||||
|
||||
def record(self, prev_out: dict[str, Any]):
|
||||
"""Record the experiment and feedback into trace"""
|
||||
feedback = prev_out["feedback"]
|
||||
exp = prev_out.get("running") or prev_out.get("coding") or prev_out.get("direct_exp_gen")
|
||||
self.trace.sync_dag_parent_and_hist((exp, feedback), prev_out[self.LOOP_IDX_KEY])
|
||||
@@ -0,0 +1,359 @@
|
||||
# =============================================================================
|
||||
# Unified Hypothesis Generation
|
||||
# =============================================================================
|
||||
# Single prompt that covers both data processing and training configuration.
|
||||
# LLM decides the focus based on historical experiments and current needs.
|
||||
|
||||
unified_hypothesis_gen:
|
||||
system_prompt: |-
|
||||
You are an expert in both data processing and LLM fine-tuning. Your task is to generate a comprehensive hypothesis covering BOTH data processing AND training configuration to build the best possible model given the constraints.
|
||||
|
||||
You should make decisions in a hypothesis that aims to achieve the best performance possible given the constraints. Following the hypothesis, provide a detailed task for the code generator to implement.
|
||||
|
||||
The user might have historical experiments to learn from. Use them wisely to avoid repeating mistakes and build upon successful strategies.
|
||||
|
||||
# Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 1: DATA PROCESSING
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 1.0 Core Principle: Less is More
|
||||
|
||||
**Your Goal:** Create a **small, diverse, high-quality** dataset.
|
||||
|
||||
### The Three Rules
|
||||
|
||||
1. **Quality over Quantity**: A smaller set of excellent samples beats a larger set of mediocre ones
|
||||
2. **Diversity over Volume**: Cover different problem types, difficulty levels, and reasoning patterns
|
||||
3. **Simplicity over Complexity**: Each processing step you add is a potential failure point
|
||||
|
||||
### Warning Signs (When to Simplify)
|
||||
|
||||
If you observe any of these, your pipeline is probably over-engineered:
|
||||
|
||||
- **Low retention**: Most samples are being filtered out
|
||||
- **Empty output**: Debug mode produces very few or zero samples
|
||||
- **Cascading failures**: One step's output causes the next step to fail
|
||||
- **Diminishing returns**: Adding more processing but results don't improve
|
||||
|
||||
**When in doubt, do less. A simple pipeline that works beats a complex one that fails.**
|
||||
|
||||
## 1.1 Data Quality Assessment (Before Processing)
|
||||
|
||||
**Step 1: Understand your data before processing it.**
|
||||
|
||||
| Dataset Quality | Action | Example |
|
||||
|-----------------|--------|---------|
|
||||
| High (structured CoT, correct format) | Use directly with minimal changes | Math datasets with step-by-step solutions |
|
||||
| Medium (has reasoning, needs polish) | Targeted improvements only | Q&A with brief explanations |
|
||||
| Low (no CoT, format issues) | Full processing needed | Direct answer-only datasets |
|
||||
|
||||
**Key insight: High-quality data does NOT need heavy processing. Over-processing good data can degrade it.**
|
||||
|
||||
## 1.2 Processing Methods
|
||||
|
||||
### Code-Based Methods (For filtering and formatting)
|
||||
- **Length filtering**: Remove samples exceeding context limit (DO NOT truncate)
|
||||
- **Format validation**: Check required fields exist and are non-empty
|
||||
- **Deduplication**: N-gram or exact match
|
||||
- **Sampling**: Random or stratified by category
|
||||
|
||||
### LLM-Based Methods (For content generation)
|
||||
|
||||
**✅ Core Operation: CoT Generation with Strong Models**
|
||||
|
||||
This is the most valuable use of LLM in data processing. High-quality CoT is essential for training reasoning ability.
|
||||
|
||||
- **Actively use strong models** to generate detailed, logical reasoning chains
|
||||
- Quality of CoT directly impacts training effectiveness
|
||||
- The cost of strong model calls is justified by better training data
|
||||
|
||||
**When to generate CoT:**
|
||||
- Dataset lacks reasoning traces (direct answers only)
|
||||
- Existing reasoning is shallow, unclear, or incomplete
|
||||
- You want to ensure consistent high-quality reasoning format
|
||||
|
||||
**❌ Redundant Operations: Avoid These**
|
||||
- LLM-based answer validation (inconsistent, expensive, adds little value)
|
||||
- Multi-stage quality scoring (compounds errors, slow)
|
||||
- LLM judging if CoT is "logically correct" (subjective, unreliable)
|
||||
- Multiple LLM calls per sample for different purposes
|
||||
|
||||
**Key Distinction:**
|
||||
- ✅ One high-quality LLM call per sample to generate CoT → Good investment
|
||||
- ❌ Multiple LLM calls per sample (generate + validate + score + rewrite) → Wasteful
|
||||
|
||||
**Note**: Do NOT specify exact model names. Describe which tier (strong/weak) for each step. Model selection is automatic.
|
||||
|
||||
## 1.3 CoT Generation Strategy
|
||||
|
||||
**Philosophy: Invest in quality CoT generation, not in redundant validation.**
|
||||
|
||||
**CRITICAL: ALL training data MUST include Chain-of-Thought reasoning. No direct answers.**
|
||||
|
||||
**How to generate CoT:**
|
||||
1. **Use strong model tier** - this is where quality matters most
|
||||
2. Generate naturally - let the model reason step by step
|
||||
3. Don't request specific format tags in the prompt (models may refuse)
|
||||
4. Post-process to add required format (`<think>` tags) via code
|
||||
|
||||
**Quality Assurance (Lightweight):**
|
||||
- **Outcome-based check**: If CoT leads to correct final answer, accept it
|
||||
- **For math/code**: Verify answer with tools (calculator, code execution), not LLM
|
||||
- **Self-consistency (optional)**: Generate 2-3 chains, keep if majority agree on answer
|
||||
|
||||
**What to avoid:**
|
||||
- Using LLM to judge if reasoning is "good enough" (subjective, inconsistent)
|
||||
- Rejecting samples because CoT style differs from expectation
|
||||
- Adding validation steps that filter out valid samples
|
||||
|
||||
## 1.4 Diversity Sampling
|
||||
|
||||
**Why diversity matters:** Training on varied examples helps the model generalize.
|
||||
|
||||
**Implementation:**
|
||||
1. Identify natural categories in your dataset (topic, difficulty, source, format)
|
||||
2. Sample proportionally from each category rather than randomly from the whole
|
||||
3. Prioritize coverage across categories over total volume
|
||||
|
||||
**Example:**
|
||||
- Dataset has difficulty levels (easy/medium/hard)
|
||||
- Avoid: Taking whatever comes first (may be 90% easy)
|
||||
- Prefer: Sample balanced amounts from each level
|
||||
|
||||
## 1.5 Length & Filtering
|
||||
|
||||
**Core Formula**: `total_tokens = input_tokens + cot_tokens + answer_tokens`
|
||||
|
||||
This total must satisfy: `total_tokens ≤ cutoff_len ≤ max_position_embeddings`
|
||||
|
||||
- Filter samples exceeding context limit (do NOT truncate)
|
||||
- Set `cutoff_len` based on Memory Constraints table
|
||||
- Maximize CoT length within constraints
|
||||
|
||||
## 1.6 Output Format
|
||||
|
||||
Output filename: `data.json` (path handled by system). Use Alpaca format:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"instruction": "problem statement",
|
||||
"input": "optional additional context",
|
||||
{% if force_think_token %}
|
||||
"output": "<think>[step-by-step reasoning]</think>[final answer]"
|
||||
{% else %}
|
||||
"output": "[step-by-step reasoning]...[final answer]"
|
||||
{% endif %}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
{% if force_think_token %}
|
||||
**Note**: `<think>` tags are added by code post-processing, not requested in LLM prompts.
|
||||
The **answer** (after `</think>`) must follow **Benchmark Description**.
|
||||
{% else %}
|
||||
**Note**: Focus on reasoning quality. Let LLM generate naturally. DO NOT include `<think>` tags.
|
||||
{% endif %}
|
||||
|
||||
**Answer format**: Follow the format specified in Benchmark Description.
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 2: TRAINING CONFIGURATION
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 2.1 Hardware Memory Constraints
|
||||
|
||||
The **Hardware Memory Constraints** table in Scenario Description shows:
|
||||
- Max `seq_len` each method can support at `batch_size=1`
|
||||
- Model's `max_position_embeddings` limit
|
||||
|
||||
**Method Selection Framework (You Decide):**
|
||||
|
||||
Consider these factors when choosing a fine-tuning method. There are NO fixed rules - learn from history and adapt:
|
||||
|
||||
1. **Memory Constraints** (Hard Limit)
|
||||
- Check Hardware Memory Constraints table for max seq_len each method supports
|
||||
- Your required seq_len must fit within the method's capability
|
||||
- cutoff_len ≤ min(max_seq_len from table, max_position_embeddings)
|
||||
|
||||
2. **Dataset Size vs Overfitting Risk** (Trade-off to Explore)
|
||||
- Smaller datasets → higher risk of overfitting with full-parameter training
|
||||
- Consider: Can you augment data? Use regularization? Early stopping?
|
||||
- PEFT methods (LoRA/QLoRA) are one option, but not the only solution
|
||||
|
||||
3. **Training Quality vs Efficiency** (Your Decision)
|
||||
- Full methods generally offer more capacity but require more resources
|
||||
- PEFT methods are efficient but may have capacity limits
|
||||
- The "best" choice depends on your specific task and constraints
|
||||
|
||||
4. **Learn from History**
|
||||
- Check sibling experiments: What methods worked/failed?
|
||||
- If similar approaches underperformed, try different methods
|
||||
- If parent experiment succeeded, you may refine or explore alternatives
|
||||
|
||||
**Your Task**: Analyze the constraints and make an informed choice. Document your reasoning in the hypothesis.
|
||||
|
||||
**Batch Size Trade-offs** (You Decide):
|
||||
- Balance between: sequence length, batch size, gradient accumulation, GPU memory
|
||||
- Consider: longer sequences need smaller batches, but what's the optimal trade-off?
|
||||
- Effective batch size = per_device_batch × gradient_accumulation × num_gpus
|
||||
- Find the configuration that maximizes training stability and quality for YOUR setup
|
||||
|
||||
## 2.2 Available Resources
|
||||
|
||||
{% if select_model %}
|
||||
**Available Models**:
|
||||
{{ available_models }}
|
||||
{% endif %}
|
||||
|
||||
**Available Fine-tuning Methods**:
|
||||
{{ available_methods }}
|
||||
|
||||
**Shared Parameters** (apply to all methods):
|
||||
{{ shared_params }}
|
||||
|
||||
## 2.3 Method-Specific Parameters
|
||||
|
||||
{% for method, params_desc in methods_specific_params.items() %}
|
||||
{{ params_desc }}{% endfor %}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 3: OUTPUT SPECIFICATION
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 3.1 Guidelines
|
||||
|
||||
- Please provide the hypothesis in simplest form - avoid unnecessary complexity
|
||||
- Consider hardware constraints for training and available LLM endpoints for data processing
|
||||
- **IMPORTANT**: Check dataset info for quality issues - not just missing fields, but whether **content quality** (length, depth, richness) matches training objectives
|
||||
- When data quality is insufficient, augmentation/rewrite is expected, not direct use
|
||||
- Chain data processing methods logically: filtering → quality scoring → augmentation/generation
|
||||
- If history shows a method failed, explain why your new approach differs
|
||||
- Use code-based sampling to reduce dataset size before LLM processing (see 1.2)
|
||||
|
||||
## 3.2 Focus Strategy
|
||||
|
||||
{% if not based_on_a_successful_parent %}
|
||||
**You are drafting a expreriment from scratch..** You must provide a comprehensive strategy covering BOTH:
|
||||
1. Data processing: How to prepare the training data
|
||||
2. Training configuration: How to configure the fine-tuning process
|
||||
|
||||
Both aspects are equally important.
|
||||
{% else %}
|
||||
**This is a subsequent experiment.** Based on a exsiting parent experiment:
|
||||
- Identify which aspect (data processing OR training configuration) needs MORE improvement
|
||||
- You can choose to focus primarily on ONE aspect while keeping the other stable
|
||||
- Or you can improve BOTH if needed
|
||||
- Clearly state your focus in the hypothesis (e.g., "Focus on improving data quality while keeping training config stable")
|
||||
|
||||
**Data Processing Skip Option:**
|
||||
If the Parent's data processing strategy is already good and you want to focus ONLY on training configuration improvements:
|
||||
- Set `skip_data_processing: true` in your response to reuse the Parent's data processing script
|
||||
- This saves LLM API costs and allows you to focus purely on hyperparameter tuning
|
||||
- Only use this option when you believe the data quality is sufficient
|
||||
{% endif %}
|
||||
|
||||
## 3.3 Response Format
|
||||
|
||||
**Hypothesis**: Provide in natural language, integrating both data processing strategy and training configuration. Structure: "[Data Processing] ... [Training] ..." or a unified narrative covering both aspects.
|
||||
|
||||
**Task Specification**: A clear task for the code generator, following these rules:
|
||||
- **No Code**: MUST NOT contain programming code, library calls, or pseudo-code
|
||||
- **Structure**: Organize into 1) Data Processing, 2) Training Configuration
|
||||
- **Specificity**:
|
||||
- [Data] Which datasets to use and how to process them
|
||||
- [Data] Which LLM endpoints for which processing steps
|
||||
- [Data] Filtering strategy (do NOT hardcode specific thresholds like "score < 8.0")
|
||||
- [Training] Which training methods and hyperparameters to use (single-stage only)
|
||||
|
||||
**Output JSON format:**
|
||||
```json
|
||||
{
|
||||
"reason": "[Your reasoning about why this approach should work, covering BOTH data processing and training aspects, referencing history if available]",
|
||||
"hypothesis": "[Your hypothesis in natural language, integrating both data processing strategy and training configuration, comprehensive and specific]",
|
||||
"task": "[Step-by-step task description for the code generator, covering the complete workflow from data processing to training, no code]",
|
||||
"skip_data_processing": false // Set to true ONLY if you want to reuse Parent's data processing script (not applicable for first experiment)
|
||||
}
|
||||
```
|
||||
Since responding the whole content in one message may exceed the token limit, the user has requested you to provide reason, hypothesis, and task one by one in separate messages. Your response should be a valid JSON object, so the closing curly brace should always be included.
|
||||
|
||||
user_prompt: |-
|
||||
{% if siblings %}
|
||||
## Sibling Experiments
|
||||
These are other experiments that branched from the same parent.
|
||||
{% for sib_exp, sib_fb in siblings %}
|
||||
### Sibling {{ loop.index }}
|
||||
- Hypothesis: {{ sib_exp.hypothesis }}
|
||||
- Result: {{ "✅ Successful" if sib_fb.decision else "❌ Failed" }}{% if sib_fb.observations %} [{{ sib_fb.observations }}]{% endif %}
|
||||
- Reason: {{ sib_fb.reason }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if parent_exp %}
|
||||
{% set parent_info = trace.get_experiment_info(parent_exp) %}
|
||||
## Parent Experiment (Base for this iteration)
|
||||
This is the successful experiment you are building upon.
|
||||
|
||||
### Parent Hypothesis
|
||||
{{ parent_info.hypothesis }}
|
||||
|
||||
{% if parent_info.config %}
|
||||
### Parent Training Configuration
|
||||
```yaml
|
||||
{{ parent_info.config }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
{% if parent_info.data_script %}
|
||||
### Parent Data Processing Script
|
||||
```python
|
||||
{{ parent_info.data_script }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
{% if parent_info.benchmark %}
|
||||
### Parent Benchmark Results
|
||||
```json
|
||||
{{ parent_info.benchmark | tojson(indent=2) }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
**Improvement Focus**: Analyze the Parent's limitations and propose improvements. Consider:
|
||||
- What aspects of the current Parent could be improved?
|
||||
- Are there any hyperparameters that seem suboptimal?
|
||||
- Could the data processing strategy be enhanced?
|
||||
- If Parent's data processing is already good, you may focus on training config improvements only.
|
||||
{% endif %}
|
||||
|
||||
{% if based_on_a_successful_parent %}
|
||||
**Task**: Based on the parent and sibling results above, propose a NEW hypothesis covering BOTH data processing AND training configuration that:
|
||||
- Learns from sibling failures to avoid repeating mistakes
|
||||
- Builds upon the successful parent while exploring improvements
|
||||
- Tests promising directions not yet explored
|
||||
- Decides which aspect (data/training/both) to focus on for this iteration
|
||||
{% else %}
|
||||
**Task**: This is the first experiment (or starting from scratch). Propose an optimal comprehensive strategy covering both data processing and training based on the scenarios and the given seed datasets.
|
||||
{% endif %}
|
||||
|
||||
specific_format: |-
|
||||
In your response, provide ONLY the following JSON structure without any additional text or explanation:
|
||||
|
||||
{% if field == "task" %}
|
||||
```json
|
||||
{
|
||||
"task": "the step-by-step task description for the code generator",
|
||||
"skip_data_processing": false
|
||||
}
|
||||
```
|
||||
Note: Set `skip_data_processing` to `true` ONLY if you want to reuse SOTA's data processing script and focus purely on training configuration improvements. This is only valid for subsequent experiments (not the first one).
|
||||
{% else %}
|
||||
```json
|
||||
{
|
||||
"{{ field }}": "the content to {{ field }} following the instruction in the previous message"
|
||||
}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
"""LLM Fine-tuning Proposal Generator
|
||||
|
||||
Unified hypothesis generation that covers both data processing and training configuration.
|
||||
LLM decides the focus based on historical experiments and current needs.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.finetune.exp import FTTask
|
||||
from rdagent.core.proposal import ExpGen, Hypothesis, Trace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.scenarios.finetune.experiment.experiment import FTExperiment
|
||||
from rdagent.scenarios.finetune.proposal.trace import FTTrace
|
||||
from rdagent.scenarios.finetune.scen.llama_factory_manager import (
|
||||
LLaMAFactory_manager,
|
||||
)
|
||||
from rdagent.scenarios.finetune.scen.scenario import LLMFinetuneScen
|
||||
from rdagent.scenarios.finetune.utils import ensure_ft_assets_exist
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class FTHypothesis(Hypothesis):
|
||||
"""LLM fine-tuning hypothesis class."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_model: str,
|
||||
hypothesis: str | None = None,
|
||||
reason: str | None = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
hypothesis,
|
||||
reason,
|
||||
concise_reason="",
|
||||
concise_observation="",
|
||||
concise_justification="",
|
||||
concise_knowledge="",
|
||||
)
|
||||
self.base_model = base_model
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.hypothesis is None:
|
||||
return f"No hypothesis available. Constructing first runnable {self.component} component."
|
||||
|
||||
lines = [
|
||||
f"Base Model: {self.base_model}",
|
||||
f"Hypothesis: {self.hypothesis}",
|
||||
]
|
||||
if self.reason:
|
||||
lines.append(f"Reason: {self.reason}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class LLMFinetuneExpGen(ExpGen):
|
||||
"""LLM fine-tuning experiment generator.
|
||||
|
||||
Generates unified hypothesis covering both data processing and training configuration.
|
||||
"""
|
||||
|
||||
def __init__(self, scen: LLMFinetuneScen):
|
||||
super().__init__(scen)
|
||||
|
||||
def gen(self, trace: Trace) -> FTExperiment:
|
||||
"""Generate LLM fine-tuning experiment."""
|
||||
base_model = FT_RD_SETTING.base_model
|
||||
logger.info(f"Generating experiment with base model: {base_model}")
|
||||
|
||||
sota_exp = trace.get_sota_experiment() # use sota_exp as the parent
|
||||
|
||||
return self._gen_hypothesis(trace, base_model, parent_exp=sota_exp)
|
||||
|
||||
def _gen_hypothesis(self, trace: Trace, base_model: str, parent_exp: FTExperiment | None = None) -> FTExperiment:
|
||||
"""Generate hypothesis covering both data processing and training configuration.
|
||||
|
||||
Args:
|
||||
trace: Experiment trace history
|
||||
base_model: Base model name
|
||||
parent_exp: Parent experiment to base this one on; usually the SOTA experiment
|
||||
|
||||
Returns:
|
||||
FTExperiment with tasks for both data processing and training
|
||||
"""
|
||||
based_on_a_successful_parent = parent_exp is not None
|
||||
logger.info(f"Generating hypothesis based on (parent_exp={parent_exp})")
|
||||
|
||||
available_models = LLaMAFactory_manager.models
|
||||
available_methods = LLaMAFactory_manager.methods
|
||||
shared_params = LLaMAFactory_manager.format_shared_params()
|
||||
methods_specific_params = {}
|
||||
for method in available_methods:
|
||||
methods_specific_params[method] = LLaMAFactory_manager.format_method_specific_params(method)
|
||||
|
||||
# Find siblings
|
||||
parent_idx = trace.exp2idx(parent_exp) if parent_exp else None
|
||||
# Handle potential list return
|
||||
if isinstance(parent_idx, list):
|
||||
parent_idx = parent_idx[0] if parent_idx else None
|
||||
|
||||
# If no parent, start from void root node
|
||||
siblings = trace.get_children(parent_idx)
|
||||
|
||||
system_prompt = T(".prompts:unified_hypothesis_gen.system_prompt").r(
|
||||
based_on_a_successful_parent=based_on_a_successful_parent,
|
||||
scenario=self.scen.get_scenario_all_desc(enable_dataset_description=True),
|
||||
available_models=available_models,
|
||||
available_methods=available_methods,
|
||||
shared_params=shared_params,
|
||||
methods_specific_params=methods_specific_params,
|
||||
select_model=base_model is None,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
|
||||
user_prompt = T(".prompts:unified_hypothesis_gen.user_prompt").r(
|
||||
parent_exp=parent_exp,
|
||||
siblings=siblings,
|
||||
trace=trace,
|
||||
based_on_a_successful_parent=based_on_a_successful_parent,
|
||||
)
|
||||
|
||||
session = APIBackend().build_chat_session(session_system_prompt=system_prompt)
|
||||
reason_dict = json.loads(
|
||||
session.build_chat_completion(
|
||||
user_prompt=user_prompt + "\n" + T(".prompts:unified_hypothesis_gen.specific_format").r(field="reason"),
|
||||
json_target_type=dict,
|
||||
)
|
||||
)
|
||||
hypothesis_dict = json.loads(
|
||||
session.build_chat_completion(
|
||||
user_prompt=T(".prompts:unified_hypothesis_gen.specific_format").r(field="hypothesis"),
|
||||
json_target_type=dict,
|
||||
)
|
||||
)
|
||||
task_dict = json.loads(
|
||||
session.build_chat_completion(
|
||||
user_prompt=T(".prompts:unified_hypothesis_gen.specific_format").r(field="task"),
|
||||
json_target_type=dict,
|
||||
)
|
||||
)
|
||||
|
||||
ensure_ft_assets_exist(model=base_model, check_model=True)
|
||||
|
||||
# Get skip_data_processing from task_dict (merged with task in 3rd LLM call)
|
||||
# Only valid for subsequent experiments, first experiment always generates data
|
||||
skip_data_processing = task_dict.get("skip_data_processing", False) if based_on_a_successful_parent else False
|
||||
if skip_data_processing:
|
||||
logger.info("Proposal decided to skip data processing, will reuse Parent's data script")
|
||||
|
||||
# Use pre-selected datasets from scenario initialization
|
||||
task = FTTask(
|
||||
base_model=base_model,
|
||||
description=task_dict.get("task"),
|
||||
benchmark=FT_RD_SETTING.target_benchmark,
|
||||
involving_datasets=self.scen.selected_datasets,
|
||||
skip_data_processing=skip_data_processing,
|
||||
)
|
||||
|
||||
hypothesis = FTHypothesis(
|
||||
base_model=base_model,
|
||||
hypothesis=hypothesis_dict.get("hypothesis"),
|
||||
reason=reason_dict.get("reason", ""),
|
||||
)
|
||||
|
||||
exp = FTExperiment(sub_tasks=[task], hypothesis=hypothesis)
|
||||
if parent_exp:
|
||||
parent_idx = trace.exp2idx(parent_exp)
|
||||
if parent_idx is not None:
|
||||
exp.local_selection = (parent_idx,)
|
||||
else:
|
||||
# If no parent, it is a experiment from scratch
|
||||
exp.local_selection = trace.NEW_ROOT
|
||||
|
||||
# Inject workspace files from Parent or SOTA experiment (if available)
|
||||
if parent_exp and (ws := parent_exp.experiment_workspace) is not None and ws.file_dict:
|
||||
exp.experiment_workspace.inject_from_workspace(ws)
|
||||
logger.info(f"Injected {len(ws.file_dict)} files from parent: {list(ws.file_dict.keys())}")
|
||||
|
||||
logger.info("Experiment created")
|
||||
|
||||
return exp
|
||||
@@ -0,0 +1,80 @@
|
||||
"""FT Trace - Specialized Trace for LLM Fine-tuning scenario.
|
||||
|
||||
Provides SOTA experiment tracking functionality.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DATA_SCRIPT_NAME,
|
||||
FT_YAML_FILE_NAME,
|
||||
)
|
||||
from rdagent.core.evolving_framework import KnowledgeBase
|
||||
from rdagent.core.proposal import ExperimentFeedback, Trace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from rdagent.scenarios.finetune.experiment.experiment import FTExperiment
|
||||
from rdagent.scenarios.finetune.scen.scenario import LLMFinetuneScen
|
||||
|
||||
|
||||
class FTTrace(Trace["LLMFinetuneScen", KnowledgeBase]):
|
||||
"""Specialized Trace for LLM Fine-tuning scenario.
|
||||
|
||||
Adds SOTA experiment tracking on top of the base Trace class.
|
||||
SOTA is explicitly managed via DAG traversal.
|
||||
"""
|
||||
|
||||
def __init__(self, scen: "LLMFinetuneScen", knowledge_base: KnowledgeBase | None = None) -> None:
|
||||
super().__init__(scen, knowledge_base)
|
||||
|
||||
# Type hint for linting
|
||||
self.hist: list[tuple[FTExperiment, ExperimentFeedback]] = []
|
||||
|
||||
def sota_benchmark(self) -> dict | None:
|
||||
"""Return SOTA experiment's benchmark results."""
|
||||
sota_exp = self.get_sota_experiment()
|
||||
if sota_exp is None:
|
||||
return None
|
||||
ws = sota_exp.experiment_workspace
|
||||
if ws is None or ws.running_info is None:
|
||||
return None
|
||||
result = getattr(ws.running_info, "result", None)
|
||||
if result and isinstance(result, dict) and "benchmark" in result:
|
||||
return result["benchmark"]
|
||||
return None
|
||||
|
||||
def get_experiment_info(self, exp: "FTExperiment") -> dict[str, Any]:
|
||||
"""Return experiment's full info for hypothesis generation."""
|
||||
info: dict[str, Any] = {
|
||||
"hypothesis": str(exp.hypothesis) if exp.hypothesis else None,
|
||||
"config": None,
|
||||
"benchmark": None,
|
||||
"data_script": None,
|
||||
}
|
||||
|
||||
ws = exp.experiment_workspace
|
||||
if ws is None:
|
||||
return info
|
||||
|
||||
if ws.file_dict:
|
||||
if FT_YAML_FILE_NAME in ws.file_dict:
|
||||
info["config"] = ws.file_dict[FT_YAML_FILE_NAME]
|
||||
if FT_DATA_SCRIPT_NAME in ws.file_dict:
|
||||
info["data_script"] = ws.file_dict[FT_DATA_SCRIPT_NAME]
|
||||
|
||||
if ws.running_info:
|
||||
result = getattr(ws.running_info, "result", None)
|
||||
if result and isinstance(result, dict) and "benchmark" in result:
|
||||
info["benchmark"] = result["benchmark"].get("accuracy_summary")
|
||||
|
||||
return info
|
||||
|
||||
def sota_info(self) -> dict[str, Any] | None:
|
||||
"""Return SOTA experiment's full info for hypothesis generation."""
|
||||
sota_exp = self.get_sota_experiment()
|
||||
if sota_exp is None:
|
||||
return None
|
||||
return self.get_experiment_info(sota_exp)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user