mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 23:47:46 +00:00
feat(rl): add AutoRL-Bench framework and benchmark integrations (#1348)
* feat: rdkit for chemcotbench * update qwen2.5&llama3.1 context * fix: force failure on validation error and remove try/except in validator * feat: unified error sample extraction (with test scripts) * feat: set conda cache with .env * feat: skip data eval if data pass in last evo * fix: rm redundant param * fix ui bug * refactor: centralize assign_code_list_to_evo in MultiProcessEvolvingStrategy * feat: add test_params.yaml generation and workspace cleanup improvements for finetune * refactor: replace get_clear_ws_cmd with clear_workspace and update prompts for hard check criteria * add bioprobench dataset * fix: handle commas in training config extraction and refactor prompt includes * bioprobench description * add bioprobench readme * feat: merge lora adapter for blackwell gpu * feat: support for multi benchmarks in one job * change dfficult aware content for training * update difficulty-aware and logging principles * fix: resolve variable name conflict in FTRunnerEvaluator * set job id accuracy to minute * feat(ui): display one selected metric per benchmark * feat: store sota exp, and fix ws_ckp bug * fix: truncate data.json in feedback * fix: opencompass data for conda env * fix: save only the last model * feat: set log path and ws path * fix: set overwrite_cache to avoid lock contention(through injecting params) * feat: redirect stdout to file in localenv * add pickle cache to dataset desc * fix CI * fix: remove redundant wrapper * feat: set python_unbuffered * move redirect stdout to env run * fix a small bug * move model folder * feat(ui): display benchmark baseline * fix: enrich scenario and benchmark description * fix: rewrite runner eval to accept easier * feat: compare with baseline when no SOTA * update tablebench readme * fix: switch back to single benchmark (for baseline) * feat(ui): add ws path in ui * refactor: update SOTA tracking to use DAG traversal and parent selection * fix: prioritize local_selection in trace and refactor sibling retrieval logic * refactor: unify error handling in feedback generation and update workspace injection * feat: add skip_loop_error_stepname to control error skip step in LoopBase * fix: set local_selection to NEW_ROOT for experiments without parent * feat: set different ports for jobs * feat: set different ports for jobs * feat: add upper data size limit for LLM fine-tuning and update related prompts * fix: replace get_truncated_stdout() with stdout for consistent output handling * refactor: remove data.json from cache and workspace logic, focus on script-based reuse * fix: rm target_scenario * feat: add selective cache extraction and custom cache key for data processing * fix(ui): bug when displaying tablebench * fix: filter config in dataset_info.json * feat: add test set, set valid set * feat(ui): update test score, and set color for final decision * feat: add test score for baseline and update ui * fix: use [-100:] as test range * feat: update data_stats in runner * feat: wait for opencompass init when run multi jobs * fix: adjust test&valid split * feat: force to generate COT(with <think> token), and add answer format in scenarios.json * feat: improve ui * fix: unify benchmark volume mounts and set extra_volumes for conda env * fix(ui): number color * fix: update GPU memory handling to use total memory in GB and streamline code * fix: set use_cot_postprocessor * feat: add env_dict to config classes and merge env vars in Env run * fix: let coder obey proposal * fix(ui): direction bug and update chemcot core metirc * fix: set consistent benchmark mount points and env vars for docker and conda * fix: addintional target for LoRA * feat: workspace dir log for benchmark running * fix: tableInstruct path bug and update benchmark description * feat: timeout for whole job * fix: align FinanceIQ import to opencompass * feat: use llm_judge for FinanceIQ * feat: switch to turn on <think> or not * feat: using scripts to redirect stdout, and run in different windows * feat: sync litellm log * fix: gpu memory format * fix: escape special characters in benchmark desc * fix: set data processing timeout to 1h * feat: set valid_loss and save_best_model * fix: inject timeout and stage * fix: loss history extract logic * feat: inject output dir * feat: inject eval batch size * feat: inject save_total_limit * feat: update data prompt * fix: escape shell special characters * fix: tablebench visualization UI * fix: move implementation validation to coder, and ignore injected params * docs: add README for RL-PostTraining evaluation system * Add AutoRL-Bench evaluation framework for RL post-training * Add architecture documentation * docs: update architecture and interface documentation for AutoRL-Bench * improve doc * fix * refactor: YAML配置驱动 * feat: add RL Docker env, workspace test, and update project structure * feat: 重命名 autorl_bench, 新增 RLWorkspace, 配置 Docker extra_volumes * Add eval-only AutoRL-Bench pipeline * sturcture clean * docs: add autorl_bench README * feat(rl): Implement RL post-training agent scaffold and example * refactor: simplify RL scenario classes and update RL CoSTEER integration * feat(rl): 调通 scaffold，mock 数据跑完 5 步循环 * feat(rl): 接入 LLM 生成代码，支持 model_path 传递 * feat(rl): Docker 执行框架，RLWorkspace.run() + RLPostTrainingRunner * feat(rl): LLM 生成假设/反馈，完整 loop 跑通 * feat: add RL post-training entry point with configurable options * refactor: simplify RL proposal and trace classes, update config and docs * Update rl eval autorl_bench layout * Update RL workflow and evaluation setup * Integrate AutoRL-Bench evaluation in RL workflow * feat(rl): 添加 --base-model/--benchmark CLI 参数，简化 RLTask * feat(rl): Docker 环境动态选择 + example_agent 完整训练评测流程（无llm） * fix(rl): 修复 feedback 传递 + 添加 verl 依赖 * refactor: remove unused validate in BenchmarkAdapter and add core utils module * feat(rl): UI * Refactor autorl_bench layout and docker entrypoint * autorl_bench: add aider autoloop tool * feat(rl): environment docker * refactor: simplify aider autoloop tooling * chore: update misc files * feat(rl): yaml-driven dataset download & auto-download on startup * feat(rl): yaml-driven dataset download & auto-download on startup * Refactor RL eval runner and clean up * Simplify RL eval runner and env * rl: include litellm in RL docker image * feat(rl): unified resource path & model repo_id structure * feat(rl): refactor eval with OpenCompass & add training code template * feat(rl): refactor eval with OpenCompass & add training code template * feat(rl): delete test bench * docs: add benchmark interface notes and TODOs for unified evaluation * feat(rl): unified benchmark eval interface + shared configs * feat(rl): 优雅 * feat(rl): prompt prososal+coder improve * feat(rl): fix eval * fix(rl): docker * fix(rl): eval * v 1.0 tmep * benchmark v1.0 * benchmark v1.1 * benchmark v1.1: grading日志+代码去重 * benchmark v1.1: grading日志+代码去重 * benchmark v1.1: grading日志+代码去重+task description * benchmark v1.2: fix * benchmark v1.3: fix，example-agent ok，rdagent test，openhands develop * benchmark v1.4: fix，example-agent ok，rdagent ok，openhands develop * benchmark : add alfworld * benchmark : update readme * benchmark : update readme * benchmark : * chore: add eval bypass block and mark TODO in grading server * benchmark * benchmark * benchmark * benchmark * alfworld * alfworld * benchmark * rdagent * rdagent * benchmark * benchmark：ui * benchmark：delete docker + log * 1 * alfworld * ui * alfworld * readme * alfworld * parallex * alfworld * run * eval gpu * alfworld * alfworld * fix conda init in start.sh for non-interactive shells Fallback to common miniconda paths when conda is not in PATH. Fixes B200 pod startup failure (conda: command not found). Made-with: Cursor * simplify start.sh: read TRAINING_PYTHON from .env No more conda detection logic. Just set TRAINING_PYTHON in .env. Fallback to conda only if not set. Made-with: Cursor * use OPENHANDS_PYTHON from .env to run agent start.sh now uses OPENHANDS_PYTHON for main.py execution, since the parent process may be in a different conda env. Made-with: Cursor * feat: register OpenCode agent into autorl_bench framework - Add agents/opencode/ with config.yaml, start.sh, README.md - Include opencode-rl pipeline code (pipeline/, runner_fsm/, benchmarks/) - Merge opencode-rl dependencies into autorl_bench requirements.txt - Remove separate venv requirement, share main environment Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update opencode agent, benchmarks, and eval configs - Sync opencode-rl runner_fsm with latest simplifications - Add smith benchmarks integration - Update opencompass configs and server with GPU support + error handling * Update OpenCode agent docs for external opencode-rl integration - Document external repo architecture (opencode-rl as independent plugin) - Add setup instructions for cloning and configuring opencode-rl - Add architecture diagram showing RD-Agent ↔ opencode-rl interaction - Document OPENCODE_RL_ROOT for custom paths * feat: add smith benchmark discovery and per-sample evaluator - Add smith/ module for dynamic benchmark discovery from rl-smith - Add PerSampleEvaluator for per-sample scoring via vLLM - Update utils.py to support script-based data download for smith benchmarks - Update opencode agent config * enforce RL-only in instructions.md; remove embedded opencode-rl - instructions.md: prohibit SFT, require RL (GRPO/PPO) for all benchmarks - remove agents/opencode/opencode-rl/ (runtime uses external OPENCODE_RL_ROOT) Made-with: Cursor * comment out OpenCode-only deps in requirements.txt openai, httpx, python-dotenv, tenacity are for OpenCode agent's separate environment. Keep peft and pydantic as shared deps. Made-with: Cursor * refactor: extract _kill_process_group, narrow exception catches - run.py: replace 2x nested 3-level try/except with shared _kill_process_group() using loop + specific exceptions - server.py: except Exception → except (RuntimeError, ValueError, OSError) - utils.py: except Exception → except requests.ConnectionError Made-with: Cursor * move kill_process_group to core/utils for reuse Extract from run.py into core/utils.py so other runners can also use it. Exported via core/__init__.py. Made-with: Cursor * add comments to run.py for workspace isolation and signal handling Made-with: Cursor * remove OpenCode-only deps from requirements.txt entirely Made-with: Cursor * allow SFT in instructions, RL as ultimate goal Made-with: Cursor * add workspace isolation rules to instructions.md Use relative paths, forbid cd outside workspace, ignore symlink targets. Made-with: Cursor * update opencode start.sh: use OPENCODE_PYTHON, add PATH for opencode CLI, remove unsupported args Made-with: Cursor * opencode start.sh: pass --run-dir to use AutoRL-Bench workspace Ensures OpenCode-FSM-Runner writes outputs into the workspace prepared by AutoRL-Bench instead of creating its own runs/ directory. Made-with: Cursor * opencode start.sh: prepend training env bin to PATH Ensures LLM agent bash calls (e.g. python3 -c "from trl import ...") resolve to the correct training environment, instead of relying on parent shell conda activation. Made-with: Cursor * opencode start.sh: restore --max-retries and --eval-timeout for opencode-rl Made-with: Cursor * add humaneval benchmark * Replace import * cleanup hack with explicit imports in OpenCompass config - Resolve dataset variable names via importlib before generating config, so the template uses `from xxx import datasets` instead of `import *` - Remove the fragile runtime cleanup hack that set leaked modules to None - Increase OpenCompass timeout from 3600s to 7200s - Fix score parsing to average across multiple subdatasets * refine opencompass config file generating * add humaneval benchmark dependency instructions human-eval package requires clone from open-compass/human-eval with a one-line patch to relax assertion for partial evaluation (test split only). Made-with: Cursor * fix: sanitize user-provided paths in RL UI (CodeQL) * fix: resolve user path relative to safe root (CodeQL) * fix: use Copilot-suggested path sanitization pattern (CodeQL) * fix: normalize and reject absolute user paths (CodeQL) * Fix training params, vLLM OOM cleanup, OpenCompass score parsing, and baseline cache logic * fix: add setuptools<75 to requirements for opencompass pkg_resources dependency uv venv does not include setuptools by default, causing OpenCompass baseline evaluation to fail with "No module named 'pkg_resources'". Made-with: Cursor * webshop * feat(autorl_bench): improve smith benchmark integration and evaluator robustness - Add smith benchmark docs to README: usage examples, discovery mechanism, SMITH_BENCH_DIR - Improve PerSampleEvaluator: vLLM GPU cleanup, test_range slicing - Refactor server.py: extract grading server from utils - Fix OpenCompass score parsing and baseline cache logic * fix: add smart fallback for OpenCompass dataset variable resolution When build_dataset_imports_explicit() fails to import an OpenCompass dataset module (common in grading server subprocess), it now guesses the correct variable name from the module path convention instead of falling back to empty names (which causes import * and breaks BBH due to leaked file handle objects). * revert: restore opencompass.py to pre-modification state Revert vLLM pid cleanup, dash-value checks, and metric-based score parsing added in 31caff2f and bb32e555. * keep metric-aware score parsing in opencompass; add baseline column to UI - opencompass.py: retain metric-type filtering (accuracy/score) instead of naive averaging, avoids polluting scores with pass/timeout counters - ui.py: add Baseline column to Agent Summary table Made-with: Cursor * fix: handle non-string answers in extract_answer to prevent TypeError arc_agi and other benchmarks can have non-string answer fields (e.g. lists), which caused a crash in re.search(). Adding str() coercion fixes this. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * update deepsearch qa tasks * fix: benchmark evaluation reliability (B1-B4) - B1: auto-detect LoRA adapters and enable vLLM LoRA mode (read base_model from adapter_config.json) - B2: serialize evaluations with threading.Lock to prevent GPU contention - B3: cache eval results by model_path to deduplicate concurrent submissions - B4: propagate error details from OpenCompass to agent (non-numeric scores, load failures) Made-with: Cursor * fix(B1): reject LoRA adapter submissions with clear merge instructions - opencompass.py: detect adapter_config.json and return error with merge_and_unload() instructions instead of broken vLLM LoRA mode - instructions.md: add requirement to submit full merged models - opencompass_template.yaml: remove unused is_lora/lora_path params Made-with: Cursor * update chat completion * update * update deepsearch * md * codex + benchmark update * codex + benchmark update * codex + benchmark update * codex * codex * fix: grading server cache key includes mtime to detect model overwrites Previously cache used only resolved_path, so overwritten models at the same path returned stale scores. Now cache key = path@max_mtime so re-evaluation is triggered when model files change. Made-with: Cursor * feat: add gemini/claude agent scaffolds, fix codex binary path - codex/start.sh: use CODEX_BIN env var instead of bare 'codex' - Add gemini/ and claude/ agent directories with config.yaml and start.sh Made-with: Cursor * chore: remove copied human_readable_trace.py from PostTrainBench Made-with: Cursor * update evaluation * benchmark * Fix log cleanup and OpenHands env * fix: webshop env pth problem * benchmark alpacaeval * style(rl): apply auto-lint fixes * fix(rl): address CI and CodeQL issues * fix(rl): make autorl bench imports CI-safe --------- Co-authored-by: Qizheng Li <jenssenlee@163.com> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: Bowen Xian <xianbowen@outlook.com> Co-authored-by: chelsea97 <zhuowbrown@gmail.com> Co-authored-by: Xu Yang <peteryang@vip.qq.com> Co-authored-by: sakura657 <yctangcse@gmail.com> Co-authored-by: shatianming5 <tianming.sha@stonybrook.edu> Co-authored-by: Yeyuqing0913 <shatianming4@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
BarryC
@@ -5,6 +5,8 @@ Pipfile
|
||||
public
|
||||
release-notes.md
|
||||
typescript*
|
||||
tmp/
|
||||
.ai/
|
||||
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
@@ -183,5 +185,6 @@ static/
|
||||
.cursor/
|
||||
.claude/
|
||||
AGENTS.md
|
||||
!rdagent/**/AGENTS.md
|
||||
|
||||
scripts/
|
||||
@@ -71,7 +71,13 @@ log_cli = true
|
||||
log_cli_level = "info"
|
||||
log_date_format = "%Y-%m-%d %H:%M:%S"
|
||||
log_format = "%(asctime)s %(levelname)s %(message)s"
|
||||
markers = [
|
||||
"offline: tests that do not require external API calls",
|
||||
]
|
||||
minversion = "6.0"
|
||||
norecursedirs = [
|
||||
"workspace",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
fix = true
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import SettingsConfigDict
|
||||
|
||||
from rdagent.core.conf import ExtendedBaseSettings
|
||||
|
||||
|
||||
class RLPostTrainingPropSetting(ExtendedBaseSettings):
|
||||
"""RL Post-training dedicated property settings.
|
||||
|
||||
Use RL_ env prefix for overrides.
|
||||
"""
|
||||
|
||||
model_config = SettingsConfigDict(env_prefix="RL_", protected_namespaces=())
|
||||
|
||||
# Main Components
|
||||
scen: str = "rdagent.scenarios.rl.scen.scenario.RLPostTrainingScen"
|
||||
hypothesis_gen: str = "rdagent.scenarios.rl.proposal.proposal.RLPostTrainingExpGen"
|
||||
coder: str = "rdagent.components.coder.rl.RLCoSTEER"
|
||||
runner: str = "rdagent.scenarios.rl.train.runner.RLPostTrainingRunner"
|
||||
summarizer: str = "rdagent.scenarios.rl.dev.feedback.RLExperiment2Feedback"
|
||||
|
||||
# Resource paths (unified directory management, similar to SFT)
|
||||
file_path: Path = Path.cwd() / "git_ignore_folder" / "rl_files"
|
||||
"""RL resource root directory. Contains datasets/ and models/ subdirectories.
|
||||
Can be overridden via RL_FILE_PATH environment variable."""
|
||||
|
||||
# Core config
|
||||
base_model: str | None = None
|
||||
"""Model name (e.g., 'Qwen2.5-Coder-0.5B-Instruct'). Docker path: /models/{base_model}"""
|
||||
|
||||
benchmark: str | None = None
|
||||
"""Benchmark/dataset name (e.g., 'gsm8k'). Docker path: /data/{benchmark}"""
|
||||
|
||||
# Benchmark evaluation
|
||||
benchmark_timeout: int = 0
|
||||
"""Benchmark evaluation timeout in seconds. 0 means no timeout."""
|
||||
|
||||
|
||||
# Global setting instance
|
||||
RL_RD_SETTING = RLPostTrainingPropSetting()
|
||||
@@ -0,0 +1,64 @@
|
||||
"""
|
||||
RL Post-training Entry Point
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Optional
|
||||
|
||||
import typer
|
||||
from typing_extensions import Annotated
|
||||
|
||||
from rdagent.app.rl.conf import RL_RD_SETTING
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.loop import RLPostTrainingRDLoop
|
||||
|
||||
|
||||
def main(
|
||||
base_model: Annotated[Optional[str], typer.Option("--base-model", "-m")] = None,
|
||||
benchmark: Annotated[Optional[str], typer.Option("--benchmark", "-b")] = None,
|
||||
step_n: Optional[int] = None,
|
||||
loop_n: Optional[int] = None,
|
||||
timeout: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
RL post-training entry point
|
||||
|
||||
Parameters
|
||||
----------
|
||||
base_model : str
|
||||
Model name (e.g., 'Qwen2.5-Coder-0.5B-Instruct')
|
||||
Docker path: /models/{base_model}
|
||||
benchmark : str
|
||||
Benchmark/dataset name (e.g., 'gsm8k')
|
||||
Docker path: /data/{benchmark}
|
||||
step_n : int, optional
|
||||
Number of steps to run; if None, runs all steps per loop
|
||||
loop_n : int, optional
|
||||
Number of loops to run; if None, runs indefinitely
|
||||
timeout : str, optional
|
||||
Maximum duration for the entire process
|
||||
|
||||
Examples
|
||||
--------
|
||||
.. code-block:: bash
|
||||
|
||||
export RL_MODELS_DIR=/path/to/models
|
||||
export RL_DATA_DIR=/path/to/data
|
||||
python rdagent/app/rl/loop.py --base-model Qwen2.5-Coder-0.5B-Instruct --benchmark gsm8k
|
||||
"""
|
||||
# Update config from CLI
|
||||
if base_model:
|
||||
RL_RD_SETTING.base_model = base_model
|
||||
if benchmark:
|
||||
RL_RD_SETTING.benchmark = benchmark
|
||||
|
||||
logger.info(f"Starting RL post-training: model={RL_RD_SETTING.base_model}, benchmark={RL_RD_SETTING.benchmark}")
|
||||
|
||||
# RDLoop 会自动根据 RL_RD_SETTING.scen 创建 Scenario
|
||||
# Scenario.__init__() 中会自动运行 baseline 评测
|
||||
loop = RLPostTrainingRDLoop(RL_RD_SETTING)
|
||||
asyncio.run(loop.run(step_n=step_n, loop_n=loop_n, all_duration=timeout))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
typer.run(main)
|
||||
@@ -0,0 +1 @@
|
||||
"""RL Post-training UI"""
|
||||
@@ -0,0 +1,176 @@
|
||||
"""
|
||||
RL Post-training Timeline Viewer
|
||||
Hierarchical view: Session > Loop > Stage > Events
|
||||
|
||||
Run:
|
||||
streamlit run rdagent/app/rl/ui/app.py
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import streamlit as st
|
||||
from streamlit import session_state as state
|
||||
|
||||
from rdagent.app.rl.ui.components import render_session, render_summary
|
||||
from rdagent.app.rl.ui.config import ALWAYS_VISIBLE_TYPES, OPTIONAL_TYPES
|
||||
from rdagent.app.rl.ui.data_loader import get_summary, get_valid_sessions, load_session
|
||||
from rdagent.app.rl.ui.rl_summary import render_job_summary
|
||||
|
||||
DEFAULT_LOG_BASE = "log/"
|
||||
|
||||
|
||||
def _safe_resolve(user_input: str | None, safe_root: Path) -> Path:
|
||||
"""Resolve user path relative to safe_root; raise ValueError if it escapes."""
|
||||
safe_root = safe_root.expanduser().resolve()
|
||||
if not user_input:
|
||||
return safe_root
|
||||
|
||||
if "\x00" in user_input:
|
||||
raise ValueError("Invalid path: contains null byte")
|
||||
|
||||
try:
|
||||
normalized = os.path.normpath(user_input)
|
||||
if os.path.splitdrive(normalized)[0]:
|
||||
raise ValueError("Absolute paths with drive letters are not allowed")
|
||||
path_obj = Path(normalized).expanduser()
|
||||
if path_obj.is_absolute():
|
||||
raise ValueError("Absolute paths are not allowed")
|
||||
candidate = (safe_root / path_obj).resolve(strict=False)
|
||||
candidate.relative_to(safe_root)
|
||||
return candidate
|
||||
except (OSError, ValueError) as exc:
|
||||
raise ValueError(f"Invalid path outside of allowed root: {user_input}") from exc
|
||||
|
||||
|
||||
def get_job_options(base_path: Path) -> list[str]:
|
||||
"""Scan directory and return job options list."""
|
||||
options = []
|
||||
has_root_tasks = False
|
||||
job_dirs = []
|
||||
|
||||
if not base_path.exists():
|
||||
return options
|
||||
|
||||
for d in base_path.iterdir():
|
||||
if not d.is_dir():
|
||||
continue
|
||||
if (d / "__session__").exists():
|
||||
has_root_tasks = True
|
||||
else:
|
||||
try:
|
||||
if any((sub / "__session__").exists() for sub in d.iterdir() if sub.is_dir()):
|
||||
job_dirs.append(d.name)
|
||||
except PermissionError:
|
||||
pass
|
||||
|
||||
job_dirs.sort(reverse=True)
|
||||
options.extend(job_dirs)
|
||||
if has_root_tasks:
|
||||
options.append(". (Current)")
|
||||
|
||||
return options
|
||||
|
||||
|
||||
def main():
|
||||
st.set_page_config(layout="wide", page_title="RL Timeline", page_icon="🤖")
|
||||
|
||||
with st.sidebar:
|
||||
view_mode = st.radio("View Mode", ["Job Summary", "Single Task"], horizontal=True)
|
||||
st.divider()
|
||||
|
||||
default_log = os.environ.get("RL_LOG_PATH", DEFAULT_LOG_BASE)
|
||||
safe_root = Path(default_log).expanduser().resolve()
|
||||
job_folder = str(safe_root)
|
||||
selected_types = ALWAYS_VISIBLE_TYPES.copy()
|
||||
is_root_job = False
|
||||
|
||||
if view_mode == "Job Summary":
|
||||
st.header("Job")
|
||||
base_folder = st.text_input("Base Folder", value=default_log, key="base_folder_input")
|
||||
try:
|
||||
base_path = _safe_resolve(base_folder, safe_root)
|
||||
except ValueError as e:
|
||||
st.error(str(e))
|
||||
return
|
||||
|
||||
job_options = get_job_options(base_path)
|
||||
if job_options:
|
||||
selected_job = st.selectbox("Select Job", job_options, key="job_select")
|
||||
if selected_job.startswith("."):
|
||||
job_folder = str(base_path)
|
||||
is_root_job = True
|
||||
else:
|
||||
job_folder = str(base_path / selected_job)
|
||||
state.selected_job_folder = job_folder
|
||||
else:
|
||||
st.warning("No jobs found in this directory")
|
||||
job_folder = str(base_path)
|
||||
|
||||
if st.button("Refresh", type="primary", key="refresh_job"):
|
||||
st.rerun()
|
||||
else:
|
||||
st.header("Session")
|
||||
default_path = getattr(state, "selected_job_folder", default_log)
|
||||
log_folder = st.text_input("Log Folder", value=default_path)
|
||||
try:
|
||||
log_path = _safe_resolve(log_folder, safe_root)
|
||||
except ValueError as e:
|
||||
st.error(str(e))
|
||||
return
|
||||
|
||||
sessions = get_valid_sessions(log_path)
|
||||
if not sessions:
|
||||
st.warning("No valid sessions found")
|
||||
return
|
||||
|
||||
selected_session = st.selectbox("Session", sessions)
|
||||
|
||||
if st.button("Load", type="primary") or "session" not in state:
|
||||
with st.spinner("Loading..."):
|
||||
state.session = load_session(log_path / selected_session)
|
||||
state.session_name = selected_session
|
||||
|
||||
st.divider()
|
||||
|
||||
st.subheader("Show More")
|
||||
selected_types = ALWAYS_VISIBLE_TYPES.copy()
|
||||
for event_type, (label, default) in OPTIONAL_TYPES.items():
|
||||
if st.toggle(label, value=default, key=f"toggle_{event_type}"):
|
||||
selected_types.append(event_type)
|
||||
|
||||
st.divider()
|
||||
|
||||
if "session" in state:
|
||||
summary = get_summary(state.session)
|
||||
st.subheader("Summary")
|
||||
st.metric("Loops", summary.get("loop_count", 0))
|
||||
st.metric("LLM Calls", summary.get("llm_call_count", 0))
|
||||
success = summary.get("docker_success", 0)
|
||||
fail = summary.get("docker_fail", 0)
|
||||
st.metric("Docker", f"{success}✓ / {fail}✗")
|
||||
|
||||
if view_mode == "Job Summary":
|
||||
st.title("📊 RL Job Summary")
|
||||
job_path = Path(job_folder).resolve()
|
||||
if job_path.exists():
|
||||
render_job_summary(job_path, is_root=is_root_job)
|
||||
else:
|
||||
st.warning(f"Job folder not found: {job_folder}")
|
||||
return
|
||||
|
||||
st.title("🤖 RL Timeline Viewer")
|
||||
|
||||
if "session" not in state:
|
||||
st.info("Select a session and click **Load** to view")
|
||||
return
|
||||
|
||||
session = state.session
|
||||
summary = get_summary(session)
|
||||
render_summary(summary)
|
||||
st.divider()
|
||||
render_session(session, selected_types)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,315 @@
|
||||
"""
|
||||
RL UI Components - Event Renderers
|
||||
Simplified version without EvoLoop
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
import streamlit as st
|
||||
|
||||
from rdagent.app.rl.ui.config import ICONS
|
||||
from rdagent.app.rl.ui.data_loader import Event, Loop, Session
|
||||
|
||||
|
||||
def format_duration(seconds: float | None) -> str:
|
||||
if seconds is None:
|
||||
return ""
|
||||
if seconds < 60:
|
||||
return f"{seconds:.1f}s"
|
||||
minutes = int(seconds // 60)
|
||||
secs = seconds % 60
|
||||
return f"{minutes}m {secs:.0f}s"
|
||||
|
||||
|
||||
def render_session(session: Session, show_types: list[str]) -> None:
|
||||
"""Render full session"""
|
||||
if session.init_events:
|
||||
filtered = [e for e in session.init_events if e.type in show_types]
|
||||
if filtered:
|
||||
with st.expander("🚀 **Initialization**", expanded=False):
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
for loop_id in sorted(session.loops.keys()):
|
||||
loop = session.loops[loop_id]
|
||||
render_loop(loop, show_types)
|
||||
|
||||
|
||||
def render_loop(loop: Loop, show_types: list[str]) -> None:
|
||||
"""Render a single loop"""
|
||||
# Get status indicators
|
||||
docker_success = None
|
||||
feedback_decision = None
|
||||
|
||||
for event in loop.running:
|
||||
if event.type == "docker_exec" and event.success is not None:
|
||||
docker_success = event.success
|
||||
|
||||
for event in loop.feedback:
|
||||
if event.type == "feedback" and event.success is not None:
|
||||
feedback_decision = event.success
|
||||
|
||||
# Build title
|
||||
parts = []
|
||||
if docker_success is not None:
|
||||
parts.append("🐳✓" if docker_success else "🐳✗")
|
||||
if feedback_decision is not None:
|
||||
parts.append("✅" if feedback_decision else "❌")
|
||||
|
||||
result_str = " ".join(parts) if parts else ""
|
||||
|
||||
with st.expander(f"🔄 **Loop {loop.loop_id}** {result_str}", expanded=False):
|
||||
# Proposal
|
||||
if loop.proposal:
|
||||
filtered = [e for e in loop.proposal if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 💡 Proposal")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Coding
|
||||
if loop.coding:
|
||||
filtered = [e for e in loop.coding if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 💻 Coding")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Running
|
||||
if loop.running:
|
||||
filtered = [e for e in loop.running if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 🏃 Running")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
# Feedback
|
||||
if loop.feedback:
|
||||
filtered = [e for e in loop.feedback if e.type in show_types]
|
||||
if filtered:
|
||||
st.markdown("#### 📊 Feedback")
|
||||
for event in filtered:
|
||||
render_event(event)
|
||||
|
||||
|
||||
def render_event(event: Event) -> None:
|
||||
"""Render a single event"""
|
||||
icon = ICONS.get(event.type, "📌")
|
||||
duration_str = f" ({format_duration(event.duration)})" if event.duration else ""
|
||||
|
||||
status = ""
|
||||
if event.success is True:
|
||||
status = "🟢 "
|
||||
elif event.success is False:
|
||||
status = "🔴 "
|
||||
|
||||
title = f"{event.time_str} {icon} {status}{event.title}{duration_str}"
|
||||
|
||||
renderers = {
|
||||
"scenario": render_scenario,
|
||||
"llm_call": render_llm_call,
|
||||
"template": render_template,
|
||||
"experiment": render_experiment,
|
||||
"code": render_code,
|
||||
"docker_exec": render_docker_exec,
|
||||
"feedback": render_feedback,
|
||||
"token": render_token,
|
||||
"time": render_time_info,
|
||||
"settings": render_settings,
|
||||
"hypothesis": render_hypothesis,
|
||||
}
|
||||
|
||||
renderer = renderers.get(event.type, render_generic)
|
||||
with st.expander(title, expanded=False):
|
||||
renderer(event.content)
|
||||
|
||||
|
||||
def render_scenario(content: Any) -> None:
|
||||
if hasattr(content, "base_model"):
|
||||
st.markdown(f"**Base Model:** `{content.base_model}`")
|
||||
if hasattr(content, "benchmark"):
|
||||
st.markdown(f"**Benchmark:** `{content.benchmark}`")
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_hypothesis(content: Any) -> None:
|
||||
if hasattr(content, "hypothesis") and content.hypothesis:
|
||||
st.markdown("**Hypothesis:**")
|
||||
st.markdown(content.hypothesis)
|
||||
if hasattr(content, "reason") and content.reason:
|
||||
with st.expander("Reason", expanded=False):
|
||||
st.markdown(content.reason)
|
||||
|
||||
|
||||
def render_settings(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
st.json(content)
|
||||
else:
|
||||
st.code(str(content), wrap_lines=True)
|
||||
|
||||
|
||||
def render_llm_call(content: Any) -> None:
|
||||
if not isinstance(content, dict):
|
||||
st.json(content) if content else st.info("No content")
|
||||
return
|
||||
|
||||
if content.get("start") and content.get("end"):
|
||||
duration = (content["end"] - content["start"]).total_seconds()
|
||||
st.caption(f"Duration: {format_duration(duration)}")
|
||||
|
||||
system = content.get("system", "")
|
||||
if system:
|
||||
with st.expander("System Prompt", expanded=False):
|
||||
st.code(system, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
user = content.get("user", "")
|
||||
if user:
|
||||
with st.expander("User Prompt", expanded=False):
|
||||
st.code(user, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
resp = content.get("resp", "")
|
||||
if resp:
|
||||
st.markdown("**Response:**")
|
||||
if resp.strip().startswith("{") or resp.strip().startswith("["):
|
||||
st.code(resp, language="json", line_numbers=True, wrap_lines=True)
|
||||
elif resp.strip().startswith("```"):
|
||||
st.markdown(resp)
|
||||
else:
|
||||
st.code(resp, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def render_template(content: Any) -> None:
|
||||
if not isinstance(content, dict):
|
||||
st.json(content) if content else st.info("No content")
|
||||
return
|
||||
|
||||
uri = content.get("uri", "")
|
||||
st.caption(f"URI: `{uri}`")
|
||||
|
||||
context = content.get("context", {})
|
||||
if context:
|
||||
with st.expander("Context Variables", expanded=False):
|
||||
st.json(context)
|
||||
|
||||
rendered = content.get("rendered", "")
|
||||
if rendered:
|
||||
with st.expander("Rendered", expanded=True):
|
||||
st.code(rendered, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
|
||||
def render_experiment(content: Any) -> None:
|
||||
if isinstance(content, list):
|
||||
for i, task in enumerate(content):
|
||||
if len(content) > 1:
|
||||
st.markdown(f"**Task {i}**")
|
||||
if hasattr(task, "description") and task.description:
|
||||
st.markdown(task.description)
|
||||
else:
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_code(content: Any) -> None:
|
||||
if isinstance(content, list):
|
||||
for ws in content:
|
||||
if hasattr(ws, "file_dict") and ws.file_dict:
|
||||
for filename, code in ws.file_dict.items():
|
||||
lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
|
||||
with st.expander(filename, expanded=False):
|
||||
st.code(code, language=lang, line_numbers=True, wrap_lines=True)
|
||||
elif hasattr(content, "file_dict") and content.file_dict:
|
||||
for filename, code in content.file_dict.items():
|
||||
lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
|
||||
with st.expander(filename, expanded=False):
|
||||
st.code(code, language=lang, line_numbers=True, wrap_lines=True)
|
||||
else:
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_docker_exec(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
exit_code = content.get("exit_code")
|
||||
if exit_code is not None:
|
||||
if exit_code == 0:
|
||||
st.success(f"Exit code: {exit_code}")
|
||||
else:
|
||||
st.error(f"Exit code: {exit_code}")
|
||||
|
||||
stdout = content.get("stdout", "")
|
||||
if stdout:
|
||||
with st.expander("Output", expanded=True):
|
||||
st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
|
||||
else:
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_feedback(content: Any) -> None:
|
||||
# Handle benchmark result (dict)
|
||||
if isinstance(content, dict):
|
||||
if "accuracy" in content or "accuracy_summary" in content:
|
||||
st.markdown("**Benchmark Result:**")
|
||||
st.json(content)
|
||||
else:
|
||||
st.json(content)
|
||||
return
|
||||
|
||||
# Handle HypothesisFeedback object
|
||||
col1, col2 = st.columns(2)
|
||||
with col1:
|
||||
decision = getattr(content, "decision", None)
|
||||
if decision is not None:
|
||||
st.metric("Decision", "Accept" if decision else "Reject")
|
||||
|
||||
reason = getattr(content, "reason", None)
|
||||
if reason:
|
||||
with st.expander("Reason", expanded=True):
|
||||
st.code(reason, language="text", line_numbers=True, wrap_lines=True)
|
||||
|
||||
code_change = getattr(content, "code_change_summary", None)
|
||||
if code_change:
|
||||
with st.expander("Code Change Summary", expanded=False):
|
||||
st.markdown(code_change)
|
||||
|
||||
|
||||
def render_token(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
col1, col2, col3 = st.columns(3)
|
||||
with col1:
|
||||
st.metric("Prompt", content.get("prompt_tokens", 0))
|
||||
with col2:
|
||||
st.metric("Completion", content.get("completion_tokens", 0))
|
||||
with col3:
|
||||
st.metric("Total", content.get("total_tokens", 0))
|
||||
else:
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_time_info(content: Any) -> None:
|
||||
if isinstance(content, dict):
|
||||
for k, v in content.items():
|
||||
st.metric(k, f"{v:.1f}s" if isinstance(v, (int, float)) else str(v))
|
||||
else:
|
||||
render_generic(content)
|
||||
|
||||
|
||||
def render_generic(content: Any) -> None:
|
||||
if hasattr(content, "__dict__"):
|
||||
st.json(vars(content))
|
||||
elif content:
|
||||
st.json(content)
|
||||
else:
|
||||
st.info("No content")
|
||||
|
||||
|
||||
def render_summary(summary: dict) -> None:
|
||||
col1, col2, col3, col4 = st.columns(4)
|
||||
with col1:
|
||||
st.metric("Loops", summary.get("loop_count", 0))
|
||||
with col2:
|
||||
st.metric("LLM Calls", summary.get("llm_call_count", 0))
|
||||
with col3:
|
||||
llm_time = summary.get("llm_total_time", 0)
|
||||
st.metric("LLM Time", format_duration(llm_time))
|
||||
with col4:
|
||||
success = summary.get("docker_success", 0)
|
||||
fail = summary.get("docker_fail", 0)
|
||||
st.metric("Docker", f"{success}✓ / {fail}✗")
|
||||
@@ -0,0 +1,54 @@
|
||||
"""
|
||||
RL UI Configuration Constants
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
# Event type definition
|
||||
EventType = Literal[
|
||||
"scenario",
|
||||
"llm_call",
|
||||
"template",
|
||||
"experiment",
|
||||
"code",
|
||||
"docker_exec",
|
||||
"feedback",
|
||||
"token",
|
||||
"time",
|
||||
"settings",
|
||||
"hypothesis",
|
||||
]
|
||||
|
||||
# Event type icons
|
||||
ICONS = {
|
||||
"scenario": "🎯",
|
||||
"llm_call": "💬",
|
||||
"template": "📋",
|
||||
"experiment": "🧪",
|
||||
"code": "📄",
|
||||
"docker_exec": "🐳",
|
||||
"feedback": "📊",
|
||||
"token": "🔢",
|
||||
"time": "⏱️",
|
||||
"settings": "⚙️",
|
||||
"hypothesis": "💡",
|
||||
}
|
||||
|
||||
# Always visible event types
|
||||
ALWAYS_VISIBLE_TYPES = [
|
||||
"scenario",
|
||||
"hypothesis",
|
||||
"llm_call",
|
||||
"experiment",
|
||||
"code",
|
||||
"docker_exec",
|
||||
"feedback",
|
||||
]
|
||||
|
||||
# Optional event types with toggle config (label, default_enabled)
|
||||
OPTIONAL_TYPES = {
|
||||
"template": ("📋 Template", False),
|
||||
"token": ("🔢 Token", False),
|
||||
"time": ("⏱️ Time", False),
|
||||
"settings": ("⚙️ Settings", False),
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
RL UI Data Loader
|
||||
Load pkl logs and convert to hierarchical timeline structure
|
||||
Simplified version: no EvoLoop (RL doesn't have evolution loops)
|
||||
"""
|
||||
|
||||
import pickle
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import streamlit as st
|
||||
|
||||
from rdagent.app.rl.ui.config import EventType
|
||||
from rdagent.log.storage import FileStorage
|
||||
|
||||
|
||||
@dataclass
|
||||
class Event:
|
||||
"""Timeline event"""
|
||||
|
||||
type: EventType
|
||||
timestamp: datetime
|
||||
tag: str
|
||||
title: str
|
||||
content: Any
|
||||
loop_id: int | None = None
|
||||
stage: str = ""
|
||||
duration: float | None = None
|
||||
success: bool | None = None
|
||||
|
||||
@property
|
||||
def time_str(self) -> str:
|
||||
return self.timestamp.strftime("%H:%M:%S")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Loop:
|
||||
"""Main loop containing stages (no EvoLoop for RL)"""
|
||||
|
||||
loop_id: int
|
||||
proposal: list[Event] = field(default_factory=list) # hypothesis generation
|
||||
coding: list[Event] = field(default_factory=list) # code generation
|
||||
running: list[Event] = field(default_factory=list) # docker training + benchmark
|
||||
feedback: list[Event] = field(default_factory=list) # feedback
|
||||
|
||||
|
||||
@dataclass
|
||||
class Session:
|
||||
"""Session containing init events and loops"""
|
||||
|
||||
init_events: list[Event] = field(default_factory=list)
|
||||
loops: dict[int, Loop] = field(default_factory=dict)
|
||||
|
||||
|
||||
def extract_loop_id(tag: str) -> int | None:
|
||||
match = re.search(r"Loop_(\d+)", tag)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def extract_stage(tag: str) -> str:
|
||||
if "proposal" in tag or "direct_exp_gen" in tag:
|
||||
return "proposal"
|
||||
if "coding" in tag:
|
||||
return "coding"
|
||||
if "running" in tag:
|
||||
return "running"
|
||||
if "feedback" in tag:
|
||||
return "feedback"
|
||||
return ""
|
||||
|
||||
|
||||
def get_valid_sessions(log_folder: Path) -> list[str]:
|
||||
if not log_folder.exists():
|
||||
return []
|
||||
sessions = []
|
||||
for d in log_folder.iterdir():
|
||||
if d.is_dir() and d.joinpath("__session__").exists():
|
||||
sessions.append(d.name)
|
||||
return sorted(sessions, reverse=True)
|
||||
|
||||
|
||||
def parse_event(tag: str, content: Any, timestamp: datetime) -> Event | None:
|
||||
loop_id = extract_loop_id(tag)
|
||||
stage = extract_stage(tag)
|
||||
|
||||
# Scenario
|
||||
if tag == "scenario":
|
||||
return Event(type="scenario", timestamp=timestamp, tag=tag, title="Scenario", content=content)
|
||||
|
||||
# Settings
|
||||
if "SETTINGS" in tag:
|
||||
name = tag.replace("_SETTINGS", "").replace("SETTINGS", "")
|
||||
return Event(type="settings", timestamp=timestamp, tag=tag, title=f"Settings: {name}", content=content)
|
||||
|
||||
# Hypothesis
|
||||
if "hypothesis" in tag:
|
||||
return Event(
|
||||
type="hypothesis",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Hypothesis",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="proposal",
|
||||
)
|
||||
|
||||
# LLM Call
|
||||
if "debug_llm" in tag:
|
||||
if isinstance(content, dict) and ("user" in content or "system" in content):
|
||||
duration = None
|
||||
if content.get("start") and content.get("end"):
|
||||
duration = (content["end"] - content["start"]).total_seconds()
|
||||
return Event(
|
||||
type="llm_call",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="LLM Call",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage,
|
||||
duration=duration,
|
||||
)
|
||||
|
||||
# Template
|
||||
if "debug_tpl" in tag:
|
||||
if isinstance(content, dict) and "uri" in content:
|
||||
uri = content.get("uri", "")
|
||||
tpl_name = uri.split(":")[-1] if ":" in uri else uri
|
||||
return Event(
|
||||
type="template",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Template: {tpl_name}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
# Experiment/Coder result
|
||||
if "coder result" in tag or "experiment generation" in tag:
|
||||
return Event(
|
||||
type="experiment",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Experiment",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage or "coding",
|
||||
)
|
||||
|
||||
# Code
|
||||
if "evolving code" in tag or "code" in tag.lower():
|
||||
return Event(
|
||||
type="code",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Code",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage or "coding",
|
||||
)
|
||||
|
||||
# Docker run
|
||||
if "docker_run" in tag:
|
||||
exit_code = content.get("exit_code") if isinstance(content, dict) else None
|
||||
success = exit_code == 0 if exit_code is not None else None
|
||||
return Event(
|
||||
type="docker_exec",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Docker Run {'✓' if success else '✗' if success is False else ''}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="running",
|
||||
success=success,
|
||||
)
|
||||
|
||||
# Benchmark result
|
||||
if "benchmark" in tag.lower():
|
||||
return Event(
|
||||
type="feedback",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Benchmark Result",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="running",
|
||||
)
|
||||
|
||||
# Feedback
|
||||
if "feedback" in tag:
|
||||
decision = getattr(content, "decision", None)
|
||||
return Event(
|
||||
type="feedback",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Feedback: {'Accept' if decision else 'Reject'}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage="feedback",
|
||||
success=decision,
|
||||
)
|
||||
|
||||
# Token cost
|
||||
if "token_cost" in tag:
|
||||
if isinstance(content, dict):
|
||||
total = content.get("total_tokens", 0)
|
||||
return Event(
|
||||
type="token",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title=f"Token: {total}",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
# Time info
|
||||
if "time_info" in tag:
|
||||
return Event(
|
||||
type="time",
|
||||
timestamp=timestamp,
|
||||
tag=tag,
|
||||
title="Time Info",
|
||||
content=content,
|
||||
loop_id=loop_id,
|
||||
stage=stage,
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
@st.cache_data(ttl=300, hash_funcs={Path: str})
|
||||
def load_session(log_path: Path) -> Session:
|
||||
"""Load events into hierarchical session structure"""
|
||||
session = Session()
|
||||
|
||||
# 手动遍历 pkl 文件，跳过无法加载的
|
||||
events = []
|
||||
pkl_files = sorted(log_path.rglob("*.pkl"))
|
||||
for pkl_file in pkl_files:
|
||||
if pkl_file.name == "debug_llm.pkl":
|
||||
continue
|
||||
try:
|
||||
with open(pkl_file, "rb") as f:
|
||||
content = pickle.load(f)
|
||||
timestamp = datetime.strptime(pkl_file.stem, "%Y-%m-%d_%H-%M-%S-%f")
|
||||
# 正确解析 tag：Loop_5/running/debug_tpl/2957404/xxx.pkl -> Loop_5.running.debug_tpl
|
||||
tag = ".".join(pkl_file.relative_to(log_path).as_posix().replace("/", ".").split(".")[:-3])
|
||||
event = parse_event(tag, content, timestamp)
|
||||
if event:
|
||||
events.append(event)
|
||||
except (ModuleNotFoundError, ImportError, pickle.UnpicklingError, ValueError):
|
||||
# 跳过无法加载的文件（不同 Python 版本或格式错误）
|
||||
continue
|
||||
|
||||
events.sort(key=lambda e: e.timestamp)
|
||||
|
||||
for event in events:
|
||||
if event.loop_id is None:
|
||||
session.init_events.append(event)
|
||||
continue
|
||||
|
||||
if event.loop_id not in session.loops:
|
||||
session.loops[event.loop_id] = Loop(loop_id=event.loop_id)
|
||||
loop = session.loops[event.loop_id]
|
||||
|
||||
if event.stage == "proposal":
|
||||
loop.proposal.append(event)
|
||||
elif event.stage == "coding":
|
||||
loop.coding.append(event)
|
||||
elif event.stage == "running":
|
||||
loop.running.append(event)
|
||||
elif event.stage == "feedback":
|
||||
loop.feedback.append(event)
|
||||
else:
|
||||
loop.proposal.append(event)
|
||||
|
||||
return session
|
||||
|
||||
|
||||
def get_summary(session: Session) -> dict:
|
||||
"""Get summary statistics"""
|
||||
llm_calls = []
|
||||
docker_execs = []
|
||||
|
||||
for e in session.init_events:
|
||||
if e.type == "llm_call":
|
||||
llm_calls.append(e)
|
||||
elif e.type == "docker_exec":
|
||||
docker_execs.append(e)
|
||||
|
||||
for loop in session.loops.values():
|
||||
for e in loop.proposal + loop.coding + loop.running + loop.feedback:
|
||||
if e.type == "llm_call":
|
||||
llm_calls.append(e)
|
||||
elif e.type == "docker_exec":
|
||||
docker_execs.append(e)
|
||||
|
||||
return {
|
||||
"loop_count": len(session.loops),
|
||||
"llm_call_count": len(llm_calls),
|
||||
"llm_total_time": sum(e.duration or 0 for e in llm_calls),
|
||||
"docker_success": sum(1 for e in docker_execs if e.success is True),
|
||||
"docker_fail": sum(1 for e in docker_execs if e.success is False),
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
"""
|
||||
RL Job Summary View
|
||||
Display summary table for all tasks in a job directory
|
||||
"""
|
||||
|
||||
import pickle
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import streamlit as st
|
||||
|
||||
|
||||
def is_valid_task(task_path: Path) -> bool:
|
||||
"""Check if directory is a valid RL task (has __session__ subdirectory)"""
|
||||
return task_path.is_dir() and (task_path / "__session__").exists()
|
||||
|
||||
|
||||
def get_loop_dirs(task_path: Path) -> list[Path]:
|
||||
"""Get sorted list of Loop directories"""
|
||||
loops = [d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")]
|
||||
return sorted(loops, key=lambda d: int(d.name.split("_")[1]))
|
||||
|
||||
|
||||
def get_loop_status(task_path: Path, loop_id: int) -> tuple[str, bool | None]:
|
||||
"""
|
||||
Get loop status and feedback decision.
|
||||
Returns: (status_str, feedback_decision)
|
||||
Status: 'C'=Coding, 'R'=Running, 'X'=Failed, 'OK'=Success
|
||||
"""
|
||||
loop_path = task_path / f"Loop_{loop_id}"
|
||||
if not loop_path.exists():
|
||||
return "-", None
|
||||
|
||||
# Check for feedback
|
||||
feedback_decision = None
|
||||
feedback_files = list(loop_path.rglob("**/feedback/**/*.pkl"))
|
||||
for f in feedback_files:
|
||||
try:
|
||||
with open(f, "rb") as fp:
|
||||
content = pickle.load(fp)
|
||||
decision = getattr(content, "decision", None)
|
||||
if decision is not None:
|
||||
feedback_decision = decision
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if feedback_decision is not None:
|
||||
return ("OK" if feedback_decision else "X"), feedback_decision
|
||||
|
||||
# Check running stage
|
||||
running_files = list(loop_path.rglob("**/running/**/*.pkl"))
|
||||
if running_files:
|
||||
return "R", None
|
||||
|
||||
# Check coding stage
|
||||
coding_files = list(loop_path.rglob("**/coding/**/*.pkl"))
|
||||
if coding_files:
|
||||
return "C", None
|
||||
|
||||
return "?", None
|
||||
|
||||
|
||||
def get_max_loops(job_path: Path) -> int:
|
||||
"""Get maximum number of loops across all tasks"""
|
||||
max_loops = 0
|
||||
for task_dir in job_path.iterdir():
|
||||
if is_valid_task(task_dir):
|
||||
loops = get_loop_dirs(task_dir)
|
||||
max_loops = max(max_loops, len(loops))
|
||||
return max_loops
|
||||
|
||||
|
||||
def get_job_summary_df(job_path: Path) -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||
"""Generate summary DataFrame for all tasks in job"""
|
||||
if not job_path.exists():
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
|
||||
if not tasks:
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
max_loops = get_max_loops(job_path)
|
||||
if max_loops == 0:
|
||||
max_loops = 10
|
||||
|
||||
data = []
|
||||
decisions_data = []
|
||||
for task_path in tasks:
|
||||
row = {"Task": task_path.name}
|
||||
decision_row = {"Task": task_path.name}
|
||||
success_count = 0
|
||||
fail_count = 0
|
||||
|
||||
for i in range(max_loops):
|
||||
status, feedback_decision = get_loop_status(task_path, i)
|
||||
row[f"L{i}"] = status
|
||||
decision_row[f"L{i}"] = feedback_decision
|
||||
if feedback_decision is True:
|
||||
success_count += 1
|
||||
elif feedback_decision is False:
|
||||
fail_count += 1
|
||||
|
||||
row["Summary"] = f"{success_count}✓/{fail_count}✗"
|
||||
decision_row["Summary"] = None
|
||||
data.append(row)
|
||||
decisions_data.append(decision_row)
|
||||
|
||||
df = pd.DataFrame(data)
|
||||
decisions_df = pd.DataFrame(decisions_data)
|
||||
if not df.empty:
|
||||
loop_cols = [c for c in df.columns if c.startswith("L")]
|
||||
cols = ["Task"] + sorted(loop_cols, key=lambda x: int(x[1:])) + ["Summary"]
|
||||
df = df[cols]
|
||||
decisions_df = decisions_df[cols]
|
||||
return df, decisions_df
|
||||
|
||||
|
||||
def style_status_cell(val: str, decision: bool | None = None) -> str:
|
||||
"""Style cell based on status value"""
|
||||
if val == "-":
|
||||
return "color: #888"
|
||||
if val == "C":
|
||||
return "color: #f0ad4e; font-weight: bold"
|
||||
if val == "R":
|
||||
return "color: #5bc0de; font-weight: bold"
|
||||
if val == "X":
|
||||
return "color: #d9534f; font-weight: bold"
|
||||
if val == "OK":
|
||||
return "color: #5cb85c; font-weight: bold"
|
||||
if val == "?":
|
||||
return "color: #888"
|
||||
return ""
|
||||
|
||||
|
||||
def style_df_with_decisions(df: pd.DataFrame, decisions_df: pd.DataFrame):
|
||||
"""Apply styling to dataframe"""
|
||||
|
||||
def apply_styles(row_idx: int, col: str) -> str:
|
||||
val = df.iloc[row_idx][col]
|
||||
decision = decisions_df.iloc[row_idx][col] if col in decisions_df.columns else None
|
||||
return style_status_cell(str(val), decision)
|
||||
|
||||
styles = pd.DataFrame("", index=df.index, columns=df.columns)
|
||||
for row_idx in range(len(df)):
|
||||
for col in df.columns:
|
||||
styles.iloc[row_idx][col] = apply_styles(row_idx, col)
|
||||
|
||||
return df.style.apply(lambda _: styles, axis=None)
|
||||
|
||||
|
||||
def render_job_summary(job_path: Path, is_root: bool = False) -> None:
|
||||
"""Render job summary UI"""
|
||||
title = "Standalone Tasks" if is_root else f"Job: {job_path.name}"
|
||||
st.subheader(title)
|
||||
|
||||
df, decisions_df = get_job_summary_df(job_path)
|
||||
if df.empty:
|
||||
st.warning("No valid tasks found in this job directory")
|
||||
return
|
||||
|
||||
st.markdown(
|
||||
"**Legend:** "
|
||||
"<span style='color:#f0ad4e'>C</span>=Coding, "
|
||||
"<span style='color:#5bc0de'>R</span>=Running, "
|
||||
"<span style='color:#5cb85c'>OK</span>=Success, "
|
||||
"<span style='color:#d9534f'>X</span>=Failed",
|
||||
unsafe_allow_html=True,
|
||||
)
|
||||
|
||||
styled_df = style_df_with_decisions(df, decisions_df)
|
||||
st.dataframe(styled_df, use_container_width=True, hide_index=True)
|
||||
|
||||
col1, col2, col3 = st.columns(3)
|
||||
with col1:
|
||||
st.metric("Tasks", len(df))
|
||||
with col2:
|
||||
loop_cols = [c for c in decisions_df.columns if c.startswith("L")]
|
||||
tasks_success = decisions_df[loop_cols].apply(lambda row: any(v is True for v in row), axis=1).sum()
|
||||
st.metric("With Success", tasks_success)
|
||||
with col3:
|
||||
total_loops = sum(1 for _, row in decisions_df.iterrows() for c in loop_cols if row[c] is not None)
|
||||
st.metric("Total Loops", total_loops)
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Shared benchmark evaluation utilities."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
# 共享配置目录
|
||||
BENCHMARK_CONFIGS_DIR = Path(__file__).parent / "configs"
|
||||
@@ -0,0 +1 @@
|
||||
"""Shared OpenCompass benchmark configurations."""
|
||||
@@ -0,0 +1,102 @@
|
||||
# Model Inference Parameters Configuration
|
||||
# Used by benchmark.py to determine inference settings for different models
|
||||
|
||||
# Default configuration (used when model is not explicitly listed)
|
||||
default:
|
||||
temperature: 0.0 # Greedy decoding for reproducible results
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
batch_size: 16
|
||||
tensor_parallel_size: auto # Will be auto-determined based on GPU count
|
||||
gpu_memory_utilization: 0.9
|
||||
repetition_penalty: 1.0
|
||||
dtype: bfloat16
|
||||
enable_thinking: false
|
||||
use_cot_postprocessor: true # Enable CoT postprocessor to extract answer from <think>...</think>answer format
|
||||
|
||||
# Model-specific configurations (override default values)
|
||||
models:
|
||||
# Qwen3 series - support thinking mode and longer sequences
|
||||
"Qwen/Qwen3-8B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true # Qwen3-specific feature
|
||||
|
||||
"Qwen/Qwen3-32B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true
|
||||
|
||||
"Qwen/Qwen3-1.7B":
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
max_seq_len: 40960
|
||||
max_out_len: 38912
|
||||
enable_thinking: true
|
||||
gpu_memory_utilization: 0.7 # It does not use too much GPU memory. But it is worth
|
||||
|
||||
# Qwen2.5 series - standard configuration with CoT postprocessor for fine-tuned models
|
||||
"Qwen/Qwen2.5-0.5B-Instruct":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
gpu_memory_utilization: 0.5 # 0.5B model is very small, no need for 0.9
|
||||
|
||||
"Qwen/Qwen2.5-0.5B":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
gpu_memory_utilization: 0.5
|
||||
|
||||
"Qwen/Qwen2.5-7B-Instruct":
|
||||
temperature: 0.0 # Greedy decoding for consistency
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
use_cot_postprocessor: true # Extract answer from CoT format after fine-tuning
|
||||
|
||||
"Qwen/Qwen2.5-32B-Instruct":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
|
||||
# Llama 3.1 series (128K context, 4K max output)
|
||||
"meta-llama/Llama-3.1-8B-Instruct":
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: 40
|
||||
max_seq_len: 32768 # 131072
|
||||
max_out_len: 4096
|
||||
|
||||
|
||||
# Mistral series
|
||||
"mistralai/Mistral-7B-Instruct-v0.3":
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: 50
|
||||
max_seq_len: 32768
|
||||
max_out_len: 8192
|
||||
|
||||
# DeepSeek series
|
||||
"deepseek-ai/deepseek-coder-33b-instruct":
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 1
|
||||
max_seq_len: 16384
|
||||
max_out_len: 4096
|
||||
@@ -0,0 +1,116 @@
|
||||
# Auto-generated OpenCompass Config for RD-Agent Benchmark
|
||||
# DO NOT EDIT MANUALLY - Generated by benchmark.py
|
||||
|
||||
template: |-
|
||||
from mmengine.config import read_base
|
||||
from opencompass.models import VLLMwithChatTemplate
|
||||
|
||||
# ==================== Dataset Import ====================
|
||||
# Use explicit imports (not `import *`) to avoid leaking non-serializable
|
||||
# objects from dataset configs into the namespace.
|
||||
with read_base():
|
||||
{% for imp in dataset_imports %}
|
||||
{% if imp.names %}
|
||||
from {{ imp.module }} import {{ imp.names | join(', ') }}
|
||||
{% else %}
|
||||
from {{ imp.module }} import *
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
# Aggregate all dataset variables
|
||||
datasets = sum([v for k, v in locals().items() if (k == 'datasets' or k.endswith('_datasets')) and isinstance(v, list)], [])
|
||||
|
||||
# Apply dataset modifications
|
||||
for ds in datasets:
|
||||
{% if test_range %}
|
||||
# Apply dataset range (e.g., "[:100]" for validation, "[-100:]" for test)
|
||||
if 'reader_cfg' not in ds:
|
||||
ds['reader_cfg'] = {}
|
||||
ds['reader_cfg']['test_range'] = '{{ test_range }}'
|
||||
|
||||
# Sync to evaluator's dataset_cfg
|
||||
if 'eval_cfg' in ds and 'evaluator' in ds['eval_cfg']:
|
||||
evaluator = ds['eval_cfg']['evaluator']
|
||||
if isinstance(evaluator, dict) and 'dataset_cfg' in evaluator:
|
||||
if 'reader_cfg' not in evaluator['dataset_cfg']:
|
||||
evaluator['dataset_cfg']['reader_cfg'] = {}
|
||||
evaluator['dataset_cfg']['reader_cfg']['test_range'] = '{{ test_range }}'
|
||||
{% endif %}
|
||||
{% if num_runs and num_runs > 1 %}
|
||||
# Multiple runs (repeat each sample n times for averaging or pass@k)
|
||||
ds['n'] = {{ num_runs }}
|
||||
{% endif %}
|
||||
{% if pass_k %}
|
||||
# Pass@k evaluation
|
||||
ds['k'] = {{ pass_k }}
|
||||
{% endif %}
|
||||
pass
|
||||
|
||||
# ==================== Model Configuration ====================
|
||||
models = [
|
||||
dict(
|
||||
type=VLLMwithChatTemplate,
|
||||
abbr='{{ model_abbr }}',
|
||||
path='{{ model_path }}',
|
||||
model_kwargs=dict(
|
||||
tensor_parallel_size={{ tensor_parallel_size }},
|
||||
gpu_memory_utilization={{ gpu_memory_utilization }},
|
||||
trust_remote_code=True,
|
||||
dtype='{{ dtype }}',
|
||||
max_model_len={{ max_seq_len }},
|
||||
),
|
||||
max_seq_len={{ max_seq_len }},
|
||||
max_out_len={{ max_out_len }},
|
||||
batch_size={{ batch_size }},
|
||||
generation_kwargs=dict(
|
||||
temperature={{ temperature }},
|
||||
top_p={{ top_p }},
|
||||
top_k={{ top_k }},
|
||||
{% if repetition_penalty != 1.0 %}
|
||||
repetition_penalty={{ repetition_penalty }},
|
||||
{% endif %}
|
||||
),
|
||||
{% if enable_thinking %}
|
||||
chat_template_kwargs=dict(enable_thinking=True),
|
||||
{% endif %}
|
||||
{% if enable_thinking or use_cot_postprocessor %}
|
||||
pred_postprocessor=dict(type='extract-non-reasoning-content'),
|
||||
{% endif %}
|
||||
run_cfg=dict(
|
||||
num_gpus={{ tensor_parallel_size }},
|
||||
num_procs=1,
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
# ==================== Inference Configuration ====================
|
||||
infer = dict(
|
||||
partitioner=dict(
|
||||
type='NaivePartitioner',
|
||||
),
|
||||
runner=dict(
|
||||
type='LocalRunner',
|
||||
max_num_workers=16,
|
||||
task=dict(
|
||||
type='OpenICLInferTask',
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# ==================== Evaluation Configuration ====================
|
||||
eval = dict(
|
||||
partitioner=dict(
|
||||
type='NaivePartitioner',
|
||||
),
|
||||
runner=dict(
|
||||
type='LocalRunner',
|
||||
max_num_workers=16,
|
||||
task=dict(
|
||||
type='OpenICLEvalTask',
|
||||
dump_details=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# ==================== Work Directory ====================
|
||||
work_dir = '{{ work_dir }}'
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Utilities shared by benchmark evaluators."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
import re
|
||||
from typing import Dict, Iterable, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _guess_dataset_var(mod_path: str) -> str:
|
||||
"""Guess the dataset variable name from an OpenCompass module path.
|
||||
|
||||
Convention: ``opencompass.configs.datasets.<name>.<name>_gen_<hash>``
|
||||
exports ``<name>_datasets``. E.g.:
|
||||
- ``bbh.bbh_gen_ee62e9`` → ``bbh_datasets``
|
||||
- ``gsm8k.gsm8k_gen_1d7fe4`` → ``gsm8k_datasets``
|
||||
- ``ARC_c.ARC_c_gen_1e0de5`` → ``ARC_c_datasets``
|
||||
"""
|
||||
# Take the parent package name (e.g. "bbh" from "...datasets.bbh.bbh_gen_xxx")
|
||||
parts = mod_path.rsplit(".", 2)
|
||||
if len(parts) >= 2:
|
||||
parent = parts[-2] # e.g. "bbh", "gsm8k", "ARC_c"
|
||||
return f"{parent}_datasets"
|
||||
return "datasets"
|
||||
|
||||
|
||||
def build_dataset_imports_explicit(dataset_imports: str | Iterable[str]) -> List[Dict[str, object]]:
|
||||
"""Build explicit dataset import specs for the OpenCompass config template.
|
||||
|
||||
Resolve explicit dataset variable names to avoid `import *`, which leaks
|
||||
non-serializable objects (e.g. `os`, `f` from BBH) and breaks mmengine's
|
||||
config dump+reload in the CLI.
|
||||
|
||||
The returned structure matches `opencompass_template.yaml` expectation:
|
||||
`[{ "module": "...", "names": ["datasets", "..._datasets"] }, ...]`.
|
||||
"""
|
||||
modules = [dataset_imports] if isinstance(dataset_imports, str) else list(dataset_imports)
|
||||
explicit: List[Dict[str, object]] = []
|
||||
for mod_path in modules:
|
||||
try:
|
||||
mod = importlib.import_module(mod_path)
|
||||
names = [
|
||||
attr
|
||||
for attr in dir(mod)
|
||||
if (attr == "datasets" or attr.endswith("_datasets")) and isinstance(getattr(mod, attr), list)
|
||||
]
|
||||
if not names:
|
||||
guessed = _guess_dataset_var(mod_path)
|
||||
logger.warning(
|
||||
"No dataset variables found in %s, guessing '%s'",
|
||||
mod_path,
|
||||
guessed,
|
||||
)
|
||||
names = [guessed]
|
||||
explicit.append({"module": mod_path, "names": names})
|
||||
except Exception as e:
|
||||
guessed = _guess_dataset_var(mod_path)
|
||||
logger.warning(
|
||||
"Failed to import %s for explicit name resolution: %s. " "Guessing variable name '%s'.",
|
||||
mod_path,
|
||||
e,
|
||||
guessed,
|
||||
)
|
||||
explicit.append({"module": mod_path, "names": [guessed]})
|
||||
return explicit
|
||||
@@ -0,0 +1 @@
|
||||
from rdagent.components.coder.rl.costeer import RLCoSTEER
|
||||
@@ -0,0 +1,140 @@
|
||||
"""RL CoSTEER - Code generation component for RL post-training"""
|
||||
|
||||
from typing import Generator
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEERMultiEvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.core.evolving_agent import EvolvingStrategy, EvoStep
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class RLCoderCoSTEERSettings(CoSTEERSettings):
|
||||
"""RL Coder settings."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class RLEvolvingStrategy(EvolvingStrategy):
|
||||
"""RL code generation strategy using LLM."""
|
||||
|
||||
def __init__(self, scen: Scenario, settings: CoSTEERSettings):
|
||||
self.scen = scen
|
||||
self.settings = settings
|
||||
|
||||
def evolve_iter(
|
||||
self,
|
||||
*,
|
||||
evo: EvolvingItem,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
evolving_trace: list[EvoStep] = [],
|
||||
**kwargs,
|
||||
) -> Generator[EvolvingItem, EvolvingItem, None]:
|
||||
"""Generate code for all tasks using LLM."""
|
||||
for index, target_task in enumerate(evo.sub_tasks):
|
||||
code = self._generate_code(target_task, evolving_trace)
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code)
|
||||
|
||||
evo = yield evo
|
||||
return
|
||||
|
||||
def _generate_code(self, task: Task, evolving_trace: list[EvoStep] = []) -> dict[str, str]:
|
||||
"""Generate RL training code using LLM."""
|
||||
from rdagent.app.rl.conf import RL_RD_SETTING
|
||||
|
||||
# 获取上轮反馈
|
||||
feedback = None
|
||||
if evolving_trace:
|
||||
last_step = evolving_trace[-1]
|
||||
if hasattr(last_step, "feedback") and last_step.feedback:
|
||||
feedback = str(last_step.feedback)
|
||||
|
||||
# 构造 prompt
|
||||
system_prompt = T(".prompts:rl_coder.system").r()
|
||||
user_prompt = T(".prompts:rl_coder.user").r(
|
||||
task_description=task.description if hasattr(task, "description") else str(task),
|
||||
base_model=RL_RD_SETTING.base_model or "",
|
||||
benchmark=RL_RD_SETTING.benchmark or "",
|
||||
hypothesis=str(task.name) if hasattr(task, "name") else "Train RL model",
|
||||
feedback=feedback,
|
||||
)
|
||||
|
||||
# 调用 LLM
|
||||
session = APIBackend().build_chat_session(session_system_prompt=system_prompt)
|
||||
code = session.build_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="python",
|
||||
)
|
||||
logger.info(f"LLM generated code:\n{code[:200]}...")
|
||||
return {"main.py": code}
|
||||
|
||||
def _mock_code(self) -> dict[str, str]:
|
||||
"""Fallback mock code."""
|
||||
return {"main.py": """import gymnasium as gym
|
||||
from stable_baselines3 import PPO
|
||||
|
||||
env = gym.make("CartPole-v1")
|
||||
model = PPO("MlpPolicy", env, verbose=1)
|
||||
model.learn(total_timesteps=1000)
|
||||
model.save("ppo_cartpole")
|
||||
print("Training completed!")
|
||||
"""}
|
||||
|
||||
|
||||
class RLCoderEvaluator:
|
||||
"""RL code evaluator (mock implementation)."""
|
||||
|
||||
def __init__(self, scen: Scenario) -> None:
|
||||
self.scen = scen
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace | None,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Evaluate RL code. Currently returns mock success."""
|
||||
# TODO: 实现真正的评估逻辑
|
||||
return CoSTEERSingleFeedback(
|
||||
execution="Mock: executed successfully",
|
||||
return_checking=None,
|
||||
code="Mock: code looks good",
|
||||
final_decision=True,
|
||||
)
|
||||
|
||||
|
||||
class RLCoSTEER(CoSTEER):
|
||||
"""RL CoSTEER - orchestrates code generation and evaluation."""
|
||||
|
||||
def __init__(self, scen: Scenario, *args, **kwargs) -> None:
|
||||
settings = RLCoderCoSTEERSettings()
|
||||
eva = CoSTEERMultiEvaluator([RLCoderEvaluator(scen=scen)], scen=scen)
|
||||
es = RLEvolvingStrategy(scen=scen, settings=settings)
|
||||
|
||||
super().__init__(
|
||||
*args,
|
||||
settings=settings,
|
||||
eva=eva,
|
||||
es=es,
|
||||
scen=scen,
|
||||
max_loop=1,
|
||||
stop_eval_chain_on_fail=False,
|
||||
with_knowledge=False,
|
||||
knowledge_self_gen=False,
|
||||
**kwargs,
|
||||
)
|
||||
@@ -0,0 +1,94 @@
|
||||
rl_coder:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责生成训练代码。
|
||||
|
||||
## 运行环境
|
||||
代码会被部署到 `$WORKSPACE/code/main.py` 并在该目录下执行。
|
||||
以下环境变量已由框架设置，代码中用 `os.environ["..."]` 读取：
|
||||
- `MODEL_PATH`: 基础模型绝对路径（只读）
|
||||
- `DATA_PATH`: 训练数据目录绝对路径（只读）
|
||||
- `OUTPUT_DIR`: 模型输出目录绝对路径（`$WORKSPACE/output/`）
|
||||
- `GRADING_SERVER_URL`: 评测服务地址（训练完后系统自动提交，代码不需要调用）
|
||||
|
||||
## 框架: trl (版本 0.27+)
|
||||
|
||||
## 可用算法
|
||||
- **GRPO**: 推荐，只需 reward function，不需要预构建偏好对
|
||||
- **DPO**: 需要 (prompt, chosen, rejected) 偏好对
|
||||
|
||||
## API 要点
|
||||
|
||||
### GRPOTrainer
|
||||
```python
|
||||
from trl import GRPOConfig, GRPOTrainer
|
||||
|
||||
trainer = GRPOTrainer(
|
||||
model=MODEL_PATH, # 模型路径
|
||||
reward_funcs=reward_fn, # reward 函数
|
||||
args=GRPOConfig(
|
||||
output_dir=OUTPUT_DIR, # 输出目录
|
||||
...
|
||||
),
|
||||
train_dataset=dataset, # 必须有 "prompt" 列
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
```
|
||||
|
||||
### reward function 签名（重要！）
|
||||
```python
|
||||
def reward_fn(completions, answer, **kwargs):
|
||||
# completions: list[str] - 模型生成的回复
|
||||
# answer: list[str] - 数据集中的 answer 列（自动传入）
|
||||
# kwargs: 数据集其他列（如 question）
|
||||
return [float(...) for ...] # 返回 reward 列表
|
||||
```
|
||||
|
||||
### GRPOConfig 关键参数
|
||||
- `num_generations`: 每个 prompt 采样次数，必须 >= 2
|
||||
- `max_completion_length`: 生成最大长度
|
||||
- `per_device_train_batch_size`: 批次大小
|
||||
|
||||
## 输出要求
|
||||
- 生成完整的 `main.py`，可直接运行
|
||||
- 路径全部通过 `os.environ` 获取，**不要硬编码路径**
|
||||
- 数据从 `$DATA_PATH` 下的 jsonl 文件加载
|
||||
- 模型保存到 `$OUTPUT_DIR`（可用子目录如 `$OUTPUT_DIR/v1`）
|
||||
|
||||
## 评测机制
|
||||
训练完成后，系统自动将 `$OUTPUT_DIR` 下最新的模型提交到 Grading Server。
|
||||
- 有模型 → 自动评测，返回 score
|
||||
- 为空 → 跳过评测
|
||||
代码只需负责训练和保存模型，**不需要**自行调用评测 API。
|
||||
|
||||
## 代码模板
|
||||
```python
|
||||
import os
|
||||
MODEL_PATH = os.environ["MODEL_PATH"]
|
||||
DATA_PATH = os.environ["DATA_PATH"]
|
||||
OUTPUT_DIR = os.environ["OUTPUT_DIR"]
|
||||
# ... 训练逻辑 ...
|
||||
trainer.save_model(OUTPUT_DIR)
|
||||
```
|
||||
|
||||
user: |-
|
||||
## 任务
|
||||
{{ task_description }}
|
||||
|
||||
## 基础模型
|
||||
- 名称: {{ base_model }}
|
||||
- 路径: 通过 $MODEL_PATH 环境变量获取
|
||||
|
||||
## 训练数据
|
||||
- 数据集: {{ benchmark }}
|
||||
- 路径: 通过 $DATA_PATH 环境变量获取
|
||||
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
{% if feedback %}
|
||||
## 上轮反馈
|
||||
{{ feedback }}
|
||||
{% endif %}
|
||||
|
||||
请根据数据格式和假设，生成完整的训练代码（main.py）。
|
||||
注意：路径全部通过 os.environ 获取，不要硬编码。
|
||||
@@ -0,0 +1,13 @@
|
||||
# 运行时生成
|
||||
workspace/
|
||||
results.csv
|
||||
log/
|
||||
logs/
|
||||
doc/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
@@ -0,0 +1,387 @@
|
||||
# AutoRL-Bench
|
||||
|
||||
让大模型（如 GPT-5.2）自主驱动 RL 训练流程，提升小模型（如 Qwen2.5-7B）在各类 Benchmark 上的表现，并评测"大模型驱动 RL"的增益效果。
|
||||
|
||||
> 核心问题：给定一个 Benchmark 及其 baseline，大模型通过 Workflow 对小模型进行 RL 训练后，小模型的分数能否超过 baseline？
|
||||
|
||||
| 角色 | 实例 | 职责 |
|
||||
|------|------|------|
|
||||
| **Benchmark** | GSM8K、HumanEval、ALFWorld 等 | 提供任务环境、自动评分 |
|
||||
| **小模型** | Qwen2.5-1.5B/7B | 被 RL 训练的 Agent |
|
||||
| **大模型** | GPT-5.2 等 | 离线驱动 RL 优化（生成 reward、调超参等） |
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 环境安装
|
||||
|
||||
```bash
|
||||
# --- 1a. Clone 代码 ---
|
||||
git clone git@github.com:microsoft/RD-Agent.git ~/RD-Agent
|
||||
cd ~/RD-Agent
|
||||
|
||||
# --- 1b. 基础 conda 环境 ---
|
||||
conda create -n autorl python=3.10 -y
|
||||
conda activate autorl
|
||||
pip install -e .
|
||||
|
||||
# 全局依赖（trl, vllm, torch, opencompass 等）
|
||||
pip install -r rdagent/scenarios/rl/autorl_bench/requirements.txt
|
||||
|
||||
# --- 1c. 按需安装 benchmark 额外依赖 ---
|
||||
|
||||
# ALFWorld
|
||||
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/alfworld/requirements.txt
|
||||
|
||||
# GSM8K：无额外依赖
|
||||
|
||||
# HumanEval
|
||||
git clone https://github.com/XianBW/human-eval.git ~/human-eval
|
||||
cd ~/human-eval && pip install -e .
|
||||
cd ~/RD-Agent
|
||||
|
||||
# WebShop（需要 Java 11+）
|
||||
conda install -c conda-forge openjdk=11 faiss-cpu -y
|
||||
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/webshop/requirements.txt
|
||||
python -c "from spacy.cli.download import download; download('en_core_web_sm')"
|
||||
|
||||
# AlpacaEval 2.0
|
||||
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/alpacaeval/requirements.txt
|
||||
|
||||
# --- 1d. OpenHands Agent（如需使用）---
|
||||
git clone git@github.com:couragec/openhands-rl.git ~/openhands-rl
|
||||
# OpenHands 需要独立 conda 环境（Python 3.12）
|
||||
conda create -n openhands python=3.12 -y
|
||||
conda run -n openhands pip install -r ~/openhands-rl/requirements.txt
|
||||
```
|
||||
|
||||
### 2. 配置 `.env`
|
||||
|
||||
```bash
|
||||
cp .env.example .env # 或手动创建
|
||||
```
|
||||
|
||||
`.env` 中需要配置的关键项：
|
||||
|
||||
```env
|
||||
# LLM API（OpenHands Agent 必需）
|
||||
OPENAI_API_KEY=your_api_key
|
||||
OPENAI_API_BASE=https://your-api-endpoint/v1
|
||||
CHAT_MODEL=gpt-5.2
|
||||
|
||||
# OpenHands 环境（可选，有默认值）
|
||||
# CONDA_ENV_OPENHANDS=openhands # 默认 openhands
|
||||
# OPENHANDS_RL_ROOT=$HOME/openhands-rl # 默认 ~/openhands-rl
|
||||
|
||||
# rl-smith benchmark 路径（可选，默认 ../rl-smith/benchmarks/）
|
||||
# SMITH_BENCH_DIR=/path/to/rl-smith/benchmarks
|
||||
```
|
||||
|
||||
### 3. 运行
|
||||
|
||||
```bash
|
||||
cd /path/to/RD-Agent
|
||||
conda activate autorl
|
||||
|
||||
# Example Agent（简单 GRPO 训练，验证流程）
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent example_agent --task gsm8k --model Qwen/Qwen2.5-1.5B --timeout 7200
|
||||
|
||||
# OpenHands Agent + GSM8K
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent openhands --task gsm8k --model Qwen/Qwen2.5-1.5B --timeout 41600
|
||||
|
||||
# OpenHands Agent + ALFWorld（首次运行自动下载 ~2GB 游戏数据）
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent openhands --task alfworld --model Qwen/Qwen2.5-1.5B-Instruct --timeout 41600
|
||||
|
||||
# Smith benchmark（来自 rl-smith，自动发现，无需手动注册）
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent openhands --task smith-bbh --model Qwen/Qwen2.5-1.5B --timeout 7200
|
||||
|
||||
# 后台运行（推荐）
|
||||
nohup python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent openhands --task alfworld --model Qwen/Qwen2.5-1.5B-Instruct \
|
||||
--timeout 41600 > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
> **数据自动下载**：首次运行某个 benchmark 时，`run.py` 会自动调用对应 `data.py` 下载训练数据，无需手动操作。
|
||||
> - GSM8K：从 HuggingFace 下载 (~5MB)
|
||||
> - HumanEval：从 HuggingFace 下载 (~164 条样本)
|
||||
> - ALFWorld：调用 `alfworld-download` 从 GitHub Releases 下载 (~2GB，含 json/pddl/tw-pddl/logic)
|
||||
|
||||
### 4. 查看结果
|
||||
|
||||
```bash
|
||||
# 实时查看运行日志
|
||||
tail -f workspace/alfworld/20260228T100000_openhands/agent.log
|
||||
|
||||
# 查看评分记录
|
||||
cat workspace/alfworld/20260228T100000_openhands/scores.json
|
||||
|
||||
# 查看全局实验汇总
|
||||
cat rdagent/scenarios/rl/autorl_bench/results.csv
|
||||
|
||||
# Web UI（Streamlit 面板）
|
||||
streamlit run rdagent/scenarios/rl/autorl_bench/core/ui.py --server.port 8511
|
||||
```
|
||||
|
||||
### 命令行参数
|
||||
|
||||
| 参数 | 说明 | 示例 |
|
||||
|------|------|------|
|
||||
| `--agent` | Agent 类型 | `example_agent`、`rdagent`、`openhands` |
|
||||
| `--task` | Benchmark 任务名（内置或 `smith-*`） | `gsm8k`、`alfworld`、`smith-bbh` |
|
||||
| `--model` | HuggingFace 模型 repo_id，首次自动下载 | `Qwen/Qwen2.5-1.5B` |
|
||||
| `--timeout` | Agent 最大运行时长（秒） | `41600`（~11.5h） |
|
||||
| `--port` | Grading Server 端口（默认 5000） | `5000` |
|
||||
|
||||
---
|
||||
|
||||
## 核心流程
|
||||
|
||||
```
|
||||
run.py 启动
|
||||
│
|
||||
├─ 1. 准备资源：下载模型（HuggingFace）+ 下载训练数据（各 benchmark 的 data.py）
|
||||
├─ 2. 构建 workspace：创建隔离目录、软链接模型和数据
|
||||
├─ 3. 挂载文件：description.md + instructions.md + benchmark 特有文件
|
||||
├─ 4. 启动 Grading Server（Flask 评测服务）
|
||||
├─ 5. 评测 baseline：用原始模型跑一次基准分（有缓存）
|
||||
├─ 6. 运行 Agent：Agent 在 workspace 内训练 + 多次提交评测
|
||||
├─ 7. 收集结果：从 Grading Server 获取所有提交记录
|
||||
└─ 8. 保存结果：追加到 results.json，更新全局 best
|
||||
```
|
||||
|
||||
### 资源存储
|
||||
|
||||
模型和数据下载后统一存储在 `git_ignore_folder/rl_files/`（可通过 `AUTORL_FILE_PATH` 覆盖）：
|
||||
|
||||
```
|
||||
git_ignore_folder/rl_files/
|
||||
├── models/Qwen/Qwen2.5-1.5B/ # 模型权重（snapshot_download）
|
||||
├── datasets/
|
||||
│ ├── gsm8k/train.jsonl # 训练数据（agent 可见）
|
||||
│ └── alfworld/train → ... # 训练游戏数据（agent 可见，评估数据不在这）
|
||||
└── baseline_workspace/ # baseline 分数缓存
|
||||
└── gsm8k_Qwen_Qwen2.5-1.5B.json
|
||||
```
|
||||
|
||||
### Workspace（每次运行隔离）
|
||||
|
||||
每次运行创建独立的 workspace 目录（`workspace/<task>/<run_id>/`），通过软链接挂载资源：
|
||||
|
||||
```
|
||||
workspace/gsm8k/
|
||||
├── 20260211T143000_openhands/ # 一次独立实验（agent 在时限内的完整生命周期）
|
||||
│ ├── code/ # Agent 代码区（所有自行编写的代码）
|
||||
│ │ ├── train.py # 训练脚本
|
||||
│ │ └── ... # 分析、处理等其他脚本
|
||||
│ ├── output/ # 模型输出（$OUTPUT_DIR）
|
||||
│ │ ├── v1/ # 第一版模型
|
||||
│ │ └── v2/ # 第二版模型（迭代优化）
|
||||
│ ├── models/Qwen/Qwen2.5-1.5B → # 软链接 → rl_files/models/...（只读）
|
||||
│ ├── data → # 软链接 → rl_files/datasets/gsm8k/（只读）
|
||||
│ ├── description.md → # 软链接 → benchmarks/gsm8k/description.md
|
||||
│ ├── instructions.md → # 软链接 → core/instructions.md
|
||||
│ ├── scores.json # 本次实验内所有提交的评分记录
|
||||
│ └── grading_server.log # Grading Server 日志
|
||||
└── 20260211T160000_rdagent/ # 另一次独立实验
|
||||
└── ...
|
||||
```
|
||||
|
||||
> **评测原则**：每次实验（一次 `run.py` 调用）是一个独立的评测单元。
|
||||
> Agent 在 `--timeout` 时限内可以多次训练、多次提交，最终取**本次实验内**的最高分。
|
||||
> 不同实验之间完全隔离，不存在跨实验的"全局最优"。
|
||||
|
||||
### results.csv（实验日志）
|
||||
|
||||
`autorl_bench/results.csv` 是纯日志记录，用于论文实验汇总，**不参与评测逻辑**：
|
||||
|
||||
```csv
|
||||
run_id,timestamp,task,agent,base_model,baseline,best_score,improvement,submissions,duration_s,success,workspace
|
||||
20260211T143000,2026-02-11 14:30:00,gsm8k,openhands,Qwen/Qwen2.5-1.5B,21.61,22.37,0.76,3,3600,True,workspace/gsm8k/...
|
||||
20260211T160000,2026-02-11 16:00:00,gsm8k,rdagent,Qwen/Qwen2.5-1.5B,21.61,23.12,1.51,7,3600,True,workspace/gsm8k/...
|
||||
```
|
||||
|
||||
每行记录一次独立实验的结果，方便对比不同 agent 在相同条件下的表现。
|
||||
|
||||
---
|
||||
|
||||
## Agent 环境变量
|
||||
|
||||
Agent 启动时（`start.sh`）可用的环境变量：
|
||||
|
||||
| 变量 | 说明 | 示例 |
|
||||
|------|------|------|
|
||||
| `TASK` | 任务名 | `gsm8k` |
|
||||
| `BASE_MODEL` | 模型名 | `Qwen/Qwen2.5-1.5B` |
|
||||
| `WORKSPACE` | 工作根目录 | `workspace/gsm8k/20260211T143000` |
|
||||
| `MODEL_PATH` | 模型路径（只读） | `$WORKSPACE/models/Qwen/Qwen2.5-1.5B` |
|
||||
| `DATA_PATH` | 数据路径（只读） | `$WORKSPACE/data` |
|
||||
| `OUTPUT_DIR` | 输出目录 | `$WORKSPACE/output` |
|
||||
| `GRADING_SERVER_URL` | 评测服务地址 | `http://localhost:5000` |
|
||||
|
||||
### Grading Server API
|
||||
|
||||
| 端点 | 方法 | 说明 |
|
||||
|------|------|------|
|
||||
| `/submit` | POST | `{"model_path": "..."}` → 返回 score + best + improvement |
|
||||
| `/set_baseline` | POST | `{"score": 21.91}` → 设置 baseline |
|
||||
| `/health` | GET | 健康检查 |
|
||||
|
||||
`/submit` 响应：
|
||||
|
||||
```json
|
||||
{
|
||||
"submission_id": 3,
|
||||
"score": 65.0,
|
||||
"baseline_score": 45.0,
|
||||
"improvement": 20.0,
|
||||
"best": {"submission_id": 2, "score": 68.0},
|
||||
"total_submissions": 3
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 代码结构
|
||||
|
||||
```
|
||||
autorl_bench/
|
||||
├── run.py # 入口脚本
|
||||
├── conf.py # 路径配置
|
||||
│
|
||||
├── core/ # 【主干代码】
|
||||
│ ├── evaluator.py # BaseEvaluator 基类
|
||||
│ ├── opencompass.py # OpenCompassEvaluator（通用评测器）
|
||||
│ ├── server.py # Grading Server（Flask）
|
||||
│ ├── utils.py # 工具函数（下载、软链接、baseline）
|
||||
│ └── instructions.md # Agent 通用指导说明
|
||||
│
|
||||
├── benchmarks/ # 【Benchmark 扩展】
|
||||
│ ├── __init__.py # 注册表 BENCHMARKS（含 smith 自动发现）
|
||||
│ ├── smith/ # rl-smith 自动发现适配
|
||||
│ │ ├── __init__.py # discover_smith_benchmarks()
|
||||
│ │ └── per_sample_eval.py# 逐条评测器
|
||||
│ ├── gsm8k/
|
||||
│ │ ├── data.py # 数据下载（train split）
|
||||
│ │ └── description.md
|
||||
│ └── alfworld/
|
||||
│ ├── data.py # 数据下载（训练游戏数据）
|
||||
│ ├── eval.py # 自定义评测器
|
||||
│ ├── requirements.txt # 额外依赖（alfworld, textworld）
|
||||
│ ├── description.md
|
||||
│ └── react_prompts.json
|
||||
│
|
||||
├── agents/ # 【Agent 扩展】
|
||||
│ ├── registry.py # 注册表（读 config.yaml）
|
||||
│ ├── example_agent/ # 简单 GRPO 训练
|
||||
│ ├── openhands/ # OpenHands SDK
|
||||
│ └── rdagent/ # RD-Agent
|
||||
│
|
||||
└── workspace/ # [运行时] 工作区 + 结果
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 扩展指南
|
||||
|
||||
### 添加新 Benchmark
|
||||
|
||||
#### 方式一：通过 rl-smith 自动生成（推荐）
|
||||
|
||||
将 benchmark 定义放在 `rl-smith/benchmarks/<name>/` 下，RD-Agent 启动时自动发现并注册为 `smith-<name>` 任务。
|
||||
|
||||
**自动生成**（给一个 GitHub URL，AI agent 自动生成全部文件）：
|
||||
|
||||
```bash
|
||||
cd /path/to/rl-smith
|
||||
python generate_benchmark.py https://github.com/suzgunmirac/BIG-Bench-Hard --name my_bbh
|
||||
|
||||
# 生成后直接可用
|
||||
cd /path/to/RD-Agent
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run --task smith-my_bbh --agent openhands --model Qwen/Qwen2.5-1.5B
|
||||
```
|
||||
|
||||
**手动创建** `rl-smith/benchmarks/<name>/` 目录，需要：
|
||||
|
||||
| 文件 | 必须 | 说明 |
|
||||
|------|------|------|
|
||||
| `config.yaml` | 是 | `name`, `eval_mode`(`per_sample`/`opencompass`), `expose_files` |
|
||||
| `eval.py` | 是 | 导出 `evaluate(question, model_answer, reference_answer, **kwargs) -> float` |
|
||||
| `data/train.jsonl` | 是 | 每行 `{"question": "...", "answer": "..."}` |
|
||||
| `download_data.py` | 否 | 数据下载脚本（幂等） |
|
||||
| `description.md` | 否 | 任务说明（挂载到 workspace） |
|
||||
|
||||
发现机制：`discover_smith_benchmarks()` 扫描 `$SMITH_BENCH_DIR/*/config.yaml`（默认 `../rl-smith/benchmarks/`），按 `eval_mode` 选择评测器（`per_sample` → `PerSampleEvaluator`，`opencompass` → `OpenCompassEvaluator`）。
|
||||
|
||||
详见 [rl-smith README](../../../../../../../rl-smith/README.md)。
|
||||
|
||||
#### 方式二：在 RD-Agent 内手动注册
|
||||
|
||||
新建 `benchmarks/new_task/` 目录，需要 3 个文件：
|
||||
|
||||
**1. `data.py` — 数据下载（只给 agent 训练数据，评估数据自己管）**
|
||||
|
||||
```python
|
||||
# benchmarks/new_task/data.py
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""下载训练数据到 target_dir，agent 只能看到这里的内容"""
|
||||
# target_dir 会被软链接到 workspace/data
|
||||
...
|
||||
```
|
||||
|
||||
**2. `description.md` — 任务描述（agent 可见）**
|
||||
|
||||
**3. 注册到 `benchmarks/__init__.py`**
|
||||
|
||||
```python
|
||||
BENCHMARKS["new_task"] = BenchmarkConfig(
|
||||
id="new_task",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.core.opencompass.OpenCompassEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.new_task.data",
|
||||
description="新任务描述",
|
||||
eval_config={"dataset": "opencompass.configs.datasets.xxx"},
|
||||
)
|
||||
```
|
||||
|
||||
如果需要自定义评测逻辑（不用 OpenCompass），再加一个 `eval.py`：
|
||||
|
||||
```python
|
||||
# benchmarks/new_task/eval.py
|
||||
from rdagent.scenarios.rl.autorl_bench.core import BaseEvaluator
|
||||
|
||||
class NewTaskEvaluator(BaseEvaluator):
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
|
||||
def run_eval(self, model_path: str, workspace_path: str, **kwargs) -> dict:
|
||||
return {"score": 85.0, "accuracy_summary": {...}}
|
||||
```
|
||||
|
||||
### 添加新 Agent
|
||||
|
||||
```yaml
|
||||
# agents/my_agent/config.yaml
|
||||
name: "My Agent"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
MY_PARAM: "value"
|
||||
```
|
||||
|
||||
```bash
|
||||
# agents/my_agent/start.sh
|
||||
#!/bin/bash
|
||||
# 在 code/ 下编写训练脚本，模型输出到 output/
|
||||
python $WORKSPACE/code/train.py --model $MODEL_PATH --data $DATA_PATH --output $OUTPUT_DIR/v1
|
||||
curl -X POST $GRADING_SERVER_URL/submit \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model_path": "'$OUTPUT_DIR'/v1"}'
|
||||
```
|
||||
|
||||
Agent 通过 `config.yaml` 自动注册，无需修改代码。
|
||||
@@ -0,0 +1,5 @@
|
||||
"""
|
||||
AutoRL-Bench: Benchmark for evaluating RL Post-training Agents
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,3 @@
|
||||
from .registry import get_agent, list_agents
|
||||
|
||||
__all__ = ["get_agent", "list_agents"]
|
||||
@@ -0,0 +1,30 @@
|
||||
# AutoRL-Bench Agent Guidelines
|
||||
|
||||
## Summary Maintenance (MANDATORY)
|
||||
|
||||
You MUST maintain a file called `summary.md` in the workspace root. Update it **after every training attempt and every submission**, not just at the end.
|
||||
|
||||
### Format
|
||||
|
||||
```markdown
|
||||
# 运行总结
|
||||
|
||||
## Attempt N (YYYY-MM-DD HH:MM)
|
||||
- **状态**: ✅ 成功 / ❌ 失败
|
||||
- **Score**: X.XX | Improvement: +Y.YY | Best: Z.ZZ
|
||||
- **训练类型**: SFT / GRPO / PPO / DPO / ...
|
||||
- **超参数**: lr=X, epochs=Y, batch_size=Z, ...
|
||||
- **做了什么**: 简述本次尝试的策略和具体操作
|
||||
- **为什么**: 为什么选择这个方法/这些超参数
|
||||
- **问题/进步**: 遇到了什么问题，相比上次有什么改进
|
||||
- **关键代码**: 关键改动的代码片段（如有）
|
||||
- **下一步建议**: 基于本次结果，下一步打算怎么做
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Append only** — never overwrite previous attempts
|
||||
2. Analyze `code/train.py` source to extract training type and hyperparameters
|
||||
3. If training fails, extract root cause from error output
|
||||
4. "做了什么" and "为什么" are the most important fields — be thorough
|
||||
5. Update summary.md IMMEDIATELY after each submission result comes back
|
||||
6. Include the grading server response (score, improvement, best) verbatim
|
||||
@@ -0,0 +1,6 @@
|
||||
name: "Claude CLI Agent"
|
||||
description: "One-shot agent：给完整 prompt，Claude Code CLI 自主完成代码编写、训练、评测提交"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
CLAUDE_MODEL: "claude-sonnet-4-6"
|
||||
CLAUDE_TIMEOUT: "36000"
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
#!/bin/bash
|
||||
# Claude Code CLI Agent wrapper for AutoRL-Bench
|
||||
|
||||
CLAUDE="${CLAUDE_BIN:-claude}"
|
||||
CLAUDE_MODEL="${CLAUDE_MODEL:-claude-sonnet-4-6}"
|
||||
CLAUDE_TIMEOUT="${CLAUDE_TIMEOUT:-36000}"
|
||||
|
||||
echo "=== Claude CLI Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
echo "Grading Server: $GRADING_SERVER_URL"
|
||||
echo "Claude Model: $CLAUDE_MODEL"
|
||||
|
||||
if [ -z "$ANTHROPIC_API_KEY" ] && [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
|
||||
echo "ERROR: No ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN set."
|
||||
echo "Set one of:"
|
||||
echo " export ANTHROPIC_API_KEY=sk-ant-..."
|
||||
echo " export CLAUDE_CODE_OAUTH_TOKEN=..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
unset GEMINI_API_KEY
|
||||
unset CODEX_API_KEY
|
||||
|
||||
export BASH_MAX_TIMEOUT_MS="36000000"
|
||||
START_EPOCH=$(date +%s)
|
||||
|
||||
# Copy AGENTS.md into workspace
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
if [ -f "$SCRIPT_DIR/AGENTS.md" ]; then
|
||||
cp "$SCRIPT_DIR/AGENTS.md" "$WORKSPACE/AGENTS.md"
|
||||
echo "AGENTS.md copied to workspace"
|
||||
fi
|
||||
|
||||
# Generate timer.sh
|
||||
cat > "$WORKSPACE/timer.sh" << TIMER
|
||||
#!/bin/bash
|
||||
DEADLINE=$((START_EPOCH + CLAUDE_TIMEOUT))
|
||||
NOW=\$(date +%s)
|
||||
REMAINING=\$((DEADLINE - NOW))
|
||||
if [ \$REMAINING -le 0 ]; then
|
||||
echo "Timer expired!"
|
||||
else
|
||||
HOURS=\$((REMAINING / 3600))
|
||||
MINUTES=\$(((REMAINING % 3600) / 60))
|
||||
printf "Remaining: %d:%02d\n" \$HOURS \$MINUTES
|
||||
fi
|
||||
TIMER
|
||||
chmod +x "$WORKSPACE/timer.sh"
|
||||
|
||||
# Build prompt
|
||||
INSTRUCTIONS=$(cat "$WORKSPACE/instructions.md" 2>/dev/null || echo "")
|
||||
DESCRIPTION=$(cat "$WORKSPACE/description.md" 2>/dev/null || echo "")
|
||||
WORKSPACE_LS=$(ls -la "$WORKSPACE" 2>/dev/null)
|
||||
DATA_SAMPLE=$(head -5 "$WORKSPACE/data/"*.jsonl 2>/dev/null || head -5 "$WORKSPACE/data/"*.json 2>/dev/null || echo "No data files found")
|
||||
|
||||
PROMPT="You are an AI researcher doing RL post-training. Complete the entire task autonomously.
|
||||
|
||||
## Task: ${TASK}
|
||||
## Base Model: ${BASE_MODEL}
|
||||
## Model Path: ${MODEL_PATH}
|
||||
## Output Dir: ${OUTPUT_DIR}
|
||||
## Grading Server: ${GRADING_SERVER_URL}
|
||||
|
||||
## Task Description
|
||||
${DESCRIPTION}
|
||||
|
||||
## Instructions
|
||||
${INSTRUCTIONS}
|
||||
|
||||
## Workspace Contents
|
||||
\`\`\`
|
||||
${WORKSPACE_LS}
|
||||
\`\`\`
|
||||
|
||||
## Data Sample (first 5 lines)
|
||||
\`\`\`
|
||||
${DATA_SAMPLE}
|
||||
\`\`\`
|
||||
|
||||
## Your Mission
|
||||
1. Read all files in the workspace to understand the task
|
||||
2. Implement your training approach (method, code structure, filenames are all up to you)
|
||||
3. Run training and save the trained model to ${OUTPUT_DIR}/ (e.g. output/v1)
|
||||
4. IMPORTANT: If you use LoRA/PEFT, you MUST merge before saving:
|
||||
model = model.merge_and_unload()
|
||||
model.save_pretrained(output_path)
|
||||
tokenizer.save_pretrained(output_path)
|
||||
5. Fix tokenizer_config.json if needed (remove extra_special_tokens list format)
|
||||
6. Submit for evaluation:
|
||||
curl -X POST ${GRADING_SERVER_URL}/submit -H 'Content-Type: application/json' -d '{\\\"model_path\\\": \\\"${OUTPUT_DIR}/v1\\\"}'
|
||||
7. Based on the score, iterate: improve your approach and submit again as v2, v3, etc.
|
||||
8. Keep iterating until you achieve the best possible score or run out of time.
|
||||
|
||||
## Time Budget
|
||||
You have ${CLAUDE_TIMEOUT} seconds total. Run \`bash timer.sh\` at any time to check remaining time.
|
||||
|
||||
IMPORTANT: Work efficiently. Start with a simple approach, get a baseline score, then iterate."
|
||||
|
||||
echo "Prompt length: ${#PROMPT} chars"
|
||||
echo "Running Claude CLI..."
|
||||
|
||||
JSONL_LOG="$WORKSPACE/agent.jsonl"
|
||||
|
||||
cd "$WORKSPACE"
|
||||
timeout "${CLAUDE_TIMEOUT}" "$CLAUDE" \
|
||||
--print \
|
||||
--verbose \
|
||||
--model "${CLAUDE_MODEL}" \
|
||||
--output-format stream-json \
|
||||
--dangerously-skip-permissions \
|
||||
"$PROMPT" \
|
||||
2>&1 | tee "$JSONL_LOG"
|
||||
|
||||
EXIT_CODE=$?
|
||||
|
||||
echo ""
|
||||
echo "--- DIAGNOSTICS ---"
|
||||
echo "exit_code: $EXIT_CODE"
|
||||
END_EPOCH=$(date +%s)
|
||||
ELAPSED=$(( END_EPOCH - START_EPOCH ))
|
||||
printf "elapsed: %02d:%02d:%02d\n" $((ELAPSED/3600)) $(((ELAPSED%3600)/60)) $((ELAPSED%60))
|
||||
echo "model_files: $(ls "$OUTPUT_DIR/" 2>/dev/null | wc -l) dirs in output/"
|
||||
echo "code_files: $(ls "$WORKSPACE/code/" 2>/dev/null | wc -l) files in code/"
|
||||
echo "summary_exists: $([ -f "$WORKSPACE/summary.md" ] && echo yes || echo no)"
|
||||
echo "gpu_memory:"
|
||||
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader 2>/dev/null || echo " nvidia-smi not available"
|
||||
echo "disk_workspace: $(du -sh "$WORKSPACE" 2>/dev/null | cut -f1)"
|
||||
echo "--- END DIAGNOSTICS ---"
|
||||
|
||||
echo "Claude CLI exited with code: $EXIT_CODE"
|
||||
exit $EXIT_CODE
|
||||
@@ -0,0 +1,30 @@
|
||||
# AutoRL-Bench Agent Guidelines
|
||||
|
||||
## Summary Maintenance (MANDATORY)
|
||||
|
||||
You MUST maintain a file called `summary.md` in the workspace root. Update it **after every training attempt and every submission**, not just at the end.
|
||||
|
||||
### Format
|
||||
|
||||
```markdown
|
||||
# 运行总结
|
||||
|
||||
## Attempt N (YYYY-MM-DD HH:MM)
|
||||
- **状态**: ✅ 成功 / ❌ 失败
|
||||
- **Score**: X.XX | Improvement: +Y.YY | Best: Z.ZZ
|
||||
- **训练类型**: SFT / GRPO / PPO / DPO / ...
|
||||
- **超参数**: lr=X, epochs=Y, batch_size=Z, ...
|
||||
- **做了什么**: 简述本次尝试的策略和具体操作
|
||||
- **为什么**: 为什么选择这个方法/这些超参数
|
||||
- **问题/进步**: 遇到了什么问题，相比上次有什么改进
|
||||
- **关键代码**: 关键改动的代码片段（如有）
|
||||
- **下一步建议**: 基于本次结果，下一步打算怎么做
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Append only** — never overwrite previous attempts
|
||||
2. Analyze `code/train.py` source to extract training type and hyperparameters
|
||||
3. If training fails, extract root cause from error output
|
||||
4. "做了什么" and "为什么" are the most important fields — be thorough
|
||||
5. Update summary.md IMMEDIATELY after each submission result comes back
|
||||
6. Include the grading server response (score, improvement, best) verbatim
|
||||
@@ -0,0 +1,7 @@
|
||||
name: "Codex CLI Agent"
|
||||
description: "One-shot agent：给完整 prompt，Codex CLI 自主完成代码编写、训练、评测提交"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
CODEX_MODEL: "gpt-5.2" # 默认 LLM（可被环境变量覆盖）
|
||||
CODEX_PROVIDER: "litellm" # 默认走 litellm proxy（可被环境变量覆盖）
|
||||
CODEX_TIMEOUT: "36000" # Codex CLI 总超时（秒）= 10小时
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
#!/bin/bash
|
||||
# Codex CLI Agent wrapper for AutoRL-Bench
|
||||
|
||||
CODEX="${CODEX_BIN:-codex}"
|
||||
|
||||
echo "=== Codex CLI Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
echo "Grading Server: $GRADING_SERVER_URL"
|
||||
|
||||
# Provider setup: litellm (default, 3x TPM via load balancing) or trapi (direct)
|
||||
CODEX_PROVIDER="${CODEX_PROVIDER:-litellm}"
|
||||
CODEX_MODEL="${CODEX_MODEL:-gpt-5.2}"
|
||||
|
||||
if [ "$CODEX_PROVIDER" = "litellm" ]; then
|
||||
export LITELLM_API_KEY="${LITELLM_API_KEY:-sk-1234}"
|
||||
echo "Provider: litellm (load-balanced across TRAPI regions)"
|
||||
echo "Model: $CODEX_MODEL"
|
||||
else
|
||||
export TRAPI_API_KEY=$(az account get-access-token --resource "api://trapi" --query accessToken --output tsv 2>/dev/null)
|
||||
if [ -z "$TRAPI_API_KEY" ]; then
|
||||
echo "ERROR: Failed to get TRAPI token. Run 'az login' first."
|
||||
exit 1
|
||||
fi
|
||||
echo "Provider: trapi (direct)"
|
||||
echo "TRAPI token: ${#TRAPI_API_KEY} chars"
|
||||
fi
|
||||
CODEX_TIMEOUT="${CODEX_TIMEOUT:-36000}"
|
||||
START_EPOCH=$(date +%s)
|
||||
|
||||
# Copy AGENTS.md into workspace (Codex CLI auto-reads it)
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
if [ -f "$SCRIPT_DIR/AGENTS.md" ]; then
|
||||
cp "$SCRIPT_DIR/AGENTS.md" "$WORKSPACE/AGENTS.md"
|
||||
echo "AGENTS.md copied to workspace"
|
||||
fi
|
||||
|
||||
# Generate timer.sh in workspace so agent can query remaining time
|
||||
cat > "$WORKSPACE/timer.sh" << TIMER
|
||||
#!/bin/bash
|
||||
DEADLINE=$((START_EPOCH + CODEX_TIMEOUT))
|
||||
NOW=\$(date +%s)
|
||||
REMAINING=\$((DEADLINE - NOW))
|
||||
if [ \$REMAINING -le 0 ]; then
|
||||
echo "Timer expired!"
|
||||
else
|
||||
HOURS=\$((REMAINING / 3600))
|
||||
MINUTES=\$(((REMAINING % 3600) / 60))
|
||||
printf "Remaining: %d:%02d\n" \$HOURS \$MINUTES
|
||||
fi
|
||||
TIMER
|
||||
chmod +x "$WORKSPACE/timer.sh"
|
||||
|
||||
# Build prompt from workspace files
|
||||
INSTRUCTIONS=$(cat "$WORKSPACE/instructions.md" 2>/dev/null || echo "")
|
||||
DESCRIPTION=$(cat "$WORKSPACE/description.md" 2>/dev/null || echo "")
|
||||
WORKSPACE_LS=$(ls -la "$WORKSPACE" 2>/dev/null)
|
||||
DATA_SAMPLE=$(head -5 "$WORKSPACE/data/"*.jsonl 2>/dev/null || head -5 "$WORKSPACE/data/"*.json 2>/dev/null || echo "No data files found")
|
||||
|
||||
PROMPT="You are an AI researcher doing RL post-training. Complete the entire task autonomously.
|
||||
|
||||
## Task: ${TASK}
|
||||
## Base Model: ${BASE_MODEL}
|
||||
## Model Path: ${MODEL_PATH}
|
||||
## Output Dir: ${OUTPUT_DIR}
|
||||
## Grading Server: ${GRADING_SERVER_URL}
|
||||
|
||||
## Task Description
|
||||
${DESCRIPTION}
|
||||
|
||||
## Instructions
|
||||
${INSTRUCTIONS}
|
||||
|
||||
## Workspace Contents
|
||||
\`\`\`
|
||||
${WORKSPACE_LS}
|
||||
\`\`\`
|
||||
|
||||
## Data Sample (first 5 lines)
|
||||
\`\`\`
|
||||
${DATA_SAMPLE}
|
||||
\`\`\`
|
||||
|
||||
## Your Mission
|
||||
1. Read all files in the workspace to understand the task
|
||||
2. Implement your training approach (method, code structure, filenames are all up to you)
|
||||
3. Run training and save the trained model to ${OUTPUT_DIR}/ (e.g. output/v1)
|
||||
4. IMPORTANT: If you use LoRA/PEFT, you MUST merge before saving:
|
||||
model = model.merge_and_unload()
|
||||
model.save_pretrained(output_path)
|
||||
tokenizer.save_pretrained(output_path)
|
||||
5. Fix tokenizer_config.json if needed (remove extra_special_tokens list format)
|
||||
6. Submit for evaluation:
|
||||
curl -X POST ${GRADING_SERVER_URL}/submit -H 'Content-Type: application/json' -d '{\\\"model_path\\\": \\\"${OUTPUT_DIR}/v1\\\"}'
|
||||
7. Based on the score, iterate: improve your approach and submit again as v2, v3, etc.
|
||||
8. Keep iterating until you achieve the best possible score or run out of time.
|
||||
|
||||
## Time Budget
|
||||
You have ${CODEX_TIMEOUT} seconds total. Run \`bash timer.sh\` at any time to check remaining time.
|
||||
|
||||
IMPORTANT: Work efficiently. Start with a simple approach, get a baseline score, then iterate."
|
||||
|
||||
echo "Prompt length: ${#PROMPT} chars"
|
||||
echo "Running Codex CLI..."
|
||||
|
||||
# JSON trace goes to agent.jsonl AND stdout (captured as agent.log by run.py)
|
||||
JSONL_LOG="$WORKSPACE/agent.jsonl"
|
||||
|
||||
timeout "${CODEX_TIMEOUT}" "$CODEX" --search exec \
|
||||
--json \
|
||||
-m "${CODEX_MODEL}" \
|
||||
-c "model_provider=\"${CODEX_PROVIDER}\"" \
|
||||
-c "model_reasoning_summary=\"detailed\"" \
|
||||
--dangerously-bypass-approvals-and-sandbox \
|
||||
--skip-git-repo-check \
|
||||
-C "$WORKSPACE" \
|
||||
"$PROMPT" \
|
||||
2>&1 | tee "$JSONL_LOG"
|
||||
|
||||
EXIT_CODE=$?
|
||||
|
||||
# --- Diagnostics ---
|
||||
echo ""
|
||||
echo "--- DIAGNOSTICS ---"
|
||||
echo "exit_code: $EXIT_CODE"
|
||||
END_EPOCH=$(date +%s)
|
||||
ELAPSED=$(( END_EPOCH - START_EPOCH ))
|
||||
printf "elapsed: %02d:%02d:%02d\n" $((ELAPSED/3600)) $(((ELAPSED%3600)/60)) $((ELAPSED%60))
|
||||
echo "model_files: $(ls "$OUTPUT_DIR/" 2>/dev/null | wc -l) dirs in output/"
|
||||
echo "code_files: $(ls "$WORKSPACE/code/" 2>/dev/null | wc -l) files in code/"
|
||||
echo "summary_exists: $([ -f "$WORKSPACE/summary.md" ] && echo yes || echo no)"
|
||||
echo "gpu_memory:"
|
||||
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader 2>/dev/null || echo " nvidia-smi not available"
|
||||
echo "disk_workspace: $(du -sh "$WORKSPACE" 2>/dev/null | cut -f1)"
|
||||
echo "--- END DIAGNOSTICS ---"
|
||||
|
||||
# Parse JSONL trace into human-readable format
|
||||
TRACE_PARSER="$(cd "$(dirname "$0")" && pwd)/human_readable_trace.py"
|
||||
if [ -f "$TRACE_PARSER" ] && [ -f "$JSONL_LOG" ]; then
|
||||
python "$TRACE_PARSER" "$JSONL_LOG" -o "$WORKSPACE/agent_trace.txt" 2>/dev/null && \
|
||||
echo "trace_parsed: yes ($(wc -l < "$WORKSPACE/agent_trace.txt") lines)" || \
|
||||
echo "trace_parsed: no (parser failed)"
|
||||
fi
|
||||
|
||||
echo "Codex CLI exited with code: $EXIT_CODE"
|
||||
exit $EXIT_CODE
|
||||
@@ -0,0 +1,6 @@
|
||||
name: "Example Agent"
|
||||
description: "GRPO 训练 + 评测"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
TRAIN_RATIO: "0.1"
|
||||
NUM_EPOCHS: "1"
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
echo "=== Example Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
python3 "$(dirname "$0")/train.py"
|
||||
@@ -0,0 +1,148 @@
|
||||
"""
|
||||
GRPO Training Loop
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
|
||||
import requests
|
||||
from datasets import Dataset
|
||||
from transformers import AutoTokenizer
|
||||
from trl import GRPOConfig, GRPOTrainer
|
||||
|
||||
|
||||
def extract_answer(text):
|
||||
if not isinstance(text, str):
|
||||
text = str(text)
|
||||
match = re.search(r"####\s*([-+]?\d[\d,]*\.?\d*)", text)
|
||||
if match:
|
||||
try:
|
||||
return float(match.group(1).replace(",", ""))
|
||||
except:
|
||||
pass
|
||||
numbers = re.findall(r"[-+]?\d[\d,]*\.?\d*", text)
|
||||
if numbers:
|
||||
try:
|
||||
return float(numbers[-1].replace(",", ""))
|
||||
except:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def load_data(file_path, ratio=1.0):
|
||||
records = []
|
||||
with open(file_path, "r") as f:
|
||||
for line in f:
|
||||
item = json.loads(line)
|
||||
prompt = f"Solve this math problem step by step. Put your final answer after ####.\n\nQuestion: {item['question']}\n\nSolution:"
|
||||
records.append({"prompt": prompt, "question": item["question"], "answer": item["answer"]})
|
||||
if ratio < 1.0:
|
||||
n = max(10, int(len(records) * ratio))
|
||||
records = records[:n]
|
||||
return records
|
||||
|
||||
|
||||
def gsm8k_reward_func(completions, answer, **kwargs):
|
||||
rewards = []
|
||||
for completion, gold_answer in zip(completions, answer):
|
||||
pred = extract_answer(completion)
|
||||
gold = extract_answer(gold_answer)
|
||||
if pred is not None and gold is not None and abs(pred - gold) < 1e-6:
|
||||
rewards.append(1.0)
|
||||
else:
|
||||
rewards.append(-1.0)
|
||||
return rewards
|
||||
|
||||
|
||||
def submit_for_grading(grading_url: str, model_path: str) -> dict | None:
|
||||
if not grading_url:
|
||||
return None
|
||||
try:
|
||||
resp = requests.post(f"{grading_url}/submit", json={"model_path": model_path}, timeout=600)
|
||||
if resp.status_code == 200:
|
||||
return resp.json()
|
||||
except Exception as e:
|
||||
print(f" Grading error: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
MODEL_PATH = os.environ.get("MODEL_PATH")
|
||||
DATA_PATH = os.environ.get("DATA_PATH")
|
||||
OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/tmp/autorl_output")
|
||||
GRADING_SERVER_URL = os.environ.get("GRADING_SERVER_URL", "")
|
||||
TRAIN_RATIO = float(os.environ.get("TRAIN_RATIO", "0.05"))
|
||||
NUM_EPOCHS = int(os.environ.get("NUM_EPOCHS", "3"))
|
||||
|
||||
if not MODEL_PATH or not DATA_PATH:
|
||||
raise ValueError("MODEL_PATH and DATA_PATH required")
|
||||
|
||||
print(f"Model: {MODEL_PATH}")
|
||||
print(f"Data: {DATA_PATH}")
|
||||
print(f"Output: {OUTPUT_DIR}")
|
||||
|
||||
train_file = f"{DATA_PATH}/train.jsonl"
|
||||
train_data = load_data(train_file, TRAIN_RATIO)
|
||||
print(f"Train samples: {len(train_data)}")
|
||||
dataset = Dataset.from_list([{"prompt": d["prompt"], "answer": d["answer"]} for d in train_data])
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
|
||||
if tokenizer.pad_token is None:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
start_time = time.time()
|
||||
|
||||
# 第一个 epoch 使用原始模型，后续 epoch 使用上一个 checkpoint
|
||||
current_model_path = MODEL_PATH
|
||||
|
||||
for epoch in range(NUM_EPOCHS):
|
||||
print(f"\n=== Epoch {epoch + 1}/{NUM_EPOCHS} ===")
|
||||
|
||||
config = GRPOConfig(
|
||||
output_dir=OUTPUT_DIR,
|
||||
max_steps=20, # 固定步数，避免小数据集报错
|
||||
per_device_train_batch_size=2, # 小 batch 避免 OOM
|
||||
gradient_accumulation_steps=4, # 梯度累积
|
||||
learning_rate=1e-5,
|
||||
max_completion_length=256,
|
||||
num_generations=4,
|
||||
logging_steps=5,
|
||||
save_strategy="no",
|
||||
report_to="none",
|
||||
bf16=True,
|
||||
)
|
||||
|
||||
# 直接传模型路径，让 GRPOTrainer 自己管理模型加载
|
||||
# 避免 vLLM colocate 模式下模型被加载两次导致 OOM
|
||||
trainer = GRPOTrainer(
|
||||
model=current_model_path,
|
||||
reward_funcs=gsm8k_reward_func,
|
||||
args=config,
|
||||
train_dataset=dataset,
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
|
||||
checkpoint_dir = f"{OUTPUT_DIR}/checkpoint-epoch{epoch + 1}"
|
||||
trainer.save_model(checkpoint_dir)
|
||||
tokenizer.save_pretrained(checkpoint_dir)
|
||||
|
||||
# 下一个 epoch 从这个 checkpoint 继续训练
|
||||
current_model_path = checkpoint_dir
|
||||
|
||||
result = submit_for_grading(GRADING_SERVER_URL, checkpoint_dir)
|
||||
if result:
|
||||
print(f" Score: {result.get('score')}")
|
||||
|
||||
trainer.save_model(OUTPUT_DIR)
|
||||
tokenizer.save_pretrained(OUTPUT_DIR)
|
||||
submit_for_grading(GRADING_SERVER_URL, OUTPUT_DIR)
|
||||
print(f"\nDone! Total: {(time.time() - start_time) / 60:.1f} min")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
# AutoRL-Bench Agent Guidelines
|
||||
|
||||
## Summary Maintenance (MANDATORY)
|
||||
|
||||
You MUST maintain a file called `summary.md` in the workspace root. Update it **after every training attempt and every submission**, not just at the end.
|
||||
|
||||
### Format
|
||||
|
||||
```markdown
|
||||
# 运行总结
|
||||
|
||||
## Attempt N (YYYY-MM-DD HH:MM)
|
||||
- **状态**: ✅ 成功 / ❌ 失败
|
||||
- **Score**: X.XX | Improvement: +Y.YY | Best: Z.ZZ
|
||||
- **训练类型**: SFT / GRPO / PPO / DPO / ...
|
||||
- **超参数**: lr=X, epochs=Y, batch_size=Z, ...
|
||||
- **做了什么**: 简述本次尝试的策略和具体操作
|
||||
- **为什么**: 为什么选择这个方法/这些超参数
|
||||
- **问题/进步**: 遇到了什么问题，相比上次有什么改进
|
||||
- **关键代码**: 关键改动的代码片段（如有）
|
||||
- **下一步建议**: 基于本次结果，下一步打算怎么做
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Append only** — never overwrite previous attempts
|
||||
2. Analyze `code/train.py` source to extract training type and hyperparameters
|
||||
3. If training fails, extract root cause from error output
|
||||
4. "做了什么" and "为什么" are the most important fields — be thorough
|
||||
5. Update summary.md IMMEDIATELY after each submission result comes back
|
||||
6. Include the grading server response (score, improvement, best) verbatim
|
||||
@@ -0,0 +1,6 @@
|
||||
name: "Gemini CLI Agent"
|
||||
description: "One-shot agent：给完整 prompt，Gemini CLI 自主完成代码编写、训练、评测提交"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
GEMINI_MODEL: "gemini-2.5-pro"
|
||||
GEMINI_TIMEOUT: "36000"
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
#!/bin/bash
|
||||
# Gemini CLI Agent wrapper for AutoRL-Bench
|
||||
|
||||
GEMINI="${GEMINI_BIN:-gemini}"
|
||||
GEMINI_MODEL="${GEMINI_MODEL:-gemini-2.5-pro}"
|
||||
GEMINI_TIMEOUT="${GEMINI_TIMEOUT:-36000}"
|
||||
|
||||
echo "=== Gemini CLI Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
echo "Grading Server: $GRADING_SERVER_URL"
|
||||
echo "Gemini Model: $GEMINI_MODEL"
|
||||
|
||||
if [ -z "$GEMINI_API_KEY" ] && [ -z "$GOOGLE_API_KEY" ]; then
|
||||
echo "WARNING: No GEMINI_API_KEY or GOOGLE_API_KEY set. Gemini CLI will use Google account login."
|
||||
fi
|
||||
|
||||
export GEMINI_SANDBOX="false"
|
||||
START_EPOCH=$(date +%s)
|
||||
|
||||
# Copy AGENTS.md into workspace
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
if [ -f "$SCRIPT_DIR/AGENTS.md" ]; then
|
||||
cp "$SCRIPT_DIR/AGENTS.md" "$WORKSPACE/AGENTS.md"
|
||||
echo "AGENTS.md copied to workspace"
|
||||
fi
|
||||
|
||||
# Copy .gemini/settings.json for workspace config
|
||||
mkdir -p "$WORKSPACE/.gemini"
|
||||
cat > "$WORKSPACE/.gemini/settings.json" << 'SETTINGS'
|
||||
{
|
||||
"general": { "defaultApprovalMode": "auto_edit" }
|
||||
}
|
||||
SETTINGS
|
||||
|
||||
# Generate timer.sh
|
||||
cat > "$WORKSPACE/timer.sh" << TIMER
|
||||
#!/bin/bash
|
||||
DEADLINE=$((START_EPOCH + GEMINI_TIMEOUT))
|
||||
NOW=\$(date +%s)
|
||||
REMAINING=\$((DEADLINE - NOW))
|
||||
if [ \$REMAINING -le 0 ]; then
|
||||
echo "Timer expired!"
|
||||
else
|
||||
HOURS=\$((REMAINING / 3600))
|
||||
MINUTES=\$(((REMAINING % 3600) / 60))
|
||||
printf "Remaining: %d:%02d\n" \$HOURS \$MINUTES
|
||||
fi
|
||||
TIMER
|
||||
chmod +x "$WORKSPACE/timer.sh"
|
||||
|
||||
# Build prompt
|
||||
INSTRUCTIONS=$(cat "$WORKSPACE/instructions.md" 2>/dev/null || echo "")
|
||||
DESCRIPTION=$(cat "$WORKSPACE/description.md" 2>/dev/null || echo "")
|
||||
WORKSPACE_LS=$(ls -la "$WORKSPACE" 2>/dev/null)
|
||||
DATA_SAMPLE=$(head -5 "$WORKSPACE/data/"*.jsonl 2>/dev/null || head -5 "$WORKSPACE/data/"*.json 2>/dev/null || echo "No data files found")
|
||||
|
||||
PROMPT="You are an AI researcher doing RL post-training. Complete the entire task autonomously.
|
||||
|
||||
## Task: ${TASK}
|
||||
## Base Model: ${BASE_MODEL}
|
||||
## Model Path: ${MODEL_PATH}
|
||||
## Output Dir: ${OUTPUT_DIR}
|
||||
## Grading Server: ${GRADING_SERVER_URL}
|
||||
|
||||
## Task Description
|
||||
${DESCRIPTION}
|
||||
|
||||
## Instructions
|
||||
${INSTRUCTIONS}
|
||||
|
||||
## Workspace Contents
|
||||
\`\`\`
|
||||
${WORKSPACE_LS}
|
||||
\`\`\`
|
||||
|
||||
## Data Sample (first 5 lines)
|
||||
\`\`\`
|
||||
${DATA_SAMPLE}
|
||||
\`\`\`
|
||||
|
||||
## Your Mission
|
||||
1. Read all files in the workspace to understand the task
|
||||
2. Implement your training approach (method, code structure, filenames are all up to you)
|
||||
3. Run training and save the trained model to ${OUTPUT_DIR}/ (e.g. output/v1)
|
||||
4. IMPORTANT: If you use LoRA/PEFT, you MUST merge before saving:
|
||||
model = model.merge_and_unload()
|
||||
model.save_pretrained(output_path)
|
||||
tokenizer.save_pretrained(output_path)
|
||||
5. Fix tokenizer_config.json if needed (remove extra_special_tokens list format)
|
||||
6. Submit for evaluation:
|
||||
curl -X POST ${GRADING_SERVER_URL}/submit -H 'Content-Type: application/json' -d '{\\\"model_path\\\": \\\"${OUTPUT_DIR}/v1\\\"}'
|
||||
7. Based on the score, iterate: improve your approach and submit again as v2, v3, etc.
|
||||
8. Keep iterating until you achieve the best possible score or run out of time.
|
||||
|
||||
## Time Budget
|
||||
You have ${GEMINI_TIMEOUT} seconds total. Run \`bash timer.sh\` at any time to check remaining time.
|
||||
|
||||
IMPORTANT: Work efficiently. Start with a simple approach, get a baseline score, then iterate."
|
||||
|
||||
echo "Prompt length: ${#PROMPT} chars"
|
||||
echo "Running Gemini CLI..."
|
||||
|
||||
JSONL_LOG="$WORKSPACE/agent.jsonl"
|
||||
|
||||
cd "$WORKSPACE"
|
||||
timeout "${GEMINI_TIMEOUT}" "$GEMINI" \
|
||||
--yolo \
|
||||
--model "${GEMINI_MODEL}" \
|
||||
--output-format stream-json \
|
||||
-p "$PROMPT" \
|
||||
2>&1 | tee "$JSONL_LOG"
|
||||
|
||||
EXIT_CODE=$?
|
||||
|
||||
echo ""
|
||||
echo "--- DIAGNOSTICS ---"
|
||||
echo "exit_code: $EXIT_CODE"
|
||||
END_EPOCH=$(date +%s)
|
||||
ELAPSED=$(( END_EPOCH - START_EPOCH ))
|
||||
printf "elapsed: %02d:%02d:%02d\n" $((ELAPSED/3600)) $(((ELAPSED%3600)/60)) $((ELAPSED%60))
|
||||
echo "model_files: $(ls "$OUTPUT_DIR/" 2>/dev/null | wc -l) dirs in output/"
|
||||
echo "code_files: $(ls "$WORKSPACE/code/" 2>/dev/null | wc -l) files in code/"
|
||||
echo "summary_exists: $([ -f "$WORKSPACE/summary.md" ] && echo yes || echo no)"
|
||||
echo "gpu_memory:"
|
||||
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader 2>/dev/null || echo " nvidia-smi not available"
|
||||
echo "disk_workspace: $(du -sh "$WORKSPACE" 2>/dev/null | cut -f1)"
|
||||
echo "--- END DIAGNOSTICS ---"
|
||||
|
||||
echo "Gemini CLI exited with code: $EXIT_CODE"
|
||||
exit $EXIT_CODE
|
||||
@@ -0,0 +1,12 @@
|
||||
name: "OpenCode Agent"
|
||||
description: "固定阶段 pipeline：代码生成→训练→评测→反馈（基于 opencode-rl）"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
MAX_ITERATIONS: "5"
|
||||
TRAINING_TIMEOUT: "7200"
|
||||
MAX_AGENT_STEPS: "25"
|
||||
MAX_RETRIES: "20"
|
||||
STALE_TIMEOUT: "1800"
|
||||
HTTP_TIMEOUT: "600"
|
||||
EVAL_TIMEOUT: "7200"
|
||||
MAX_STEPS: "20"
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/bin/bash
|
||||
# OpenCode Agent wrapper for AutoRL-Bench
|
||||
|
||||
echo "=== OpenCode Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
echo "Grading Server: $GRADING_SERVER_URL"
|
||||
echo "Output Dir: $OUTPUT_DIR"
|
||||
|
||||
# 加载 .env 配置（启动时已在 RD-Agent 目录）
|
||||
if [ -f .env ]; then
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
echo "Loaded .env"
|
||||
fi
|
||||
|
||||
# opencode-rl 路径：默认用外部独立目录
|
||||
OPENCODE_RL_ROOT="${OPENCODE_RL_ROOT:-/data/userdata/v-tiansha/opencode-rl}"
|
||||
|
||||
# OPENCODE_MODEL 优先从 config.yaml 传入，否则用 CHAT_MODEL，默认 gpt-5
|
||||
export OPENCODE_MODEL="${OPENCODE_MODEL:-${CHAT_MODEL:-gpt-5}}"
|
||||
echo "OpenCode Model: $OPENCODE_MODEL"
|
||||
|
||||
export PYTHONUNBUFFERED=1
|
||||
|
||||
# opencode CLI 可能装在 ~/.opencode/bin，确保在 PATH 中
|
||||
export PATH="$HOME/.opencode/bin:$PATH"
|
||||
|
||||
# 把训练环境的 bin 目录加到 PATH，这样 LLM agent 的 bash 工具调用
|
||||
# (python3 -c "from trl import ...") 也能用到正确的训练依赖
|
||||
if [ -n "$TRAINING_PYTHON" ]; then
|
||||
TRAINING_BIN_DIR="$(dirname "$TRAINING_PYTHON")"
|
||||
export PATH="$TRAINING_BIN_DIR:$PATH"
|
||||
echo "Training env bin: $TRAINING_BIN_DIR (prepended to PATH)"
|
||||
fi
|
||||
|
||||
# Python 解释器：优先用 .env 中的 OPENCODE_PYTHON，否则用 python3
|
||||
PYTHON="${OPENCODE_PYTHON:-python3}"
|
||||
echo "Python: $PYTHON"
|
||||
|
||||
# 生成 opencode config（用 RD-Agent 根 .env 中的 API 配置）
|
||||
export XDG_CONFIG_HOME="${OPENCODE_RL_ROOT}/.opencode-config"
|
||||
mkdir -p "$XDG_CONFIG_HOME/opencode"
|
||||
cat > "$XDG_CONFIG_HOME/opencode/opencode.json" <<EOCFG
|
||||
{
|
||||
"\$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"openai": {
|
||||
"npm": "@ai-sdk/openai",
|
||||
"name": "Auto-configured",
|
||||
"options": {
|
||||
"baseURL": "${OPENAI_API_BASE}",
|
||||
"apiKey": "${OPENAI_API_KEY}"
|
||||
},
|
||||
"models": {
|
||||
"${OPENCODE_MODEL}": { "name": "${OPENCODE_MODEL}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOCFG
|
||||
|
||||
# 运行 opencode-rl pipeline
|
||||
cd "$OPENCODE_RL_ROOT"
|
||||
|
||||
# Use exec to REPLACE bash with python3, so signals go directly to python3
|
||||
# without an intermediate bash process. This avoids double signal delivery.
|
||||
exec "$PYTHON" main.py \
|
||||
--benchmark "$TASK" \
|
||||
--base-model "$BASE_MODEL" \
|
||||
--run-dir "$WORKSPACE" \
|
||||
--max-iterations ${MAX_ITERATIONS:-5} \
|
||||
--max-retries ${MAX_RETRIES:-20} \
|
||||
--training-timeout ${TRAINING_TIMEOUT:-7200} \
|
||||
--stale-timeout ${STALE_TIMEOUT:-1800} \
|
||||
--http-timeout ${HTTP_TIMEOUT:-600} \
|
||||
--eval-timeout ${EVAL_TIMEOUT:-7200} \
|
||||
--max-agent-steps ${MAX_AGENT_STEPS:-25}
|
||||
@@ -0,0 +1,8 @@
|
||||
name: "OpenHands Agent"
|
||||
description: "固定阶段 pipeline：每轮 代码生成→训练→评测→反馈（参考 openhands-magic）"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
MAX_ITERATIONS: "30" # Pipeline 迭代次数（每轮=写代码+训练+评测）
|
||||
TRAINING_TIMEOUT: "36000" # 每轮训练超时（秒）= 10小时
|
||||
MAX_AGENT_STEPS: "20" # 每轮代码生成 agent 最大步数
|
||||
LLM_MODEL: "gpt-5.2"
|
||||
@@ -0,0 +1,54 @@
|
||||
#!/bin/bash
|
||||
# OpenHands Agent wrapper for AutoRL-Bench
|
||||
|
||||
echo "=== OpenHands Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
echo "Grading Server: $GRADING_SERVER_URL"
|
||||
echo "Output Dir: $OUTPUT_DIR"
|
||||
|
||||
# 加载 .env 配置（启动时已在 RD-Agent 目录）
|
||||
if [ -f .env ]; then
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
echo "Loaded .env"
|
||||
fi
|
||||
|
||||
# 映射环境变量（rdagent 用 OPENAI_API_KEY，openhands 用 LLM_API_KEY）
|
||||
if [ -z "$OPENAI_API_KEY" ] && [ -n "$LLM_API_KEY" ]; then
|
||||
export OPENAI_API_KEY="$LLM_API_KEY"
|
||||
fi
|
||||
if [ -z "$LLM_API_KEY" ] && [ -n "$OPENAI_API_KEY" ]; then
|
||||
export LLM_API_KEY="$OPENAI_API_KEY"
|
||||
fi
|
||||
if [ -z "$LLM_API_KEY" ] && [ -z "$OPENAI_API_KEY" ]; then
|
||||
echo "ERROR: LLM_API_KEY or OPENAI_API_KEY required"
|
||||
exit 2
|
||||
fi
|
||||
# LLM_MODEL 优先从 config.yaml 传入，否则用 CHAT_MODEL，默认 gpt-5
|
||||
export LLM_MODEL="${LLM_MODEL:-${CHAT_MODEL:-gpt-5}}"
|
||||
export LLM_BASE_URL="${OPENAI_API_BASE}"
|
||||
echo "LLM API key length: ${#LLM_API_KEY}"
|
||||
echo "LLM Model: $LLM_MODEL"
|
||||
|
||||
# 训练环境 Python 路径（.env 中设 TRAINING_PYTHON 即可，无需 conda）
|
||||
if [ -z "$TRAINING_PYTHON" ]; then
|
||||
echo "WARNING: TRAINING_PYTHON not set in .env, trying conda fallback..."
|
||||
source "$(conda info --base 2>/dev/null || echo /root/miniconda3)/etc/profile.d/conda.sh" 2>/dev/null
|
||||
conda activate "${CONDA_ENV_TRAINING:-autorl}" 2>/dev/null
|
||||
export TRAINING_PYTHON="$(which python)"
|
||||
conda activate "${CONDA_ENV_OPENHANDS:-openhands}" 2>/dev/null
|
||||
fi
|
||||
echo "Training Python: $TRAINING_PYTHON"
|
||||
|
||||
# 运行 openhands-rl pipeline
|
||||
cd "${OPENHANDS_RL_ROOT:-$HOME/openhands-rl}"
|
||||
OPENHANDS_PYTHON="${OPENHANDS_PYTHON:-python}"
|
||||
|
||||
"$OPENHANDS_PYTHON" main.py \
|
||||
--benchmark "$TASK" \
|
||||
--base-model "$BASE_MODEL" \
|
||||
--workspace "$WORKSPACE" \
|
||||
--max-iterations ${MAX_ITERATIONS:-10} \
|
||||
--training-timeout ${TRAINING_TIMEOUT:-7200} \
|
||||
--max-agent-steps ${MAX_AGENT_STEPS:-50}
|
||||
@@ -0,0 +1,6 @@
|
||||
name: "RD-Agent"
|
||||
description: "RD-Agent RL Post-training Loop (自动假设生成 + 代码生成 + 验证迭代)"
|
||||
start: "start.sh"
|
||||
env_vars:
|
||||
STEP_N: "200"
|
||||
LOOP_N: "40"
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/bin/bash
|
||||
# RD-Agent wrapper for AutoRL-Bench
|
||||
|
||||
echo "=== RD-Agent ==="
|
||||
echo "Task: $TASK"
|
||||
echo "Model: $BASE_MODEL"
|
||||
echo "Workspace: $WORKSPACE"
|
||||
|
||||
# 加载 .env 配置（启动时已在 RD-Agent 目录）
|
||||
if [ -f .env ]; then
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
echo "Loaded .env"
|
||||
fi
|
||||
|
||||
# 设置 rdagent 数据目录（命令行会传 base_model 和 benchmark）
|
||||
export RL_FILE_PATH=$(dirname $(dirname $MODEL_PATH))
|
||||
echo "RL_FILE_PATH: $RL_FILE_PATH"
|
||||
|
||||
# 运行 rdagent（内部每次迭代会自动调用 grading server 评测）
|
||||
python -m rdagent.app.rl.loop \
|
||||
--base-model "$BASE_MODEL" \
|
||||
--benchmark "$TASK" \
|
||||
--step-n $STEP_N \
|
||||
--loop-n $LOOP_N
|
||||
@@ -0,0 +1,42 @@
|
||||
"""
|
||||
Agent Registry
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
AGENTS_DIR = Path(__file__).parent
|
||||
|
||||
|
||||
@dataclass
|
||||
class Agent:
|
||||
id: str
|
||||
name: str
|
||||
start: Path
|
||||
env_vars: dict = None
|
||||
|
||||
def __post_init__(self):
|
||||
self.env_vars = self.env_vars or {}
|
||||
|
||||
|
||||
def get_agent(agent_id: str) -> Agent:
|
||||
agent_dir = AGENTS_DIR / agent_id
|
||||
config_file = agent_dir / "config.yaml"
|
||||
|
||||
if not config_file.exists():
|
||||
raise ValueError(f"Agent not found: {agent_id}")
|
||||
|
||||
data = yaml.safe_load(config_file.read_text())
|
||||
|
||||
return Agent(
|
||||
id=agent_id,
|
||||
name=data.get("name", agent_id),
|
||||
start=agent_dir / data.get("start", "start.sh"),
|
||||
env_vars=data.get("env_vars", {}),
|
||||
)
|
||||
|
||||
|
||||
def list_agents() -> list[str]:
|
||||
return [d.name for d in AGENTS_DIR.iterdir() if d.is_dir() and (d / "config.yaml").exists()]
|
||||
@@ -0,0 +1,136 @@
|
||||
"""
|
||||
AutoRL-Bench Benchmarks Registry
|
||||
|
||||
注册表，管理所有可用的 benchmark 评测器。
|
||||
添加新 benchmark 时，在此注册。
|
||||
"""
|
||||
|
||||
import importlib
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Type
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
BENCHMARKS_DIR = Path(__file__).parent
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkConfig:
|
||||
"""Benchmark 配置
|
||||
|
||||
每个 benchmark 的数据下载/处理逻辑写在各自目录的 data.py 里，
|
||||
不在这里统一处理。这样新增 benchmark 时只需在自己目录下实现即可。
|
||||
"""
|
||||
|
||||
id: str
|
||||
evaluator_class: str # 评测器类的完整路径
|
||||
data_module: str = "" # 数据模块路径（实现 download_train_data 函数）
|
||||
description: str = ""
|
||||
eval_config: Optional[Dict[str, Any]] = field(default=None)
|
||||
expose_files: list = field(
|
||||
default_factory=list
|
||||
) # benchmark 特有的额外文件（description.md 和 instructions.md 由 run.py 统一挂载）
|
||||
bench_dir: Optional[str] = None # 自定义 benchmark 目录路径（默认 None 则用 BENCHMARKS_DIR / id）
|
||||
|
||||
|
||||
# Benchmark 注册表
|
||||
BENCHMARKS: Dict[str, BenchmarkConfig] = {
|
||||
"gsm8k": BenchmarkConfig(
|
||||
id="gsm8k",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.core.opencompass.OpenCompassEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.gsm8k.data",
|
||||
description="Grade School Math 8K - 小学数学推理",
|
||||
eval_config={
|
||||
"dataset": "opencompass.configs.datasets.gsm8k.gsm8k_gen_1d7fe4",
|
||||
},
|
||||
),
|
||||
"humaneval": BenchmarkConfig(
|
||||
id="humaneval",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.core.opencompass.OpenCompassEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.humaneval.data",
|
||||
description="HumanEval - Python 代码生成",
|
||||
eval_config={
|
||||
"dataset": "opencompass.configs.datasets.humaneval.humaneval_gen",
|
||||
"test_range": "[82:]",
|
||||
},
|
||||
),
|
||||
"alpacaeval": BenchmarkConfig(
|
||||
id="alpacaeval",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.benchmarks.alpacaeval.eval.AlpacaEvalEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.alpacaeval.data",
|
||||
description="AlpacaEval 2.0 - 指令遵循与偏好评测（LLM Judge）",
|
||||
eval_config={
|
||||
"reference_file": "alpaca_eval_gpt4_baseline.json",
|
||||
"annotators_config": "annotators_gpt52_fn",
|
||||
"max_model_len": 4096,
|
||||
"max_tokens": 512,
|
||||
},
|
||||
expose_files=["eval.py"],
|
||||
),
|
||||
"alfworld": BenchmarkConfig(
|
||||
id="alfworld",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.benchmarks.alfworld.eval.ALFWorldEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.alfworld.data",
|
||||
description="ALFWorld - 文本游戏交互环境（ReAct agent，支持 vLLM/API）",
|
||||
eval_config={
|
||||
"max_steps": 50,
|
||||
"env_num": 134, # 完整评测集（valid_unseen），之前调试时设为 1
|
||||
},
|
||||
expose_files=["eval.py", "react_prompts.json"],
|
||||
),
|
||||
"webshop": BenchmarkConfig(
|
||||
id="webshop",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.benchmarks.webshop.eval.WebShopEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.webshop.data",
|
||||
description="WebShop - 在线购物网站交互环境（ReAct agent，支持 vLLM/API）",
|
||||
eval_config={
|
||||
"max_steps": 50,
|
||||
"num_instructions": 100,
|
||||
"webshop_port": 8080,
|
||||
},
|
||||
expose_files=["eval.py"],
|
||||
),
|
||||
"deepsearchqa": BenchmarkConfig(
|
||||
id="deepsearchqa",
|
||||
evaluator_class="rdagent.scenarios.rl.autorl_bench.benchmarks.deepsearchqa.eval.DeepSearchQAEvaluator",
|
||||
data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.deepsearchqa.data",
|
||||
description="DeepSearchQA - Google DeepMind 多步信息检索基准（900题，17领域）",
|
||||
eval_config={
|
||||
"num_samples": 100, # 快速评测用 100，完整评测用 900
|
||||
"max_steps": 6, # ReAct 最大搜索轮次
|
||||
# api_key": "...", # 可选，不填则用 DuckDuckGo
|
||||
},
|
||||
expose_files=["eval.py"],
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks.smith import discover_smith_benchmarks
|
||||
|
||||
BENCHMARKS.update(discover_smith_benchmarks())
|
||||
|
||||
|
||||
def get_benchmark(benchmark_id: str) -> BenchmarkConfig:
|
||||
"""获取 benchmark 配置"""
|
||||
if benchmark_id not in BENCHMARKS:
|
||||
available = list(BENCHMARKS.keys())
|
||||
raise ValueError(f"Unknown benchmark: {benchmark_id}. Available: {available}")
|
||||
return BENCHMARKS[benchmark_id]
|
||||
|
||||
|
||||
def get_evaluator(benchmark_id: str) -> BaseEvaluator:
|
||||
"""获取 benchmark 的评测器实例"""
|
||||
config = get_benchmark(benchmark_id)
|
||||
|
||||
# 动态导入评测器类
|
||||
module_path, class_name = config.evaluator_class.rsplit(".", 1)
|
||||
module = importlib.import_module(module_path)
|
||||
evaluator_class: Type[BaseEvaluator] = getattr(module, class_name)
|
||||
|
||||
return evaluator_class(config)
|
||||
|
||||
|
||||
def list_benchmarks() -> list[str]:
|
||||
"""列出所有可用的 benchmark"""
|
||||
return list(BENCHMARKS.keys())
|
||||
@@ -0,0 +1 @@
|
||||
"""ALFWorld Benchmark"""
|
||||
@@ -0,0 +1,36 @@
|
||||
# ALFWorld base config (from alfworld official repo)
|
||||
# $ALFWORLD_DATA is set by eval.py -> data._ensure_alfworld_data()
|
||||
|
||||
dataset:
|
||||
data_path: '$ALFWORLD_DATA/json_2.1.1/train'
|
||||
eval_id_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_seen'
|
||||
eval_ood_data_path: '$ALFWORLD_DATA/json_2.1.1/valid_unseen'
|
||||
num_train_games: -1
|
||||
num_eval_games: -1
|
||||
|
||||
logic:
|
||||
domain: '$ALFWORLD_DATA/logic/alfred.pddl'
|
||||
grammar: '$ALFWORLD_DATA/logic/alfred.twl2'
|
||||
|
||||
env:
|
||||
type: 'AlfredTWEnv'
|
||||
domain_randomization: False
|
||||
task_types: [1, 2, 3, 4, 5, 6]
|
||||
expert_timeout_steps: 150
|
||||
expert_type: "handcoded"
|
||||
goal_desc_human_anns_prob: 0.0
|
||||
|
||||
controller:
|
||||
type: 'oracle'
|
||||
debug: False
|
||||
load_receps: True
|
||||
|
||||
general:
|
||||
random_seed: 42
|
||||
use_cuda: True
|
||||
task: 'alfred'
|
||||
training_method: 'dagger'
|
||||
|
||||
dagger:
|
||||
training:
|
||||
max_nb_steps_per_episode: 50
|
||||
@@ -0,0 +1,78 @@
|
||||
"""
|
||||
ALFWorld 数据准备
|
||||
|
||||
官方 alfworld-download 一次性下载所有数据（json + pddl + game.tw-pddl + logic）
|
||||
到 ~/.cache/alfworld/，然后只把训练数据 symlink 给 agent。
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def _run_alfworld_download() -> None:
|
||||
"""调用 alfworld-download，兼容 conda env PATH 问题"""
|
||||
import subprocess
|
||||
|
||||
bin_dir = Path(sys.executable).parent
|
||||
script = bin_dir / "alfworld-download"
|
||||
if script.exists():
|
||||
subprocess.run([sys.executable, str(script)], check=True)
|
||||
else:
|
||||
subprocess.run(["alfworld-download"], check=True)
|
||||
|
||||
|
||||
def _ensure_alfworld_data() -> Path:
|
||||
"""确保 alfworld 完整数据已下载，返回数据根目录
|
||||
|
||||
alfworld-download 下载三个 zip 到 ~/.cache/alfworld/:
|
||||
- json_2.1.1_json.zip -> traj_data.json
|
||||
- json_2.1.1_pddl.zip -> initial_state.pddl
|
||||
- json_2.1.3_tw-pddl.zip -> game.tw-pddl
|
||||
+ logic/alfred.pddl, logic/alfred.twl2
|
||||
"""
|
||||
cache_dir = Path.home() / ".cache" / "alfworld"
|
||||
json_dir = cache_dir / "json_2.1.1"
|
||||
|
||||
tw_pddl_ok = json_dir.exists() and any(json_dir.rglob("game.tw-pddl"))
|
||||
pddl_ok = json_dir.exists() and any(json_dir.rglob("initial_state.pddl"))
|
||||
logic_ok = (cache_dir / "logic" / "alfred.pddl").exists()
|
||||
|
||||
if tw_pddl_ok and pddl_ok and logic_ok:
|
||||
logger.info(f"ALFWorld data already complete: {cache_dir}")
|
||||
return cache_dir
|
||||
|
||||
logger.info("Running alfworld-download (downloads ~2GB, first time only)...")
|
||||
_run_alfworld_download()
|
||||
|
||||
if not any(json_dir.rglob("game.tw-pddl")):
|
||||
raise RuntimeError(
|
||||
f"alfworld-download finished but game.tw-pddl not found in {json_dir}. "
|
||||
"Check network connectivity to GitHub releases."
|
||||
)
|
||||
logger.info(f"ALFWorld data ready: {cache_dir}")
|
||||
return cache_dir
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""准备 ALFWorld 训练数据（agent 可见）"""
|
||||
marker = target_dir / ".downloaded"
|
||||
if marker.exists():
|
||||
logger.info(f"ALFWorld train data exists: {target_dir}")
|
||||
return
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cache_dir = _ensure_alfworld_data()
|
||||
train_src = cache_dir / "json_2.1.1" / "train"
|
||||
|
||||
if not train_src.exists():
|
||||
raise FileNotFoundError(f"ALFWorld train data not found: {train_src}")
|
||||
|
||||
train_dst = target_dir / "train"
|
||||
if not train_dst.exists():
|
||||
train_dst.symlink_to(train_src)
|
||||
logger.info(f"ALFWorld train data linked: {train_dst} -> {train_src}")
|
||||
|
||||
marker.touch()
|
||||
@@ -0,0 +1,62 @@
|
||||
# ALFWorld 任务
|
||||
|
||||
## 目标
|
||||
训练模型在 ALFWorld 文本游戏环境中获得更高的任务成功率。这是一个**交互式**任务：模型需要在环境中多步决策（rollout），而非一次性生成答案。
|
||||
|
||||
## 环境概述
|
||||
ALFWorld 是一个文本模拟的家庭环境（TextWorld 引擎）。模型扮演 agent，通过文本指令在房间中导航、操作物品来完成任务。
|
||||
|
||||
## 任务类型（6 种）
|
||||
1. **pick_and_place**: 拿起物品放到指定位置
|
||||
2. **pick_clean_then_place**: 清洁物品后放到指定位置
|
||||
3. **pick_heat_then_place**: 加热物品后放到指定位置
|
||||
4. **pick_cool_then_place**: 冷却物品后放到指定位置
|
||||
5. **look_at_obj_in_light**: 在灯光下查看物品
|
||||
6. **pick_two_obj_and_place**: 拿起两个物品放到指定位置
|
||||
|
||||
## Rollout 流程
|
||||
|
||||
每局游戏的交互循环：
|
||||
|
||||
```
|
||||
初始化：ob, info = env.reset() # 获取初始观察（房间描述 + 任务目标）
|
||||
|
||||
循环（每步）：
|
||||
action = model(观察历史) # 模型根据历史生成动作（文本）
|
||||
ob, reward, done, info = env.step([action]) # 环境执行动作，返回新观察
|
||||
if done:
|
||||
break
|
||||
```
|
||||
|
||||
**一个 rollout 示例（pick_and_place）：**
|
||||
```
|
||||
任务: "put a pencil in/on shelf."
|
||||
|
||||
Step 1: 观察: "You are in the middle of a room. Looking around you, you see a bed 1, a desk 1, a shelf 1..."
|
||||
动作: "go to desk 1"
|
||||
Step 2: 观察: "On the desk 1, you see a pencil 1, a book 2."
|
||||
动作: "take pencil 1 from desk 1"
|
||||
Step 3: 观察: "You pick up the pencil 1 from the desk 1."
|
||||
动作: "go to shelf 1"
|
||||
Step 4: 观察: "You arrive at shelf 1. On the shelf 1, you see nothing."
|
||||
动作: "put pencil 1 in/on shelf 1"
|
||||
Step 5: 观察: "You put the pencil 1 in/on the shelf 1."
|
||||
结果: 任务完成
|
||||
```
|
||||
|
||||
## 可用动作空间
|
||||
Agent 的动作是自由文本，常见动作包括：
|
||||
- 导航: `go to {object} {id}`（如 `go to desk 1`, `go to fridge 1`）
|
||||
- 拿取: `take {object} {id} from {location} {id}`
|
||||
- 放置: `put {object} {id} in/on {location} {id}`
|
||||
- 打开/关闭: `open {object} {id}`, `close {object} {id}`
|
||||
- 加热/冷却: `heat {object} {id} with microwave {id}`, `cool {object} {id} with fridge {id}`
|
||||
- 清洁: `clean {object} {id} with sinkbasin {id}`
|
||||
- 使用: `use {object} {id}`（如 `use desklamp 1`）
|
||||
- 思考: `think: {reasoning}`（不影响环境状态）
|
||||
|
||||
## 评测指标
|
||||
- **成功率** = 成功任务数 / 总任务数
|
||||
|
||||
## 参考代码
|
||||
环境交互和评测的完整实现见 `eval.py`。
|
||||
@@ -0,0 +1,373 @@
|
||||
"""
|
||||
ALFWorld Evaluator - 交互式文本游戏环境
|
||||
|
||||
使用 ReAct agent（few-shot + 完整历史）在 ALFWorld 中评测 LLM。
|
||||
支持两种后端：
|
||||
- vllm: 本地模型推理（text completion，和 ReAct 原版一致）
|
||||
- api: OpenAI 兼容 API（chat completion）
|
||||
|
||||
ReAct 官方代码: https://github.com/ysymyth/ReAct/blob/main/alfworld.ipynb
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
# 日志目录
|
||||
LOG_DIR = Path(__file__).resolve().parent.parent.parent / "log"
|
||||
|
||||
|
||||
class _Tee:
|
||||
"""同时输出到终端和日志文件"""
|
||||
|
||||
def __init__(self, filepath):
|
||||
self.terminal = sys.__stdout__
|
||||
self.log = open(filepath, "w", encoding="utf-8")
|
||||
|
||||
def write(self, message):
|
||||
self.terminal.write(message)
|
||||
self.log.write(message)
|
||||
self.log.flush()
|
||||
|
||||
def flush(self):
|
||||
self.terminal.flush()
|
||||
self.log.flush()
|
||||
|
||||
def isatty(self):
|
||||
return False
|
||||
|
||||
def fileno(self):
|
||||
return self.terminal.fileno()
|
||||
|
||||
|
||||
def _log(msg: str):
|
||||
"""简单的 print 日志（会被 Tee 同时写入文件）"""
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# ReAct agent 核心逻辑（来自官方 alfworld.ipynb）
|
||||
# ============================================================
|
||||
|
||||
# 任务类型 → few-shot prompt key 的映射
|
||||
TASK_PREFIXES = {
|
||||
"pick_and_place": "put",
|
||||
"pick_clean_then_place": "clean",
|
||||
"pick_heat_then_place": "heat",
|
||||
"pick_cool_then_place": "cool",
|
||||
"look_at_obj": "examine",
|
||||
"pick_two_obj": "puttwo",
|
||||
}
|
||||
|
||||
|
||||
def process_ob(ob: str) -> str:
|
||||
"""官方 ReAct 的 observation 清洗"""
|
||||
if ob.startswith("You arrive at loc "):
|
||||
ob = ob[ob.find(". ") + 2 :]
|
||||
return ob
|
||||
|
||||
|
||||
def alfworld_run(llm_fn: Callable, env, prompt: str, ob: str, max_steps: int = 50) -> tuple:
|
||||
"""
|
||||
ReAct 官方的单局评测逻辑。
|
||||
|
||||
Args:
|
||||
llm_fn: llm(prompt, stop) -> str
|
||||
env: ALFWorld 环境实例
|
||||
prompt: few-shot prompt（含 2 个示例）
|
||||
ob: 初始 observation
|
||||
max_steps: 最大步数
|
||||
|
||||
Returns:
|
||||
(reward, steps): reward=1 表示成功，steps 为实际步数
|
||||
"""
|
||||
init_prompt = prompt + ob + "\n>"
|
||||
history = ""
|
||||
for i in range(1, max_steps + 1):
|
||||
action = llm_fn(init_prompt + history, stop=["\n"]).strip()
|
||||
observation, reward, done, info = env.step([action])
|
||||
observation = process_ob(observation[0])
|
||||
reward = info["won"][0]
|
||||
done = done[0]
|
||||
if action.startswith("think:"):
|
||||
observation = "OK."
|
||||
_log(f" Act {i}: {action}")
|
||||
_log(f" Obs {i}: {observation}")
|
||||
history += f" {action}\n{observation}\n>"
|
||||
if done:
|
||||
return reward, i
|
||||
return 0, max_steps
|
||||
|
||||
|
||||
# ============================================================
|
||||
# LLM 后端工厂
|
||||
# ============================================================
|
||||
|
||||
|
||||
def create_llm_fn(backend: str, model_path: str, **kwargs) -> tuple:
|
||||
"""
|
||||
创建统一的 llm(prompt, stop) 函数。
|
||||
|
||||
backend="vllm": 本地模型，text completion（和 ReAct 原版行为一致）
|
||||
backend="api": OpenAI 兼容 chat API
|
||||
|
||||
Returns:
|
||||
(llm_fn, cleanup_fn): cleanup_fn 释放 GPU 显存
|
||||
"""
|
||||
if backend == "vllm":
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.distributed.parallel_state import destroy_model_parallel
|
||||
|
||||
llm_engine = LLM(
|
||||
model=model_path, tensor_parallel_size=kwargs.get("tensor_parallel_size", 1), trust_remote_code=True
|
||||
)
|
||||
|
||||
def vllm_fn(prompt: str, stop: List[str] = None) -> str:
|
||||
params = SamplingParams(temperature=0, max_tokens=100, stop=stop or ["\n"])
|
||||
outputs = llm_engine.generate([prompt], params)
|
||||
return outputs[0].outputs[0].text
|
||||
|
||||
def cleanup():
|
||||
nonlocal llm_engine
|
||||
import gc
|
||||
|
||||
import torch
|
||||
|
||||
destroy_model_parallel()
|
||||
llm_engine = None
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
_log("vLLM engine released, GPU memory freed.")
|
||||
|
||||
return vllm_fn, cleanup
|
||||
|
||||
elif backend == "api":
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key=kwargs.get("api_key", os.getenv("OPENAI_API_KEY")),
|
||||
base_url=kwargs.get("api_base", os.getenv("OPENAI_API_BASE")),
|
||||
)
|
||||
model_name = model_path
|
||||
|
||||
system_msg = (
|
||||
"You are playing a text-based household game. "
|
||||
"You will be given a task and interaction history. "
|
||||
"Output ONLY the next action (e.g. 'go to desk 1', 'take mug 1 from desk 1', "
|
||||
"'use desklamp 1', 'think: I need to find...') with NO extra text, "
|
||||
"NO prefix like '>' or 'Action:', just the raw action string."
|
||||
)
|
||||
|
||||
def api_fn(prompt: str, stop: List[str] = None) -> str:
|
||||
response = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": system_msg},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=100,
|
||||
stop=stop or ["\n"],
|
||||
)
|
||||
text = response.choices[0].message.content or ""
|
||||
text = text.strip()
|
||||
if text.startswith("> "):
|
||||
text = text[2:]
|
||||
return text
|
||||
|
||||
return api_fn, lambda: None
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unknown backend: {backend}. Use 'vllm' or 'api'.")
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Evaluator
|
||||
# ============================================================
|
||||
|
||||
|
||||
class ALFWorldEvaluator(BaseEvaluator):
|
||||
"""
|
||||
ALFWorld 评测器（ReAct agent）
|
||||
|
||||
eval_config 字段：
|
||||
max_steps: 每局最大步数（默认 50）
|
||||
env_num: 评测局数（默认 134）
|
||||
react_prompts: ReAct few-shot prompts 文件路径
|
||||
backend: "vllm" 或 "api"（默认自动判断）
|
||||
api_key: API 密钥（backend=api 时）
|
||||
api_base: API 地址（backend=api 时）
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
"""运行 ALFWorld 评测"""
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "alfworld"
|
||||
|
||||
# 合并 kwargs 到 eval_config
|
||||
cfg = {**self.eval_config, **kwargs}
|
||||
max_steps = cfg.get("max_steps", 50)
|
||||
env_num = cfg.get("env_num", 134)
|
||||
|
||||
# --- 设置日志 Tee ---
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
model_safe = model_path.replace("/", "_")
|
||||
log_file = LOG_DIR / f"alfworld_{model_safe}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
sys.stdout = _Tee(log_file)
|
||||
|
||||
# --- 判断 backend ---
|
||||
backend = cfg.get("backend")
|
||||
if backend is None:
|
||||
backend = "api" if not Path(model_path).exists() else "vllm"
|
||||
_log(f"Log: {log_file}")
|
||||
_log(f"ALFWorld eval: backend={backend}, model={model_path}")
|
||||
|
||||
# --- 创建 LLM 函数 ---
|
||||
llm_fn, llm_cleanup = create_llm_fn(
|
||||
backend=backend,
|
||||
model_path=model_path,
|
||||
api_key=cfg.get("api_key"),
|
||||
api_base=cfg.get("api_base"),
|
||||
tensor_parallel_size=cfg.get("tensor_parallel_size", 1),
|
||||
)
|
||||
|
||||
# --- 加载 ReAct few-shot prompts ---
|
||||
prompts_path = cfg.get("react_prompts")
|
||||
if prompts_path is None:
|
||||
# 默认路径：和 eval.py 同目录下的 react_prompts.json
|
||||
prompts_path = Path(__file__).parent / "react_prompts.json"
|
||||
with open(prompts_path) as f:
|
||||
react_prompts = json.load(f)
|
||||
|
||||
# --- 确保 ALFWorld 游戏数据已下载 ---
|
||||
self._ensure_alfworld_data()
|
||||
|
||||
# --- 初始化 ALFWorld 环境 ---
|
||||
workspace = Path(workspace_path)
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks.alfworld.data import (
|
||||
_ensure_alfworld_data,
|
||||
)
|
||||
|
||||
alfworld_data = str(_ensure_alfworld_data())
|
||||
os.environ["ALFWORLD_DATA"] = alfworld_data
|
||||
|
||||
# env_config: 读同目录下官方 base_config.yaml，展开 $ALFWORLD_DATA
|
||||
config_yaml = Path(__file__).parent / "base_config.yaml"
|
||||
with open(config_yaml) as f:
|
||||
import yaml
|
||||
|
||||
env_config = yaml.safe_load(f)
|
||||
env_config = self._expand_env_vars(env_config)
|
||||
|
||||
from alfworld.agents.environment import get_environment
|
||||
|
||||
split = cfg.get("split", "eval_out_of_distribution")
|
||||
env_type = env_config.get("env", {}).get("type", "AlfredTWEnv")
|
||||
alfred_env = get_environment(env_type)(env_config, train_eval=split)
|
||||
env = alfred_env.init_env(batch_size=1)
|
||||
|
||||
num_games = min(env_num, alfred_env.num_games)
|
||||
_log(f"ALFWorld: {num_games} games, max {max_steps} steps, split={split}")
|
||||
|
||||
# --- 评测循环（ReAct 官方逻辑） ---
|
||||
cnts = [0] * 6
|
||||
rs = [0] * 6
|
||||
|
||||
for game_no in range(num_games):
|
||||
ob, info = env.reset()
|
||||
ob = "\n".join(ob[0].split("\n\n")[1:])
|
||||
name = "/".join(info["extra.gamefile"][0].split("/")[-3:-1])
|
||||
_log(f"\n[Game {game_no + 1}/{num_games}] {name}")
|
||||
|
||||
matched = False
|
||||
for i, (prefix, prompt_key) in enumerate(TASK_PREFIXES.items()):
|
||||
if name.startswith(prefix):
|
||||
prompt = (
|
||||
"Interact with a household to solve a task. Here are two examples.\n"
|
||||
+ react_prompts[f"react_{prompt_key}_1"]
|
||||
+ react_prompts[f"react_{prompt_key}_0"]
|
||||
+ "\nHere is the task.\n"
|
||||
)
|
||||
reward, steps = alfworld_run(llm_fn, env, prompt, ob, max_steps)
|
||||
rs[i] += reward
|
||||
cnts[i] += 1
|
||||
matched = True
|
||||
_log(f" Result: {'WON' if reward else 'LOST'} ({steps} steps)")
|
||||
break
|
||||
|
||||
if not matched:
|
||||
_log(f" WARNING: Unknown task type: {name}, skipping")
|
||||
continue
|
||||
|
||||
total_r, total_c = sum(rs), sum(cnts)
|
||||
_log(f" Running: {total_r}/{total_c} = {total_r / max(total_c, 1):.1%}")
|
||||
|
||||
env.close()
|
||||
llm_cleanup()
|
||||
|
||||
# --- 汇总结果 ---
|
||||
total_success = sum(rs)
|
||||
total_count = sum(cnts)
|
||||
success_rate = total_success / total_count if total_count > 0 else 0.0
|
||||
|
||||
per_task = {}
|
||||
for (prefix, _), s, c in zip(TASK_PREFIXES.items(), rs, cnts):
|
||||
if c > 0:
|
||||
per_task[prefix] = {"success": s, "total": c, "rate": s / c}
|
||||
|
||||
result["score"] = success_rate * 100
|
||||
result["accuracy_summary"] = {
|
||||
"success_count": total_success,
|
||||
"total_count": total_count,
|
||||
"success_rate": success_rate,
|
||||
"per_task": per_task,
|
||||
}
|
||||
|
||||
_log(f"\nALFWorld done: {total_success}/{total_count} = {success_rate:.2%}")
|
||||
for prefix, stats in per_task.items():
|
||||
_log(f" {prefix:30s} {stats['success']}/{stats['total']} = {stats['rate']:.0%}")
|
||||
|
||||
# 恢复 stdout
|
||||
sys.stdout = sys.__stdout__
|
||||
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _ensure_alfworld_data():
|
||||
"""检查 ALFWorld 游戏数据（~2.1GB），没有就自动下载"""
|
||||
import subprocess
|
||||
|
||||
cache_dir = Path.home() / ".cache" / "alfworld"
|
||||
if (cache_dir / "json_2.1.1").exists():
|
||||
return
|
||||
_log("Downloading ALFWorld game data (~2.1GB, first time only)...")
|
||||
subprocess.run(["alfworld-download"], check=True)
|
||||
_log(f"ALFWorld data downloaded to {cache_dir}")
|
||||
|
||||
def _expand_env_vars(self, obj):
|
||||
"""递归展开 $ENV_VAR"""
|
||||
if isinstance(obj, str):
|
||||
return os.path.expandvars(obj)
|
||||
elif isinstance(obj, dict):
|
||||
return {k: self._expand_env_vars(v) for k, v in obj.items()}
|
||||
elif isinstance(obj, list):
|
||||
return [self._expand_env_vars(x) for x in obj]
|
||||
return obj
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,5 @@
|
||||
# ALFWorld benchmark 额外依赖
|
||||
# pip install -r benchmarks/alfworld/requirements.txt
|
||||
alfworld
|
||||
textworld
|
||||
openai
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
<|im_start|>system
|
||||
You are a highly efficient assistant, who evaluates and rank large language models (LLMs) based on the quality of their responses to given prompts. This process will create a leaderboard reflecting the most accurate and human-preferred answers.
|
||||
<|im_end|>
|
||||
<|im_start|>user
|
||||
I require a leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding responses. Your task is to assess these responses, ranking the models in order of preference from a human perspective. Once ranked, please output the results in a structured JSON format for the make_partial_leaderboard function.
|
||||
|
||||
## Prompt
|
||||
|
||||
{
|
||||
"instruction": """{instruction}""",
|
||||
}
|
||||
|
||||
## Model Outputs
|
||||
|
||||
Here are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.
|
||||
|
||||
{
|
||||
{
|
||||
"model": "m",
|
||||
"output": """{output_1}"""
|
||||
},
|
||||
{
|
||||
"model": "M",
|
||||
"output": """{output_2}"""
|
||||
}
|
||||
}
|
||||
|
||||
## Task
|
||||
|
||||
Evaluate and rank the models based on the quality and relevance of their outputs. The ranking should be such that the model with the highest quality output is ranked first.
|
||||
<|im_end|>
|
||||
+1202
File diff suppressed because it is too large
Load Diff
+44
@@ -0,0 +1,44 @@
|
||||
alpaca_eval_gpt5_2_fn:
|
||||
prompt_template: "annotators_gpt52_fn/alpaca_eval_fn.txt"
|
||||
fn_completions: "openai_completions"
|
||||
completions_kwargs:
|
||||
model_name: "gpt-5.2"
|
||||
max_tokens: 100
|
||||
temperature: 0
|
||||
tool_choice:
|
||||
type: function
|
||||
function:
|
||||
name: "make_partial_leaderboard"
|
||||
tools:
|
||||
- type: function
|
||||
function:
|
||||
name: "make_partial_leaderboard"
|
||||
description: "Make a leaderboard of models given a list of the models ordered by the preference of their outputs."
|
||||
strict: true
|
||||
parameters:
|
||||
type: "object"
|
||||
properties:
|
||||
ordered_models:
|
||||
type: "array"
|
||||
description: "A list of models ordered by the preference of their outputs. The first model in the list has the best output."
|
||||
items:
|
||||
type: "object"
|
||||
properties:
|
||||
model:
|
||||
type: "string"
|
||||
description: "The name of the model"
|
||||
rank:
|
||||
type: "number"
|
||||
description: "Order of preference of the model, 1 has the best output"
|
||||
additionalProperties: false
|
||||
required: ["model", "rank"]
|
||||
additionalProperties: false
|
||||
required: ["ordered_models"]
|
||||
fn_completion_parser: "pipeline_meta_parser"
|
||||
completion_parser_kwargs:
|
||||
parsers_to_kwargs:
|
||||
json_parser:
|
||||
annotation_key: "ordered_models"
|
||||
ranking_parser:
|
||||
model_1_name: "m"
|
||||
batch_size: 1
|
||||
@@ -0,0 +1,53 @@
|
||||
"""
|
||||
AlpacaEval 训练数据准备
|
||||
|
||||
使用独立指令数据作为训练集（避免与评测集泄漏）。
|
||||
默认使用 tatsu-lab/alpaca 的前 N 条样本。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from datasets import load_dataset
|
||||
from loguru import logger
|
||||
|
||||
DATASET_REPO = "tatsu-lab/alpaca"
|
||||
TRAIN_SAMPLES = int(os.getenv("ALPACAEVAL_TRAIN_SAMPLES", "2000"))
|
||||
|
||||
|
||||
def _convert_row(row: dict) -> dict:
|
||||
instruction = row.get("instruction", "")
|
||||
user_input = row.get("input", "")
|
||||
output = row.get("output", "")
|
||||
question = instruction if not user_input else f"{instruction}\n\n{user_input}"
|
||||
return {
|
||||
"instruction": instruction,
|
||||
"input": user_input,
|
||||
"output": output,
|
||||
"question": question,
|
||||
"answer": output,
|
||||
}
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""下载指令数据（agent 可见）。"""
|
||||
output_file = target_dir / "train.jsonl"
|
||||
|
||||
if output_file.exists():
|
||||
with open(output_file, "r", encoding="utf-8") as f:
|
||||
line_count = sum(1 for _ in f)
|
||||
if line_count == TRAIN_SAMPLES:
|
||||
logger.info(f"AlpacaEval train data exists: {output_file} ({line_count} samples)")
|
||||
return
|
||||
logger.warning(f"AlpacaEval train data has {line_count} samples, expected {TRAIN_SAMPLES}. Rebuilding...")
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"Downloading {DATASET_REPO} (first {TRAIN_SAMPLES} samples)...")
|
||||
dataset = load_dataset(DATASET_REPO, split=f"train[:{TRAIN_SAMPLES}]")
|
||||
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for item in dataset:
|
||||
f.write(json.dumps(_convert_row(item), ensure_ascii=False) + "\n")
|
||||
|
||||
logger.info(f"Saved {len(dataset)} samples to {output_file}")
|
||||
@@ -0,0 +1,17 @@
|
||||
# AlpacaEval 2.0 任务
|
||||
|
||||
## 目标
|
||||
评估模型的指令遵循与回答偏好表现（LLM Judge）。
|
||||
|
||||
## 评测集
|
||||
- AlpacaEval 2.0（`tatsu-lab/alpaca_eval` / `alpaca_eval_gpt4_baseline.json`）
|
||||
- 规模：805 条
|
||||
- 评测指标：Length-Controlled Win Rate（默认）
|
||||
|
||||
## 训练数据（agent 可见）
|
||||
- 默认使用 `tatsu-lab/alpaca` 前 2000 条指令样本
|
||||
- 可通过环境变量 `ALPACAEVAL_TRAIN_SAMPLES` 调整样本数
|
||||
|
||||
## 说明
|
||||
- 评测使用 GPT-4 Turbo 作为裁判（需配置 `OPENAI_API_KEY` / `OPENAI_API_BASE`）
|
||||
- 评测集与训练集独立，避免泄漏
|
||||
@@ -0,0 +1,201 @@
|
||||
"""
|
||||
AlpacaEval 2.0 Evaluator
|
||||
|
||||
流程：
|
||||
1. 读取 AlpacaEval 2.0 参考输出（gpt4 baseline）
|
||||
2. 用 vLLM 生成模型输出
|
||||
3. 调用 alpaca_eval 进行 head-to-head 评测（Length-Controlled Win Rate）
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
DEFAULT_REFERENCE_FILE = "alpaca_eval_gpt4_baseline.json"
|
||||
DEFAULT_ANNOTATORS_CONFIG = "weighted_alpaca_eval_gpt4_turbo"
|
||||
|
||||
|
||||
class AlpacaEvalEvaluator(BaseEvaluator):
|
||||
"""AlpacaEval 2.0 评测器（LLM Judge）"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
model_name: str = "",
|
||||
gpu_count: int = 1,
|
||||
test_range: str = "[:]",
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "alpacaeval"
|
||||
|
||||
if not self.validate_model(model_path):
|
||||
result["error"] = f"Model not found: {model_path}"
|
||||
return result
|
||||
|
||||
try:
|
||||
from alpaca_eval import evaluate as alpaca_evaluate
|
||||
except Exception as e:
|
||||
result["error"] = f"alpaca_eval import failed: {e}"
|
||||
return result
|
||||
|
||||
# 1) Load reference outputs (AlpacaEval 2.0)
|
||||
reference_file = self.eval_config.get("reference_file", DEFAULT_REFERENCE_FILE)
|
||||
reference_outputs = self._load_reference_outputs(reference_file)
|
||||
|
||||
# Optionally limit instances for quick eval
|
||||
max_instances = self.eval_config.get("max_instances")
|
||||
if isinstance(max_instances, int) and max_instances > 0:
|
||||
reference_outputs = reference_outputs[:max_instances]
|
||||
|
||||
# 2) Generate model outputs with vLLM
|
||||
work_dir = Path(workspace_path) / "benchmark_results" / "alpacaeval"
|
||||
work_dir.mkdir(parents=True, exist_ok=True)
|
||||
model_outputs = self._generate_model_outputs(
|
||||
model_path=model_path,
|
||||
model_name=model_name,
|
||||
reference_outputs=reference_outputs,
|
||||
gpu_count=gpu_count,
|
||||
)
|
||||
try:
|
||||
(work_dir / "model_outputs.json").write_text(json.dumps(model_outputs, ensure_ascii=False, indent=2))
|
||||
except Exception:
|
||||
logger.warning("Failed to save AlpacaEval model outputs")
|
||||
|
||||
# 3) AlpacaEval scoring
|
||||
annotators_config = self.eval_config.get("annotators_config", DEFAULT_ANNOTATORS_CONFIG)
|
||||
config_path = Path(annotators_config)
|
||||
if not config_path.is_absolute():
|
||||
local_path = Path(__file__).parent / annotators_config
|
||||
if local_path.exists():
|
||||
annotators_config = str(local_path)
|
||||
|
||||
try:
|
||||
df_leaderboard, all_crossannotations = alpaca_evaluate(
|
||||
model_outputs=model_outputs,
|
||||
reference_outputs=reference_outputs,
|
||||
annotators_config=annotators_config,
|
||||
name=model_name or "model",
|
||||
output_path=str(work_dir),
|
||||
is_return_instead_of_print=True,
|
||||
)
|
||||
except Exception as e:
|
||||
result["error"] = f"alpaca_eval failed: {e}"
|
||||
return result
|
||||
|
||||
# Extract score
|
||||
score, summary = self._extract_score(df_leaderboard, model_name or "model")
|
||||
result["score"] = score
|
||||
summary.update(
|
||||
{
|
||||
"num_samples": len(reference_outputs),
|
||||
"annotators_config": annotators_config,
|
||||
"reference_file": reference_file,
|
||||
}
|
||||
)
|
||||
result["accuracy_summary"] = summary
|
||||
|
||||
logger.info(f"AlpacaEval score: {result['score']}")
|
||||
return result
|
||||
|
||||
def _load_reference_outputs(self, filename: str) -> List[dict]:
|
||||
path = hf_hub_download(
|
||||
repo_id="tatsu-lab/alpaca_eval",
|
||||
repo_type="dataset",
|
||||
filename=filename,
|
||||
)
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
# Expect list of dicts with keys: instruction, output, generator
|
||||
return data
|
||||
|
||||
def _format_prompt(self, instruction: str, tokenizer) -> str:
|
||||
if hasattr(tokenizer, "apply_chat_template"):
|
||||
messages = [{"role": "user", "content": instruction}]
|
||||
try:
|
||||
return tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
except Exception:
|
||||
pass
|
||||
return instruction
|
||||
|
||||
def _generate_model_outputs(
|
||||
self,
|
||||
model_path: str,
|
||||
model_name: str,
|
||||
reference_outputs: List[dict],
|
||||
gpu_count: int,
|
||||
) -> List[dict]:
|
||||
from transformers import AutoTokenizer
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
max_model_len = int(self.eval_config.get("max_model_len", 4096))
|
||||
max_tokens = int(self.eval_config.get("max_tokens", 512))
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
||||
tp_size = 1
|
||||
if gpu_count and gpu_count > 0:
|
||||
power = 0
|
||||
while (1 << (power + 1)) <= gpu_count:
|
||||
power += 1
|
||||
tp_size = 1 << power
|
||||
llm = LLM(
|
||||
model=model_path,
|
||||
tensor_parallel_size=tp_size,
|
||||
trust_remote_code=True,
|
||||
max_model_len=max_model_len,
|
||||
)
|
||||
sampling_params = SamplingParams(
|
||||
temperature=0.0,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
|
||||
prompts = [self._format_prompt(item["instruction"], tokenizer) for item in reference_outputs]
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
model_outputs = []
|
||||
for item, out in zip(reference_outputs, outputs):
|
||||
text = out.outputs[0].text.strip() if out.outputs else ""
|
||||
model_outputs.append(
|
||||
{
|
||||
"instruction": item.get("instruction", ""),
|
||||
"output": text,
|
||||
"generator": model_name or Path(model_path).name,
|
||||
"dataset": item.get("dataset", ""),
|
||||
}
|
||||
)
|
||||
|
||||
# Save raw outputs for debugging
|
||||
return model_outputs
|
||||
|
||||
def _extract_score(self, df_leaderboard, name: str) -> tuple[float, dict]:
|
||||
row = None
|
||||
if name in df_leaderboard.index:
|
||||
row = df_leaderboard.loc[name]
|
||||
elif "model" in df_leaderboard.columns:
|
||||
matched = df_leaderboard[df_leaderboard["model"] == name]
|
||||
if not matched.empty:
|
||||
row = matched.iloc[0]
|
||||
|
||||
if row is None:
|
||||
return 0.0, {"error": "model not found in leaderboard"}
|
||||
|
||||
summary = row.to_dict() if hasattr(row, "to_dict") else dict(row)
|
||||
score = None
|
||||
for key in ("length_controlled_winrate", "win_rate", "winrate"):
|
||||
if key in summary:
|
||||
score = summary[key]
|
||||
break
|
||||
if score is None:
|
||||
score = 0.0
|
||||
return float(score), summary
|
||||
@@ -0,0 +1 @@
|
||||
alpaca-eval
|
||||
@@ -0,0 +1 @@
|
||||
"DeepSearchQA Benchmark"
|
||||
@@ -0,0 +1,14 @@
|
||||
# benchmarks/deepsearchqa/data.py
|
||||
from pathlib import Path
|
||||
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path):
|
||||
"""下载 deepsearchqa 数据到本地"""
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 只下载 eval split（DeepSearchQA 只有 eval split）
|
||||
dataset = load_dataset("google/deepsearchqa", split="eval")
|
||||
dataset.save_to_disk(str(target_dir / "deepsearchqa"))
|
||||
print(f"DeepSearchQA saved to {target_dir}")
|
||||
@@ -0,0 +1,32 @@
|
||||
# DeepSearchQA 任务
|
||||
|
||||
## 目标
|
||||
回答需要多步网络搜索的复杂问题，涵盖 17 个领域。
|
||||
|
||||
## 数据集
|
||||
- 来源: google/deepsearchqa (HuggingFace)
|
||||
- 规模: 900 题
|
||||
- 答案类型: Single Answer (35%) / Set Answer (65%)
|
||||
|
||||
## Rollout 流程
|
||||
|
||||
模型通过 ReAct 格式交替搜索和推理：
|
||||
|
||||
Question: "Which countries had GDP > X and..."
|
||||
Thought: I need to find GDP data first.
|
||||
Action: search[GDP per capita rankings 2023]
|
||||
Observation: [search result summarization]
|
||||
Thought: Now I need to filter by condition Y.
|
||||
Action: search[condition Y countries list]
|
||||
Observation: [search result summarization]
|
||||
Thought: I have enough information.
|
||||
Action: answer[Country A, Country B]
|
||||
|
||||
## 评测指标
|
||||
- 答案由 LLM Judge 打分（推荐 gemini-2.5-flash）
|
||||
- Set Answer 需包含 gold 中的所有项目
|
||||
- 最终分数 = 正确数 / 总题数 × 100
|
||||
|
||||
## 搜索后端配置
|
||||
- 默认使用 `duckduckgo-search` 包（无需配置，但可能有频率限制）
|
||||
- 推荐配置 `SERPAPI_KEY` 环境变量以获得更稳定的搜索结果
|
||||
@@ -0,0 +1,330 @@
|
||||
# benchmarks/deepsearchqa/eval.py
|
||||
|
||||
"""
|
||||
DeepSearchQA Evaluator
|
||||
|
||||
使用 vLLM 加载本地模型，结合 Web Search 工具，
|
||||
在 DeepSearchQA 数据集上评测模型的多步信息检索能力。
|
||||
|
||||
数据集: https://huggingface.co/datasets/google/deepsearchqa
|
||||
评测方式: LLM Judge (推荐 gemini-2.5-flash)
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
import requests
|
||||
from datasets import load_dataset
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
REACT_SYSTEM_PROMPT = """You are a research assistant that answers questions by searching the web.
|
||||
|
||||
You must follow this format strictly:
|
||||
Thought: [your reasoning]
|
||||
Action: search[your search query] <- to search the web
|
||||
OR
|
||||
Action: answer[Paris] <- to give final answer
|
||||
|
||||
Rules:
|
||||
- Always start with a Thought
|
||||
- Use search[] to find information
|
||||
- Use answer[] ONLY when you have enough information
|
||||
- For Set Answer questions, list all items separated by commas
|
||||
- Be concise and factual
|
||||
"""
|
||||
|
||||
|
||||
class DeepSearchQAEvaluator(BaseEvaluator):
|
||||
"""
|
||||
DeepSearchQA 评测器
|
||||
|
||||
流程：
|
||||
1. 从 HuggingFace 加载数据集
|
||||
2. 对每道题运行 ReAct 循环（模型 + 搜索工具）
|
||||
3. 用 LLM Judge 对比模型答案与 gold answer
|
||||
4. 返回 F1/EM 分数
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(self, model_path: str, workspace_path: str, **kwargs) -> Dict[str, Any]:
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "deepsearchqa"
|
||||
|
||||
if not self.validate_model(model_path):
|
||||
result["error"] = f"Model not found: {model_path}"
|
||||
return result
|
||||
|
||||
# load datasets
|
||||
num_samples = self.eval_config.get("num_samples", 100)
|
||||
dataset = load_dataset(
|
||||
"google/deepsearchqa",
|
||||
split="eval",
|
||||
# 如果已下载到本地可用 data_dir 参数
|
||||
)
|
||||
samples = list(dataset.select(range(min(num_samples, len(dataset)))))
|
||||
logger.info(f"DeepSearchQA: {len(samples)} samples")
|
||||
|
||||
# load model (vLLM)
|
||||
logger.info(f"Loading model: {model_path}")
|
||||
llm = LLM(
|
||||
model=model_path,
|
||||
tensor_parallel_size=1,
|
||||
trust_remote_code=True,
|
||||
max_model_len=4096,
|
||||
)
|
||||
sampling_params = SamplingParams(
|
||||
temperature=0.0,
|
||||
max_tokens=256,
|
||||
stop=["\nAction:", "\nThought:", "\nObservation:"],
|
||||
)
|
||||
|
||||
# search tool
|
||||
search_fn = self._get_search_function()
|
||||
|
||||
# evaluation loop
|
||||
correct = 0
|
||||
results_detail = []
|
||||
|
||||
for i, sample in enumerate(samples):
|
||||
question = sample["problem"]
|
||||
gold_answer = sample["answer"]
|
||||
answer_type = sample.get("answer_type", "Single Answer")
|
||||
|
||||
logger.info(f"\n[{i+1}/{len(samples)}] {question[:80]}...")
|
||||
|
||||
# ReAct loop
|
||||
predicted = self._react_loop(
|
||||
llm,
|
||||
sampling_params,
|
||||
search_fn,
|
||||
question,
|
||||
answer_type,
|
||||
)
|
||||
|
||||
# LLM Judge score
|
||||
score = self._judge_answer(predicted, gold_answer, answer_type)
|
||||
if score:
|
||||
correct += 1
|
||||
|
||||
results_detail.append(
|
||||
{
|
||||
"question": question[:100],
|
||||
"gold": gold_answer,
|
||||
"predicted": predicted,
|
||||
"answer_type": answer_type,
|
||||
"correct": score,
|
||||
}
|
||||
)
|
||||
logger.info(f" Predicted: {predicted[:80]}")
|
||||
logger.info(f" Gold: {gold_answer[:80]}")
|
||||
logger.info(f" Correct: {score}")
|
||||
running_acc = correct / (i + 1)
|
||||
logger.info(f" Running accuracy: {correct}/{i+1} = {running_acc:.2%}")
|
||||
|
||||
accuracy = correct / len(samples) if samples else 0.0
|
||||
result["score"] = accuracy * 100
|
||||
result["accuracy_summary"] = {
|
||||
"correct": correct,
|
||||
"total": len(samples),
|
||||
"accuracy": accuracy,
|
||||
"details": results_detail,
|
||||
}
|
||||
logger.info(f"\nDeepSearchQA done: {correct}/{len(samples)} = {accuracy:.2%}")
|
||||
return result
|
||||
|
||||
# ----------------------------------------------------------
|
||||
# ReAct loop
|
||||
# ----------------------------------------------------------
|
||||
|
||||
def _react_loop(
|
||||
self,
|
||||
llm: "LLM",
|
||||
sampling_params: "SamplingParams",
|
||||
search_fn,
|
||||
question: str,
|
||||
answer_type: str,
|
||||
) -> str:
|
||||
"""ReAct multi-step reasoning loop, return final answer string"""
|
||||
max_steps = self.eval_config.get("max_steps", 6)
|
||||
|
||||
conversation = f"Question: {question}\n" f"Answer type: {answer_type}\n\n" "Thought:"
|
||||
full_prompt = f"{REACT_SYSTEM_PROMPT}\n\n{conversation}"
|
||||
|
||||
# for step in range(max_steps):
|
||||
# outputs = llm.generate([full_prompt], sampling_params)
|
||||
# generated = outputs[0].outputs[0].text.strip()
|
||||
# full_prompt += f" {generated}"
|
||||
|
||||
# # parse Action
|
||||
# action_match = re.search(r"Action:\s*(search|answer)\[(.+?)\]", full_prompt, re.DOTALL)
|
||||
# if not action_match:
|
||||
# # force append an answer
|
||||
# full_prompt += "\nAction: answer["
|
||||
# outputs2 = llm.generate([full_prompt], SamplingParams(temperature=0, max_tokens=128, stop=["]"]))
|
||||
# return outputs2[0].outputs[0].text.strip()
|
||||
|
||||
# action_type = action_match.group(1)
|
||||
# action_content = action_match.group(2).strip()
|
||||
|
||||
# if action_type == "answer":
|
||||
# return action_content
|
||||
|
||||
# # execute search
|
||||
# observation = search_fn(action_content)
|
||||
# logger.info(f" Step {step+1} | Search: {action_content[:60]}")
|
||||
# logger.info(f" Observation: {observation[:120]}")
|
||||
|
||||
# full_prompt += (
|
||||
# f"\nObservation: {observation}\n"
|
||||
# "Thought:"
|
||||
# )
|
||||
# exceed max steps, extract last answer
|
||||
# last_answer = re.findall(r"Action:\s*answer\[(.+?)\]", full_prompt, re.DOTALL)
|
||||
# return last_answer[-1].strip() if last_answer else "I don't know"
|
||||
|
||||
# ...existing code...
|
||||
model_trace = ""
|
||||
|
||||
for step in range(max_steps):
|
||||
outputs = llm.generate([full_prompt], sampling_params)
|
||||
generated = outputs[0].outputs[0].text.strip()
|
||||
model_trace += ("\n" + generated) if model_trace else generated
|
||||
full_prompt += f" {generated}"
|
||||
|
||||
# parse Action ONLY from current model output
|
||||
action_match = re.search(r"Action:\s*(search|answer)\[(.+?)\]", generated, re.DOTALL)
|
||||
if not action_match:
|
||||
# force append an answer
|
||||
full_prompt += "\nAction: answer["
|
||||
outputs2 = llm.generate([full_prompt], SamplingParams(temperature=0, max_tokens=128, stop=["]"]))
|
||||
generated2 = outputs2[0].outputs[0].text.strip()
|
||||
model_trace += ("\n" + generated2) if generated2 else ""
|
||||
# reject template placeholder
|
||||
if generated2.strip().lower() == "your final answer":
|
||||
continue
|
||||
return generated2.strip()
|
||||
|
||||
action_type = action_match.group(1)
|
||||
action_content = action_match.group(2).strip()
|
||||
|
||||
if action_type == "answer":
|
||||
# reject template placeholder
|
||||
if action_content.lower() == "your final answer":
|
||||
# treat as no valid action, let loop continue
|
||||
full_prompt += "\nThat is not a valid answer. Please think again.\nThought:"
|
||||
continue
|
||||
return action_content
|
||||
|
||||
# execute search
|
||||
observation = search_fn(action_content)
|
||||
logger.info(f" Step {step+1} | Search: {action_content[:60]}")
|
||||
logger.info(f" Observation: {observation[:120]}")
|
||||
|
||||
full_prompt += f"\nObservation: {observation}\n" "Thought:"
|
||||
|
||||
# exceed max steps, extract last answer from model output only
|
||||
last_answer = re.findall(r"Action:\s*answer\[(.+?)\]", model_trace, re.DOTALL)
|
||||
# filter out template placeholder
|
||||
real_answers = [a.strip() for a in last_answer if a.strip().lower() != "your final answer"]
|
||||
return real_answers[-1] if real_answers else "I don't know"
|
||||
|
||||
# ----------------------------------------------------------
|
||||
# search tool
|
||||
# ----------------------------------------------------------
|
||||
|
||||
def _get_search_function(self):
|
||||
"""返回搜索函数，优先使用 SerpAPI，降级到 DuckDuckGo"""
|
||||
import os
|
||||
|
||||
serpapi_key = os.environ.get("SERPAPI_KEY") or self.eval_config.get("serpapi_key")
|
||||
|
||||
if serpapi_key:
|
||||
logger.info("Using SerpAPI for web search")
|
||||
return lambda q: self._serpapi_search(q, serpapi_key)
|
||||
else:
|
||||
logger.info("Using DuckDuckGo for web search (no SERPAPI_KEY)")
|
||||
return self._duckduckgo_search
|
||||
|
||||
def _serpapi_search(self, query: str, api_key: str) -> str:
|
||||
"""SerpAPI 搜索，返回摘要文本"""
|
||||
try:
|
||||
resp = requests.get(
|
||||
"https://serpapi.com/search",
|
||||
params={"q": query, "api_key": api_key, "num": 3},
|
||||
timeout=10,
|
||||
)
|
||||
data = resp.json()
|
||||
snippets = [r.get("snippet", "") for r in data.get("organic_results", [])[:3]]
|
||||
return " | ".join(snippets) or "No results found."
|
||||
except Exception as e:
|
||||
return f"Search error: {e}"
|
||||
|
||||
def _duckduckgo_search(self, query: str) -> str:
|
||||
"""DuckDuckGo 即时答案 API（免费，但结果较少）"""
|
||||
try:
|
||||
resp = requests.get(
|
||||
"https://api.duckduckgo.com/",
|
||||
params={"q": query, "format": "json", "no_html": 1},
|
||||
timeout=10,
|
||||
)
|
||||
data = resp.json()
|
||||
abstract = data.get("AbstractText", "")
|
||||
related = " | ".join(r.get("Text", "") for r in data.get("RelatedTopics", [])[:2] if isinstance(r, dict))
|
||||
return abstract or related or "No results found."
|
||||
except Exception as e:
|
||||
return f"Search error: {e}"
|
||||
|
||||
# ----------------------------------------------------------
|
||||
# LLM Judge
|
||||
# ----------------------------------------------------------
|
||||
|
||||
def _judge_answer(
|
||||
self,
|
||||
predicted: str,
|
||||
gold: str,
|
||||
answer_type: str,
|
||||
) -> bool:
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
|
||||
judge_prompt = f"""You are an answer evaluator. Compare the predicted answer to the gold answer.
|
||||
Question answer type: {answer_type}
|
||||
Gold answer: {gold}
|
||||
Predicted answer: {predicted}
|
||||
For "Single Answer": The predicted answer is correct if it contains the same key information as the gold answer.
|
||||
For "Set Answer": The predicted answer is correct if it contains ALL items from the gold answer (order doesn't matter, minor wording variations are OK).
|
||||
Reply with ONLY "correct" or "incorrect". No explanation."""
|
||||
|
||||
try:
|
||||
response = (
|
||||
APIBackend()
|
||||
.build_messages_and_create_chat_completion(
|
||||
user_prompt=judge_prompt,
|
||||
system_prompt="You are a strict answer evaluator.",
|
||||
)
|
||||
.strip()
|
||||
.lower()
|
||||
)
|
||||
return "correct" in response
|
||||
except Exception as e:
|
||||
logger.warning(f"Judge failed: {e}, falling back to string match")
|
||||
return self._string_match(predicted, gold, answer_type)
|
||||
|
||||
def _string_match(self, predicted: str, gold: str, answer_type: str) -> bool:
|
||||
"""fallback: simple string matching"""
|
||||
pred = predicted.strip().lower()
|
||||
gold = gold.strip().lower()
|
||||
if answer_type == "Single Answer":
|
||||
return gold in pred or pred in gold
|
||||
else:
|
||||
gold_items = [x.strip() for x in gold.split(",")]
|
||||
return all(item in pred for item in gold_items)
|
||||
@@ -0,0 +1,28 @@
|
||||
"""
|
||||
GSM8K 数据下载
|
||||
|
||||
Agent 只能看到 train split。
|
||||
评估（OpenCompass）用 test split，由 OpenCompass 自己内部加载。
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from datasets import load_dataset
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""下载 GSM8K 训练数据（agent 可见）"""
|
||||
output_file = target_dir / "train.jsonl"
|
||||
if output_file.exists():
|
||||
logger.info(f"GSM8K train data exists: {output_file}")
|
||||
return
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Downloading GSM8K train split...")
|
||||
dataset = load_dataset("openai/gsm8k", "main", split="train")
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for item in dataset:
|
||||
f.write(json.dumps(item, ensure_ascii=False) + "\n")
|
||||
logger.info(f"Saved {len(dataset)} samples to {output_file}")
|
||||
@@ -0,0 +1,16 @@
|
||||
# GSM8K 任务
|
||||
|
||||
## 目标
|
||||
训练模型在 GSM8K 数学问题上获得更高准确率。
|
||||
|
||||
## 数据格式
|
||||
```json
|
||||
{"question": "...", "answer": "... #### 42"}
|
||||
```
|
||||
|
||||
## 评测指标
|
||||
- 答案准确率（exact match）
|
||||
|
||||
## 提示
|
||||
- 答案格式: `#### 数字`
|
||||
- 使用 GRPO/PPO 等 RL 方法训练
|
||||
@@ -0,0 +1 @@
|
||||
"""HumanEval benchmark."""
|
||||
@@ -0,0 +1,50 @@
|
||||
"""
|
||||
HumanEval 数据下载
|
||||
|
||||
HumanEval 官方数据只有 test split，这里固定按 1:1 划分：
|
||||
- 前半（82 条）导出到 train.jsonl，给 agent 训练使用
|
||||
- 后半（82 条）留给评测使用（由 evaluator 通过 test_range 控制）
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from datasets import load_dataset
|
||||
from loguru import logger
|
||||
|
||||
_TOTAL_SAMPLES = 164
|
||||
_TRAIN_SAMPLES = _TOTAL_SAMPLES // 2
|
||||
|
||||
|
||||
def _convert_row(row: dict) -> dict:
|
||||
"""将 openai/openai_humaneval 统一为 autorl_bench 常用字段。"""
|
||||
return {
|
||||
"question": row.get("prompt", ""),
|
||||
"answer": row.get("canonical_solution", ""),
|
||||
"task_id": row.get("task_id", ""),
|
||||
"entry_point": row.get("entry_point", ""),
|
||||
"test": row.get("test", ""),
|
||||
}
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""下载 HumanEval 数据（agent 可见）。"""
|
||||
output_file = target_dir / "train.jsonl"
|
||||
if output_file.exists():
|
||||
with open(output_file, "r", encoding="utf-8") as f:
|
||||
line_count = sum(1 for _ in f)
|
||||
if line_count == _TRAIN_SAMPLES:
|
||||
logger.info(f"HumanEval train data exists: {output_file} ({line_count} samples)")
|
||||
return
|
||||
logger.warning(f"HumanEval train data has {line_count} samples, expected {_TRAIN_SAMPLES}. Rebuilding...")
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Downloading HumanEval split...")
|
||||
dataset = load_dataset("openai/openai_humaneval", split="test")
|
||||
train_split = dataset.select(range(_TRAIN_SAMPLES))
|
||||
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for item in train_split:
|
||||
f.write(json.dumps(_convert_row(item), ensure_ascii=False) + "\n")
|
||||
|
||||
logger.info(f"Saved {len(train_split)} train samples to {output_file}")
|
||||
@@ -0,0 +1,27 @@
|
||||
# HumanEval 任务
|
||||
|
||||
## 目标
|
||||
训练模型在 HumanEval 的 Python 函数补全任务上获得更高 pass@1。
|
||||
|
||||
## 数据格式
|
||||
```json
|
||||
{
|
||||
"question": "函数签名与 docstring（prompt）",
|
||||
"answer": "参考实现（canonical_solution）",
|
||||
"task_id": "HumanEval/0",
|
||||
"entry_point": "目标函数名",
|
||||
"test": "用于校验实现正确性的测试代码"
|
||||
}
|
||||
```
|
||||
|
||||
## 评测指标
|
||||
- pass@1（由 OpenCompass HumanEval 配置执行）
|
||||
|
||||
## 数据划分
|
||||
- HumanEval 原始 `test` 共 164 条。
|
||||
- 训练可见数据固定为前 82 条（`[:82]`）。
|
||||
- 自动评测固定为后 82 条（`[82:]`），与训练集不重叠。
|
||||
|
||||
## 提示
|
||||
- 生成可执行的 Python 函数实现，优先保证正确性。
|
||||
- 注意函数名必须与 `entry_point` 一致。
|
||||
@@ -0,0 +1,4 @@
|
||||
# HumanEval benchmark 额外依赖
|
||||
# 安装方式：
|
||||
# git clone https://github.com/XianBW/human-eval.git ~/human-eval
|
||||
# cd ~/human-eval && pip install -e .
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Smith benchmarks — dynamic discovery via config.yaml.
|
||||
|
||||
Scans SMITH_BENCH_DIR/*/config.yaml and builds BenchmarkConfig entries
|
||||
automatically. The actual benchmark code/data lives outside the repo;
|
||||
default location is ``<repo-root>/../rl-smith/benchmarks/``.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import BenchmarkConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Default: rl-smith/benchmarks as a sibling of the repo root
|
||||
import rdagent
|
||||
|
||||
_REPO_ROOT = Path(rdagent.__path__[0]).resolve().parent # rdagent pkg dir → RD-Agent/
|
||||
_SMITH_BENCH_DIR = Path(os.environ.get("SMITH_BENCH_DIR", str(_REPO_ROOT.parent / "rl-smith" / "benchmarks")))
|
||||
_PKG = "rdagent.scenarios.rl.autorl_bench"
|
||||
|
||||
|
||||
def discover_smith_benchmarks() -> dict[str, BenchmarkConfig]:
|
||||
"""Scan SMITH_BENCH_DIR/*/config.yaml and build BenchmarkConfig dict."""
|
||||
if not _SMITH_BENCH_DIR.is_dir():
|
||||
logger.warning(
|
||||
"SMITH_BENCH_DIR=%s does not exist; returning empty smith registry",
|
||||
_SMITH_BENCH_DIR,
|
||||
)
|
||||
return {}
|
||||
|
||||
result = {}
|
||||
for cfg_path in sorted(_SMITH_BENCH_DIR.glob("*/config.yaml")):
|
||||
bench_dir = cfg_path.parent
|
||||
if bench_dir.name.startswith("_"):
|
||||
continue
|
||||
raw = yaml.safe_load(cfg_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict) or not raw.get("name"):
|
||||
continue
|
||||
|
||||
name = raw["name"]
|
||||
eval_mode = raw.get("eval_mode", "per_sample")
|
||||
bench_id = f"smith-{name}"
|
||||
|
||||
if eval_mode == "opencompass":
|
||||
evaluator_class = f"{_PKG}.core.opencompass.OpenCompassEvaluator"
|
||||
eval_config = {"dataset": raw.get("opencompass_dataset", "")}
|
||||
elif eval_mode == "per_sample":
|
||||
evaluator_class = f"{_PKG}.benchmarks.smith.per_sample_eval.PerSampleEvaluator"
|
||||
eval_config = {"eval_script": str(bench_dir / "eval.py")}
|
||||
else:
|
||||
# Skip benchmarks with unsupported eval modes (e.g. custom_model)
|
||||
# that are already registered as standalone benchmarks.
|
||||
logger.info("Skipping smith-%s: unsupported eval_mode=%s", name, eval_mode)
|
||||
continue
|
||||
|
||||
result[bench_id] = BenchmarkConfig(
|
||||
id=bench_id,
|
||||
evaluator_class=evaluator_class,
|
||||
data_module="",
|
||||
description=raw.get("description", ""),
|
||||
eval_config=eval_config,
|
||||
expose_files=raw.get("expose_files", []),
|
||||
bench_dir=str(bench_dir),
|
||||
)
|
||||
return result
|
||||
@@ -0,0 +1,192 @@
|
||||
"""Per-sample evaluator for smith benchmarks (arc_agi, zero_shot_cot).
|
||||
|
||||
Loads a model via vLLM, runs inference on each test sample, then uses the
|
||||
benchmark's eval.py to score each prediction individually.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
|
||||
class PerSampleEvaluator(BaseEvaluator):
|
||||
"""Evaluator that scores each sample individually using benchmark-specific eval.py."""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
model_name: str = "",
|
||||
gpu_count: int = 1,
|
||||
test_range: str = "[:]",
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "per_sample"
|
||||
|
||||
if not self.validate_model(model_path):
|
||||
result["error"] = f"Model not found: {model_path}"
|
||||
return result
|
||||
|
||||
# Load the benchmark-specific eval module
|
||||
eval_script = self.eval_config.get("eval_script", "")
|
||||
eval_module_path = self.eval_config.get("eval_module", "")
|
||||
if not eval_script and not eval_module_path:
|
||||
result["error"] = "No eval_script or eval_module configured"
|
||||
return result
|
||||
|
||||
try:
|
||||
if eval_script:
|
||||
spec = importlib.util.spec_from_file_location("eval", eval_script)
|
||||
eval_mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(eval_mod)
|
||||
else:
|
||||
eval_mod = importlib.import_module(eval_module_path)
|
||||
except Exception as e:
|
||||
result["error"] = f"Cannot load eval module: {e}"
|
||||
return result
|
||||
|
||||
# Load test data
|
||||
workspace = Path(workspace_path)
|
||||
data_dir = workspace / "data"
|
||||
test_file = data_dir / "train.jsonl"
|
||||
if not test_file.exists():
|
||||
result["error"] = f"Test data not found: {test_file}"
|
||||
return result
|
||||
|
||||
test_data = []
|
||||
with open(test_file, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
test_data.append(json.loads(line))
|
||||
|
||||
# Apply test_range slicing
|
||||
test_data = _apply_range(test_data, test_range)
|
||||
|
||||
if not test_data:
|
||||
result["error"] = "No test data after applying range"
|
||||
return result
|
||||
|
||||
logger.info(f"[{self.benchmark_id}] Running per-sample eval on {len(test_data)} samples")
|
||||
|
||||
# Load model and run inference via vLLM
|
||||
try:
|
||||
import vllm
|
||||
from vllm import SamplingParams
|
||||
|
||||
llm = vllm.LLM(model=model_path, tensor_parallel_size=gpu_count)
|
||||
sampling_params = SamplingParams(temperature=0, max_tokens=2048)
|
||||
|
||||
prompts = []
|
||||
for item in test_data:
|
||||
q = item.get("question", "")
|
||||
if isinstance(q, dict):
|
||||
# For arc_agi: question is a JSON object, stringify it
|
||||
q = json.dumps(q)
|
||||
prompts.append(q)
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
except Exception as e:
|
||||
# Clean up vLLM GPU memory even on failure
|
||||
if "llm" in locals():
|
||||
_cleanup_vllm(llm)
|
||||
result["error"] = f"vLLM inference failed: {e}"
|
||||
return result
|
||||
|
||||
# Release vLLM GPU memory to avoid OOM for subsequent evaluations
|
||||
_cleanup_vllm(llm)
|
||||
|
||||
# Score each sample
|
||||
total = 0
|
||||
correct = 0.0
|
||||
for item, output in zip(test_data, outputs):
|
||||
model_answer = output.outputs[0].text
|
||||
question = item.get("question", "")
|
||||
reference = item.get("answer", "")
|
||||
|
||||
# Pass extra kwargs from the item (e.g. answer_type for zero_shot_cot)
|
||||
extra = {k: v for k, v in item.items() if k not in ("question", "answer")}
|
||||
try:
|
||||
score = eval_mod.evaluate(question, model_answer, reference, **extra)
|
||||
except Exception as e:
|
||||
logger.warning(f"Eval error on sample: {e}")
|
||||
score = 0.0
|
||||
|
||||
correct += score
|
||||
total += 1
|
||||
|
||||
accuracy = (correct / total) * 100 if total > 0 else 0.0
|
||||
result["score"] = accuracy
|
||||
result["accuracy_summary"] = {
|
||||
"correct": correct,
|
||||
"total": total,
|
||||
"accuracy": accuracy,
|
||||
}
|
||||
|
||||
logger.info(f"[{self.benchmark_id}] Score: {accuracy:.2f}% ({correct}/{total})")
|
||||
return result
|
||||
|
||||
|
||||
def _cleanup_vllm(llm) -> None:
|
||||
"""Release vLLM GPU memory without initializing CUDA in the main process.
|
||||
|
||||
We delete the LLM object and run torch.cuda.empty_cache() inside a
|
||||
*spawned* subprocess so that the main process never touches CUDA directly.
|
||||
This avoids the 'Cannot re-initialize CUDA in forked subprocess' error
|
||||
that OpenCompass would hit later when it forks inference workers.
|
||||
"""
|
||||
import multiprocessing as mp
|
||||
|
||||
def _gpu_cleanup():
|
||||
try:
|
||||
import gc
|
||||
|
||||
import torch
|
||||
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
del llm
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
ctx = mp.get_context("spawn")
|
||||
p = ctx.Process(target=_gpu_cleanup)
|
||||
p.start()
|
||||
p.join(timeout=30)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _apply_range(data: list, test_range: str) -> list:
|
||||
"""Apply a Python-style slice string like '[:]' or '[:100]' to a list."""
|
||||
test_range = test_range.strip()
|
||||
if not test_range or test_range == "[:]":
|
||||
return data
|
||||
try:
|
||||
# Parse "[start:stop]" or "[:stop]" etc.
|
||||
inner = test_range.strip("[]")
|
||||
parts = inner.split(":")
|
||||
start = int(parts[0]) if parts[0] else None
|
||||
stop = int(parts[1]) if len(parts) > 1 and parts[1] else None
|
||||
return data[start:stop]
|
||||
except Exception:
|
||||
return data
|
||||
@@ -0,0 +1,6 @@
|
||||
"""WebShop Benchmark"""
|
||||
|
||||
from .data import download_train_data
|
||||
from .eval import WebShopEvaluator
|
||||
|
||||
__all__ = ["WebShopEvaluator", "download_train_data"]
|
||||
@@ -0,0 +1,154 @@
|
||||
"""
|
||||
WebShop 数据准备
|
||||
|
||||
注意：WebShop PyPI 包不完整（缺少 web_agent_site 模块），需要从 GitHub 克隆完整仓库。
|
||||
为避免 setup.sh 破坏当前环境依赖，我们手动下载数据。
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from loguru import logger
|
||||
|
||||
WEBSHOP_CACHE_DIR = Path.home() / ".cache" / "webshop"
|
||||
WEBSHOP_REPO_DIR = WEBSHOP_CACHE_DIR / "repo"
|
||||
|
||||
|
||||
def _clone_webshop_repo() -> Path:
|
||||
"""克隆 WebShop 仓库到缓存目录"""
|
||||
if WEBSHOP_REPO_DIR.exists() and (WEBSHOP_REPO_DIR / ".git").exists():
|
||||
logger.info(f"WebShop repo exists: {WEBSHOP_REPO_DIR}")
|
||||
return WEBSHOP_REPO_DIR
|
||||
|
||||
WEBSHOP_CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Cloning WebShop repository...")
|
||||
|
||||
subprocess.run(
|
||||
["git", "clone", "--depth", "1", "https://github.com/princeton-nlp/webshop.git", str(WEBSHOP_REPO_DIR)],
|
||||
check=True,
|
||||
)
|
||||
|
||||
logger.info(f"WebShop repo cloned to: {WEBSHOP_REPO_DIR}")
|
||||
return WEBSHOP_REPO_DIR
|
||||
|
||||
|
||||
def _ensure_repo_in_path():
|
||||
"""确保 webshop 仓库在 Python 路径中（优先于 PyPI 包）。
|
||||
|
||||
同时向 venv site-packages 写入 webshop.pth，使任何子进程（accelerate launch 等）
|
||||
都能直接 import web_agent_site，无需手动设置 sys.path。
|
||||
"""
|
||||
import site
|
||||
|
||||
repo_str = str(WEBSHOP_REPO_DIR)
|
||||
if repo_str not in sys.path:
|
||||
sys.path.insert(0, repo_str)
|
||||
|
||||
# Write a .pth file so subprocesses inherit the path without extra setup.
|
||||
pth_content = repo_str + "\n"
|
||||
for sp in site.getsitepackages():
|
||||
pth_file = Path(sp) / "webshop.pth"
|
||||
try:
|
||||
if not pth_file.exists() or pth_file.read_text() != pth_content:
|
||||
pth_file.write_text(pth_content)
|
||||
logger.info(f"Registered webshop path via {pth_file}")
|
||||
break
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
|
||||
def _download_webshop_data():
|
||||
"""下载 WebShop 数据（手动下载，避免 setup.sh 破坏环境依赖）"""
|
||||
data_dir = WEBSHOP_REPO_DIR / "data"
|
||||
marker = data_dir / ".download_complete"
|
||||
|
||||
if marker.exists():
|
||||
logger.info(f"WebShop data already downloaded: {data_dir}")
|
||||
return
|
||||
|
||||
logger.info("Downloading WebShop data (~500MB, first time only)...")
|
||||
data_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 使用 gdown 下载 Google Drive 文件（small 数据集，1000个产品）
|
||||
files = [
|
||||
("1EgHdxQ_YxqIQlvvq5iKlCrkEKR6-j0Ib", "items_shuffle_1000.json"),
|
||||
("1IduG0xl544V_A_jv3tHXC0kyFi7PnyBu", "items_ins_v2_1000.json"),
|
||||
("14Kb5SPBk_jfdLZ_CDBNitW98QLDlKR5O", "items_human_ins.json"),
|
||||
]
|
||||
|
||||
for file_id, filename in files:
|
||||
filepath = data_dir / filename
|
||||
if not filepath.exists():
|
||||
try:
|
||||
subprocess.run(["gdown", file_id, "-O", str(filepath)], check=True, timeout=120)
|
||||
logger.info(f"Downloaded {filename}")
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
|
||||
logger.warning(f"Failed to download {filename}: {e}")
|
||||
|
||||
# 构建搜索引擎索引
|
||||
_build_search_index()
|
||||
|
||||
marker.touch()
|
||||
logger.info(f"WebShop data ready: {data_dir}")
|
||||
|
||||
|
||||
def _build_search_index():
|
||||
"""构建 WebShop 搜索引擎索引"""
|
||||
search_engine_dir = WEBSHOP_REPO_DIR / "search_engine"
|
||||
marker = search_engine_dir / ".index_built"
|
||||
|
||||
if marker.exists():
|
||||
return
|
||||
|
||||
logger.info("Building WebShop search index...")
|
||||
|
||||
# 创建 convert_product_file_format.py 需要的所有目录
|
||||
for d in ["resources_100", "resources", "resources_1k", "resources_100k", "indexes"]:
|
||||
(search_engine_dir / d).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
# 转换产品文件格式
|
||||
convert_script = search_engine_dir / "convert_product_file_format.py"
|
||||
if convert_script.exists():
|
||||
subprocess.run([sys.executable, str(convert_script)], cwd=search_engine_dir, check=True, timeout=60)
|
||||
|
||||
# 构建索引
|
||||
index_script = search_engine_dir / "run_indexing.sh"
|
||||
if index_script.exists():
|
||||
subprocess.run(["bash", str(index_script)], cwd=search_engine_dir, check=True, timeout=120)
|
||||
|
||||
marker.touch()
|
||||
logger.info("Search index built successfully")
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
|
||||
raise RuntimeError(f"Failed to build search index: {e}") from e
|
||||
|
||||
|
||||
def download_train_data(target_dir: Path) -> None:
|
||||
"""准备 WebShop 训练数据（agent 可见）
|
||||
|
||||
流程：
|
||||
1. 克隆 WebShop 仓库（如果不存在）
|
||||
2. 下载产品数据（手动方式，避免 setup.sh 破坏依赖）
|
||||
3. 将训练数据链接到 target_dir
|
||||
"""
|
||||
marker = target_dir / ".downloaded"
|
||||
if marker.exists():
|
||||
logger.info(f"WebShop train data exists: {target_dir}")
|
||||
return
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
_clone_webshop_repo()
|
||||
_ensure_repo_in_path()
|
||||
_download_webshop_data()
|
||||
|
||||
# 链接训练数据给 agent
|
||||
human_traj_src = WEBSHOP_REPO_DIR / "data" / "human_trajectories"
|
||||
if human_traj_src.exists():
|
||||
human_traj_dst = target_dir / "human_trajectories"
|
||||
if not human_traj_dst.exists():
|
||||
human_traj_dst.symlink_to(human_traj_src)
|
||||
logger.info(f"Linked human_trajectories: {human_traj_dst}")
|
||||
|
||||
marker.touch()
|
||||
@@ -0,0 +1,97 @@
|
||||
# WebShop 任务
|
||||
|
||||
## 目标
|
||||
训练模型在 WebShop 电商网站环境中获得更高的购物任务成功率。这是一个**交互式**任务：模型需要在网页环境中多步决策（rollout），根据用户指令搜索并购买匹配的产品。
|
||||
|
||||
## 环境概述
|
||||
|
||||
WebShop 是一个模拟电商网站环境，包含 118 万真实产品和用户指令。Agent 需要根据文本指令完成购物任务。
|
||||
|
||||
环境有 4 种页面状态：
|
||||
- **search** - 搜索页面，包含搜索框
|
||||
- **results** - 搜索结果页，列出匹配的产品
|
||||
- **item** - 产品详情页
|
||||
- **item-detail** - 产品详细信息页
|
||||
|
||||
## 动作空间
|
||||
|
||||
Agent 的动作是文本格式，有两种类型：
|
||||
|
||||
1. **搜索**: `search[query]` - 在搜索页面使用
|
||||
- 示例：`search[red running shoes]`
|
||||
|
||||
2. **选择**: `choose[option]` - 根据当前页面选择选项
|
||||
- `choose[Back to Search]` - 返回搜索页
|
||||
- `choose[Next >]` / `choose[< Prev]` - 翻页
|
||||
- `choose[Product Title]` - 选择产品
|
||||
- `choose[Option]` - 选择颜色/尺寸等变体
|
||||
- `choose[Description]` - 查看详情
|
||||
- `choose[Buy Now]` - 购买产品
|
||||
|
||||
## Rollout 流程
|
||||
|
||||
每轮购物任务的交互循环：
|
||||
|
||||
```python
|
||||
# 初始化
|
||||
obs, info = env.reset(idx=instruction_idx) # 获取初始观察（搜索页面）
|
||||
|
||||
done = False
|
||||
for step in range(max_steps):
|
||||
# 1. 模型根据指令、历史、当前观察生成动作
|
||||
action = model(instruction, history, obs)
|
||||
|
||||
# 2. 环境执行动作
|
||||
obs, reward, done, info = env.step(action)
|
||||
|
||||
# 3. 记录历史
|
||||
history.append((action, obs))
|
||||
|
||||
if done:
|
||||
break
|
||||
|
||||
# reward: 最终奖励 (0-1)，反映产品匹配程度
|
||||
```
|
||||
|
||||
**一个 rollout 示例**：
|
||||
|
||||
```
|
||||
指令: "I'm looking for a quick-release replacement fitness strap band;
|
||||
it should match my chic teal fitbit, and price lower than 40.00 dollars"
|
||||
|
||||
Step 1: 观察: "WebShop [SEP] Search [SEP]"
|
||||
动作: "search[quick-release fitness strap band teal fitbit]"
|
||||
|
||||
Step 2: 观察: "WebShop [SEP] Results [SEP] [Back to Search] [Next >]
|
||||
[Teal Silicone Sport Band for Fitbit... $12.99]
|
||||
[Quick Release Nylon Band Teal... $15.99]..."
|
||||
动作: "choose[Teal Silicone Sport Band for Fitbit Charge 2, Large, $12.99]"
|
||||
|
||||
Step 3: 观察: "WebShop [SEP] Item [SEP] Teal Silicone Sport Band...
|
||||
[Buy Now] [Back to Search] [Description] [Size Large] [Size Small]"
|
||||
动作: "choose[Buy Now]"
|
||||
|
||||
Step 4: 观察: "WebShop [SEP] Episode finished [SEP] reward = 0.95"
|
||||
结果: 任务完成，奖励 0.95（高匹配度）
|
||||
```
|
||||
|
||||
## 观测格式
|
||||
|
||||
环境返回的观测是文本格式：
|
||||
|
||||
```
|
||||
WebShop [SEP] {Page Type} [SEP] {Content}
|
||||
```
|
||||
|
||||
- `WebShop` - 固定前缀
|
||||
- `{Page Type}` - 页面类型：Search / Results / Item
|
||||
- `{Content}` - 页面内容，包括可用选项
|
||||
|
||||
## 评测指标
|
||||
|
||||
- **成功率** = 成功购买匹配产品的比例（reward >= 0.5 视为成功）
|
||||
- **平均奖励** = 所有任务的平均奖励值（0-1），基于产品类型、属性、价格匹配度计算
|
||||
|
||||
## 参考代码
|
||||
|
||||
环境交互和评测的完整实现见 `eval.py`。
|
||||
@@ -0,0 +1,426 @@
|
||||
"""
|
||||
WebShop Evaluator - 电商网站交互环境
|
||||
|
||||
使用 ReAct agent 在 WebShop 环境中评测 LLM。
|
||||
支持两种后端：
|
||||
- vllm: 本地模型推理
|
||||
- api: OpenAI 兼容 API
|
||||
|
||||
WebShop 官方代码: https://github.com/princeton-nlp/webshop
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Tuple
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
|
||||
from .data import WEBSHOP_REPO_DIR, _clone_webshop_repo, _ensure_repo_in_path
|
||||
|
||||
# 日志目录
|
||||
LOG_DIR = Path(__file__).resolve().parent.parent.parent / "log"
|
||||
|
||||
|
||||
class _Tee:
|
||||
"""同时输出到终端和日志文件"""
|
||||
|
||||
def __init__(self, filepath):
|
||||
self.terminal = sys.__stdout__
|
||||
self.log = open(filepath, "w", encoding="utf-8")
|
||||
|
||||
def write(self, message):
|
||||
self.terminal.write(message)
|
||||
self.log.write(message)
|
||||
self.log.flush()
|
||||
|
||||
def flush(self):
|
||||
self.terminal.flush()
|
||||
self.log.flush()
|
||||
|
||||
def isatty(self):
|
||||
return False
|
||||
|
||||
def fileno(self):
|
||||
return self.terminal.fileno()
|
||||
|
||||
|
||||
def _log(msg: str):
|
||||
"""简单的 print 日志（会被 Tee 同时写入文件）"""
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# LLM 后端工厂
|
||||
# ============================================================
|
||||
|
||||
|
||||
def create_llm_fn(backend: str, model_path: str, **kwargs) -> Tuple[Callable, Callable]:
|
||||
"""
|
||||
创建统一的 llm(prompt, stop) 函数。
|
||||
|
||||
backend="vllm": 本地模型，text completion
|
||||
backend="api": OpenAI 兼容 chat API
|
||||
|
||||
Returns:
|
||||
(llm_fn, cleanup_fn): cleanup_fn 释放资源
|
||||
"""
|
||||
if backend == "vllm":
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.distributed.parallel_state import destroy_model_parallel
|
||||
|
||||
llm_engine = LLM(
|
||||
model=model_path, tensor_parallel_size=kwargs.get("tensor_parallel_size", 1), trust_remote_code=True
|
||||
)
|
||||
|
||||
_vllm_sys_msg = (
|
||||
"You are a helpful shopping assistant browsing an e-commerce website. "
|
||||
"Given a user instruction, current observation, and available actions, "
|
||||
"pick the best action to find and purchase a matching product. "
|
||||
"Output ONLY one action (e.g., 'search[red shoes]', 'click[buy now]') "
|
||||
"with NO extra text, NO explanation."
|
||||
)
|
||||
|
||||
def vllm_fn(prompt: str, stop: List[str] = None) -> str:
|
||||
messages = [
|
||||
{"role": "system", "content": _vllm_sys_msg},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
params = SamplingParams(temperature=0, max_tokens=100, stop=stop or ["\n"])
|
||||
outputs = llm_engine.chat(messages, sampling_params=params)
|
||||
return outputs[0].outputs[0].text
|
||||
|
||||
def cleanup():
|
||||
nonlocal llm_engine
|
||||
import gc
|
||||
|
||||
import torch
|
||||
|
||||
destroy_model_parallel()
|
||||
llm_engine = None
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
_log("vLLM engine released, GPU memory freed.")
|
||||
|
||||
return vllm_fn, cleanup
|
||||
|
||||
elif backend == "api":
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key=kwargs.get("api_key") or os.getenv("OPENAI_API_KEY"),
|
||||
base_url=kwargs.get("api_base") or os.getenv("OPENAI_API_BASE"),
|
||||
)
|
||||
model_name = model_path
|
||||
|
||||
system_msg = (
|
||||
"You are a helpful shopping assistant browsing an e-commerce website. "
|
||||
"Given a user instruction, current observation, and available actions, "
|
||||
"pick the best action to find and purchase a matching product. "
|
||||
"Output ONLY one action (e.g., 'search[red shoes]', 'click[buy now]') "
|
||||
"with NO extra text, NO explanation."
|
||||
)
|
||||
|
||||
def api_fn(prompt: str, stop: List[str] = None) -> str:
|
||||
response = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": system_msg},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=100,
|
||||
stop=stop or ["\n"],
|
||||
)
|
||||
text = response.choices[0].message.content or ""
|
||||
return text.strip()
|
||||
|
||||
return api_fn, lambda: None
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unknown backend: {backend}. Use 'vllm' or 'api'.")
|
||||
|
||||
|
||||
# ============================================================
|
||||
# ReAct Agent 核心逻辑
|
||||
# ============================================================
|
||||
|
||||
|
||||
def _format_available_actions(avail: dict) -> str:
|
||||
"""将环境返回的 available_actions 格式化为文本列表"""
|
||||
lines = []
|
||||
if avail.get("has_search_bar"):
|
||||
lines.append("search[<your query>]")
|
||||
for txt in avail.get("clickables", []):
|
||||
lines.append(f"click[{txt}]")
|
||||
return "\n".join(f" {a}" for a in lines)
|
||||
|
||||
|
||||
def build_react_prompt(
|
||||
instruction: str,
|
||||
history: List[Tuple[str, str]],
|
||||
observation: str,
|
||||
available_actions: str = "",
|
||||
history_window: int = 5,
|
||||
) -> str:
|
||||
"""构建 ReAct 风格的提示词，包含 available_actions 和有限历史窗口"""
|
||||
prompt = f"""You are shopping on an e-commerce website. Find and purchase a product matching the user's instruction.
|
||||
|
||||
Instruction: {instruction}
|
||||
|
||||
Available actions:
|
||||
{available_actions}
|
||||
|
||||
Rules:
|
||||
- Output ONLY one action from the available actions list above.
|
||||
- For search, use: search[your query]
|
||||
- For clicking, use: click[exact text from the list]
|
||||
- Do NOT output anything other than the action.
|
||||
|
||||
Now it's your turn:
|
||||
"""
|
||||
|
||||
recent = history[-history_window:] if len(history) > history_window else history
|
||||
offset = len(history) - len(recent)
|
||||
|
||||
for i, (action, obs) in enumerate(recent):
|
||||
step_num = offset + i + 1
|
||||
prompt += f"\nObservation {step_num}: {obs}\n"
|
||||
prompt += f"Action {step_num}: {action}\n"
|
||||
|
||||
prompt += f"\nObservation {len(history)+1}: {observation}\n"
|
||||
prompt += f"Action {len(history)+1}:"
|
||||
|
||||
return prompt
|
||||
|
||||
|
||||
def webshop_run(
|
||||
llm_fn: Callable,
|
||||
env,
|
||||
instruction: str,
|
||||
observation: str,
|
||||
max_steps: int = 50,
|
||||
history_window: int = 5,
|
||||
) -> Tuple[float, int, bool]:
|
||||
"""
|
||||
单轮 WebShop 评测逻辑。
|
||||
|
||||
Args:
|
||||
llm_fn: llm(prompt, stop) -> str
|
||||
env: WebShop 环境实例
|
||||
instruction: 用户指令
|
||||
observation: 初始观察
|
||||
max_steps: 最大步数
|
||||
history_window: prompt 中保留的最近历史步数
|
||||
|
||||
Returns:
|
||||
(reward, steps, success): reward为最终奖励, steps为实际步数, success是否成功
|
||||
"""
|
||||
history = []
|
||||
|
||||
for step in range(1, max_steps + 1):
|
||||
avail = env.get_available_actions()
|
||||
avail_text = _format_available_actions(avail)
|
||||
|
||||
prompt = build_react_prompt(
|
||||
instruction,
|
||||
history,
|
||||
observation,
|
||||
available_actions=avail_text,
|
||||
history_window=history_window,
|
||||
)
|
||||
|
||||
action = llm_fn(prompt, stop=["\n"]).strip()
|
||||
|
||||
# 清理动作前缀
|
||||
if action.startswith("Action:"):
|
||||
action = action[7:].strip()
|
||||
if action.startswith("choose["):
|
||||
action = "click[" + action[7:]
|
||||
|
||||
_log(f" Step {step}: {action}")
|
||||
|
||||
observation, reward, done, info = env.step(action)
|
||||
|
||||
_log(f" Obs {step}: {observation[:200]}...")
|
||||
_log(f" Reward: {reward}, Done: {done}")
|
||||
|
||||
history.append((action, observation))
|
||||
|
||||
if done:
|
||||
success = reward >= 0.5
|
||||
return reward, step, success
|
||||
|
||||
return 0.0, max_steps, False
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Evaluator
|
||||
# ============================================================
|
||||
|
||||
|
||||
class WebShopEvaluator(BaseEvaluator):
|
||||
"""
|
||||
WebShop 评测器（ReAct agent）
|
||||
|
||||
eval_config 字段：
|
||||
max_steps: 每任务最大步数（默认 50）
|
||||
num_instructions: 评测指令数量（默认 100）
|
||||
backend: "vllm" 或 "api"（默认自动判断）
|
||||
api_key: API 密钥（backend=api 时）
|
||||
api_base: API 地址（backend=api 时）
|
||||
num_products: 加载的产品数量（默认 1000，可选 1000 或全部）
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
"""运行 WebShop 评测"""
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "webshop"
|
||||
|
||||
# 合并 kwargs 到 eval_config
|
||||
cfg = {**self.eval_config, **kwargs}
|
||||
max_steps = cfg.get("max_steps", 50)
|
||||
num_instructions = cfg.get("num_instructions", 100)
|
||||
num_products = cfg.get("num_products", 1000)
|
||||
|
||||
# --- 设置日志 Tee ---
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
model_safe = model_path.replace("/", "_").replace("\\", "_")
|
||||
log_file = LOG_DIR / f"webshop_{model_safe}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
old_stdout = sys.stdout
|
||||
sys.stdout = _Tee(log_file)
|
||||
|
||||
try:
|
||||
_log(f"Log: {log_file}")
|
||||
|
||||
# --- 确保 WebShop 仓库可用 ---
|
||||
_clone_webshop_repo()
|
||||
_ensure_repo_in_path()
|
||||
|
||||
# --- 判断 backend ---
|
||||
backend = cfg.get("backend")
|
||||
if backend is None:
|
||||
backend = "api" if not Path(model_path).exists() else "vllm"
|
||||
_log(f"WebShop eval: backend={backend}, model={model_path}")
|
||||
|
||||
# --- 创建 LLM 函数 ---
|
||||
llm_fn, llm_cleanup = create_llm_fn(
|
||||
backend=backend,
|
||||
model_path=model_path,
|
||||
api_key=cfg.get("api_key"),
|
||||
api_base=cfg.get("api_base"),
|
||||
tensor_parallel_size=cfg.get("tensor_parallel_size", 1),
|
||||
)
|
||||
|
||||
# --- 初始化 WebShop 环境 ---
|
||||
try:
|
||||
from web_agent_site.envs.web_agent_text_env import WebAgentTextEnv
|
||||
except ImportError as e:
|
||||
result["error"] = f"Failed to import WebShop: {e}. Please check WebShop installation."
|
||||
sys.stdout = old_stdout
|
||||
return result
|
||||
|
||||
env = WebAgentTextEnv(
|
||||
observation_mode="text",
|
||||
num_products=num_products,
|
||||
)
|
||||
|
||||
# --- 加载评测指令 ---
|
||||
instruction_idxs = list(range(min(num_instructions, 12000)))
|
||||
|
||||
_log(f"WebShop: {len(instruction_idxs)} instructions, max {max_steps} steps each")
|
||||
|
||||
# --- 评测循环 ---
|
||||
total_reward = 0.0
|
||||
success_count = 0
|
||||
total_steps = 0
|
||||
|
||||
for idx, instr_idx in enumerate(instruction_idxs):
|
||||
try:
|
||||
# 重置环境
|
||||
observation, _ = env.reset(session=instr_idx)
|
||||
instruction = env.get_instruction_text()
|
||||
|
||||
_log(f"\n[Task {idx + 1}/{len(instruction_idxs)}] {instruction[:80]}...")
|
||||
|
||||
# 运行 agent
|
||||
reward, steps, success = webshop_run(
|
||||
llm_fn=llm_fn,
|
||||
env=env,
|
||||
instruction=instruction,
|
||||
observation=observation,
|
||||
max_steps=max_steps,
|
||||
)
|
||||
|
||||
total_reward += reward
|
||||
total_steps += steps
|
||||
if success:
|
||||
success_count += 1
|
||||
|
||||
_log(f" Result: {'SUCCESS' if success else 'FAIL'} (reward={reward:.2f}, steps={steps})")
|
||||
|
||||
# 打印进度
|
||||
current_success_rate = success_count / (idx + 1)
|
||||
_log(f" Running: {success_count}/{idx + 1} = {current_success_rate:.1%}")
|
||||
|
||||
except Exception as e:
|
||||
_log(f" ERROR: {e}")
|
||||
import traceback
|
||||
|
||||
_log(traceback.format_exc())
|
||||
continue
|
||||
|
||||
# --- 汇总结果 ---
|
||||
total_count = len(instruction_idxs)
|
||||
success_rate = success_count / total_count if total_count > 0 else 0.0
|
||||
avg_reward = total_reward / total_count if total_count > 0 else 0.0
|
||||
avg_steps = total_steps / total_count if total_count > 0 else 0.0
|
||||
|
||||
result["score"] = success_rate * 100 # 转为百分比
|
||||
result["accuracy_summary"] = {
|
||||
"success_count": success_count,
|
||||
"total_count": total_count,
|
||||
"success_rate": success_rate,
|
||||
"avg_reward": avg_reward,
|
||||
"avg_steps": avg_steps,
|
||||
"total_reward": total_reward,
|
||||
}
|
||||
|
||||
_log(f"\nWebShop done: {success_count}/{total_count} = {success_rate:.2%}")
|
||||
_log(f" Average reward: {avg_reward:.3f}")
|
||||
_log(f" Average steps: {avg_steps:.1f}")
|
||||
|
||||
except Exception as e:
|
||||
result["error"] = str(e)
|
||||
_log(f"ERROR: {e}")
|
||||
import traceback
|
||||
|
||||
_log(traceback.format_exc())
|
||||
|
||||
finally:
|
||||
# --- 清理 ---
|
||||
if "env" in locals():
|
||||
env.close()
|
||||
|
||||
# 释放 LLM 资源
|
||||
if "llm_cleanup" in locals():
|
||||
llm_cleanup()
|
||||
|
||||
# 恢复 stdout
|
||||
sys.stdout = old_stdout
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,29 @@
|
||||
# WebShop benchmark 依赖
|
||||
#
|
||||
# 前置要求：Java 11+ (JDK) 和 faiss-cpu
|
||||
# conda install -c conda-forge openjdk=11 faiss-cpu
|
||||
#
|
||||
# 安装命令:
|
||||
# pip install -r benchmarks/webshop/requirements.txt
|
||||
# python -m spacy download en_core_web_sm
|
||||
#
|
||||
# 注意：Flask/Werkzeug 已在主 requirements.txt 中固定为 2.x 版本
|
||||
|
||||
# WebShop PyPI 包
|
||||
webshop
|
||||
|
||||
# 数据下载工具
|
||||
gdown
|
||||
|
||||
# WebShop 特有依赖
|
||||
gym==0.24.0
|
||||
beautifulsoup4==4.11.1
|
||||
cleantext==1.1.4
|
||||
pyserini==0.17.0
|
||||
rank_bm25==0.2.2
|
||||
thefuzz==0.19.0
|
||||
spacy==3.7.2
|
||||
|
||||
# 注意：Flask/Werkzeug 固定为 2.x（Flask 3.x 与 WebShop 不兼容）
|
||||
flask==2.2.5
|
||||
Werkzeug==2.2.3
|
||||
@@ -0,0 +1,51 @@
|
||||
"""
|
||||
AutoRL-Bench 配置
|
||||
|
||||
独立配置，不依赖 RL_RD_SETTING，只复用 rdagent 基类。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import SettingsConfigDict
|
||||
|
||||
from rdagent.core.conf import ExtendedBaseSettings
|
||||
|
||||
|
||||
class AutoRLBenchSettings(ExtendedBaseSettings):
|
||||
"""AutoRL-Bench 配置
|
||||
|
||||
环境变量前缀: AUTORL_
|
||||
例如: AUTORL_FILE_PATH=/data/autorl_bench
|
||||
"""
|
||||
|
||||
model_config = SettingsConfigDict(env_prefix="AUTORL_", protected_namespaces=())
|
||||
|
||||
file_path: Path = Path.cwd() / "git_ignore_folder" / "rl_files"
|
||||
rdagent_root: Path = Path.cwd() # Docker 挂载用，可通过 AUTORL_RDAGENT_ROOT 覆盖
|
||||
|
||||
|
||||
AUTORL_BENCH_SETTING = AutoRLBenchSettings()
|
||||
|
||||
|
||||
def get_autorl_bench_dir() -> Path:
|
||||
return Path(__file__).parent
|
||||
|
||||
|
||||
def get_workspace_dir() -> Path:
|
||||
return get_autorl_bench_dir() / "workspace"
|
||||
|
||||
|
||||
def get_instructions_file() -> Path:
|
||||
return get_autorl_bench_dir() / "core" / "instructions.md"
|
||||
|
||||
|
||||
def get_models_dir() -> Path:
|
||||
return AUTORL_BENCH_SETTING.file_path / "models"
|
||||
|
||||
|
||||
def get_data_dir() -> Path:
|
||||
return AUTORL_BENCH_SETTING.file_path / "datasets"
|
||||
|
||||
|
||||
def get_baseline_cache_dir() -> Path:
|
||||
return AUTORL_BENCH_SETTING.file_path / "baseline_workspace"
|
||||
@@ -0,0 +1,86 @@
|
||||
"""
|
||||
AutoRL-Bench Core Module
|
||||
|
||||
主干代码，定义统一的评测接口和服务。
|
||||
开发新 benchmark 或 agent 时不需要修改此模块。
|
||||
|
||||
================================================================================
|
||||
面向开发者的接口约定
|
||||
================================================================================
|
||||
|
||||
评测器基类: BaseEvaluator (evaluator.py)
|
||||
所有 benchmark 评测器继承此类并实现 run_eval 方法。
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str, # 训练后的模型路径（本地目录）
|
||||
workspace_path: str, # 工作目录路径
|
||||
model_name: str = "", # 模型名称（用于配置推理参数）
|
||||
gpu_count: int = 1, # 可用 GPU 数量
|
||||
test_range: str = "[:]", # 测试数据范围
|
||||
**kwargs,
|
||||
) -> EvalResult
|
||||
|
||||
评测结果: EvalResult (evaluator.py)
|
||||
TypedDict，必须字段: benchmark, model_path, score, accuracy_summary
|
||||
|
||||
具体实现:
|
||||
- OpenCompassEvaluator (opencompass.py) — 基于 OpenCompass 的评测
|
||||
- PerSampleEvaluator (benchmarks/smith/) — 逐样本评测
|
||||
|
||||
服务:
|
||||
- GradingServer (server.py) — 评测服务器
|
||||
- create_grading_server (server.py) — 创建服务上下文管理器
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
from .evaluator import (
|
||||
BaseEvaluator,
|
||||
EvalResult,
|
||||
)
|
||||
from .metrics import run_workspace_metrics
|
||||
from .opencompass import OpenCompassEvaluator
|
||||
from .server import create_grading_server
|
||||
from .utils import (
|
||||
append_result,
|
||||
detect_driver_model,
|
||||
download_data,
|
||||
download_model,
|
||||
ensure_symlink,
|
||||
get_baseline_score,
|
||||
init_run_meta,
|
||||
kill_process_group,
|
||||
print_summary,
|
||||
read_run_meta,
|
||||
set_baseline_to_server,
|
||||
setup_workspace,
|
||||
submit_to_grading_server,
|
||||
update_run_meta,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# 数据结构
|
||||
"EvalResult",
|
||||
# 评测器
|
||||
"BaseEvaluator",
|
||||
"OpenCompassEvaluator",
|
||||
# 服务
|
||||
"create_grading_server",
|
||||
# 工具函数
|
||||
"ensure_symlink",
|
||||
"download_model",
|
||||
"download_data",
|
||||
"get_baseline_score",
|
||||
"submit_to_grading_server",
|
||||
"set_baseline_to_server",
|
||||
# workspace & results
|
||||
"setup_workspace",
|
||||
"append_result",
|
||||
"detect_driver_model",
|
||||
"print_summary",
|
||||
"kill_process_group",
|
||||
"init_run_meta",
|
||||
"update_run_meta",
|
||||
"read_run_meta",
|
||||
"run_workspace_metrics",
|
||||
]
|
||||
@@ -0,0 +1,151 @@
|
||||
"""
|
||||
AutoRL-Bench Evaluator Base Class
|
||||
|
||||
所有 benchmark 评测器的基类，定义统一的评测接口。
|
||||
|
||||
开发新 benchmark 时，继承 BaseEvaluator 并实现 run_eval 方法。
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
from typing_extensions import NotRequired, TypedDict
|
||||
|
||||
# ============================================================
|
||||
# 数据结构定义（Schema）
|
||||
# ============================================================
|
||||
|
||||
|
||||
class EvalResult(TypedDict):
|
||||
"""
|
||||
评测输出结果
|
||||
|
||||
必须字段:
|
||||
benchmark: benchmark 名称
|
||||
model_path: 评测的模型路径
|
||||
score: 评测分数 (0-100)
|
||||
accuracy_summary: 详细指标字典
|
||||
|
||||
可选字段:
|
||||
eval_type: 评测类型 ("opencompass" / "alfworld" / ...)
|
||||
error: 错误信息（评测失败时）
|
||||
raw_output: 原始输出日志
|
||||
"""
|
||||
|
||||
# 必须字段
|
||||
benchmark: str
|
||||
model_path: str
|
||||
score: float
|
||||
accuracy_summary: Dict[str, Any]
|
||||
|
||||
# 可选字段
|
||||
eval_type: NotRequired[str]
|
||||
error: NotRequired[str]
|
||||
raw_output: NotRequired[str]
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 抽象基类
|
||||
# ============================================================
|
||||
|
||||
|
||||
class BaseEvaluator(ABC):
|
||||
"""
|
||||
Benchmark 评测器基类
|
||||
|
||||
所有自定义 benchmark 必须继承此类并实现 run_eval 方法。
|
||||
|
||||
=====================================================
|
||||
最简单的方式：调用 benchmark 自带的评测代码
|
||||
=====================================================
|
||||
|
||||
大多数 benchmark（如 HumanEval、MBPP、ALFWorld）都有官方评测脚本，
|
||||
只需要：
|
||||
1. 下载 benchmark repo
|
||||
2. 调用它的评测函数
|
||||
3. 把结果转成 EvalResult 格式
|
||||
|
||||
Example（包装现有评测）:
|
||||
class MyBenchmarkEvaluator(BaseEvaluator):
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
|
||||
def run_eval(self, model_path, workspace_path, **kwargs) -> EvalResult:
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
|
||||
# 1. 调用 benchmark 自带的评测
|
||||
from some_benchmark import evaluate # benchmark 官方库
|
||||
raw_result = evaluate(model_path) # 调用官方评测
|
||||
|
||||
# 2. 转换成统一格式
|
||||
result["score"] = raw_result["accuracy"] * 100
|
||||
result["accuracy_summary"] = raw_result
|
||||
return result
|
||||
|
||||
=====================================================
|
||||
完整示例：自定义评测逻辑
|
||||
=====================================================
|
||||
|
||||
如果需要完全自定义评测（如交互式环境）：
|
||||
|
||||
Example:
|
||||
class InteractiveEvaluator(BaseEvaluator):
|
||||
def run_eval(self, model_path, workspace_path, **kwargs) -> EvalResult:
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
|
||||
# 1. 加载模型
|
||||
model = load_model(model_path)
|
||||
|
||||
# 2. 运行评测循环
|
||||
success = 0
|
||||
for task in tasks:
|
||||
output = model.generate(task.prompt)
|
||||
if task.check(output):
|
||||
success += 1
|
||||
|
||||
# 3. 返回结果
|
||||
result["score"] = success / len(tasks) * 100
|
||||
result["accuracy_summary"] = {"success": success, "total": len(tasks)}
|
||||
return result
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
model_name: str = "",
|
||||
gpu_count: int = 1,
|
||||
test_range: str = "[:]",
|
||||
**kwargs,
|
||||
) -> EvalResult:
|
||||
"""
|
||||
执行评测
|
||||
|
||||
Args:
|
||||
model_path: 训练后的模型路径（本地目录）
|
||||
workspace_path: 工作目录路径
|
||||
model_name: 模型名称（用于配置推理参数）
|
||||
gpu_count: 可用 GPU 数量
|
||||
test_range: 测试数据范围，如 "[:]" 或 "[:100]"
|
||||
**kwargs: 其他评测参数
|
||||
|
||||
Returns:
|
||||
EvalResult: 评测结果
|
||||
"""
|
||||
pass
|
||||
|
||||
def validate_model(self, model_path: str) -> bool:
|
||||
"""验证模型路径是否有效"""
|
||||
return Path(model_path).exists()
|
||||
|
||||
def get_default_result(self, benchmark_name: str, model_path: str) -> EvalResult:
|
||||
"""返回默认的结果结构"""
|
||||
return {
|
||||
"benchmark": benchmark_name,
|
||||
"model_path": model_path,
|
||||
"score": 0.0,
|
||||
"accuracy_summary": {},
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
# AutoRL-Bench 任务说明
|
||||
|
||||
你是一个强化学习训练 Agent，目标是通过 RL Post-Training 提升模型表现。
|
||||
|
||||
**核心目标**：在固定时间内（默认 12 小时）尽可能提高分数，可多次提交并利用反馈迭代。
|
||||
|
||||
## 关键信息（先读）
|
||||
- **工作区限制**：当前目录就是工作区，只用相对路径，禁止 `cd` 到外部
|
||||
- **时间单一事实源**：优先读取 `./run_meta.json`
|
||||
- **评测入口**：`POST $GRADING_SERVER_URL/submit`
|
||||
|
||||
## 环境变量
|
||||
- TASK: 任务名称
|
||||
- BASE_MODEL: 基础模型名称
|
||||
- MODEL_PATH: 基础模型路径（只读）
|
||||
- DATA_PATH: 训练数据路径（只读）
|
||||
- OUTPUT_DIR: 模型输出目录（提交评测时指定此目录下的模型路径）
|
||||
- GRADING_SERVER_URL: 评测服务地址
|
||||
|
||||
## 时间与预算信号（单一事实源）
|
||||
默认预算 **12 小时（43200 秒）**，以 `run_meta.json` 为准。
|
||||
|
||||
`run_meta.json` 字段：
|
||||
- start_time: 任务开始时间（unix timestamp, 秒）
|
||||
- timeout_s: 总时长上限（秒）
|
||||
- last_submit_time: 最后一次提交时间（unix timestamp, 秒）
|
||||
- end_time: 任务结束时间（unix timestamp, 秒）
|
||||
|
||||
可选 API：
|
||||
- GET $GRADING_SERVER_URL/time
|
||||
- 返回字段：`start_time / timeout_s / last_submit_time / end_time / now / remaining`
|
||||
|
||||
## 工作区与目录
|
||||
**你的当前目录就是工作区。所有需要的文件都在当前目录下。**
|
||||
- **禁止 `cd` 到当前目录之外**（不要访问父目录或其他路径）
|
||||
- **只使用相对路径**（如 `./code/train.py`，而非绝对路径）
|
||||
- 如果看到 symlink 指向外部路径，忽略它——直接用相对路径访问即可
|
||||
|
||||
目录结构：
|
||||
```
|
||||
./
|
||||
├── code/ # 你的代码区（所有自行编写的代码放在此处）
|
||||
├── data/ # 训练数据（只读）
|
||||
├── models/ # 基础模型（只读）
|
||||
├── output/ # 模型输出（训练好的模型保存在此）
|
||||
├── description.md # 任务描述（必读）
|
||||
├── instructions.md # 本文件
|
||||
├── run_meta.json # 时间与预算信号（单一事实源）
|
||||
└── ... # benchmark 特有文件（用 ls 查看完整列表）
|
||||
```
|
||||
|
||||
**先 `ls` 查看当前目录所有可用文件。** 不同类型的 benchmark 会提供不同的额外文件：
|
||||
- **交互式环境类**（如 ALFWorld）：会提供 `eval.py`（环境交互 + 评测逻辑）、prompt 模板、配置文件等——这些是编写训练代码的关键参考
|
||||
- **静态数据集类**（如 GSM8K）：主要通过 `data/` 下的数据文件提供训练样本
|
||||
|
||||
**说明**：
|
||||
- `code/`：在此编写代码（文件名和结构自由组织）
|
||||
- `output/`：训练产出的模型存放处。可存放多个版本（如 `output/v1/`、`output/v2/`），提交时指定具体路径
|
||||
|
||||
## 任务流程（固定时间内刷分）
|
||||
1. 探索工作区，阅读 `description.md`、`instructions.md` 和相关文件（如有 `eval.py` 务必仔细阅读），了解任务目标和可用资源
|
||||
2. 在 `code/` 下编写代码，训练模型（SFT、GRPO、PPO 等均可）
|
||||
3. 保存模型到 $OUTPUT_DIR（如 `output/v1`）
|
||||
4. 提交评测：POST $GRADING_SERVER_URL/submit
|
||||
5. 根据返回的 score 调整策略，**在剩余时间内持续迭代并提交更高分模型**
|
||||
|
||||
## API
|
||||
```bash
|
||||
# 提交评测（指定模型路径，返回 score、improvement、best）
|
||||
curl -X POST "$GRADING_SERVER_URL/submit" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model_path": "'$OUTPUT_DIR'/v1"}'
|
||||
|
||||
# 指定 GPU 评测（可选，默认使用 GPU 0）
|
||||
curl -X POST "$GRADING_SERVER_URL/submit" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model_path": "'$OUTPUT_DIR'/v1", "gpu": "0"}'
|
||||
|
||||
# 多卡评测
|
||||
curl -X POST "$GRADING_SERVER_URL/submit" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model_path": "'$OUTPUT_DIR'/v1", "gpu": "2,3"}'
|
||||
|
||||
# 查询时间与预算（优先使用 run_meta.json；此 API 仅做补充）
|
||||
curl "$GRADING_SERVER_URL/time"
|
||||
|
||||
# 健康检查（返回可用 GPU 列表等信息）
|
||||
curl "$GRADING_SERVER_URL/health"
|
||||
```
|
||||
|
||||
### /submit 参数
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| model_path | string | 是 | 模型路径 |
|
||||
| gpu | string | 否 | 指定 GPU（如 "0"、"1"、"0,1"），必须是可用 GPU 之一。不传则默认使用第一个可用 GPU。可通过 /health 查看可用列表 |
|
||||
|
||||
### /submit 响应示例
|
||||
```json
|
||||
{
|
||||
"submission_id": 3,
|
||||
"score": 65.0,
|
||||
"baseline_score": 45.0,
|
||||
"improvement": 20.0,
|
||||
"best": {"submission_id": 2, "score": 68.0},
|
||||
"total_submissions": 3
|
||||
}
|
||||
```
|
||||
|
||||
## 注意
|
||||
- **不要直接复制/symlink 基座模型提交**——未经训练的基座模型只会得到 baseline 分数（improvement = 0），这是浪费提交机会。必须先训练再提交。
|
||||
- 可多次提交不同版本的模型，系统自动跟踪最高分
|
||||
- 合理利用时间，根据 score 反馈迭代优化
|
||||
- **必须提交完整模型**：评测系统不支持 LoRA adapter 目录。如果用 LoRA/PEFT 训练，保存前必须合并：`model = model.merge_and_unload(); model.save_pretrained(output_path); tokenizer.save_pretrained(output_path)`
|
||||
- trl 保存模型后，`tokenizer_config.json` 中的 `extra_special_tokens` 会被保存为 list 格式，但 vLLM/transformers 加载时需要 dict 格式。保存模型后需删除该字段，否则评测会失败。
|
||||
@@ -0,0 +1,156 @@
|
||||
"""
|
||||
AutoRL-Bench Metrics
|
||||
|
||||
计算 run 级别过程指标，并生成可视化。
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.core.utils import read_run_meta
|
||||
|
||||
|
||||
def _parse_iso_time(value: str) -> Optional[datetime]:
|
||||
try:
|
||||
return datetime.fromisoformat(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _safe_div(numerator: float, denominator: float) -> Optional[float]:
|
||||
if denominator == 0:
|
||||
return None
|
||||
return numerator / denominator
|
||||
|
||||
|
||||
def compute_metrics(
|
||||
workspace: Path,
|
||||
baseline: Optional[float],
|
||||
base_model_path: Optional[str],
|
||||
) -> dict[str, Any]:
|
||||
scores_file = workspace / "scores.json"
|
||||
scores = json.loads(scores_file.read_text()) if scores_file.exists() else []
|
||||
score_values = [entry.get("score", 0.0) for entry in scores]
|
||||
|
||||
valid_scores = [s for s in score_values if s and s > 0]
|
||||
valid_submission_rate = _safe_div(len(valid_scores), len(score_values))
|
||||
|
||||
first_valid_idx = None
|
||||
for idx, score in enumerate(score_values, start=1):
|
||||
if score and score > 0:
|
||||
first_valid_idx = idx
|
||||
break
|
||||
|
||||
run_meta = read_run_meta(workspace)
|
||||
start_time = run_meta.get("start_time")
|
||||
end_time = run_meta.get("end_time")
|
||||
timeout_s = run_meta.get("timeout_s")
|
||||
last_submit_time = run_meta.get("last_submit_time")
|
||||
|
||||
first_valid_delay = None
|
||||
if start_time and first_valid_idx:
|
||||
first_time = _parse_iso_time(scores[first_valid_idx - 1].get("timestamp"))
|
||||
if first_time:
|
||||
first_valid_delay = int(first_time.timestamp() - start_time)
|
||||
|
||||
time_to_first_improvement = None
|
||||
if baseline is not None and start_time:
|
||||
for entry in scores:
|
||||
if entry.get("score", 0.0) > baseline:
|
||||
ts = _parse_iso_time(entry.get("timestamp"))
|
||||
if ts:
|
||||
time_to_first_improvement = int(ts.timestamp() - start_time)
|
||||
break
|
||||
|
||||
monotonic_ratio = None
|
||||
if len(score_values) >= 2:
|
||||
increases = sum(1 for prev, cur in zip(score_values, score_values[1:]) if cur > prev)
|
||||
monotonic_ratio = _safe_div(increases, len(score_values) - 1)
|
||||
|
||||
copy_model_count = None
|
||||
if base_model_path:
|
||||
base_path = Path(base_model_path).resolve()
|
||||
copy_model_count = 0
|
||||
for entry in scores:
|
||||
model_path = entry.get("model_path")
|
||||
if model_path and Path(model_path).resolve() == base_path:
|
||||
copy_model_count += 1
|
||||
|
||||
time_used_ratio = None
|
||||
if start_time and end_time and timeout_s:
|
||||
time_used_ratio = _safe_div(end_time - start_time, timeout_s)
|
||||
|
||||
time_to_best = None
|
||||
if start_time and scores:
|
||||
best_entry = max(scores, key=lambda x: x.get("score", 0.0))
|
||||
best_ts = _parse_iso_time(best_entry.get("timestamp"))
|
||||
if best_ts:
|
||||
time_to_best = int(best_ts.timestamp() - start_time)
|
||||
|
||||
last_submit_gap = None
|
||||
if end_time and last_submit_time:
|
||||
last_submit_gap = int(end_time - last_submit_time)
|
||||
|
||||
return {
|
||||
"valid_submission_rate": valid_submission_rate,
|
||||
"first_valid_idx": first_valid_idx,
|
||||
"first_valid_delay": first_valid_delay,
|
||||
"score_trajectory": score_values,
|
||||
"time_to_first_improvement": time_to_first_improvement,
|
||||
"time_to_best": time_to_best,
|
||||
"monotonic_ratio": monotonic_ratio,
|
||||
"copy_model_count": copy_model_count,
|
||||
"time_used_ratio": time_used_ratio,
|
||||
"last_submit_gap": last_submit_gap,
|
||||
}
|
||||
|
||||
|
||||
def write_metrics_json(workspace: Path, metrics: dict[str, Any]) -> Path:
|
||||
reports_dir = workspace / "reports"
|
||||
reports_dir.mkdir(exist_ok=True)
|
||||
target = reports_dir / "metrics.json"
|
||||
target.write_text(json.dumps(metrics, indent=2, ensure_ascii=False))
|
||||
return target
|
||||
|
||||
|
||||
def plot_score_trajectory(workspace: Path, metrics: dict[str, Any]) -> Optional[Path]:
|
||||
try:
|
||||
import matplotlib.pyplot as plt
|
||||
except ImportError:
|
||||
return None
|
||||
scores = metrics.get("score_trajectory", [])
|
||||
reports_dir = workspace / "reports"
|
||||
figures_dir = reports_dir / "figures"
|
||||
figures_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = figures_dir / "score_trajectory.png"
|
||||
|
||||
plt.figure(figsize=(6, 4))
|
||||
if scores:
|
||||
plt.plot(list(range(1, len(scores) + 1)), scores, marker="o")
|
||||
plt.xlabel("submission")
|
||||
plt.ylabel("score")
|
||||
plt.title("score trajectory")
|
||||
else:
|
||||
plt.text(0.5, 0.5, "no submissions", ha="center", va="center")
|
||||
plt.axis("off")
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig(target)
|
||||
plt.close()
|
||||
return target
|
||||
|
||||
|
||||
def run_workspace_metrics(
|
||||
workspace: Path,
|
||||
baseline: Optional[float],
|
||||
base_model_path: Optional[str],
|
||||
*,
|
||||
plot: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
metrics = compute_metrics(workspace, baseline, base_model_path)
|
||||
write_metrics_json(workspace, metrics)
|
||||
if plot:
|
||||
plot_score_trajectory(workspace, metrics)
|
||||
return metrics
|
||||
@@ -0,0 +1,204 @@
|
||||
"""
|
||||
OpenCompass Evaluator
|
||||
|
||||
用于所有使用 OpenCompass 评测的 benchmark（gsm8k, math 等）。
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
from rdagent.components.benchmark import BENCHMARK_CONFIGS_DIR
|
||||
from rdagent.components.benchmark.utils import build_dataset_imports_explicit
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.evaluator import BaseEvaluator
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class OpenCompassEvaluator(BaseEvaluator):
|
||||
"""
|
||||
OpenCompass 通用评测器
|
||||
|
||||
适用于所有使用 OpenCompass 评测的 benchmark。
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.benchmark_id = config.id
|
||||
self.eval_config = config.eval_config or {}
|
||||
|
||||
def run_eval(
|
||||
self,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
model_name: str = "",
|
||||
gpu_count: int = 1,
|
||||
test_range: str = "[:]",
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
"""使用 OpenCompass 评测"""
|
||||
result = self.get_default_result(self.benchmark_id, model_path)
|
||||
result["eval_type"] = "opencompass"
|
||||
|
||||
if not self.validate_model(model_path):
|
||||
result["error"] = f"Model not found: {model_path}"
|
||||
return result
|
||||
|
||||
workspace = Path(workspace_path)
|
||||
model_path = str(Path(model_path).resolve())
|
||||
work_dir = workspace / "benchmark_results"
|
||||
work_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 获取评测配置
|
||||
dataset_import = self.eval_config.get("dataset", f"opencompass.configs.datasets.{self.benchmark_id}")
|
||||
# 允许 benchmark 在配置中声明默认评测切片（例如 HumanEval 仅评后半）
|
||||
effective_test_range = test_range
|
||||
if test_range == "[:]" and self.eval_config.get("test_range"):
|
||||
effective_test_range = self.eval_config["test_range"]
|
||||
|
||||
# 从 models.yaml 获取模型推理配置
|
||||
inference_config = self._get_model_inference_config(model_name, gpu_count)
|
||||
|
||||
dataset_imports_explicit = build_dataset_imports_explicit(dataset_import)
|
||||
|
||||
# B1 fix: 拒绝 LoRA adapter，提示 agent 合并后再提交
|
||||
adapter_cfg_file = Path(model_path) / "adapter_config.json"
|
||||
if adapter_cfg_file.exists():
|
||||
result["error"] = (
|
||||
"LoRA adapter detected — the evaluation system requires a full merged model. "
|
||||
"Please merge before saving: "
|
||||
"model = model.merge_and_unload(); "
|
||||
"model.save_pretrained(output_path); "
|
||||
"tokenizer.save_pretrained(output_path)"
|
||||
)
|
||||
return result
|
||||
|
||||
# 生成 OpenCompass 配置
|
||||
template_vars = {
|
||||
"model_abbr": f"rl-{self.benchmark_id}",
|
||||
"model_path": model_path,
|
||||
"dataset_imports": dataset_imports_explicit,
|
||||
"test_range": effective_test_range,
|
||||
"num_runs": 1,
|
||||
"pass_k": None,
|
||||
"work_dir": str(work_dir),
|
||||
**inference_config,
|
||||
}
|
||||
|
||||
config_content = T("rdagent.components.benchmark.configs.opencompass_template:template").r(**template_vars)
|
||||
config_path = workspace / "opencompass_config.py"
|
||||
config_path.write_text(config_content)
|
||||
|
||||
logger.info(f"Running OpenCompass benchmark: {self.benchmark_id}")
|
||||
logger.info(f"Model: {model_path}")
|
||||
|
||||
# 运行 OpenCompass
|
||||
cmd = ["opencompass", str(config_path), "--work-dir", str(work_dir)]
|
||||
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=7200)
|
||||
except subprocess.TimeoutExpired:
|
||||
result["error"] = "OpenCompass timeout (7200s)"
|
||||
return result
|
||||
|
||||
if proc.returncode != 0:
|
||||
error_msg = proc.stderr[:1000] if proc.stderr else proc.stdout[:1000] if proc.stdout else "No output"
|
||||
logger.warning(f"OpenCompass failed: {error_msg}")
|
||||
result["error"] = f"OpenCompass exit code: {proc.returncode}"
|
||||
result["raw_output"] = error_msg
|
||||
return result
|
||||
|
||||
# 解析结果
|
||||
result = self._parse_results(work_dir, result)
|
||||
logger.info(f"Benchmark score: {result['score']}")
|
||||
return result
|
||||
|
||||
def _get_model_inference_config(self, model_name: str, gpu_count: int) -> dict:
|
||||
"""从 models.yaml 加载模型推理配置"""
|
||||
config_data = yaml.safe_load(open(BENCHMARK_CONFIGS_DIR / "models.yaml", "r"))
|
||||
|
||||
default_config = config_data.get("default", {})
|
||||
models_config = config_data.get("models", {})
|
||||
|
||||
model_specific = models_config.get(model_name, {})
|
||||
if not model_specific:
|
||||
best_match_len = 5
|
||||
for configured_model in models_config:
|
||||
if model_name.startswith(configured_model) and len(configured_model) > best_match_len:
|
||||
model_specific = models_config[configured_model]
|
||||
best_match_len = len(configured_model)
|
||||
|
||||
final_config = {**default_config, **model_specific}
|
||||
|
||||
# 处理 auto tensor_parallel_size
|
||||
if final_config.get("tensor_parallel_size") == "auto":
|
||||
if gpu_count <= 0:
|
||||
final_config["tensor_parallel_size"] = 1
|
||||
else:
|
||||
power = 0
|
||||
while (1 << (power + 1)) <= gpu_count:
|
||||
power += 1
|
||||
final_config["tensor_parallel_size"] = 1 << power
|
||||
|
||||
return final_config
|
||||
|
||||
def _parse_results(self, work_dir: Path, result: dict) -> dict:
|
||||
"""解析 OpenCompass 输出结果"""
|
||||
timestamped_dirs = sorted([d for d in work_dir.glob("202*_*") if d.is_dir()], reverse=True)
|
||||
|
||||
if not timestamped_dirs:
|
||||
result["error"] = "No results directory found"
|
||||
return result
|
||||
|
||||
summary_dir = timestamped_dirs[0] / "summary"
|
||||
csv_files = list(summary_dir.rglob("*.csv"))
|
||||
|
||||
if not csv_files:
|
||||
result["error"] = "No results CSV found"
|
||||
return result
|
||||
|
||||
df = pd.read_csv(csv_files[0])
|
||||
score_col = [c for c in df.columns if c not in ["dataset", "version", "metric", "mode"]]
|
||||
|
||||
if not score_col:
|
||||
return result
|
||||
|
||||
col = score_col[0]
|
||||
|
||||
# If CSV has a 'metric' column, pick only the primary metric rows
|
||||
# (avoids averaging in pass/timeout/failed counters)
|
||||
if "metric" in df.columns:
|
||||
for m in ("accuracy", "score"):
|
||||
rows = df[df["metric"] == m]
|
||||
if not rows.empty:
|
||||
vals = []
|
||||
for raw in rows[col].dropna().values:
|
||||
try:
|
||||
vals.append(float(raw))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
if vals:
|
||||
result["score"] = sum(vals) / len(vals)
|
||||
result["accuracy_summary"] = {"accuracy": result["score"], "num_subdatasets": len(vals)}
|
||||
return result
|
||||
|
||||
# Fallback: take the first numeric value
|
||||
non_numeric_values = []
|
||||
for raw in df[col].dropna().values:
|
||||
try:
|
||||
result["score"] = float(raw)
|
||||
result["accuracy_summary"] = {"accuracy": result["score"], "num_subdatasets": 1}
|
||||
return result
|
||||
except (ValueError, TypeError):
|
||||
non_numeric_values.append(str(raw))
|
||||
logger.warning(f"OpenCompass returned non-numeric score: {raw!r}, skipping")
|
||||
|
||||
if non_numeric_values:
|
||||
result["error"] = (
|
||||
f"Evaluation failed: OpenCompass returned non-numeric scores {non_numeric_values}. This usually means vLLM failed to load the model (missing config.json, GPU OOM, or engine crash)."
|
||||
)
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,449 @@
|
||||
"""
|
||||
AutoRL-Bench Grading Server (Simplified)
|
||||
|
||||
精简的评测服务，主要提供 submit 接口。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Set
|
||||
|
||||
import requests
|
||||
from flask import Flask, jsonify, request
|
||||
from werkzeug.serving import make_server
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.core.utils import read_run_meta, update_run_meta
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
|
||||
def _get_available_gpus() -> Set[str]:
|
||||
"""从 CUDA_VISIBLE_DEVICES 获取可用 GPU 集合"""
|
||||
cuda_env = os.environ.get("CUDA_VISIBLE_DEVICES", "")
|
||||
if not cuda_env.strip():
|
||||
return set()
|
||||
return {g.strip() for g in cuda_env.split(",") if g.strip()}
|
||||
|
||||
|
||||
def _validate_gpu(gpu: str, available: Set[str]) -> Optional[str]:
|
||||
"""校验 gpu 参数，返回错误信息或 None（合法）"""
|
||||
requested = {g.strip() for g in gpu.split(",") if g.strip()}
|
||||
if not requested:
|
||||
return "gpu parameter is empty"
|
||||
invalid = requested - available
|
||||
if invalid:
|
||||
return f"GPU {invalid} not in available GPUs {sorted(available)} (from CUDA_VISIBLE_DEVICES)"
|
||||
return None
|
||||
|
||||
|
||||
class GradingServer:
|
||||
"""评测服务器"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
task: str,
|
||||
base_model: str,
|
||||
workspace: Path,
|
||||
):
|
||||
self.task = task
|
||||
self.base_model = base_model
|
||||
self.workspace = Path(workspace)
|
||||
self.scores_file = self.workspace / "scores.json"
|
||||
self.baseline_score: Optional[float] = None
|
||||
self.available_gpus: Set[str] = _get_available_gpus()
|
||||
self._eval_lock = threading.Lock()
|
||||
self._eval_cache: dict[str, dict] = {}
|
||||
|
||||
@staticmethod
|
||||
def _make_cache_key(resolved_path: Path) -> str:
|
||||
"""用路径 + safetensors/bin 文件最新 mtime 组合作为 cache key。
|
||||
模型被覆盖后 mtime 变化，cache 自动失效。"""
|
||||
mtime = 0.0
|
||||
if resolved_path.is_dir():
|
||||
for f in resolved_path.rglob("*"):
|
||||
if f.suffix in (".safetensors", ".bin", ".json") and f.is_file():
|
||||
mt = f.stat().st_mtime
|
||||
if mt > mtime:
|
||||
mtime = mt
|
||||
elif resolved_path.is_file():
|
||||
mtime = resolved_path.stat().st_mtime
|
||||
return f"{resolved_path}@{mtime}"
|
||||
|
||||
def load_scores(self) -> list[dict]:
|
||||
if self.scores_file.exists():
|
||||
return json.loads(self.scores_file.read_text())
|
||||
return []
|
||||
|
||||
def save_scores(self, scores: list[dict]):
|
||||
self.scores_file.write_text(json.dumps(scores, indent=2, ensure_ascii=False))
|
||||
|
||||
def get_evaluator(self):
|
||||
"""获取当前 task 的评测器"""
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import get_evaluator
|
||||
|
||||
return get_evaluator(self.task)
|
||||
|
||||
def resolve_model_path(self, model_path: str) -> Path:
|
||||
"""将模型路径约束在 workspace 下，防止访问任意文件系统路径。"""
|
||||
if "\x00" in model_path:
|
||||
raise ValueError("Invalid model_path")
|
||||
|
||||
workspace_root = self.workspace.expanduser().resolve()
|
||||
normalized = os.path.normpath(model_path)
|
||||
if os.path.splitdrive(normalized)[0]:
|
||||
raise ValueError("Invalid model_path")
|
||||
|
||||
if os.path.isabs(normalized):
|
||||
candidate = normalized
|
||||
else:
|
||||
candidate = os.path.join(str(workspace_root), normalized)
|
||||
|
||||
resolved_path = Path(candidate).expanduser().resolve(strict=False)
|
||||
try:
|
||||
resolved_path.relative_to(workspace_root)
|
||||
except ValueError as exc:
|
||||
raise ValueError("Invalid model_path") from exc
|
||||
return resolved_path
|
||||
|
||||
def submit(self, model_path: str, gpu: Optional[str] = None) -> dict:
|
||||
"""
|
||||
提交模型评测
|
||||
|
||||
Args:
|
||||
model_path: 模型路径
|
||||
gpu: 指定 GPU（如 "0", "1", "0,1"），必须是 CUDA_VISIBLE_DEVICES 中的子集。
|
||||
None 则使用 CUDA_VISIBLE_DEVICES 中的第一个 GPU。
|
||||
|
||||
Returns:
|
||||
包含 score、best、improvement 等完整信息的结果
|
||||
|
||||
Raises:
|
||||
ValueError: gpu 不在 CUDA_VISIBLE_DEVICES 范围内，或 model_path 非法
|
||||
"""
|
||||
if self.available_gpus:
|
||||
if gpu is None:
|
||||
gpu = sorted(self.available_gpus, key=int)[0]
|
||||
else:
|
||||
err = _validate_gpu(gpu, self.available_gpus)
|
||||
if err:
|
||||
raise ValueError(err)
|
||||
|
||||
# B3 fix: 同一 model_path + 同一内容去重，直接返回缓存结果
|
||||
# 用路径 + 模型文件最新 mtime 作为 cache key，模型文件被覆盖后自动失效
|
||||
resolved_path = self.resolve_model_path(model_path)
|
||||
cache_key = self._make_cache_key(resolved_path)
|
||||
if cache_key in self._eval_cache:
|
||||
cached = self._eval_cache[cache_key]
|
||||
logger.info(f"[SUBMIT] Cache hit for {model_path}, score={cached.get('score')}")
|
||||
return cached
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# B2 fix: 串行化评测，防止多个 vLLM 实例同时抢 GPU
|
||||
with self._eval_lock:
|
||||
# Double-check: 等锁期间可能已被其他线程评完
|
||||
cache_key = self._make_cache_key(resolved_path)
|
||||
if cache_key in self._eval_cache:
|
||||
cached = self._eval_cache[cache_key]
|
||||
logger.info(f"[SUBMIT] Cache hit (after lock) for {model_path}, score={cached.get('score')}")
|
||||
return cached
|
||||
|
||||
scores = self.load_scores()
|
||||
submission_id = len(scores) + 1
|
||||
|
||||
logger.info(f"[SUBMIT #{submission_id}] Started | model_path={model_path} | gpu={gpu}")
|
||||
|
||||
old_cuda = os.environ.get("CUDA_VISIBLE_DEVICES")
|
||||
if gpu is not None:
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu)
|
||||
|
||||
try:
|
||||
evaluator = self.get_evaluator()
|
||||
gpu_count = len(self.available_gpus) if self.available_gpus else 1
|
||||
result = evaluator.run_eval(
|
||||
model_path=str(resolved_path),
|
||||
workspace_path=str(self.workspace),
|
||||
model_name=self.base_model,
|
||||
gpu_count=gpu_count,
|
||||
)
|
||||
finally:
|
||||
if old_cuda is not None:
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = old_cuda
|
||||
elif "CUDA_VISIBLE_DEVICES" in os.environ:
|
||||
del os.environ["CUDA_VISIBLE_DEVICES"]
|
||||
|
||||
elapsed_seconds = time.time() - start_time
|
||||
|
||||
# 解析分数
|
||||
score = result.get("score", 0.0)
|
||||
error = result.get("error")
|
||||
|
||||
# 计算 improvement
|
||||
improvement = None
|
||||
if self.baseline_score is not None:
|
||||
improvement = round(score - self.baseline_score, 6)
|
||||
|
||||
# 构建结果
|
||||
entry = {
|
||||
"submission_id": submission_id,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"model_path": model_path,
|
||||
"score": score,
|
||||
"baseline_score": self.baseline_score,
|
||||
"improvement": improvement,
|
||||
"elapsed_seconds": round(elapsed_seconds, 2),
|
||||
}
|
||||
# B4 fix: 透传 error 字段
|
||||
if error:
|
||||
entry["error"] = error
|
||||
|
||||
scores.append(entry)
|
||||
self.save_scores(scores)
|
||||
update_run_meta(self.workspace, last_submit_time=int(time.time()))
|
||||
|
||||
# 查找最高分
|
||||
best_entry = max(scores, key=lambda x: x.get("score", 0))
|
||||
|
||||
logger.info(f"[SUBMIT #{submission_id}] Done | score={score}, best={best_entry['score']}")
|
||||
|
||||
response = {
|
||||
**entry,
|
||||
"best": best_entry,
|
||||
"total_submissions": len(scores),
|
||||
}
|
||||
|
||||
# 只缓存成功的评测结果（失败的不缓存，允许重试）
|
||||
if not error:
|
||||
self._eval_cache[self._make_cache_key(resolved_path)] = response
|
||||
|
||||
return response
|
||||
|
||||
def set_baseline(self, score: float):
|
||||
"""设置 baseline 分数"""
|
||||
self.baseline_score = score
|
||||
logger.info(f"[BASELINE] Set to {score}")
|
||||
|
||||
|
||||
# 全局服务器实例
|
||||
_server: Optional[GradingServer] = None
|
||||
|
||||
|
||||
def get_server() -> GradingServer:
|
||||
global _server
|
||||
if _server is None:
|
||||
raise RuntimeError("Server not initialized. Call init_server() first.")
|
||||
return _server
|
||||
|
||||
|
||||
def init_server(task: str, base_model: str, workspace: str) -> GradingServer:
|
||||
"""初始化服务器"""
|
||||
global _server
|
||||
_server = GradingServer(task, base_model, Path(workspace))
|
||||
return _server
|
||||
|
||||
|
||||
# Flask 路由
|
||||
@app.route("/submit", methods=["POST"])
|
||||
def submit():
|
||||
"""
|
||||
提交模型评测
|
||||
|
||||
Request:
|
||||
{"model_path": "/path/to/model"}
|
||||
|
||||
Response:
|
||||
{
|
||||
"submission_id": 1,
|
||||
"score": 85.0,
|
||||
"improvement": 5.0,
|
||||
"best": {...},
|
||||
"total_submissions": 10
|
||||
}
|
||||
"""
|
||||
data = request.get_json() or {}
|
||||
model_path = data.get("model_path")
|
||||
gpu = data.get("gpu")
|
||||
|
||||
if not model_path:
|
||||
return jsonify({"error": "Missing model_path"}), 400
|
||||
|
||||
server = get_server()
|
||||
if gpu is not None:
|
||||
gpu = str(gpu)
|
||||
err = _validate_gpu(gpu, server.available_gpus)
|
||||
if err:
|
||||
return (
|
||||
jsonify(
|
||||
{
|
||||
"error": err,
|
||||
"available_gpus": sorted(server.available_gpus, key=int),
|
||||
}
|
||||
),
|
||||
400,
|
||||
)
|
||||
|
||||
try:
|
||||
result = server.submit(model_path, gpu=gpu)
|
||||
return jsonify(result)
|
||||
except ValueError:
|
||||
logger.warning("[SUBMIT] Invalid request", exc_info=True)
|
||||
return jsonify({"error": "Invalid request"}), 400
|
||||
except (RuntimeError, OSError):
|
||||
logger.exception("[SUBMIT] Internal server error")
|
||||
return jsonify({"error": "Internal server error"}), 500
|
||||
|
||||
|
||||
@app.route("/health", methods=["GET"])
|
||||
def health():
|
||||
"""健康检查"""
|
||||
server = get_server()
|
||||
return jsonify(
|
||||
{
|
||||
"status": "ok",
|
||||
"task": server.task,
|
||||
"workspace": str(server.workspace),
|
||||
"available_gpus": sorted(server.available_gpus, key=int) if server.available_gpus else [],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@app.route("/time", methods=["GET"])
|
||||
def time_status():
|
||||
"""时间与预算信号"""
|
||||
server = get_server()
|
||||
meta = read_run_meta(server.workspace)
|
||||
now = int(time.time())
|
||||
timeout_s = meta.get("timeout_s")
|
||||
start_time = meta.get("start_time")
|
||||
remaining = None
|
||||
if isinstance(timeout_s, int) and isinstance(start_time, int):
|
||||
remaining = max(timeout_s - (now - start_time), 0)
|
||||
return jsonify(
|
||||
{
|
||||
**meta,
|
||||
"now": now,
|
||||
"remaining": remaining,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@app.route("/set_baseline", methods=["POST"])
|
||||
def set_baseline():
|
||||
"""设置 baseline 分数"""
|
||||
data = request.get_json() or {}
|
||||
score = data.get("score")
|
||||
|
||||
if score is None:
|
||||
return jsonify({"error": "Missing score"}), 400
|
||||
|
||||
server = get_server()
|
||||
server.set_baseline(float(score))
|
||||
return jsonify({"baseline_score": score, "status": "set"})
|
||||
|
||||
|
||||
def run_server(task: str, base_model: str, workspace: str, host: str = "0.0.0.0", port: int = 5000):
|
||||
"""启动服务器"""
|
||||
init_server(task, base_model, workspace)
|
||||
logger.info(f"Grading Server | task={task} | {host}:{port}")
|
||||
app.run(host=host, port=port, debug=False, threaded=True)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Grading Server 上下文管理器
|
||||
# ============================================================
|
||||
|
||||
|
||||
class GradingServerContext:
|
||||
"""Grading Server 基类"""
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
pass
|
||||
|
||||
def get_baseline(self, task: str, model_name: str, model_path: str, workspace_path: str) -> float:
|
||||
raise NotImplementedError
|
||||
|
||||
def load_scores(self) -> list:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class LocalServerContext(GradingServerContext):
|
||||
"""本地 Flask Server"""
|
||||
|
||||
def __init__(self, task: str, base_model: str, workspace: str, port: int):
|
||||
self.task = task
|
||||
self.base_model = base_model
|
||||
self.workspace = workspace
|
||||
self.port = port
|
||||
self.server = None
|
||||
self._http_server = None
|
||||
self._thread = None
|
||||
|
||||
def __enter__(self):
|
||||
logger.info(f"[Local Mode] Starting evaluation server on port {self.port}...")
|
||||
self.server = init_server(self.task, self.base_model, self.workspace)
|
||||
|
||||
self._http_server = make_server("0.0.0.0", self.port, app, threaded=True)
|
||||
self._thread = threading.Thread(target=self._http_server.serve_forever, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
# Poll /health for up to 15 seconds instead of blind sleep(2)
|
||||
deadline = time.time() + 15
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
resp = requests.get(f"http://localhost:{self.port}/health", timeout=2)
|
||||
if resp.status_code == 200:
|
||||
break
|
||||
except requests.ConnectionError:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise RuntimeError(f"Grading server failed to start on port {self.port}")
|
||||
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
if self._http_server:
|
||||
self._http_server.shutdown()
|
||||
self._http_server = None
|
||||
|
||||
def get_baseline(self, task: str, model_name: str, model_path: str, workspace_path: str) -> float:
|
||||
from rdagent.scenarios.rl.autorl_bench.core.utils import get_baseline_score
|
||||
|
||||
baseline = get_baseline_score(task, model_name, model_path, workspace_path)
|
||||
self.server.set_baseline(baseline)
|
||||
return baseline
|
||||
|
||||
def load_scores(self) -> list:
|
||||
return self.server.load_scores() if self.server else []
|
||||
|
||||
|
||||
def create_grading_server(benchmark, workspace: Path, port: int, base_model: str) -> GradingServerContext:
|
||||
"""创建 Grading Server 上下文"""
|
||||
return LocalServerContext(
|
||||
task=benchmark.id,
|
||||
base_model=base_model,
|
||||
workspace=str(workspace),
|
||||
port=port,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--task", type=str, required=True)
|
||||
parser.add_argument("--base-model", type=str, default="")
|
||||
parser.add_argument("--workspace", type=str, default=".")
|
||||
parser.add_argument("--port", type=int, default=5000)
|
||||
parser.add_argument("--host", type=str, default="0.0.0.0")
|
||||
args = parser.parse_args()
|
||||
|
||||
run_server(args.task, args.base_model, args.workspace, args.host, args.port)
|
||||
@@ -0,0 +1,45 @@
|
||||
你负责维护实验的累积运行总结文件 `reports/summary.md`。
|
||||
|
||||
每轮实验结束后，你需要用 file_editor 在 `reports/summary.md` **末尾追加**一个新 section。
|
||||
|
||||
如果 `reports/summary.md` 不存在，先创建文件，首行写 `# 运行总结`。
|
||||
|
||||
每轮追加的 section 格式（严格遵守，不增删字段）：
|
||||
|
||||
```
|
||||
## Iteration N (YYYY-MM-DD HH:MM, 耗时 Xs)
|
||||
- **状态**: ✅ 成功 / ❌ 失败 (exit_code=X)
|
||||
- **Score**: X | Improvement: X | Best: X (iter N)
|
||||
- **训练类型**: GRPO / SFT / PPO / copy_model / placeholder / unknown
|
||||
- **关键配置**: lr=X, epochs=X, batch=X, ...（从代码中提取）
|
||||
- **做了什么**: 具体策略（训练方法、reward 函数设计、数据处理等）
|
||||
- **为什么**: 为什么选择这个策略（基于上轮结果的推理）
|
||||
- **问题/进步**: 发现了什么问题，或相比上轮取得了什么进步
|
||||
- **关键代码**: 最能体现本轮策略的 3-5 行代码
|
||||
- **代码片段（上下文）**: 从 train.py 复制 15-40 行可运行上下文，标注函数名/行号范围；若与上轮相同写“与 Iteration N 相同，无变更”
|
||||
```
|
||||
|
||||
数据来源建议（按优先级）：
|
||||
- Score/Improvement/Best: scores.json 或服务器返回
|
||||
- 状态/耗时/exit_code: run.log
|
||||
- 训练类型/关键配置/关键代码: code/train.py
|
||||
- 失败根因: agent.log + run.log
|
||||
|
||||
规则：
|
||||
1. **追加**，不要覆盖已有内容
|
||||
2. 必须按 Iteration 递增写入：读取 `reports/summary.md` 中最后一个迭代号，当前必须是 N+1；若不满足先修正再写
|
||||
3. 分析 code下面的 源码，提取训练类型和超参数；若无法确定写 unknown，不留空
|
||||
4. 如果训练失败，必须给出可定位的根因（日志片段或错误类型），并标注 failure_type（如：code_error_runtime / rollout_logic_wrong / timeout_no_submission / copy_model_fallback / training_diverged / unknown）
|
||||
5. “做了什么”“为什么”是最重要字段，必须可复现、可检验，且“为什么”必须引用上轮证据（scores.json/run.log/agent.log）
|
||||
6. **问题/进步** 必须包含过程指标或失败类型（如：valid_submission_rate / first_valid_idx / time_to_first_improvement / time_used_ratio / failure_type）
|
||||
7. 长代码片段每轮最多 1 段，最多 40 行；优先贴本轮新增或改动处；若与上轮相同写“与 Iteration N 相同，无变更”
|
||||
8. 若与上一轮根因相同，避免整段重复，明确写出“新增证据/新增尝试/无新增”
|
||||
9. 同步追加 `reports/summary.jsonl`（同目录），每轮一行 JSON，字段建议：
|
||||
- iteration, timestamp, duration_s, status, exit_code
|
||||
- score, improvement
|
||||
- train_type, failure_type
|
||||
- 仅保留可量化字段（图表使用），不写 why/what/next_step 等长文本
|
||||
- metrics 由 benchmark 后处理统一计算（不在 jsonl 中填写）
|
||||
- 示例：
|
||||
{"iteration": 3, "timestamp": "2026-03-10 12:34", "duration_s": 812, "status": "success", "exit_code": 0, "score": 65.0, "improvement": 20.0, "train_type": "GRPO", "failure_type": "unknown"}
|
||||
10. 写入后自检 section 完整性：上述字段必须齐全，缺失则补全后再结束
|
||||
@@ -0,0 +1,165 @@
|
||||
"""
|
||||
AutoRL-Bench Results Dashboard
|
||||
|
||||
Usage:
|
||||
streamlit run rdagent/scenarios/rl/autorl_bench/core/ui.py --server.port=8510 --server.address=0.0.0.0
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
import streamlit as st
|
||||
|
||||
CSV_PATH = Path(__file__).resolve().parent.parent / "results.csv"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# ---------- 页面配置 ----------
|
||||
st.set_page_config(page_title="AutoRL-Bench", page_icon="🧪", layout="wide")
|
||||
|
||||
# ---------- 自定义样式 ----------
|
||||
st.markdown(
|
||||
"""
|
||||
<style>
|
||||
/* 指标卡片 */
|
||||
div[data-testid="stMetric"] {
|
||||
background: linear-gradient(135deg, #667eea11, #764ba211);
|
||||
border: 1px solid #e0e0e0;
|
||||
border-radius: 10px;
|
||||
padding: 10px 14px;
|
||||
}
|
||||
div[data-testid="stMetric"] label {
|
||||
font-size: 0.72rem;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
opacity: 0.7;
|
||||
}
|
||||
div[data-testid="stMetric"] div[data-testid="stMetricValue"] {
|
||||
font-size: 1.3rem;
|
||||
font-weight: 700;
|
||||
}
|
||||
/* 表格行高亮 */
|
||||
.stDataFrame td {
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
</style>
|
||||
""",
|
||||
unsafe_allow_html=True,
|
||||
)
|
||||
|
||||
# ---------- 标题 ----------
|
||||
st.markdown("# 🧪 AutoRL-Bench Results")
|
||||
st.divider()
|
||||
|
||||
# ---------- 加载数据 ----------
|
||||
if not CSV_PATH.exists():
|
||||
st.info("No results yet. Run an experiment first.")
|
||||
st.stop()
|
||||
|
||||
df = pd.read_csv(CSV_PATH)
|
||||
df["timestamp"] = pd.to_datetime(df["timestamp"])
|
||||
df["duration_min"] = (df["duration_s"] / 60).round(1)
|
||||
|
||||
# ---------- 侧栏 ----------
|
||||
with st.sidebar:
|
||||
st.markdown("### Filters")
|
||||
agents = ["All"] + sorted(df["agent"].unique().tolist())
|
||||
sel_agent = st.selectbox("Agent", agents)
|
||||
|
||||
tasks = ["All"] + sorted(df["task"].unique().tolist())
|
||||
sel_task = st.selectbox("Task", tasks)
|
||||
|
||||
st.divider()
|
||||
st.markdown("### About")
|
||||
st.markdown("Evaluating LLM-driven agents that optimize smaller LLMs " "via RL post-training.")
|
||||
|
||||
filtered = df.copy()
|
||||
if sel_agent != "All":
|
||||
filtered = filtered[filtered["agent"] == sel_agent]
|
||||
if sel_task != "All":
|
||||
filtered = filtered[filtered["task"] == sel_task]
|
||||
|
||||
# ---------- Agent 对比 ----------
|
||||
if len(filtered) > 1:
|
||||
st.markdown("#### Agent Summary")
|
||||
summary = (
|
||||
filtered.groupby(["agent", "task", "base_model"])
|
||||
.agg(
|
||||
runs=("agent", "size"),
|
||||
success=("success", "sum"),
|
||||
baseline=("baseline", "first"),
|
||||
best=("best_score", "max"),
|
||||
best_improve=("improvement", "max"),
|
||||
subs=("submissions", "sum"),
|
||||
)
|
||||
.round(2)
|
||||
.reset_index()
|
||||
.sort_values("best", ascending=False)
|
||||
)
|
||||
summary.columns = [
|
||||
"Agent",
|
||||
"Task",
|
||||
"Base Model",
|
||||
"Runs",
|
||||
"Success",
|
||||
"Baseline",
|
||||
"Best",
|
||||
"Best Impr.",
|
||||
"Submissions",
|
||||
]
|
||||
st.dataframe(summary, use_container_width=True, hide_index=True)
|
||||
|
||||
st.divider()
|
||||
|
||||
# ---------- 结果表格 ----------
|
||||
st.markdown("#### Run History")
|
||||
display = filtered[
|
||||
[
|
||||
"timestamp",
|
||||
"agent",
|
||||
"driver_model",
|
||||
"base_model",
|
||||
"task",
|
||||
"baseline",
|
||||
"best_score",
|
||||
"improvement",
|
||||
"submissions",
|
||||
"duration_min",
|
||||
"success",
|
||||
"workspace",
|
||||
]
|
||||
].sort_values("timestamp", ascending=False)
|
||||
|
||||
display.columns = [
|
||||
"Time",
|
||||
"Agent",
|
||||
"Driver LLM",
|
||||
"Base Model",
|
||||
"Task",
|
||||
"Baseline",
|
||||
"Best Score",
|
||||
"Improvement",
|
||||
"Submissions",
|
||||
"Duration(min)",
|
||||
"Success",
|
||||
"Workspace",
|
||||
]
|
||||
|
||||
st.dataframe(
|
||||
display,
|
||||
use_container_width=True,
|
||||
hide_index=True,
|
||||
column_config={
|
||||
"Time": st.column_config.DatetimeColumn(format="YYYY-MM-DD HH:mm"),
|
||||
"Best Score": st.column_config.NumberColumn(format="%.2f"),
|
||||
"Baseline": st.column_config.NumberColumn(format="%.2f"),
|
||||
"Improvement": st.column_config.NumberColumn(format="%.2f"),
|
||||
"Duration(min)": st.column_config.NumberColumn(format="%.0f"),
|
||||
"Success": st.column_config.CheckboxColumn(),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,369 @@
|
||||
"""
|
||||
AutoRL-Bench Core Utilities
|
||||
|
||||
统一的工具函数：下载、baseline、grading client、workspace、results
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.conf import (
|
||||
get_baseline_cache_dir,
|
||||
get_data_dir,
|
||||
get_models_dir,
|
||||
)
|
||||
|
||||
|
||||
def kill_process_group(proc: "subprocess.Popen") -> None:
|
||||
"""尽力杀掉进程组：SIGTERM → SIGKILL → proc.kill()"""
|
||||
import signal as _signal
|
||||
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
for sig in (_signal.SIGTERM, _signal.SIGKILL):
|
||||
try:
|
||||
os.killpg(os.getpgid(proc.pid), sig)
|
||||
proc.wait(timeout=10)
|
||||
return
|
||||
except ProcessLookupError:
|
||||
return
|
||||
except subprocess.TimeoutExpired:
|
||||
continue
|
||||
except OSError:
|
||||
break
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 文件工具
|
||||
# ============================================================
|
||||
|
||||
|
||||
def ensure_symlink(src: Path, dst: Path):
|
||||
"""创建软链接（已存在则跳过，并发安全）"""
|
||||
if not src.exists():
|
||||
return
|
||||
try:
|
||||
dst.symlink_to(src)
|
||||
except FileExistsError:
|
||||
pass
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 下载相关
|
||||
# ============================================================
|
||||
|
||||
|
||||
def download_model(model_name: str, model_dir: Optional[str] = None) -> str:
|
||||
"""下载模型（已存在则跳过）"""
|
||||
base_dir = Path(model_dir) if model_dir else get_models_dir()
|
||||
target_dir = base_dir / model_name
|
||||
|
||||
if target_dir.exists() and any(target_dir.iterdir()):
|
||||
logger.info(f"Model exists: {target_dir}")
|
||||
return str(target_dir)
|
||||
|
||||
logger.info(f"Downloading model: {model_name}...")
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
snapshot_download(repo_id=model_name, local_dir=str(target_dir), local_dir_use_symlinks=False)
|
||||
logger.info(f"Model downloaded to {target_dir}")
|
||||
return str(target_dir)
|
||||
|
||||
|
||||
def download_data(task: str, data_dir: Optional[str] = None) -> str:
|
||||
"""下载训练数据（agent 可见部分）
|
||||
|
||||
支持两种模式：
|
||||
1. data_module 模式（传统）：调用 data.py 中的 download_train_data()
|
||||
2. download_data.py 脚本模式（smith benchmarks）：直接运行脚本
|
||||
"""
|
||||
import importlib
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import (
|
||||
BENCHMARKS_DIR,
|
||||
get_benchmark,
|
||||
)
|
||||
|
||||
config = get_benchmark(task)
|
||||
base_dir = Path(data_dir) if data_dir else get_data_dir()
|
||||
target_dir = base_dir / task
|
||||
|
||||
if config.data_module:
|
||||
# 传统方式（gsm8k、alfworld 等）
|
||||
module = importlib.import_module(config.data_module)
|
||||
module.download_train_data(target_dir)
|
||||
else:
|
||||
# 脚本方式（所有 smith benchmarks）
|
||||
bench_dir = Path(config.bench_dir) if config.bench_dir else BENCHMARKS_DIR / task
|
||||
script = bench_dir / "download_data.py"
|
||||
if script.exists():
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
[sys.executable, str(script)],
|
||||
cwd=str(bench_dir),
|
||||
check=True,
|
||||
)
|
||||
# 脚本输出到 bench_dir/data/train.jsonl，拷贝到 target_dir
|
||||
src = bench_dir / "data" / "train.jsonl"
|
||||
dst = target_dir / "train.jsonl"
|
||||
if src.exists() and not dst.exists():
|
||||
shutil.copy2(src, dst)
|
||||
else:
|
||||
# No download script — copy pre-existing data from bench_dir/data/
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
src = bench_dir / "data" / "train.jsonl"
|
||||
dst = target_dir / "train.jsonl"
|
||||
if src.exists() and not dst.exists():
|
||||
shutil.copy2(src, dst)
|
||||
logger.info(f"Copied {src} → {dst}")
|
||||
elif not src.exists():
|
||||
logger.warning(f"Benchmark {task} has no data_module, download_data.py, or train.jsonl")
|
||||
|
||||
return str(target_dir)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Baseline 相关
|
||||
# ============================================================
|
||||
|
||||
|
||||
def _safe_model_name(model_name: str) -> str:
|
||||
"""将模型名转为安全的文件名"""
|
||||
return re.sub(r"[/\\:*?\"<>|]", "_", model_name)
|
||||
|
||||
|
||||
def get_baseline_score(
|
||||
task: str,
|
||||
model_name: str,
|
||||
model_path: str,
|
||||
workspace_path: str,
|
||||
gpu_count: int = 1,
|
||||
test_range: str = "[:]",
|
||||
force_rerun: bool = False,
|
||||
) -> float:
|
||||
"""获取 baseline score（有缓存则读缓存，没有则评测）"""
|
||||
safe_name = _safe_model_name(model_name)
|
||||
cache_file = get_baseline_cache_dir() / f"{task}_{safe_name}.json"
|
||||
|
||||
# 检查缓存
|
||||
if not force_rerun and cache_file.exists():
|
||||
data = json.loads(cache_file.read_text())
|
||||
score = data.get("score", 0.0)
|
||||
logger.info(f"Baseline cache hit: {cache_file.name}, score={score}")
|
||||
return score
|
||||
|
||||
# 执行评测
|
||||
logger.info(f"Running baseline evaluation: task={task}, model={model_name}")
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import get_evaluator
|
||||
|
||||
evaluator = get_evaluator(task)
|
||||
result = evaluator.run_eval(
|
||||
model_path=model_path,
|
||||
workspace_path=workspace_path,
|
||||
model_name=model_name,
|
||||
gpu_count=gpu_count,
|
||||
test_range=test_range,
|
||||
)
|
||||
|
||||
score = result.get("score", 0.0)
|
||||
error = result.get("error")
|
||||
logger.info(f"Baseline score: {score}")
|
||||
|
||||
# Only cache successful evaluations — failed ones should be retried next time
|
||||
if not error:
|
||||
cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
cache_data = {
|
||||
"task": task,
|
||||
"model_name": model_name,
|
||||
"score": score,
|
||||
"test_range": test_range,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
}
|
||||
cache_file.write_text(json.dumps(cache_data, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
logger.warning(f"Baseline evaluation failed ({error}), result NOT cached")
|
||||
|
||||
return score
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Grading Server Client
|
||||
# ============================================================
|
||||
|
||||
|
||||
def submit_to_grading_server(
|
||||
model_path: str,
|
||||
grading_url: Optional[str] = None,
|
||||
timeout: int = 600,
|
||||
) -> dict | None:
|
||||
"""提交模型到 grading server 评测"""
|
||||
url = grading_url or os.environ.get("GRADING_SERVER_URL")
|
||||
if not url:
|
||||
return None
|
||||
|
||||
logger.info(f"Submitting to grading server: {url}/submit")
|
||||
resp = requests.post(f"{url}/submit", json={"model_path": model_path}, timeout=timeout)
|
||||
resp.raise_for_status()
|
||||
result = resp.json()
|
||||
logger.info(f"Grading result: score={result.get('score')}")
|
||||
return result
|
||||
|
||||
|
||||
def set_baseline_to_server(score: float, grading_url: Optional[str] = None) -> bool:
|
||||
"""设置 baseline score 到 grading server"""
|
||||
url = grading_url or os.environ.get("GRADING_SERVER_URL")
|
||||
if not url:
|
||||
return False
|
||||
|
||||
resp = requests.post(f"{url}/set_baseline", json={"score": score}, timeout=30)
|
||||
resp.raise_for_status()
|
||||
return True
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Workspace 搭建
|
||||
# ============================================================
|
||||
|
||||
|
||||
def init_run_meta(workspace: Path, timeout_s: int) -> Path:
|
||||
"""初始化 run_meta.json（单一事实源）。"""
|
||||
run_meta = workspace / "run_meta.json"
|
||||
payload = {
|
||||
"start_time": int(datetime.now().timestamp()),
|
||||
"timeout_s": int(timeout_s),
|
||||
"last_submit_time": None,
|
||||
"end_time": None,
|
||||
}
|
||||
run_meta.write_text(json.dumps(payload, indent=2, ensure_ascii=False))
|
||||
return run_meta
|
||||
|
||||
|
||||
def update_run_meta(workspace: Path, **fields) -> Path:
|
||||
"""更新 run_meta.json 的部分字段。"""
|
||||
run_meta = workspace / "run_meta.json"
|
||||
data = json.loads(run_meta.read_text()) if run_meta.exists() else {}
|
||||
data.update(fields)
|
||||
run_meta.write_text(json.dumps(data, indent=2, ensure_ascii=False))
|
||||
return run_meta
|
||||
|
||||
|
||||
def read_run_meta(workspace: Path) -> dict:
|
||||
"""读取 run_meta.json。"""
|
||||
run_meta = workspace / "run_meta.json"
|
||||
return json.loads(run_meta.read_text()) if run_meta.exists() else {}
|
||||
|
||||
|
||||
def setup_workspace(
|
||||
run_id: str,
|
||||
agent_id: str,
|
||||
task: str,
|
||||
base_model: str,
|
||||
model_path: str,
|
||||
data_path: str,
|
||||
benchmark,
|
||||
) -> Path:
|
||||
"""创建隔离的 workspace 目录并挂载资源文件，返回 workspace 路径。"""
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import BENCHMARKS_DIR
|
||||
from rdagent.scenarios.rl.autorl_bench.conf import (
|
||||
get_instructions_file,
|
||||
get_workspace_dir,
|
||||
)
|
||||
|
||||
workspace = get_workspace_dir() / task / f"{run_id}_{agent_id}"
|
||||
workspace.mkdir(parents=True, exist_ok=True)
|
||||
(workspace / "code").mkdir(exist_ok=True)
|
||||
(workspace / "output").mkdir(exist_ok=True)
|
||||
(workspace / "reports").mkdir(exist_ok=True)
|
||||
|
||||
# 模型 & 数据 symlink
|
||||
model_link = workspace / "models" / base_model
|
||||
data_link = workspace / "data"
|
||||
model_link.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
ensure_symlink(Path(model_path), model_link)
|
||||
ensure_symlink(Path(data_path), data_link)
|
||||
|
||||
# 挂载文件：任务描述 + 通用说明 + benchmark 特有文件
|
||||
bench_dir = Path(benchmark.bench_dir) if benchmark.bench_dir else BENCHMARKS_DIR / task
|
||||
ensure_symlink(bench_dir / "description.md", workspace / "description.md")
|
||||
ensure_symlink(get_instructions_file(), workspace / "instructions.md")
|
||||
|
||||
for fname in benchmark.expose_files:
|
||||
ensure_symlink(bench_dir / fname, workspace / fname)
|
||||
|
||||
return workspace
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Results CSV 记录
|
||||
# ============================================================
|
||||
|
||||
RESULTS_CSV_COLUMNS = [
|
||||
"run_id",
|
||||
"timestamp",
|
||||
"task",
|
||||
"agent",
|
||||
"driver_model",
|
||||
"base_model",
|
||||
"baseline",
|
||||
"best_score",
|
||||
"improvement",
|
||||
"submissions",
|
||||
"duration_s",
|
||||
"success",
|
||||
"workspace",
|
||||
]
|
||||
|
||||
|
||||
def detect_driver_model(env: dict) -> str:
|
||||
"""从环境变量检测驱动 agent 的 LLM 模型名。"""
|
||||
return env.get("LLM_MODEL") or os.environ.get("CHAT_MODEL") or os.environ.get("OPENAI_MODEL") or "unknown"
|
||||
|
||||
|
||||
def append_result(row: dict) -> Path:
|
||||
"""追加一行到全局 results.csv，返回文件路径。"""
|
||||
from rdagent.scenarios.rl.autorl_bench.conf import get_autorl_bench_dir
|
||||
|
||||
results_csv = get_autorl_bench_dir() / "results.csv"
|
||||
write_header = not results_csv.exists()
|
||||
with open(results_csv, "a", newline="") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=RESULTS_CSV_COLUMNS)
|
||||
if write_header:
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
return results_csv
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 运行摘要
|
||||
# ============================================================
|
||||
|
||||
|
||||
def print_summary(
|
||||
baseline: float,
|
||||
best: dict | None,
|
||||
scores: list,
|
||||
workspace,
|
||||
) -> None:
|
||||
"""打印运行摘要。"""
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"Baseline: {baseline}")
|
||||
if best:
|
||||
logger.info(f"Best Score: {best.get('score', 0)}")
|
||||
logger.info(f"Improvement: {best.get('improvement')}")
|
||||
logger.info(f"Total Submissions: {len(scores)}")
|
||||
logger.info(f"Workspace: {workspace}")
|
||||
logger.info("=" * 60)
|
||||
@@ -0,0 +1,34 @@
|
||||
# AutoRL-Bench 依赖
|
||||
# conda 环境: (Python 3.10)
|
||||
|
||||
# RL 训练（核心）
|
||||
trl>=0.27.0
|
||||
accelerate>=1.0.0
|
||||
datasets>=3.0.0
|
||||
peft>=0.18.1
|
||||
|
||||
# 评测
|
||||
opencompass==0.5.1
|
||||
setuptools<75 # uv venv 不自带, opencompass 依赖 pkg_resources
|
||||
|
||||
# 推理加速（可选，TRL 支持 0.10.2-0.12.0）
|
||||
vllm>=0.12.0
|
||||
|
||||
# 数据处理
|
||||
numpy>=1.26.0
|
||||
pandas>=1.5.0
|
||||
pydantic>=2.0.0
|
||||
|
||||
# 模型
|
||||
torch>=2.0.0
|
||||
transformers>=4.40.0
|
||||
huggingface_hub>=0.20.0
|
||||
|
||||
# Web 服务
|
||||
flask
|
||||
flask-cors
|
||||
|
||||
# 工具
|
||||
loguru
|
||||
requests
|
||||
pyyaml
|
||||
@@ -0,0 +1,229 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
AutoRL-Bench Runner
|
||||
|
||||
入口脚本。
|
||||
|
||||
Usage:
|
||||
python -m rdagent.scenarios.rl.autorl_bench.run \
|
||||
--agent example_agent --task gsm8k --model Qwen/Qwen2.5-0.5B
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger as loguru_logger
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.rl.autorl_bench.agents import get_agent
|
||||
from rdagent.scenarios.rl.autorl_bench.benchmarks import get_benchmark
|
||||
from rdagent.scenarios.rl.autorl_bench.core import (
|
||||
append_result,
|
||||
create_grading_server,
|
||||
detect_driver_model,
|
||||
download_data,
|
||||
download_model,
|
||||
init_run_meta,
|
||||
kill_process_group,
|
||||
print_summary,
|
||||
run_workspace_metrics,
|
||||
setup_workspace,
|
||||
update_run_meta,
|
||||
)
|
||||
|
||||
|
||||
def run(
|
||||
agent_id: str,
|
||||
task: str,
|
||||
base_model: str,
|
||||
timeout: int = 3600,
|
||||
port: int = 5000,
|
||||
) -> dict:
|
||||
"""运行 Agent 评测"""
|
||||
from rdagent.scenarios.rl.autorl_bench.conf import get_workspace_dir
|
||||
|
||||
start_time = datetime.now()
|
||||
run_id = start_time.strftime("%Y%m%dT%H%M%S")
|
||||
if port != 5000:
|
||||
run_id = f"{run_id}_p{port}"
|
||||
benchmark = get_benchmark(task)
|
||||
|
||||
# 每次 run 独立 workspace + 独立日志文件
|
||||
workspace = get_workspace_dir() / task / f"{run_id}_{agent_id}"
|
||||
workspace.mkdir(parents=True, exist_ok=True)
|
||||
log_file = workspace / "run.log"
|
||||
_sink_id = loguru_logger.add(log_file, format="{time:YYYY-MM-DD HH:mm:ss} | {level} | {message}", level="DEBUG")
|
||||
|
||||
# 用 mutable 容器让闭包能访问后续赋值的 agent 子进程
|
||||
_agent_proc = [None]
|
||||
|
||||
# 收到 SIGTERM/SIGINT 时杀掉整棵进程树再退出
|
||||
def _on_signal(signum, frame):
|
||||
sig_name = signal.Signals(signum).name
|
||||
logger.warning(f"Received {sig_name}, terminating...")
|
||||
proc = _agent_proc[0]
|
||||
if proc is not None:
|
||||
kill_process_group(proc)
|
||||
logger.info(f"Run interrupted by {sig_name} at {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
loguru_logger.remove(_sink_id)
|
||||
sys.exit(128 + signum)
|
||||
|
||||
signal.signal(signal.SIGTERM, _on_signal)
|
||||
signal.signal(signal.SIGINT, _on_signal)
|
||||
|
||||
logger.info(f"=== AutoRL-Bench ===")
|
||||
logger.info(f"Agent: {agent_id}, Task: {task}, Model: {base_model}")
|
||||
logger.info(f"Workspace: {workspace}")
|
||||
logger.info(f"Start: {start_time.strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
|
||||
# 1. 准备资源（已有则跳过下载）
|
||||
logger.info("Preparing resources...")
|
||||
model_path = download_model(base_model)
|
||||
data_path = download_data(task)
|
||||
|
||||
# 2. 搭建 workspace（补充 symlink 挂载）
|
||||
workspace = setup_workspace(
|
||||
run_id,
|
||||
agent_id,
|
||||
task,
|
||||
base_model,
|
||||
model_path,
|
||||
data_path,
|
||||
benchmark,
|
||||
)
|
||||
init_run_meta(workspace, timeout)
|
||||
|
||||
# 3. 启动 Grading Server + 运行 Agent
|
||||
with create_grading_server(benchmark, workspace, port, base_model) as grading:
|
||||
logger.info("Evaluating baseline...")
|
||||
baseline = grading.get_baseline(
|
||||
task,
|
||||
base_model,
|
||||
str(workspace / "models" / base_model),
|
||||
str(workspace),
|
||||
)
|
||||
logger.info(f"Baseline Score: {baseline}")
|
||||
|
||||
agent = get_agent(agent_id)
|
||||
logger.info(f"Running agent: {agent.name}")
|
||||
|
||||
env = {
|
||||
**agent.env_vars,
|
||||
**os.environ,
|
||||
"TASK": task,
|
||||
"BASE_MODEL": base_model,
|
||||
"WORKSPACE": str(workspace),
|
||||
"MODEL_PATH": str(workspace / "models" / base_model),
|
||||
"DATA_PATH": str(workspace / "data"),
|
||||
"OUTPUT_DIR": str(workspace / "output"),
|
||||
"GRADING_SERVER_URL": f"http://localhost:{port}",
|
||||
}
|
||||
|
||||
agent_log = workspace / "agent.log"
|
||||
success = False
|
||||
with open(agent_log, "w", encoding="utf-8") as af:
|
||||
proc = subprocess.Popen(
|
||||
["bash", str(agent.start)],
|
||||
env=env,
|
||||
stdout=af,
|
||||
stderr=subprocess.STDOUT,
|
||||
start_new_session=True,
|
||||
)
|
||||
_agent_proc[0] = proc
|
||||
try:
|
||||
proc.wait(timeout=timeout)
|
||||
success = proc.returncode == 0
|
||||
logger.info(f"Agent finished, exit_code={proc.returncode}, log: {agent_log}")
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning(f"Agent timed out after {timeout}s, killing process group...")
|
||||
kill_process_group(proc)
|
||||
|
||||
scores = grading.load_scores()
|
||||
|
||||
# 4. 保存结果
|
||||
end_time = datetime.now()
|
||||
update_run_meta(workspace, end_time=int(end_time.timestamp()))
|
||||
best = max(scores, key=lambda x: x.get("score", 0)) if scores else None
|
||||
|
||||
result = {
|
||||
"success": success,
|
||||
"agent_id": agent_id,
|
||||
"task": task,
|
||||
"base_model": base_model,
|
||||
"baseline_score": baseline,
|
||||
"best": best,
|
||||
"total_submissions": len(scores),
|
||||
"duration_seconds": (end_time - start_time).total_seconds(),
|
||||
}
|
||||
|
||||
# 追加到全局 results.csv
|
||||
append_result(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"timestamp": start_time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"task": task,
|
||||
"agent": agent_id,
|
||||
"driver_model": detect_driver_model(env),
|
||||
"base_model": base_model,
|
||||
"baseline": baseline,
|
||||
"best_score": best.get("score", 0) if best else 0,
|
||||
"improvement": best.get("improvement") if best else None,
|
||||
"submissions": len(scores),
|
||||
"duration_s": round((end_time - start_time).total_seconds()),
|
||||
"success": success,
|
||||
"workspace": str(workspace),
|
||||
}
|
||||
)
|
||||
|
||||
try:
|
||||
run_workspace_metrics(
|
||||
workspace=workspace,
|
||||
baseline=baseline,
|
||||
base_model_path=str(workspace / "models" / base_model),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to write workspace metrics")
|
||||
|
||||
print_summary(baseline, best, scores, workspace)
|
||||
|
||||
logger.info(f"Log saved to: {log_file}")
|
||||
|
||||
# 移除本次 run 添加的 file sink（避免异常导致进程退出）
|
||||
if _sink_id is not None:
|
||||
try:
|
||||
loguru_logger.remove(_sink_id)
|
||||
except Exception:
|
||||
logger.exception(f"Failed to remove log sink id={_sink_id}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
load_dotenv(".env")
|
||||
|
||||
parser = argparse.ArgumentParser(description="AutoRL-Bench Runner")
|
||||
parser.add_argument("--agent", "-a", required=True, help="Agent ID (openhands, rdagent)")
|
||||
parser.add_argument("--task", "-t", required=True, help="Task name (gsm8k, math, alfworld)")
|
||||
parser.add_argument("--model", "-m", required=True, help="Base model name")
|
||||
parser.add_argument("--timeout", type=int, default=3600, help="Timeout in seconds")
|
||||
parser.add_argument("--port", type=int, default=5000, help="Grading server port")
|
||||
args = parser.parse_args()
|
||||
|
||||
result = run(
|
||||
agent_id=args.agent,
|
||||
task=args.task,
|
||||
base_model=args.model,
|
||||
timeout=args.timeout,
|
||||
port=args.port,
|
||||
)
|
||||
|
||||
sys.exit(0 if result["success"] else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
# AutoRL-Bench 测试模块
|
||||
@@ -0,0 +1,112 @@
|
||||
"""
|
||||
测试 benchmark 评测功能
|
||||
|
||||
用法:
|
||||
python -m rdagent.scenarios.rl.autorl_bench.test.test_benchmark \
|
||||
--model-path /path/to/model \
|
||||
--task gsm8k
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--model-path", required=True, help="本地模型路径")
|
||||
parser.add_argument("--model-name", default=None, help="模型名称（默认从路径推断）")
|
||||
parser.add_argument("--task", default="gsm8k", help="评测任务")
|
||||
parser.add_argument("--port", type=int, default=15000, help="grading server 端口")
|
||||
args = parser.parse_args()
|
||||
|
||||
model_path = Path(args.model_path).resolve()
|
||||
if not model_path.exists():
|
||||
print(f"[ERROR] Model not found: {model_path}")
|
||||
return 1
|
||||
|
||||
model_name = args.model_name or model_path.name
|
||||
grading_url = f"http://localhost:{args.port}"
|
||||
|
||||
print(f"Model Path: {model_path}")
|
||||
print(f"Model Name: {model_name}")
|
||||
print(f"Task: {args.task}")
|
||||
print(f"Grading URL: {grading_url}")
|
||||
print("-" * 50)
|
||||
|
||||
# 使用固定 workspace
|
||||
from rdagent.scenarios.rl.autorl_bench.conf import get_workspace_dir
|
||||
|
||||
workspace = get_workspace_dir() / args.task
|
||||
workspace.mkdir(parents=True, exist_ok=True)
|
||||
print(f"Workspace: {workspace}")
|
||||
|
||||
# 启动 grading_server
|
||||
import threading
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.core.server import app, init_server
|
||||
|
||||
server = init_server(args.task, model_name, str(workspace))
|
||||
|
||||
print(f"Starting grading server on port {args.port}...")
|
||||
server_thread = threading.Thread(
|
||||
target=lambda: app.run(host="0.0.0.0", port=args.port, debug=False, threaded=False), daemon=True
|
||||
)
|
||||
server_thread.start()
|
||||
|
||||
# 等待 server 启动
|
||||
for i in range(10):
|
||||
time.sleep(0.5)
|
||||
try:
|
||||
resp = requests.get(f"{grading_url}/health", timeout=2)
|
||||
if resp.status_code == 200:
|
||||
print(f"Grading server started.")
|
||||
break
|
||||
except:
|
||||
pass
|
||||
else:
|
||||
print("[ERROR] Grading server failed to start")
|
||||
return 1
|
||||
|
||||
# 提交评测
|
||||
print("-" * 50)
|
||||
print("Submitting model for evaluation...")
|
||||
print(f"POST {grading_url}/submit")
|
||||
|
||||
start_time = time.time()
|
||||
resp = requests.post(
|
||||
f"{grading_url}/submit",
|
||||
json={"model_path": str(model_path)},
|
||||
timeout=3600,
|
||||
)
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
print("-" * 50)
|
||||
print(f"Response status: {resp.status_code}")
|
||||
print(f"Elapsed: {elapsed:.2f}s")
|
||||
print("Result:")
|
||||
|
||||
if resp.status_code == 200:
|
||||
result = resp.json()
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
score = result.get("score", 0)
|
||||
print("-" * 50)
|
||||
if score > 0:
|
||||
print(f"[SUCCESS] Score: {score}")
|
||||
else:
|
||||
print(f"[FAILED] Score: {score}")
|
||||
else:
|
||||
print(f"Error response: {resp.text}")
|
||||
print("-" * 50)
|
||||
print(f"[ERROR] Server returned {resp.status_code}")
|
||||
|
||||
print("Done.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,319 @@
|
||||
"""
|
||||
测试 B1-B4 修复
|
||||
|
||||
验证:
|
||||
B1: LoRA adapter 自动检测
|
||||
B2: 评测锁（串行化 GPU 访问）
|
||||
B3: model_path 去重缓存
|
||||
B4: error 字段透传
|
||||
|
||||
运行: python -m rdagent.scenarios.rl.autorl_bench.test.test_fixes
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
PASS = 0
|
||||
FAIL = 0
|
||||
|
||||
|
||||
def report(name: str, ok: bool, detail: str = ""):
|
||||
global PASS, FAIL
|
||||
status = "PASS" if ok else "FAIL"
|
||||
if ok:
|
||||
PASS += 1
|
||||
else:
|
||||
FAIL += 1
|
||||
print(f" [{status}] {name}" + (f" — {detail}" if detail else ""))
|
||||
|
||||
|
||||
# ============================================================
|
||||
# B1: LoRA adapter 自动检测
|
||||
# ============================================================
|
||||
def test_b1_lora_detection():
|
||||
print("\n=== B1: LoRA adapter detection ===")
|
||||
from rdagent.scenarios.rl.autorl_bench.core.opencompass import OpenCompassEvaluator
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
adapter_dir = Path(tmpdir) / "lora_output"
|
||||
adapter_dir.mkdir()
|
||||
base_model_dir = Path(tmpdir) / "base_model"
|
||||
base_model_dir.mkdir()
|
||||
(base_model_dir / "config.json").write_text("{}")
|
||||
|
||||
# Case 1: adapter_config.json 存在且 base model 存在
|
||||
(adapter_dir / "adapter_config.json").write_text(json.dumps({"base_model_name_or_path": str(base_model_dir)}))
|
||||
|
||||
config = MagicMock()
|
||||
config.id = "gsm8k"
|
||||
config.eval_config = {}
|
||||
evaluator = OpenCompassEvaluator(config)
|
||||
|
||||
with (
|
||||
patch.object(
|
||||
evaluator,
|
||||
"_get_model_inference_config",
|
||||
return_value={
|
||||
"tensor_parallel_size": 1,
|
||||
"gpu_memory_utilization": 0.9,
|
||||
"dtype": "auto",
|
||||
"max_seq_len": 4096,
|
||||
"max_out_len": 512,
|
||||
"batch_size": 8,
|
||||
"temperature": 0.0,
|
||||
"top_p": 1.0,
|
||||
"top_k": -1,
|
||||
"repetition_penalty": 1.0,
|
||||
"enable_thinking": False,
|
||||
"use_cot_postprocessor": False,
|
||||
},
|
||||
),
|
||||
patch("subprocess.run") as mock_run,
|
||||
):
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="test", stdout="")
|
||||
result = evaluator.run_eval(
|
||||
model_path=str(adapter_dir),
|
||||
workspace_path=tmpdir,
|
||||
model_name="test-model",
|
||||
)
|
||||
if mock_run.called:
|
||||
config_path = Path(tmpdir) / "opencompass_config.py"
|
||||
if config_path.exists():
|
||||
content = config_path.read_text()
|
||||
report(
|
||||
"LoRA detected → is_lora=True in config",
|
||||
"enable_lora=True" in content,
|
||||
f"config has enable_lora={'enable_lora=True' in content}",
|
||||
)
|
||||
report(
|
||||
"lora_path set in config", "lora_path=" in content, f"lora_path found={'lora_path=' in content}"
|
||||
)
|
||||
report(
|
||||
"model_path points to base model",
|
||||
str(base_model_dir) in content,
|
||||
f"base_model in config={str(base_model_dir) in content}",
|
||||
)
|
||||
else:
|
||||
report("OpenCompass config generated", False, "config file not found")
|
||||
else:
|
||||
report("OpenCompass was called", False, "subprocess.run not called")
|
||||
|
||||
# Case 2: adapter_config.json with missing base model
|
||||
bad_adapter_dir = Path(tmpdir) / "bad_lora"
|
||||
bad_adapter_dir.mkdir()
|
||||
(bad_adapter_dir / "adapter_config.json").write_text(
|
||||
json.dumps({"base_model_name_or_path": "/nonexistent/model"})
|
||||
)
|
||||
result = evaluator.run_eval(
|
||||
model_path=str(bad_adapter_dir),
|
||||
workspace_path=tmpdir,
|
||||
model_name="test-model",
|
||||
)
|
||||
report(
|
||||
"Missing base model → returns error",
|
||||
"error" in result and "not found" in result["error"],
|
||||
result.get("error", "no error"),
|
||||
)
|
||||
|
||||
# Case 3: normal model (no adapter_config.json) — should NOT set is_lora
|
||||
normal_dir = Path(tmpdir) / "normal_model"
|
||||
normal_dir.mkdir()
|
||||
(normal_dir / "config.json").write_text("{}")
|
||||
with (
|
||||
patch.object(
|
||||
evaluator,
|
||||
"_get_model_inference_config",
|
||||
return_value={
|
||||
"tensor_parallel_size": 1,
|
||||
"gpu_memory_utilization": 0.9,
|
||||
"dtype": "auto",
|
||||
"max_seq_len": 4096,
|
||||
"max_out_len": 512,
|
||||
"batch_size": 8,
|
||||
"temperature": 0.0,
|
||||
"top_p": 1.0,
|
||||
"top_k": -1,
|
||||
"repetition_penalty": 1.0,
|
||||
"enable_thinking": False,
|
||||
"use_cot_postprocessor": False,
|
||||
},
|
||||
),
|
||||
patch("subprocess.run") as mock_run,
|
||||
):
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="test", stdout="")
|
||||
evaluator.run_eval(
|
||||
model_path=str(normal_dir),
|
||||
workspace_path=tmpdir,
|
||||
model_name="test-model",
|
||||
)
|
||||
config_path = Path(tmpdir) / "opencompass_config.py"
|
||||
if config_path.exists():
|
||||
content = config_path.read_text()
|
||||
report(
|
||||
"Normal model → no enable_lora",
|
||||
"enable_lora" not in content,
|
||||
f"enable_lora absent={'enable_lora' not in content}",
|
||||
)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# B2+B3: 评测锁 + 去重缓存
|
||||
# ============================================================
|
||||
def test_b2b3_lock_and_cache():
|
||||
print("\n=== B2+B3: Eval lock + dedup cache ===")
|
||||
from rdagent.scenarios.rl.autorl_bench.core.server import GradingServer
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
server = GradingServer("gsm8k", "test-model", Path(tmpdir))
|
||||
|
||||
report("Server has _eval_lock", hasattr(server, "_eval_lock"))
|
||||
report("Server has _eval_cache", hasattr(server, "_eval_cache"))
|
||||
|
||||
# Mock evaluator to track concurrency
|
||||
call_log = []
|
||||
active_count = [0]
|
||||
max_concurrent = [0]
|
||||
|
||||
def mock_run_eval(**kwargs):
|
||||
active_count[0] += 1
|
||||
max_concurrent[0] = max(max_concurrent[0], active_count[0])
|
||||
call_log.append(kwargs.get("model_path", ""))
|
||||
time.sleep(0.3)
|
||||
active_count[0] -= 1
|
||||
return {"score": 85.0, "accuracy_summary": {}}
|
||||
|
||||
mock_evaluator = MagicMock()
|
||||
mock_evaluator.run_eval = mock_run_eval
|
||||
|
||||
with patch.object(server, "get_evaluator", return_value=mock_evaluator):
|
||||
# B2 test: concurrent submits should be serialized
|
||||
model_a = Path(tmpdir) / "model_a"
|
||||
model_b = Path(tmpdir) / "model_b"
|
||||
model_a.mkdir()
|
||||
model_b.mkdir()
|
||||
(model_a / "config.json").write_text("{}")
|
||||
(model_b / "config.json").write_text("{}")
|
||||
|
||||
threads = []
|
||||
results = []
|
||||
|
||||
def submit_wrapper(mp):
|
||||
r = server.submit(str(mp))
|
||||
results.append(r)
|
||||
|
||||
t1 = threading.Thread(target=submit_wrapper, args=(model_a,))
|
||||
t2 = threading.Thread(target=submit_wrapper, args=(model_b,))
|
||||
t1.start()
|
||||
t2.start()
|
||||
t1.join()
|
||||
t2.join()
|
||||
|
||||
report(
|
||||
"B2: max concurrent evals = 1 (lock works)",
|
||||
max_concurrent[0] == 1,
|
||||
f"max_concurrent={max_concurrent[0]}",
|
||||
)
|
||||
report("B2: both evaluations completed", len(results) == 2, f"results={len(results)}")
|
||||
|
||||
# B3 test: same model_path should hit cache
|
||||
call_log.clear()
|
||||
server.submit(str(model_a)) # should hit cache
|
||||
|
||||
report(
|
||||
"B3: duplicate submit uses cache (no re-eval)",
|
||||
str(model_a.resolve()) not in [str(Path(p).resolve()) for p in call_log],
|
||||
f"call_log after cache hit: {call_log}",
|
||||
)
|
||||
|
||||
# B3 test: failed eval should NOT be cached
|
||||
def mock_fail_eval(**kwargs):
|
||||
return {"score": 0.0, "error": "GPU OOM", "accuracy_summary": {}}
|
||||
|
||||
mock_evaluator.run_eval = mock_fail_eval
|
||||
fail_model = Path(tmpdir) / "fail_model"
|
||||
fail_model.mkdir()
|
||||
(fail_model / "config.json").write_text("{}")
|
||||
|
||||
r1 = server.submit(str(fail_model))
|
||||
report(
|
||||
"B3: failed eval not cached",
|
||||
str(fail_model.resolve()) not in server._eval_cache,
|
||||
f"cached={str(fail_model.resolve()) in server._eval_cache}",
|
||||
)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# B4: error 字段透传
|
||||
# ============================================================
|
||||
def test_b4_error_passthrough():
|
||||
print("\n=== B4: Error field passthrough ===")
|
||||
from rdagent.scenarios.rl.autorl_bench.core.server import GradingServer
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
server = GradingServer("gsm8k", "test-model", Path(tmpdir))
|
||||
|
||||
def mock_error_eval(**kwargs):
|
||||
return {
|
||||
"score": 0.0,
|
||||
"error": "vLLM model load failed: config.json not found",
|
||||
"accuracy_summary": {},
|
||||
}
|
||||
|
||||
mock_evaluator = MagicMock()
|
||||
mock_evaluator.run_eval = mock_error_eval
|
||||
|
||||
model_dir = Path(tmpdir) / "error_model"
|
||||
model_dir.mkdir()
|
||||
(model_dir / "config.json").write_text("{}")
|
||||
|
||||
with patch.object(server, "get_evaluator", return_value=mock_evaluator):
|
||||
result = server.submit(str(model_dir))
|
||||
report("Error field present in response", "error" in result, f"error={result.get('error', 'MISSING')}")
|
||||
report("Score is 0.0", result.get("score") == 0.0)
|
||||
|
||||
# Test _parse_results with non-numeric values (B4 in opencompass)
|
||||
import pandas as pd
|
||||
|
||||
from rdagent.scenarios.rl.autorl_bench.core.opencompass import OpenCompassEvaluator
|
||||
|
||||
config = MagicMock()
|
||||
config.id = "gsm8k"
|
||||
config.eval_config = {}
|
||||
evaluator = OpenCompassEvaluator(config)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
work_dir = Path(tmpdir)
|
||||
ts_dir = work_dir / "20260306_120000"
|
||||
summary_dir = ts_dir / "summary"
|
||||
summary_dir.mkdir(parents=True)
|
||||
|
||||
csv_path = summary_dir / "results.csv"
|
||||
df = pd.DataFrame({"dataset": ["gsm8k"], "rl-gsm8k": ["-"]})
|
||||
df.to_csv(csv_path, index=False)
|
||||
|
||||
result = {"score": 0.0, "accuracy_summary": {}, "benchmark": "gsm8k", "model_path": "/test"}
|
||||
result = evaluator._parse_results(work_dir, result)
|
||||
|
||||
report("Non-numeric score → error field set", "error" in result, result.get("error", "MISSING")[:80])
|
||||
report("Score remains 0.0 on parse failure", result["score"] == 0.0)
|
||||
|
||||
|
||||
def main():
|
||||
test_b1_lora_detection()
|
||||
test_b2b3_lock_and_cache()
|
||||
test_b4_error_passthrough()
|
||||
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Results: {PASS} passed, {FAIL} failed")
|
||||
print(f"{'='*50}")
|
||||
return 1 if FAIL > 0 else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,116 @@
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from rdagent.core.proposal import Experiment2Feedback, HypothesisFeedback
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class RLExperiment2Feedback(Experiment2Feedback):
|
||||
"""Generate feedback for RL post-training experiments using LLM."""
|
||||
|
||||
def __init__(self, scen: Scenario, version: str = "exp_feedback") -> None:
|
||||
super().__init__(scen)
|
||||
self.version = version
|
||||
|
||||
def generate_feedback(
|
||||
self, exp: Any, trace: Any | None = None, exception: Exception | None = None
|
||||
) -> HypothesisFeedback:
|
||||
"""Generate feedback using LLM."""
|
||||
# 获取实验结果
|
||||
result = getattr(exp, "result", {}) or {}
|
||||
exit_code = result.get("exit_code", -1)
|
||||
stdout = result.get("stdout", "")
|
||||
running_time = result.get("running_time", 0)
|
||||
benchmark = result.get("benchmark")
|
||||
benchmark_summary = None
|
||||
if benchmark:
|
||||
try:
|
||||
benchmark_summary = json.dumps(benchmark, ensure_ascii=False, indent=2)
|
||||
except TypeError:
|
||||
benchmark_summary = str(benchmark)
|
||||
|
||||
# 获取假设和任务描述
|
||||
hypothesis = str(exp.hypothesis) if exp.hypothesis else "N/A"
|
||||
task_desc = exp.sub_tasks[0].get_task_information() if exp.sub_tasks else "N/A"
|
||||
|
||||
if exception is not None:
|
||||
return self._gen_error_feedback(hypothesis, str(exception))
|
||||
|
||||
return self._gen_feedback_with_llm(
|
||||
hypothesis=hypothesis,
|
||||
task_desc=task_desc,
|
||||
exit_code=exit_code,
|
||||
stdout=stdout,
|
||||
running_time=running_time,
|
||||
benchmark=benchmark_summary,
|
||||
)
|
||||
|
||||
def _gen_feedback_with_llm(
|
||||
self,
|
||||
hypothesis: str,
|
||||
task_desc: str,
|
||||
exit_code: int,
|
||||
stdout: str,
|
||||
running_time: float,
|
||||
benchmark: str | None,
|
||||
) -> HypothesisFeedback:
|
||||
"""Generate feedback using LLM."""
|
||||
system_prompt = T(".prompts:exp_feedback.system").r()
|
||||
user_prompt = T(".prompts:exp_feedback.user").r(
|
||||
hypothesis=hypothesis,
|
||||
task_desc=task_desc,
|
||||
exit_code=exit_code,
|
||||
stdout=stdout,
|
||||
running_time=running_time,
|
||||
benchmark=benchmark,
|
||||
exception=None,
|
||||
)
|
||||
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=True,
|
||||
)
|
||||
resp_dict = json.loads(resp)
|
||||
|
||||
decision = resp_dict.get("decision", exit_code == 0)
|
||||
reason = resp_dict.get("reason", "")
|
||||
suggestions = resp_dict.get("suggestions", "")
|
||||
|
||||
logger.info(f"Feedback: decision={decision}, reason={reason[:100]}...")
|
||||
|
||||
return HypothesisFeedback(
|
||||
decision=decision,
|
||||
reason=reason,
|
||||
code_change_summary=suggestions,
|
||||
)
|
||||
|
||||
def _gen_error_feedback(self, hypothesis: str, error_info: str) -> HypothesisFeedback:
|
||||
"""Generate feedback for failed experiments."""
|
||||
system_prompt = T(".prompts:exp_feedback_error.system").r()
|
||||
user_prompt = T(".prompts:exp_feedback_error.user").r(
|
||||
hypothesis=hypothesis,
|
||||
error_info=error_info,
|
||||
)
|
||||
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=True,
|
||||
)
|
||||
resp_dict = json.loads(resp)
|
||||
|
||||
error_type = resp_dict.get("error_type", "Unknown")
|
||||
root_cause = resp_dict.get("root_cause", error_info)
|
||||
fix_suggestion = resp_dict.get("fix_suggestion", "")
|
||||
|
||||
logger.error(f"Error feedback: {error_type} - {root_cause[:100]}...")
|
||||
|
||||
return HypothesisFeedback(
|
||||
decision=False,
|
||||
reason=f"[{error_type}] {root_cause}",
|
||||
code_change_summary=fix_suggestion,
|
||||
)
|
||||
@@ -0,0 +1,63 @@
|
||||
exp_feedback:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责分析实验结果并生成反馈。
|
||||
|
||||
## 分析维度
|
||||
1. 训练是否成功完成
|
||||
2. 代码质量和实现正确性
|
||||
3. 是否达成假设目标
|
||||
4. 改进建议
|
||||
|
||||
## 输出要求
|
||||
JSON 格式：{"decision": true/false, "reason": "...", "suggestions": "..."}
|
||||
- decision: true 表示接受当前实验，false 表示拒绝
|
||||
- reason: 决策原因
|
||||
- suggestions: 下一步改进建议
|
||||
|
||||
user: |-
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
## 任务描述
|
||||
{{ task_desc }}
|
||||
|
||||
## 执行结果
|
||||
- exit_code: {{ exit_code }}
|
||||
- running_time: {{ running_time }}s
|
||||
{% if stdout %}
|
||||
- stdout (前1000字符):
|
||||
{{ stdout[:1000] }}
|
||||
{% endif %}
|
||||
{% if benchmark %}
|
||||
## Benchmark 结果
|
||||
{{ benchmark }}
|
||||
{% endif %}
|
||||
|
||||
{% if exception %}
|
||||
## 异常信息
|
||||
{{ exception }}
|
||||
{% endif %}
|
||||
|
||||
请分析实验结果并给出反馈。
|
||||
|
||||
exp_feedback_error:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责分析失败的实验。
|
||||
|
||||
## 常见错误类型
|
||||
- ImportError: 缺少依赖库
|
||||
- SyntaxError: 代码语法错误
|
||||
- RuntimeError: 运行时错误（OOM、CUDA 等）
|
||||
- API 不兼容: 库版本问题
|
||||
|
||||
## 输出要求
|
||||
JSON 格式：{"error_type": "...", "root_cause": "...", "fix_suggestion": "..."}
|
||||
|
||||
user: |-
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
## 错误信息
|
||||
{{ error_info }}
|
||||
|
||||
请分析错误原因并给出修复建议。
|
||||
Vendored
+8
@@ -0,0 +1,8 @@
|
||||
"""RL Environment Configuration"""
|
||||
|
||||
from rdagent.scenarios.rl.env.conf import (
|
||||
RL_DATA_DIR,
|
||||
RL_MODELS_DIR,
|
||||
)
|
||||
|
||||
__all__ = ["RL_DATA_DIR", "RL_MODELS_DIR"]
|
||||
Vendored
+15
@@ -0,0 +1,15 @@
|
||||
"""
|
||||
RL Training Environment Configuration
|
||||
|
||||
autorl_bench 模式下，run.py 已完成环境搭建，不需要 Docker。
|
||||
保留基础路径配置供其他模块引用。
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.rl.conf import RL_RD_SETTING
|
||||
|
||||
# RL 资源路径（从 env var 优先，fallback 到 RL_RD_SETTING）
|
||||
RL_MODELS_DIR = Path(os.environ.get("MODEL_PATH", str(RL_RD_SETTING.file_path / "models")))
|
||||
RL_DATA_DIR = Path(os.environ.get("DATA_PATH", str(RL_RD_SETTING.file_path / "datasets")))
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
# Base 镜像：PyTorch 2.9.1 + TRL + transformers（训练+评测通用）
|
||||
FROM pytorch/pytorch:2.9.1-cuda12.6-cudnn9-runtime
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
# System dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
git ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# LLM post-training 库（trl 会自动安装兼容的 transformers、accelerate、datasets）
|
||||
# Also include `litellm` for AutoRL-Bench evaluation adapters (e.g. GSM8K).
|
||||
# 注意：transformers 4.57.x 解决 tokenizer save_pretrained 与 vLLM 的兼容性问题
|
||||
# transformers 5.0 移除了 Qwen2TokenizerFast，导致保存格式不兼容
|
||||
RUN pip install --no-cache-dir trl==0.27.0 peft verl==0.7.0 litellm>=1.73 "transformers>=4.50,<5.0"
|
||||
|
||||
# 默认入口
|
||||
CMD ["bash"]
|
||||
@@ -0,0 +1,9 @@
|
||||
# EvalPlus 训练+评测镜像
|
||||
FROM autorl-bench/base:latest
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
# 额外安装：evalplus
|
||||
RUN pip install --no-cache-dir evalplus
|
||||
|
||||
CMD ["bash"]
|
||||
@@ -0,0 +1,10 @@
|
||||
# GSM8K 训练镜像
|
||||
FROM autorl-bench/base:latest
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
# GSM8K 不需要额外依赖，base 镜像已包含所有
|
||||
# agent 生成的 main.py 会被挂载到 /workspace
|
||||
|
||||
CMD ["python", "main.py"]
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# MiniWoB 训练+评测镜像
|
||||
FROM autorl-bench/base:latest
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
# 额外安装：浏览器 + selenium + miniwob
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
chromium chromium-driver \
|
||||
fonts-liberation \
|
||||
libnss3 libxss1 libasound2 libgbm1 \
|
||||
libx11-6 libxext6 libxrender1 libxtst6 \
|
||||
libgtk-3-0 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install --no-cache-dir \
|
||||
miniwob \
|
||||
gymnasium \
|
||||
selenium
|
||||
|
||||
ENV CHROME_BIN=/usr/bin/chromium \
|
||||
CHROMEDRIVER_BIN=/usr/bin/chromedriver
|
||||
|
||||
CMD ["bash"]
|
||||
@@ -0,0 +1,22 @@
|
||||
"""RL Post-training Experiment"""
|
||||
|
||||
from rdagent.core.experiment import Experiment, Task
|
||||
from rdagent.scenarios.rl.experiment.workspace import RLWorkspace
|
||||
|
||||
|
||||
class RLTask(Task):
|
||||
"""RDLoop 内部的任务描述（每次迭代一个）。
|
||||
|
||||
仅用于 rdagent 框架内部流转，和 autorl_bench 的 benchmark 无关。
|
||||
"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class RLExperiment(Experiment[RLTask, RLWorkspace, RLWorkspace]):
|
||||
"""RL post-training experiment with workspace initialization."""
|
||||
|
||||
def __init__(self, sub_tasks: list[RLTask], *args, **kwargs) -> None:
|
||||
super().__init__(sub_tasks=sub_tasks, *args, **kwargs)
|
||||
# Initialize experiment workspace (required by CoSTEER)
|
||||
self.experiment_workspace = RLWorkspace()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user