mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 23:47:46 +00:00
feat: add LLM-finetune scenario (#1314)
* refine prompt * small update * fix a small bug * remove debug config after execution * fix: only remove <think> at start * feat: support creating dataset & multi-eval frame (#1302) * feat: add iterative evolve and evaluation support with partial chain stop * feat: add FTDataEvaluator and support multiple implement functions in finetune * feat: data implement for pre-proposal and proposal and add datasets (#1303) * feat:(1) support for multi layer dataset extraction (2) add category.json for dataset in datasets/ * fix: fix bug for generate category.json * feat: add get_dataset_folder_desc * init data proposal and merge qzli/ft * update data proposal prompts and add max_position_embeddings and resolve confilcts * remove sample counts in data proposal * turn data and train to unified hypo_gen * refine prompts * remove category.json and add it to dataset_info * fix jinja problem and proposal done * lint * add ai-generated description and raw readme into dataset_info.json * update prompt for description * add datasets * initial fix for proposal of data * final version for data proposal * lint * feat: add stats in dataset_info, and enable data coder (#1306) * refactor(dataset): add stats into dataset_info.json, and remove dataset from gitignore_folder * feat: enable data coder and run data process * feat: Merge data coder (#1307) * feat: implement finetune data coding, evaluation, and config improvements * fix: deepspeed config path * fix: dataset info columns --------- Co-authored-by: Young <afe.young@gmail.com> * replace str length with token_limit * add readme to dataset_info and remove useless blank lines in scenario description * feat: dataset prepare * fix: extract prams script name * feat: add loss&predictions samples to feedback * remove duplicate envs and and add llm_api_preferences and enhance reasoning token limits * feat: network for ft_env * fix: remove gpt-4o, which has low quota * feat: a simple ui * feat: merge data and train task type (#1309) * feat: filter redundant prams of lf * fix: ui bug caused by removing task_type * fix: force agent to use high concurrency, and remove redundant prompt * feat: extract info from llama factory log, and check data exists before download * fix: add compatibility rules * feat: llm evaluator for data coder * feat: openai package in ft docker, and refine prompt * feat: refine ft ui, add more info * feat: add raw logs * refine data coder prompt(for feedback debug) * feat: select dataset in scen init * fix: ui for docker log seperately * feat: sync log through blob * improve ui, and add llm feedback in Runner&Exp2FB (#1312) * fix: ui bug to visualize docker log, and lint * feat: unified docker log for ft env, and some refactor * fix bugs and improve ui * feat: save log of evaluator(single feedback) * feat: add evaluator, set cleanup docker log * feat: call llm in RunnerEvaluator and Feedback * fix: extract structured error message in RunnerEvaluator * feat: feedback improve, and fix some bugs * feat: feedback improve when runner fails * small update * feat(UI): add running info and benchmark metric in loop expander * feat(UI): add render markdown toggle * feat: refine prompts and add error type in exp2fb * feat: add filterd params reason, set default benchmark timeout to infinite, and refine train loss express * recover dataset deepscaler * feat: set timeout in .env * refactor: unifiied ft_env timeout * feat: debug mode for data coder * feat: deliver data_stats after generate debug_data * feat: use gpt-5.1 as judge model, set judge_retry, and refine debug mode prompt * refine prompt * refactor: llama factory manager logic, and refine data processing prompt * feat(DockerEnv): support GPU selection via CUDA_VISIBLE_DEVICES * feat: set api concurrency via .env * fix: ft env timeout bug * feat: enable CondaEnv run * fix: can't update bin path in first run, and path bug in lf manager * feat(ui): set log path through .env * refactor(ui): wrap_lines, remove css * feat(coder): retry when parse code-block fail * fix: refine single-fb in ui, and fix path bug(not allow proposal to decide path) * fix: opencompass CondaEnv torch compatible with vllm * fix: refine error text in coding * feat: deepspeed config for CondaEnv * feat: memory estimator * fix: deepspeed package for condaenv * fix: use `client.chat.completions.create()` only * feat: flash attention for condaenv * feat: strong and weak models interface * fix: condaenv package dependency * use multi round conversation in llm finetune proposal * refine prompt for data processing * enable evolving in data coder * maximize output token size * fix: refine ui * fix: optional packages for llama factory * fix: torch denpendency for b200 * fix: opencompass dependency * update cot prompts * skip the sub implement * skip conda preparation if env exists * update chemcot datasets * fix: unify docker to use litellm * update readme and instructions * fix: set CUDA_VISIBLE_DEVICES for CondaEnv * feat: add panorama dataset, refactor dataset interface * feat: calculate token using tiktoken, and ndarray bug * fix: download subtasks of chemcotdataset seperately * feat: customized prepare func for datasets * feat: update new benchmarks * add datasets package * docs: readme for llm finetune * feat: download raw data directly, with post-process function * feat: analyze raw dataset * suppress litellm debug info * feat(ui): summary page * feat: run multi-jobs * feat: improve ui * feat: add path and checkout options to LLM finetune loop entrypoint * feat: add FinanceIQ_ppl benchmark with auto-download and dataset desc rendering * refactor: remove unused imports and dead code, fix session folder logging * feat: enable tablebench and tableInstruct dataset * refine dataset readme, and coder prompt * refine proposal and coder prompt * fix: ui path (default log path) * feat: add automatic LoRA model merging for benchmarking with vLLM * refactor: reorganize finetune benchmark and merge modules under benchmark dir * refactor: modularize benchmark config and error extraction for finetune scenario * fix: update benchmark import paths and disable env cache for device info * refactor docke&conda env and fix import bugs * modify init python file * feat: add FinanceIQ dataset split utility and integrate with pipeline * feat: set weak and strong model by env, distribute workload across models * feat: sample dataset and rm params for tensorboard, wandb * update script to run jobs * refine proposal prompt, remove specific dataset name * fix(ui): auto switch log folder * fix: estimate the processed full data after sample * feat: filter raw data more aggressively, and lower data_eval standard * feat: sync workspace to blob * feat: rdkit for chemcotbench * update qwen2.5&llama3.1 context * fix: force failure on validation error and remove try/except in validator * feat: unified error sample extraction (with test scripts) * feat: set conda cache with .env * feat: skip data eval if data pass in last evo * fix: rm redundant param * fix ui bug * refactor: centralize assign_code_list_to_evo in MultiProcessEvolvingStrategy * feat: add test_params.yaml generation and workspace cleanup improvements for finetune * refactor: replace get_clear_ws_cmd with clear_workspace and update prompts for hard check criteria * add bioprobench dataset * fix: handle commas in training config extraction and refactor prompt includes * bioprobench description * add bioprobench readme * feat: merge lora adapter for blackwell gpu * feat: support for multi benchmarks in one job * change dfficult aware content for training * update difficulty-aware and logging principles * fix: resolve variable name conflict in FTRunnerEvaluator * set job id accuracy to minute * feat(ui): display one selected metric per benchmark * feat: store sota exp, and fix ws_ckp bug * fix: truncate data.json in feedback * fix: opencompass data for conda env * fix: save only the last model * feat: set log path and ws path * fix: set overwrite_cache to avoid lock contention(through injecting params) * feat: redirect stdout to file in localenv * add pickle cache to dataset desc * fix CI * fix: remove redundant wrapper * feat: set python_unbuffered * move redirect stdout to env run * fix a small bug * move model folder * feat(ui): display benchmark baseline * fix: enrich scenario and benchmark description * fix: rewrite runner eval to accept easier * feat: compare with baseline when no SOTA * update tablebench readme * fix: switch back to single benchmark (for baseline) * feat(ui): add ws path in ui * refactor: update SOTA tracking to use DAG traversal and parent selection * fix: prioritize local_selection in trace and refactor sibling retrieval logic * refactor: unify error handling in feedback generation and update workspace injection * feat: add skip_loop_error_stepname to control error skip step in LoopBase * fix: set local_selection to NEW_ROOT for experiments without parent * feat: set different ports for jobs * feat: set different ports for jobs * feat: add upper data size limit for LLM fine-tuning and update related prompts * fix: replace get_truncated_stdout() with stdout for consistent output handling * refactor: remove data.json from cache and workspace logic, focus on script-based reuse * fix: rm target_scenario * feat: add selective cache extraction and custom cache key for data processing * fix(ui): bug when displaying tablebench * fix: filter config in dataset_info.json * feat: add test set, set valid set * feat(ui): update test score, and set color for final decision * feat: add test score for baseline and update ui * fix: use [-100:] as test range * feat: update data_stats in runner * feat: wait for opencompass init when run multi jobs * fix: adjust test&valid split * feat: force to generate COT(with <think> token), and add answer format in scenarios.json * feat: improve ui * fix: unify benchmark volume mounts and set extra_volumes for conda env * fix(ui): number color * fix: update GPU memory handling to use total memory in GB and streamline code * fix: set use_cot_postprocessor * feat: add env_dict to config classes and merge env vars in Env run * fix: let coder obey proposal * fix(ui): direction bug and update chemcot core metirc * fix: set consistent benchmark mount points and env vars for docker and conda * fix: addintional target for LoRA * feat: workspace dir log for benchmark running * fix: tableInstruct path bug and update benchmark description * feat: timeout for whole job * fix: align FinanceIQ import to opencompass * feat: use llm_judge for FinanceIQ * feat: switch to turn on <think> or not * feat: using scripts to redirect stdout, and run in different windows * feat: sync litellm log * fix: gpu memory format * fix: escape special characters in benchmark desc * fix: set data processing timeout to 1h * feat: set valid_loss and save_best_model * fix: inject timeout and stage * fix: loss history extract logic * feat: inject output dir * feat: inject eval batch size * feat: inject save_total_limit * feat: update data prompt * fix: escape shell special characters * fix: tablebench visualization UI * fix: move implementation validation to coder, and ignore injected params * feat: README for FinanceIQ dataset * fix: bioprobench desc error * fix: remove task alignment when coder eval * fix: FinanceIQ now extracts last capital as answer * fix: stdout contains binary data * feat: recover estimate full output and set eval setting automatically * fix(ui): precision for summary table * fix(ui): import error * feat: try to use lora * fix(api): fix litellm bug for code block * fix: refine prompts to give agent more decision space * chore(ci): fix mypy typing issues * chore(ci): format code with black * chore(ci): fix ruff lint violations * chore(ci): sort imports with isort * chore(ci): format code with black * test: temporarily skip extract_parameters imports due to numpy pin * fix: compatibility issues for qlib scenarios on finetune branch * fix(fin_factor): skip to fb for coder error * fix(loop): default skip to feedback step on skip_loop_error When skip_loop_error exception happens and skip_loop_error_stepname is not explicitly set, default to jumping to 'feedback' step if it exists, otherwise fall back to the last step (record). This prevents KeyError when record step tries to access feedback data that doesn't exist because we skipped the feedback phase. Also removed redundant skip_loop_error_stepname from finetune loop since it's now the default behavior. * add 'skip to record' to DS scenario like other scenarios * fix 2 scenarios bug about rd_loop class * fix: lint(mypy, ruff, black) error * fix: mypy lint error * fix data science scenario bug --------- Co-authored-by: Xu Yang <peteryang@vip.qq.com> Co-authored-by: Qizheng Li <jenssenlee@163.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com> Co-authored-by: amstrongzyf <201840057@smail.nju.edu.cn> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: amstrongzyf <amstrongzyf@126.com> Co-authored-by: chelsea97 <zhuowbrown@gmail.com> Co-authored-by: SunsetWolf <Lv.Linlang@hotmail.com>
This commit is contained in:
XianBW
@@ -28,6 +28,7 @@ class CoSTEER(Developer[Experiment]):
|
||||
with_knowledge: bool = True,
|
||||
knowledge_self_gen: bool = True,
|
||||
max_loop: int | None = None,
|
||||
stop_eval_chain_on_fail: bool = False,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
@@ -46,6 +47,7 @@ class CoSTEER(Developer[Experiment]):
|
||||
self.evolving_strategy = es
|
||||
self.evaluator = eva
|
||||
self.evolving_version = evolving_version
|
||||
self.stop_eval_chain_on_fail = stop_eval_chain_on_fail
|
||||
|
||||
# init rag method
|
||||
self.rag = (
|
||||
@@ -99,10 +101,10 @@ class CoSTEER(Developer[Experiment]):
|
||||
evolving_strategy=self.evolving_strategy,
|
||||
rag=self.rag,
|
||||
with_knowledge=self.with_knowledge,
|
||||
with_feedback=True,
|
||||
knowledge_self_gen=self.knowledge_self_gen,
|
||||
enable_filelock=self.settings.enable_filelock,
|
||||
filelock_path=self.settings.filelock_path,
|
||||
stop_eval_chain_on_fail=self.stop_eval_chain_on_fail,
|
||||
)
|
||||
|
||||
# Evolving the solution
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import json
|
||||
from abc import abstractmethod
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, List
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Dict, Generator, List
|
||||
|
||||
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.evaluation import Evaluator, Feedback
|
||||
from rdagent.core.evolving_agent import RAGEvaluator
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import Task, Workspace
|
||||
from rdagent.core.utils import multiprocessing_wrapper
|
||||
@@ -37,12 +39,16 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
It is design align the phases of the implemented code
|
||||
- Execution -> Return Value -> Code -> Final Decision
|
||||
"""
|
||||
execution: str
|
||||
execution: str # Summarized execution feedback
|
||||
# execution_feedback
|
||||
return_checking: str | None # including every check in the testing (constraints about the generated value)
|
||||
# value_feedback, shape_feedback, value_generated_flag
|
||||
code: str
|
||||
final_decision: bool | None = None
|
||||
raw_execution: str = "" # Full raw stdout for UI display
|
||||
source_feedback: Dict[str, bool] = field(
|
||||
default_factory=dict
|
||||
) # Record the source of the feedback since it might be merged from multiple feedbacks, stores the mapping from source tag to its final_decision, this dict also includes the feedback source of itself
|
||||
|
||||
@staticmethod
|
||||
def val_and_update_init_dict(data: dict) -> dict:
|
||||
@@ -72,8 +78,8 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
raise ValueError(f"'final_decision' must be a boolean, not {type(data['final_decision'])}")
|
||||
|
||||
for attr in "execution", "return_checking", "code":
|
||||
if data[attr] is not None and not isinstance(data[attr], str):
|
||||
raise ValueError(f"'{attr}' must be a string, not {type(data[attr])}")
|
||||
if data.get(attr) is not None and not isinstance(data[attr], str):
|
||||
data[attr] = json.dumps(data[attr], indent=2, ensure_ascii=False)
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
@@ -95,6 +101,10 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
attr,
|
||||
"\n\n".join([getattr(_fb, attr) for _fb in feedback_li if getattr(_fb, attr) is not None]),
|
||||
)
|
||||
fb.source_feedback = {}
|
||||
for _fb in feedback_li:
|
||||
for tag, decision in _fb.source_feedback.items():
|
||||
fb.source_feedback[tag] = decision
|
||||
return fb
|
||||
|
||||
def __str__(self) -> str:
|
||||
@@ -125,6 +135,7 @@ class CoSTEERSingleFeedbackDeprecated(CoSTEERSingleFeedback):
|
||||
final_feedback: str = None,
|
||||
value_generated_flag: bool = None,
|
||||
final_decision_based_on_gt: bool = None,
|
||||
source_feedback: dict = None,
|
||||
) -> None:
|
||||
self.execution_feedback = execution_feedback
|
||||
self.code_feedback = code_feedback
|
||||
@@ -133,6 +144,7 @@ class CoSTEERSingleFeedbackDeprecated(CoSTEERSingleFeedback):
|
||||
self.final_feedback = final_feedback
|
||||
self.value_generated_flag = value_generated_flag
|
||||
self.final_decision_based_on_gt = final_decision_based_on_gt
|
||||
self.source_feedback = source_feedback if source_feedback is not None else {}
|
||||
|
||||
# TODO:
|
||||
# Not general enough. So we should not put them in the general costeer feedback
|
||||
@@ -226,6 +238,7 @@ class CoSTEEREvaluator(Evaluator):
|
||||
# TODO:
|
||||
# I think we should have unified interface for all evaluates, for examples.
|
||||
# So we should adjust the interface of other factors
|
||||
# Based on the implementation, I think a better name is some name like task-implement evaluator
|
||||
@abstractmethod
|
||||
def evaluate(
|
||||
self,
|
||||
@@ -237,19 +250,23 @@ class CoSTEEREvaluator(Evaluator):
|
||||
raise NotImplementedError("Please implement the `evaluator` method")
|
||||
|
||||
|
||||
class CoSTEERMultiEvaluator(CoSTEEREvaluator):
|
||||
class CoSTEERMultiEvaluator(RAGEvaluator):
|
||||
"""This is for evaluation of experiment. Due to we have multiple tasks, so we will return a list of evaluation feebacks"""
|
||||
|
||||
def __init__(self, single_evaluator: CoSTEEREvaluator | list[CoSTEEREvaluator], *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
def __init__(self, single_evaluator: CoSTEEREvaluator | list[CoSTEEREvaluator], scen: "Scenario") -> None:
|
||||
super().__init__()
|
||||
self.scen = scen
|
||||
self.single_evaluator = single_evaluator
|
||||
|
||||
def evaluate(
|
||||
def evaluate_iter(
|
||||
self,
|
||||
evo: EvolvingItem,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERMultiFeedback:
|
||||
) -> Generator[CoSTEERMultiFeedback, EvolvingItem | None, CoSTEERMultiFeedback]:
|
||||
evo = yield CoSTEERMultiFeedback(
|
||||
[]
|
||||
) # it will receive the evo first, so the first yield is for get the sent evo instead of generate useful feedback
|
||||
|
||||
eval_l = self.single_evaluator if isinstance(self.single_evaluator, list) else [self.single_evaluator]
|
||||
|
||||
# 1) Evaluate each sub_task
|
||||
@@ -279,7 +296,12 @@ class CoSTEERMultiEvaluator(CoSTEEREvaluator):
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
# None received, we skip the rest and return the overall feedback directly
|
||||
evo_next_iter = yield CoSTEERMultiFeedback(multi_implementation_feedback)
|
||||
task_li_feedback_li.append(multi_implementation_feedback)
|
||||
if evo_next_iter is None:
|
||||
break
|
||||
evo = evo_next_iter
|
||||
|
||||
# 2) merge the feedbacks along the sub_tasks to aggregate the multiple evaluation feedbacks
|
||||
merged_task_feedback = []
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import abstractmethod
|
||||
from typing import Callable, Generator
|
||||
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
@@ -26,7 +27,6 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
self.settings = settings
|
||||
self.improve_mode = improve_mode # improve mode means we only implement the task which has failed before. The main diff is the first loop will not implement all tasks.
|
||||
|
||||
@abstractmethod
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: Task,
|
||||
@@ -58,6 +58,15 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def implement_func_list(self) -> list[Callable]:
|
||||
"""
|
||||
One evolve solution will be divided into multiple implement functions.
|
||||
The functions will be called sequentially.
|
||||
|
||||
`implement_one_task` is the default implementation. Please refer to its signature for more details.
|
||||
"""
|
||||
return [self.implement_one_task]
|
||||
|
||||
@abstractmethod
|
||||
def assign_code_list_to_evo(self, code_list: list[dict], evo: EvolvingItem) -> None:
|
||||
"""
|
||||
@@ -66,19 +75,48 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
Due to the implement_one_task take `workspace` as input and output the `modification`.
|
||||
We should apply implementation to evo
|
||||
|
||||
Assumptions:
|
||||
- The modidication on evo should happen in-place!!
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def evolve(
|
||||
def assign_code_list_to_evo(self, code_list: list[dict | None], evo) -> None:
|
||||
"""Assign code modifications to evolving item.
|
||||
|
||||
For runner, coder already generated full training config, so typically no modifications.
|
||||
But this method is required by the abstract base class.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
|
||||
# If there are any modifications (usually empty for runner)
|
||||
if code_list[index]:
|
||||
# Handle change summary if present
|
||||
if self.KEY_CHANGE_SUMMARY in code_list[index]:
|
||||
evo.sub_workspace_list[index].change_summary = code_list[index].pop(self.KEY_CHANGE_SUMMARY)
|
||||
# Inject any modified files
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
|
||||
return evo
|
||||
|
||||
def evolve_iter(
|
||||
self,
|
||||
*,
|
||||
evo: EvolvingItem,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
evolving_trace: list[EvoStep] = [],
|
||||
**kwargs,
|
||||
) -> EvolvingItem:
|
||||
) -> Generator[EvolvingItem, EvolvingItem, None]:
|
||||
if queried_knowledge is None:
|
||||
raise ValueError(
|
||||
"MultiProcessEvolvingStrategy requires queried_knowledge for efficient implementation. Please set with_knowledge=True in CoSTEER constructor."
|
||||
)
|
||||
code_list = [None for _ in range(len(evo.sub_tasks))]
|
||||
|
||||
last_feedback = None
|
||||
@@ -111,24 +149,24 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
{}
|
||||
) # empty implementation for skipped task, but assign_code_list_to_evo will still assign it
|
||||
|
||||
result = multiprocessing_wrapper(
|
||||
[
|
||||
(
|
||||
self.implement_one_task,
|
||||
for implement_func in self.implement_func_list():
|
||||
result = multiprocessing_wrapper(
|
||||
[
|
||||
(
|
||||
evo.sub_tasks[target_index],
|
||||
queried_knowledge,
|
||||
evo.experiment_workspace,
|
||||
None if last_feedback is None else last_feedback[target_index],
|
||||
),
|
||||
)
|
||||
for target_index in to_be_finished_task_index
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
for index, target_index in enumerate(to_be_finished_task_index):
|
||||
code_list[target_index] = result[index]
|
||||
implement_func,
|
||||
(
|
||||
evo.sub_tasks[target_index],
|
||||
queried_knowledge,
|
||||
evo.experiment_workspace,
|
||||
None if last_feedback is None else last_feedback[target_index],
|
||||
),
|
||||
)
|
||||
for target_index in to_be_finished_task_index
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
)
|
||||
for index, target_index in enumerate(to_be_finished_task_index):
|
||||
code_list[target_index] = result[index]
|
||||
|
||||
evo = self.assign_code_list_to_evo(code_list, evo)
|
||||
|
||||
return evo
|
||||
self.assign_code_list_to_evo(code_list, evo)
|
||||
yield evo
|
||||
|
||||
@@ -100,6 +100,38 @@ class CoSTEERRAGStrategy(RAGStrategy):
|
||||
|
||||
|
||||
class CoSTEERQueriedKnowledge(QueriedKnowledge):
|
||||
"""
|
||||
Data container for knowledge retrieved from the CoSTEER knowledge base during a query operation.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
success_task_to_knowledge_dict : dict, optional
|
||||
A mapping between task information strings and their corresponding `CoSTEERKnowledge` objects
|
||||
for tasks that were successfully completed.
|
||||
Type: dict[str, CoSTEERKnowledge]
|
||||
Example:
|
||||
{
|
||||
"task_info_1": CoSTEERKnowledge(target_task=Task(...),
|
||||
implementation=FBWorkspace(...),
|
||||
feedback=CoSTEERSingleFeedback(...)),
|
||||
"task_info_2": CoSTEERKnowledge(...)
|
||||
}
|
||||
failed_task_info_set : set, optional
|
||||
A set containing task information strings that were attempted but failed repeatedly beyond
|
||||
the allowed trial limit.
|
||||
Type: set[str]
|
||||
Example:
|
||||
{
|
||||
"failed_task_info_1",
|
||||
"failed_task_info_2"
|
||||
}
|
||||
|
||||
Returns
|
||||
-------
|
||||
None
|
||||
This class is a data holder, initialization does not return any value.
|
||||
"""
|
||||
|
||||
def __init__(self, success_task_to_knowledge_dict: dict = {}, failed_task_info_set: set = set()) -> None:
|
||||
self.success_task_to_knowledge_dict = success_task_to_knowledge_dict
|
||||
self.failed_task_info_set = failed_task_info_set
|
||||
@@ -134,6 +166,8 @@ class CoSTEERQueriedKnowledgeV1(CoSTEERQueriedKnowledge):
|
||||
|
||||
|
||||
class CoSTEERRAGStrategyV1(CoSTEERRAGStrategy):
|
||||
"""it is deprecated"""
|
||||
|
||||
def __init__(self, settings: CoSTEERSettings, *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
self.current_generated_trace_count = 0
|
||||
@@ -245,6 +279,62 @@ class CoSTEERRAGStrategyV1(CoSTEERRAGStrategy):
|
||||
|
||||
|
||||
class CoSTEERQueriedKnowledgeV2(CoSTEERQueriedKnowledgeV1):
|
||||
"""
|
||||
Aggregation subclass of `CoSTEERQueriedKnowledgeV1` that extends the queried knowledge to also
|
||||
include mappings between tasks and knowledge related to similar errors from successful executions.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
task_to_former_failed_traces : dict, optional
|
||||
Mapping from task information strings to a tuple containing:
|
||||
- A list of `CoSTEERKnowledge` objects representing the most recent failed attempts for that task.
|
||||
- An optional `CoSTEERKnowledge` object of the latest failed attempt after a successful execution,
|
||||
or `None` if not applicable.
|
||||
Type: dict[str, tuple[list[CoSTEERKnowledge], CoSTEERKnowledge | None]]
|
||||
Example:
|
||||
{
|
||||
"task_info_A": ([CoSTEERKnowledge(...), CoSTEERKnowledge(...)], None),
|
||||
"task_info_B": ([CoSTEERKnowledge(...), CoSTEERKnowledge(...)], CoSTEERKnowledge(...))
|
||||
}
|
||||
|
||||
task_to_similar_task_successful_knowledge : dict, optional
|
||||
Mapping from task information strings to a list of `CoSTEERKnowledge` objects representing
|
||||
knowledge from similar tasks that have been successfully completed.
|
||||
Type: dict[str, list[CoSTEERKnowledge]]
|
||||
Example:
|
||||
{
|
||||
"task_info_A": [CoSTEERKnowledge(...), CoSTEERKnowledge(...)],
|
||||
"task_info_C": []
|
||||
}
|
||||
|
||||
task_to_similar_error_successful_knowledge : dict, optional
|
||||
Mapping from task information strings to a list of tuples, each containing:
|
||||
- A string describing the error(s) encountered.
|
||||
- A tuple of two `CoSTEERKnowledge` objects:
|
||||
* The first corresponds to the trace where that error was encountered.
|
||||
* The second is related to a successful implementation that had the same error in a prior attempt.
|
||||
Type: dict[str, list[tuple[str, tuple[CoSTEERKnowledge, CoSTEERKnowledge]]]]
|
||||
Example:
|
||||
{
|
||||
"task_info_B": [
|
||||
(
|
||||
"1. ErrorType: ValueError; Error line: some_function_call()",
|
||||
(CoSTEERKnowledge(...), CoSTEERKnowledge(...))
|
||||
)
|
||||
]
|
||||
}
|
||||
|
||||
**kwargs : dict
|
||||
Additional keyword arguments passed to the parent constructor, such as:
|
||||
- success_task_to_knowledge_dict: dict[str, CoSTEERKnowledge]
|
||||
- failed_task_info_set: set[str]
|
||||
|
||||
Returns
|
||||
-------
|
||||
None
|
||||
This class is purely a data container and does not return a value upon initialization.
|
||||
"""
|
||||
|
||||
# Aggregation of knowledge
|
||||
def __init__(
|
||||
self,
|
||||
@@ -338,7 +428,7 @@ class CoSTEERRAGStrategyV2(CoSTEERRAGStrategy):
|
||||
self.current_generated_trace_count = len(evolving_trace)
|
||||
return None
|
||||
|
||||
def query(self, evo: EvolvableSubjects, evolving_trace: list[EvoStep]) -> CoSTEERQueriedKnowledge | None:
|
||||
def query(self, evo: EvolvableSubjects, evolving_trace: list[EvoStep]) -> CoSTEERQueriedKnowledge:
|
||||
conf_knowledge_sampler = self.settings.v2_knowledge_sampler
|
||||
queried_knowledge_v2 = CoSTEERQueriedKnowledgeV2(
|
||||
success_task_to_knowledge_dict=self.knowledgebase.success_task_to_knowledge_dict,
|
||||
|
||||
@@ -67,7 +67,7 @@ class EnsembleCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
|
||||
stdout += f"\nNOTE: the above scripts run with return code {ret_code}"
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
@@ -13,7 +10,6 @@ from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
|
||||
from rdagent.utils.fmt import shrink_text
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
@@ -69,7 +65,7 @@ class FeatureCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
workflow_code=implementation.all_codes,
|
||||
)
|
||||
user_prompt = T(".prompts:feature_eval.user").r(
|
||||
stdout=result.get_truncated_stdout(),
|
||||
stdout=result.stdout,
|
||||
workflow_stdout=workflow_stdout,
|
||||
)
|
||||
|
||||
|
||||
@@ -71,7 +71,7 @@ class ModelGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
) # only check the model changed this time
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
|
||||
if stdout is None:
|
||||
|
||||
@@ -163,7 +163,7 @@ class PipelineCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
result = implementation.run(
|
||||
env=env, entry=f"strace -e trace=file -f -o trace.log python -m coverage run main.py"
|
||||
)
|
||||
result_stdout = result.get_truncated_stdout()
|
||||
result_stdout = result.stdout
|
||||
|
||||
nb_conversion_ret_code = 0
|
||||
nb_conversion_check_text = ""
|
||||
@@ -261,7 +261,7 @@ class PipelineCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
implementation.inject_files(**{"test/submission_format_test.py": base_check_code})
|
||||
# stdout += "----Submission Check 1-----\n"
|
||||
submission_result = implementation.run(env=env, entry="python test/submission_format_test.py")
|
||||
submission_check_out = submission_result.get_truncated_stdout()
|
||||
submission_check_out = submission_result.stdout
|
||||
submission_ret_code = submission_result.exit_code
|
||||
stdout += "\n" + submission_check_out
|
||||
|
||||
|
||||
@@ -56,7 +56,7 @@ class DataLoaderCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
test_code = (DIRNAME / "eval_tests" / "data_loader_test.txt").read_text()
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
result = implementation.run(env=env, entry=f"python {fname}")
|
||||
stdout = result.get_truncated_stdout()
|
||||
stdout = result.stdout
|
||||
ret_code = result.exit_code
|
||||
match = re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===(.*)", stdout, re.DOTALL)
|
||||
stdout_part_1, eda_output, stdout_part_2 = match.groups() if match else (stdout, None, "")
|
||||
|
||||
@@ -125,7 +125,7 @@ class WorkflowGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
implementation.inject_files(**{"test/submission_format_test.py": base_check_code})
|
||||
# stdout += "----Submission Check 1-----\n"
|
||||
submission_result = implementation.run(env=env, entry="python test/submission_format_test.py")
|
||||
submission_check_out = submission_result.get_truncated_stdout()
|
||||
submission_check_out = submission_result.stdout
|
||||
submission_ret_code = submission_result.exit_code
|
||||
stdout += "\n" + submission_check_out
|
||||
|
||||
|
||||
@@ -0,0 +1,391 @@
|
||||
"""
|
||||
LLM Fine-tuning CoSTEER Implementation
|
||||
|
||||
This module provides fine-tuning specific components for the CoSTEER framework,
|
||||
including evaluators and evolving strategies.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
import yaml
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEERMultiEvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DATA_SCRIPT_NAME,
|
||||
FT_PATHS,
|
||||
FT_TEST_PARAMS_FILE_NAME,
|
||||
FT_YAML_FILE_NAME,
|
||||
FTCoderCoSTEERSettings,
|
||||
)
|
||||
from rdagent.components.coder.finetune.eval import FTCoderEvaluator, FTDataEvaluator
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.scenarios.finetune.scen.llama_factory_manager import LLaMAFactory_manager
|
||||
from rdagent.scenarios.finetune.scen.utils import FinetuneDatasetDescriptor
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
|
||||
class LLMFinetuneEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
"""LLM Fine-tuning specific evolving strategy"""
|
||||
|
||||
def __init__(self, scen: Scenario, settings, *args, **kwargs):
|
||||
super().__init__(scen, settings)
|
||||
self.llama_factory_manager = LLaMAFactory_manager
|
||||
|
||||
def implement_func_list(self) -> list[Callable]:
|
||||
return [self.implement_data, self.implement_lf_config]
|
||||
|
||||
def implement_data(
|
||||
self,
|
||||
target_task: Task,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
prev_task_feedback: CoSTEERSingleFeedback | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Generate data processing script based on task.
|
||||
|
||||
This method generates a Python script that processes seed datasets
|
||||
and outputs a data.json file in Alpaca format.
|
||||
|
||||
Returns:
|
||||
dict with "process_data.py" key containing the script code,
|
||||
or empty dict if data already exists.
|
||||
"""
|
||||
# Check if proposal decided to skip data processing (reuse SOTA's data processing script)
|
||||
if getattr(target_task, "skip_data_processing", False):
|
||||
# Defensive check: ensure data script actually exists before skipping
|
||||
script_exists = False
|
||||
if workspace is not None:
|
||||
script_exists = FT_DATA_SCRIPT_NAME in workspace.file_dict
|
||||
|
||||
if script_exists:
|
||||
logger.info("Proposal decided to skip data processing, reusing SOTA's data script")
|
||||
return {}
|
||||
else:
|
||||
logger.warning(
|
||||
"skip_data_processing=True but process_data.py not found in workspace, "
|
||||
"this indicates SOTA injection failed - system design issue"
|
||||
)
|
||||
# Don't fallback silently, let it fail early to expose the issue
|
||||
|
||||
# check whether the current code passes evaluation
|
||||
if (
|
||||
prev_task_feedback is not None
|
||||
and "FTDataEvaluator" in prev_task_feedback.source_feedback
|
||||
and prev_task_feedback.source_feedback["FTDataEvaluator"]
|
||||
):
|
||||
logger.info("Previous data processing code passed evaluation, skipping regeneration")
|
||||
return {}
|
||||
|
||||
# build former failed trace
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[target_task.get_task_information()]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(FT_YAML_FILE_NAME)
|
||||
!= workspace.file_dict.get(FT_YAML_FILE_NAME)
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# Get dataset information for the task
|
||||
involving_datasets = getattr(target_task, "involving_datasets", [])
|
||||
dataset_info = self._get_dataset_info(involving_datasets, datasets_path=FT_PATHS.datasets)
|
||||
|
||||
# Generate data processing script using LLM
|
||||
system_prompt = T(".prompts:data_coder.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=target_task.get_task_information(),
|
||||
dataset_info=dataset_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
api_max_workers=FT_RD_SETTING.api_max_workers,
|
||||
datasets_path=FT_PATHS.datasets,
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
|
||||
user_prompt = T(".prompts:data_coder.user").r(
|
||||
datasets_path=FT_PATHS.datasets,
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
latest_code=workspace.file_dict.get(FT_DATA_SCRIPT_NAME, "") if workspace else "",
|
||||
latest_feedback=prev_task_feedback,
|
||||
involved_dataset_folder_desc={
|
||||
ds_name: FinetuneDatasetDescriptor().describe_dataset_folder(
|
||||
Path(FT_RD_SETTING.file_path) / "datasets" / ds_name, include_dataset_readme=True
|
||||
)
|
||||
for ds_name in involving_datasets
|
||||
},
|
||||
)
|
||||
|
||||
script_code = APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="python",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
logger.info(f"Generated data processing script ({len(script_code)} chars)")
|
||||
|
||||
return {FT_DATA_SCRIPT_NAME: script_code}
|
||||
|
||||
def _get_dataset_info(self, involving_datasets: list[str], datasets_path: str = None) -> str:
|
||||
"""Read dataset_info.json and return information for specified datasets.
|
||||
|
||||
Handles unified tasks structure:
|
||||
- readme: Dataset README content
|
||||
- file_tree: Directory structure
|
||||
- total_samples: Total sample count
|
||||
- tasks: Dict of task info (use "_root" for root-level data files)
|
||||
|
||||
Args:
|
||||
involving_datasets: List of dataset names to include
|
||||
datasets_path: Base path for datasets (e.g., "/assets/datasets/")
|
||||
"""
|
||||
datasets_dir = Path(FT_RD_SETTING.file_path) / "datasets"
|
||||
dataset_info_path = datasets_dir / "dataset_info.json"
|
||||
|
||||
# Use provided path or get from config
|
||||
if datasets_path is None:
|
||||
datasets_path = FT_PATHS.datasets
|
||||
|
||||
if not dataset_info_path.exists():
|
||||
logger.warning(f"dataset_info.json not found at {dataset_info_path}")
|
||||
return "No dataset information available."
|
||||
|
||||
try:
|
||||
with open(dataset_info_path, "r", encoding="utf-8") as f:
|
||||
all_dataset_info = json.load(f)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read dataset_info.json: {e}")
|
||||
return f"Error reading dataset info: {e}"
|
||||
|
||||
# Filter to only involved datasets, or use all if none specified
|
||||
if involving_datasets:
|
||||
filtered_info = {name: info for name, info in all_dataset_info.items() if name in involving_datasets}
|
||||
else:
|
||||
filtered_info = all_dataset_info
|
||||
|
||||
if not filtered_info:
|
||||
return "No matching datasets found in dataset_info.json."
|
||||
|
||||
# Format dataset info for the prompt
|
||||
info_parts = []
|
||||
for name, info in filtered_info.items():
|
||||
info_text = f"### Dataset: {name}\n"
|
||||
# IMPORTANT: Tell LLM the full path to dataset directory
|
||||
dataset_full_path = f"{datasets_path}{name}/"
|
||||
info_text += f"- **Dataset path**: `{dataset_full_path}` (each dataset has its own subdirectory)\n"
|
||||
info_text += f"- Total samples: {info.get('total_samples', 'N/A')}\n"
|
||||
info_text += f"- Size: {info.get('total_size_mb', 'N/A')} MB\n"
|
||||
|
||||
# File tree for understanding directory structure
|
||||
if info.get("file_tree"):
|
||||
file_tree = info["file_tree"]
|
||||
# Truncate if too long
|
||||
if len(file_tree) > 1000:
|
||||
file_tree = file_tree[:1000] + "\n..."
|
||||
info_text += f"\n**File Structure** (relative to `{dataset_full_path}`):\n```\n{file_tree}\n```\n"
|
||||
|
||||
# Handle unified tasks structure
|
||||
tasks = info.get("tasks", {})
|
||||
if tasks:
|
||||
info_text += "\n**Tasks:**\n"
|
||||
for task_name, task_info in tasks.items():
|
||||
# "_root" indicates data files are in root directory
|
||||
display_name = "(root)" if task_name == "_root" else task_name
|
||||
info_text += f"\n#### {display_name}\n"
|
||||
# Show full paths for data files
|
||||
files = task_info.get("files", [])
|
||||
info_text += f"- Files: {files}\n"
|
||||
if files:
|
||||
info_text += f" - Full path example: `{dataset_full_path}{files[0]}`\n"
|
||||
info_text += f"- Sample count: {task_info.get('sample_count', 'N/A')}\n"
|
||||
if task_info.get("column_stats"):
|
||||
# Show key token stats
|
||||
stats_summary = []
|
||||
for col, stats in task_info["column_stats"].items():
|
||||
if stats.get("p50_tokens", 0) > 0:
|
||||
stats_summary.append(f"{col}: p50={stats['p50_tokens']}, p99={stats['p99_tokens']}")
|
||||
if stats_summary:
|
||||
info_text += f"- Token stats: {'; '.join(stats_summary[:5])}\n"
|
||||
|
||||
# README excerpt
|
||||
if info.get("readme"):
|
||||
readme = info["readme"]
|
||||
if len(readme) > 500:
|
||||
readme = readme[:500] + "..."
|
||||
info_text += f"\n**README:**\n{readme}\n"
|
||||
|
||||
info_parts.append(info_text)
|
||||
|
||||
return "\n".join(info_parts)
|
||||
|
||||
def implement_lf_config(
|
||||
self,
|
||||
target_task: Task,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
prev_task_feedback: CoSTEERSingleFeedback | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Implement a single fine-tuning task by generating LlamaFactory config"""
|
||||
if prev_task_feedback is not None and prev_task_feedback.source_feedback.get("FTCoderEvaluator", False):
|
||||
logger.info("Previous training code passed evaluation, skipping regeneration")
|
||||
return {}
|
||||
|
||||
task_info = target_task.get_task_information()
|
||||
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[task_info] if queried_knowledge is not None else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(FT_YAML_FILE_NAME)
|
||||
!= workspace.file_dict.get(FT_YAML_FILE_NAME)
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# Get task parameters from the task object
|
||||
base_model = getattr(target_task, "base_model")
|
||||
|
||||
# Use LLM to generate LlamaFactory config YAML
|
||||
# Coder will decide method based on hypothesis and available parameters
|
||||
config_files = self._generate_llamafactory_config_with_llm(
|
||||
base_model=base_model,
|
||||
task_info=task_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge,
|
||||
prev_feedback=prev_task_feedback,
|
||||
workspace=workspace,
|
||||
)
|
||||
|
||||
# Return generated config files directly - validation happens in evaluator
|
||||
return config_files
|
||||
|
||||
def _generate_llamafactory_config_with_llm(
|
||||
self,
|
||||
base_model: str,
|
||||
task_info: str = "",
|
||||
queried_former_failed_knowledge: tuple = None,
|
||||
prev_feedback=None,
|
||||
workspace=None,
|
||||
) -> dict[str, str]:
|
||||
"""Generate LlamaFactory configuration YAML using LLM"""
|
||||
|
||||
# Query LLaMA Factory parameters: shared params once + method-specific params
|
||||
available_methods = self.llama_factory_manager.methods
|
||||
shared_params = self.llama_factory_manager.format_shared_params()
|
||||
|
||||
# Format method-specific parameters only (no duplication of shared params)
|
||||
methods_specific_params = {}
|
||||
for method in available_methods:
|
||||
methods_specific_params[method] = self.llama_factory_manager.format_method_specific_params(method)
|
||||
|
||||
# Use environment-aware paths (Docker vs Conda)
|
||||
# Note: datasets_path in finetune_coder uses workspace path where processed
|
||||
# data.json and dataset_info.json are located (generated by FTDataEvaluator)
|
||||
|
||||
# Generate prompts using templates with all required parameters
|
||||
system_prompt = T(".prompts:finetune_coder.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=task_info,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
available_methods=", ".join(available_methods),
|
||||
shared_params=shared_params,
|
||||
methods_specific_params=methods_specific_params,
|
||||
)
|
||||
|
||||
# Read data_stats.json from workspace (injected by FTDataEvaluator)
|
||||
data_stats = workspace.file_dict.get("data_stats.json", "")
|
||||
|
||||
user_prompt = T(".prompts:finetune_coder.user").r(
|
||||
latest_code=workspace.file_dict.get(FT_YAML_FILE_NAME, ""),
|
||||
latest_feedback=prev_feedback,
|
||||
base_model=base_model,
|
||||
models_path=FT_PATHS.models,
|
||||
datasets_path=FT_PATHS.workspace, # Training config uses workspace path for processed data
|
||||
workspace_path=FT_PATHS.workspace,
|
||||
deepspeed_path=FT_PATHS.deepspeed,
|
||||
data_stats=data_stats,
|
||||
has_think_token=self.scen.model_info.get("has_think_token", False),
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
|
||||
# Call LLM to generate config (multi-turn)
|
||||
session = APIBackend().build_chat_session(session_system_prompt=system_prompt)
|
||||
|
||||
# Turn 1: Generate main training config
|
||||
train_config_yaml = session.build_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="yaml",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
|
||||
# Validate main config YAML syntax
|
||||
yaml.safe_load(train_config_yaml)
|
||||
logger.info("Extracted main YAML config successfully")
|
||||
|
||||
# Turn 2: Generate test parameters (test_params.yaml)
|
||||
test_params_prompt = T(".prompts:finetune_coder.user_test_params").r(workspace_path=FT_PATHS.workspace)
|
||||
test_params_yaml = session.build_chat_completion(
|
||||
user_prompt=test_params_prompt,
|
||||
json_mode=False,
|
||||
code_block_language="yaml",
|
||||
code_block_fallback=False,
|
||||
)
|
||||
|
||||
# Validate test params YAML syntax
|
||||
yaml.safe_load(test_params_yaml)
|
||||
logger.info("Extracted test params YAML successfully")
|
||||
|
||||
return {FT_YAML_FILE_NAME: train_config_yaml, FT_TEST_PARAMS_FILE_NAME: test_params_yaml}
|
||||
|
||||
|
||||
class LLMFinetuneCoSTEER(CoSTEER):
|
||||
"""LLM Fine-tuning CoSTEER implementation"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
settings = FTCoderCoSTEERSettings()
|
||||
eva = CoSTEERMultiEvaluator([FTDataEvaluator(scen=scen), FTCoderEvaluator(scen=scen)], scen=scen)
|
||||
es = LLMFinetuneEvolvingStrategy(scen=scen, settings=settings)
|
||||
|
||||
super().__init__(
|
||||
*args,
|
||||
settings=settings,
|
||||
eva=eva,
|
||||
es=es,
|
||||
evolving_version=2,
|
||||
scen=scen,
|
||||
max_loop=FT_RD_SETTING.coder_max_loop if hasattr(FT_RD_SETTING, "coder_max_loop") else 5,
|
||||
stop_eval_chain_on_fail=True, # finetune involve partial implementation.
|
||||
**kwargs,
|
||||
)
|
||||
@@ -0,0 +1,387 @@
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.scen.utils import _compute_column_stats
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import (
|
||||
BenchmarkCondaConf,
|
||||
BenchmarkCondaEnv,
|
||||
BenchmarkDockerConf,
|
||||
BenchmarkDockerEnv,
|
||||
DockerEnv,
|
||||
Env,
|
||||
FTCondaConf,
|
||||
FTCondaEnv,
|
||||
FTDockerEnv,
|
||||
)
|
||||
|
||||
|
||||
def is_docker_env(env: Env) -> bool:
|
||||
"""Check if the environment is Docker-based."""
|
||||
return isinstance(env, DockerEnv)
|
||||
|
||||
|
||||
def get_workspace_prefix(env: Env) -> str:
|
||||
"""Return workspace path prefix based on env type.
|
||||
|
||||
Docker uses /workspace as mount point, conda uses current directory.
|
||||
"""
|
||||
return "/workspace" if is_docker_env(env) else "."
|
||||
|
||||
|
||||
FT_YAML_FILE_NAME = "train.yaml"
|
||||
FT_DATA_PROC_FILE_NAME = "data_process.py"
|
||||
FT_DEBUG_YAML_FILE_NAME = "debug_train.yaml"
|
||||
FT_TEST_PARAMS_FILE_NAME = "test_params.yaml"
|
||||
FT_DATA_FILE_NAME = "data.json"
|
||||
FT_DATA_SCRIPT_NAME = "process_data.py"
|
||||
|
||||
# ENV Info: the path of the model and dataset in the container/environment
|
||||
FT_MODEL_PATH = "/assets/models"
|
||||
FT_DATASET_PATH = "/assets/datasets"
|
||||
|
||||
|
||||
def get_data_processing_cache_key(local_path: str | Path) -> list[list[str]]:
|
||||
"""Generate cache key based only on data processing script and dataset info.
|
||||
|
||||
This ensures that data processing results are reused as long as the script
|
||||
and dataset configuration remain unchanged, even if other files in the
|
||||
workspace (like training config) have been modified.
|
||||
"""
|
||||
content = []
|
||||
local_path = Path(local_path)
|
||||
# We only care about the script that generates data and the dataset configuration
|
||||
for filename in [FT_DATA_SCRIPT_NAME, "dataset_info.json"]:
|
||||
file_path = local_path / filename
|
||||
if file_path.exists():
|
||||
content.append([filename, file_path.read_text()])
|
||||
content.sort(key=lambda x: x[0])
|
||||
return content
|
||||
|
||||
|
||||
class FTPathConfig:
|
||||
"""Centralized path configuration for FT scenario.
|
||||
|
||||
Provides environment-aware paths for Docker vs Conda modes.
|
||||
Uses lazy evaluation (properties) to avoid import-time errors.
|
||||
|
||||
Usage:
|
||||
from rdagent.components.coder.finetune.conf import FT_PATHS
|
||||
|
||||
models_path = FT_PATHS.models # e.g., "/assets/models/" or "/path/to/finetune/models/"
|
||||
datasets_path = FT_PATHS.datasets # e.g., "/assets/datasets/" or "/path/to/finetune/datasets/"
|
||||
workspace_path = FT_PATHS.workspace # e.g., "/workspace/" or "./"
|
||||
"""
|
||||
|
||||
@property
|
||||
def is_docker(self) -> bool:
|
||||
"""Check if current environment is Docker-based."""
|
||||
# FIXME: the env should work in same way for docker and conda env.
|
||||
# We should not expose the env type everywhere.
|
||||
return FTCoderCoSTEERSettings().env_type == "docker"
|
||||
|
||||
@property
|
||||
def models(self) -> str:
|
||||
"""Model directory path (with trailing slash)."""
|
||||
if self.is_docker:
|
||||
return FT_MODEL_PATH + "/"
|
||||
return str(FT_RD_SETTING.file_path / "models") + "/"
|
||||
|
||||
@property
|
||||
def datasets(self) -> str:
|
||||
"""Dataset directory path for raw datasets (with trailing slash)."""
|
||||
if self.is_docker:
|
||||
return FT_DATASET_PATH + "/"
|
||||
return str(FT_RD_SETTING.file_path / "datasets") + "/"
|
||||
|
||||
@property
|
||||
def workspace(self) -> str:
|
||||
"""Workspace path prefix for prompts (with trailing slash)."""
|
||||
return "/workspace/" if self.is_docker else "./"
|
||||
|
||||
@property
|
||||
def deepspeed(self) -> str:
|
||||
"""DeepSpeed config directory."""
|
||||
if self.is_docker:
|
||||
return "/app/examples/deepspeed/"
|
||||
# Conda mode: use bundled deepspeed configs in project
|
||||
# Path: conf.py -> finetune -> coder -> components -> rdagent -> scenarios/finetune/env/conda/deepspeed
|
||||
rdagent_root = Path(__file__).parent.parent.parent.parent
|
||||
deepspeed_path = rdagent_root / "scenarios" / "finetune" / "env" / "conda" / "deepspeed"
|
||||
return str(deepspeed_path) + "/" if deepspeed_path.exists() else ""
|
||||
|
||||
|
||||
# Singleton instance for path configuration
|
||||
FT_PATHS = FTPathConfig()
|
||||
|
||||
|
||||
class FTCoderCoSTEERSettings(CoSTEERSettings):
|
||||
"""LLM Fine-tuning CoSTEER settings"""
|
||||
|
||||
class Config:
|
||||
env_prefix = "FT_Coder_CoSTEER_"
|
||||
|
||||
max_seconds_multiplier: int = 8
|
||||
"""LLM training takes longer, use higher multiplier"""
|
||||
|
||||
env_type: str = "docker"
|
||||
"""Environment type for LLM fine-tuning (docker/conda)"""
|
||||
|
||||
extra_eval: list[str] = []
|
||||
"""Extra evaluators"""
|
||||
|
||||
|
||||
def _get_standard_ft_volumes() -> dict:
|
||||
"""Get standard mount volume configuration for LLM finetune environments.
|
||||
|
||||
Creates standard directory mappings:
|
||||
- models -> /assets/models (ro)
|
||||
- datasets -> /assets/datasets (ro)
|
||||
|
||||
Returns:
|
||||
Dictionary of local_path -> docker_mount_config mappings
|
||||
"""
|
||||
base_path = Path(FT_RD_SETTING.file_path)
|
||||
volumes = {}
|
||||
|
||||
# Read-only mounts for data and models
|
||||
readonly_mounts = [
|
||||
("models", FT_MODEL_PATH),
|
||||
("datasets", FT_DATASET_PATH),
|
||||
]
|
||||
|
||||
for local_dir, docker_path in readonly_mounts:
|
||||
local_path = base_path / local_dir
|
||||
volumes[str(local_path)] = {"bind": docker_path, "mode": "ro"}
|
||||
|
||||
return volumes
|
||||
|
||||
|
||||
def get_ft_env(
|
||||
extra_volumes: dict = {},
|
||||
operation: str = "full_training",
|
||||
enable_cache: bool | None = None,
|
||||
) -> Env:
|
||||
"""LLM finetune dedicated environment construction function.
|
||||
|
||||
Automatically includes standard finetune volume mounts:
|
||||
- models -> /assets/models (ro)
|
||||
- datasets -> /assets/datasets (ro)
|
||||
- output -> /workspace/output (rw, auto-created)
|
||||
|
||||
Note: .llama_factory_info is no longer automatically mounted.
|
||||
Pass llama_factory_info volume via extra_volumes when needed.
|
||||
|
||||
Args:
|
||||
extra_volumes: Additional volume mounts beyond standard ones
|
||||
operation: Operation type for timeout selection.
|
||||
- "data_processing": Data processing (data_processing_timeout)
|
||||
- "micro_batch": Micro-batch test (micro_batch_timeout)
|
||||
- "full_training": Full training (full_timeout)
|
||||
enable_cache: Whether to enable caching (None means use config value)
|
||||
|
||||
Returns:
|
||||
Configured environment ready for use
|
||||
"""
|
||||
|
||||
conf = FTCoderCoSTEERSettings()
|
||||
|
||||
# Select timeout based on operation type
|
||||
timeout_map = {
|
||||
"data_processing": FT_RD_SETTING.data_processing_timeout,
|
||||
"debug_data_processing": FT_RD_SETTING.debug_data_processing_timeout,
|
||||
"micro_batch": FT_RD_SETTING.micro_batch_timeout,
|
||||
"full_training": FT_RD_SETTING.full_timeout,
|
||||
}
|
||||
running_timeout_period = timeout_map.get(operation, FT_RD_SETTING.full_timeout)
|
||||
|
||||
# Use config value if enable_cache is not explicitly provided
|
||||
if enable_cache is None:
|
||||
enable_cache = FT_RD_SETTING.docker_enable_cache
|
||||
|
||||
# Use dedicated LLM docker or conda env based on config
|
||||
if conf.env_type == "docker":
|
||||
env = FTDockerEnv()
|
||||
# Docker mode: setup volume mounts for models/datasets
|
||||
standard_volumes = _get_standard_ft_volumes()
|
||||
combined_volumes = standard_volumes.copy()
|
||||
combined_volumes.update(extra_volumes)
|
||||
env.conf.extra_volumes = combined_volumes
|
||||
elif conf.env_type == "conda":
|
||||
env = FTCondaEnv(conf=FTCondaConf()) # Auto-installs dependencies if env doesn't exist
|
||||
# Conda mode: no volume mounts needed, use local paths directly
|
||||
# extra_volumes are ignored in conda mode
|
||||
else:
|
||||
raise ValueError(f"Unknown env type: {conf.env_type}")
|
||||
|
||||
env.conf.running_timeout_period = running_timeout_period
|
||||
env.conf.enable_cache = enable_cache
|
||||
env.prepare()
|
||||
return env
|
||||
|
||||
|
||||
def get_data_processing_env(
|
||||
enable_cache: bool | None = None,
|
||||
is_debug: bool = False,
|
||||
) -> tuple[Env, dict]:
|
||||
"""Get environment for data processing scripts with LLM API access.
|
||||
|
||||
This environment is configured for running data processing scripts that may
|
||||
need to call LLM APIs. It includes:
|
||||
- Standard finetune volume mounts (datasets, models)
|
||||
- LLM API environment variables (OPENAI_API_KEY, OPENAI_BASE_URL, etc.)
|
||||
|
||||
Args:
|
||||
enable_cache: Whether to enable Docker caching
|
||||
is_debug: Whether running in debug mode (shorter timeout, default 20 min vs 1 hour)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_vars) where env_vars contains LLM API keys
|
||||
to be passed to env.run() as the env parameter
|
||||
"""
|
||||
env = get_ft_env(
|
||||
operation="debug_data_processing" if is_debug else "data_processing",
|
||||
enable_cache=enable_cache,
|
||||
)
|
||||
|
||||
# Collect LLM API environment variables to pass to env.run()
|
||||
llm_env_vars = {"PYTHONPATH": "./"} # Base env var
|
||||
|
||||
# Pass OPENAI_API_KEY directly
|
||||
if api_key := os.getenv("OPENAI_API_KEY"):
|
||||
llm_env_vars["OPENAI_API_KEY"] = api_key
|
||||
|
||||
# Read OPENAI_API_BASE from env, but pass as OPENAI_BASE_URL (OpenAI SDK expects this name)
|
||||
if api_base := os.getenv("OPENAI_API_BASE"):
|
||||
llm_env_vars["OPENAI_BASE_URL"] = api_base
|
||||
|
||||
# Pass model pools as JSON environment variables for load balancing
|
||||
llm_env_vars["STRONG_MODEL_POOL"] = json.dumps(FT_RD_SETTING.strong_models)
|
||||
llm_env_vars["WEAK_MODEL_POOL"] = json.dumps(FT_RD_SETTING.weak_models)
|
||||
|
||||
return env, llm_env_vars
|
||||
|
||||
|
||||
def clear_workspace(workspace: FBWorkspace, env: Env) -> None:
|
||||
"""
|
||||
Clean the files in LLM finetune workspace.
|
||||
Only keeps the files that are injected by the coder (in workspace.file_dict) and `logs`.
|
||||
|
||||
Args:
|
||||
workspace: The workspace object containing the file dictionary.
|
||||
env: The environment to execute the clean command in.
|
||||
"""
|
||||
target_path = workspace.workspace_path
|
||||
if not target_path.exists():
|
||||
return
|
||||
|
||||
# The cache_path is created when mounting, so the permissions changes does not work.
|
||||
keep_items = {"logs", T("scenarios.data_science.share:scen.cache_path").r()}
|
||||
|
||||
for file_path in workspace.file_dict.keys():
|
||||
top_level = Path(file_path).parts[0]
|
||||
keep_items.add(top_level)
|
||||
|
||||
remove_items = []
|
||||
for item in target_path.iterdir():
|
||||
if item.name in keep_items:
|
||||
continue
|
||||
remove_items.append(item.name)
|
||||
|
||||
if remove_items:
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
# Construct rm command with all items to remove
|
||||
# Items are relative to workspace root inside the env
|
||||
items_str = " ".join([f"'{ws_prefix}/{item}'" for item in remove_items])
|
||||
cmd = f"rm -rf {items_str}"
|
||||
workspace.execute(env=env, entry=cmd)
|
||||
|
||||
|
||||
def get_benchmark_env(
|
||||
extra_volumes: dict = {},
|
||||
timeout: int | None = None,
|
||||
) -> Env:
|
||||
"""OpenCompass benchmark environment construction function.
|
||||
|
||||
Supports both Docker and conda environments based on FT_Coder_CoSTEER_env_type.
|
||||
|
||||
Args:
|
||||
extra_volumes: Additional volume mounts (only used in Docker mode)
|
||||
timeout: Running timeout in seconds (None uses config default)
|
||||
|
||||
Returns:
|
||||
Configured environment ready for benchmark evaluation
|
||||
"""
|
||||
conf = FTCoderCoSTEERSettings()
|
||||
|
||||
# Use benchmark-specific timeout or config default
|
||||
if timeout is None:
|
||||
# 0 means no timeout, use 7 days as practical "infinite"
|
||||
timeout = FT_RD_SETTING.benchmark_timeout if FT_RD_SETTING.benchmark_timeout > 0 else 86400 * 7
|
||||
|
||||
benchmark_volumes = {}
|
||||
# Setup finetune share folder mount for models
|
||||
(FT_RD_SETTING.file_path / "benchmarks").mkdir(parents=True, exist_ok=True)
|
||||
# NOTE: we choose a folder in the workspace as the mount point due to we may run multiple instances in same
|
||||
# host machine. If conda env is used, the mount point will conflict with each other.
|
||||
benchmark_volumes[str((FT_RD_SETTING.file_path / "benchmarks").resolve())] = {
|
||||
"bind": "./benchmarks",
|
||||
"mode": "rw",
|
||||
}
|
||||
env_dict = {"COMPASS_DATA_CACHE": "./benchmarks/opencompass_data"}
|
||||
# Mount models directory for LoRA base model access (vLLM needs base model config)
|
||||
models_path = FT_RD_SETTING.file_path / "models"
|
||||
if models_path.exists():
|
||||
benchmark_volumes[str(models_path.resolve())] = {"bind": FT_MODEL_PATH, "mode": "ro"}
|
||||
benchmark_volumes.update(extra_volumes)
|
||||
|
||||
if conf.env_type == "docker":
|
||||
docker_conf = BenchmarkDockerConf()
|
||||
docker_conf.running_timeout_period = timeout
|
||||
docker_conf.extra_volumes = benchmark_volumes
|
||||
docker_conf.env_dict = env_dict
|
||||
env = BenchmarkDockerEnv(conf=docker_conf)
|
||||
elif conf.env_type == "conda":
|
||||
# NOTE:
|
||||
# We assume user has the permissions to create the softlink in the target directory.
|
||||
# If we have requirements in the future, we suggest make the target directory configurable in BenchmarkCondaConf.
|
||||
conda_conf = BenchmarkCondaConf()
|
||||
conda_conf.running_timeout_period = timeout
|
||||
conda_conf.extra_volumes = benchmark_volumes
|
||||
conda_conf.env_dict = env_dict
|
||||
env = BenchmarkCondaEnv(conf=conda_conf) # Auto-installs dependencies if env doesn't exist
|
||||
else:
|
||||
raise ValueError(f"Unknown env type: {conf.env_type}")
|
||||
|
||||
env.prepare()
|
||||
return env
|
||||
|
||||
|
||||
def inject_data_stats(implementation: FBWorkspace, data: list, stdout: str) -> None:
|
||||
"""Compute token statistics and inject data_stats.json.
|
||||
|
||||
Used by both FTDataEvaluator (coding stage) and FTRunnerEvaluator (running stage).
|
||||
|
||||
Args:
|
||||
implementation: The workspace to inject data_stats.json into
|
||||
data: The data list from data.json
|
||||
stdout: The stdout from process_data.py execution
|
||||
"""
|
||||
token_stats = _compute_column_stats(data)
|
||||
|
||||
data_stats = {
|
||||
"total_samples": len(data),
|
||||
"token_stats": token_stats,
|
||||
"stdout_summary": stdout,
|
||||
}
|
||||
|
||||
implementation.inject_files(**{"data_stats.json": json.dumps(data_stats, indent=2)})
|
||||
logger.info(f"Injected data_stats.json with {len(data)} samples")
|
||||
@@ -0,0 +1,399 @@
|
||||
"""
|
||||
LLM Fine-tuning Evaluation Components
|
||||
|
||||
Provides simplified evaluation: parameter filtering + micro-batch testing.
|
||||
No redundant LLM feedback generation - test results speak for themselves.
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from rdagent.app.finetune.llm.conf import FT_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DATA_FILE_NAME,
|
||||
FT_DATA_SCRIPT_NAME,
|
||||
FT_YAML_FILE_NAME,
|
||||
clear_workspace,
|
||||
get_data_processing_cache_key,
|
||||
get_data_processing_env,
|
||||
get_ft_env,
|
||||
get_workspace_prefix,
|
||||
inject_data_stats,
|
||||
)
|
||||
from rdagent.components.coder.finetune.unified_validator import (
|
||||
SYSTEM_MANAGED_PARAMS,
|
||||
LLMConfigValidator,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
|
||||
class FTDataEvaluator(CoSTEEREvaluator):
|
||||
"""Evaluator for data processing results.
|
||||
|
||||
This evaluator:
|
||||
1. Executes the process_data.py script in Docker
|
||||
2. Validates the output data.json file
|
||||
3. Generates dataset_info.json for LlamaFactory
|
||||
"""
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: Optional[QueriedKnowledge] = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Evaluate data processing implementation with LLM feedback."""
|
||||
|
||||
script_code = implementation.file_dict.get(FT_DATA_SCRIPT_NAME, "")
|
||||
data_json_path = implementation.workspace_path / FT_DATA_FILE_NAME
|
||||
execution_output = ""
|
||||
exit_code = 0
|
||||
data = None
|
||||
error_msg = None
|
||||
|
||||
# Step 1: Check script exists
|
||||
if not script_code:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution=f"No {FT_DATA_SCRIPT_NAME} found",
|
||||
return_checking="Data processing script missing",
|
||||
code="Please generate a data processing script first.",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTDataEvaluator")
|
||||
return feedback
|
||||
|
||||
# NOTE: we depends cache for speeding up the process of data generation.
|
||||
# So we clear the workspace every time.
|
||||
|
||||
# Step 3: Execute script in DEBUG mode (generates ~10 samples for fast validation)
|
||||
env, env_vars = get_data_processing_env(is_debug=True)
|
||||
|
||||
# Clear workspace (except logs and file_dict items) before data processing
|
||||
clear_workspace(implementation, env=env)
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
|
||||
# Use FTWorkspace.run() for unified Docker logging
|
||||
# --debug flag tells the script to generate only ~10 samples
|
||||
result = implementation.run(
|
||||
env=env,
|
||||
entry=f"python {ws_prefix}/{FT_DATA_SCRIPT_NAME} --debug",
|
||||
env_vars=env_vars,
|
||||
cache_key_extra_func=get_data_processing_cache_key,
|
||||
cache_files_to_extract=[FT_DATA_FILE_NAME],
|
||||
)
|
||||
execution_output = result.stdout if hasattr(result, "stdout") else str(result)
|
||||
exit_code = result.exit_code if hasattr(result, "exit_code") else -1
|
||||
|
||||
# Step 4: Validate output
|
||||
if not data_json_path.exists():
|
||||
error_msg = f"{FT_DATA_FILE_NAME} not generated"
|
||||
else:
|
||||
validation_result = self._validate_data_json(data_json_path)
|
||||
if not validation_result["valid"]:
|
||||
error_msg = validation_result["error"]
|
||||
else:
|
||||
self._update_dataset_info(implementation, validation_result["sample_count"])
|
||||
|
||||
# Step 5: Load data if valid
|
||||
if error_msg is None and data_json_path.exists():
|
||||
with open(data_json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Step 5.5: Compute token stats and inject data_stats for yaml coder
|
||||
if data is not None and error_msg is None:
|
||||
inject_data_stats(implementation, data, execution_output)
|
||||
|
||||
# Step 6: Generate LLM feedback
|
||||
# Truncate stdout from end for LLM (summary at the end is more useful)
|
||||
stdout_summary = execution_output[-1500:] if execution_output else ""
|
||||
return self._generate_llm_feedback(
|
||||
target_task=target_task,
|
||||
script_code=script_code if error_msg else "", # Only show script on error
|
||||
stdout=stdout_summary, # Always show summary (truncated from end)
|
||||
exit_code=exit_code,
|
||||
data=data,
|
||||
error_msg=error_msg,
|
||||
queried_knowledge=queried_knowledge,
|
||||
raw_stdout=execution_output, # Full log for UI
|
||||
)
|
||||
|
||||
def _generate_llm_feedback(
|
||||
self,
|
||||
target_task: Task,
|
||||
script_code: str,
|
||||
stdout: str,
|
||||
exit_code: int,
|
||||
data: Optional[list],
|
||||
error_msg: Optional[str],
|
||||
queried_knowledge: Optional[QueriedKnowledge],
|
||||
raw_stdout: str = "",
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Generate LLM-based feedback for data processing evaluation."""
|
||||
|
||||
# Prepare data statistics and samples
|
||||
if data:
|
||||
stats = self._analyze_data_quality(data)
|
||||
data_stats = json.dumps(stats, indent=2)
|
||||
sampled_data = self._sample_data(data)
|
||||
data_samples = json.dumps(sampled_data, indent=2, ensure_ascii=False)
|
||||
sample_count = len(sampled_data)
|
||||
total_samples = len(data)
|
||||
else:
|
||||
data_stats = json.dumps({"error": error_msg or "No data generated"})
|
||||
data_samples = "[]"
|
||||
sample_count = 0
|
||||
total_samples = 0
|
||||
|
||||
# Extract similar successful knowledge
|
||||
queried_similar_successful_knowledge = []
|
||||
if queried_knowledge is not None:
|
||||
task_info = target_task.get_task_information()
|
||||
queried_similar_successful_knowledge = queried_knowledge.task_to_similar_task_successful_knowledge.get(
|
||||
task_info, []
|
||||
)
|
||||
|
||||
# Build prompts
|
||||
system_prompt = T(".prompts:data_eval.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
upper_data_size_limit=FT_RD_SETTING.upper_data_size_limit,
|
||||
force_think_token=FT_RD_SETTING.force_think_token,
|
||||
)
|
||||
user_prompt = T(".prompts:data_eval.user").r(
|
||||
task_desc=target_task.get_task_information(),
|
||||
script_code=script_code,
|
||||
exit_code=exit_code,
|
||||
stdout=stdout[:3000] if stdout else "", # Empty string triggers {% if stdout %} = false
|
||||
data_stats=data_stats,
|
||||
sample_count=sample_count,
|
||||
total_samples=total_samples,
|
||||
data_samples=data_samples,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Generating LLM feedback for data evaluation (samples: {total_samples}, has_error: {bool(error_msg)})"
|
||||
)
|
||||
|
||||
feedback = build_cls_from_json_with_retry(
|
||||
CoSTEERSingleFeedback,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
init_kwargs_update_func=CoSTEERSingleFeedback.val_and_update_init_dict,
|
||||
)
|
||||
|
||||
# NOTE: 0 exit code is a hard criteria for success
|
||||
if exit_code != 0:
|
||||
feedback.final_decision = False
|
||||
|
||||
feedback.raw_execution = raw_stdout
|
||||
feedback.source_feedback[self.__class__.__name__] = feedback.final_decision
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTDataEvaluator")
|
||||
return feedback
|
||||
|
||||
def _validate_data_json(self, data_json_path: Path) -> dict:
|
||||
"""Validate data.json file format and content."""
|
||||
try:
|
||||
with open(data_json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Must be a non-empty list
|
||||
if not isinstance(data, list):
|
||||
return {"valid": False, "error": "data.json must be a JSON array", "sample_count": 0}
|
||||
|
||||
if len(data) == 0:
|
||||
return {"valid": False, "error": "data.json is empty", "sample_count": 0}
|
||||
|
||||
# Check required fields in samples
|
||||
required_fields = ["instruction", "output"]
|
||||
for i, sample in enumerate(data[:10]): # Check first 10 samples
|
||||
if not isinstance(sample, dict):
|
||||
return {"valid": False, "error": f"Sample {i} is not a dict", "sample_count": 0}
|
||||
|
||||
missing = [f for f in required_fields if f not in sample]
|
||||
if missing:
|
||||
return {"valid": False, "error": f"Sample {i} missing fields: {missing}", "sample_count": 0}
|
||||
|
||||
# Check for empty required fields
|
||||
for field in required_fields:
|
||||
if not sample.get(field):
|
||||
return {
|
||||
"valid": False,
|
||||
"error": f"Sample {i} has empty '{field}' field",
|
||||
"sample_count": 0,
|
||||
}
|
||||
|
||||
return {"valid": True, "error": None, "sample_count": len(data)}
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
return {"valid": False, "error": f"Invalid JSON: {e}", "sample_count": 0}
|
||||
except Exception as e:
|
||||
return {"valid": False, "error": f"Error reading file: {e}", "sample_count": 0}
|
||||
|
||||
def _update_dataset_info(self, implementation: FBWorkspace, sample_count: int):
|
||||
"""Generate dataset_info.json for LlamaFactory to use the processed data.
|
||||
|
||||
Note: LlamaFactory's columns mapping uses internal names (prompt, query, response)
|
||||
that map to the actual column names in the data file (instruction, input, output).
|
||||
See: https://github.com/hiyouga/LLaMA-Factory/blob/main/src/llamafactory/data/parser.py
|
||||
"""
|
||||
dataset_info = {
|
||||
"processed_data": {
|
||||
"file_name": FT_DATA_FILE_NAME,
|
||||
"formatting": "alpaca",
|
||||
"columns": {
|
||||
"prompt": "instruction",
|
||||
"query": "input",
|
||||
"response": "output",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
implementation.inject_files(**{"dataset_info.json": json.dumps(dataset_info, indent=2)})
|
||||
logger.info(f"Updated dataset_info.json with processed_data ({sample_count} samples)")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to update dataset_info.json: {e}")
|
||||
|
||||
def _sample_data(self, data: list, n: int = 5) -> list:
|
||||
"""Random sampling for LLM evaluation."""
|
||||
if len(data) <= n:
|
||||
return data
|
||||
return random.sample(data, n)
|
||||
|
||||
def _analyze_data_quality(self, data: list) -> dict:
|
||||
"""Analyze data quality statistics for all fields."""
|
||||
if not data:
|
||||
return {"total_samples": 0, "error": "Empty data"}
|
||||
|
||||
# Analyze length stats for all standard fields
|
||||
fields = ["instruction", "input", "output"]
|
||||
stats = {"total_samples": len(data)}
|
||||
|
||||
for field in fields:
|
||||
lens = [len(str(d.get(field, ""))) for d in data]
|
||||
empty_count = sum(1 for d in data if not d.get(field))
|
||||
stats[f"{field}_len"] = {
|
||||
"min": min(lens),
|
||||
"max": max(lens),
|
||||
"avg": round(sum(lens) / len(lens), 1),
|
||||
}
|
||||
stats[f"{field}_empty_ratio"] = round(empty_count / len(data) * 100, 1)
|
||||
|
||||
# Detect duplicates by full record (instruction + input + output)
|
||||
record_set = set(
|
||||
(str(d.get("instruction", "")), str(d.get("input", "")), str(d.get("output", ""))) for d in data
|
||||
)
|
||||
duplicate_count = len(data) - len(record_set)
|
||||
stats["duplicate_count"] = duplicate_count
|
||||
stats["duplicate_ratio"] = round(duplicate_count / len(data) * 100, 1)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
class FTCoderEvaluator(CoSTEEREvaluator):
|
||||
"""Evaluator for LLM fine-tuning implementations with simplified validation"""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERSingleFeedback:
|
||||
"""Evaluate LLM fine-tuning implementation with two-step validation"""
|
||||
|
||||
task_info = target_task.get_task_information()
|
||||
|
||||
# Check task history
|
||||
if queried_knowledge is not None:
|
||||
if task_info in queried_knowledge.success_task_to_knowledge_dict:
|
||||
return queried_knowledge.success_task_to_knowledge_dict[task_info].feedback
|
||||
elif task_info in queried_knowledge.failed_task_info_set:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution="Task failed too many times, skipping.",
|
||||
return_checking="Task failed too many times, skipping.",
|
||||
code="Task failed too many times, skipping.",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
|
||||
env = get_ft_env(operation="micro_batch")
|
||||
config_yaml = implementation.file_dict.get(FT_YAML_FILE_NAME, "")
|
||||
if not config_yaml:
|
||||
feedback = CoSTEERSingleFeedback(
|
||||
execution=f"No {FT_YAML_FILE_NAME} found",
|
||||
return_checking="Configuration file missing",
|
||||
code="No valid configuration file",
|
||||
final_decision=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
|
||||
# Two-step validation: parameter filtering + micro-batch test
|
||||
validation_result = LLMConfigValidator().validate_and_test(
|
||||
config_yaml=config_yaml, workspace=implementation, env=env
|
||||
)
|
||||
# NOTE: Docker execution is logged by FTWorkspace.run() automatically
|
||||
|
||||
# Update config with filtered version
|
||||
if validation_result.filtered_config != config_yaml:
|
||||
implementation.inject_files(**{FT_YAML_FILE_NAME: validation_result.filtered_config})
|
||||
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[target_task.get_task_information()]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
|
||||
system_prompt = T(".prompts:finetune_eval.system").r(
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
system_managed_params=SYSTEM_MANAGED_PARAMS,
|
||||
)
|
||||
user_prompt = T(".prompts:finetune_eval.user").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=target_task.get_task_information(),
|
||||
stdout=validation_result.execution_output or "No output",
|
||||
code_yaml=implementation.file_dict[FT_YAML_FILE_NAME],
|
||||
workspace_files="\n".join(
|
||||
[
|
||||
f"- {file.name} ({file.stat().st_size} bytes)"
|
||||
for file in implementation.workspace_path.rglob("*")
|
||||
if file.is_file() and "checkpoint" not in file.absolute().as_posix()
|
||||
]
|
||||
),
|
||||
)
|
||||
feedback = build_cls_from_json_with_retry(
|
||||
CoSTEERSingleFeedback,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
init_kwargs_update_func=CoSTEERSingleFeedback.val_and_update_init_dict,
|
||||
)
|
||||
|
||||
# Force failure if validation failed programmatically
|
||||
if not validation_result.success:
|
||||
feedback.final_decision = False
|
||||
logger.warning("FTCoderEvaluator: Forced final_decision=False due to validation failure")
|
||||
|
||||
feedback.raw_execution = validation_result.raw_stdout or ""
|
||||
feedback.source_feedback[self.__class__.__name__] = feedback.final_decision
|
||||
logger.log_object(feedback, tag="evaluator_feedback.FTCoderEvaluator")
|
||||
return feedback
|
||||
@@ -0,0 +1,39 @@
|
||||
"""
|
||||
LLM Fine-tuning Experiment Components
|
||||
|
||||
Defines tasks for LLM fine-tuning following data science pattern.
|
||||
"""
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
|
||||
|
||||
class FTTask(CoSTEERTask):
|
||||
"""Training task class for LLM fine-tuning operations - follows data science pattern"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_model: str,
|
||||
description: str,
|
||||
benchmark: str,
|
||||
involving_datasets: Optional[List[str]] = None,
|
||||
skip_data_processing: bool = False,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(name="LLM-Fine-Tuning", description=description, *args, **kwargs)
|
||||
self.base_model = base_model
|
||||
self.benchmark = benchmark
|
||||
self.involving_datasets = involving_datasets or []
|
||||
self.skip_data_processing = skip_data_processing # If True, reuse SOTA's data processing script
|
||||
|
||||
def get_task_information(self) -> str:
|
||||
"""Get task information for coder prompt generation"""
|
||||
task_desc = f"""name: {self.name}
|
||||
description: {self.description}
|
||||
base_model: {self.base_model}
|
||||
"""
|
||||
if self.involving_datasets:
|
||||
task_desc += f"involving_datasets: {self.involving_datasets}\n"
|
||||
return task_desc
|
||||
@@ -0,0 +1,742 @@
|
||||
data_coder:
|
||||
system: |-
|
||||
You are a world-class data engineer specializing in preparing training data for large language model fine-tuning.
|
||||
Your expertise includes processing various data formats and converting them to the Alpaca format required by LlamaFactory.
|
||||
|
||||
# Part 1: Context
|
||||
|
||||
## 1.1 Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
## 1.2 Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## 1.3 Available Datasets
|
||||
The following datasets are available for processing:
|
||||
{{ dataset_info }}
|
||||
|
||||
## 1.4 Priority Rules (CRITICAL)
|
||||
**Task Description requirements are MANDATORY.** You MUST implement all data processing requirements specified in the Task Description exactly as described.
|
||||
|
||||
# Part 2: Output Specification
|
||||
|
||||
## 2.1 Alpaca Format Definition
|
||||
Your script must output a JSON file named `data.json` in the current working directory (`{{ workspace_path }}`).
|
||||
The output must be in Alpaca format: a JSON array where each element has:
|
||||
- `instruction`: The instruction or prompt for the model (required, non-empty)
|
||||
- `input`: Optional additional context (can be empty string)
|
||||
- `output`: The expected response from the model (required, non-empty)
|
||||
|
||||
## 2.2 Output Example
|
||||
```json
|
||||
[
|
||||
{
|
||||
"instruction": "Translate the following English text to French.",
|
||||
"input": "Hello, how are you?",
|
||||
"output": "Bonjour, comment allez-vous?"
|
||||
},
|
||||
{
|
||||
"instruction": "Summarize the following article.",
|
||||
"input": "Article content here...",
|
||||
"output": "Summary of the article..."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## 2.3 Data Quality Awareness (IMPORTANT)
|
||||
- Raw datasets may contain low-quality, noisy, or incorrect samples
|
||||
- It is better to DISCARD questionable samples than to include them in training data
|
||||
- When encountering samples that are ambiguous, malformed, or have inconsistent answers, prefer filtering them out
|
||||
- A smaller but high-quality dataset is more valuable than a larger noisy one
|
||||
- High filtering rate is acceptable and expected - it means the script is doing quality control properly
|
||||
|
||||
## 2.4 Data Validation Rules
|
||||
Before writing the final data.json, implement these validations:
|
||||
|
||||
### 2.4.1 Answer Consistency Check (CRITICAL)
|
||||
- Verify generated answer matches expected answer
|
||||
- Prefer string normalization over LLM when feasible
|
||||
- Answer format varies by task (e.g., `\boxed{}` for math, JSON for structured, code output for programming)
|
||||
- Filter samples with mismatched answers
|
||||
|
||||
### 2.4.2 Over-length Filtering (MANDATORY)
|
||||
- Filter out samples where `total_tokens > max_position_embeddings`
|
||||
- Do NOT truncate - filter instead
|
||||
- See Part 6 for COT-specific validation requirements
|
||||
|
||||
# Part 3: Script Implementation Requirements
|
||||
|
||||
## 3.1 Basic Conventions
|
||||
1. Read data from `{{ datasets_path }}` directory (mounted read-only)
|
||||
2. Use standard Python libraries (json, csv, os, pathlib) when possible
|
||||
3. Handle file encoding properly (use utf-8)
|
||||
4. Include error handling for file operations
|
||||
5. Print progress information to stdout for debugging
|
||||
6. **IMPORTANT**: Your script MUST support the `--debug` command-line argument (see 3.2). Other than `--debug`, do NOT expect any other command-line arguments.
|
||||
|
||||
## 3.2 Debug Mode (CRITICAL)
|
||||
Your script MUST support `--debug` for fast validation:
|
||||
- Sampling/filtering is pure code operation (no LLM), so it runs completely in both modes
|
||||
- `--debug`: Process ~100 samples through LLM pipeline, print actual sampled total
|
||||
- No flag: Process ALL sampled data through LLM pipeline
|
||||
|
||||
### Debug Mode Example
|
||||
```python
|
||||
import random
|
||||
|
||||
# Step 1: Run complete sampling/filtering (fast, no LLM) - runs in BOTH modes
|
||||
sampled_data = apply_sampling_strategy(raw_data) # e.g., 50000 → 2000
|
||||
|
||||
# Step 2: Limit LLM processing in debug mode only
|
||||
if args.debug:
|
||||
samples_to_process = random.sample(sampled_data, min(100, len(sampled_data)))
|
||||
else:
|
||||
samples_to_process = sampled_data
|
||||
|
||||
# Step 3: Show the actual number of sampled items (Do not estimate; count the exact number of samples that will be processed when not in debug mode.)
|
||||
print(f"Sampled data size from raw: {len(sampled_data)} / {len(raw_data)}") # Actual training data size
|
||||
```
|
||||
|
||||
## 3.3 Logging Convention
|
||||
Only print progress at 20%, 40%, 60%, 80%, 100%. No per-item logs.
|
||||
|
||||
## 3.4 Output Statistics Format
|
||||
Your script should print statistics at the end of execution:
|
||||
|
||||
### Script Execution Summary (REQUIRED)
|
||||
```
|
||||
# Debug mode (--debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Debug samples processed: {debug_processed_count}
|
||||
Estimated full output: ~{int(actual_output / debug_processed_count * sampled_count)}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
|
||||
# Full mode (no --debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
```
|
||||
|
||||
### CoT Quality Statistics (REQUIRED for COT tasks)
|
||||
```
|
||||
========== COT QUALITY STATS ==========
|
||||
COT format check: {with_think_tags}/{total} have <think> tags
|
||||
Over-length filtered: {count} ({percentage}%)
|
||||
Answer consistency check: {passed}/{total} passed
|
||||
Length distribution: p25={}, p50={}, p75={}, p99={}
|
||||
=======================================
|
||||
```
|
||||
|
||||
# Part 4: Scope Clarification (IMPORTANT)
|
||||
**Your script should ONLY handle data processing and output data.json.**
|
||||
- DO NOT generate training configuration files (e.g., train.yaml, training_config.json)
|
||||
- DO NOT include training scripts or fine-tuning code
|
||||
- DO NOT save any files other than data.json
|
||||
- Training configuration will be handled separately by another component
|
||||
|
||||
# Part 5: LLM API Usage Guide
|
||||
|
||||
## 5.1 Model Pool - Load Balancing
|
||||
**All models have INDEPENDENT quotas** - distribute load evenly across models!
|
||||
|
||||
```python
|
||||
import os, json
|
||||
import litellm; litellm.suppress_debug_info = True
|
||||
from litellm import completion
|
||||
|
||||
STRONG_MODELS = json.loads(os.getenv("STRONG_MODEL_POOL", "[]")) # CoT generation
|
||||
WEAK_MODELS = json.loads(os.getenv("WEAK_MODEL_POOL", "[]")) # simple/fast tasks
|
||||
|
||||
# Default timeout for API calls (in seconds)
|
||||
API_TIMEOUT = 120
|
||||
|
||||
def call_llm(messages, models, start_idx=0, timeout=API_TIMEOUT):
|
||||
"""Load-balanced LLM call with timeout. Use start_idx to distribute across models."""
|
||||
if not models:
|
||||
raise RuntimeError("Model pool is empty. Set STRONG_MODEL_POOL/WEAK_MODEL_POOL env vars.")
|
||||
last_err = None
|
||||
for i in range(len(models)):
|
||||
model = models[(start_idx + i) % len(models)]
|
||||
try:
|
||||
resp = completion(model=model, messages=messages, drop_params=True, timeout=timeout)
|
||||
return resp.choices[0].message.content
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
continue
|
||||
raise RuntimeError(f"All models failed. Last error: {last_err}")
|
||||
```
|
||||
|
||||
## 5.2 Timeout & Efficiency (CRITICAL)
|
||||
- Set `timeout=120` for API calls to prevent blocking on complex problems
|
||||
- If timeout after retries, skip sample and continue
|
||||
- Prefer string/regex over LLM for validation (answer check, structure check) when possible
|
||||
|
||||
## 5.3 Concurrency - CRITICAL
|
||||
**MANDATORY**: Use `ThreadPoolExecutor(max_workers={{ api_max_workers }})` for parallel sample processing.
|
||||
- DO NOT use `os.cpu_count()` - it limits parallelism unnecessarily
|
||||
- The value {{ api_max_workers }} is intentional for maximizing API throughput
|
||||
- Pass `start_idx=sample_index % len(models)` to distribute load evenly
|
||||
|
||||
```python
|
||||
with ThreadPoolExecutor(max_workers={{ api_max_workers }}) as executor: # NOT os.cpu_count()!
|
||||
futures = {executor.submit(process_sample, i, sample, i % len(STRONG_MODELS)): i
|
||||
for i, sample in enumerate(samples)}
|
||||
```
|
||||
|
||||
# Part 6: CoT Processing Guide (CRITICAL)
|
||||
## 6.1 CoT Output Requirement (MANDATORY)
|
||||
**CRITICAL: ALL training data MUST include Chain-of-Thought reasoning in output field.**
|
||||
|
||||
### Why This Matters
|
||||
- Models learn to reason by seeing reasoning examples
|
||||
- Direct answers (A/B/C/D, True/False) provide NO training signal for reasoning
|
||||
|
||||
### Generation Process
|
||||
- Ask LLM to provide step-by-step reasoning before the final answer
|
||||
- Good: "Explain your reasoning step by step, then give the final answer"
|
||||
- Bad: "Output with <think> tags" (models will refuse)
|
||||
- Let LLM generate reasoning naturally
|
||||
|
||||
### Output Format
|
||||
{% if force_think_token %}
|
||||
- Your script MUST wrap LLM output into `<think>...</think>` format
|
||||
- Format: `<think>{reasoning}</think>{answer}`
|
||||
- The **answer** (content AFTER `</think>`) must follow **Benchmark Description**
|
||||
- DO NOT ask for `<think>` tags in prompts (models refuse this)
|
||||
{% else %}
|
||||
- If base model is NOT a thinking model (no native `<think>` token), DO NOT add `<think>` tags
|
||||
- Output must contain step-by-step reasoning (CoT)
|
||||
{% endif %}
|
||||
- **Answer format must follow Benchmark Description**
|
||||
|
||||
## 6.2 Post-Processing Validation
|
||||
{% if force_think_token %}
|
||||
- **Structure check**: `"<think>" in output and "</think>" in output`
|
||||
{% endif %}
|
||||
- **Content check**: Output must contain reasoning (not just direct answer)
|
||||
- **Answer check**: Answer format must match Benchmark Description
|
||||
|
||||
# Part 7: Previous Failed Attempts
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Part 8: Response Format
|
||||
Provide ONLY the Python script in a markdown code block:
|
||||
```python
|
||||
# Your complete Python script here
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the code block.
|
||||
|
||||
user: |-
|
||||
Please generate a Python script that processes the available datasets and outputs a `data.json` file in Alpaca format.
|
||||
|
||||
The script will be executed in two modes:
|
||||
1. **Debug mode (coding phase):** `python {{ workspace_path }}process_data.py --debug` - process 100 samples for fast validation
|
||||
2. **Full mode (running phase):** `python {{ workspace_path }}process_data.py` - generates all samples for training
|
||||
|
||||
Dataset files are located at: {{ datasets_path }}
|
||||
|
||||
## Detailed Dataset Descriptions
|
||||
{% for ds_name, ds_desc in involved_dataset_folder_desc.items() %}
|
||||
### Dataset: {{ ds_name }}
|
||||
(Note: All file paths for this dataset are relative to `{{ datasets_path }}{{ ds_name }}/`)
|
||||
{{ ds_desc }}
|
||||
{% endfor %}
|
||||
|
||||
Output file should be: {{ workspace_path }}data.json
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Data Processing Script
|
||||
```python
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Script
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the 'Previous Data Processing Script' based on the feedback above. Do not create a new script. Consider the feedback carefully and make necessary corrections. If the feedback asks for more information or logging, make sure to include that in your revised script to help the evaluator to better assess your implementation.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new Data Processing Script based on the task description.
|
||||
{% endif %}
|
||||
|
||||
**IMPORTANT**: Make sure your script supports the `--debug` argument as described in the system prompt.
|
||||
|
||||
finetune_coder:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in large language model fine-tuning using LlamaFactory.
|
||||
Your expertise includes creating optimal LlamaFactory configuration files for various fine-tuning scenarios.
|
||||
|
||||
# Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
## Previous Failed Attempts
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Available Fine-tuning Methods
|
||||
{{ available_methods }}
|
||||
|
||||
## Shared Parameters
|
||||
These parameters apply to all fine-tuning methods:
|
||||
{{ shared_params }}
|
||||
|
||||
## Method-Specific Parameters
|
||||
{% for method, params_desc in methods_specific_params.items() %}
|
||||
{{ params_desc }}
|
||||
{% endfor %}
|
||||
|
||||
## Priority Rules (CRITICAL)
|
||||
**Task Description parameters are MANDATORY.** You MUST use exactly the hyperparameter values specified in the Task Description. Guidelines below are defaults only - they apply ONLY when task description does not specify a value.
|
||||
|
||||
## Requirements
|
||||
1. Create a LlamaFactory configuration file named `train.yaml`
|
||||
2. Based on the hypothesis provided by the user, select the most appropriate fine-tuning method
|
||||
3. Generate full training configuration (no sample limit)
|
||||
4. Ensure all parameters are valid for LlamaFactory
|
||||
5. **Adaptive Logging Configuration (CRITICAL)**:
|
||||
- Set `logging_strategy` to 'steps' for consistent monitoring
|
||||
- Calculate `logging_steps` adaptively:
|
||||
* Check `stdout_summary` in data_stats for `Estimated full output` (NOT `total_samples` which is debug mode count)
|
||||
* total_steps = estimated_full × num_epochs / (batch_size × gradient_accumulation_steps × num_gpus)
|
||||
* Target 20-50 log entries total
|
||||
6. **Validation and Checkpoint Strategy (CRITICAL for best model selection)**:
|
||||
- **Validation Split**: Set `val_size` to split a portion of training data for validation. Choose ratio based on dataset size and task needs.
|
||||
- **Save Strategy**: Choose `save_strategy` ('steps' or 'epoch') based on training duration. MUST ensure `eval_strategy` == `save_strategy`.
|
||||
- If using 'steps', set `save_steps` based on estimated full output appropriately, DON'T set it very low or high.
|
||||
- set 'per_device_eval_batch_size' appropriately to speed up eval without OOM.
|
||||
- **Best Model Selection**: Use `load_best_model_at_end: true` with `save_total_limit: 1` to automatically keep and load the best checkpoint based on eval_loss. Note: `save_total_limit` will be force-injected to 1.
|
||||
7. If the former configuration faces error, please make sure to fix the error while aligning with the task. If these two goals conflict, please prioritize fixing the error.
|
||||
|
||||
## Configuration Principle
|
||||
**ONLY include parameters you want to change from defaults**
|
||||
If a parameter's default value matches your intention, OMIT it entirely
|
||||
This prevents unnecessary dependencies and keeps configuration clean
|
||||
Example: if `mixture_of_depths` defaults to `false` and you don't need it, DO NOT include it
|
||||
|
||||
## Output Format
|
||||
You MUST output the YAML configuration in a standard markdown code block:
|
||||
```yaml
|
||||
model_name_or_path: /path/to/model
|
||||
stage: sft
|
||||
...
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the YAML block.
|
||||
|
||||
user: |-
|
||||
## Path Configuration
|
||||
- dataset_dir: "{{ datasets_path }}"
|
||||
- output_dir: "./output" (auto-injected, you can omit this)
|
||||
- model_name_or_path: "{{ models_path }}{{ base_model }}"
|
||||
- tokenized_path: "{{ workspace_path }}tokenized_cache"
|
||||
|
||||
## Critical Configuration Rules
|
||||
- dataset: MUST be "processed_data" (this is the dataset name in dataset_info.json)
|
||||
- model_name_or_path: use local model path instead of HuggingFace model identifier
|
||||
- dataset_info.json is located at: "{{ datasets_path }}dataset_info.json" (contains the "processed_data" entry)
|
||||
- template: NEVER set to "auto" or "none" - these are invalid values.
|
||||
- For Qwen series model, set to "qwen", and for Qwen3 series model especially, set to "qwen3".
|
||||
- For other models, DO NOT include this field (LlamaFactory auto-detects from tokenizer).
|
||||
- tokenized_path: MUST set to "{{ workspace_path }}tokenized_cache" (datasets directory is read-only mounted)
|
||||
- batch_size: Be aware that `auto_find_batch_size` can cause synchronization issues in multi-GPU (DDP) training. Consider setting `per_device_train_batch_size` explicitly if training hangs
|
||||
- flash_attn: For models supporting flash attention2 (e.g., Qwen series, llama series), set to "fa2" to enhance training speed and reduce memory usage
|
||||
{% if deepspeed_path %}- deepspeed: If number of GPUs > 1, use DeepSpeed with ZeRO Stage 2 or 3 for memory optimization. specifically, set to "{{ deepspeed_path }}ds_z3_config.json" for ZeRO Stage 3, otherwise use "{{ deepspeed_path }}ds_z2_config.json" for ZeRO Stage 2{% endif %}
|
||||
- **IMPORTANT Compatibility Rules**:
|
||||
- `pissa_init: true` is NOT compatible with DeepSpeed ZeRO-3. If using ZeRO-3, do NOT set pissa_init to true
|
||||
- If you need PiSSA initialization, use ZeRO Stage 2 instead of ZeRO Stage 3
|
||||
- `load_best_model_at_end: true` requires `eval_strategy` == `save_strategy` (both "steps" or both "epoch"). Always set both to the same value.
|
||||
|
||||
{% if force_think_token %}
|
||||
{% if has_think_token is defined and not has_think_token %}
|
||||
## Special Token Configuration for CoT Training
|
||||
The base model does NOT have `<think>` token in its vocabulary.
|
||||
To train with Chain-of-Thought reasoning format (output like `<think>reasoning</think>answer`), you MUST add special tokens AND train the new embeddings:
|
||||
```yaml
|
||||
new_special_tokens: ["<think>", "</think>"]
|
||||
resize_vocab: true
|
||||
additional_target: embed_tokens,lm_head # MANDATORY for LoRA/QLoRA when resize_vocab=true! And Full Training does not need this field.
|
||||
```
|
||||
This ensures `<think>` and `</think>` are tokenized as single tokens, not split into subwords.
|
||||
{% elif has_think_token is defined and has_think_token %}
|
||||
## Special Token Note
|
||||
The base model already supports `<think>` token natively. No need to add special tokens for CoT training.
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{# When force_think_token=false, no special token configuration needed #}
|
||||
|
||||
{% if data_stats %}
|
||||
## Processed Data Statistics (from debug mode)
|
||||
{{ data_stats }}
|
||||
|
||||
**Your Task**: Implement the training configuration specified in the task description.
|
||||
|
||||
- Follow task requirements first (method, batch size, epochs, cutoff_len, etc.)
|
||||
- Apply technical constraints only when task doesn't specify:
|
||||
- `cutoff_len`: ≤ min(max_position_embeddings, memory limit, data p99)
|
||||
- `per_device_train_batch_size`: Choose based on Memory Estimates table
|
||||
- `gradient_accumulation_steps`: Adjust for stable training (effective_batch = batch × accum × gpus)
|
||||
- Validation setup: `val_size`, `eval_strategy` == `save_strategy`, `load_best_model_at_end: true`
|
||||
{% endif %}
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Configuration
|
||||
```yaml
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Configuration
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the configuration based on the feedback above and the hypothesis.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new configuration for the model {{ base_model }} based on the hypothesis above.
|
||||
|
||||
**Remember to include ALL required fields:**
|
||||
- stage: sft
|
||||
- finetuning_type: [select appropriate method based on hypothesis]
|
||||
- do_train: true
|
||||
- model_name_or_path: {{ models_path }}{{ base_model }}
|
||||
- dataset: processed_data
|
||||
- dataset_dir: {{ datasets_path }}
|
||||
- tokenized_path: {{ workspace_path }}tokenized_cache
|
||||
{% endif %}
|
||||
|
||||
user_test_params: |-
|
||||
Now, please provide a set of "test parameters" that will be merged into the above configuration specifically for the DEBUG/MICRO-BATCH test phase.
|
||||
|
||||
The debug phase runs on a very small subset (~10 samples).
|
||||
You need to override parameters that adapt to the dataset for quick debugging the yaml config.
|
||||
|
||||
**Example for Test Parameters:**
|
||||
- Set `num_train_epochs` to 1.
|
||||
- Set `max_samples` to a very small number.
|
||||
|
||||
**Output Format:**
|
||||
Output ONLY the YAML block for these test parameters:
|
||||
```yaml
|
||||
num_train_epochs: 1
|
||||
...
|
||||
```
|
||||
|
||||
finetune_eval:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in evaluating fine-tuning configurations for large language models using LlamaFactory.
|
||||
Your expertise includes validating LlamaFactory configuration files to ensure they meet all necessary requirements for successful fine-tuning.
|
||||
|
||||
You will be provided with:
|
||||
1. A detailed scenario description which requires a fine-tuning LLM.
|
||||
2. A yaml configuration file named `train.yaml` created for LlamaFactory fine-tuning.
|
||||
3. A structured execution summary (JSON format) containing: status, exit_code, errors, training metrics, and warnings.
|
||||
4. The files generated during the execution.
|
||||
5. Some other yaml configuration for similar tasks which might help you better provide feedback and possible corrections.
|
||||
|
||||
Your task is to:
|
||||
1. Check the execution summary to determine if the run succeeded.
|
||||
2. validate the provided `train.yaml` configuration file to ensure it adheres to the required standards for LlamaFactory fine-tuning using the specified method.
|
||||
3. Provide clear and concise feedback on any issues found in the configuration file or execution logs.
|
||||
4. Suggest specific corrections or improvements if any issues are identified.
|
||||
|
||||
You must give a false final decision only if:
|
||||
- The execution fails with non-zero exit code.
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
### Similar Successful Implementations to help training config Improvement
|
||||
The user has done several similar tasks and get some successful implementations. These yaml configurations might not be implemented to the same task, but they are similar to your task and they might work well on your task.
|
||||
Please refer to these successful implementation and provide your suggestions in your response on how to correct your current code based on these successful implementations.
|
||||
## Successful Implementations for Similar Tasks
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Similar Task {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Yaml configurations:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Important Notice
|
||||
- You may find that the execution is short with limited data and iterations. This is expected as we are only validating the configuration file's correctness and not performing full-scale training. Don't treat this as a failure. Also do not put this information in your feedback.
|
||||
|
||||
## Output Format
|
||||
Please respond with your feedback in the following JSON format without anything else.
|
||||
```json
|
||||
{
|
||||
"execution": "State if run succeeded. If errors, include all messages verbatim. Classify cause: algorithm, implementation, or environment."
|
||||
"return_checking": "Plain text. Examine the generated files from the user input. Does the output contains a fine-tuned model or expected artifacts? If not, specify what is missing or incorrect.",
|
||||
"code": "Plain text. Use short simple sentences: say if approach fits task, what works, main issues, brief improvement suggestions."
|
||||
"final_decision": <true/false>, # Final decision on whether the configuration is acceptable for full data fine-tuning
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
# Scenario Information
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Yaml Configuration File
|
||||
```yaml
|
||||
{{ code_yaml }}
|
||||
|
||||
## Execution Summary (Structured)
|
||||
```json
|
||||
{{ stdout }}
|
||||
```
|
||||
|
||||
## Workspace Files
|
||||
{{ workspace_files }}
|
||||
|
||||
data_eval:
|
||||
system: |-
|
||||
You are a data quality expert for LLM fine-tuning using LlamaFactory.
|
||||
Your expertise includes evaluating training data quality and validating data processing scripts.
|
||||
|
||||
You will evaluate:
|
||||
1. **Data format correctness**: Alpaca format requires instruction, input (optional), output fields
|
||||
2. **Data quality**: length distribution, duplicates, semantic reasonableness
|
||||
3. **Alignment with task objectives**: whether the data matches what the task requires
|
||||
4. **Code logic correctness**: whether the processing script is well-designed
|
||||
|
||||
## The Main Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
## Similar Successful Data Processing Examples
|
||||
The following are successful data processing implementations for similar tasks:
|
||||
{% for knowledge in queried_similar_successful_knowledge %}
|
||||
### Example {{ loop.index }}:
|
||||
**Task:** {{ knowledge.target_task.get_task_information() }}
|
||||
**Code:**
|
||||
```python
|
||||
{{ knowledge.implementation.file_dict.get("process_data.py", "N/A") }}
|
||||
```
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Debug Mode Context (IMPORTANT)
|
||||
This evaluation runs during the CODING phase in DEBUG MODE.
|
||||
- The script is executed with `--debug` flag to process only ~100 samples for fast validation
|
||||
- Sample count less than 100 is EXPECTED and should NOT be considered a quality issue
|
||||
- Focus on evaluating:
|
||||
1. Data format correctness (Alpaca format)
|
||||
2. Data quality of the generated samples
|
||||
3. Script logic correctness (will it work in full mode?)
|
||||
- Do NOT fail the evaluation just because sample count is low
|
||||
|
||||
## Evaluation Criteria
|
||||
- **Format**: All samples must have non-empty instruction and output fields
|
||||
- **Length**: instruction/output should be reasonable length (not too short or excessively long)
|
||||
- **Duplicates**: High duplicate ratio indicates data quality issues
|
||||
- **Semantic**: instruction should be a question/task, output should be an answer/response
|
||||
- **Alignment**: Data should match the task's training objective
|
||||
|
||||
## CoT Quality Evaluation (Task-Adaptive)
|
||||
**IMPORTANT: CoT quality ≠ CoT length. Adapt criteria based on task type from README metadata.**
|
||||
|
||||
**Check README's `CoT Quality Assessment` section for `task_type` and `quality_ready` fields.**
|
||||
|
||||
1. **Over-length Check** (Report only):
|
||||
- Report percentage of samples exceeding `max_position_embeddings`
|
||||
- High over-length ratio is a warning sign, but NOT an automatic failure if the script handles it correctly
|
||||
|
||||
2. **Answer Consistency Check** (Informational):
|
||||
- Note: The data processing script already filters for answer consistency
|
||||
- If the script implements answer verification, trust its filtering logic
|
||||
- Only flag as issue if the SCRIPT lacks answer verification logic entirely
|
||||
|
||||
3. **Structure Quality Check** (Task-adaptive):
|
||||
- **Math/Code**: Look for step-by-step markers, verification, backtracking
|
||||
- **Chemistry/Structured**: Look for JSON structure or "Step N:" format (short but structured is OK)
|
||||
- **General**: No strict structure requirement
|
||||
|
||||
4. **Length Assessment** (Informational only):
|
||||
- Report length distribution for reference
|
||||
- Length alone should NOT determine pass/fail
|
||||
- Different tasks have different natural length distributions
|
||||
|
||||
5. **Polish Quality Assessment**:
|
||||
- All data must be polished before use
|
||||
- If README shows `baseline_quality: high`: verify enrichment was applied
|
||||
- If README shows `baseline_quality: low`: verify full generation/rewrite was done
|
||||
- Check polish met the requirements in `polish_strategy`
|
||||
|
||||
**Include in return_checking:**
|
||||
- "Task type: {type}, Quality ready: {ready}"
|
||||
- "CoT stats: p50={}, over-length={X}%, structure quality={Y}%"
|
||||
- Assessment based on task-appropriate criteria
|
||||
|
||||
## Hard Check Criteria (AUTOMATIC FAIL if not met)
|
||||
{% if force_think_token %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- EVERY sample MUST contain `<think>` and `</think>` tags
|
||||
- Content AFTER `</think>` must be non-empty
|
||||
|
||||
**Rejection:** "FAIL: {X} samples missing <think> tags."
|
||||
{% else %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- Output must contain reasoning content (not just a direct answer)
|
||||
- Answer format must match **Benchmark Description**
|
||||
- Do NOT reject for reasoning quality or answer correctness
|
||||
|
||||
**Rejection:** "FAIL: {X}% of samples are direct answers without reasoning."
|
||||
{% endif %}
|
||||
|
||||
### 2. Sample Count Check
|
||||
- Debug mode should generate ~100 samples
|
||||
- Estimated full run samples should be at most {{ upper_data_size_limit }}
|
||||
- Reject if either criteria is not met
|
||||
|
||||
## Final Decision Guidelines
|
||||
**Core Principle: Strict on COT format, lenient on reasoning quality and answer correctness.**
|
||||
|
||||
- **Approve (true)** if:
|
||||
- Script runs successfully (exit_code == 0)
|
||||
- At least 1 sample is generated
|
||||
{% if force_think_token %}- ALL samples have `<think>` and `</think>` tags (MANDATORY){% else %}- ALL samples contain reasoning content (not just direct answers){% endif %}
|
||||
- Data format is correct (Alpaca format with instruction/output)
|
||||
|
||||
- **Reject (false)** if ANY of these:
|
||||
- Script fails to run (exit_code != 0)
|
||||
- Zero samples are generated
|
||||
{% if force_think_token %}- **ANY sample missing `<think>` or `</think>` tags (HARD FAIL)**{% else %}- **ANY sample missing reasoning content (just direct answer)**{% endif %}
|
||||
- Data format is fundamentally broken
|
||||
- **Data does NOT match task description requirements**
|
||||
|
||||
- **Do NOT reject** for:
|
||||
- Low sample count in debug mode (expected)
|
||||
- Moderate quality variations in individual samples
|
||||
- Length distribution not matching ideal patterns
|
||||
- High filtering rate (script doing its job)
|
||||
|
||||
## Important Note
|
||||
- Do not summarize the code into your feedback and DO NOT copy the task description also. Only provide new insights based on your evaluation.
|
||||
- If you think the current logging information is not sufficient to find out the issues, please specify what additional logging information is needed in your feedback and put this information in 'code' block. The user will add further provide you the additional logging information in the next iteration.
|
||||
- Do not write any code in your response, use plain text only.
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only (no markdown code block):
|
||||
{
|
||||
"execution": "Script execution status and data generation result. Include exit code and any errors.",
|
||||
"return_checking": "Data quality analysis: format validation, length distribution assessment, duplicate ratio, semantic issues found; Hard check criteria: does the solution meet the hard check criteria",
|
||||
"code": "Code issues and specific improvement suggestions. What works well, what needs fixing.",
|
||||
"final_decision": true/false
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
{% if script_code %}
|
||||
|
||||
# Data Processing Script (for debugging)
|
||||
```python
|
||||
{{ script_code }}
|
||||
```
|
||||
{% endif %}
|
||||
{% if stdout %}
|
||||
|
||||
# Execution Output ({% if exit_code != 0 %}error logs{% else %}summary{% endif %})
|
||||
```
|
||||
Exit code: {{ exit_code }}
|
||||
{{ stdout }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
# Data Statistics
|
||||
```json
|
||||
{{ data_stats }}
|
||||
```
|
||||
|
||||
# Sample Data ({{ sample_count }} samples from total {{ total_samples }}) [DEBUG MODE]
|
||||
```json
|
||||
{{ data_samples }}
|
||||
```
|
||||
|
||||
runner_eval:
|
||||
system: |-
|
||||
You are a world-class ML engineer evaluating LLM fine-tuning results.
|
||||
|
||||
## Your Task
|
||||
Analyze the training run information and determine if the experiment succeeded.
|
||||
|
||||
## Evaluation Criteria (for final_decision)
|
||||
1. **Execution Success**: Did training complete without errors? Check exit_code and model outputs.
|
||||
2. **Benchmark Execution**: Did benchmark run successfully? Check benchmark results availability.
|
||||
|
||||
## Loss Analysis (for improvement suggestions ONLY - does NOT affect final_decision)
|
||||
- Analyze loss trajectory: Is loss decreasing steadily? Any signs of overfitting?
|
||||
- Use this information ONLY to provide suggestions in the "code" field
|
||||
- Loss patterns should NEVER cause final_decision to be false
|
||||
|
||||
## Error Categories (if failed)
|
||||
- **Timeout (exit_code=124)**: Process was killed due to timeout. Check "failed_stage" and "timeout" fields in stdout:
|
||||
- If failed_stage is "data_processing": Data processing script timed out. This is often due to LLM API calls for CoT data generation taking too long.
|
||||
- If failed_stage is "training": Training timed out.
|
||||
- **OOM**: GPU memory exhaustion - suggest batch size/model changes
|
||||
- **CUDA**: Driver/device issues - suggest environment checks
|
||||
- **Config**: Invalid parameters - suggest specific fixes
|
||||
- **Data**: Dataset issues - suggest data pipeline fixes
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only:
|
||||
{
|
||||
"execution": "Execution status: SUCCESS or FAILED with category [OOM/CUDA/Config/Data]. Include key metrics or error details.",
|
||||
"return_checking": "If success: benchmark analysis. If failed: what failed and expected behavior.",
|
||||
"code": "Configuration assessment and improvement suggestions",
|
||||
"final_decision": true/false // Set to true as long as training succeeded (exit_code=0) and benchmark ran successfully
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Training Configuration
|
||||
```yaml
|
||||
{{ config_yaml }}
|
||||
```
|
||||
|
||||
# Execution Info
|
||||
- Exit Code: {{ exit_code }}
|
||||
- Model Output Files: {{ model_files_status }}
|
||||
{% if failed_stage %}- Failed Stage: {{ failed_stage }}
|
||||
- Stage Timeout Config: {{ timeout_seconds }} seconds
|
||||
{% endif %}
|
||||
|
||||
# Benchmark Results
|
||||
```json
|
||||
{{ benchmark_result }}
|
||||
```
|
||||
|
||||
# Loss History (train loss and eval_loss if validation enabled)
|
||||
```json
|
||||
{{ loss_history }}
|
||||
```
|
||||
{% include "components.coder.finetune.prompts:runner_eval.train_output" %}
|
||||
|
||||
train_output: |-
|
||||
# Training Output (key information extracted from stdout)
|
||||
```
|
||||
{{ stdout }}
|
||||
```
|
||||
@@ -0,0 +1,304 @@
|
||||
"""
|
||||
Simplified LLM Fine-tuning Configuration Validator
|
||||
|
||||
Two-step validation:
|
||||
1. Parameter filtering - Remove unsupported parameters
|
||||
2. Micro-batch testing - Runtime validation with small dataset
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Set
|
||||
|
||||
import yaml
|
||||
|
||||
from rdagent.components.coder.finetune.conf import (
|
||||
FT_DEBUG_YAML_FILE_NAME,
|
||||
FT_TEST_PARAMS_FILE_NAME,
|
||||
get_ft_env,
|
||||
get_workspace_prefix,
|
||||
)
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.scenarios.finetune.scen.llama_factory_manager import LLaMAFactory_manager
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
# System-managed parameters that are automatically injected during validation.
|
||||
# These should NOT be checked for alignment in eval prompts.
|
||||
# Single source of truth: modify here to change injected parameters.
|
||||
SYSTEM_MANAGED_PARAMS = {
|
||||
"overwrite_cache": True, # Avoid HF datasets cache lock contention
|
||||
"save_only_model": True, # Save disk space
|
||||
# "save_total_limit": 1, # Limit checkpoint count to save disk space
|
||||
"output_dir": "./output", # Standardize model output location
|
||||
"per_device_eval_batch_size": 1, # Prevent OOM during evaluation
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
"""Configuration validation result"""
|
||||
|
||||
success: bool
|
||||
filtered_config: str
|
||||
execution_output: str = "" # Parsed/summarized output for LLM
|
||||
raw_stdout: str = "" # Full raw stdout for UI display
|
||||
errors: List[str] = field(default_factory=list)
|
||||
execution_time: float = 0.0
|
||||
|
||||
|
||||
class LLMConfigValidator:
|
||||
"""LLM configuration validator with two-step validation:
|
||||
|
||||
1. Parameter filtering - Remove unsupported parameters
|
||||
2. Micro-batch test - Runtime validation with small dataset
|
||||
|
||||
The micro-batch test inherently validates completeness, so no separate completeness check is needed.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._supported_params_cache: Optional[Set[str]] = None
|
||||
|
||||
def validate_and_test(self, config_yaml: str, workspace: FBWorkspace, env) -> ValidationResult:
|
||||
"""Three-step validation: parameter filtering + injection + micro-batch testing"""
|
||||
start_time = time.time()
|
||||
|
||||
# Step 1: Parameter filtering
|
||||
filtered_config, removed_params = self._filter_parameters(config_yaml)
|
||||
|
||||
# Step 2: Inject required parameters for multi-task environments
|
||||
injected_config = self._inject_required_parameters(filtered_config)
|
||||
|
||||
# Step 3: Micro-batch testing (validates everything at runtime)
|
||||
result = self._run_micro_batch_test(injected_config, workspace, env)
|
||||
result.execution_time = time.time() - start_time
|
||||
|
||||
# Add filtered params info to execution_output for agent learning
|
||||
if removed_params:
|
||||
filter_info = (
|
||||
f"\n\n[Filtered Parameters] {len(removed_params)} unsupported params removed: {removed_params}"
|
||||
)
|
||||
result.execution_output += filter_info
|
||||
|
||||
return result
|
||||
|
||||
def _filter_parameters(self, config_yaml: str) -> tuple[str, List[str]]:
|
||||
"""Filter configuration parameters to only include supported ones.
|
||||
|
||||
Returns:
|
||||
tuple: (filtered_yaml, removed_params_list)
|
||||
"""
|
||||
config_dict = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config_dict, dict):
|
||||
return config_yaml, []
|
||||
|
||||
supported_params = self._get_supported_parameters()
|
||||
|
||||
filtered_config = {}
|
||||
removed_params = []
|
||||
for k, v in config_dict.items():
|
||||
if k in supported_params:
|
||||
filtered_config[k] = v
|
||||
else:
|
||||
removed_params.append(k)
|
||||
|
||||
if removed_params:
|
||||
logger.info(f"Filtered out {len(removed_params)} unsupported parameters: {removed_params}")
|
||||
|
||||
return yaml.dump(filtered_config, default_flow_style=False, sort_keys=False), removed_params
|
||||
|
||||
def _inject_required_parameters(self, config_yaml: str) -> str:
|
||||
"""Inject required parameters for multi-task environments.
|
||||
|
||||
Uses SYSTEM_MANAGED_PARAMS as the single source of truth.
|
||||
"""
|
||||
config = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config, dict):
|
||||
return config_yaml
|
||||
|
||||
config.update(SYSTEM_MANAGED_PARAMS)
|
||||
|
||||
logger.info(f"Injected required parameters: {SYSTEM_MANAGED_PARAMS}")
|
||||
return yaml.dump(config, default_flow_style=False, sort_keys=False)
|
||||
|
||||
def _get_supported_parameters(self) -> Set[str]:
|
||||
"""Get supported parameters from LlamaFactory Manager"""
|
||||
if self._supported_params_cache is not None:
|
||||
return self._supported_params_cache
|
||||
|
||||
all_params = LLaMAFactory_manager.get_parameters()
|
||||
|
||||
# Extract all parameter names from all parameter types (including nested structures)
|
||||
supported_params = set()
|
||||
for param_type, params_dict in all_params.items():
|
||||
if isinstance(params_dict, dict):
|
||||
# Recursively extract parameter names from nested dictionaries
|
||||
for key, value in params_dict.items():
|
||||
if isinstance(value, dict) and "name" in value:
|
||||
# This is a parameter definition with metadata
|
||||
supported_params.add(key)
|
||||
elif isinstance(value, dict):
|
||||
# This is a nested category (e.g., BaseModelArguments, LoraArguments)
|
||||
# Extract parameter names from the nested structure
|
||||
for nested_key, nested_value in value.items():
|
||||
if isinstance(nested_value, dict) and "name" in nested_value:
|
||||
supported_params.add(nested_key)
|
||||
|
||||
if not supported_params:
|
||||
raise RuntimeError("No parameters found in LlamaFactory Manager")
|
||||
|
||||
logger.info(f"Loaded {len(supported_params)} parameters from LlamaFactory Manager")
|
||||
self._supported_params_cache = supported_params
|
||||
return supported_params
|
||||
|
||||
def _parse_execution_log(self, stdout: str, exit_code: int, failed_stage: str = None) -> str:
|
||||
"""Parse execution log and extract key information for LLM evaluation.
|
||||
|
||||
Reduces log from ~36k tokens to ~500 tokens by extracting only:
|
||||
- Status and exit code
|
||||
- Error messages (if any)
|
||||
- Training metrics (if successful)
|
||||
- Warnings (limited)
|
||||
- Timeout and stage information (if applicable)
|
||||
|
||||
Args:
|
||||
stdout: The execution output
|
||||
exit_code: The process exit code
|
||||
failed_stage: Which stage failed - "data_processing" or "training"
|
||||
"""
|
||||
result = {
|
||||
"status": "success" if exit_code == 0 else "failed",
|
||||
"exit_code": exit_code,
|
||||
}
|
||||
|
||||
# Handle timeout (exit_code 124)
|
||||
if exit_code == 124:
|
||||
result["timeout"] = True
|
||||
if failed_stage:
|
||||
result["failed_stage"] = failed_stage
|
||||
|
||||
# 1. Extract error information (highest priority)
|
||||
# Strategy: extract rank0's error block (each line prefixed with [rank0]:)
|
||||
error_text = None
|
||||
|
||||
# Method A: Extract [rank0]: prefixed lines and reconstruct traceback
|
||||
rank0_lines = re.findall(r"\[rank0\]:[^\n]+", stdout)
|
||||
if rank0_lines:
|
||||
rank0_block = "\n".join(line.replace("[rank0]: ", "").replace("[rank0]:", "") for line in rank0_lines)
|
||||
# Find traceback in rank0 block
|
||||
tb_match = re.search(
|
||||
r"Traceback \(most recent call last\):.*?(?:Error|Exception):[^\n]+", rank0_block, re.DOTALL
|
||||
)
|
||||
if tb_match:
|
||||
error_text = tb_match.group(0)
|
||||
|
||||
# Method B: Fallback to generic traceback (no rank prefix)
|
||||
# Use findall to get ALL tracebacks, then keep the first one (root cause)
|
||||
if not error_text:
|
||||
all_tracebacks = re.findall(
|
||||
r"Traceback \(most recent call last\):.*?(?:Error|Exception):[^\n]+", stdout, re.DOTALL
|
||||
)
|
||||
if all_tracebacks:
|
||||
# First traceback is usually the root cause
|
||||
error_text = all_tracebacks[0]
|
||||
if len(all_tracebacks) > 1:
|
||||
error_text += f"\n\n[Note: {len(all_tracebacks)} total errors, showing root cause]"
|
||||
|
||||
if error_text:
|
||||
# Limit length but keep from the END (actual error type/message is at the end of traceback)
|
||||
result["error"] = error_text[-4000:] if len(error_text) > 4000 else error_text
|
||||
|
||||
# 2. Extract training information
|
||||
if "Running training" in stdout:
|
||||
result["training_started"] = True
|
||||
|
||||
# Extract training config
|
||||
# NOTE: we may have log like "Num examples = 1,000,000" and "Num Epochs = 1,000"; So we need to handle ","
|
||||
num_examples = re.search(r"Num examples\s*=\s*([\d,]+)", stdout)
|
||||
num_epochs = re.search(r"Num Epochs\s*=\s*([\d,]+)", stdout)
|
||||
if num_examples:
|
||||
result["num_examples"] = int(num_examples.group(1).replace(",", ""))
|
||||
if num_epochs:
|
||||
result["num_epochs"] = int(num_epochs.group(1).replace(",", ""))
|
||||
|
||||
# Extract final metrics (JSON format from trainer output)
|
||||
final_metrics = re.search(r"\{'train_runtime':[^}]+\}", stdout)
|
||||
if final_metrics:
|
||||
try:
|
||||
metrics = eval(final_metrics.group(0)) # Safe: only numbers and strings
|
||||
result["final_metrics"] = {
|
||||
"train_loss": metrics.get("train_loss"),
|
||||
"train_runtime": metrics.get("train_runtime"),
|
||||
"train_samples_per_second": metrics.get("train_samples_per_second"),
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Check completion
|
||||
if "Training completed" in stdout:
|
||||
result["completed"] = True
|
||||
|
||||
# 3. Extract warnings (limit to 20)
|
||||
warnings = re.findall(r"\[WARNING[^\]]*\][^\n]+", stdout)
|
||||
if warnings:
|
||||
result["warnings"] = list(set(warnings))[:20]
|
||||
|
||||
# 4. Fallback: if parsing failed, include truncated raw log
|
||||
if not result.get("error") and not result.get("training_started"):
|
||||
result["raw_log_tail"] = stdout[-2000:] if len(stdout) > 2000 else stdout
|
||||
|
||||
return json.dumps(result, indent=2, ensure_ascii=False)
|
||||
|
||||
def _run_micro_batch_test(self, config_yaml: str, workspace: FBWorkspace, env) -> ValidationResult:
|
||||
"""Run micro-batch training test for runtime validation"""
|
||||
result = ValidationResult(success=True, filtered_config=config_yaml)
|
||||
ws_prefix = get_workspace_prefix(env)
|
||||
|
||||
# Create micro-batch test configuration
|
||||
config = yaml.safe_load(config_yaml)
|
||||
if not isinstance(config, dict):
|
||||
result.success = False
|
||||
result.execution_output = "Invalid YAML configuration"
|
||||
result.errors.append("Invalid configuration for micro-batch test")
|
||||
return result
|
||||
|
||||
test_config = config.copy()
|
||||
|
||||
# Load extra test parameters from workspace (generated by coder in 2nd turn)
|
||||
extra_test_params = yaml.safe_load(workspace.file_dict[FT_TEST_PARAMS_FILE_NAME])
|
||||
|
||||
# Merge extra test parameters (overrides previous settings)
|
||||
if extra_test_params:
|
||||
test_config.update(extra_test_params)
|
||||
|
||||
# Run micro-batch training
|
||||
workspace.inject_files(**{FT_DEBUG_YAML_FILE_NAME: yaml.dump(test_config, default_flow_style=False)})
|
||||
training_result = workspace.run(
|
||||
env=env,
|
||||
entry=f"llamafactory-cli train {FT_DEBUG_YAML_FILE_NAME}",
|
||||
)
|
||||
|
||||
# Remove micro-batch test files
|
||||
workspace.remove_files([FT_DEBUG_YAML_FILE_NAME, FT_TEST_PARAMS_FILE_NAME])
|
||||
|
||||
# Parse and store structured execution output (reduces ~36k tokens to ~500)
|
||||
raw_stdout = training_result.stdout if training_result.stdout else ""
|
||||
result.raw_stdout = raw_stdout # Keep full log for UI
|
||||
result.execution_output = self._parse_execution_log(raw_stdout, training_result.exit_code)
|
||||
|
||||
# Check results
|
||||
progress_indicators = ["train_loss", "Training:", "Epoch", "loss:", "step"]
|
||||
has_progress = any(ind.lower() in training_result.stdout.lower() for ind in progress_indicators)
|
||||
|
||||
if training_result.exit_code == 0 and has_progress:
|
||||
logger.info("Micro-batch test passed")
|
||||
result.success = True
|
||||
else:
|
||||
result.success = False
|
||||
result.errors.append(f"Micro-batch test failed (exit_code={training_result.exit_code})")
|
||||
|
||||
return result
|
||||
@@ -7,14 +7,14 @@ class BasePropSetting(ExtendedBaseSettings):
|
||||
You can add following config in the subclass to distinguish the environment variables.
|
||||
"""
|
||||
|
||||
scen: str = ""
|
||||
knowledge_base: str = ""
|
||||
knowledge_base_path: str = ""
|
||||
hypothesis_gen: str = ""
|
||||
interactor: str = ""
|
||||
hypothesis2experiment: str = ""
|
||||
coder: str = ""
|
||||
runner: str = ""
|
||||
summarizer: str = ""
|
||||
scen: str | None = None
|
||||
knowledge_base: str | None = None
|
||||
knowledge_base_path: str | None = None
|
||||
hypothesis_gen: str | None = None
|
||||
interactor: str | None = None
|
||||
hypothesis2experiment: str | None = None
|
||||
coder: str | None = None
|
||||
runner: str | None = None
|
||||
summarizer: str | None = None
|
||||
|
||||
evolving_n: int = 10
|
||||
|
||||
@@ -30,14 +30,30 @@ class RDLoop(LoopBase, metaclass=LoopMeta):
|
||||
logger.log_object(scen, tag="scenario")
|
||||
logger.log_object(PROP_SETTING.model_dump(), tag="RDLOOP_SETTINGS")
|
||||
logger.log_object(RD_AGENT_SETTINGS.model_dump(), tag="RD_AGENT_SETTINGS")
|
||||
self.hypothesis_gen: HypothesisGen = import_class(PROP_SETTING.hypothesis_gen)(scen)
|
||||
self.hypothesis_gen: HypothesisGen = (
|
||||
import_class(PROP_SETTING.hypothesis_gen)(scen)
|
||||
if hasattr(PROP_SETTING, "hypothesis_gen") and PROP_SETTING.hypothesis_gen
|
||||
else None
|
||||
)
|
||||
|
||||
self.hypothesis2experiment: Hypothesis2Experiment = import_class(PROP_SETTING.hypothesis2experiment)()
|
||||
self.hypothesis2experiment: Hypothesis2Experiment = (
|
||||
import_class(PROP_SETTING.hypothesis2experiment)()
|
||||
if hasattr(PROP_SETTING, "hypothesis2experiment") and PROP_SETTING.hypothesis2experiment
|
||||
else None
|
||||
)
|
||||
|
||||
self.coder: Developer = import_class(PROP_SETTING.coder)(scen)
|
||||
self.runner: Developer = import_class(PROP_SETTING.runner)(scen)
|
||||
self.coder: Developer = (
|
||||
import_class(PROP_SETTING.coder)(scen) if hasattr(PROP_SETTING, "coder") and PROP_SETTING.coder else None
|
||||
)
|
||||
self.runner: Developer = (
|
||||
import_class(PROP_SETTING.runner)(scen) if hasattr(PROP_SETTING, "runner") and PROP_SETTING.runner else None
|
||||
)
|
||||
|
||||
self.summarizer: Experiment2Feedback = import_class(PROP_SETTING.summarizer)(scen)
|
||||
self.summarizer: Experiment2Feedback = (
|
||||
import_class(PROP_SETTING.summarizer)(scen)
|
||||
if hasattr(PROP_SETTING, "summarizer") and PROP_SETTING.summarizer
|
||||
else None
|
||||
)
|
||||
self.trace = Trace(scen=scen)
|
||||
super().__init__()
|
||||
|
||||
@@ -72,21 +88,21 @@ class RDLoop(LoopBase, metaclass=LoopMeta):
|
||||
return exp
|
||||
|
||||
def feedback(self, prev_out: dict[str, Any]):
|
||||
# TODO: the logic branch of exception should be moved to summarizer
|
||||
e = prev_out.get(self.EXCEPTION_KEY, None)
|
||||
if e is not None:
|
||||
feedback = HypothesisFeedback(
|
||||
observations=str(e),
|
||||
hypothesis_evaluation="",
|
||||
new_hypothesis="",
|
||||
reason="",
|
||||
reason=str(e),
|
||||
decision=False,
|
||||
code_change_summary="",
|
||||
acceptable=False,
|
||||
)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["direct_exp_gen"]["exp_gen"], feedback))
|
||||
else:
|
||||
feedback = self.summarizer.generate_feedback(prev_out["running"], self.trace)
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
self.trace.hist.append((prev_out["running"], feedback))
|
||||
logger.log_object(feedback, tag="feedback")
|
||||
return feedback
|
||||
|
||||
# TODO: `def record(self, prev_out: dict[str, Any]):` has already been hard coded into LoopBase
|
||||
# So we should add it into RDLoop class to make sure every RDLoop Sub Class be aware of it.
|
||||
def record(self, prev_out: dict[str, Any]):
|
||||
feedback = prev_out["feedback"]
|
||||
exp = prev_out.get("running") or prev_out.get("coding") or prev_out.get("direct_exp_gen", {}).get("exp_gen")
|
||||
self.trace.sync_dag_parent_and_hist((exp, feedback), prev_out[self.LOOP_IDX_KEY])
|
||||
|
||||
Reference in New Issue
Block a user