mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-08-06 11:37:44 +00:00
feat: refactor for general data science (#498)
* refine ds modal for more cases: eval and es * update model template * prompts for model and ensemble * fix a bug * fix a bug * init: ds workflow evovingstrategy * Adding ensemble (#505) * Initial Draft * Updating logic for init * Revising * Successful Testing * Updating to use the latest & right class * bug: bug-fixing for testing * data science loop changes * data science loop base * ds loop feedback * fix * remove measure_time because it's duplicated (in LoopBase) * add the knowledge query for data_loader & feature * edit ds workflow evaluator * data_loader bug fix * stop evolving when all tasks completed * llm app change * fix break all complete strategy * Adding queried knowledge (#508) Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com> * fix loop bug * ds workflow evaluator; test; refine prompts * workflow spec * fix ci * feature task changes * ds loop change * fix a bug in feat * add query knowledge for model and workflow * llm_debug info(for show) using pickle instead of json * remove NextLoopException * loop change * coder raise CoderError when all sub_tasks failed * rename code_dict to file_dict in FBWorkspace * add CoSTEER unittest * now show self.version in Task.get_task_information(), simplify CoSTEER sub tasks definition * remove some properties in ModelTask, add model_type in it. * fix llm app bug * llm web app bug fix * ds loop bug fix * fix: give component code to feature&ens eval * loop catch error bug * rename load_from_raw_data to load_data * feat: Add debug data creation functionality for data science scenarios * support local folder (#511) * support local folder * remove unnecessary random * KaggleScen Subclass * small fix * use template for style description * update default scen to kaggle * update sample data script * make sure frac < 1 * fix a bug * feature spec changes * fix * changeimport order * clear unnecessary std outputs * fix a typo * create sample folder after unzip kaggle data * feature/model test script update * Align the data types across modules. * fix a bug in model eval * show line number * move sample entry point to app * spec & model prompt changes * Refine the competition specification to address the data type problem and the coherence issue. * fix some bugs * add file filter in FBworkspace.code property * support non-binary prediction * avoid too much warnings * fix a bug in ensemble module * filtered the knowledge query in all modules * delete RAG in idea proposal * refine the code in ensemble * show exp workspace in llm_st * exp_gen bug fix * feedback bug fix * use `feature` instead of `feat01` * Trace & method of judging if exp is completed change * fix a bug in package calling and execute ci * fix code * bug fix * bug fix * fix a bug * fix some bugs * fix a bug * refactor: Enhance error handling and feedback in data science loop * support different use_azure on chat and embedding models * multi-model proposal logic * fix a small syntax error * loopBase and some changes * ensemble scores change * fbworkspace.code -> .all_codes * use all model codes in workflow coder * check scores.csv's keys(model_names) * model name changes * add a todo in ensemble test * sota_exp changes * give model info in exp gen * add runner time limit * config using debug data or not in evals * exp to feedback base * add feature code when writing model task * small problem * copying during sampling * update * refactor: Simplify code handling and improve workspace management * model part output fix * print model's execution time * bug fix * ensemble test fix * ens small change * ens_test bug fix * Refine partial expansion logic to display only a few subfolders when their structure is uniform, improving readability in nested directories. * several update on prompts * sample subfolders * Filter the stdout after code execution to remove irrelevant information e.g. progress bars, whitespace characters, excessive line breaks. * Add some more prompts and comments * several update on the first init rounds * model timeout as error * fix pattern of getting model codes in workspace * small bux fix on model prompts * remove get_code_with_key since we have regex pattern * fix: Correct tqdm progress bar update logic in LoopBase class * feat: Add diff generation and enhance feedback mechanism in data science loop * update some fix to model and workflow prompts * refine the logic of progress bar filter * add last_successful_exp in exp_gen * fix a one line bug * add a hint in prompt * fix data sample for bms * fix data sample for bms * hypothesis small fix * crawler readme update * fix component gen * fix bug * annotation change * load description.md if it exists * refactor: Simplify SOTA description handling in feedback and prompts * refactor: Use shared templates for feedback and experiment descriptions * change webapp for model codes changes * update proposal * add timeout message for docker run output * fix * refine the code in docker time processing * use .shape instead of len() when do shape eval * won't change size during iteration * support bson sample * sample support jsonl and bson * add former_code to coder prompts * a little speed us in debug data creating * filter progress bar when eval ens and main * avoid costeer makes no change to former code * fix several log error * add timeout judge threshold * fix some bugs in the evaluation of component output shapes * File structure for supporting litellm (#517) Co-authored-by: Young <afe.young@gmail.com> * ignore submission and show processing * ignore submission and show processing * add efficiency notice * refactor: Enhance error message with detailed feedback summary * refactor: Simplify component handling in DSExpGen class * refactor: Update code structure and add docstring for clarity * reserve one sample to each label in data sampling * add Evaluation info * refine costeer code to avoid giving same code twice * use raw_description as plain text * add a prompt hint to avoid same dict key * model task name bug in first model exp gen * fix a typo * add some debug info in costeer tests * task init change * enhance data sampling * refine the code in data_loader * more reasonable loop * fix a bug in data folder description * add error msg & traceback to execution feedback * fix llm error msg detection * add task information to costeer eval & add cache to docker run(use zipfile to store the whole workspace) * fix CI first round * fix CI second round * use txt to store test script to avoid pytest * remove zipfile in requirements * add azure.identity to requirements * ignore debug web page * component test changes * remove redundent task_desc in model coder * feat: Add APE module and prompts for automated prompt engineering * fix: Update .gitignore and improve text formatting in eval.py * refactor: Update print output and improve code comments and imports * style: Fix string formatting and import order in ape.py and fmt.py * exclude ape * add a data folder notice * reduce unnecessary output to stdout * refine the code of describe_data_folder * fix ci * style: streamlit style update (#522) * streamlit style update * fix import * fix format * fix llm_st loop progress bar * debugapp small change * fix model str * refine some prompts * fix model str * fix CI * refine the logic associated with the data_folder * fix ci * small change * set filter_progress_bar as default in execute * model proposal with workflow * add submission check in workflow eval * fix bug * small change * fix CI * fix CI * refactor: Move generate_diff to utils and update DSExpGen logic * more reasonable prompt describing metric direction * fix a minor jinja2 bug * quick fix exp_gen bugs * fix the following bug * fix * fix some bugs * remove workflow from model * add pending_tasks_list in data science to enable coding model and workflow * refine the code for handling JSON-formatted data descriptions * assert with information * ensure correct csv file name * add logging to help record the output * log competition * add log tag for debug llm app * test: Test ds refactor ll (#523) * fix bugs to former scenario * fix a bug because coding in rdloop changed * fix the bug when feedback gets no hypothesis * fix trace structure * change all trace hist when merging hypothesis to experiments * ignore some error in ruff * fix kaggle scenario bugs * refine one line * another bug * another small bug * fix ui bugs * chage kaggle train.py path --------- Co-authored-by: Xu Yang <peteryang@vip.qq.com> * fix CI * Update rdagent/app/data_science/loop.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add samplecsv into spec prompts * fix CI --------- Co-authored-by: TPLin22 <tplin2@163.com> Co-authored-by: yuanteli <1957922024@qq.com> Co-authored-by: Xisen Wang <118058822+xisen-w@users.noreply.github.com> Co-authored-by: Bowen Xian <xianbowen@outlook.com> Co-authored-by: Xu Yang <peteryang@vip.qq.com> Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com> Co-authored-by: Tim <illking@foxmail.com> Co-authored-by: 炼金术师华华 <37462254+YeewahChan@users.noreply.github.com> Co-authored-by: Linlang <30293408+SunsetWolf@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
you-n-g
@@ -105,4 +105,5 @@ class CoSTEER(Developer[Experiment]):
|
||||
pickle.dump(self.knowledge_base, open(self.new_knowledge_base_path, "wb"))
|
||||
logger.info(f"New knowledge base saved to {self.new_knowledge_base_path}")
|
||||
exp.sub_workspace_list = experiment.sub_workspace_list
|
||||
exp.experiment_workspace = experiment.experiment_workspace
|
||||
return exp
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from abc import abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
|
||||
@@ -10,8 +11,52 @@ from rdagent.core.scenario import Task
|
||||
from rdagent.core.utils import multiprocessing_wrapper
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
|
||||
# TODO:
|
||||
# 1. It seems logically sound, but we currently lack a scenario to apply it.
|
||||
# 2. If it proves to be useful, relocate it to a more general location.
|
||||
#
|
||||
# class FBWorkspaceExeFeedback(Feedback):
|
||||
# """
|
||||
# It pairs with FBWorkspace in the abstract level.
|
||||
# """
|
||||
# # ws: FBWorkspace # potential
|
||||
# stdout: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class CoSTEERSingleFeedback(Feedback):
|
||||
# TODO: (xiao)
|
||||
# it should be more general class for FBWorkspaceExeFeedback
|
||||
# A better name of it may be NormalFeedback
|
||||
# TODO: It should be a general feeddback for CoSTEERR
|
||||
"""
|
||||
The feedback for the data loader evaluation.
|
||||
It is design align the phases of the implemented code
|
||||
- Execution -> Return Value -> Code -> Final Decision
|
||||
"""
|
||||
execution: str
|
||||
# execution_feedback
|
||||
return_checking: str | None # including every check in the testing (constraints about the generated value)
|
||||
# value_feedback, shape_feedback, value_generated_flag
|
||||
code: str
|
||||
final_decision: bool
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"""------------------Execution------------------
|
||||
{self.execution}
|
||||
------------------Return Checking------------------
|
||||
{self.return_checking if self.return_checking is not None else 'No return checking'}
|
||||
------------------Code------------------
|
||||
{self.code}
|
||||
------------------Final Decision------------------
|
||||
This implementation is {'SUCCESS' if self.final_decision else 'FAIL'}.
|
||||
"""
|
||||
|
||||
def __bool__(self):
|
||||
return self.final_decision
|
||||
|
||||
|
||||
class CoSTEERSingleFeedbackDeprecated(CoSTEERSingleFeedback):
|
||||
"""This class is a base class for all code generator feedback to single implementation"""
|
||||
|
||||
def __init__(
|
||||
@@ -26,7 +71,6 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
final_decision_based_on_gt: bool = None,
|
||||
) -> None:
|
||||
self.execution_feedback = execution_feedback
|
||||
self.shape_feedback = shape_feedback
|
||||
self.code_feedback = code_feedback
|
||||
self.value_feedback = value_feedback
|
||||
self.final_decision = final_decision
|
||||
@@ -34,6 +78,26 @@ class CoSTEERSingleFeedback(Feedback):
|
||||
self.value_generated_flag = value_generated_flag
|
||||
self.final_decision_based_on_gt = final_decision_based_on_gt
|
||||
|
||||
# TODO:
|
||||
# Not general enough. So we should not put them in the general costeer feedback
|
||||
# Instead, we should create subclass for it.
|
||||
self.shape_feedback = shape_feedback # Not general enough. So
|
||||
|
||||
# TODO: @property
|
||||
@property
|
||||
def execution(self):
|
||||
return self.execution_feedback
|
||||
|
||||
@property
|
||||
def return_checking(self):
|
||||
if self.value_generated_flag:
|
||||
return f"value feedback: {self.value_feedback}\n\nshape feedback: {self.shape_feedback}"
|
||||
return None
|
||||
|
||||
@property
|
||||
def code(self):
|
||||
return self.code_feedback
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"""------------------Execution Feedback------------------
|
||||
{self.execution_feedback if self.execution_feedback is not None else 'No execution feedback'}
|
||||
@@ -73,6 +137,8 @@ class CoSTEEREvaluator(Evaluator):
|
||||
|
||||
|
||||
class CoSTEERMultiEvaluator(Evaluator):
|
||||
"""This is for evaluation of experiment. Due to we have multiple tasks, so we will return a list of evaluation feebacks"""
|
||||
|
||||
def __init__(self, single_evaluator: CoSTEEREvaluator, *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
self.single_evaluator = single_evaluator
|
||||
|
||||
@@ -1,18 +1,32 @@
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
|
||||
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
|
||||
from rdagent.core.evolving_agent import RAGEvoAgent
|
||||
from rdagent.core.evolving_framework import EvolvableSubjects
|
||||
from rdagent.core.exception import CoderError
|
||||
|
||||
|
||||
class FilterFailedRAGEvoAgent(RAGEvoAgent):
|
||||
|
||||
def filter_evolvable_subjects_by_feedback(
|
||||
self, evo: EvolvableSubjects, feedback: CoSTEERSingleFeedback
|
||||
self, evo: EvolvableSubjects, feedback: CoSTEERSingleFeedbackDeprecated
|
||||
) -> EvolvableSubjects:
|
||||
assert isinstance(evo, EvolvingItem)
|
||||
# FIXME: the list does not align with the annotation; It should be MultipleFeedback instead of a list of feedbacks
|
||||
assert isinstance(feedback, list)
|
||||
assert len(evo.sub_workspace_list) == len(feedback)
|
||||
|
||||
for index in range(len(evo.sub_workspace_list)):
|
||||
if evo.sub_workspace_list[index] is not None and feedback[index] and not feedback[index].final_decision:
|
||||
if evo.sub_workspace_list[index] is not None and feedback[index] is not None and not feedback[index]:
|
||||
evo.sub_workspace_list[index].clear()
|
||||
|
||||
failed_feedbacks = [
|
||||
f"- feedback{index + 1:02d}:\n - execution: {f.execution}\n - return_checking: {f.return_checking}\n - code: {f.code}"
|
||||
for index, f in enumerate(feedback)
|
||||
if f is not None and not f.final_decision
|
||||
]
|
||||
|
||||
if len(failed_feedbacks) == len(feedback):
|
||||
feedback_summary = "\n".join(failed_feedbacks)
|
||||
raise CoderError(f"All tasks are failed:\n{feedback_summary}")
|
||||
|
||||
return evo
|
||||
|
||||
@@ -12,7 +12,7 @@ from rdagent.components.coder.CoSTEER.scheduler import random_select
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.evaluation import Scenario
|
||||
from rdagent.core.evolving_framework import EvolvingStrategy, QueriedKnowledge
|
||||
from rdagent.core.experiment import Workspace
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.prompts import Prompts
|
||||
from rdagent.core.scenario import Task
|
||||
from rdagent.core.utils import multiprocessing_wrapper
|
||||
@@ -30,7 +30,17 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
self,
|
||||
target_task: Task,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
) -> Workspace:
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]: # FIXME: fix interface of previous implement
|
||||
"""
|
||||
This method will input the task & current workspace,
|
||||
and output the modification to applied to the workspace.
|
||||
(i.e. replace the content <filename> with <content>)
|
||||
|
||||
Return
|
||||
------
|
||||
The new files {<filename>: <content>} to update the workspace.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def select_one_round_tasks(
|
||||
@@ -86,7 +96,10 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
|
||||
|
||||
result = multiprocessing_wrapper(
|
||||
[
|
||||
(self.implement_one_task, (evo.sub_tasks[target_index], queried_knowledge))
|
||||
(
|
||||
self.implement_one_task,
|
||||
(evo.sub_tasks[target_index], queried_knowledge, evo.experiment_workspace),
|
||||
)
|
||||
for target_index in to_be_finished_task_index
|
||||
],
|
||||
n=RD_AGENT_SETTINGS.multi_proc_n,
|
||||
|
||||
@@ -48,7 +48,7 @@ class CoSTEERKnowledge(Knowledge):
|
||||
|
||||
def get_implementation_and_feedback_str(self) -> str:
|
||||
return f"""------------------implementation code:------------------
|
||||
{self.implementation.code}
|
||||
{self.implementation.all_codes}
|
||||
------------------implementation feedback:------------------
|
||||
{self.feedback!s}
|
||||
"""
|
||||
@@ -269,15 +269,15 @@ class CoSTEERRAGStrategyV2(RAGStrategy):
|
||||
else:
|
||||
# generate error node and store into knowledge base
|
||||
error_analysis_result = []
|
||||
if not single_feedback.value_generated_flag:
|
||||
if single_feedback.return_checking:
|
||||
error_analysis_result = self.analyze_error(
|
||||
single_feedback.execution_feedback,
|
||||
feedback_type="execution",
|
||||
single_feedback.return_checking,
|
||||
feedback_type="value",
|
||||
)
|
||||
else:
|
||||
error_analysis_result = self.analyze_error(
|
||||
single_feedback.value_feedback,
|
||||
feedback_type="value",
|
||||
single_feedback.execution,
|
||||
feedback_type="execution",
|
||||
)
|
||||
self.knowledgebase.working_trace_error_analysis.setdefault(
|
||||
target_task_information,
|
||||
@@ -425,8 +425,8 @@ class CoSTEERRAGStrategyV2(RAGStrategy):
|
||||
current_index = 1
|
||||
while current_index < len(former_trace_knowledge):
|
||||
if (
|
||||
not former_trace_knowledge[current_index].feedback.value_generated_flag
|
||||
and former_trace_knowledge[current_index - 1].feedback.value_generated_flag
|
||||
not former_trace_knowledge[current_index].feedback.return_checking
|
||||
and former_trace_knowledge[current_index - 1].feedback.return_checking
|
||||
):
|
||||
former_trace_knowledge.pop(current_index)
|
||||
else:
|
||||
@@ -718,7 +718,7 @@ class CoSTEERKnowledgeBaseV2(EvolvingKnowledgeBase):
|
||||
Load knowledge, offer brief information of knowledge and common handle interfaces
|
||||
"""
|
||||
self.graph: UndirectedGraph = UndirectedGraph(Path.cwd() / "graph.pkl")
|
||||
logger.info(f"Knowledge Graph loaded, size={self.graph.size()}")
|
||||
logger.info(f"CoSTEER Knowledge Graph loaded, size={self.graph.size()}")
|
||||
|
||||
if init_component_list:
|
||||
for component in init_component_list:
|
||||
|
||||
@@ -4,4 +4,6 @@ from rdagent.core.experiment import Task
|
||||
class CoSTEERTask(Task):
|
||||
def __init__(self, base_code: str = None, *args, **kwargs) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
# TODO: we may upgrade the base_code into a workspace-like thing to know previous.
|
||||
# NOTE: (xiao) think we don't need the base_code anymore. The information should be retrieved from the workspace.
|
||||
self.base_code = base_code
|
||||
|
||||
@@ -1,19 +1,131 @@
|
||||
# from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
# from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
# from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
# from rdagent.core.scenario import Scenario
|
||||
"""
|
||||
File structure
|
||||
- ___init__.py: the entrance/agent of coder
|
||||
- evaluator.py
|
||||
- conf.py
|
||||
- exp.py: everything under the experiment, e.g.
|
||||
- Task
|
||||
- Experiment
|
||||
- Workspace
|
||||
- test.py
|
||||
- Each coder could be tested.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.data_science.ensemble.eval import EnsembleCoSTEEREvaluator
|
||||
from rdagent.components.coder.data_science.ensemble.exp import EnsembleTask
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
# class ModelEnsembleCoSTEER(CoSTEER):
|
||||
# def __init__(
|
||||
# self,
|
||||
# scen: Scenario,
|
||||
# *args,
|
||||
# **kwargs,
|
||||
# ) -> None:
|
||||
# eva = CoSTEERMultiEvaluator(
|
||||
# ModelEnsembleCoSTEEREvaluator(scen=scen), scen=scen
|
||||
# ) # Please specify whether you agree running your eva in parallel or not
|
||||
# es = ModelEnsembleMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
class EnsembleMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: EnsembleTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]:
|
||||
# Get task information for knowledge querying
|
||||
ensemble_information_str = target_task.get_task_information()
|
||||
|
||||
# super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=1, scen=scen, **kwargs)
|
||||
# Query knowledge
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[ensemble_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[ensemble_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
latest_code_feedback = [
|
||||
knowledge.feedback
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("ensemble.py") is not None
|
||||
and knowledge.implementation.file_dict.get("ensemble.py") == workspace.file_dict.get("ensemble.py")
|
||||
]
|
||||
if len(latest_code_feedback) > 0:
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("ensemble.py") != workspace.file_dict.get("ensemble.py")
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# Generate code with knowledge integration
|
||||
competition_info = self.scen.get_scenario_all_desc()
|
||||
system_prompt = T(".prompts:ensemble_coder.system").r(
|
||||
task_desc=ensemble_information_str,
|
||||
competition_info=competition_info,
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
queried_former_failed_knowledge=(
|
||||
queried_former_failed_knowledge[0] if queried_former_failed_knowledge else None
|
||||
),
|
||||
)
|
||||
user_prompt = T(".prompts:ensemble_coder.user").r(
|
||||
ensemble_spec=workspace.file_dict["spec/ensemble.md"],
|
||||
latest_code=workspace.file_dict.get("ensemble.py"),
|
||||
latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None,
|
||||
)
|
||||
|
||||
for _ in range(5):
|
||||
ensemble_code = json.loads(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt, system_prompt=system_prompt, json_mode=True
|
||||
)
|
||||
)["code"]
|
||||
if ensemble_code != workspace.file_dict.get("ensemble.py"):
|
||||
break
|
||||
else:
|
||||
user_prompt = user_prompt + "\nPlease avoid generating same code to former code!"
|
||||
else:
|
||||
raise CoderError("Failed to generate a new ensemble code.")
|
||||
|
||||
return {
|
||||
"ensemble.py": ensemble_code,
|
||||
}
|
||||
|
||||
def assign_code_list_to_evo(self, code_list: list[dict[str, str]], evo):
|
||||
"""
|
||||
Assign the code list to the evolving item.
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
# evo.sub_workspace_list[index] = FBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
return evo
|
||||
|
||||
|
||||
class EnsembleCoSTEER(CoSTEER):
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
eva = CoSTEERMultiEvaluator(EnsembleCoSTEEREvaluator(scen=scen), scen=scen)
|
||||
es = EnsembleMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
|
||||
super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=2, scen=scen, **kwargs)
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
# Configuration file for ensemble component
|
||||
# Currently empty as no specific configuration is needed
|
||||
@@ -0,0 +1,75 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Environment, StrictUndefined
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
EnsembleEvalFeedback = CoSTEERSingleFeedback
|
||||
|
||||
|
||||
class EnsembleCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> EnsembleEvalFeedback:
|
||||
|
||||
target_task_information = target_task.get_task_information()
|
||||
if (
|
||||
queried_knowledge is not None
|
||||
and target_task_information in queried_knowledge.success_task_to_knowledge_dict
|
||||
):
|
||||
return queried_knowledge.success_task_to_knowledge_dict[target_task_information].feedback
|
||||
elif queried_knowledge is not None and target_task_information in queried_knowledge.failed_task_info_set:
|
||||
return EnsembleEvalFeedback(
|
||||
execution="This task has failed too many times, skip implementation.",
|
||||
code="This task has failed too many times, skip implementation.",
|
||||
return_checking="This task has failed too many times, skip implementation.",
|
||||
final_decision=False,
|
||||
)
|
||||
|
||||
ds_docker_conf = DSDockerConf()
|
||||
ds_docker_conf.extra_volumes = {
|
||||
f"{DS_RD_SETTING.local_data_path}/sample/{self.scen.competition}": "/kaggle/input"
|
||||
}
|
||||
de = DockerEnv(conf=ds_docker_conf)
|
||||
|
||||
fname = "ensemble_test.txt"
|
||||
test_code = (DIRNAME / "eval_tests" / "ensemble_test.txt").read_text()
|
||||
test_code = (
|
||||
Environment(undefined=StrictUndefined)
|
||||
.from_string(test_code)
|
||||
.render(
|
||||
model_names=[
|
||||
fn[:-3] for fn in implementation.file_dict.keys() if fn.startswith("model_") and "test" not in fn
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
stdout = implementation.execute(env=de, entry=f"python {fname}")
|
||||
|
||||
system_prompt = T(".prompts:ensemble_eval.system").r(
|
||||
task_desc=target_task_information,
|
||||
test_code=test_code,
|
||||
code=implementation.file_dict["ensemble.py"],
|
||||
)
|
||||
user_prompt = T(".prompts:ensemble_eval.user").r(stdout=stdout)
|
||||
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(user_prompt, system_prompt, json_mode=True)
|
||||
return EnsembleEvalFeedback(**json.loads(resp))
|
||||
@@ -0,0 +1,65 @@
|
||||
"""
|
||||
A qualified ensemble implementation should:
|
||||
- Successfully run
|
||||
- Return predictions
|
||||
- Have correct shapes for inputs and outputs
|
||||
- Use validation data appropriately
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
from sklearn.model_selection import train_test_split
|
||||
from load_data import load_data
|
||||
from feature import feat_eng
|
||||
from ensemble import ens_and_decision
|
||||
|
||||
X, y, test_X, test_ids = load_data()
|
||||
X, y, test_X = feat_eng(X, y, test_X)
|
||||
train_X, val_X, train_y, val_y = train_test_split(X, y, test_size=0.2, random_state=42)
|
||||
|
||||
test_preds_dict = {}
|
||||
val_preds_dict = {}
|
||||
{% for mn in model_names %}
|
||||
from {{mn}} import model_workflow as {{mn}}_workflow
|
||||
val_preds_dict["{{mn}}"], test_preds_dict["{{mn}}"], _ = {{mn}}_workflow(
|
||||
X=train_X,
|
||||
y=train_y,
|
||||
val_X=val_X,
|
||||
val_y=val_y,
|
||||
test_X=test_X
|
||||
)
|
||||
{% endfor %}
|
||||
|
||||
for key in val_preds_dict.keys():
|
||||
if val_preds_dict[key] is None:
|
||||
print(f"Model {key} validation predictions (val_preds_dict[key]) is None.")
|
||||
elif isinstance(val_preds_dict[key], list):
|
||||
print(f"Model {key} validation predictions (val_preds_dict[key]) (list type) length: {len(val_preds_dict[key])}")
|
||||
else:
|
||||
print(f"Model {key} validation predictions (val_preds_dict[key]) shape: {val_preds_dict[key].shape}")
|
||||
|
||||
if test_preds_dict[key] is None:
|
||||
print(f"Model {key} test predictions (test_preds_dict[key]) is None.")
|
||||
elif isinstance(test_preds_dict[key], list):
|
||||
print(f"Model {key} test predictions (test_preds_dict[key]) (list type) length: {len(test_preds_dict[key])}")
|
||||
else:
|
||||
print(f"Model {key} test predictions (test_preds_dict[key]) shape: {test_preds_dict[key].shape}")
|
||||
|
||||
# Run ensemble
|
||||
final_pred = ens_and_decision(test_preds_dict, val_preds_dict, val_y)
|
||||
|
||||
# Check shape
|
||||
if isinstance(final_pred, list):
|
||||
assert len(final_pred) == len(test_X), (
|
||||
f"Wrong output sample size: len(final_pred) ({len(final_pred)}) and len(test_X) ({len(test_X)})"
|
||||
)
|
||||
else:
|
||||
assert final_pred.shape[0] == test_X.shape[0], (
|
||||
f"Wrong output sample size: final_pred.shape[0] ({final_pred.shape[0]}) and test_X.shape[0] ({test_X.shape[0]})"
|
||||
)
|
||||
|
||||
# check if scores.csv is generated
|
||||
assert Path("scores.csv").exists(), "scores.csv is not generated"
|
||||
|
||||
print("Ensemble test passed successfully.")
|
||||
print(f"Output shape: {final_pred.shape}")
|
||||
@@ -0,0 +1,10 @@
|
||||
import pickle
|
||||
import site
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
|
||||
EnsembleTask = CoSTEERTask
|
||||
@@ -0,0 +1,79 @@
|
||||
ensemble_coder:
|
||||
system: |-
|
||||
You are a Python data scientist working on model ensemble implementation. Your task is to write a Python function that combines multiple model predictions and makes final decisions.
|
||||
|
||||
Your specific task as follows:
|
||||
{{task_desc}}
|
||||
|
||||
You should follow the provided specifications to complete this task.
|
||||
|
||||
-----------Competition Information-----------
|
||||
{{ competition_info }}
|
||||
|
||||
Please respond with the code in the following json format:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
-----------Here is the relevant information for this task-----------
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Successful Implementations for Similar Models:--------------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["ensemble.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Previous Failed Attempts:--------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["ensemble.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
Please implement an ensemble function with the following specification:
|
||||
-----------Ensemble Specification-----------
|
||||
{{ ensemble_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
---------Former code---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
---------Feedback to former code---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
||||
{% endif %}
|
||||
ensemble_eval:
|
||||
system: |-
|
||||
You are a data scientist evaluating an ensemble implementation.
|
||||
The main code generation task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
The ensemble code is:
|
||||
```python
|
||||
{{code}}
|
||||
```
|
||||
|
||||
You are testing the ensemble with the following code:
|
||||
```python
|
||||
{{test_code}}
|
||||
```
|
||||
|
||||
You'll be given the stdout of your testing scripts.
|
||||
Please respond with your feedback in the following JSON format:
|
||||
{
|
||||
"execution": "Describe how well the ensemble executed, including any errors or issues encountered. Please keep the error message and tracking information",
|
||||
"return_checking": "Detail the checks performed on the ensemble results, including shape and value validation.",
|
||||
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
user: |-
|
||||
{{stdout}}
|
||||
@@ -0,0 +1,58 @@
|
||||
"""
|
||||
Helper functions for testing the ensemble coder(CoSTEER-based) component.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.components.coder.data_science.ensemble import EnsembleCoSTEER
|
||||
from rdagent.components.coder.data_science.ensemble.exp import EnsembleTask
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.scen import KaggleScen
|
||||
|
||||
# Add the competition folder to path
|
||||
COMPETITION_PATH = (
|
||||
Path(__file__).parent.parent.parent.parent.parent
|
||||
/ "scenarios"
|
||||
/ "kaggle"
|
||||
/ "tpl_ex"
|
||||
/ "aerial-cactus-identification"
|
||||
)
|
||||
sys.path.append(str(COMPETITION_PATH))
|
||||
|
||||
EnsembleExperiment = DSExperiment
|
||||
|
||||
|
||||
def load_ensemble_spec():
|
||||
spec_path = COMPETITION_PATH / "spec" / "ensemble.md"
|
||||
with open(spec_path, "r") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def develop_one_competition(competition: str):
|
||||
# Initialize scenario and coder
|
||||
scen = KaggleScen(competition=competition)
|
||||
ensemble_coder = EnsembleCoSTEER(scen)
|
||||
# Load ensemble specification
|
||||
ensemble_spec = load_ensemble_spec()
|
||||
|
||||
# Create the ensemble task with actual data context and specification
|
||||
task = EnsembleTask(
|
||||
name="EnsembleTask",
|
||||
description="""
|
||||
Implement ensemble and decision making for model predictions.
|
||||
""",
|
||||
)
|
||||
|
||||
exp = EnsembleExperiment(sub_tasks=[task])
|
||||
|
||||
# Injecting the corresponding specification
|
||||
exp.experiment_workspace.inject_files(**{"spec/ensemble.md": ensemble_spec})
|
||||
|
||||
# Develop the experiment
|
||||
exp = ensemble_coder.develop(exp)
|
||||
return exp
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
develop_one_competition("aerial-cactus-identification")
|
||||
@@ -0,0 +1,118 @@
|
||||
import json
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.data_science.feature.eval import FeatureCoSTEEREvaluator
|
||||
from rdagent.components.coder.data_science.feature.exp import FeatureTask
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class FeatureMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: FeatureTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]:
|
||||
# return a workspace with "load_data.py", "spec/load_data.md" inside
|
||||
# assign the implemented code to the new workspace.
|
||||
feature_information_str = target_task.get_task_information()
|
||||
|
||||
# 1. query
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[feature_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[feature_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
latest_code_feedback = [
|
||||
knowledge.feedback
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("feature.py") is not None
|
||||
and knowledge.implementation.file_dict.get("feature.py") == workspace.file_dict.get("feature.py")
|
||||
]
|
||||
if len(latest_code_feedback) > 0:
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("feature.py") != workspace.file_dict.get("feature.py")
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# 2. code
|
||||
system_prompt = T(".prompts:feature.system").r(
|
||||
task_desc=feature_information_str,
|
||||
data_loader_code=workspace.file_dict.get("load_data.py"),
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
)
|
||||
user_prompt = T(".prompts:feature.user").r(
|
||||
feature_spec=workspace.file_dict["spec/feature.md"],
|
||||
latest_code=workspace.file_dict.get("feature.py"),
|
||||
latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None,
|
||||
)
|
||||
|
||||
for _ in range(5):
|
||||
feature_code = json.loads(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt, system_prompt=system_prompt, json_mode=True
|
||||
)
|
||||
)["code"]
|
||||
if feature_code != workspace.file_dict.get("feature.py"):
|
||||
break
|
||||
else:
|
||||
user_prompt = user_prompt + "\nPlease avoid generating same code to former code!"
|
||||
else:
|
||||
raise CoderError("Failed to generate a new feature code.")
|
||||
|
||||
return {
|
||||
"feature.py": feature_code,
|
||||
}
|
||||
|
||||
def assign_code_list_to_evo(self, code_list: list[dict[str, str]], evo):
|
||||
"""
|
||||
Assign the code list to the evolving item.
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
# evo.sub_workspace_list[index] = FBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
return evo
|
||||
|
||||
|
||||
class FeatureCoSTEER(CoSTEER):
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
eva = CoSTEERMultiEvaluator(
|
||||
FeatureCoSTEEREvaluator(scen=scen), scen=scen
|
||||
) # Please specify whether you agree running your eva in parallel or not
|
||||
es = FeatureMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
|
||||
super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=2, scen=scen, **kwargs)
|
||||
@@ -0,0 +1,68 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
from rdagent.utils.fmt import shrink_text
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
FeatureEvalFeedback = CoSTEERSingleFeedback
|
||||
|
||||
|
||||
class FeatureCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> FeatureEvalFeedback:
|
||||
|
||||
target_task_information = target_task.get_task_information()
|
||||
if (
|
||||
queried_knowledge is not None
|
||||
and target_task_information in queried_knowledge.success_task_to_knowledge_dict
|
||||
):
|
||||
return queried_knowledge.success_task_to_knowledge_dict[target_task_information].feedback
|
||||
elif queried_knowledge is not None and target_task_information in queried_knowledge.failed_task_info_set:
|
||||
return FeatureEvalFeedback(
|
||||
execution="This task has failed too many times, skip implementation.",
|
||||
return_checking="This task has failed too many times, skip implementation.",
|
||||
code="This task has failed too many times, skip implementation.",
|
||||
final_decision=False,
|
||||
)
|
||||
|
||||
ds_docker_conf = DSDockerConf()
|
||||
# TODO: we should /= 20 for the timeout period on debug component
|
||||
ds_docker_conf.extra_volumes = {
|
||||
f"{DS_RD_SETTING.local_data_path}/sample/{self.scen.competition}": "/kaggle/input"
|
||||
}
|
||||
de = DockerEnv(conf=ds_docker_conf)
|
||||
|
||||
# TODO: do we need to clean the generated temporary content?
|
||||
fname = "feature_test.py"
|
||||
test_code = (DIRNAME / "eval_tests" / "feature_test.txt").read_text()
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
|
||||
stdout = implementation.execute(env=de, entry=f"python {fname}")
|
||||
|
||||
system_prompt = T(".prompts:feature_eval.system").r(
|
||||
task_desc=target_task.get_task_information(),
|
||||
test_code=test_code,
|
||||
code=implementation.file_dict["feature.py"],
|
||||
)
|
||||
user_prompt = T(".prompts:feature_eval.user").r(stdout=shrink_text(stdout))
|
||||
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(user_prompt, system_prompt, json_mode=True)
|
||||
return FeatureEvalFeedback(**json.loads(resp))
|
||||
@@ -0,0 +1,35 @@
|
||||
"""
|
||||
A qualified data loader should support following features
|
||||
- successfully run
|
||||
- len(test) == len(test_ids) == submission length
|
||||
- len(train) == len(y)
|
||||
|
||||
Please make sure the stdout is rich enough to support informative feedback
|
||||
"""
|
||||
|
||||
import pickle
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from feature import feat_eng
|
||||
from load_data import load_data
|
||||
|
||||
X, y, X_test, test_ids = load_data()
|
||||
print(f"X.shape: {X.shape}")
|
||||
print(f"y.shape: {y.shape}" if not isinstance(y, list) else f"y(list)'s length: {len(y)}")
|
||||
print(f"X_test.shape: {X_test.shape}")
|
||||
print(f"test_ids length: {len(test_ids)}")
|
||||
X, y, X_test = feat_eng(X, y, X_test)
|
||||
|
||||
|
||||
def get_length(data):
|
||||
return len(data) if isinstance(data, list) else data.shape[0]
|
||||
|
||||
assert get_length(X_test) == get_length(test_ids), (
|
||||
f"Mismatch in length of test images and test IDs: X_test ({get_length(X_test)}) and test_ids ({get_length(test_ids)})"
|
||||
)
|
||||
assert get_length(X) == get_length(y), (
|
||||
f"Mismatch in length of training images and labels: X ({get_length(X)}) and y ({get_length(y)})"
|
||||
)
|
||||
|
||||
print("Feature Engineering test passed successfully. Length of test images matches length of test IDs.")
|
||||
@@ -0,0 +1,10 @@
|
||||
import pickle
|
||||
import site
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
|
||||
FeatureTask = CoSTEERTask
|
||||
@@ -0,0 +1,91 @@
|
||||
feature:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
Your task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
This project involves implementing feature engineering techniques to prepare data for machine learning models, and this project code will be written by GPT.
|
||||
Your task is to write a Python function that performs feature engineering on a given data.
|
||||
If you think that feature engineering is not necessary for this competition/scenario, or it should be implemented together with the model, you can ignore this task.
|
||||
You should follow the provided specifications to complete this task.
|
||||
|
||||
Your function input is the output of a data loading function, the data loader function code is as follows:
|
||||
```python
|
||||
{{data_loader_code}}
|
||||
```
|
||||
Please understand the code and try to implement the feature engineering function based on the data loader output.
|
||||
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
-----------Here is the relevant information for this task-----------
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Successful Implementations for Similar Models:--------------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["feature.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Previous Failed Attempts:--------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["feature.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
---------Feature Processing Specification---------
|
||||
{{ feature_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
---------Former code---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
---------Feedback to former code---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
||||
{% endif %}
|
||||
|
||||
|
||||
feature_eval:
|
||||
system: |-
|
||||
You are data scientist whose job is to evaluate the feature processing code generation.
|
||||
The main code generation task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
The feature code is:
|
||||
```python
|
||||
{{code}}
|
||||
```
|
||||
|
||||
You are testing the feature processing code with the following code
|
||||
```python
|
||||
{{test_code}}
|
||||
```
|
||||
|
||||
You'll be given the stdout of your testing scripts.
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the feature processing executed, including any errors or issues encountered. Please keep the error message and tracking information",
|
||||
"return_checking": "Detail the checks performed on the data after feature processing, including data integrity and correctness.",
|
||||
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
user: |-
|
||||
```
|
||||
{{stdout}}
|
||||
```
|
||||
@@ -0,0 +1,37 @@
|
||||
"""
|
||||
Helper functions for testing the feature coder(CoSTEER-based) component.
|
||||
- Does the developer loop work correctly
|
||||
|
||||
It is NOT:
|
||||
- it is not interface unittest(i.e. workspace evaluator in the CoSTEER Loop)
|
||||
"""
|
||||
|
||||
from rdagent.components.coder.data_science.feature import FeatureCoSTEER
|
||||
from rdagent.components.coder.data_science.feature.exp import FeatureTask
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.scen import KaggleScen
|
||||
|
||||
|
||||
def develop_one_competition(competition: str): # -> experiment
|
||||
scen = KaggleScen(competition=competition)
|
||||
feature_coder = FeatureCoSTEER(scen)
|
||||
|
||||
with open("./rdagent/scenarios/kaggle/tpl_ex/aerial-cactus-identification/spec/feature.md", "r") as file:
|
||||
feat_spec = file.read()
|
||||
|
||||
# Create the experiment
|
||||
ft = FeatureTask(name="FeatureTask", description=scen.get_competition_full_desc())
|
||||
exp = DSExperiment(
|
||||
sub_tasks=[ft],
|
||||
)
|
||||
|
||||
with open("./rdagent/scenarios/kaggle/tpl_ex/aerial-cactus-identification/load_data.py", "r") as file:
|
||||
load_data_code = file.read()
|
||||
exp.experiment_workspace.inject_files(**{"load_data.py": load_data_code, "spec/feature.md": feat_spec})
|
||||
|
||||
# Develop the experiment
|
||||
exp = feature_coder.develop(exp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
develop_one_competition("aerial-cactus-identification")
|
||||
@@ -1,19 +0,0 @@
|
||||
# from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
# from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
# from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
# from rdagent.core.scenario import Scenario
|
||||
|
||||
|
||||
# class FeatureCoSTEER(CoSTEER):
|
||||
# def __init__(
|
||||
# self,
|
||||
# scen: Scenario,
|
||||
# *args,
|
||||
# **kwargs,
|
||||
# ) -> None:
|
||||
# eva = CoSTEERMultiEvaluator(
|
||||
# FeatureCoSTEEREvaluator(scen=scen), scen=scen
|
||||
# ) # Please specify whether you agree running your eva in parallel or not
|
||||
# es = FeatureMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
|
||||
# super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=1, scen=scen, **kwargs)
|
||||
@@ -1,19 +1,144 @@
|
||||
# from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
# from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
# from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
# from rdagent.core.scenario import Scenario
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Environment, StrictUndefined
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.data_science.model.eval import (
|
||||
ModelGeneralCaseSpecEvaluator,
|
||||
)
|
||||
from rdagent.components.coder.data_science.model.exp import ModelTask
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.ret import BatchEditOut
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
# class ModelCoSTEER(CoSTEER):
|
||||
# def __init__(
|
||||
# self,
|
||||
# scen: Scenario,
|
||||
# *args,
|
||||
# **kwargs,
|
||||
# ) -> None:
|
||||
# eva = CoSTEERMultiEvaluator(
|
||||
# ModelCoSTEEREvaluator(scen=scen), scen=scen
|
||||
# ) # Please specify whether you agree running your eva in parallel or not
|
||||
# es = ModelMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
class ModelMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: ModelTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]:
|
||||
model_information_str = target_task.get_task_information()
|
||||
|
||||
# super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=1, scen=scen, **kwargs)
|
||||
# 1. query
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[model_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[model_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
latest_code_feedback = [
|
||||
knowledge.feedback
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(f"{target_task.name}.py") is not None
|
||||
and knowledge.implementation.file_dict.get(f"{target_task.name}.py")
|
||||
== workspace.file_dict.get(f"{target_task.name}.py")
|
||||
]
|
||||
if len(latest_code_feedback) > 0:
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get(f"{target_task.name}.py")
|
||||
!= workspace.file_dict.get(f"{target_task.name}.py")
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# 2. code
|
||||
system_prompt = T(".prompts:model_coder.system").r(
|
||||
task_desc=model_information_str,
|
||||
data_loader_code=workspace.file_dict.get("load_data.py"),
|
||||
feature_code=workspace.file_dict["feature.py"],
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
out_spec=BatchEditOut.get_spec(),
|
||||
)
|
||||
# user_prompt = T(".prompts:model_coder.user").r(
|
||||
# model_spec=workspace.file_dict["spec/model.md"],
|
||||
# feature_code=workspace.file_dict["feature.py"],
|
||||
# latest_code=workspace.file_dict.get(f"{target_task.name}.py", None),
|
||||
# )
|
||||
# We want to use a simpler way to
|
||||
user_prompt = T(".prompts:model_coder.user_general").r(
|
||||
model_spec=workspace.file_dict["spec/model.md"],
|
||||
workspace_code=workspace.get_codes(
|
||||
r"^model_(?!test)\w+\.py$"
|
||||
), # TODO: If we have high failure rate here, we should clean this step with less information.
|
||||
latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None,
|
||||
)
|
||||
|
||||
for _ in range(5):
|
||||
batch_edit = BatchEditOut.extract_output(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt,
|
||||
system_prompt=system_prompt,
|
||||
json_mode=BatchEditOut.json_mode,
|
||||
)
|
||||
)
|
||||
|
||||
# 3. post process to align file name to the task name
|
||||
batch_edit = {
|
||||
(f"{target_task.name}.py" if value != "__DEL__" and key != f"{target_task.name}.py" else key): value
|
||||
for key, value in batch_edit.items()
|
||||
}
|
||||
|
||||
if batch_edit[f"{target_task.name}.py"] != "__DEL__" and batch_edit[
|
||||
f"{target_task.name}.py"
|
||||
] != workspace.file_dict.get(f"{target_task.name}.py"):
|
||||
break
|
||||
else:
|
||||
user_prompt = user_prompt + "\nPlease avoid generating same code to former code!"
|
||||
else:
|
||||
raise CoderError("Failed to generate a new model code.")
|
||||
|
||||
return batch_edit
|
||||
|
||||
def assign_code_list_to_evo(self, code_list: list[dict[str, str]], evo):
|
||||
"""
|
||||
Assign the code list to the evolving item.
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
# evo.sub_workspace_list[index] = FBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
return evo
|
||||
|
||||
|
||||
class ModelCoSTEER(CoSTEER):
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
eva = CoSTEERMultiEvaluator(
|
||||
ModelGeneralCaseSpecEvaluator(scen=scen), scen=scen
|
||||
) # Please specify whether you agree running your eva in parallel or not
|
||||
# eva = ModelGeneralCaseSpecEvaluator(scen=scen)
|
||||
es = ModelMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
|
||||
super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=2, scen=scen, **kwargs)
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
"""
|
||||
Beyond previous tests
|
||||
-
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
ModelSingleFeedback = CoSTEERSingleFeedback
|
||||
|
||||
|
||||
# Below are unit tests for testing the specification of the implemented model ------------------
|
||||
class ModelGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
"""
|
||||
Motivation case:
|
||||
- Simplest case, we already split the data into train_data, valid_data, and test_data. We require the model to learn (optionally validate on valid data), and infer on test data.
|
||||
|
||||
Test workflow:
|
||||
- Build train, valid, and test data to run it, and test the output (e.g., shape, etc.)
|
||||
"""
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> ModelSingleFeedback:
|
||||
target_task_information = target_task.get_task_information()
|
||||
if (
|
||||
queried_knowledge is not None
|
||||
and target_task_information in queried_knowledge.success_task_to_knowledge_dict
|
||||
):
|
||||
return queried_knowledge.success_task_to_knowledge_dict[target_task_information].feedback
|
||||
elif queried_knowledge is not None and target_task_information in queried_knowledge.failed_task_info_set:
|
||||
return ModelSingleFeedback(
|
||||
execution="This task has failed too many times, skip implementation.",
|
||||
return_checking="This task has failed too many times, skip implementation.",
|
||||
code="This task has failed too many times, skip implementation.",
|
||||
final_decision=False,
|
||||
)
|
||||
|
||||
ds_docker_conf = DSDockerConf()
|
||||
ds_docker_conf.extra_volumes = {
|
||||
f"{DS_RD_SETTING.local_data_path}/sample/{self.scen.competition}": "/kaggle/input"
|
||||
}
|
||||
de = DockerEnv(conf=ds_docker_conf)
|
||||
|
||||
fname = "model_test.py"
|
||||
test_code = (
|
||||
(DIRNAME / "eval_tests" / "model_test.txt").read_text().replace("model01", target_task.name)
|
||||
) # only check the model changed this time
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
stdout = implementation.execute(env=de, entry=f"python {fname}")
|
||||
|
||||
if stdout is None:
|
||||
raise CoderError(
|
||||
"The execution output contains too many progress bars and results in the LLM's token size exceeding the limit."
|
||||
)
|
||||
|
||||
system_prompt = T(".prompts:model_eval.system").r(
|
||||
task_desc=target_task.get_task_information(),
|
||||
test_code=test_code,
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
spec=implementation.file_dict["spec/model.md"],
|
||||
)
|
||||
user_prompt = T(".prompts:model_eval.user").r(
|
||||
stdout=stdout,
|
||||
code=implementation.file_dict[f"{target_task.name}.py"],
|
||||
)
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(user_prompt, system_prompt, json_mode=True)
|
||||
return ModelSingleFeedback(**json.loads(resp))
|
||||
@@ -0,0 +1,53 @@
|
||||
import time
|
||||
|
||||
from feature import feat_eng
|
||||
from load_data import load_data
|
||||
from model01 import model_workflow
|
||||
from sklearn.model_selection import train_test_split
|
||||
|
||||
|
||||
def log_execution_results(start_time, val_pred, test_pred, hypers, execution_label):
|
||||
"""Log the results of a single model execution."""
|
||||
feedback_str = f"{execution_label} successful.\n"
|
||||
feedback_str += f"Validation predictions shape: {val_pred.shape if val_pred is not None else 'None'}\n"
|
||||
feedback_str += f"Test predictions shape: {test_pred.shape if test_pred is not None else 'None'}\n"
|
||||
feedback_str += f"Hyperparameters: {hypers if hypers is not None else 'None'}\n"
|
||||
feedback_str += f"Execution time: {time.time() - start_time:.2f} seconds.\n"
|
||||
print(feedback_str)
|
||||
|
||||
|
||||
# Load and preprocess data
|
||||
X, y, test_X, test_ids = load_data()
|
||||
X, y, test_X = feat_eng(X, y, test_X)
|
||||
train_X, val_X, train_y, val_y = train_test_split(X, y, test_size=0.8, random_state=42)
|
||||
print(f"train_X.shape: {train_X.shape}")
|
||||
print(f"train_y.shape: {train_y.shape}" if not isinstance(train_y, list) else f"train_y(list)'s length: {len(train_y)}")
|
||||
print(f"val_X.shape: {val_X.shape}")
|
||||
print(f"val_y.shape: {val_y.shape}" if not isinstance(val_y, list) else f"val_y(list)'s length: {len(val_y)}")
|
||||
|
||||
# First execution
|
||||
print("The first execution begins.\n")
|
||||
start_time = time.time()
|
||||
val_pred, test_pred, hypers = model_workflow(
|
||||
X=train_X,
|
||||
y=train_y,
|
||||
val_X=val_X,
|
||||
val_y=val_y,
|
||||
test_X=None,
|
||||
)
|
||||
log_execution_results(start_time, val_pred, test_pred, hypers, "The first execution")
|
||||
|
||||
# Second execution
|
||||
print("The second execution begins.\n")
|
||||
start_time = time.time()
|
||||
val_pred, test_pred, final_hypers = model_workflow(
|
||||
X=train_X,
|
||||
y=train_y,
|
||||
val_X=None,
|
||||
val_y=None,
|
||||
test_X=test_X,
|
||||
hyper_params=hypers,
|
||||
)
|
||||
log_execution_results(start_time, val_pred, test_pred, final_hypers, "The second execution")
|
||||
|
||||
print("Model code test passed successfully.")
|
||||
@@ -0,0 +1,43 @@
|
||||
import pickle
|
||||
import site
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
from rdagent.core.experiment import Experiment, FBWorkspace
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
from rdagent.oai.llm_utils import md5_hash
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
|
||||
class ModelTask(CoSTEERTask):
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
description: str,
|
||||
architecture: str = "",
|
||||
*args,
|
||||
hyperparameters: Dict[str, str] = {},
|
||||
model_type: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
self.architecture: str = architecture
|
||||
self.hyperparameters: str = hyperparameters
|
||||
self.model_type: str | None = (
|
||||
model_type # Tabular for tabular model, TimesSeries for time series model, Graph for graph model, XGBoost for XGBoost model
|
||||
# TODO: More Models Supported
|
||||
)
|
||||
super().__init__(name=name, description=description, *args, **kwargs)
|
||||
|
||||
def get_task_information(self):
|
||||
task_desc = f"""name: {self.name}
|
||||
description: {self.description}
|
||||
"""
|
||||
if self.architecture:
|
||||
task_desc += f"architecture: {self.architecture}\n"
|
||||
if self.hyperparameters:
|
||||
task_desc += f"hyperparameters: {self.hyperparameters}\n"
|
||||
if self.model_type:
|
||||
task_desc += f"model_type: {self.model_type}\n"
|
||||
return task_desc
|
||||
@@ -0,0 +1,136 @@
|
||||
model_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
Your task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
The user's ultimate goal is to obtain accurate predictions from the model on input data. Follow the instructions below to ensure your response is correct and aligned with the user's expectations.
|
||||
|
||||
Your function's input is from the output of a feature engineering function whose input is the output of a data loading function. The raw data loader function and feature engineer function code is as follows:
|
||||
--------- Raw Data Loader Code: ---------
|
||||
{{data_loader_code}}
|
||||
--------- Feature Engineering Code: ---------
|
||||
{{feature_code}}
|
||||
|
||||
Instructions for Code Generation:
|
||||
Leveraging User Inputs:
|
||||
The user may provide various forms of additional information to guide you:
|
||||
|
||||
Successful Examples: Correct implementations of similar models.
|
||||
Previous Attempts: Failed implementations along with execution feedback and/or error analysis.
|
||||
Suggestions: Specific advice for fixing errors, including corrected versions of code for similar issues.
|
||||
Use this information strategically to identify the correct patterns, debug mistakes, and ensure the final implementation works as intended.
|
||||
|
||||
Preserving Correct Code:
|
||||
If the user has shared their latest code, carefully analyze it and only modify parts that require changes. Do not alter correct sections of the code.
|
||||
|
||||
Error Learning:
|
||||
If previous failed attempts and their feedback are available, learn from them. Understand what went wrong and avoid repeating similar mistakes in your new implementation.
|
||||
The failure knowledge may include the code unrelated to the model, such as data loading, preprocessing, or feature engineering. Focus only on the model implementation part.
|
||||
|
||||
{% if out_spec %}
|
||||
{{out_spec}}
|
||||
The file name should be the model name described in the model task in the format "{task_name}.py". You should always follow this name format.
|
||||
{% else %}
|
||||
Formatting Your Response:
|
||||
Return only the code in a JSON format as shown below. Do not include any explanations or extra text. Example:
|
||||
{
|
||||
"code": "Your corrected or newly implemented Python code as a single string"
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
-----------Here is the relevant information for this task-----------
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Successful Implementations for Similar Models:--------------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict[similar_successful_knowledge.target_task.name ~ '.py'] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Previous Failed Attempts:--------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict[former_failed_knowledge.target_task.name ~ '.py'] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
---------Model Specification---------
|
||||
{{ model_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
---------Former Code---------
|
||||
Former Code: {{ latest_code }}
|
||||
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
||||
{% endif %}
|
||||
|
||||
user_general: |-
|
||||
--------- Workspace code---------
|
||||
{% if workspace_code|length == 0 %}
|
||||
So far the workspace is empty. No model code has been implemented yet.
|
||||
{% else %}
|
||||
{{ workspace_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
---------Feedback to former code---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
---------Model Specification---------
|
||||
When you are implementing the code, you should follow the spec
|
||||
{{ model_spec }}
|
||||
|
||||
|
||||
model_eval:
|
||||
system: |-
|
||||
You are a data scientist.
|
||||
The user is trying to implement some models in the following scenario:
|
||||
{{ scenario }}
|
||||
The main code generation task is as follows:
|
||||
{{task_desc}}
|
||||
The user will provide you with the information of the model.
|
||||
The information about how to implement the model is given in spec.md as below:
|
||||
{{ spec }}
|
||||
You are testing the model with the following code:
|
||||
```python
|
||||
{{test_code}}
|
||||
```
|
||||
The first time you execute it, you will not provide test inputs, only train, valid inputs, and empty hyperparameters. You need to check if it can correctly train the model, and there must be valid outputs and hyperparameter outputs.
|
||||
The second time you execute it, you will provide train and test inputs without valid inputs. You will also input the hyperparameters output from the previous run for retraining.
|
||||
Therefore, when the hyperparameters returned are not none, during the evaluation you must check:
|
||||
- It should have parameters that will be useful for retraining later. It must include the early stop round.
|
||||
- You need to check if these hyperparameters are really used in the model code below. The early stop round must be used if given.
|
||||
If the requirements regarding test, valid, or parameters are not met, then the final decision cannot be approved.
|
||||
|
||||
You should evaluate the code given by the user. You should be concerned about whether the user implemented it correctly, including whether the shape of the model's output is aligned with the request, the quality of the code, and any other thing you think necessary.
|
||||
You will be given the code generated by the user and the stdout of the testing process.
|
||||
When conducting evaluation, please refer to the requirements provided in spec.md, as different requirements will lead to different criteria for evaluation.
|
||||
|
||||
Only if there is "Model code test passed successfully." in the stdout, then the model is considered successful, or else there must be some issues with the model.
|
||||
If no stdout is provided, the model is considered to have failed due to a timeout. Please check if there are any ways to improve the model's execution speed.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order:
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the model executed successfully, including any errors or issues encountered. Please keep the error message and tracking information",
|
||||
"return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.",
|
||||
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process. Check whether the hyperparameters from the previous run are used in the model code, compare the parameter names in stdout and if they are used in the retraining part of the code. It is acceptable when hyperparameters is None.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------------Code generated by user:---------------
|
||||
{{ code }}
|
||||
--------------stdoutput:---------------
|
||||
'''
|
||||
{{ stdout }}
|
||||
'''
|
||||
@@ -0,0 +1,67 @@
|
||||
"""
|
||||
Generate dataset to test the model workflow output
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.data_science.model import ModelCoSTEER
|
||||
from rdagent.components.coder.data_science.model.eval import (
|
||||
ModelGeneralCaseSpecEvaluator,
|
||||
)
|
||||
from rdagent.components.coder.data_science.model.exp import ModelTask
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.scen import KaggleScen
|
||||
|
||||
|
||||
# Take tasks, spec.md and feat as input, generate a feedback as output
|
||||
def develop_one_competition(competition: str):
|
||||
scen = KaggleScen(competition=competition)
|
||||
model_coder = ModelCoSTEER(scen)
|
||||
|
||||
# Create the task
|
||||
mt = ModelTask(
|
||||
name="ModelTask",
|
||||
description="A CNN Model",
|
||||
model_type="CNN",
|
||||
architecture="\hat{y}_u = CNN(X_u)",
|
||||
# variables="variables: {'\\hat{y}_u': 'The predicted output for node u', 'X_u': 'The input features for node u'}",
|
||||
hyperparameters="...",
|
||||
base_code="",
|
||||
)
|
||||
|
||||
tpl_ex_path = Path(__file__).resolve() / Path("rdagent/scenarios/kaggle/tpl_ex").resolve() / competition
|
||||
injected_file_names = ["spec/model.md", "load_data.py", "feature.py", "model01.py"]
|
||||
|
||||
modelexp = FBWorkspace()
|
||||
for file_name in injected_file_names:
|
||||
file_path = tpl_ex_path / file_name
|
||||
modelexp.inject_files(**{file_name: file_path.read_text()})
|
||||
|
||||
mt.base_code += modelexp.file_dict["model01.py"]
|
||||
exp = DSExperiment(
|
||||
sub_tasks=[mt],
|
||||
)
|
||||
|
||||
# Test the evaluator:
|
||||
"""eva = ModelGeneralCaseSpecEvaluator(scen=scen)
|
||||
exp.feedback = eva.evaluate(target_task=mt, queried_knowledge=None, implementation=modelexp, gt_implementation=None)
|
||||
print(exp.feedback)"""
|
||||
|
||||
# Test the evolving strategy:
|
||||
"""es = ModelMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
new_code = es.implement_one_task(target_task=mt, queried_knowledge=None, workspace=modelexp)
|
||||
print(new_code)"""
|
||||
|
||||
# Run the experiment
|
||||
for file_name in injected_file_names:
|
||||
file_path = tpl_ex_path / file_name
|
||||
exp.experiment_workspace.inject_files(**{file_name: file_path.read_text()})
|
||||
|
||||
exp = model_coder.develop(exp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
develop_one_competition("aerial-cactus-identification")
|
||||
# dotenv run -- python rdagent/components/coder/data_science/model/test.py
|
||||
@@ -0,0 +1,15 @@
|
||||
# CoSTEER
|
||||
|
||||
- subworkspace使用主experiment_workspace `RD-Agent/rdagent/scenarios/data_science/experiment/experiment.py`
|
||||
|
||||
## evolving_strategy ( implement_one_task() )
|
||||
|
||||
1. xxxTask (in exp.py)
|
||||
- spec
|
||||
- description
|
||||
2.
|
||||
|
||||
## evaluator
|
||||
|
||||
1. queried_knowledge部分 共用
|
||||
2. eval_test脚本
|
||||
@@ -1,19 +1,196 @@
|
||||
# from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
# from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
# from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
# from rdagent.core.scenario import Scenario
|
||||
"""
|
||||
|
||||
Loop should not large change exclude
|
||||
- Action Choice[current data loader & spec]
|
||||
- other should share
|
||||
- Propose[choice] => Task[Choice] => CoSTEER =>
|
||||
-
|
||||
|
||||
Extra feature:
|
||||
- cache
|
||||
|
||||
|
||||
# class DataLoaderCoSTEER(CoSTEER):
|
||||
# def __init__(
|
||||
# self,
|
||||
# scen: Scenario,
|
||||
# *args,
|
||||
# **kwargs,
|
||||
# ) -> None:
|
||||
# eva = CoSTEERMultiEvaluator(
|
||||
# DataLoaderCoSTEEREvaluator(scen=scen), scen=scen
|
||||
# ) # Please specify whether you agree running your eva in parallel or not
|
||||
# es = DataLoaderMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
File structure
|
||||
- ___init__.py: the entrance/agent of coder
|
||||
- evaluator.py
|
||||
- conf.py
|
||||
- exp.py: everything under the experiment, e.g.
|
||||
- Task
|
||||
- Experiment
|
||||
- Workspace
|
||||
- test.py
|
||||
- Each coder could be tested.
|
||||
"""
|
||||
|
||||
# super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=1, scen=scen, **kwargs)
|
||||
import json
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
CoSTEERQueriedKnowledgeV2,
|
||||
)
|
||||
from rdagent.components.coder.data_science.raw_data_loader.eval import (
|
||||
DataLoaderCoSTEEREvaluator,
|
||||
)
|
||||
from rdagent.components.coder.data_science.raw_data_loader.exp import DataLoaderTask
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
class DataLoaderMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: DataLoaderTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]:
|
||||
# return a workspace with "load_data.py", "spec/load_data.md" inside
|
||||
# assign the implemented code to the new workspace.
|
||||
competition_info = self.scen.get_scenario_all_desc()
|
||||
data_folder_info = self.scen.processed_data_folder_description
|
||||
data_loader_task_info = target_task.get_task_information()
|
||||
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[data_loader_task_info]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[data_loader_task_info]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
latest_code_feedback = [
|
||||
knowledge.feedback
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("load_data.py") is not None
|
||||
and knowledge.implementation.file_dict.get("load_data.py") == workspace.file_dict.get("load_data.py")
|
||||
]
|
||||
if len(latest_code_feedback) > 0:
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("load_data.py") != workspace.file_dict.get("load_data.py")
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# 1. specifications
|
||||
# TODO: We may move spec into a separated COSTEER task
|
||||
if "spec/data_loader.md" not in workspace.file_dict: # Only generate the spec once
|
||||
system_prompt = T(".prompts:spec.system").r(
|
||||
task_desc=data_loader_task_info,
|
||||
competition_info=competition_info,
|
||||
folder_spec=data_folder_info,
|
||||
)
|
||||
data_loader_prompt = T(".prompts:spec.user.data_loader").r(
|
||||
latest_spec=workspace.file_dict.get("spec/data_loader.md")
|
||||
)
|
||||
feature_prompt = T(".prompts:spec.user.feature").r(latest_spec=workspace.file_dict.get("spec/feature.md"))
|
||||
model_prompt = T(".prompts:spec.user.model").r(latest_spec=workspace.file_dict.get("spec/model.md"))
|
||||
ensemble_prompt = T(".prompts:spec.user.ensemble").r(
|
||||
latest_spec=workspace.file_dict.get("spec/ensemble.md")
|
||||
)
|
||||
workflow_prompt = T(".prompts:spec.user.workflow").r(
|
||||
latest_spec=workspace.file_dict.get("spec/workflow.md")
|
||||
)
|
||||
|
||||
spec_session = APIBackend().build_chat_session(session_system_prompt=system_prompt)
|
||||
|
||||
data_loader_spec = json.loads(
|
||||
spec_session.build_chat_completion(user_prompt=data_loader_prompt, json_mode=True)
|
||||
)["spec"]
|
||||
feature_spec = json.loads(spec_session.build_chat_completion(user_prompt=feature_prompt, json_mode=True))[
|
||||
"spec"
|
||||
]
|
||||
model_spec = json.loads(spec_session.build_chat_completion(user_prompt=model_prompt, json_mode=True))[
|
||||
"spec"
|
||||
]
|
||||
ensemble_spec = json.loads(spec_session.build_chat_completion(user_prompt=ensemble_prompt, json_mode=True))[
|
||||
"spec"
|
||||
]
|
||||
workflow_spec = json.loads(spec_session.build_chat_completion(user_prompt=workflow_prompt, json_mode=True))[
|
||||
"spec"
|
||||
]
|
||||
else:
|
||||
data_loader_spec = workspace.file_dict["spec/data_loader.md"]
|
||||
feature_spec = workspace.file_dict["spec/feature.md"]
|
||||
model_spec = workspace.file_dict["spec/model.md"]
|
||||
ensemble_spec = workspace.file_dict["spec/ensemble.md"]
|
||||
workflow_spec = workspace.file_dict["spec/workflow.md"]
|
||||
|
||||
# 2. code
|
||||
system_prompt = T(".prompts:data_loader_coder.system").r(
|
||||
task_desc=data_loader_task_info,
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
)
|
||||
user_prompt = T(".prompts:data_loader_coder.user").r(
|
||||
competition_info=competition_info,
|
||||
data_loader_spec=data_loader_spec,
|
||||
folder_spec=data_folder_info,
|
||||
latest_code=workspace.file_dict.get("load_data.py"),
|
||||
latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None,
|
||||
)
|
||||
|
||||
for _ in range(5):
|
||||
data_loader_code = json.loads(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt, system_prompt=system_prompt, json_mode=True
|
||||
)
|
||||
)["code"]
|
||||
if data_loader_code != workspace.file_dict.get("load_data.py"):
|
||||
break
|
||||
else:
|
||||
user_prompt = user_prompt + "\nPlease avoid generating same code to former code!"
|
||||
else:
|
||||
raise CoderError("Failed to generate a new data loader code.")
|
||||
|
||||
return {
|
||||
"spec/data_loader.md": data_loader_spec,
|
||||
"spec/feature.md": feature_spec,
|
||||
"spec/model.md": model_spec,
|
||||
"spec/ensemble.md": ensemble_spec,
|
||||
"spec/workflow.md": workflow_spec,
|
||||
"load_data.py": data_loader_code,
|
||||
}
|
||||
|
||||
def assign_code_list_to_evo(self, code_list: list[dict[str, str]], evo):
|
||||
"""
|
||||
Assign the code list to the evolving item.
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
# evo.sub_workspace_list[index] = FBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
return evo
|
||||
|
||||
|
||||
class DataLoaderCoSTEER(CoSTEER):
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
eva = CoSTEERMultiEvaluator(
|
||||
DataLoaderCoSTEEREvaluator(scen=scen), scen=scen
|
||||
) # Please specify whether you agree running your eva in parallel or not
|
||||
es = DataLoaderMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
|
||||
super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=2, scen=scen, **kwargs)
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# tess successfully running.
|
||||
# (GPT) if it aligns with the spec & rationality of the spec.
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERSingleFeedback,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledgeV2,
|
||||
)
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
DataLoaderEvalFeedback = CoSTEERSingleFeedback
|
||||
|
||||
|
||||
class DataLoaderCoSTEEREvaluator(CoSTEEREvaluator):
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: CoSTEERQueriedKnowledgeV2 = None,
|
||||
**kwargs,
|
||||
) -> DataLoaderEvalFeedback:
|
||||
|
||||
target_task_information = target_task.get_task_information()
|
||||
if (
|
||||
queried_knowledge is not None
|
||||
and target_task_information in queried_knowledge.success_task_to_knowledge_dict
|
||||
):
|
||||
return queried_knowledge.success_task_to_knowledge_dict[target_task_information].feedback
|
||||
elif queried_knowledge is not None and target_task_information in queried_knowledge.failed_task_info_set:
|
||||
return DataLoaderEvalFeedback(
|
||||
execution="This task has failed too many times, skip implementation.",
|
||||
return_checking="This task has failed too many times, skip implementation.",
|
||||
code="This task has failed too many times, skip implementation.",
|
||||
final_decision=False,
|
||||
)
|
||||
|
||||
ds_docker_conf = DSDockerConf()
|
||||
ds_docker_conf.extra_volumes = {
|
||||
f"{DS_RD_SETTING.local_data_path}/sample/{self.scen.competition}": "/kaggle/input"
|
||||
}
|
||||
de = DockerEnv(conf=ds_docker_conf)
|
||||
|
||||
# TODO: do we need to clean the generated temporary content?
|
||||
fname = "data_loader_test.py"
|
||||
test_code = (DIRNAME / "eval_tests" / "data_loader_test.txt").read_text()
|
||||
implementation.inject_files(**{fname: test_code})
|
||||
stdout = implementation.execute(env=de, entry=f"python {fname}")
|
||||
|
||||
system_prompt = T(".prompts:data_loader_eval.system").r(
|
||||
task_desc=target_task.get_task_information(),
|
||||
test_code=test_code,
|
||||
code=implementation.file_dict["load_data.py"],
|
||||
)
|
||||
user_prompt = T(".prompts:data_loader_eval.user").r(stdout=stdout)
|
||||
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(user_prompt, system_prompt, json_mode=True)
|
||||
return DataLoaderEvalFeedback(**json.loads(resp))
|
||||
@@ -0,0 +1,25 @@
|
||||
"""
|
||||
A qualified data loader should support following features
|
||||
- successfully run
|
||||
- len(test) == len(test_ids) == submission length
|
||||
- len(train) == len(y)
|
||||
|
||||
Please make sure the stdout is rich enough to support informative feedback
|
||||
"""
|
||||
|
||||
import pickle
|
||||
from load_data import load_data
|
||||
|
||||
X, y, X_test, test_ids = load_data()
|
||||
|
||||
def get_length(data):
|
||||
return len(data) if isinstance(data, list) else data.shape[0]
|
||||
|
||||
assert get_length(X_test) == get_length(test_ids), (
|
||||
f"Mismatch in length of test images and test IDs: X_test ({get_length(X_test)}) and test_ids ({get_length(test_ids)})"
|
||||
)
|
||||
assert get_length(X) == get_length(y), (
|
||||
f"Mismatch in length of training images and labels: X ({get_length(X)}) and y ({get_length(y)})"
|
||||
)
|
||||
|
||||
print("Data loader test passed successfully. Length of test images matches length of test IDs.")
|
||||
@@ -0,0 +1,14 @@
|
||||
import pickle
|
||||
import site
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
from rdagent.core.experiment import Experiment, FBWorkspace
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
from rdagent.oai.llm_utils import md5_hash
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
DataLoaderTask = CoSTEERTask
|
||||
@@ -0,0 +1,386 @@
|
||||
spec:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
Currently, you are working on a Kaggle competition project.
|
||||
This project involves analyzing data and building models to beat other competitors, with the code being generated by large language models.
|
||||
|
||||
Your overall task is provided below:
|
||||
{{ task_desc }}
|
||||
|
||||
Your task is to write five specification texts (in markdown format) for the following tasks, based on the competition information provided
|
||||
- Data loading (and preprocessing)
|
||||
- Feature Engineering
|
||||
- Model Building
|
||||
- Ensemble
|
||||
- The overall workflow
|
||||
|
||||
The specifications for each step should be tailored to the competition information provided.
|
||||
|
||||
Your specification should consists two parts:
|
||||
1. The function definition in code format with detailed annotation to each parameter and return value.
|
||||
2. A detailed docstring to the function that explains the purpose of the function, the input parameters, and the output.
|
||||
3. Additional information or notes that the coder should consider while implementing the function.
|
||||
Your specifications should not include any code implementation, only the function definition and docstring.
|
||||
|
||||
-----------Competition Information-----------
|
||||
{{ competition_info }}
|
||||
|
||||
-----------Folder Description---------(All path are relative to the data folder)
|
||||
- Ensure that all columns in sample_submission can be generated.
|
||||
{{ folder_spec }}
|
||||
|
||||
user:
|
||||
data_loader: |-
|
||||
Data loader specification text should follow these detailed requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `load_data`
|
||||
- Input: No input arguments.
|
||||
- Output:
|
||||
- `X` (DT, define based on competition information): Feature matrix for training data.
|
||||
- `y` (DT): Target vector for training data.
|
||||
- `X_test` (DT): Feature matrix for test data.
|
||||
- `test_ids` (DT): Identifiers for the test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Specify the data source location (`/kaggle/input/`).
|
||||
- Clearly define the structure and type of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Precautions for Data Loading and Preprocessing:
|
||||
- File Handling:
|
||||
- Ensure proper file encoding (e.g., UTF-8) and delimiters (e.g., CSV comma-separated).
|
||||
- Combine or process multiple files if necessary.
|
||||
- Data Preprocessing:
|
||||
- Convert data types correctly (e.g., numeric, categorical, date parsing).
|
||||
- Handle missing values appropriately (e.g., impute, drop rows/columns).
|
||||
- Optimize memory usage for large datasets using techniques like downcasting or reading data in chunks if necessary.
|
||||
- Domain-Specific Handling:
|
||||
- Apply competition-specific preprocessing steps as needed (e.g., text tokenization, image resizing).
|
||||
|
||||
3. Code Standards:
|
||||
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
4. Notes:
|
||||
- Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
|
||||
- Extend domain-specific handling steps based on the competition information.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
|
||||
|
||||
Please respond with a JSON structure as follows:
|
||||
{
|
||||
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
||||
}
|
||||
|
||||
feature: |-
|
||||
Feature engineering specification text should adhere to the following requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `feat_eng`
|
||||
- Parameters:
|
||||
- `X` (DT): Train data to be transformed.
|
||||
- `y` (DT): Train label data.
|
||||
- `X_test` (DT): Test data.
|
||||
- Output:
|
||||
- `X_transformed` (DT): Transformed train data.
|
||||
- `y_transformed` (DT): Transformed train label data.
|
||||
- `X_test_transformed` (DT): Transformed test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Precautions for Feature Engineering:
|
||||
- Well handle the shape of the data
|
||||
- The sample size of the train data and the test data should be the same in all scenarios.
|
||||
- To most of the scenario, the input shape and the output shape should be exactly the same.
|
||||
- To some tabular data, you may add or remove some columns so your inferred column number may be unsure.
|
||||
- Integration with Model Pipeline
|
||||
- If feature engineering is strictly part of the model pipeline, state explicitly that it will be handled at the model stage.
|
||||
- If integrated here, ensure this function applies all required transformations while avoiding data leakage.
|
||||
- General Considerations:
|
||||
- Ensure scalability for large datasets.
|
||||
- Handle missing values and outliers appropriately (e.g., impute, remove, or replace).
|
||||
- Ensure consistency between feature data types and transformations.
|
||||
- Avoid data leakage: Only use features derived from training data, excluding information from test or validation sets.
|
||||
- Domain-Specific Features:
|
||||
- Apply logic for competition-specific features (e.g., text vectorization, image augmentations, categorical encoding).
|
||||
|
||||
3. Code Standards:
|
||||
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) definitions with those in the Data Loader specification.
|
||||
- Extend or adjust domain-specific transformations based on competition requirements.
|
||||
- The device has GPU support, so you can use it for feature engineering if necessary to accelerate the process.
|
||||
- Multi processing or parallel processing can be used to speed up the feature engineering process.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
Please respond with a JSON structure as follows:
|
||||
{
|
||||
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
||||
}
|
||||
|
||||
model: |-
|
||||
Model building specification text should adhere to the following requirements:
|
||||
|
||||
1. Function Interface:
|
||||
- Function Name: `model_workflow`
|
||||
- Parameters:
|
||||
- `X` (DT): Training feature data.
|
||||
- `y` (DT): Training label data.
|
||||
- `val_X` (Optional[DT]): Validation feature data.
|
||||
- `val_y` (Optional[DT]): Validation label data.
|
||||
- `test_X` (Optional[DT]): Test feature data.
|
||||
- `hyper_params` (dict): Dictionary of hyperparameters for model configuration.
|
||||
- Output:
|
||||
- `pred_val` (Optional[DT]): Predictions on validation data.
|
||||
- `pred_test` (Optional[DT]): Predictions on test data.
|
||||
- `hyper_params` (dict): Updated dictionary of hyperparameters after training.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Code Standards:
|
||||
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
3. Precautions:
|
||||
- Ensure input arrays (`X`, `y`, `val_X`, `val_y`, `test_X`) have consistent dimensions and shapes.
|
||||
- Use default values for hyperparameters if `hyper_params` is not provided.
|
||||
- Train the model on `X` and `y`.
|
||||
- Evaluate the model using `val_X` and `val_y` if validation data is available.
|
||||
- If `test_X` is provided, generate predictions for it.
|
||||
- Do not use progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) with the definitions used in Feature Engineering specifications.
|
||||
- The device has GPU support, so you can use it for training if necessary to accelerate the process.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
Please respond in the following JSON format:
|
||||
{
|
||||
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
||||
}
|
||||
|
||||
ensemble: |-
|
||||
Ensemble specification text adhere to the following requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `ens_and_decision`
|
||||
- Parameters:
|
||||
- `test_preds_dict` (Dict[str, DT]): A dictionary of test predictions from different models. The key is the model file name.
|
||||
- `val_preds_dict` (Dict[str, DT]): A dictionary of validation predictions from different models. The key is the model file name.
|
||||
- `val_label` (DT): Validation label.
|
||||
- Output:
|
||||
- `final_pred` (DT): Ensemble prediction for the test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Precautions:
|
||||
- Validation of Inputs:
|
||||
- Ensure all predictions in `test_preds_dict` and `val_preds_dict` have consistent shapes and dimensions.
|
||||
- Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions.
|
||||
- Handle empty or invalid inputs gracefully with appropriate error messages.
|
||||
- You should calculate the metric for each model and ensemble strategy, and save the results in `scores.csv`, e.g.:
|
||||
```python
|
||||
scores = {}
|
||||
for model_name, val_pred in val_preds_dict.items():
|
||||
scores[model_name] = calculate_metric(val_label, val_pred)
|
||||
|
||||
... some code about ensemble strategy ...
|
||||
ensemble_score = calculate_metric(val_label, ensemble_pred)
|
||||
scores[<ensemble_strategy_name>] = ensemble_score
|
||||
|
||||
scores_df = pd.DataFrame(scores.items(), columns=['Model', <metric_name>])
|
||||
scores_df.to_csv("scores.csv", index=False)
|
||||
```
|
||||
- Consensus Strategy:
|
||||
- Clearly define how the ensemble predictions are aggregated (e.g., majority voting, weighted average).
|
||||
- Avoid introducing biases or overfitting during decision-making.
|
||||
|
||||
3. Code Standards:
|
||||
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) definitions with those used in model specifications.
|
||||
- Ensure flexibility to handle multiple ensemble strategies based on competition requirements.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
Please respond in the following JSON format:
|
||||
{
|
||||
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
||||
}
|
||||
|
||||
workflow: |-
|
||||
Your task is to implement the main workflow script (`main.py`) for a Kaggle-style machine learning competition project.
|
||||
Follow the provided project structure and specifications to ensure consistency and maintainability:
|
||||
1. Workflow Integration:
|
||||
- Integrate the following components into the workflow:
|
||||
- Data loading (`load_data.py`).
|
||||
- Feature engineering (`feature.py`).
|
||||
- Model workflow for training and testing (`model_*.py`).
|
||||
- Ensemble and decision-making (`ensemble.py`).
|
||||
- Treat each component as a modular and callable Python function.
|
||||
2. Feature Engineering
|
||||
- The feature engineering should be called only once. For example:
|
||||
`X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)`
|
||||
- It should be called before dataset splitting.
|
||||
|
||||
3. Dataset Splitting
|
||||
- The dataset returned by `load_data` is not split into training and testing sets, so the dataset splitting should happen after calling `feat_eng`.
|
||||
- By default, split the dataset into 80% for training and 20% for testing.
|
||||
- You can also use cross-validation or other splitting methods as you deem more useful and appropriate based on the Competition Information.
|
||||
|
||||
4. Submission File:
|
||||
- Save the final predictions as `submission.csv` in the format required by the competition.
|
||||
- Present the required submission format explicitly and ensure the output adheres to it.
|
||||
|
||||
5. Code Standards:
|
||||
- Use consistent naming conventions and type annotations.
|
||||
- Document the workflow with clear comments and docstring.
|
||||
- Do not use progress bars (e.g., tqdm) in the code.
|
||||
|
||||
6. Ensemble Strategy:
|
||||
Put all the model's return into a dict, using the model file name as key, and the return as value.
|
||||
Sample code:
|
||||
{% raw %}
|
||||
{% for mn in model_names %}
|
||||
from {{mn}} import model_workflow as {{mn}}_workflow
|
||||
val_preds_dict["{{mn}}"], test_preds_dict["{{mn}}"], _ = {{mn}}_workflow(
|
||||
X=train_X,
|
||||
y=train_y,
|
||||
val_X=val_X,
|
||||
val_y=val_y,
|
||||
test_X=test_X
|
||||
)
|
||||
{% endfor %}
|
||||
final_pred = ens_and_decision(test_preds_dict, val_preds_dict, val_y)
|
||||
{% endraw %}
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
Please response the specification in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"spec": "The corresponding specification string as described above. You should create the rules based on the competition information instead of copying the requirements."
|
||||
}
|
||||
|
||||
data_loader_coder:
|
||||
system: |-
|
||||
You are a Python data scientist working on a new project. This project will be used to analyze data and build models to predict future outcomes, and this project codes will be written by GPT.
|
||||
Your task is described below:
|
||||
{{ task_desc }}
|
||||
You should follow the provided specifications to complete this task.
|
||||
You need to write the corresponding data loading code based on the information provided in the user's Data Folder Description, rather than relying on any suggestions that might exist in the spec.
|
||||
|
||||
Notice, the data files are stored in the data folder located at `/kaggle/input/`, and the data folder is structured as described in the Data Folder Description. Please don't load the data from the current directory.
|
||||
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
-----------Here is the relevant information for this task-----------
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Successful Implementations for Similar Models:--------------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Previous Failed Attempts:--------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
---------Competition Information---------
|
||||
{{ competition_info }}
|
||||
|
||||
---------Data Loader Specification---------
|
||||
{{ data_loader_spec }}
|
||||
|
||||
---------Data Folder Description---------(All path are relative to the data folder)
|
||||
{{ folder_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
---------Former code---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
---------Feedback to former code---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
||||
{% endif %}
|
||||
|
||||
You should strictly follow the function interface specifications provided by the specification to implement the function.
|
||||
|
||||
|
||||
data_loader_eval:
|
||||
system: |-
|
||||
You are data scientist writing some data loader code for a Kaggle-style machine learning competition project.
|
||||
The main code generation task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
The data loader code is in a file named "load_data.py":
|
||||
```python
|
||||
{{code}}
|
||||
```
|
||||
|
||||
You are testing the data_loader with the following code
|
||||
```python
|
||||
{{test_code}}
|
||||
```
|
||||
|
||||
You'll be given the stdout of your testing scripts.
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the data loader executed, including any errors or issues encountered. Please keep the error message and tracking information",
|
||||
"return_checking": "Detail the checks performed on the data loaded, including data integrity and correctness.",
|
||||
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
user: |-
|
||||
```
|
||||
{{stdout}}
|
||||
```
|
||||
@@ -0,0 +1,30 @@
|
||||
"""
|
||||
Helper functions for testing the raw_data_loader coder(CoSTEER-based) component.
|
||||
- Does the developer loop work correctly
|
||||
|
||||
It is NOT:
|
||||
- it is not interface unittest(i.e. workspace evaluator in the CoSTEER Loop)
|
||||
"""
|
||||
|
||||
from rdagent.components.coder.data_science.raw_data_loader import DataLoaderCoSTEER
|
||||
from rdagent.components.coder.data_science.raw_data_loader.exp import DataLoaderTask
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.scen import KaggleScen
|
||||
|
||||
|
||||
def develop_one_competition(competition: str): # -> experiment
|
||||
scen = KaggleScen(competition=competition)
|
||||
data_loader_coder = DataLoaderCoSTEER(scen)
|
||||
|
||||
# Create the experiment
|
||||
dlt = DataLoaderTask(name="DataLoaderTask", description="")
|
||||
exp = DSExperiment(
|
||||
sub_tasks=[dlt],
|
||||
)
|
||||
|
||||
# Develop the experiment
|
||||
exp = data_loader_coder.develop(exp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
develop_one_competition("aerial-cactus-identification")
|
||||
@@ -1,19 +1,120 @@
|
||||
# from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
# from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
# from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
# from rdagent.core.scenario import Scenario
|
||||
import json
|
||||
|
||||
from rdagent.components.coder.CoSTEER import CoSTEER
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
|
||||
from rdagent.components.coder.CoSTEER.evolving_strategy import (
|
||||
MultiProcessEvolvingStrategy,
|
||||
)
|
||||
from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
CoSTEERQueriedKnowledge,
|
||||
)
|
||||
from rdagent.components.coder.data_science.workflow.eval import (
|
||||
WorkflowGeneralCaseSpecEvaluator,
|
||||
)
|
||||
from rdagent.components.coder.data_science.workflow.exp import WorkflowTask
|
||||
from rdagent.core.exception import CoderError
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.scenario import Scenario
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
|
||||
|
||||
# class WorkflowCoSTEER(CoSTEER):
|
||||
# def __init__(
|
||||
# self,
|
||||
# scen: Scenario,
|
||||
# *args,
|
||||
# **kwargs,
|
||||
# ) -> None:
|
||||
# eva = CoSTEERMultiEvaluator(
|
||||
# WorkflowCoSTEEREvaluator(scen=scen), scen=scen
|
||||
# ) # Please specify whether you agree running your eva in parallel or not
|
||||
# es = WorkflowMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
class WorkflowMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
def implement_one_task(
|
||||
self,
|
||||
target_task: WorkflowTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge | None = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> dict[str, str]:
|
||||
# competition_info = self.scen.competition_descriptions
|
||||
workflow_information_str = target_task.get_task_information()
|
||||
|
||||
# super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=1, scen=scen, **kwargs)
|
||||
# 1. query
|
||||
queried_similar_successful_knowledge = (
|
||||
queried_knowledge.task_to_similar_task_successful_knowledge[workflow_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
queried_former_failed_knowledge = (
|
||||
queried_knowledge.task_to_former_failed_traces[workflow_information_str]
|
||||
if queried_knowledge is not None
|
||||
else []
|
||||
)
|
||||
latest_code_feedback = [
|
||||
knowledge.feedback
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("main.py") is not None
|
||||
and knowledge.implementation.file_dict.get("main.py") == workspace.file_dict.get("main.py")
|
||||
]
|
||||
if len(latest_code_feedback) > 0:
|
||||
queried_former_failed_knowledge = (
|
||||
[
|
||||
knowledge
|
||||
for knowledge in queried_former_failed_knowledge[0]
|
||||
if knowledge.implementation.file_dict.get("main.py") != workspace.file_dict.get("main.py")
|
||||
],
|
||||
queried_former_failed_knowledge[1],
|
||||
)
|
||||
|
||||
# 2. code
|
||||
system_prompt = T(".prompts:workflow_coder.system").r(
|
||||
task_desc=workflow_information_str,
|
||||
competition_info=self.scen.get_competition_full_desc(),
|
||||
queried_similar_successful_knowledge=queried_similar_successful_knowledge,
|
||||
queried_former_failed_knowledge=queried_former_failed_knowledge[0],
|
||||
)
|
||||
user_prompt = T(".prompts:workflow_coder.user").r(
|
||||
load_data_code=workspace.file_dict["load_data.py"],
|
||||
feature_code=workspace.file_dict["feature.py"],
|
||||
model_codes=workspace.get_codes(r"^model_(?!test)\w+\.py$"),
|
||||
ensemble_code=workspace.file_dict["ensemble.py"],
|
||||
latest_code=workspace.file_dict.get("main.py"),
|
||||
workflow_spec=workspace.file_dict["spec/workflow.md"],
|
||||
latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None,
|
||||
)
|
||||
|
||||
for _ in range(5):
|
||||
workflow_code = json.loads(
|
||||
APIBackend().build_messages_and_create_chat_completion(
|
||||
user_prompt=user_prompt, system_prompt=system_prompt, json_mode=True
|
||||
)
|
||||
)["code"]
|
||||
if workflow_code != workspace.file_dict.get("main.py"):
|
||||
break
|
||||
else:
|
||||
user_prompt = user_prompt + "\nPlease avoid generating same code to former code!"
|
||||
else:
|
||||
raise CoderError("Failed to generate a new workflow code.")
|
||||
|
||||
return {"main.py": workflow_code}
|
||||
|
||||
def assign_code_list_to_evo(self, code_list: list[dict[str, str]], evo):
|
||||
"""
|
||||
Assign the code list to the evolving item.
|
||||
|
||||
The code list is aligned with the evolving item's sub-tasks.
|
||||
If a task is not implemented, put a None in the list.
|
||||
"""
|
||||
for index in range(len(evo.sub_tasks)):
|
||||
if code_list[index] is None:
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
# evo.sub_workspace_list[index] = FBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index] = evo.experiment_workspace
|
||||
evo.sub_workspace_list[index].inject_files(**code_list[index])
|
||||
return evo
|
||||
|
||||
|
||||
class WorkflowCoSTEER(CoSTEER):
|
||||
def __init__(
|
||||
self,
|
||||
scen: Scenario,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
eva = CoSTEERMultiEvaluator(
|
||||
WorkflowGeneralCaseSpecEvaluator(scen=scen), scen=scen
|
||||
) # Please specify whether you agree running your eva in parallel or not
|
||||
es = WorkflowMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
super().__init__(*args, settings=CoSTEER_SETTINGS, eva=eva, es=es, evolving_version=2, scen=scen, **kwargs)
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERMultiFeedback,
|
||||
CoSTEERSingleFeedback,
|
||||
CoSTEERSingleFeedbackDeprecated,
|
||||
)
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import FBWorkspace, Task
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
from rdagent.utils.agent.tpl import T
|
||||
from rdagent.utils.env import DockerEnv, DSDockerConf
|
||||
|
||||
DIRNAME = Path(__file__).absolute().resolve().parent
|
||||
|
||||
WorkflowSingleFeedback = CoSTEERSingleFeedback
|
||||
WorkflowMultiFeedback = CoSTEERMultiFeedback
|
||||
|
||||
|
||||
class WorkflowGeneralCaseSpecEvaluator(CoSTEEREvaluator):
|
||||
"""
|
||||
Motivation case:
|
||||
- Simplest case, we already split the data into train_data, valid_data, and test_data. We require the model to learn (optionally validate on valid data), and infer on test data.
|
||||
|
||||
Test workflow:
|
||||
- Build train, valid, and test data to run it, and test the output (e.g., shape, etc.)
|
||||
"""
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
target_task: Task,
|
||||
implementation: FBWorkspace,
|
||||
gt_implementation: FBWorkspace,
|
||||
queried_knowledge: QueriedKnowledge = None,
|
||||
**kwargs,
|
||||
) -> CoSTEERSingleFeedbackDeprecated:
|
||||
target_task_information = target_task.get_task_information()
|
||||
if (
|
||||
queried_knowledge is not None
|
||||
and target_task_information in queried_knowledge.success_task_to_knowledge_dict
|
||||
):
|
||||
return queried_knowledge.success_task_to_knowledge_dict[target_task_information].feedback
|
||||
elif queried_knowledge is not None and target_task_information in queried_knowledge.failed_task_info_set:
|
||||
return WorkflowSingleFeedback(
|
||||
execution="This task has failed too many times, skip implementation.",
|
||||
return_checking="This task has failed too many times, skip implementation.",
|
||||
code="This task has failed too many times, skip implementation.",
|
||||
final_decision=False,
|
||||
)
|
||||
ds_docker_conf = DSDockerConf()
|
||||
ds_docker_conf.extra_volumes = {
|
||||
f"{DS_RD_SETTING.local_data_path}/sample/{self.scen.competition}": "/kaggle/input"
|
||||
}
|
||||
de = DockerEnv(conf=ds_docker_conf)
|
||||
fname = "main.py"
|
||||
stdout = implementation.execute(env=de, entry=f"python {fname}")
|
||||
|
||||
# Check score file
|
||||
score_fp = implementation.workspace_path / "scores.csv"
|
||||
if not score_fp.exists():
|
||||
stdout += "\nMetrics file (scores.csv) is not generated."
|
||||
else:
|
||||
score_df = pd.read_csv(score_fp, index_col=0)
|
||||
model_set_in_scores = set(score_df.index)
|
||||
model_set_in_folder = set(
|
||||
f[:-3] for f in implementation.file_dict.keys() if re.match(r"^model_(?!test)\w+\.py$", f)
|
||||
)
|
||||
for model in model_set_in_folder:
|
||||
if model not in model_set_in_scores:
|
||||
stdout += (
|
||||
f"\nModel {model} is not evaluated in the scores.csv. The scores.csv has {model_set_in_scores}."
|
||||
)
|
||||
|
||||
# Check submission file
|
||||
submission_fp = implementation.workspace_path / "submission.csv"
|
||||
if not submission_fp.exists():
|
||||
stdout += "\nSubmission file (submission.csv) is not generated."
|
||||
else:
|
||||
check_code = (DIRNAME / "eval_tests" / "submission_check.txt").read_text()
|
||||
implementation.inject_files(**{"submission_check.py": check_code})
|
||||
stdout += implementation.execute(env=de, entry="python submission_check.py")
|
||||
|
||||
system_prompt = T(".prompts:workflow_eval.system").r(
|
||||
scenario=self.scen.get_scenario_all_desc(),
|
||||
task_desc=target_task.get_task_information(),
|
||||
spec=implementation.file_dict["spec/workflow.md"],
|
||||
)
|
||||
user_prompt = T(".prompts:workflow_eval.user").r(
|
||||
stdout=stdout.strip(),
|
||||
code=implementation.file_dict["main.py"],
|
||||
)
|
||||
resp = APIBackend().build_messages_and_create_chat_completion(user_prompt, system_prompt, json_mode=True)
|
||||
return WorkflowSingleFeedback(**json.loads(resp))
|
||||
@@ -0,0 +1,18 @@
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
|
||||
# Check if the sample submission file exists
|
||||
if not Path("/kaggle/input/sample_submission.csv").exists():
|
||||
exit(0)
|
||||
|
||||
sample_submission = pd.read_csv('/kaggle/input/sample_submission.csv')
|
||||
our_submission = pd.read_csv('submission.csv')
|
||||
|
||||
success = True
|
||||
for col in sample_submission.columns:
|
||||
if col not in our_submission.columns:
|
||||
success = False
|
||||
print(f'Column {col} not found in submission.csv')
|
||||
|
||||
if success:
|
||||
print('submission.csv is valid.')
|
||||
@@ -0,0 +1,10 @@
|
||||
import pickle
|
||||
import site
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
|
||||
WorkflowTask = CoSTEERTask
|
||||
@@ -0,0 +1,115 @@
|
||||
workflow_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
Your specific task is as follows:
|
||||
{{task_desc}}
|
||||
|
||||
Your current competition information is as follows:
|
||||
{{ competition_info }}
|
||||
|
||||
The user has written different Python functions that can load and preprocess data, execute feature engineering, train models, and ensemble them.
|
||||
|
||||
These Python codes with different functionalities are written separately in different Python files.
|
||||
You don't need to edit the existing code. Your task is to integrate the existing processes of load_data, feature, model, and ensemble into a complete workflow.
|
||||
This workflow code is also a Python file, and it functions similarly to a main process that calls the sub-files for each step and ultimately outputs a prediction file.
|
||||
|
||||
The user will also provide specifications on how to organize the code and give instructions.
|
||||
|
||||
The code you implement should align with the framework given in the specifications.
|
||||
After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code.
|
||||
|
||||
Please respond with the code in the following JSON format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
-----------Here is the relevant information for this task-----------
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Successful Implementations for Similar Models:--------------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["main.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Previous Failed Attempts:--------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["main.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
---------Workflow Specification---------
|
||||
{{ workflow_spec }}
|
||||
|
||||
---------load data code---------
|
||||
file: load_data.py
|
||||
{{ load_data_code }}
|
||||
|
||||
---------feature engineering code---------
|
||||
file: feature.py
|
||||
{{ feature_code }}
|
||||
|
||||
---------model training code---------
|
||||
Attention: The input and output of the model function is flexible. Training dataset is necessary, but validation and test dateset might be optional. The hyperparameters can either be passed as arguments or be set as default values in the function. You need to use the function correctly.
|
||||
All model files share the same function name. Please import the model files with their name like: from {file_name} import {function_name}
|
||||
{{ model_codes }}
|
||||
|
||||
---------ensemble code---------
|
||||
Note, we will check the index of the score.csv, so please use the model name as the index to feed into ensemble function.
|
||||
file: ensemble.py
|
||||
{{ ensemble_code }}
|
||||
|
||||
{% if latest_code %}
|
||||
---------Former code---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
---------Feedback to former code---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
||||
{% endif %}
|
||||
|
||||
workflow_eval:
|
||||
system: |-
|
||||
You are a data scientist.
|
||||
The user is trying to build a workflow in the following scenario:
|
||||
{{ scenario }}
|
||||
The main code generation task is as follows:
|
||||
{{task_desc}}
|
||||
The user will provide you with the information of the workflow and its components.
|
||||
The information about how to build the workflow is given in the specification file as below:
|
||||
{{ spec }}
|
||||
This workflow will import all the codes including data loading, feature engineering, model tuning, and ensembling.
|
||||
You are testing it by running the workflow code. The results will be collected as the stdout and it will help you evaluate the code.
|
||||
|
||||
Your job is to evaluate the workflow code given by the user. You should be concerned about whether the code executes successfully, generates predictions correctly, and satisfies other requirements in the specification.
|
||||
The components have already been evaluated by the user, so you only need to evaluate and improve the workflow code unless there are very serious issues with the components.
|
||||
|
||||
Your evaluation should only consider whether the code executes successfully, generates well formatted predictions, and aligns with the target task. The performance of the model is not a concern in this task.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order:
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the model executed successfully, including any errors or issues encountered. Please keep the error message and tracking information",
|
||||
"return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in the specification. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.",
|
||||
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Check whether the hyperparameters from the previous run are used in the model code, compare the parameter names in stdout and if they are used in the retraining part of the code.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
user: |-
|
||||
--------------Code generated by user:---------------
|
||||
{{ code }}
|
||||
--------------stdoutput:---------------
|
||||
'''
|
||||
{{ stdout }}
|
||||
'''
|
||||
@@ -0,0 +1,59 @@
|
||||
"""
|
||||
Generate dataset to test the workflow output
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
|
||||
from rdagent.components.coder.data_science.workflow import WorkflowCoSTEER
|
||||
from rdagent.components.coder.data_science.workflow.eval import (
|
||||
WorkflowGeneralCaseSpecEvaluator,
|
||||
)
|
||||
from rdagent.components.coder.data_science.workflow.exp import WorkflowTask
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
|
||||
from rdagent.scenarios.data_science.scen import KaggleScen
|
||||
|
||||
|
||||
def develop_one_competition(competition: str):
|
||||
scen = KaggleScen(competition=competition)
|
||||
workflow_coder = WorkflowCoSTEER(scen)
|
||||
|
||||
wt = WorkflowTask(
|
||||
name="WorkflowTask",
|
||||
description="Integrate the existing processes of load_data, feature, model, and ensemble into a complete workflow.",
|
||||
base_code="",
|
||||
)
|
||||
|
||||
tpl_ex_path = Path(__file__).resolve() / Path("rdagent/scenarios/kaggle/tpl_ex").resolve() / competition
|
||||
injected_file_names = ["spec/workflow.md", "load_data.py", "feature.py", "model01.py", "ensemble.py", "main.py"]
|
||||
|
||||
workflowexp = FBWorkspace()
|
||||
for file_name in injected_file_names:
|
||||
file_path = tpl_ex_path / file_name
|
||||
workflowexp.inject_files(**{file_name: file_path.read_text()})
|
||||
|
||||
wt.base_code += workflowexp.file_dict["main.py"]
|
||||
exp = DSExperiment(
|
||||
sub_tasks=[wt],
|
||||
)
|
||||
|
||||
"""es = WorkflowMultiProcessEvolvingStrategy(scen=scen, settings=CoSTEER_SETTINGS)
|
||||
new_code = es.implement_one_task(target_task=wt, queried_knowledge=None, workspace = workflowexp)
|
||||
print(new_code)"""
|
||||
|
||||
"""eva = WorkflowGeneralCaseSpecEvaluator(scen=scen)
|
||||
exp.feedback = eva.evaluate(target_task=wt, queried_knowledge=None, implementation=workflowexp, gt_implementation=None)
|
||||
print(exp.feedback)"""
|
||||
|
||||
# Run the experiment
|
||||
for file_name in injected_file_names:
|
||||
file_path = tpl_ex_path / file_name
|
||||
exp.experiment_workspace.inject_files(**{file_name: file_path.read_text()})
|
||||
|
||||
exp = workflow_coder.develop(exp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
develop_one_competition("aerial-cactus-identification")
|
||||
# dotenv run -- python rdagent/components/coder/data_science/workflow/test.py
|
||||
@@ -79,7 +79,7 @@ class FactorCodeEvaluator(FactorEvaluator):
|
||||
**kwargs,
|
||||
):
|
||||
factor_information = target_task.get_task_information()
|
||||
code = implementation.code
|
||||
code = implementation.all_codes
|
||||
|
||||
system_prompt = (
|
||||
Environment(undefined=StrictUndefined)
|
||||
|
||||
@@ -3,7 +3,7 @@ import re
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERMultiFeedback,
|
||||
CoSTEERSingleFeedback,
|
||||
CoSTEERSingleFeedbackDeprecated,
|
||||
)
|
||||
from rdagent.components.coder.factor_coder.eva_utils import (
|
||||
FactorCodeEvaluator,
|
||||
@@ -14,7 +14,7 @@ from rdagent.components.coder.factor_coder.factor import FactorTask
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import Workspace
|
||||
|
||||
FactorSingleFeedback = CoSTEERSingleFeedback
|
||||
FactorSingleFeedback = CoSTEERSingleFeedbackDeprecated
|
||||
FactorMultiFeedback = CoSTEERMultiFeedback
|
||||
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ from rdagent.components.coder.CoSTEER.knowledge_management import (
|
||||
)
|
||||
from rdagent.components.coder.factor_coder.config import FACTOR_COSTEER_SETTINGS
|
||||
from rdagent.components.coder.factor_coder.factor import FactorFBWorkspace, FactorTask
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.prompts import Prompts
|
||||
from rdagent.oai.llm_conf import LLM_SETTINGS
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
@@ -72,6 +73,7 @@ class FactorMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
self,
|
||||
target_task: FactorTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> str:
|
||||
target_factor_task_information = target_task.get_task_information()
|
||||
|
||||
@@ -179,5 +181,5 @@ class FactorMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
evo.sub_workspace_list[index] = FactorFBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index].inject_code(**{"factor.py": code_list[index]})
|
||||
evo.sub_workspace_list[index].inject_files(**{"factor.py": code_list[index]})
|
||||
return evo
|
||||
|
||||
@@ -34,12 +34,16 @@ class FactorTask(CoSTEERTask):
|
||||
self.factor_name = (
|
||||
factor_name # TODO: remove it in the later version. Keep it only for pickle version compatibility
|
||||
)
|
||||
self.factor_description = factor_description
|
||||
self.factor_formulation = factor_formulation
|
||||
self.variables = variables
|
||||
self.factor_resources = resource
|
||||
self.factor_implementation = factor_implementation
|
||||
super().__init__(name=factor_name, *args, **kwargs)
|
||||
super().__init__(name=factor_name, description=factor_description, *args, **kwargs)
|
||||
|
||||
@property
|
||||
def factor_description(self):
|
||||
"""for compatibility"""
|
||||
return self.description
|
||||
|
||||
def get_task_information(self):
|
||||
return f"""factor_name: {self.factor_name}
|
||||
@@ -88,8 +92,8 @@ class FactorFBWorkspace(FBWorkspace):
|
||||
|
||||
def hash_func(self, data_type: str = "Debug") -> str:
|
||||
return (
|
||||
md5_hash(data_type + self.code_dict["factor.py"])
|
||||
if ("factor.py" in self.code_dict and not self.raise_exception)
|
||||
md5_hash(data_type + self.file_dict["factor.py"])
|
||||
if ("factor.py" in self.file_dict and not self.raise_exception)
|
||||
else None
|
||||
)
|
||||
|
||||
@@ -114,7 +118,7 @@ class FactorFBWorkspace(FBWorkspace):
|
||||
|
||||
"""
|
||||
super().execute()
|
||||
if self.code_dict is None or "factor.py" not in self.code_dict:
|
||||
if self.file_dict is None or "factor.py" not in self.file_dict:
|
||||
if self.raise_exception:
|
||||
raise CodeFormatError(self.FB_CODE_NOT_SET)
|
||||
else:
|
||||
|
||||
@@ -52,7 +52,7 @@ evolving_strategy_factor_implementation_v1_system: |-
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Your former latest attempt:---------------
|
||||
=====Code to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].implementation.code }}
|
||||
{{ queried_former_failed_knowledge[-1].implementation.all_codes }}
|
||||
=====Feedback to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].feedback }}
|
||||
{% endif %}
|
||||
@@ -74,9 +74,9 @@ evolving_strategy_factor_implementation_v2_user: |-
|
||||
--------------Factor information to similar error ({{error_content}}):---------------
|
||||
{{ similar_error_knowledge[0].target_task.get_task_information() }}
|
||||
=====Code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[0].implementation.code }}
|
||||
{{ similar_error_knowledge[0].implementation.all_codes }}
|
||||
=====Success code to former code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[1].implementation.code }}
|
||||
{{ similar_error_knowledge[1].implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
Recall your last failure, your implementation met some errors.
|
||||
@@ -91,13 +91,13 @@ evolving_strategy_factor_implementation_v2_user: |-
|
||||
=====Factor {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.code }}
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if latest_attempt_to_latest_successful_execution is not none %}
|
||||
You have tried to correct your former failed code but still met some errors. Here is the latest attempt to the latest successful execution, try not to get the same error to your new code:
|
||||
=====Your latest attempt=====
|
||||
{{ latest_attempt_to_latest_successful_execution.implementation.code }}
|
||||
{{ latest_attempt_to_latest_successful_execution.implementation.all_codes }}
|
||||
=====Feedback to your latest attempt=====
|
||||
{{ latest_attempt_to_latest_successful_execution.feedback }}
|
||||
{% endif %}
|
||||
@@ -126,9 +126,9 @@ evolving_strategy_error_summary_v2_user: |-
|
||||
--------------Factor information to similar error ({{error_content}}):---------------
|
||||
{{ similar_error_knowledge[0].target_task.get_task_information() }}
|
||||
=====Code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[0].implementation.code }}
|
||||
{{ similar_error_knowledge[0].implementation.all_codes }}
|
||||
=====Success code to former code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[1].implementation.code }}
|
||||
{{ similar_error_knowledge[1].implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
@@ -158,7 +158,7 @@ select_implementable_factor_user: |-
|
||||
--------------Your former attempt:---------------
|
||||
{% for former_attempt in factor_info[2] %}
|
||||
=====Code to attempt {{ loop.index }}=====
|
||||
{{ former_attempt.implementation.code }}
|
||||
{{ former_attempt.implementation.all_codes }}
|
||||
=====Feedback to attempt {{ loop.index }}=====
|
||||
{{ former_attempt.feedback }}
|
||||
{% endfor %}
|
||||
|
||||
@@ -15,6 +15,7 @@ from rdagent.oai.llm_utils import APIBackend
|
||||
evaluate_prompts = Prompts(file_path=Path(__file__).parent / "prompts.yaml")
|
||||
|
||||
|
||||
# This shape evaluator is also used in data_science
|
||||
def shape_evaluator(prediction: np.ndarray, target_shape: Tuple = None) -> Tuple[str, bool]:
|
||||
if target_shape is None or prediction is None:
|
||||
return (
|
||||
@@ -67,7 +68,7 @@ class ModelCodeEvaluator(Evaluator):
|
||||
assert isinstance(gt_implementation, ModelFBWorkspace)
|
||||
|
||||
model_task_information = target_task.get_task_information()
|
||||
code = implementation.code
|
||||
code = implementation.all_codes
|
||||
|
||||
system_prompt = (
|
||||
Environment(undefined=StrictUndefined)
|
||||
@@ -93,7 +94,7 @@ class ModelCodeEvaluator(Evaluator):
|
||||
code=code,
|
||||
model_execution_feedback=execution_feedback_to_render,
|
||||
model_value_feedback=model_value_feedback,
|
||||
gt_code=gt_implementation.code if gt_implementation else None,
|
||||
gt_code=gt_implementation.all_codes if gt_implementation else None,
|
||||
)
|
||||
)
|
||||
if (
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from rdagent.components.coder.CoSTEER.evaluators import (
|
||||
CoSTEEREvaluator,
|
||||
CoSTEERMultiFeedback,
|
||||
CoSTEERSingleFeedback,
|
||||
CoSTEERSingleFeedbackDeprecated,
|
||||
)
|
||||
from rdagent.components.coder.model_coder.eva_utils import (
|
||||
ModelCodeEvaluator,
|
||||
@@ -13,7 +13,7 @@ from rdagent.components.coder.model_coder.model import ModelFBWorkspace, ModelTa
|
||||
from rdagent.core.evolving_framework import QueriedKnowledge
|
||||
from rdagent.core.experiment import Task, Workspace
|
||||
|
||||
ModelSingleFeedback = CoSTEERSingleFeedback
|
||||
ModelSingleFeedback = CoSTEERSingleFeedbackDeprecated
|
||||
ModelMultiFeedback = CoSTEERMultiFeedback
|
||||
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ from rdagent.components.coder.model_coder.model import (
|
||||
ModelFBWorkspace,
|
||||
ModelTask,
|
||||
)
|
||||
from rdagent.core.experiment import FBWorkspace
|
||||
from rdagent.core.prompts import Prompts
|
||||
from rdagent.oai.llm_conf import LLM_SETTINGS
|
||||
from rdagent.oai.llm_utils import APIBackend
|
||||
@@ -28,6 +29,7 @@ class ModelMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
self,
|
||||
target_task: ModelTask,
|
||||
queried_knowledge: CoSTEERQueriedKnowledge = None,
|
||||
workspace: FBWorkspace | None = None,
|
||||
) -> str:
|
||||
model_information_str = target_task.get_task_information()
|
||||
|
||||
@@ -102,5 +104,5 @@ class ModelMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy):
|
||||
continue
|
||||
if evo.sub_workspace_list[index] is None:
|
||||
evo.sub_workspace_list[index] = ModelFBWorkspace(target_task=evo.sub_tasks[index])
|
||||
evo.sub_workspace_list[index].inject_code(**{"model.py": code_list[index]})
|
||||
evo.sub_workspace_list[index].inject_files(**{"model.py": code_list[index]})
|
||||
return evo
|
||||
|
||||
@@ -24,7 +24,6 @@ class ModelTask(CoSTEERTask):
|
||||
model_type: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
self.description: str = description
|
||||
self.formulation: str = formulation
|
||||
self.architecture: str = architecture
|
||||
self.variables: str = variables
|
||||
@@ -32,7 +31,7 @@ class ModelTask(CoSTEERTask):
|
||||
self.model_type: str = (
|
||||
model_type # Tabular for tabular model, TimesSeries for time series model, Graph for graph model, XGBoost for XGBoost model
|
||||
)
|
||||
super().__init__(name=name, *args, **kwargs)
|
||||
super().__init__(name=name, description=description, *args, **kwargs)
|
||||
|
||||
def get_task_information(self):
|
||||
task_desc = f"""name: {self.name}
|
||||
@@ -84,8 +83,8 @@ class ModelFBWorkspace(FBWorkspace):
|
||||
param_init_value: float = 1.0,
|
||||
) -> str:
|
||||
target_file_name = f"{batch_size}_{num_features}_{num_timesteps}_{input_value}_{param_init_value}"
|
||||
for code_file_name in sorted(list(self.code_dict.keys())):
|
||||
target_file_name = f"{target_file_name}_{self.code_dict[code_file_name]}"
|
||||
for code_file_name in sorted(list(self.file_dict.keys())):
|
||||
target_file_name = f"{target_file_name}_{self.file_dict[code_file_name]}"
|
||||
return md5_hash(target_file_name)
|
||||
|
||||
@cache_with_pickle(hash_func)
|
||||
@@ -124,7 +123,7 @@ PARAM_INIT_VALUE = {param_init_value}
|
||||
env={},
|
||||
code_dump_file_py_name="model_test",
|
||||
)
|
||||
if results is None:
|
||||
if len(results) == 0:
|
||||
raise RuntimeError(f"Error in running the model code: {log}")
|
||||
[execution_feedback_str, execution_model_output] = results
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ class ModelCodeWriter(Developer[ModelExperiment]):
|
||||
# Extract the code part from the response
|
||||
match = re.search(r".*```[Pp]ython\n(.*)\n```.*", resp, re.DOTALL)
|
||||
code = match.group(1)
|
||||
mti.inject_code(**{"model.py": code})
|
||||
mti.inject_files(**{"model.py": code})
|
||||
mti_l.append(mti)
|
||||
exp.sub_workspace_list = mti_l
|
||||
return exp
|
||||
|
||||
@@ -65,7 +65,7 @@ evolving_strategy_model_coder:
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Your former latest attempt:---------------
|
||||
=====Code to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].implementation.code }}
|
||||
{{ queried_former_failed_knowledge[-1].implementation.all_codes }}
|
||||
=====Feedback to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].feedback }}
|
||||
{% endif %}
|
||||
@@ -85,7 +85,7 @@ evolving_strategy_model_coder:
|
||||
=====Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.code }}
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
@@ -93,7 +93,7 @@ evolving_strategy_model_coder:
|
||||
--------------Former failed code:---------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %}
|
||||
=====Code to implementation {{ loop.index }}=====
|
||||
{{ former_failed_knowledge.implementation.code }}
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback to implementation {{ loop.index }}=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
|
||||
Reference in New Issue
Block a user