diff --git a/rdagent/components/coder/data_science/ensemble/prompts.yaml b/rdagent/components/coder/data_science/ensemble/prompts.yaml index d78d10cd..c7258696 100644 --- a/rdagent/components/coder/data_science/ensemble/prompts.yaml +++ b/rdagent/components/coder/data_science/ensemble/prompts.yaml @@ -1,26 +1,24 @@ ensemble_coder: system: |- - You are a Python data scientist working on model ensemble implementation. Your task is to write a Python function that combines multiple model predictions and makes final decisions. + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. + + ## Task Description + Currently, you are working on model ensemble implementation. Your task is to write a Python function that combines multiple model predictions and makes final decisions. Your specific task as follows: - {{task_desc}} + {{ task_desc }} - You should follow the provided specifications to complete this task. - - -----------Competition Information----------- + Here is the competition information for this task: {{ competition_info }} - Please respond with the code in the following json format: - { - "code": "The Python code as a string." - } - {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} - -----------Here is the relevant information for this task----------- + ## Relevant Information for This Task {% endif %} + {% if queried_similar_successful_knowledge|length != 0 %} - --------------Successful Implementations for Similar Models:-------------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:===== + --------- Successful Implementations for Similar Models --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== {{ similar_successful_knowledge.target_task.get_task_information() }} =====Code:===== {{ similar_successful_knowledge.implementation.file_dict["ensemble.py"] }} @@ -28,7 +26,7 @@ ensemble_coder: {% endif %} {% if queried_former_failed_knowledge|length != 0 %} - --------------Previous Failed Attempts:-------------- + --------- Previous Failed Attempts --------- {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: =====Code:===== {{ former_failed_knowledge.implementation.file_dict["ensemble.py"] }} @@ -37,55 +35,74 @@ ensemble_coder: {% endfor %} {% endif %} + ## Output Format + Please response the code in the following json format. Here is an example structure for the JSON output: + { + "code": "The Python code as a string." + } + user: |- - Please implement an ensemble function with the following specification: - -----------Ensemble Specification----------- + --------- Ensemble Specification --------- {{ ensemble_spec }} {% if latest_code %} - ---------Former code--------- - {{ latest_code }} + --------- Former code --------- + {{ latest_code }} {% if latest_code_feedback is not none %} - ---------Feedback to former code--------- - {{ latest_code_feedback }} + --------- Feedback to former code --------- + {{ latest_code_feedback }} {% endif %} - The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code. + The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes. {% endif %} + + ensemble_eval: system: |- - You are a data scientist evaluating an ensemble implementation. - The main code generation task is as follows: - {{task_desc}} - - The ensemble code is: + You are a data scientist responsible for evaluating ensemble implementation code generation. + + ## Task Description + {{ task_desc }} + + ## Ensemble Code ```python - {{code}} + {{ code }} ``` - You are testing the ensemble with the following code: + ## Testing Process + The ensemble code is tested using the following script: ```python - {{test_code}} + {{ test_code }} ``` + You will analyze the execution results based on the test output provided. {% if workflow_stdout is not none %} - Your ensemble code is also part of the whole workflow, the user also tested the whole workflow and provided you the stdout. - The whole workflow code is: - {{workflow_code}} - Please consider both stdout and approve the code when both the ensemble test and the whole workflow test pass. + ### Whole Workflow Consideration + The ensemble code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout. + + **Workflow Code:** + ```python + {{ workflow_code }} + ``` + + You should evaluate both the ensemble test results and the overall workflow results. **Approve the code only if both tests pass.** {% endif %} + + ## Evaluation Criteria + You will be given the standard output (`stdout`) from the ensemble test and, if applicable, the workflow test. - You'll be given the stdout of your testing scripts. - Please respond with your feedback in the following JSON format: + Please respond with your feedback in the following JSON format and order + ```json { - "execution": "Describe how well the ensemble executed, including any errors or issues encountered. Please keep the error message and tracking information", + "execution": "Describe how well the ensemble executed, including any errors or issues encountered. Retain all error messages and traceback details.", "return_checking": "Detail the checks performed on the ensemble results, including shape and value validation.", - "code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.", + "code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.", "final_decision": - } + } + ``` user: |- - Ensemble test stdout: - {{stdout}} + --------- Ensemble test stdout --------- + {{ stdout }} {% if workflow_stdout is not none %} - Whole workflow test stdout: - {{workflow_stdout}} + --------- Whole workflow test stdout --------- + {{ workflow_stdout }} {% endif %} diff --git a/rdagent/components/coder/data_science/feature/__init__.py b/rdagent/components/coder/data_science/feature/__init__.py index 6020acf7..a06115ed 100644 --- a/rdagent/components/coder/data_science/feature/__init__.py +++ b/rdagent/components/coder/data_science/feature/__init__.py @@ -57,13 +57,13 @@ class FeatureMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy): ) # 2. code - system_prompt = T(".prompts:feature.system").r( + system_prompt = T(".prompts:feature_coder.system").r( task_desc=feature_information_str, data_loader_code=workspace.file_dict.get("load_data.py"), queried_similar_successful_knowledge=queried_similar_successful_knowledge, queried_former_failed_knowledge=queried_former_failed_knowledge[0], ) - user_prompt = T(".prompts:feature.user").r( + user_prompt = T(".prompts:feature_coder.user").r( feature_spec=workspace.file_dict["spec/feature.md"], latest_code=workspace.file_dict.get("feature.py"), latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None, diff --git a/rdagent/components/coder/data_science/feature/prompts.yaml b/rdagent/components/coder/data_science/feature/prompts.yaml index 6a01f21f..5198d23d 100644 --- a/rdagent/components/coder/data_science/feature/prompts.yaml +++ b/rdagent/components/coder/data_science/feature/prompts.yaml @@ -1,33 +1,18 @@ -feature: +feature_coder: system: |- - You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. - Your task is as follows: - {{task_desc}} - - This project involves implementing feature engineering techniques to prepare data for machine learning models, and this project code will be written by GPT. - Your task is to write a Python function that performs feature engineering on a given data. - If you think that feature engineering is not necessary for this competition/scenario, or it should be implemented together with the model, you can ignore this task. - You should follow the provided specifications to complete this task. - - Your function input is the output of a data loading function, the data loader function code is as follows: - ```python - {{data_loader_code}} - ``` - Please understand the code and try to implement the feature engineering function based on the data loader output. - - Please response the code in the following json format. Here is an example structure for the JSON output: - { - "code": "The Python code as a string." - } + ## Task Description + {{ task_desc }} {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} - -----------Here is the relevant information for this task----------- + ## Relevant Information for This Task {% endif %} + {% if queried_similar_successful_knowledge|length != 0 %} - --------------Successful Implementations for Similar Models:-------------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:===== + --------- Successful Implementations for Similar Models --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== {{ similar_successful_knowledge.target_task.get_task_information() }} =====Code:===== {{ similar_successful_knowledge.implementation.file_dict["feature.py"] }} @@ -35,7 +20,7 @@ feature: {% endif %} {% if queried_former_failed_knowledge|length != 0 %} - --------------Previous Failed Attempts:-------------- + --------- Previous Failed Attempts --------- {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: =====Code:===== {{ former_failed_knowledge.implementation.file_dict["feature.py"] }} @@ -44,58 +29,85 @@ feature: {% endfor %} {% endif %} + ## Guidelines + 1. If feature engineering is unnecessary or should be combined with model training, you may skip this step. + 2. The function input is the output of the following data loader: + ```python + {{ data_loader_code }} + ``` + 3. **Additional Guidance:** + - If a previous attempt exists, improve upon it without repeating mistakes. + - If errors indicate a missing file, find a way to download it or implement an alternative solution. + + ## Output Format + Please response the code in the following json format. Here is an example structure for the JSON output: + { + "code": "The Python code as a string." + } + user: |- - ---------Feature Processing Specification--------- + --------- Feature Processing Specification --------- {{ feature_spec }} {% if latest_code %} - ---------Former code--------- - {{ latest_code }} + --------- Former code --------- + {{ latest_code }} {% if latest_code_feedback is not none %} - ---------Feedback to former code--------- - {{ latest_code_feedback }} + --------- Feedback to former code --------- + {{ latest_code_feedback }} + {% endif %} + The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes. {% endif %} - The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code. - {% endif %} feature_eval: system: |- - You are data scientist whose job is to evaluate the feature processing code generation. - The main code generation task is as follows: - {{task_desc}} + You are a data scientist responsible for evaluating feature engineering code generation. - The feature code is: + ## Task Description + {{ task_desc }} + + ## Feature Engineering Code ```python - {{code}} + {{ code }} ``` - You are testing the feature processing code with the following code + ## Testing Process + The feature engineering code is tested using the following script: ```python - {{test_code}} + {{ test_code }} ``` - + You will analyze the execution results based on the test output provided. + {% if workflow_stdout is not none %} - Your feature engineering code is also part of the whole workflow, the user also tested the whole workflow and provided you the stdout. - The whole workflow code is: - {{workflow_code}} - Please consider both stdout and approve the code when both the feature engineering test and the whole workflow test pass. + ### Whole Workflow Consideration + The feature engineering code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout. + + **Workflow Code:** + ```python + {{ workflow_code }} + ``` + + You should evaluate both the feature engineering test results and the overall workflow results. **Approve the code only if both tests pass.** {% endif %} - You'll be given the stdout of your testing scripts. + ## Evaluation Criteria + You will be given the standard output (`stdout`) from the feature engineering test and, if applicable, the workflow test. + Please respond with your feedback in the following JSON format and order ```json { - "execution": "Describe how well the feature processing executed, including any errors or issues encountered. Please keep the error message and tracking information", - "return_checking": "Detail the checks performed on the data after feature processing, including data integrity and correctness.", - "code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.", + "execution": "Describe how well the feature engineering executed, including any errors or issues encountered. Retain all error messages and traceback details.", + "return_checking": "Evaluate the correctness and integrity of processed data, checking for missing values, incorrect transformations, and data consistency.", + "code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.", "final_decision": } ``` + user: |- - Feature engineering test stdout: - {{stdout}} + --------- Feature engineering test stdout --------- + {{ stdout }} {% if workflow_stdout is not none %} - Whole workflow test stdout: - {{workflow_stdout}} + --------- Whole workflow test stdout --------- + {{ workflow_stdout }} {% endif %} diff --git a/rdagent/components/coder/data_science/model/__init__.py b/rdagent/components/coder/data_science/model/__init__.py index 537d3fab..812a71c6 100644 --- a/rdagent/components/coder/data_science/model/__init__.py +++ b/rdagent/components/coder/data_science/model/__init__.py @@ -79,7 +79,7 @@ class ModelMultiProcessEvolvingStrategy(MultiProcessEvolvingStrategy): # We want to use a simpler way to user_prompt = T(".prompts:model_coder.user_general").r( model_spec=workspace.file_dict["spec/model.md"], - workspace_code=workspace.get_codes( + latest_model_code=workspace.get_codes( r"^model_(?!test)\w+\.py$" ), # TODO: If we have high failure rate here, we should clean this step with less information. latest_code_feedback=latest_code_feedback[0] if len(latest_code_feedback) > 0 else None, diff --git a/rdagent/components/coder/data_science/model/eval.py b/rdagent/components/coder/data_science/model/eval.py index 6acd7fa3..ce353cd9 100644 --- a/rdagent/components/coder/data_science/model/eval.py +++ b/rdagent/components/coder/data_science/model/eval.py @@ -82,6 +82,7 @@ class ModelGeneralCaseSpecEvaluator(CoSTEEREvaluator): system_prompt = T(".prompts:model_eval.system").r( task_desc=target_task.get_task_information(), test_code=test_code, + code=implementation.file_dict[f"{target_task.name}.py"], scenario=self.scen.get_scenario_all_desc(), spec=implementation.file_dict["spec/model.md"], workflow_stdout=workflow_stdout, @@ -89,7 +90,6 @@ class ModelGeneralCaseSpecEvaluator(CoSTEEREvaluator): ) user_prompt = T(".prompts:model_eval.user").r( stdout=stdout, - code=implementation.file_dict[f"{target_task.name}.py"], workflow_stdout=workflow_stdout, ) return build_cls_from_json_with_retry(ModelSingleFeedback, system_prompt=system_prompt, user_prompt=user_prompt) diff --git a/rdagent/components/coder/data_science/model/prompts.yaml b/rdagent/components/coder/data_science/model/prompts.yaml index 1cc2e11a..47b5306c 100644 --- a/rdagent/components/coder/data_science/model/prompts.yaml +++ b/rdagent/components/coder/data_science/model/prompts.yaml @@ -1,145 +1,136 @@ model_coder: - system: |- - You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. - Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. - - Your task is as follows: - {{task_desc}} + system: |- + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. - The user's ultimate goal is to obtain accurate predictions from the model on input data. Follow the instructions below to ensure your response is correct and aligned with the user's expectations. + ## Task Description + {{ task_desc }} - Your function's input is from the output of a feature engineering function whose input is the output of a data loading function. The raw data loader function and feature engineer function code is as follows: - --------- Raw Data Loader Code: --------- - {{data_loader_code}} - --------- Feature Engineering Code: --------- - {{feature_code}} - - Instructions for Code Generation: - Leveraging User Inputs: - The user may provide various forms of additional information to guide you: + {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} + ## Relevant Information for This Task + {% endif %} + + {% if queried_similar_successful_knowledge|length != 0 %} + --------- Successful Implementations for Similar Models --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== + {{ similar_successful_knowledge.target_task.get_task_information() }} + =====Code:===== + {{ similar_successful_knowledge.implementation.file_dict[similar_successful_knowledge.target_task.name ~ '.py'] }} + {% endfor %} + {% endif %} - Successful Examples: Correct implementations of similar models. - Previous Attempts: Failed implementations along with execution feedback and/or error analysis. - Suggestions: Specific advice for fixing errors, including corrected versions of code for similar issues. - Use this information strategically to identify the correct patterns, debug mistakes, and ensure the final implementation works as intended. + {% if queried_former_failed_knowledge|length != 0 %} + --------- Previous Failed Attempts --------- + {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: + =====Code:===== + {{ former_failed_knowledge.implementation.file_dict[former_failed_knowledge.target_task.name ~ '.py'] }} + =====Feedback:===== + {{ former_failed_knowledge.feedback }} + {% endfor %} + {% endif %} - Preserving Correct Code: - If the user has shared their latest code, carefully analyze it and only modify parts that require changes. Do not alter correct sections of the code. + ## Guidelines + 1. The function's input is from the output of a feature engineering function whose input is the output of a data loading function. The data loader function and feature engineering function code is as follows: + --------- Data Loader Code --------- + {{ data_loader_code }} + --------- Feature Engineering Code --------- + {{ feature_code }} - Error Learning: - If previous failed attempts and their feedback are available, learn from them. Understand what went wrong and avoid repeating similar mistakes in your new implementation. - The failure knowledge may include the code unrelated to the model, such as data loading, preprocessing, or feature engineering. Focus only on the model implementation part. + ## Output Format + {% if out_spec %} + {{ out_spec }} + The file name should be the model name described in the model task in the format "{task_name}.py". You should always follow this name format. + {% else %} + Please response the code in the following json format. Here is an example structure for the JSON output: + { + "code": "The Python code as a string." + } + {% endif %} - {% if out_spec %} - {{out_spec}} - The file name should be the model name described in the model task in the format "{task_name}.py". You should always follow this name format. - {% else %} - Formatting Your Response: - Return only the code in a JSON format as shown below. Do not include any explanations or extra text. Example: - { - "code": "Your corrected or newly implemented Python code as a single string" - } - {% endif %} - - {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} - -----------Here is the relevant information for this task----------- - {% endif %} - {% if queried_similar_successful_knowledge|length != 0 %} - --------------Successful Implementations for Similar Models:-------------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:===== - {{ similar_successful_knowledge.target_task.get_task_information() }} - =====Code:===== - {{ similar_successful_knowledge.implementation.file_dict[similar_successful_knowledge.target_task.name ~ '.py'] }} - {% endfor %} - {% endif %} - - {% if queried_former_failed_knowledge|length != 0 %} - --------------Previous Failed Attempts:-------------- - {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: - =====Code:===== - {{ former_failed_knowledge.implementation.file_dict[former_failed_knowledge.target_task.name ~ '.py'] }} - =====Feedback:===== - {{ former_failed_knowledge.feedback }} - {% endfor %} - {% endif %} - - user: |- - ---------Model Specification--------- - {{ model_spec }} - - {% if latest_code %} - ---------Former Code--------- - Former Code: {{ latest_code }} - The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code. - {% endif %} - - user_general: |- - --------- Workspace code--------- - {% if workspace_code|length == 0 %} - So far the workspace is empty. No model code has been implemented yet. - {% else %} - {{ workspace_code }} - {% if latest_code_feedback is not none %} - ---------Feedback to former code--------- - {{ latest_code_feedback }} - {% endif %} - {% endif %} - ---------Model Specification--------- - When you are implementing the code, you should follow the spec - {{ model_spec }} + user_general: |- + --------- Model Building Specification --------- + {{ model_spec }} + --------- Former model code --------- + {% if latest_model_code|length == 0 %} + So far the workspace is empty. No model code has been implemented yet. + {% else %} + {{ latest_model_code }} + {% if latest_code_feedback is not none %} + --------- Feedback to former code --------- + {{ latest_code_feedback }} + {% endif %} + The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes. + {% endif %} model_eval: - system: |- - You are a data scientist. - The user is trying to implement some models in the following scenario: - {{ scenario }} - The main code generation task is as follows: - {{task_desc}} - The user will provide you with the information of the model. - The information about how to implement the model is given in spec.md as below: - {{ spec }} - You are testing the model with the following code: - ```python - {{test_code}} - ``` - The first time you execute it, you will not provide test inputs, only train, valid inputs, and empty hyperparameters. You need to check if it can correctly train the model, and there must be valid outputs and hyperparameter outputs. - The second time you execute it, you will provide train and test inputs without valid inputs. You will also input the hyperparameters output from the previous run for retraining. - Therefore, when the hyperparameters returned are not none, during the evaluation you must check: - - It should have parameters that will be useful for retraining later. It must include the early stop round. - - You need to check if these hyperparameters are really used in the model code below. The early stop round must be used if given. - If the requirements regarding test, valid, or parameters are not met, then the final decision cannot be approved. - - You should evaluate the code given by the user. You should be concerned about whether the user implemented it correctly, including whether the shape of the model's output is aligned with the request, the quality of the code, and any other thing you think necessary. - You will be given the code generated by the user and the stdout of the testing process. - When conducting evaluation, please refer to the requirements provided in spec.md, as different requirements will lead to different criteria for evaluation. + system: |- + You are a data scientist responsible for evaluating model building code generation. - Only if there is "Model code test passed successfully." in the stdout, then the model is considered successful, or else there must be some issues with the model. - If no stdout is provided, the model is considered to have failed due to a timeout. Please check if there are any ways to improve the model's execution speed. - - {% if workflow_stdout is not none %} - Your model code is also part of the whole workflow, the user also tested the whole workflow and provided you the stdout. - The whole workflow code is: - {{workflow_code}} - Please consider both stdout and approve the code when both the model test and the whole workflow test pass. - {% endif %} + ## Task Description + {{ task_desc }} + + ## Model Building Code + ```python + {{ code }} + ``` + + ## Testing Process + The model building code is tested using the following script: + ```python + {{ test_code }} + ``` + + ### Execution Phases + The model is tested in two phases: + + 1. Initial Training Phase: + - The model receives **train and valid inputs** with **empty hyperparameters**. + - The focus is on verifying whether the model successfully trains and produces **valid outputs and hyperparameter outputs**. + + 2. Retraining Phase: + - The model receives **train and test inputs** (without valid inputs). + - The hyperparameters generated from the first phase are passed back for **retraining**. + + + ### Key Requirements for Approval + A model can only be approved if it meets all of the following conditions: + 1. Hyperparameter Handling + - If hyperparameters are returned, they must include an early stop round. + - The hyperparameters must be correctly utilized in the model for retraining. + - If the early stop round is provided, it must be used in the model implementation. + 2. The model output shape must strictly match the specifications in `spec.md`. + + {% if workflow_stdout is not none %} + ### Whole Workflow Consideration + The model building code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout. + + **Workflow Code:** + ```python + {{ workflow_code }} + ``` + + You should evaluate both the model building test results and the overall workflow results. **Approve the code only if both tests pass.** + {% endif %} - Please respond with your feedback in the following JSON format and order: - ```json - { - "execution": "Describe whether the model executed successfully, including any errors or issues encountered. Please keep the error message and tracking information", - "return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.", - "code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process. Check whether the hyperparameters from the previous run are used in the model code, compare the parameter names in stdout and if they are used in the retraining part of the code. It is acceptable when hyperparameters is None.", - "final_decision": - } - ``` + ## Evaluation Criteria + You will be given the standard output (`stdout`) from the model building test and, if applicable, the workflow test. + [Note] If no stdout for model buidling test is provided, the model failed due to a timeout or out-of-memory error. You should analyze potential optimizations. - user: |- - ---------------Code generated by user:--------------- - {{ code }} - ---------------Model test stdout:--------------- - {{stdout}} - {% if workflow_stdout is not none %} - ---------------Whole workflow test stdout:--------------- - {{workflow_stdout}} - {% endif %} + Please respond with your feedback in the following JSON format and order + ```json + { + "execution": "Describe how well the model building executed, including any errors or issues encountered. Retain all error messages and traceback details.", + "return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.", + "code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.", + "final_decision": + } + ``` + + user: |- + --------- Model building test stdout --------- + {{ stdout }} + {% if workflow_stdout is not none %} + --------- Whole workflow test stdout --------- + {{ workflow_stdout }} + {% endif %} diff --git a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml index cd74e447..e2fc0ed9 100644 --- a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml +++ b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml @@ -1,6 +1,6 @@ spec: system: |- - You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. Currently, you are working on a Kaggle competition project. @@ -76,7 +76,7 @@ spec: You should follow the provided specifications to improve this task. {% endif %} - + Please respond with a JSON structure as follows: { @@ -305,25 +305,19 @@ spec: data_loader_coder: system: |- - You are a Python data scientist working on a new project. This project will be used to analyze data and build models to predict future outcomes, and this project codes will be written by GPT. - Your task is described below: + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. + + ## Task Description {{ task_desc }} - You should follow the provided specifications to complete this task. - You need to write the corresponding data loading code based on the information provided in the user's Data Folder Description, rather than relying on any suggestions that might exist in the spec. - - Notice, the data files are stored in the data folder located at `/kaggle/input/`, and the data folder is structured as described in the Data Folder Description. Please don't load the data from the current directory. - - Please response the code in the following json format. Here is an example structure for the JSON output: - { - "code": "The Python code as a string." - } {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} - -----------Here is the relevant information for this task----------- + ## Relevant Information for This Task {% endif %} + {% if queried_similar_successful_knowledge|length != 0 %} - --------------Successful Implementations for Similar Models:-------------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:===== + --------- Successful Implementations for Similar Models --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== {{ similar_successful_knowledge.target_task.get_task_information() }} =====Code:===== {{ similar_successful_knowledge.implementation.all_codes }} @@ -331,7 +325,7 @@ data_loader_coder: {% endif %} {% if queried_former_failed_knowledge|length != 0 %} - --------------Previous Failed Attempts:-------------- + --------- Previous Failed Attempts --------- {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: =====Code:===== {{ former_failed_knowledge.implementation.all_codes }} @@ -340,66 +334,86 @@ data_loader_coder: {% endfor %} {% endif %} + ## Guidelines + 1. Ensure that the dataset is loaded strictly from `/kaggle/input/`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`). + + ## Output Format + Please response the code in the following json format. Here is an example structure for the JSON output: + { + "code": "The Python code as a string." + } + user: |- - ---------Competition Information--------- + --------- Competition Information --------- {{ competition_info }} - ---------Data Loader Specification--------- + --------- Data Loader Specification --------- {{ data_loader_spec }} - ---------Data Folder Description---------(All path are relative to the data folder) + --------- Data Folder Description (All path are relative to the data folder) --------- {{ folder_spec }} {% if latest_code %} - ---------Former code--------- - {{ latest_code }} + --------- Former code --------- + {{ latest_code }} {% if latest_code_feedback is not none %} - ---------Feedback to former code--------- - {{ latest_code_feedback }} + --------- Feedback to former code --------- + {{ latest_code_feedback }} {% endif %} - The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code. - {% endif %} + The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes. + {% endif %} You should strictly follow the function interface specifications provided by the specification to implement the function. data_loader_eval: system: |- - You are data scientist writing some data loader code for a Kaggle-style machine learning competition project. - The main code generation task is as follows: - {{task_desc}} + You are a data scientist responsible for evaluating data loader code for a Kaggle-style machine learning competition project. + + ## Task Description + {{ task_desc }} - The data loader code is in a file named "load_data.py": + ## Data Loader Code + The data loader code is located in `load_data.py`: ```python - {{code}} + {{ code }} ``` - You are testing the data_loader with the following code + ## Testing Process + The data loader is tested using the following script: ```python - {{test_code}} + {{ test_code }} ``` {% if workflow_stdout is not none %} - Your feature engineering code is also part of the whole workflow, the user also tested the whole workflow and provided you the stdout. - The whole workflow code is: - {{workflow_code}} - Please consider both stdout and approve the code when both the feature engineering test and the whole workflow test pass. + ### Whole Workflow Consideration + The data loader is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout. + + **Workflow Code:** + ```python + {{ workflow_code }} + ``` + + You should evaluate both the data loader test results and the overall workflow execution. **Approve the code only if both tests pass.** {% endif %} - You'll be given the stdout of your testing scripts. - Please respond with your feedback in the following JSON format and order + ## Evaluation Criteria + You will be given the standard output (`stdout`) from the data loader test and, if applicable, the workflow test. + + Your response must follow this structured JSON format: ```json { - "execution": "Describe how well the data loader executed, including any errors or issues encountered. Please keep the error message and tracking information", - "return_checking": "Detail the checks performed on the data loaded, including data integrity and correctness.", - "code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.", + "execution": "Describe how well the data loader executed, including any errors or issues encountered. Retain all error messages and traceback details.", + "return_checking": "Evaluate the correctness and integrity of the loaded data. Check for issues like missing values, incorrect data types, outliers, or formatting inconsistencies.", + "code": "Assess code quality, readability, and adherence to best practices. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for faster data loading.", "final_decision": } ``` + user: |- - Data loader test stdout: - {{stdout}} + --------- Data loader test stdout --------- + {{ stdout }} {% if workflow_stdout is not none %} - Whole workflow test stdout: - {{workflow_stdout}} + --------- Whole workflow test stdout --------- + {{ workflow_stdout }} {% endif %} diff --git a/rdagent/components/coder/data_science/workflow/prompts.yaml b/rdagent/components/coder/data_science/workflow/prompts.yaml index 4110510f..a292ed63 100644 --- a/rdagent/components/coder/data_science/workflow/prompts.yaml +++ b/rdagent/components/coder/data_science/workflow/prompts.yaml @@ -1,36 +1,21 @@ workflow_coder: system: |- - You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. + You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science. Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems. - Your specific task is as follows: - {{task_desc}} + ## Task Description + {{ task_desc }} - Your current competition information is as follows: + Here is the competition information for this task: {{ competition_info }} - - The user has written different Python functions that can load and preprocess data, execute feature engineering, train models, and ensemble them. - - These Python codes with different functionalities are written separately in different Python files. - You don't need to edit the existing code. Your task is to integrate the existing processes of load_data, feature, model, and ensemble into a complete workflow. - This workflow code is also a Python file, and it functions similarly to a main process that calls the sub-files for each step and ultimately outputs a prediction file. - - The user will also provide specifications on how to organize the code and give instructions. - - The code you implement should align with the framework given in the specifications. - After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code. - - Please respond with the code in the following JSON format. Here is an example structure for the JSON output: - { - "code": "The Python code as a string." - } {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %} - -----------Here is the relevant information for this task----------- + ## Relevant Information for This Task {% endif %} + {% if queried_similar_successful_knowledge|length != 0 %} - --------------Successful Implementations for Similar Models:-------------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:===== + --------- Successful Implementations for Similar Models --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== {{ similar_successful_knowledge.target_task.get_task_information() }} =====Code:===== {{ similar_successful_knowledge.implementation.file_dict["main.py"] }} @@ -38,7 +23,7 @@ workflow_coder: {% endif %} {% if queried_former_failed_knowledge|length != 0 %} - --------------Previous Failed Attempts:-------------- + --------- Previous Failed Attempts --------- {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}: =====Code:===== {{ former_failed_knowledge.implementation.file_dict["main.py"] }} @@ -47,69 +32,98 @@ workflow_coder: {% endfor %} {% endif %} + ## Guidelines + 1. Understand the User's Code Structure + - The user has written different Python functions that can load and preprocess data, execute feature engineering, train models, and ensemble them. + - Each functionality is in a separate Python file. + 2. Your task is only to integrate the existing processes of load_data, feature, model, and ensemble into a complete workflow. Do not edit or modify the existing Python files. The final step should output the predictions in the required format. + 3. The user may provide specific code organization rules and instructions. Ensure that the integration follows the given framework and structure. + 4. After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code. + + ## Output Format + Please response the code in the following json format. Here is an example structure for the JSON output: + { + "code": "The Python code as a string." + } + user: |- - ---------Workflow Specification--------- + --------- Workflow Specification --------- {{ workflow_spec }} - ---------load data code--------- + --------- load data code --------- file: load_data.py {{ load_data_code }} - ---------feature engineering code--------- + --------- feature engineering code --------- file: feature.py {{ feature_code }} - ---------model training code--------- + --------- model training code --------- Attention: The input and output of the model function is flexible. Training dataset is necessary, but validation and test dateset might be optional. The hyperparameters can either be passed as arguments or be set as default values in the function. You need to use the function correctly. All model files share the same function name. Please import the model files with their name like: from {file_name} import {function_name} {{ model_codes }} - ---------ensemble code--------- + --------- ensemble code --------- Note, we will check the index of the score.csv, so please use the model name as the index to feed into ensemble function. file: ensemble.py {{ ensemble_code }} {% if latest_code %} - ---------Former code--------- - {{ latest_code }} + --------- Former code --------- + {{ latest_code }} {% if latest_code_feedback is not none %} - ---------Feedback to former code--------- - {{ latest_code_feedback }} + --------- Feedback to former code --------- + {{ latest_code_feedback }} + {% endif %} + The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes. {% endif %} - The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code. - {% endif %} workflow_eval: system: |- - You are a data scientist. + You are a data scientist responsible for evaluating workflow code generation. + + ## Task Description The user is trying to build a workflow in the following scenario: {{ scenario }} + The main code generation task is as follows: - {{task_desc}} - The user will provide you with the information of the workflow and its components. - The information about how to build the workflow is given in the specification file as below: + {{ task_desc }} + + The user provides workflow information and its components. + The details on how to structure the workflow are given in the specification file: + ```python {{ spec }} - This workflow will import all the codes including data loading, feature engineering, model tuning, and ensembling. - You are testing it by running the workflow code. The results will be collected as the stdout and it will help you evaluate the code. + ``` + This workflow integrates multiple stages, including: + - Data loading + - Feature engineering + - Model training + - Ensembling - Your job is to evaluate the workflow code given by the user. You should be concerned about whether the code executes successfully, generates predictions correctly, and satisfies other requirements in the specification. - The components have already been evaluated by the user, so you only need to evaluate and improve the workflow code unless there are very serious issues with the components. + ## Evaluation Scope + Your focus is to check whether the workflow code: + 1. Executes successfully, correctly organizing components and generating a final submission. + 2. Generates predictions in the correct format, ensuring they align with the **sample submission** structure! - Your evaluation should only consider whether the code executes successfully, generates well formatted predictions, and aligns with the target task. The performance of the model is not a concern in this task. + [Note] + 1. The individual components (data loading, feature engineering, model tuning, etc.) have already been evaluated by the user. You should only evaluate and improve the workflow code, unless there are critical issues in the components. + 2. Model performance is NOT a concern in this evaluation—only correct execution and formatting matter. - Please respond with your feedback in the following JSON format and order: + ## Evaluation Criteria + You will be given the workflow execution output (`stdout`) to determine correctness. + + Please respond with your feedback in the following JSON format and order ```json { - "execution": "Describe whether the main code executed successfully(well organizing different components and generating the final submission), including any errors or issues encountered. Please keep the error message and tracking information", - "return_checking": "Review the generated files, specifically the submission file, to ensure the format matches the sample submission. Pay attention to the index, columns, and content of the CSV files in the stdout." - "code": "Provide feedback on the code quality, readability, and adherence to specifications.", + "execution": "Describe whether the main workflow executed successfully, correctly integrating all components and generating the final submission. Include any errors or issues encountered, and retain all error messages and traceback details.", + "return_checking": "Verify the generated files, particularly the submission file. Ensure that its format matches the sample submission, checking the index, column names, and CSV content.", + "code": "Provide feedback on code quality, readability, and adherence to the given specifications.", "final_decision": } ``` + user: |- - --------------Code generated by user:--------------- - {{ code }} - --------------stdoutput:--------------- - ''' + --------- Workflow test stdout --------- {{ stdout }} - ''' + --------- Workflow code generated by user --------- + {{ code }} \ No newline at end of file