diff --git a/rdagent/components/coder/data_science/ensemble/prompts.yaml b/rdagent/components/coder/data_science/ensemble/prompts.yaml index bf874cb8..b7db201b 100644 --- a/rdagent/components/coder/data_science/ensemble/prompts.yaml +++ b/rdagent/components/coder/data_science/ensemble/prompts.yaml @@ -39,6 +39,7 @@ ensemble_coder: 1. The function's code is associated with several other functions including a data loader, feature engineering, and model training. all codes are as follows: {{ all_code }} 2. You should avoid using logging module to output information in your generated code, and instead use the print() function. + {% include "scenarios.data_science.share:guidelines.coding" %} ## Output Format {% if out_spec %} diff --git a/rdagent/components/coder/data_science/feature/prompts.yaml b/rdagent/components/coder/data_science/feature/prompts.yaml index 949c0fac..047d9c8a 100644 --- a/rdagent/components/coder/data_science/feature/prompts.yaml +++ b/rdagent/components/coder/data_science/feature/prompts.yaml @@ -48,6 +48,10 @@ feature_coder: from joblib import Memory memory = Memory(location='/tmp/cache', verbose=0) @memory.cache``` + 6. Coding tricks: + - If the input consists of a batch of file paths and you need to modify the file contents to complete your feature engineering task, you can accomplish your feature engineering task by modifying these files and creating new files in a subfolder within "/tmp/cache" (this path is persistent, otherwise you may lose your created file). Then the new file paths are returned. + + {% include "scenarios.data_science.share:guidelines.coding" %} ## Output Format {% if out_spec %} diff --git a/rdagent/components/coder/data_science/model/prompts.yaml b/rdagent/components/coder/data_science/model/prompts.yaml index 39fa2cc4..c0af430e 100644 --- a/rdagent/components/coder/data_science/model/prompts.yaml +++ b/rdagent/components/coder/data_science/model/prompts.yaml @@ -45,6 +45,7 @@ model_coder: from joblib import Memory memory = Memory(location='/tmp/cache', verbose=0) @memory.cache`` + {% include "scenarios.data_science.share:guidelines.coding" %} ## Output Format {% if out_spec %} diff --git a/rdagent/components/coder/data_science/pipeline/prompts.yaml b/rdagent/components/coder/data_science/pipeline/prompts.yaml index c785c794..df0f8a53 100644 --- a/rdagent/components/coder/data_science/pipeline/prompts.yaml +++ b/rdagent/components/coder/data_science/pipeline/prompts.yaml @@ -57,6 +57,7 @@ pipeline_coder: User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1] - An evaluation agent will help to check whether the EDA part is added correctly. - During the EDA part, you should try to avoid any irrelevant information sending to the standard output. + {% include "scenarios.data_science.share:guidelines.coding" %} {% if enable_model_dump %} ## Model Dumping diff --git a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml index a45a1e4d..ffce8f61 100644 --- a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml +++ b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml @@ -224,71 +224,7 @@ spec: You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring. workflow: |- - Your task is to implement the main workflow script (`main.py`) for a Kaggle-style machine learning competition project. - Follow the provided project structure and specifications to ensure consistency and maintainability: - 1. Workflow Integration: - - Integrate the following components into the workflow: - - Data loading (`load_data.py`). - - Feature engineering (`feature.py`). - - Model workflow for training and testing (`model_*.py`). - - Ensemble workflow that combines results from the model workflow to obtain the final prediction (`ensemble.py`). - - Treat each component as a modular and callable Python function. - - The workflow script should be flexible enough to handle either a single model or multiple models, with filenames (model_*.py) that are not determined at the outset. - For multiple model selection, utilize Python code to identify eligible models based on filenames, for example: - ```python - available_models = [f for f in os.listdir('.') if f.startswith('model_') and 'test' not in f] - ``` - 2. Feature Engineering - - The feature engineering should be called only once. For example: - `X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)` - - It should be called before dataset splitting. - - 3. Dataset Splitting - - The dataset returned by `load_data` is not pre-split. After calling `feat_eng`, split the data into training and test sets. - - [Notice] If feasible, apply cross-validation on the training set (`X_transformed`, `y_transformed`) to ensure a reliable assessment of model performance. - - Keep the test set (`X_test_transformed`) unchanged, as it is only used for generating the final predictions. - - Pseudocode logic for reference: - ``` - Set number of splits and initialize KFold cross-validator. - Create dictionaries for validation and test predictions. - For each model file: - Import the model dynamically. - Initialize arrays for out-of-fold (OOF) and test predictions. - For each fold in KFold: - Split data into training and validation sets. - Run model workflow to get validation and test predictions. - Validate shapes. - Store validation and test predictions. - Compute average test predictions across folds. - Save OOF and averaged test predictions. - Ensemble predictions from all models and print the final shape. - ``` - - 4. Submission File: - - Save the final predictions as `submission.csv`, ensuring the format matches the competition requirements (refer to `sample_submission` in the Folder Description for the correct structure). - - Present the required submission format explicitly and ensure the output adheres to it. - - 5. Code Standards: - - Do not use progress bars (e.g., tqdm) in the code. - - 6. Ensemble Strategy: - Consolidate all model outputs into a dictionary, where each key is the model's filename (excluding the .py extension) and its corresponding value is the model's output. - Sample code: - {% raw %} - {% for model_name in model_names %} - model_module = __import__(model_name.replace('.py', '')) - val_pred, test_pred, _ = model_module.model_workflow( - X=train_X, - y=train_y, - val_X=val_X, - val_y=val_y, - test_X=X_test_transformed - ) - val_preds_dict[model_module.__name__] = val_pred - test_preds_dict[model_module.__name__] = test_pred - {% endfor %} - final_pred = ensemble_workflow(test_preds_dict, val_preds_dict, val_y) - {% endraw %} + {% include "scenarios.data_science.share:component_spec.Workflow" %} {% if latest_spec %} 7. Former Specification: @@ -313,8 +249,8 @@ data_loader_coder: {% endif %} {% if queried_similar_successful_knowledge|length != 0 %} - --------- Successful Implementations for Similar Models --------- - ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:===== + --------- Successful Implementation Examples for Similar Task --------- + ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Example {{ loop.index }}:===== {{ similar_successful_knowledge.target_task.get_task_information() }} =====Code:===== {{ similar_successful_knowledge.implementation.all_codes }} @@ -339,6 +275,7 @@ data_loader_coder: from joblib import Memory memory = Memory(location='/tmp/cache', verbose=0) @memory.cache``` + {% include "scenarios.data_science.share:guidelines.coding" %} ## Exploratory Data Analysis (EDA) part(Required): - Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better. diff --git a/rdagent/components/coder/data_science/workflow/prompts.yaml b/rdagent/components/coder/data_science/workflow/prompts.yaml index ed314fda..6a312448 100644 --- a/rdagent/components/coder/data_science/workflow/prompts.yaml +++ b/rdagent/components/coder/data_science/workflow/prompts.yaml @@ -40,6 +40,7 @@ workflow_coder: 3. The user may provide specific code organization rules and instructions. Ensure that the integration follows the given framework and structure. 4. After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code. 5. You should avoid using logging module to output information in your generated code, and instead use the print() function. + {% include "scenarios.data_science.share:guidelines.coding" %} ## Output Format {% if out_spec %} @@ -133,4 +134,4 @@ workflow_eval: --------- Workflow test stdout --------- {{ stdout }} --------- Workflow code generated by user --------- - {{ code }} \ No newline at end of file + {{ code }} diff --git a/rdagent/scenarios/data_science/proposal/exp_gen/prompts.yaml b/rdagent/scenarios/data_science/proposal/exp_gen/prompts.yaml index 86e99f3b..99dfe86f 100644 --- a/rdagent/scenarios/data_science/proposal/exp_gen/prompts.yaml +++ b/rdagent/scenarios/data_science/proposal/exp_gen/prompts.yaml @@ -70,11 +70,7 @@ hypothesis_and_feedback: |- task_gen: system: |- - {% if hypothesis is not none %} The user is trying to generate new {{ targets }} based on the hypothesis generated in the previous step. - {% else %} - The user is trying to generate a very simple new {{ targets }} based on the information provided. - {% endif %} The {{ targets }} are used in certain scenario, the scenario is as follows: {{ scenario }} @@ -85,7 +81,7 @@ task_gen: {% endif %} {% if hypothesis is none %} - Since we are at the very beginning stage, we plan to start from a very simple task. To each component, please only generate the task to implement the most simple and basic function of the component. For example, the feature engineering should only implement the function which output the raw data without any transformation. The model component only uses the most basic and easy to implement model without any tuning (but the model type should suit the task). The ensemble component only uses the simplest ensemble method. The main focus at this stage is to build the first runnable version of the solution. + Since we are at the very beginning stage, we plan to start with a very simple task. For example, the feature engineering can only implement the function that outputs the raw data without any transformation. The model component uses the most suitable type of model for the task, but a relatively basic version. The ensemble component only uses the simplest ensemble method. The main focus at this stage is to build the first runnable version of the solution. {% else %} The user will use the {{ targets }} generated to do some experiments. The user will provide this information to you: 1. The target hypothesis you are targeting to generate {{ targets }} for. @@ -269,7 +265,7 @@ component_gen: {{ component_desc }} Please generate the output in JSON format following the format below: - {{ component_output_format }} + {% include "scenarios.data_science.proposal.exp_gen.prompts:output_format.component" %} user: |- Here are the former experiments and their feedbacks: @@ -298,7 +294,7 @@ hypothesis_specification: |- output_format: component: |- { - "reason": "The reason why you choose this component. Based on the current status and former trials, why this component is the most promising one to focus on.", + "reason": "The reason why you chose this component. Based on the current status and former trials, 1) why this component is the most promising one to focus on. 2) Why the component is the right place to apply your idea." "component": "The component you suggest to focus on. It must be one of ['DataLoadSpec', 'FeatureEng', 'Model', 'Ensemble', 'Workflow']." } hypothesis: |- diff --git a/rdagent/scenarios/data_science/proposal/exp_gen/proposal.py b/rdagent/scenarios/data_science/proposal/exp_gen/proposal.py index d7579cde..ddf52ad2 100644 --- a/rdagent/scenarios/data_science/proposal/exp_gen/proposal.py +++ b/rdagent/scenarios/data_science/proposal/exp_gen/proposal.py @@ -106,7 +106,6 @@ class DSProposalV1ExpGen(ExpGen): for key, value in T("scenarios.data_science.share:component_description").template.items() ] ), - component_output_format=T(".prompts:output_format.component").r(), ) component_user_prompt = T(".prompts:component_gen.user").r( diff --git a/rdagent/scenarios/data_science/share.yaml b/rdagent/scenarios/data_science/share.yaml index 464346aa..3eb99581 100644 --- a/rdagent/scenarios/data_science/share.yaml +++ b/rdagent/scenarios/data_science/share.yaml @@ -91,16 +91,24 @@ describe: # some template to describe some object component_description: DataLoadSpec: |- Loads raw competition data, ensuring proper data types, and providing an exploratory data analysis summary. + - When focusing on this component, the corresponding editing files will be: "load_data.py" FeatureEng: |- Transforms raw data into meaningful features while maintaining shape consistency, avoiding data leakage, and optimizing for model performance. It should be model-agnostic (data transformations/augmentations that apply only to specific model frameworks should not be included here). + Ensure that any changes you suggest for feature engineering can be implemented without altering the model's code. If the changes require modifications to the model's code, they are considered specific to the model. We should focus on the model component to apply these changes. + - When focusing on this component, the corresponding editing files will be: "feature.py" Model: |- Perform one of three tasks: model building, which develops a model to address the problem; model tuning, which optimizes an existing model for better performance; or model removal, which discards models that do not contribute effectively. - Handle data operations or augmentations that are closely tied to the model framework, such as tools provided by PyTorch or TensorFlow. + Handle data operations or augmentations that are + 1) closely tied to the model framework, such as tools (e.g., PyTorch's Datasets & DataLoaders) provided by PyTorch or TensorFlow. + 2) cannot be applied in feature engineering ("feature.py") without modifying the model code. + - When focusing on this component, the corresponding editing files will be: "model_*.py" Ensemble: |- Combines predictions from multiple models using ensemble strategies, evaluates their performance, and generates the final test predictions. + - When focusing on this component, the corresponding editing files will be: "ensemble.py" Workflow: |- Integrates all pipeline components, from data loading to ensemble prediction, ensuring efficient execution and correct output formatting. + - When focusing on this component, the corresponding editing files will be: "main.py" component_spec: general: |- @@ -115,7 +123,7 @@ component_spec: - Handle file encoding and delimiters appropriately. - Combine or process multiple files if necessary. - Avoid using the sample submission file to infer test indices. If a dedicated test index file is available, use that. If not, use the order in the test file as the test index. - - Ensure you load the actual data from the files, not just the filenames or paths. Do not postpone data loading to later steps. + - If each prediction sample is linked to a file on disk, simply load the file paths (please load the full path to make it easier to write the loader in following workflows) as X/features without any additional processing. 2. Data Preprocessing: - Convert data types correctly (e.g., numeric, categorical, date parsing). @@ -160,11 +168,14 @@ component_spec: 5. Notes: - GPU and multiprocessing are available and are encouraged to use for accelerating transformations. - Feature engineering should be executed **once** and reused across all models to ensure consistency: `X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)` + - If the data loader returns the file path directly, we can skip feature engineering and return original values directly. Model: |- - Do not use progress bars (e.g., `tqdm`) in the implementation. - The device has GPU support, so you are encouraged to use it for training if necessary to accelerate the process. - Some data transformations/augmentations can be included in this step (e.g., data tools provided by TensorFlow and Torch) + - Please correctly handle data transformations/augmentations, especially when the dataloader loads the file path directly. + - Ensure dynamic handling of feature dimensions to accommodate potential enhancements in input features without requiring code modifications. Ensemble: |- 1. Input Validation: @@ -211,6 +222,7 @@ component_spec: ```python available_models = [f for f in os.listdir('.') if f.startswith('model_') and 'test' not in f] ``` + - The workflow script should be directly executable. We will run your script as is, so do not assume that your functions will be imported and called separately. 2. Feature Engineering - The feature engineering should be called only once. For example: `X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)` @@ -310,3 +322,7 @@ component_spec: 8. Submission File: - Save the final predictions as `submission.csv`, ensuring the format matches the competition requirements (refer to `sample_submission` in the Folder Description for the correct structure). - Present the required submission format explicitly and ensure the output adheres to it. + +guidelines: + coding: |- + You might receive exploratory data analysis (EDA) details about the source data. Do not use this EDA information to create assertions or raise errors. We might generate sample data for quick coding (so your code may run on sample data which is part of the full-size data), but remember that the EDA details are based on the full-size data. \ No newline at end of file