feat: refine prompt (#760)

* style: Simplify language and improve clarity in prompts and share.yaml

* style: Update prompt wording for clarity in raw_data_loader

* style: Simplify conditional logic in task_gen system prompt

* refactor: Update prompts and proposal for component output format handling

* fix: Correct grammar and add clarification in prompts.yaml

* feat: Include coding guidelines in data science component prompts

* lint

rdagent/components/coder/data_science/ensemble/prompts.yaml

@@ -39,6 +39,7 @@ ensemble_coder:
     1. The function's code is associated with several other functions including a data loader, feature engineering, and model training. all codes are as follows:
     {{ all_code }}
     2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
+    {% include "scenarios.data_science.share:guidelines.coding" %}
 
     ## Output Format
     {% if out_spec %}

rdagent/components/coder/data_science/feature/prompts.yaml

@@ -48,6 +48,10 @@ feature_coder:
     from joblib import Memory
     memory = Memory(location='/tmp/cache', verbose=0)
     @memory.cache```
+    6. Coding tricks:
+      - If the input consists of a batch of file paths and you need to modify the file contents to complete your feature engineering task, you can accomplish your feature engineering task by modifying these files and creating new files in a subfolder within "/tmp/cache" (this path is persistent, otherwise you may lose your created file). Then the new file paths are returned.
+
+    {% include "scenarios.data_science.share:guidelines.coding" %}
 
     ## Output Format
     {% if out_spec %}

rdagent/components/coder/data_science/model/prompts.yaml

@@ -45,6 +45,7 @@ model_coder:
     from joblib import Memory
     memory = Memory(location='/tmp/cache', verbose=0)
     @memory.cache``
+    {% include "scenarios.data_science.share:guidelines.coding" %}
 
     ## Output Format
     {% if out_spec %}

rdagent/components/coder/data_science/pipeline/prompts.yaml

@@ -57,6 +57,7 @@ pipeline_coder:
       User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
     - An evaluation agent will help to check whether the EDA part is added correctly.
     - During the EDA part, you should try to avoid any irrelevant information sending to the standard output.
+    {% include "scenarios.data_science.share:guidelines.coding" %}
 
     {% if enable_model_dump %}
     ## Model Dumping

rdagent/components/coder/data_science/raw_data_loader/prompts.yaml

@@ -224,71 +224,7 @@ spec:
       You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.
 
     workflow: |-
-      Your task is to implement the main workflow script (`main.py`) for a Kaggle-style machine learning competition project. 
-      Follow the provided project structure and specifications to ensure consistency and maintainability:
-      1. Workflow Integration:
-        - Integrate the following components into the workflow:
-          - Data loading (`load_data.py`).
-          - Feature engineering (`feature.py`).
-          - Model workflow for training and testing (`model_*.py`). 
-          - Ensemble workflow that combines results from the model workflow to obtain the final prediction (`ensemble.py`).
-        - Treat each component as a modular and callable Python function.
-        - The workflow script should be flexible enough to handle either a single model or multiple models, with filenames (model_*.py) that are not determined at the outset.
-          For multiple model selection, utilize Python code to identify eligible models based on filenames, for example:
-          ```python
-          available_models = [f for f in os.listdir('.') if f.startswith('model_') and 'test' not in f]
-          ```
-      2. Feature Engineering
-        - The feature engineering should be called only once. For example:
-          `X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)`
-        - It should be called before dataset splitting.
-
-      3. Dataset Splitting
-        - The dataset returned by `load_data` is not pre-split. After calling `feat_eng`, split the data into training and test sets.
-        - [Notice] If feasible, apply cross-validation on the training set (`X_transformed`, `y_transformed`) to ensure a reliable assessment of model performance.
-        - Keep the test set (`X_test_transformed`) unchanged, as it is only used for generating the final predictions.
-        - Pseudocode logic for reference:
-          ```
-          Set number of splits and initialize KFold cross-validator.
-          Create dictionaries for validation and test predictions.
-          For each model file:
-              Import the model dynamically.
-              Initialize arrays for out-of-fold (OOF) and test predictions.
-              For each fold in KFold:
-                  Split data into training and validation sets.
-                  Run model workflow to get validation and test predictions.
-                  Validate shapes.
-                  Store validation and test predictions.
-              Compute average test predictions across folds.
-              Save OOF and averaged test predictions.
-          Ensemble predictions from all models and print the final shape.
-          ```
-
-      4. Submission File:
-        - Save the final predictions as `submission.csv`, ensuring the format matches the competition requirements (refer to `sample_submission` in the Folder Description for the correct structure).
-        - Present the required submission format explicitly and ensure the output adheres to it.
-
-      5. Code Standards:
-        - Do not use progress bars (e.g., tqdm) in the code.
-
-      6. Ensemble Strategy:
-        Consolidate all model outputs into a dictionary, where each key is the model's filename (excluding the .py extension) and its corresponding value is the model's output.
-        Sample code:
-        {% raw %}
-        {% for model_name in model_names %}
-        model_module = __import__(model_name.replace('.py', ''))
-        val_pred, test_pred, _ = model_module.model_workflow(
-          X=train_X,
-          y=train_y,
-          val_X=val_X,
-          val_y=val_y,
-          test_X=X_test_transformed
-        )
-        val_preds_dict[model_module.__name__] = val_pred
-        test_preds_dict[model_module.__name__] = test_pred
-        {% endfor %}
-        final_pred = ensemble_workflow(test_preds_dict, val_preds_dict, val_y)
-        {% endraw %}
+      {% include "scenarios.data_science.share:component_spec.Workflow" %}
 
       {% if latest_spec %}
       7. Former Specification:
@@ -313,8 +249,8 @@ data_loader_coder:
     {% endif %}
     
     {% if queried_similar_successful_knowledge|length != 0 %}
-    --------- Successful Implementations for Similar Models ---------
-    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
+    --------- Successful Implementation Examples for Similar Task ---------
+    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Example {{ loop.index }}:=====
     {{ similar_successful_knowledge.target_task.get_task_information() }}
     =====Code:=====
     {{ similar_successful_knowledge.implementation.all_codes }}
@@ -339,6 +275,7 @@ data_loader_coder:
     from joblib import Memory
     memory = Memory(location='/tmp/cache', verbose=0)
     @memory.cache```
+    {% include "scenarios.data_science.share:guidelines.coding" %}
     
     ## Exploratory Data Analysis (EDA) part(Required):
     - Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better.

rdagent/components/coder/data_science/workflow/prompts.yaml

@@ -40,6 +40,7 @@ workflow_coder:
     3. The user may provide specific code organization rules and instructions. Ensure that the integration follows the given framework and structure.
     4. After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code.
     5. You should avoid using logging module to output information in your generated code, and instead use the print() function.
+    {% include "scenarios.data_science.share:guidelines.coding" %}
 
     ## Output Format
     {% if out_spec %}
@@ -133,4 +134,4 @@ workflow_eval:
     --------- Workflow test stdout ---------
     {{ stdout }}
     --------- Workflow code generated by user ---------
-    {{ code }}
+    {{ code }}