NexQuant/rdagent/components/coder/data_science/pipeline/prompts.yaml

pipeline_coder:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    ## Task Description
    {{ task_desc }}

    ## The runtime environment your code will running on
    {{ runtime_environment }}

    ## Specification your code should follow
    {{ spec }}

    {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
    ## Relevant Information for This Task
    {% endif %}
    
    {% if queried_similar_successful_knowledge|length != 0 %}
    --------- Successful Implementations for Similar Models ---------
    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
    {{ similar_successful_knowledge.target_task.get_task_information() }}
    =====Code:=====
    {{ similar_successful_knowledge.implementation.all_codes }}
    {% endfor %} 
    {% endif %}

    {% if queried_former_failed_knowledge|length != 0 %}
    --------- Previous Failed Attempts ---------
    {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
    =====Code:=====
    {{ former_failed_knowledge.implementation.all_codes }}
    =====Feedback:=====
    {{ former_failed_knowledge.feedback }}
    {% endfor %}
    {% endif %}


    ## Guidelines
    1. Ensure that the dataset is loaded strictly from `{% include "scenarios.data_science.share:scen.input_path" %}`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).
    2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
    
    ## Exploratory Data Analysis (EDA) part(Required):
    - Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better.
    - The EDA part should include but not limited in the following information in plain text:
      - The shape of the data.
      - The first 5 rows of the data.
      - The data types of each column.
      - The number of missing values in each column.
      - The number of unique values in each column.
      - The distribution of the target variable.
      - Any other information that you think is important for the following steps.
    - The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema: 
      === Start of EDA part ===
      { You EDA output content }
      === End of EDA part ===
      User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
    - An evaluation agent will help to check whether the EDA part is added correctly.
    - During the EDA part, you should try to avoid any irrelevant information sending to the standard output.
    {% include "scenarios.data_science.share:guidelines.coding" %}

    {% if enable_model_dump %}
    ## Model Dumping
    {% include "components.coder.data_science.share.prompts:dump_model_coder.guideline" %}
    {% endif %}



    ## Output Format
    {% if out_spec %}
    {{ out_spec }}
    {% else %}
    Please response the code in the following json format. Here is an example structure for the JSON output:
    {
        "code": "The Python code as a string."
    }
    {% endif %}

  user: |-
    --------- Competition Information ---------
    {{ competition_info }}

    --------- Data Folder Description (All path are relative to the data folder) ---------
    {{ folder_spec }}
    
    {% if latest_code %}
    --------- Former code ---------
    {{ latest_code }}
    {% if latest_code_feedback is not none %}
    --------- Feedback to former code ---------
    {{ latest_code_feedback }}
    The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
    {% else %}
    The former code is correct. You should try to improve the code based on the provided task while not changing the irrelevant parts.
    {% endif %}
    {% endif %} 

    You should strictly follow the code specifications provided by the specification to implement the function.


pipeline_eval:
  system: |-
    You are a data scientist responsible for evaluating code generation.

    ## Task Description
    The user is trying to build a code in the following scenario:
    {{ scenario }}

    The main code generation task is as follows:
    {{ task_desc }}

    The details on how to structure the code are given in the specification:
    {{ spec }}
    
    {% if is_sub_enabled %}
    ## Evaluation Scope
    Your focus is to check whether the workflow code:
    Step 1: Executes successfully without any errors. Please distinguish between the errors and warnings.

    Step 2: Correctly generates a final submission in the correct format, ensuring: they align with the submission structure, the index names and column names should match the sample, and the items should not be empty or apparently incorrect.
    
    Step 3: Aligns with the competition requirements. This includes:
    - CAREFULLY ANALYZE WHETHER THE EXPERIMENTAL SETUP AND CODE MAY CAUSE MISALIGNMENT BETWEEN VALIDATION AND TEST PERFORMANCE.
    - Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
      - Exact match between the implementation code of metric and the requirements of the scenario. The metric number is not the focus.
      - Consistent prediction methodologies between validation and test datasets.
      - No shortcuts or fold-specific strategies applied inconsistently.
      - Rigorous checks for corner-case consistency.
    - If such discrepancies or risks are found:
      - Clearly document these issues in `code`.
      - Begin your `code` with `[Evaluation error]`, explicitly stating the evaluation alignment issues causing experiment failure.
    - If no issues are found, begin your `code` with `[Code analysis]`, providing a detailed analysis of the code quality, readability, and adherence to specifications.

    ## Evaluation Criteria
    You will be given the execution output (`stdout`) to determine correctness.  

    [Note] 
    1. Model performance is NOT a concern in this evaluation—only correct execution and formatting matter.
    2. You only check the format of the submission since we only feed you part of the data, so the submission might has different index to the sample submission data.
    
    Please respond with your feedback in the following JSON format and order
    ```json
    {
        "execution": "Describe whether the code executed successfully, correctly integrating all components and generating the final submission. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Verify the generated files, particularly the submission file. Ensure that its format matches the sample submission, checking the index, column names, and CSV content.",
        "code": "Begin explicitly with [Code analysis] or [Evaluation error]. Provide feedback on code quality, readability, adherence to the given specifications, and alignment with competition requirements.",
        "final_decision": <true/false>
    }
    ```
    {% else %}
    ## Evaluation Scope
    Your focus is to check whether the workflow code executes successfully.

    You will be given the execution output (`stdout`) to determine correctness.  

    [Note] 
    1. Model performance is NOT a concern in this evaluation—only correct execution and formatting matter.

    Please respond with your feedback in the following JSON format and order
    ```json
    {
        "execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Describe the expected file to be generated.",
        "code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
        "final_decision": <true/false>
    }
    ```
    {% endif %}
# NOTE: when is_sub_enabled == False, we don't have any checking about the return. So it is just placeholder currently

  user: |-
    --------- code generated by user ---------
    {{ code }}

    --------- code running stdout ---------
    {{ stdout }}