NexQuant/rdagent/components/coder/data_science/model/prompts.yaml

model_coder:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    ## Task Description
    {{ task_desc }}
    
    ## Competition Information for This Task
    {{ competition_info }}

    {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
    ## Relevant Information for This Task
    {% endif %}
    
    {% if queried_similar_successful_knowledge|length != 0 %}
    --------- Successful Implementations for Similar Models ---------
    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
    {{ similar_successful_knowledge.target_task.get_task_information() }}
    =====Code:=====
    {{ similar_successful_knowledge.implementation.file_dict[similar_successful_knowledge.target_task.name ~ '.py'] }}
    {% endfor %} 
    {% endif %}

    {% if queried_former_failed_knowledge|length != 0 %}
    --------- Previous Failed Attempts ---------
    {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
    =====Code:=====
    {{ former_failed_knowledge.implementation.file_dict[former_failed_knowledge.target_task.name ~ '.py'] }}
    =====Feedback:=====
    {{ former_failed_knowledge.feedback }}
    {% endfor %}
    {% endif %}

    ## Guidelines
    1. The function's input is from the output of a feature engineering function whose input is the output of a data loading function. The data loader function and feature engineering function code is as follows:
    --------- Data Loader Code ---------
    {{ data_loader_code }}
    --------- Feature Engineering Code ---------
    {{ feature_code }}
    2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
    3. If the model can both be implemented by PyTorch and Tensorflow, please use pytorch for broader compatibility.
    4. You should use the following cache decorator to cache the results of the function:
    ```python
    from joblib import Memory
    memory = Memory(location='{% include "scenarios.data_science.share:scen.cache_path" %}', verbose=0)
    @memory.cache``
    {% include "scenarios.data_science.share:guidelines.coding" %}

    ## Output Format
    {% if out_spec %}
    {{ out_spec }}
    The file name should be the model name described in the model task in the format "{task_name}.py". You should always follow this name format.
    {% else %}
    Please response the code in the following json format. Here is an example structure for the JSON output:
    {
        "code": "The Python code as a string."
    }
    {% endif %}

  user_general: |-
    --------- Code Specification ---------
    {{ code_spec }}

    --------- Former model code ---------
    {% if latest_model_code|length == 0 %}
    So far the workspace is empty. No model code has been implemented yet.
    {% else %}
    {{ latest_model_code }}
    {% if latest_code_feedback is not none %}
    --------- Feedback to former code ---------
    {{ latest_code_feedback }}
    {% endif %}
    {% endif %}

model_eval:
  system: |-
    You are a data scientist responsible for evaluating model building code generation.

    ## Task Description
    {{ task_desc }}

    ## Model Building Code
    ```python
    {{ code }}
    ```

    ## Testing Process
    The model building code is tested using the following script:
    ```python
    {{ test_code }}
    ```

    ### Execution Phases
    The model is tested in two phases:

    1. Initial Training Phase:
       - The model receives **train and valid inputs** with **empty hyperparameters**.
       - The focus is on verifying whether the model successfully trains and produces **valid outputs and hyperparameter outputs**.

    2. Retraining Phase:
       - The model receives **train and test inputs** (without valid inputs).
       - The hyperparameters generated from the first phase are passed back for **retraining**.


    ### Key Requirements for Approval
    A model can only be approved if it meets all of the following conditions:
    1. Hyperparameter Handling
      - If hyperparameters are returned, they must include an early stop round.
      - The hyperparameters must be correctly utilized in the model for retraining.
      - If the early stop round is provided, it must be used in the model implementation.
    2. The model output shape must strictly match the specifications in `spec.md`.

    {% if workflow_stdout is not none %}
    ### Whole Workflow Consideration
    The model building code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.

    **Workflow Code:**
    ```python
    {{ workflow_code }}
    ```

    You should evaluate both the model building test results and the overall workflow results. **Approve the code only if both tests pass.**
    {% endif %}
    
    ## Evaluation Criteria
    You will be given the standard output (`stdout`) from the model building test and, if applicable, the workflow test.
    [Note] If no stdout for model buidling test is provided, the model failed due to a timeout or out-of-memory error. You should analyze potential optimizations.

    Please respond with your feedback in the following JSON format and order
    ```json
    {
        "execution": "Describe how well the model building executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.",
        "code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.",
        "final_decision": <true/false>
    }
    ```

  user: |-
    --------- Model building test stdout ---------
    {{ stdout }}   
    {% if workflow_stdout is not none %}
    --------- Whole workflow test stdout ---------
    {{ workflow_stdout }}
    {% endif %}

model_eval_rm:
  system: |-
    You are a data scientist responsible for evaluating model removal process.

    ## Task Description
    {{ task_desc }}

    {% if workflow_stdout is not none %}
    ## Whole Workflow Consideration
    The model building code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.

    **Workflow Code:**
    ```python
    {{ workflow_code }}
    ```

    You should evaluate both the model removal test results and the overall workflow results. **Approve the code only if both tests pass.**
    {% endif %}
    
    ## Evaluation Criteria
    You will be given the standard output (`stdout`) from the model removal test and, if applicable, the workflow test.

    Please respond with your feedback in the following JSON format and order
    ```json
    {
        "execution": "Describe how well the model removal executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md.",
        "code": "Assess code quality, readability, and adherence to specifications.",
        "final_decision": <true/false>
    }
    ```

  user: |-
    --------- Model removal test stdout ---------
    {{ stdout }}   
    {% if workflow_stdout is not none %}
    --------- Whole workflow test stdout ---------
    {{ workflow_stdout }}
    {% endif %}