NexQuant/rdagent/scenarios/data_science/dev/runner/prompts.yaml

DSCoSTEER_eval:
  system: |-
    {% include "scenarios.data_science.share:scen.role" %}
    {% if is_sub_enabled %}
    You will be provided with:
    1. `Code base`: The code base of the solution
    2. `The stdout of code execution and testing`: The generated stdout when executing the code base and corresponding testing
    3, `The time spent on code execution`: The time spent on the code execution
    4. `The timeout of code execution`: the time limitation of the code execution
    5. `The percent of timeout used`: the percentage of the time limitation used
    Your task is to perform the following evaluation(s):

    # Evalution 1: Code Correctness
    ## Scenario
    The code is focusing on the following scenario:
    {{ scenario }}

    ## Target Task Description
    The code is focusing on the following task
    {{ task_desc }}

    ## Runtime Environment
    You have following environment to run the code:
    {{ runtime_environment }}

    ## Evaluation Guidelines
    1. Evaluate the code base based on several aspects, including execution correctness, return checking, and code quality.
    2. Ensure the code does not contain any incorrect, fabricated, or deceptive operations, such as mocking data, scores, or results.
    3. Confirm that the prediction file (`submission.csv`) is generated using only the test dataset, and its format matches the sample submission.
    If the code does not satisfy the requirements:
    - Set "final_decision" to false.
    - set "hyperparameter_tuning_decision" to false.
    - Set "hyperparameter_tuning_suggestion" to an empty string.
    If the code satisfy the requirements:
    - Proceed to the next evaluation.

    # Evaluation 2: Hyperparameter
    ## Evaluation Description
    The user will provide you the time spent on the whole code execution and the timeout of the code execution. You should decide whether the hyperparameter is reasonable based on the time.
    For example, if the code uses only a very small portion of the allowed time, and hyperparameters like `n_estimators` or `epochs` have low values, with early stopping not being triggered and possible signs of underfitting, you should suggest increasing these hyperparameters.
    You should also notice other resources utilization hyper-parameters,
    For example, if you are using a GPU with large memory, and the batch size is set very low, you should suggest increasing the batch size if it is not reasonable.

    ## Evaluation Guidelines
    1. The code execution time or resource utilization suggest that there is room for improvement in the hyperparameters.
    2. The code must apply early stopping strategy already (in order to prevent overfitting).
    3. Your suggestion should have a strong chance of improving the model's performance. Focus on the most obvious and impactful opportunities for quick improvement by leveraging more training time. Don't explore hyperparameters with low confidence.  If there are no obvious and impactful opportunities and the code runs well, please accept it.
    If the code satisfy the requirements:
    - Set "hyperparameter_tuning_decision" to true.
    - Set "final_decision" to false.
    - Provide a reasonable suggestion in "hyperparameter_tuning_suggestion". The "hyperparameter_tuning_suggestion" should begin with a clear observation, followed by your suggestion. For example: "[Observation] The maximum number of epochs was reached, but the validation loss is still going down and early stopping was not activated. Only 15% of the allowed time was used. [Suggestion] We recommend increasing epochs to 100 to avoid underfitting and further improve model performance."
    If the code does not satisfy the requirements:
    - Set "hyperparameter_tuning_decision" to false.
    - Set "hyperparameter_tuning_suggestion" to an empty string.

    ## Output format
    Please respond with your feedback in the following JSON format and order without anything else:
    ```json
    {
        "execution": "Describe whether the whole code base executed successfully and generating the final submission. Include any errors or issues encountered, and retain all error messages and traceback details.",
        "return_checking": "Verify the generated files, particularly the submission file. Ensure that its format matches the sample submission",
        "code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
        "final_decision": <true/false>,
        "hyperparameter_tuning_decision": <true/false>,
        "hyperparameter_tuning_suggestion": <suggestion in plain text for hyperparameter tuning>,
    }
    ```
    {% else %}
    The user will provide you the whole code base, some logs generated during the execution of the whole workflow. Your evaluation scope includes whether the workflow code executes successfully.
    No need to check the detail of submission file.

    Please respond with your feedback in the following JSON format and order
    ```json
    {
        "execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Describe the expected file to be generated.",
        "code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
        "final_decision": <true/false>,
        "hyperparameter_tuning_decision": <true/false>,
        "hyperparameter_tuning_suggestion": <suggestion in plain text for hyperparameter tuning>,
    }
    ```
    {% endif %}
# NOTE: when is_sub_enabled == False, we don't have any checking about the return. So it is just placeholder currently

  user: |-
    # Code base
    {{ code }}

    ## Stdout of code execution and testing
    {{ stdout }}

    # The time spend on code execution and timeout
    {{ time_spent }}

    ## The timeout of code execution
    {{ timeout }}

    ## The percent of timeout used
    {{ percent_of_timeout_used }}

DSCoSTEER:
  system_debugger: |-
    {% include "scenarios.data_science.share:scen.role" %}
    You have finished the implementation of the whole workflow which has executed well on a sampled dataset. Now we are working on the full dataset.
    The user has reported that the workflow failed to execute on the full dataset.
    Your will be provided with:
    1. Code base.
    2. Task description, which is the task the code is trying to solve.
    3. Feedback generated during the execution of the whole workflow.
    4. Suggestions for hyperparameter tuning.
    Your job is to debug the whole code base, try to correct the errors, and ensure that the workflow can execute successfully on the full dataset.

    ## Task description
    {{ task_desc }}

    ## Instructions
    1. Minimal changes principle: only modify the code that is necessary to fix the issues but not affect any other parts of the code. Try to correct as less files as possible since files are interdependent.
    {% if diff_mode %}
    2. You must output in Code Diff format. The detailed format specification is as follows.
    {% else %}
    2. You must output the COMPLETE and FULL code. Do not truncate, summarize, or omit any parts of the code. Include all imports, functions, classes, and the entire workflow from start to finish.
    {% endif %}

    ## Output Format
    {% if out_spec %}
    {{ out_spec }}
    {% else %}
    Please response the code in the following JSON format without anything else.
    {
        "code": "The Python code as a string."
    }
    {% endif %}

  system_refine: |-
    {% include "scenarios.data_science.share:scen.role" %}
    You have finished the implementation of the whole workflow which has executed well on a sampled dataset. Now we are working on the full dataset.
    The user has reported that the hyperparameters are not reasonable and the code didn't make the best use of the time limit.
    Your will be provided with:
    1. Code base.
    2. Feedback generated during the execution of the whole workflow.
    3. Suggestions for hyperparameter tuning.
    Your task is to refine the code base and modify the hyperparameters based on the feedback and suggestions.

    ## Instructions
    1. Minimal changes principle: only modify necessary hyperparameters based on the feedback and suggestions.
    {% if diff_mode %}
    2. You must output in Code Diff format. The detailed format specification is as follows.
    {% else %}
    2. You must output the COMPLETE and FULL code. Do not truncate, summarize, or omit any parts of the code. Include all imports, functions, classes, and the entire workflow from start to finish.
    {% endif %}

    ## Output Format
    {% if out_spec %}
    {{ out_spec }}
    {% else %}
    Please response the code in the following JSON format without anything else.
    {
        "code": "The Python code as a string."
    }
    {% endif %}

  user: |-
    # Code Base
    {{ code }}

    ## Feedback
    {{ feedback }}

    {% if hyperparameter_tuning_suggestion is not none %}
    ## Hyperparameter Tuning Suggestion
    {{ hyperparameter_tuning_suggestion }}
    {% endif %}