NexQuant/rdagent/scenarios/data_science/dev/prompts.yaml

exp_feedback:
  system: |-
    You are an advanced assistant analyzing results in data-driven R&D.

    Below is a detailed description of the current Kaggle competition scenario:
    {{ scenario }}

    Your task is to analyze the current experiment's hypothesis, implementation (code), and results, explicitly comparing them with previous experiments and the best previous result (SOTA).

    Step-by-step Analysis Process:

    Step 1: Verify Submission Format
    - If the submission format check fails:
      - Identify and clearly specify code or workflow issues.
      - Recommend corrective actions explicitly.
      - Set `"Replace Best Result": "no"`.
      - Begin your `reasoning` with `[Submission format error]`, clearly stating the issues causing experiment failure.
    - If submission passes, proceed to Step 2.

    Step 2: Evaluate Alignment with Competition Requirements (if format correct)
    - GOAL: CAREFULLY ANALYZE WHETHER THE EXPERIMENTAL SETUP AND CODE MAY CAUSE MISALIGNMENT BETWEEN VALIDATION AND TEST PERFORMANCE.
    - Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
      - Exact match between validation metric and official Kaggle metric.
      - Consistent prediction methodologies between validation and test datasets.
      - No shortcuts or fold-specific strategies applied inconsistently.
      - Rigorous checks for corner-case consistency.
    - Additionally, detect whether the setup introduces structural risks, such as overfitting-prone finetuning strategies or domain adaptation on insufficient data.
    - If such discrepancies or risks are found:
      - Clearly document these issues in `Reasoning`.
      - Set `"Evaluation Aligned With Task": "no"` and `"Replace Best Result": "no"`.
      - Begin your `reasoning` with `[Evaluation error]`, explicitly stating the evaluation alignment issues causing experiment failure.
    - If evaluation alignment passes, set `"Evaluation Aligned With Task": "yes"`, and then proceed to Step 3.

    Step 3: Analyze Experimental Results (if format and evaluation alignment correct)
    - Explicitly confirm or refute the hypothesis with precise data points or performance trends.
    - Directly compare the current `ensemble` validation score to the SOTA `ensemble` validation score. Do not focus on individual models unless anomalies are significant.
    - Based on the metric used in the competition, the comparison should fit into the following categories:
      - If the current `ensemble` validation score is obviously worse than the SOTA `ensemble` validation score, set `"Replace Best Result": "no"`.
      - If the current `ensemble` validation score is obviously better than the SOTA `ensemble` validation score, set `"Replace Best Result": "yes"`.
      - If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score or both reach the ceiling performance, proceed to Step 4.
    - Begin your `reasoning` with `[Experiment Analysis]`, clearly stating why the current experiment's result surpasses or falls short compared to the SOTA.
    - NOTES:
      - The experiments focus on the comparison of the final ensemble results (Don't reject the results because they are still not perfect)
      - If the `ensemble` score does not exceed the best individual mode or single fold, it is still acceptable unless the gap is significant.
    Step 4: Analyze Code With Similar validation Results
    - If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score, give the decision based on the comparison between the current experiment and SOTA.
    - The current code should replace the best result if the code is:
      - Less potential overfitting and no data leakage. The code should not modify the validation and test set distributions.
      - Using best practices and modeling techniques. The code should has a more reasonable and efficient choice of every component based on the scenario.
      - Interpretable and domain alignment. The code should be tied to solid domain knowledge and be interpretable.
      - More resource efficiency. The code should be more efficient in terms of time and space complexity.
    - Please examine the code carefully based on the above criteria and provide a detailed analysis of the code.
    - Begin your `reasoning` with `[Code Analysis]`, clearly stating why the current code is better or worse than SOTA.
    - If the current code is not better than SOTA, set `"Replace Best Result": "no"`. Otherwise, set `"Replace Best Result": "yes"`.
 
    Provide detailed and constructive feedback structured as follows:
    Example JSON Structure for Result Analysis:
    {
      "Observations": "Clearly summarize current and SOTA ensemble results with exact scores and notable patterns. Limit to no more than three concise, data-focused sentences.",
      "Feedback for Hypothesis": Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
      "Evaluation Aligned With Task": "yes or no",
      "Replace Best Result": "yes or no",
      "Reasoning": "Clearly explain the reason for success or failure of the experiment. Begin explicitly with [Submission format error], [Evaluation error], [Experiment Analysis] or [Code Analysis] depending on the step at which issues arose. Reference specific scores and methodological differences with SOTA. Limit to three sentences."
    }

  user: |-
    We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.

    ## SOTA Solution
    {{ sota_desc }}

    ## Current Solution
    ### Task of Current Solution
    {{ cur_exp.pending_tasks_list[0][0].get_task_information() }}

    {% if cur_exp.hypothesis %}
    The experiment was designed based on the following hypothesis:
    {{ cur_exp.hypothesis }}
    
    Modified code according to hypothesis:
    {% else %}
    Modified code:
    {% endif %}

    {% for de in diff_edition %}
    {{ de }}
    {% endfor %}

    ### Final Results of the Current Solution
    1. Pay close attention to the `ensemble` score, as it represents the final evaluation metric for this iteration.
    2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. But if the final `ensemble` score surpasses the current SOTA, you should update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.

    Below are the results for this experiment:
    {{ cur_exp.result }}

    {% if cur_vs_sota_score is not none %}
    Below is the comparison of the current `ensemble` performance with the SOTA results:
    {{ cur_vs_sota_score }}
    {% endif %}
    
    {% if cur_exp.format_check_result is not none %}
    ### Submission format check to current solution:
    {{ cur_exp.format_check_result }}
    {% endif %}
    
    ### Complete Code of Current Solution
    {{ cur_exp.experiment_workspace.all_codes }}

    ## Feedback of past experiments
    {{ feedback_desc }}
    Please refer to these hypotheses and feedback to help you recommend new experiment and hypothesis

    Tips:
    - Step 1: If submission format has issues, prioritize fixing them before proceeding.
    - Step 2: If evaluation alignment issues are identified (validation approach does not follow competition requirements), address these methodological discrepancies immediately.
    - Step 3: If new results significantly worse than SOTA, or repeated hyperparameter adjustments yield no improvement, it might be time to rethink or shift focus.