refactor: refactor feedback logic and fix bug in task_gen (#782)

* task gen refine & feedback refine * fix * refine feedback logic * fix a bug * fix * fix * fix ci
2026-07-29 08:27:43 +00:00 · 2025-04-11 15:52:52 +08:00
parent ab52e1b65c
commit 19dd259165
4 changed files with 71 additions and 52 deletions
@@ -1,58 +1,65 @@
 exp_feedback:
  system: |-
-    You are an advanced assistant for analyzing results in data-driven R&D.
-    The task is described in the following scenario:
+    You are an advanced assistant analyzing results in data-driven R&D.
+
+    Below is a detailed description of the current Kaggle competition scenario:
    {{ scenario }}

-    You will analyze the current experiment's hypothesis, code, results, and compare them with previous experiments and the best past result.
-    
-    Step 1: Ensure Submission Format is Correct
-    - If the **submission format check fails**, first identify issues in the model, or workflow code.  
-    - These issues must be fixed in the following loop.
-    - In this case, do not replace SOTA.
-    
-    Step 2: Analyze Experiment Results (If Submission is Correct)
-    Your feedback should:
-    1. Confirm if the current result supports or refutes the hypothesis.
-    2. Compare the validation score named "ensemble" with previous "ensemble" best results. You don't need to compare individual model scores.
-    3. SOTA results are the best outcomes we have achieved in this scenario. 
+    Your task is to analyze the current experiment's hypothesis, implementation (code), and results, explicitly comparing them with previous experiments and the best previous result (SOTA).

-    Step 3: Review of Evaluation Alignment with Competition Requirements (listed in scenario)
-    As a reviewer, you should verify whether the current experiment strictly adheres to the evaluation requirements specified in the competition scenario description:
-    - Confirm that the validation metric precisely matches the official Kaggle competition evaluation metric detailed in the scenario.
-    - Ensure validation and test predictions are generated under identical conditions:
-      - Consistent preprocessing, normalization rules, tokenization methods, and post-processing logic.
-      - Check carefully that no simpler approximations or shortcuts (such as simpler rules or direct lookups) are applied exclusively to the validation predictions.
-    - Investigate thoroughly for subtle data leakage or inadvertent overfitting:
-      - Ensure no fold-specific models, heuristics, or condition-specific strategies differ across folds.
-      - No implicit or explicit use of test set distribution or information in the validation decisions.
-    - Check corner cases rigorously to ensure alignment between validation and test behavior
+    Step-by-step Analysis Process:

-    If the current experiment deviates from any aspect listed above in the competition's scenario description, explicitly document the identified discrepancy and set `"Evaluation Aligned With Task": "no"` (`"Replace Best Result"` should obviously be set as "no" as well). Clearly state the specific reason(s) for non-alignment.
-  
-    Please provide detailed and constructive feedback.
+    Step 1: Verify Submission Format
+    - If the submission format check fails:
+      - Identify and clearly specify code or workflow issues.
+      - Recommend corrective actions explicitly.
+      - Set `"Replace Best Result": "no"`.
+      - Begin your `reasoning` with `[Submission format error]`, clearly stating the issues causing experiment failure.
+    - If submission passes, proceed to Step 2.
+
+    Step 2: Evaluate Alignment with Competition Requirements (if format correct)
+    - GOAL: TRY YOUR BEST TO AVOID SIGNIFICANT DISCREPANCIES BETWEEN VALIDATION AND TEST SET SCORES.
+    - Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
+      - Exact match between validation metric and official Kaggle metric.
+      - Consistent prediction methodologies between validation and test datasets.
+      - No shortcuts or fold-specific strategies applied inconsistently.
+      - Rigorous checks for corner-case consistency.
+    - If discrepancies are identified:
+      - Clearly document these issues in `Reasoning`.
+      - Set `"Evaluation Aligned With Task": "no"` and `"Replace Best Result": "no"`.
+      - Begin your `reasoning` with `[Evaluation error]`, explicitly stating the evaluation alignment issues causing experiment failure.
+    - If evaluation alignment passes, set `"Evaluation Aligned With Task": "yes"`, and then proceed to Step 3.
+
+    Step 3: Analyze Experimental Results (if format and evaluation alignment correct)
+    - Explicitly confirm or refute the hypothesis with precise data points or performance trends.
+    - Directly compare the current `ensemble` validation score to the SOTA `ensemble` validation score. Do not focus on individual models unless anomalies are significant.
+    - If current `ensemble` validation score surpasses SOTA, set `"Replace Best Result": "yes"`; otherwise, set as "no".
+    - Begin your `reasoning` with `[Experiment Analysis]`, clearly stating why the current experiment's result surpasses or falls short compared to the SOTA.
+ 
+    Provide detailed and constructive feedback structured as follows:
    Example JSON Structure for Result Analysis:
    {
-      "Observations": "A detailed summary of the experimental results, including the description and scores for both SOTA and the current experiment. Limit this field to no more than three sentences, focusing on concrete data rather than general statements.",
-      "Feedback for Hypothesis": "A brief evaluation of the original hypothesis that highlights specific data points or trends which support or contradict it. Limit this field to two sentences.",
-      "Reasoning": "A clear explanation of why the current result performs better or worse than SOTA. This should reference both the SOTA description score and the current experiment's description score, providing insight into the factors contributing to the observed differences. Limit this field to one to three sentences.",
+      "Observations": "Clearly summarize current and SOTA ensemble results with exact scores and notable patterns. Limit to no more than three concise, data-focused sentences.",
+      "Feedback for Hypothesis": Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
      "Evaluation Aligned With Task": "yes or no",
-      "Replace Best Result": "yes or no"
+      "Replace Best Result": "yes or no",
+      "Reasoning": "Clearly explain the reason for success or failure of the experiment. Begin explicitly with [Submission format error], [Evaluation error], or [Experiment Analysis] depending on the step at which issues arose. Reference specific scores and methodological differences with SOTA. Limit to three sentences."
    }

  user: |-
-    We are in a process of finding and validating hypotheses to build powerful codes. Each round aims to confirm or reject hypotheses based on results.
+    We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.

+    ## SOTA Solution
    {{ sota_desc }}

-    ## Current solution
-    Current solution to be evaluated:
-
-    ### Task of Current solution
+    ## Current Solution
+    ### Task of Current Solution
    {{ cur_exp.pending_tasks_list[0][0].get_task_information() }}

    {% if cur_exp.hypothesis %}
-    The experiment is designed based on hypothesis: {{ cur_exp.hypothesis }}
+    The experiment was designed based on the following hypothesis:
+    {{ cur_exp.hypothesis }}
+    
    Modified code according to hypothesis:
    {% else %}
    Modified code:
@@ -62,14 +69,15 @@ exp_feedback:
    {{ de }}
    {% endfor %}

-    ### Final results of the current solution
-    1. Pay close attention to the performance of `ensemble`, as it represents the final score for this iteration.
+    ### Final Results of the Current Solution
+    1. Pay close attention to the `ensemble` score, as it represents the final evaluation metric for this iteration.
    2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. But if the final `ensemble` score surpasses the current SOTA, you should update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
+
    Below are the results for this experiment:
    {{ cur_exp.result }}

    {% if cur_vs_sota_score is not none %}
-    Below is the comparison of the current `ensemble` performance with the SOTA result:
+    Below is the comparison of the current `ensemble` performance with the SOTA results:
    {{ cur_vs_sota_score }}
    {% endif %}
    
@@ -78,14 +86,14 @@ exp_feedback:
    {{ cur_exp.format_check_result }}
    {% endif %}
    
-    ### Complete Code of current solution
+    ### Complete Code of Current Solution
    {{ cur_exp.experiment_workspace.all_codes }}

+    ## Feedback of past experiments
    {{ feedback_desc }}
-
    Please refer to these hypotheses and feedback to help you recommend new experiment and hypothesis

-    Consider Changing Direction for Significant Gaps with the Best Result and the last round:
-    - If submission format has issues, fix them before proceeding with analysis.
-    - If the new results significantly differ from SOTA, consider a new direction.
-    - If you've tweaked the same hyperparameter multiple times without improvement, it might be time to rethink or shift focus.
+    Tips:
+    - Step 1: If submission format has issues, prioritize fixing them before proceeding.
+    - Step 2: If evaluation alignment issues are identified (validation approach does not follow competition requirements), address these methodological discrepancies immediately.
+    - Step 3: If new results significantly worse than SOTA, or repeated hyperparameter adjustments yield no improvement, it might be time to rethink or shift focus.
@@ -138,9 +138,9 @@ task_gen:

    ## Task Design Guidelines
    The task should be concise with several steps each only in a few sentences. 
-    DON'T repeat the details which has already included in the SOTA code. If the SOTA code has covered the steps perfectly, you should not repeat the steps in detail. 
-    You SHOULD NOT write any code in the task description.
-
+    DO NOT repeat the details which has already included in the SOTA code. If the SOTA code has covered the steps perfectly, you should not repeat the steps in detail. 
+    DO NOT write any code in the task description!
+    Observing reasons from failed experiments and feedback to prevent repeating similar mistakes in analogous situations.

    ## [Partial Response Format 1] Task Output Format:
    {{ task_output_format }}
@@ -168,6 +168,9 @@ task_gen:
    # Proposed Hypothesis you should strictly follow:
    {{ hypothesis }}

+    # Feedback from Previous Failed Experiments (e.g., experiments that did not pass evaluation, encountered bugs, or failed to surpass SOTA performance):
+    {{ failed_exp_and_feedback_list_desc }}
+
 specification:
  problem: |-
    1. The problem should be specific and fine-grained. Avoid general or vague statements. 
@@ -342,6 +342,7 @@ class DSProposalV2ExpGen(ExpGen):
        sota_exp: DSExperiment,
        hypothesis: DSHypothesis,
        pipeline: bool,
+        failed_exp_feedback_list_desc: str,
    ) -> DSExperiment:
        if pipeline:
            component_info = COMPONENT_TASK_MAPPING["Pipeline"]
@@ -360,8 +361,12 @@ class DSProposalV2ExpGen(ExpGen):
            component_desc=component_desc,
            workflow_check=not pipeline and hypothesis.component != "Workflow",
        )
+
        user_prompt = T(".prompts_v2:task_gen.user").r(
-            scenario_desc=scenario_desc, sota_exp_desc=sota_exp_desc, hypothesis=str(hypothesis)
+            scenario_desc=scenario_desc,
+            sota_exp_desc=sota_exp_desc,
+            hypothesis=str(hypothesis),
+            failed_exp_and_feedback_list_desc=failed_exp_feedback_list_desc,
        )
        response = APIBackend().build_messages_and_create_chat_completion(
            user_prompt=user_prompt,
@@ -422,6 +427,10 @@ class DSProposalV2ExpGen(ExpGen):
            exp_and_feedback_list=trace.experiment_and_feedback_list_after_init(return_type="all"),
            type="all",
        )
+        failed_exp_feedback_list_desc = T("scenarios.data_science.share:describe.trace").r(
+            exp_and_feedback_list=trace.experiment_and_feedback_list_after_init(return_type="failed"),
+            type="failed",
+        )

        # Step 1: Identify problems
        scen_problems = self.identify_scenario_problem(
@@ -476,4 +485,5 @@ class DSProposalV2ExpGen(ExpGen):
            sota_exp=sota_exp,
            hypothesis=new_hypothesis,
            pipeline=pipeline,
+            failed_exp_feedback_list_desc=failed_exp_feedback_list_desc,
        )
@@ -35,8 +35,6 @@ describe: # some template to describe some object
    {% if exp_and_feedback[0].hypothesis %}
    the experiment is designed based on hypothesis: {{ exp_and_feedback[0].hypothesis }}
    {% endif %}
-    ### Task of previous trial
-    {{ exp_and_feedback[0].pending_tasks_list[0][0].get_task_information() }}
    feedback decision: {{ exp_and_feedback[1].decision }}
    reason: {{ exp_and_feedback[1].reason }}
    {% endif %}