scenario_problem:
  system: |-
    {% include "scenarios.data_science.share:scen.role" %}
    The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace. If new trace surpasses the current SOTA, it will be the new SOTA. If not, it will be a failed experiment.
    You will be provide with: 
      1. A detailed competition scenario description;
      2. Previous SOTA experiments and feedbacks, which are past SOTA experiments indexed from oldest to newest;
      3. Previous failed experiments and feedbacks, which are ordered attempts that did not surpass the current SOTA implementation;
    Your task is to analyze the given information and extract the **Scenario Problems** from the given materials.

    ## Scenario Problems
    ### Definition
    Scenario problems are specific, context-dependent challenges arising from a competition's dataset or domain. They fall into two categories:
    1. Dataset Characteristics: Inherent structural or statistical properties of the dataset (such as imbalance, high dimensionality, collinearity, outliers, missing data, skewed distribution, time-based patterns, etc.).
    2. Domain-specific Insights: Actionable knowledge derived from expertise in the competition's domain, enabling correct interpretation of data patterns or constraints. These insights are not evident from the data alone and require external context to resolve ambiguities, engineer features, or avoid invalid assumptions.

    ### Specification
    {{ problem_spec }}
    
    ### Core Analysis Dimensions
    1. SOTA Mismatch Diagnosis: Systematically compare current implementations against both data properties and domain knowledge to identify critical discrepancies.
    2. Gap Forensic Analysis: Examine successful solutions to reveal unstated problems they implicitly address through workarounds.
    3. Domain-Implementation Conflict Detection: Identify instances where technical approaches violate domain constraints or oversimplify complex relationships.
    4. In case there is no SOTA implementation, your scenario problem should focus on the scenario itself.

    ### Output Format
    {{ problem_output_format }}

  user: |-
    # Scenario Description
    {{ scenario_desc }}

    # Current SOTA Implementation
    {{ sota_exp_desc }}

feedback_problem:
  system: |-
    {% include "scenarios.data_science.share:scen.role" %}
    The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace. If new trace surpasses the current SOTA, it will be the new SOTA. If not, it will be a failed experiment.
    You will be provided with: 
      1. A detailed competition scenario description;
      2. Previous SOTA experiments and feedbacks, which are past SOTA experiments indexed from oldest to newest;
      3. Previous failed experiments and feedbacks, which are ordered attempts that did not surpass the current SOTA implementation;
      4. The current SOTA implementation and feedback, which is the latest SOTA experiments from the previous experiments;
    Your task is to analyze the given information and extract the **Feedback Problems** from the previous experiments or the current SOTA implementation.

    
    {% if inject_diverse %}
    ### Focus on Diversity!!
    Diversity is very critical in the analysis of scenario problems. You should closely check the history of previous experiments and feedbacks, and try to explore the problems/hypotheses that are not covered by the previous experiments.
    1. Check the previous experiments and feedbacks to find the problems that are not covered by the previous experiments.
    2. Check the current SOTA implementation and feedback to find the problems that are not covered by the current SOTA implementation.
    3. Do not do incremental exploration on the previous problems.
    {% endif %}

    ## Feedback Problems
    ### Definition
    Feedback problems are specific and fine-grained technical, or methodological issues within the previous experiments or the current SOTA implementation.

    ### Guidelines
    Here are few guidelines to help you identify the feedback problems:
    1. Feedback Analysis
      - Extract explicit issues directly stated in the feedback.
      - Infer implicit issues from feedback context.
    2. Code Review
      - Feature Engineering. Check for missing, redundant, or improper features and the mismatch between features and models.
      - Model Architecture. Assess the compatibility between the model type and the problem domain. Verify model architecture and hyperparameters.
      - Ensemble. Check if ensemble methods are optimized. Identify underperforming base models to remove from the ensemble.
      - Training. Validate hyperparameter (e.g., learning rate, batch size), loss functions, and regularization. Determine if hyperparameters are optimized based on prior experiment traces.
    3. Trace History Analysis
      - Flag unresolved and persistent issues recurring across traces.
      - Highlight partial fixes (e.g., inappropriate feature engineering).
      - Identify unexplored directions (e.g. new features, new model structures) from prior traces.
      - Identify potential unsolved time/memory constraints from previous experiments trace.
    
    ### Specification
    {{ problem_spec }}

    ### Output Format
    {{ problem_output_format }}

  user: |-
    # Scenario Description
    {{ scenario_desc }}
    
    # Previous Experiments and Feedbacks
    {{ exp_and_feedback_list_desc }}    

    # Current SOTA Implementation
    {{ sota_exp_desc }}

hypothesis_gen:
  system: |-
    {% include "scenarios.data_science.share:scen.role" %}
    The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace. If new trace surpasses the current SOTA, it will be the new SOTA. If not, it will be a failed experiment.
    You will be provided with: 
      1. A detailed competition scenario description;
      2. Previous SOTA experiments and feedbacks, which are past SOTA experiments indexed from oldest to newest;
      3. Previous failed experiments and feedbacks, which are ordered attempts that did not surpass the current SOTA implementation;
      4. The current SOTA implementation and feedback, which is the latest SOTA experiments from the previous experiments;
      5. A list of identified problems, which are specific technical or methodological issues within the previous experiments;
    Your task is to:
      1. **Hypothesis Proposal**: Propose testable hypotheses to address the identified problems.
      2. **Hypothesis Evaluation**: Evaluate the proposed hypotheses across multiple dimensions.

    {% if enable_idea_pool %}
    In order to assist you in the hypothesis proposal, the user has sampled a list of ideas for each of the identified problems.
    The ideas are extracted methods or techniques from previous SOTA implementations of other competitions.
    These ideas can potentially tackle the identified problems and improve the current SOTA implementation but you should decide whether to use them or not.
    To specific problem, if you choose to use the given idea, you should modify it to a proper hypothesis and also mark the inspired flag as True.
    {% endif %}

    # Task 1: Hypothesis Proposal
    For each identified problem, propose a hypothesis to improve the current SOTA implementation.

    ## Hypothesis Guidelines
    Here are few guidelines to help you formulate hypotheses:
    1. Problem Impact Analysis
      - Quantify how the problem degrades performance.
    2. Previous Experiments Analysis
      - For previous SOTA experiments, analyze insights and implicit patterns that can be leveraged to improve the current SOTA implementation.
      - For failed experiments, think about the persistent problems they facing. If these experiments consistently failed due to time/memory constraints, prioritize changes on efficiency.
    3. Actionable Changes
      - If the problem relates to time/memory constraints, consider smaller model sizes or alternative algorithms with reduced complexity.
      - If the problem involves underperforming models, propose removing or replacing models of significantly worse performance.
      - If the problem relates to hyperparameter tuning, recommend a specific method or strategy for tuning.
    4. Note on Time/Memory Constraints
      - If prior experiments failed due to time/memory limitations, assume your new hypothesis will face the same constraints. In this case, prioritize efficiency and **ONLY** response to the problems related to time/memory constraints in your response dictionary.
      - Besides, do not compromise performance merely for efficiency since the current SOTA implementation do not encounter the constraints. You should think about how to balance the efficiency and performance so that your new hypothesis can be executed successfully and achieve satisfactory performance. 
    5. Note on Drafting the First Implementation
      - In case there is no SOTA implementation, you should draft the first implementation. In this case, your hypothesis should not only cover the problem but also on the overall design such as how to do the feature engineering and the model selection.
    {% if enable_idea_pool %}
    6. Idea Reference
      - Each idea is a method, technique or trick that contributes to high performance from other competition implementation under similar problem. You are free to use them as an inspiration for your hypothesis proposal.
    {% endif %}

    ## Hypothesis Specification
    {{ hypothesis_spec }}


    # Task 2: Hypothesis Evaluation
    After proposing the hypothesis, your second task is to evaluate the hypothesis from multiple dimensions.

    ## Evaluation Instruction
    Firstly, you should tag the hypothesis with one of the following components. If the hypothesis is related to multiple components, you should choose the most relevant one.
    {{ component_desc }}

    Secondly, please score the proposed hypothesis from 1 to 10 for each of the following dimensions (where 1 means lowest and 10 means highest):
    1. Problem-Hypothesis Alignment: How well the hypothesis addresses the identified problem.
    2. Expected Impact: The estimated improvement after applying the hypothesis to current SOTA implementation.
    3. Novelty: Degree of innovation compared to previous attempts. If the proposed hypothesis is similar to previous experiments' hypothesis, assign novelty score to one.
    4. Feasibility: The ease of implementing the proposed hypothesis in the current SOTA implementation.
    5. Risk-Reward Balance: The exploration-exploitation balance of the proposed hypothesis.

    {% if inject_diverse %}
    # Focus on Diversity!!
    Diversity is very critical in the analysis of scenario problems. You should closely check the history of previous experiments and feedbacks, and try to explore the problems/hypotheses that are not covered by the previous experiments.
    1. Check the previous experiments and feedbacks to find the problems that are not covered by the previous experiments.
    2. Check the current SOTA implementation and feedback to find the problems that are not covered by the current SOTA implementation.
    3. Think out of the box and explore the hypothesis that are not covered by the previous experiments and feedbacks, but are reasonable and aligned with the identified problems. 
    4. Do not do incremental exploration on the previous problems, like lightgbm -> xgboost, or 1dCNN -> 2dCNN. Totally different hypothesis on model\data\feature\ensemble\workflow level are welcomed.
    {% endif %}


    ## Final Output Format in JSON Schema:
    {{ hypothesis_output_format }}
    
  user: |-
    # Scenario Description
    {{ scenario_desc }}

    # Previous Experiments and Feedbacks
    {{ exp_and_feedback_list_desc }}

    # Current SOTA Implementation
    {{ sota_exp_desc }}

    # Identified Problems{% if enable_idea_pool %} with Sampled Ideas{% endif %}
    {{ problems }}

task_gen:
  system: |-
    {% include "scenarios.data_science.share:scen.role" %}
    The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace, not necessarily the immediate predecessor.
    You will be provided with: 
      1. A detailed competition scenario description;
      2. Previous SOTA experiments and feedbacks, which are past SOTA experiments indexed from oldest to newest;
      3. Previous failed experiments and feedbacks, which are ordered attempts that did not surpass the current SOTA implementation;
      4. The current SOTA implementation and feedback, which is the latest SOTA experiments from the previous experiments;
      5. A proposed hypothesis to improve the current SOTA implementation;

    # Step 1: Task Design
    Your first task is to generate new {{ targets }} based on the proposed hypothesis. Your task should very detailed with specific steps and instructions. The task should be specific and fine-grained, avoiding general or vague statements.

    ## Specification
    {{ task_specification }}

    ## Task Design Guidelines
    1. The task should be concise with several steps each only in a few sentences. 
    2. DO NOT repeat the details which has already included in the SOTA code. If the SOTA code has covered the steps perfectly, you should not repeat the steps in detail. 
    3. DO NOT write any code in the task description!
    4. Observe reasons from failed experiments and feedback to prevent repeating similar mistakes in analogous situations.
    5. Specific and Non-Vague
      - Avoid vague statements like "choose a proper model" Instead, specify the exact task to be made.
      - No phrases like "for example" or "eg.," should be used in the task. Give a clear decision in the task.
    6. Resource limitations
      - The user will give you some failed experiments and feedbacks. If the former experiments faced time/memory constraints, it means it's very likely that your generated task will also face the same constraints. In this case, you should design a task that prioritize efficiency in terms of time and space complexity.
      - If you plan to prioritize efficiency, you can modify the parts which is not related to the hypothesis. Which means your task should still able to validate the hypothesis.
      - Add [EFFICIENCY AS PRIORITY] tag in the task description to indicate that the task takes efficiency as a priority.
      - Although the task should prioritize efficiency, it should not be the only focus. The task should also be aligned with the proposed hypothesis and the current SOTA implementation.

    ## [Partial Response Format 1] Task Output Format:
    {{ task_output_format }}

    {% if workflow_check %}
    # Step 2: Workflow Update
    Since components have dependencies, your second task is to update the workflow to reflect the changes made to the target component. Please also decide whether the workflow needs to be updated and provide a brief description of the change task.
    {{ component_desc }}
    [Partial Response Format 2] Your generated workflow description should be a simple text and the following agent will do the implementation. If you think the workflow should not be updated, just respond with "No update needed".
    {% endif %}

    Your final output should strictly adhere to the following JSON format. 
    {
      "task_design": ---The dict corresponding to task output format---,
      {% if workflow_check %}"workflow_update": ---A string corresponding to workflow description--- {% endif %}
    }
    
  user: |-
    # Scenario Description
    {{ scenario_desc }}

    # Current SOTA Implementation
    {{ sota_exp_desc }}

    # Proposed Hypothesis you should strictly follow:
    {{ hypothesis }}

    # Feedback from Previous Failed Experiments (e.g., experiments that did not pass evaluation, encountered bugs, or failed to surpass SOTA performance):
    {{ failed_exp_and_feedback_list_desc }}

idea_sample:
  system: |-
    You are a Kaggle Grandmaster and expert ML engineer with deep expertise in statistics, machine learning, and competition optimization.
    The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace, not necessarily the immediate predecessor.
    You will be given a competition scenario, previous SOTA and failed experiments and feedbacks, and the current SOTA implementation and feedback.
    The user has identified potential problems in the current SOTA implementation and sampled few ideas for possible improvement direction for each of the problem.
    Your task is to identify the most useful and potential idea for each of the problem according to the impact, alignment, and novelty of the ideas.

    The user provided ideas might not be the suitable solution for the identified problems. If all ideas to one problem are not useful, please ignore this problem in your response dict.

    ### Specification
    {{ idea_spec }}

    ### Output Format
    {{ idea_output_format }}

  user: |-
    # Scenario Description
    {{ scenario_desc }}
    
    # Previous Experiments and Feedbacks
    {{ exp_feedback_list_desc }}    

    # Current SOTA Implementation
    {{ sota_exp_desc }}

    # Problem-Ideas Pairs
    {{ problem_ideas }}

specification:
  problem: |-
    1. The problem should be specific and fine-grained. Avoid general or vague statements. 
    2. The problem should technical or methodological. Focus on design and implementation flaws, not runtime errors.
    3. The problem should be strictly aligned with the improvement of target metric. The problem should fit the template: "IF THE PROBLEM IS SOLVED, THEN THE TARGET METRIC WILL IMPROVE."
  
  hypothesis: |-
    1. Each hypothesis should be specific and non-vague.
      - Avoid vague statements like "improve the model" or "optimize the pipeline." Instead, specify the exact changes to be made. Do not use ambiguous changes like "try method A or method B". 
      - No phrases like "for example" or "eg.," should be used in the hypothesis. Give a clear decision in the hypothesis.
    2. Each hypothesis should be testable and actionable. It should clearly state the expected change or improvement in the component's performance. For example, "tuning a model" is too broad, whereas "increasing the learning rate to 0.1 in the LightGBM model will improve performance" is testable and actionable.
    3. Each hypothesis should be aligned with the current SOTA implementation. It should be a potential solution to the identified problem.
    4. All the changes in the hypothesis should be correlated and relevant to each other. Avoid proposing multiple independent ideas in a single hypothesis.
    {% if not pipeline %}5. Each hypothesis should focus on a single direction per experiment. Avoid proposing multiple possibilities within the same hypothesis, such as "this may work in case A or case B." Research and development can be approached at different levels (shallow or deep), but each experimental loop should validate only one specific idea.
    6. Each hypothesis should focus on one component. The components will be described in the evaluation stage.
    {% else %}5. The hypothesis should focus on the whole pipeline. If needed, the hypothesis may propose changes across multiple parts in the SOTA implementation.
    {% endif %}

  idea: |-
    1. Alignment: The idea should be aligned with the identified problem. It should be a potential solution to the problem.
    2. Novelty: The idea should be novel and not previously explored in the current SOTA implementation. Avoid ideas that have already been tried and failed.
    3. Impact: The idea should have the potential to significantly improve the current SOTA implementation. It should be a promising direction for further exploration.
    4. You should identify the most useful and potential idea for each of the problem. If none of the provided ideas are useful, please ignore this problem in your response dict.

output_format:
  problem: |-
    For each of the identified problem, you should strictly adhere to the following JSON schema. 
    Your final output should be a dict containing all the identified problem without anything else.
    Please respond at most five problems FEWER BUT BETTER considering the most valuable and recently not explored. Don't respond problems not relevant to the improvement of target metric.
    {
      "problem name 1": {
        "problem": "Description of the first issue in no more than three sentences.",
        "reason": "Brief explanation of why this is a problem, based on the feedback or inferred from provided materials in no more than two sentences."
      },
      "problem name 2": {
        "problem": "Description of the second issue in no more than three sentences.",
        "reason": "Brief explanation of why this is a problem, based on the feedback or inferred from provided materials in no more than two sentences."
      }
    }
  hypothesis: |-
    For each of the identified problem, you should propose a hypothesis strictly following to the JSON schema. Your final output should be a dict containing all the proposed hypothesis.
    {
      "problem name 1 (should be exactly same as the problem name provided)": {
        {% if enable_idea_pool %}"inspired": "True or False. Set to True if the hypothesis is inspired by the user provided ideas. Otherwise, set it to False.",{% endif %}
        "reason": "Provide a clear, logical progression from problem identification to hypothesis formulation, grounded in evidence (e.g., trace history, domain principles, or competition constraints). Refer to the Hypothesis Guidelines for better understanding. Reason should be short with no more than two sentences.",
        "component": "The component tag of the hypothesis. Must be one of ('DataLoadSpec', 'FeatureEng', 'Model', 'Ensemble', 'Workflow').",
        "hypothesis": "A concise, testable statement derived from previous experimental outcomes. Limit it to one or two sentences that clearly specify the expected change or improvement in the <component>'s performance.",
        "evaluation": {
          "alignment_score": "The alignment of the proposed hypothesis with the identified problem.",
          "impact_score": "The expected impact of the proposed hypothesis on the current SOTA implementation.",
          "novelty_score": "The novelty of the proposed hypothesis compared to existing solutions.",
          "feasibility_score": "The feasibility of implementing the proposed hypothesis in the current SOTA implementation.",
          "risk_reward_balance_score": "The risk-reward balance of implementing the proposed hypothesis.",
        }
      },
    }
  idea: |-
    For each of the problems, you should identified the most useful and potential idea strictly following to the JSON schema.
    Your final output should be a dict containing the problems and corresponding identified ideas pairs without anything else.
    Please respond at most five problem-ideas pairs considering the most valuable and recently not explored.
    {
      "problem name 1 (should be exactly same as the problem name provided)": 1, # The index which is same to the idea index provided in the input and must be integer.
      "problem name 2 (should be exactly same as the problem name provided)": 2, # The index which is same to the idea index provided in the input and must be integer.
    }


