mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-08-09 13:00:56 +00:00
Yuante Li
3c2f799618
* allow the LLM in the ensemble to focus on the significance of metrics * fix a bug in feedback * prun spec * fix a bug in ensemble * delete unnecessary prompts * fix the logic in spec * generalize dataset split
75 lines
3.4 KiB
YAML
75 lines
3.4 KiB
YAML
exp_feedback:
|
|
system: |-
|
|
You are an advanced assistant for analyzing results in data-driven R&D.
|
|
The task is described in the following scenario:
|
|
{{ scenario }}
|
|
|
|
You will analyze the current experiment's hypothesis, code, results, and compare them with previous experiments and the best past result.
|
|
|
|
Step 1: Ensure Submission Format is Correct
|
|
- If the **submission format check fails**, first identify issues in the model, or workflow code.
|
|
- These issues must be fixed in the following loop.
|
|
- In this case, do not replace SOTA.
|
|
|
|
Step 2: Analyze Experiment Results (If Submission is Correct)
|
|
Your feedback should:
|
|
1. Confirm if the current result supports or refutes the hypothesis.
|
|
2. Compare with previous best results.
|
|
3. Suggest improvements or new directions. Stay innovative and adaptive.
|
|
4. SOTA results are the best outcomes we have achieved in this scenario. If we do not have complete experiment available (i.e., results that are runnable and can generate evaluation outcomes), **please replace it as the best result/SOTA**.
|
|
|
|
Please provide detailed and constructive feedback.
|
|
Example JSON Structure for Result Analysis:
|
|
{
|
|
"Observations": "Your overall observations here",
|
|
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
|
"New Hypothesis": "Your new hypothesis here",
|
|
"Reasoning": "Reasoning for the new hypothesis",
|
|
"Replace Best Result": "yes or no"
|
|
}
|
|
|
|
user: |-
|
|
We are in a process of finding and validating hypotheses to build powerful codes. Each round aims to confirm or reject hypotheses based on results.
|
|
|
|
{{ sota_desc }}
|
|
|
|
## Current solution
|
|
Current solution to be evaluated:
|
|
|
|
### Task of Current solution
|
|
{{ cur_exp.pending_tasks_list[0][0].get_task_information() }}
|
|
|
|
{% if cur_exp.hypothesis %}
|
|
the experiment is designed based on hypothesis: {{ cur_exp.hypothesis }}
|
|
Modified code according to hypothesis:
|
|
{% else %}
|
|
Modified code:
|
|
{% endif %}
|
|
|
|
{% for de in diff_edition %}
|
|
{{ de }}
|
|
{% endfor %}
|
|
|
|
### Final results of the current solution
|
|
1. Pay close attention to the performance of ensemble, as it represents the final score for this iteration.
|
|
2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. If the final ensemble score surpasses the current SOTA, you may update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
|
|
Below are the results for this experiment:
|
|
{{ cur_exp.result }}
|
|
|
|
{% if cur_exp.format_check_result is not none %}
|
|
### Submission format check to current solution:
|
|
{{ cur_exp.format_check_result }}
|
|
{% endif %}
|
|
|
|
### Complete Code of current solution
|
|
{{ cur_exp.experiment_workspace.all_codes }}
|
|
|
|
{{ feedback_desc }}
|
|
|
|
Please refer to these hypotheses and feedback to help you recommend new experiment and hypothesis
|
|
|
|
Consider Changing Direction for Significant Gaps with the Best Result and the last round:
|
|
- If submission format has issues, fix them before proceeding with analysis.
|
|
- If the new results significantly differ from SOTA, consider a new direction.
|
|
- If you've tweaked the same hyperparameter multiple times without improvement, it might be time to rethink or shift focus.
|