gavindiaz/NexQuant
1
0
Fork 0
mirror of https://github.com/NicolasBohn/NexQuant.git synced 2026-07-27 23:47:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: sota comparison logic (#608)

Browse Source 
* concat sota and current result

* fix the bug in current_exp's score vs sota's score

* fix ci

---------

Co-authored-by: yuanteli <1957922024@qq.com>
This commit is contained in:
Haoran Pan
2025-02-18 20:43:28 +08:00
committed by GitHub
parent bf6eb6f15d
commit 45f022cd32
2 changed files with 18 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rdagent/scenarios/data_science/dev/feedback.py
+11
View File
@@ -1,5 +1,7 @@
import json
import pandas as pd
from rdagent.components.knowledge_management.graph import UndirectedNode
from rdagent.core.experiment import Experiment
from rdagent.core.prompts import Prompts
@@ -51,6 +53,14 @@ class DSExperiment2Feedback(Experiment2Feedback):
# assumption:
# The feedback should focus on experiment **improving**.
# Assume that all the the sota exp is based on the previous sota experiment
cur_vs_sota_score = None
if sota_exp:
cur_score = pd.DataFrame(exp.result).loc["ensemble"].iloc[0]
sota_score = pd.DataFrame(sota_exp.result).loc["ensemble"].iloc[0]
cur_vs_sota_score = (
f"The current score is {cur_score}, while the SOTA score is {sota_score}. "
f"{'In this competition, higher is better.' if self.scen.metric_direction else 'In this competition, lower is better.'}"
)
system_prompt = T(".prompts:exp_feedback.system").r(scenario=self.scen.get_scenario_all_desc())
user_prompt = T(".prompts:exp_feedback.user").r(
@@ -58,6 +68,7 @@ class DSExperiment2Feedback(Experiment2Feedback):
cur_exp=exp,
diff_edition=diff_edition,
feedback_desc=feedback_desc,
cur_vs_sota_score=cur_vs_sota_score,
)
resp_dict = json.loads(
rdagent/scenarios/data_science/dev/prompts.yaml
+7 -2
View File
@@ -51,10 +51,15 @@ exp_feedback:
{% endfor %}
### Final results of the current solution
1. Pay close attention to the performance of ensemble, as it represents the final score for this iteration.
2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. If the final ensemble score surpasses the current SOTA, you may update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
1. Pay close attention to the performance of `ensemble`, as it represents the final score for this iteration.
2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. But if the final `ensemble` score surpasses the current SOTA, you should update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
Below are the results for this experiment:
{{ cur_exp.result }}
{% if cur_vs_sota_score is not none %}
Below is the comparison of the current `ensemble` performance with the SOTA result:
{{ cur_vs_sota_score }}
{% endif %}
{% if cur_exp.format_check_result is not none %}
### Submission format check to current solution: