gavindiaz/NexQuant
1
0
Fork 0
mirror of https://github.com/NicolasBohn/NexQuant.git synced 2026-07-27 23:47:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: improve eval alignment check (e.g. small-scale finetuning) (#802)

Browse Source 
* fix

* fix
This commit is contained in:
Yuante Li
2025-04-22 18:47:24 +08:00
committed by GitHub
parent 0f2399667b
commit 39e25d28a1
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rdagent/scenarios/data_science/dev/prompts.yaml
+3 -2
View File
@@ -18,13 +18,14 @@ exp_feedback:
- If submission passes, proceed to Step 2.
Step 2: Evaluate Alignment with Competition Requirements (if format correct)
- GOAL: TRY YOUR BEST TO AVOID SIGNIFICANT DISCREPANCIES BETWEEN VALIDATION AND TEST SET SCORES.
- GOAL: CAREFULLY ANALYZE WHETHER THE EXPERIMENTAL SETUP AND CODE MAY CAUSE MISALIGNMENT BETWEEN VALIDATION AND TEST PERFORMANCE.
- Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
- Exact match between validation metric and official Kaggle metric.
- Consistent prediction methodologies between validation and test datasets.
- No shortcuts or fold-specific strategies applied inconsistently.
- Rigorous checks for corner-case consistency.
- If discrepancies are identified:
- Additionally, detect whether the setup introduces structural risks, such as overfitting-prone finetuning strategies or domain adaptation on insufficient data.
- If such discrepancies or risks are found:
- Clearly document these issues in `Reasoning`.
- Set `"Evaluation Aligned With Task": "no"` and `"Replace Best Result": "no"`.
- Begin your `reasoning` with `[Evaluation error]`, explicitly stating the evaluation alignment issues causing experiment failure.