mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-08-06 11:37:44 +00:00
Tim
6f9843a7df
* custom data * fix: simplify competition check and log local description file * no sample data * feat: add test evaluation module with error handling support * fix: update eval path to use eval_sub_dir and add valid_check TODO * refactor: add MLETestEval check to conditionally run grading steps * avoid blank stdout * valid in testeval * rename test.csv to avoid conflict * Support Disabling sample submission * refactoring * fix: remove DS_KAGGLE_DATA and update prompt instructions * add try for grade * ignore submission * fix: remove tee from eval command and warn about pipeline exit code detection * optional to use raw description * support old data * add execution result to stdout * add metric to raw description * custom data explain * add debug_path * rst update --------- Co-authored-by: Young <afe.young@gmail.com>
82 lines
3.8 KiB
YAML
82 lines
3.8 KiB
YAML
DSCoSTEER_eval:
|
|
system: |-
|
|
You are a data scientist responsible for evaluating all the code.
|
|
|
|
## Task Description
|
|
The user is trying to build a data science solution in the following scenario:
|
|
{{ scenario }}
|
|
|
|
The task is as follows:
|
|
{{ task_desc }}
|
|
|
|
The whole workflow includes multiple stages, such as:
|
|
- Data loading
|
|
- Feature engineering
|
|
- Model training
|
|
- Ensembling
|
|
|
|
{% if is_sub_enabled %}
|
|
The user will provide you the whole code base, some logs generated during the execution of the whole workflow. Your evaluation scope includes whether the workflow code:
|
|
1. Executes successfully, correctly organizing components and generating a final submission.
|
|
2. Generates predictions in the correct format, ensuring they align with the **sample submission** structure!
|
|
|
|
|
|
Please respond with your feedback in the following JSON format and order
|
|
```json
|
|
{
|
|
"execution": "Describe whether the whole code base executed successfully and generating the final submission. Include any errors or issues encountered, and retain all error messages and traceback details.",
|
|
"return_checking": "Verify the generated files, particularly the submission file. Ensure that its format matches the sample submission",
|
|
"code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
|
|
"final_decision": <true/false>
|
|
}
|
|
```
|
|
{% else %}
|
|
The user will provide you the whole code base, some logs generated during the execution of the whole workflow. Your evaluation scope includes whether the workflow code executes successfully.
|
|
No need to check the detail of submission file.
|
|
|
|
Please respond with your feedback in the following JSON format and order
|
|
```json
|
|
{
|
|
"execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information.",
|
|
"return_checking": "Describe the expected file to be generated.",
|
|
"code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
|
|
"final_decision": <true/false>
|
|
}
|
|
```
|
|
{% endif %}
|
|
# NOTE: when is_sub_enabled == False, we don't have any checking about the return. So it is just placeholder currently
|
|
|
|
user: |-
|
|
--------- code base ---------
|
|
{{ code }}
|
|
--------- test stdout ---------
|
|
{{ stdout }}
|
|
|
|
DSCoSTEER_debugger:
|
|
system: |-
|
|
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
|
You have finished the implementation of the whole workflow which has executed well on a sampled dataset. However, the user has reported that the workflow failed to execute on the full dataset.
|
|
|
|
Your current job is to debug the whole code base, try to correct the errors, and ensure that the workflow can execute successfully on the full dataset.
|
|
The user will provide your the whole code base and some feedback generated during the execution of the whole workflow. Please identify the issues and provide the corrected code.
|
|
|
|
Task description:
|
|
{{ task_desc }}
|
|
|
|
Your modified code should follow the minimal changes principle. You should only modify the code that is necessary to fix the issues but not affect any other parts of the code. Try to correct as less files as possible since files are interdependent.
|
|
|
|
## Output Format
|
|
{% if out_spec %}
|
|
{{ out_spec }}
|
|
{% else %}
|
|
Please response the code in the following json format. Here is an example structure for the JSON output:
|
|
{
|
|
"code": "The Python code as a string."
|
|
}
|
|
{% endif %}
|
|
user: |-
|
|
--------- code base ---------
|
|
{{ code }}
|
|
--------- feedback ---------
|
|
{{ feedback }}
|