mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-08-03 18:27:43 +00:00
you-n-g
b662f45021
* feat: add model dump flag and multi-evaluator support * tmp code * refactor: update evaluator feedback and FBWorkspace types * feat: add get_clear_ws_cmd and CPU count in Docker environment * feat: Add model dump check level and enhance evaluator functionality fix data type bug * fix: Ensure required files exist before model dump evaluation * refactor: streamline prompt and file checks in model dump evaluation * fix: add assertions and reorder file reads in model dump evaluator * feat: remove EDA part from evaluation output * docs: update dump_model guidelines and eval prompt to include template * style: reformat multiline dicts and lists in conf and eval files * fix: add DOTALL flag to EDA removal regex
55 lines
3.2 KiB
YAML
55 lines
3.2 KiB
YAML
dump_model_coder:
|
|
guideline: |-
|
|
Please dump the model in a "models/" subfolder in the first running, and the script rerun performs inference without needing to retrain the model.
|
|
If there are parameters generated from the training data that might be needed for inference on test data, please save them in the "models/" subfolder as well.
|
|
Make sure that the required files, like submission.csv and scores.csv, are created even if you bypass the model training step by loading the saved model file directly.
|
|
|
|
dump_model_eval:
|
|
system: |-
|
|
You are a data scientist tasked with evaluating code generation. You've developed a Kaggle competition code that can produce a submission file.
|
|
The code should follow the guideline below:
|
|
{% include "components.coder.data_science.share.prompts:dump_model_coder.guideline" %}
|
|
|
|
You will receive the following information:
|
|
- The implemented code
|
|
- The stdout from running the code
|
|
- The file list in "models/" subfolder
|
|
- The scores.csv file generated during both training and inference (if it exists)
|
|
|
|
Focus on these aspects:
|
|
- Check if the code saves the model in the "models/" subfolder.
|
|
- Ensure that when the code is rerun, it skips the training process and loads the model from the "models/" subfolder for direct inference.
|
|
- Verify that there is no training activity in the output.
|
|
- Ensure that even if you skip the model training by loading saved models, the files like scores.csv and submission.csv are still correctly created.
|
|
- The model's performance should remain consistent and not vary unreasonably between training and inference.
|
|
|
|
Please respond with your feedback in the following JSON format and order
|
|
```json
|
|
{
|
|
"execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. Carefully check the stdout to ensure that when the code is rerun, it skips the training process and loads the model from the 'models/' subfolder for direct inference. Append the information that makes you think that the model is still being retrained when rerunning the code."
|
|
"return_checking": "Verify the generated files include necessary files. Make sure scores.csv file does not change unreasonably between training and inference",
|
|
"code": "The code has explicity dump the model into 'models/' subfolder; When the modes files are already in 'models/' subfolder, the code will explicity skip the training process.",
|
|
"final_decision": <true or false in boolean type; only return true when ensuring that the code saves the model in a 'models/' subfolder, and the script rerun performs inference without needing to retrain the model.>
|
|
}
|
|
```
|
|
|
|
user: |-
|
|
------------ The implemented code ------------
|
|
{{code}}
|
|
|
|
------------ The stdout from running the code ------------
|
|
{{stdout}}
|
|
|
|
------------ The file list in "models/" subfolder ------------
|
|
{% for f in model_folder_files %}
|
|
- {{ f }}
|
|
{% endfor %}
|
|
|
|
------------ The scores.csv file generated ------------
|
|
# Training:
|
|
{{scores_content_before}}
|
|
|
|
# Inference:
|
|
{{scores_content_after}}
|
|
|