mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 23:47:46 +00:00
Xu Yang
401 lines
20 KiB
YAML
401 lines
20 KiB
YAML
spec:
|
|
system: |-
|
|
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
|
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
|
|
|
Currently, you are working on a Kaggle competition project.
|
|
This project involves analyzing data and building models to beat other competitors, with the code being generated by large language models.
|
|
|
|
The runtime environment you are working in includes the following libraries and their respective versions:
|
|
{{ runtime_environment }}
|
|
|
|
Your overall task is provided below:
|
|
{{ task_desc }}
|
|
|
|
Your task is to write five specification texts (in markdown format) for the following tasks, based on the competition information provided
|
|
- Data loading (and preprocessing)
|
|
- Feature Engineering
|
|
- Model Building
|
|
- Ensemble
|
|
- The overall workflow
|
|
|
|
The specifications for each step should be tailored to the competition information provided.
|
|
|
|
Your specification should consists two parts:
|
|
1. The function definition in code format with detailed annotation to each parameter and return value.
|
|
2. A detailed docstring to the function that explains the purpose of the function, the input parameters, and the output.
|
|
3. Additional information or notes that the coder should consider while implementing the function.
|
|
Your specifications should not include any code implementation, only the function definition and docstring.
|
|
|
|
-----------Competition Information-----------
|
|
{{ competition_info }}
|
|
|
|
-----------Folder Description---------(All path are relative to the data folder)
|
|
- Ensure that all columns in sample_submission can be generated.
|
|
{{ folder_spec }}
|
|
|
|
user:
|
|
data_loader: |-
|
|
Data loader specification text should follow these detailed requirements:
|
|
1. Function Interface:
|
|
- Function Name: `load_data`
|
|
- Input: No input arguments.
|
|
- Output:
|
|
- `X` (DT, define based on competition information): Feature matrix for training data.
|
|
- `y` (DT): Target vector for training data.
|
|
- `X_test` (DT): Feature matrix for test data.
|
|
- `test_ids` (DT): Identifiers for the test data.
|
|
- Docstring Requirements:
|
|
- Describe the purpose of the function.
|
|
- Specify the data source location (`/kaggle/input/`).
|
|
- Clearly define the structure and type of the output.
|
|
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
|
|
|
2. Precautions for Data Loading and Preprocessing:
|
|
- File Handling:
|
|
- Ensure proper file encoding (e.g., UTF-8) and delimiters (e.g., CSV comma-separated).
|
|
- Combine or process multiple files if necessary.
|
|
- Data Preprocessing:
|
|
- Convert data types correctly (e.g., numeric, categorical, date parsing).
|
|
- Handle missing values appropriately (e.g., impute, drop rows/columns).
|
|
- Optimize memory usage for large datasets using techniques like downcasting or reading data in chunks if necessary.
|
|
- Domain-Specific Handling:
|
|
- Apply competition-specific preprocessing steps as needed (e.g., text tokenization, image resizing).
|
|
|
|
3. Code Standards:
|
|
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
|
|
|
4. Notes:
|
|
- Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
|
|
- Extend domain-specific handling steps based on the competition information.
|
|
- Never use sample submission as the test index, as it may not be the same as the test data. Use the test index file or test data source to get the test index.
|
|
|
|
{% if latest_spec %}
|
|
5. Former Specification:
|
|
{{ latest_spec }}
|
|
You should follow the provided specifications to improve this task.
|
|
{% endif %}
|
|
|
|
|
|
|
|
Please respond with a JSON structure as follows:
|
|
{
|
|
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
|
}
|
|
|
|
feature: |-
|
|
Feature engineering specification text should adhere to the following requirements:
|
|
1. Function Interface:
|
|
- Function Name: `feat_eng`
|
|
- Parameters:
|
|
- `X` (DT): Train data to be transformed.
|
|
- `y` (DT): Train label data.
|
|
- `X_test` (DT): Test data.
|
|
- Output:
|
|
- `X_transformed` (DT): Transformed train data.
|
|
- `y_transformed` (DT): Transformed train label data.
|
|
- `X_test_transformed` (DT): Transformed test data.
|
|
- Docstring Requirements:
|
|
- Describe the purpose of the function.
|
|
- Clarify the input parameters and their data types.
|
|
- Define the structure and format of the output.
|
|
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
|
|
|
2. Precautions for Feature Engineering:
|
|
- Well handle the shape of the data
|
|
- The sample size of the train data and the test data should be the same in all scenarios.
|
|
- To most of the scenario, the input shape and the output shape should be exactly the same.
|
|
- To some tabular data, you may add or remove some columns so your inferred column number may be unsure.
|
|
- Integration with Model Pipeline
|
|
- If feature engineering is strictly part of the model pipeline, state explicitly that it will be handled at the model stage.
|
|
- If integrated here, ensure this function applies all required transformations while avoiding data leakage.
|
|
- General Considerations:
|
|
- Ensure scalability for large datasets.
|
|
- Handle missing values and outliers appropriately (e.g., impute, remove, or replace).
|
|
- Ensure consistency between feature data types and transformations.
|
|
- Avoid data leakage: Only use features derived from training data, excluding information from test or validation sets.
|
|
- Domain-Specific Features:
|
|
- Apply logic for competition-specific features (e.g., text vectorization, image augmentations, categorical encoding).
|
|
|
|
3. Code Standards:
|
|
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
|
|
|
4. Notes:
|
|
- Align `DT` (data type) definitions with those in the Data Loader specification.
|
|
- Extend or adjust domain-specific transformations based on competition requirements.
|
|
- The device has GPU support, so you can use it for feature engineering if necessary to accelerate the process.
|
|
- Multi processing or parallel processing can be used to speed up the feature engineering process.
|
|
|
|
{% if latest_spec %}
|
|
5. Former Specification:
|
|
{{ latest_spec }}
|
|
You should follow the provided specifications to improve this task.
|
|
{% endif %}
|
|
|
|
Please respond with a JSON structure as follows:
|
|
{
|
|
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
|
}
|
|
|
|
model: |-
|
|
Model building specification text should adhere to the following requirements:
|
|
|
|
1. Function Interface:
|
|
- Function Name: `model_workflow`
|
|
- Parameters:
|
|
- `X` (DT): Training feature data.
|
|
- `y` (DT): Training label data.
|
|
- `val_X` (Optional[DT]): Validation feature data.
|
|
- `val_y` (Optional[DT]): Validation label data.
|
|
- `test_X` (Optional[DT]): Test feature data.
|
|
- `hyper_params` (dict): Dictionary of hyperparameters for model configuration.
|
|
- Output:
|
|
- `pred_val` (Optional[DT]): Predictions on validation data.
|
|
- `pred_test` (Optional[DT]): Predictions on test data.
|
|
- `hyper_params` (dict): Updated dictionary of hyperparameters after training.
|
|
- Docstring Requirements:
|
|
- Describe the purpose of the function.
|
|
- Clarify the input parameters and their data types.
|
|
- Define the structure and format of the output.
|
|
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
|
|
|
2. Code Standards:
|
|
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
|
|
|
3. Precautions:
|
|
- Ensure input arrays (`X`, `y`, `val_X`, `val_y`, `test_X`) have consistent dimensions and shapes.
|
|
- Use default values for hyperparameters if `hyper_params` is not provided.
|
|
- Train the model on `X` and `y`.
|
|
- Evaluate the model using `val_X` and `val_y` if validation data is available.
|
|
- If `test_X` is provided, generate predictions for it.
|
|
- Do not use progress bars (e.g., `tqdm`) in the implementation.
|
|
|
|
4. Notes:
|
|
- Align `DT` (data type) with the definitions used in Feature Engineering specifications.
|
|
- The device has GPU support, so you can use it for training if necessary to accelerate the process.
|
|
|
|
{% if latest_spec %}
|
|
5. Former Specification:
|
|
{{ latest_spec }}
|
|
You should follow the provided specifications to improve this task.
|
|
{% endif %}
|
|
|
|
Please respond in the following JSON format:
|
|
{
|
|
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
|
}
|
|
|
|
ensemble: |-
|
|
Ensemble specification text adhere to the following requirements:
|
|
1. Function Interface:
|
|
- Function Name: `ens_and_decision`
|
|
- Parameters:
|
|
- `test_preds_dict` (Dict[str, DT]): A dictionary of test predictions from different models. The key is the model file name.
|
|
- `val_preds_dict` (Dict[str, DT]): A dictionary of validation predictions from different models. The key is the model file name.
|
|
- `val_label` (DT): Validation label.
|
|
- Output:
|
|
- `final_pred` (DT): Ensemble prediction for the test data.
|
|
- Docstring Requirements:
|
|
- Describe the purpose of the function.
|
|
- Clarify the input parameters and their data types.
|
|
- Define the structure and format of the output.
|
|
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
|
|
|
2. Precautions:
|
|
- Validation of Inputs:
|
|
- Ensure all predictions in `test_preds_dict` and `val_preds_dict` have consistent shapes and dimensions.
|
|
- Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions.
|
|
- Handle empty or invalid inputs gracefully with appropriate error messages.
|
|
- You should calculate the metric for each model and ensemble strategy, and save the results in `scores.csv`, e.g.:
|
|
```python
|
|
scores = {}
|
|
for model_name, val_pred in val_preds_dict.items():
|
|
scores[model_name] = calculate_metric(val_label, val_pred)
|
|
|
|
... some code about ensemble strategy ...
|
|
ensemble_score = calculate_metric(val_label, ensemble_pred)
|
|
scores[<ensemble_strategy_name>] = ensemble_score
|
|
|
|
scores_df = pd.DataFrame(scores.items(), columns=['Model', <metric_name>])
|
|
scores_df.to_csv("scores.csv", index=False)
|
|
```
|
|
- Consensus Strategy:
|
|
- Clearly define how the ensemble predictions are aggregated (e.g., majority voting, weighted average).
|
|
- Avoid introducing biases or overfitting during decision-making.
|
|
|
|
3. Code Standards:
|
|
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
|
|
|
4. Notes:
|
|
- Align `DT` (data type) definitions with those used in model specifications.
|
|
- Ensure flexibility to handle multiple ensemble strategies based on competition requirements.
|
|
|
|
{% if latest_spec %}
|
|
5. Former Specification:
|
|
{{ latest_spec }}
|
|
You should follow the provided specifications to improve this task.
|
|
{% endif %}
|
|
|
|
Please respond in the following JSON format:
|
|
{
|
|
"spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
|
|
}
|
|
|
|
workflow: |-
|
|
Your task is to implement the main workflow script (`main.py`) for a Kaggle-style machine learning competition project.
|
|
Follow the provided project structure and specifications to ensure consistency and maintainability:
|
|
1. Workflow Integration:
|
|
- Integrate the following components into the workflow:
|
|
- Data loading (`load_data.py`).
|
|
- Feature engineering (`feature.py`).
|
|
- Model workflow for training and testing (`model_*.py`).
|
|
- Ensemble and decision-making (`ensemble.py`).
|
|
- Treat each component as a modular and callable Python function.
|
|
2. Feature Engineering
|
|
- The feature engineering should be called only once. For example:
|
|
`X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)`
|
|
- It should be called before dataset splitting.
|
|
|
|
3. Dataset Splitting
|
|
- The dataset returned by `load_data` is not split into training and testing sets, so the dataset splitting should happen after calling `feat_eng`.
|
|
- By default, split the dataset into 80% for training and 20% for testing.
|
|
- You can also use cross-validation or other splitting methods as you deem more useful and appropriate based on the Competition Information.
|
|
|
|
4. Submission File:
|
|
- Save the final predictions as `submission.csv` in the format required by the competition.
|
|
- Present the required submission format explicitly and ensure the output adheres to it.
|
|
|
|
5. Code Standards:
|
|
- Use consistent naming conventions and type annotations.
|
|
- Document the workflow with clear comments and docstring.
|
|
- Do not use progress bars (e.g., tqdm) in the code.
|
|
|
|
6. Ensemble Strategy:
|
|
Put all the model's return into a dict, using the model file name as key, and the return as value.
|
|
Sample code:
|
|
{% raw %}
|
|
{% for mn in model_names %}
|
|
from {{mn}} import model_workflow as {{mn}}_workflow
|
|
val_preds_dict["{{mn}}"], test_preds_dict["{{mn}}"], _ = {{mn}}_workflow(
|
|
X=train_X,
|
|
y=train_y,
|
|
val_X=val_X,
|
|
val_y=val_y,
|
|
test_X=test_X
|
|
)
|
|
{% endfor %}
|
|
final_pred = ens_and_decision(test_preds_dict, val_preds_dict, val_y)
|
|
{% endraw %}
|
|
|
|
{% if latest_spec %}
|
|
5. Former Specification:
|
|
{{ latest_spec }}
|
|
You should follow the provided specifications to improve this task.
|
|
{% endif %}
|
|
|
|
Please response the specification in the following json format. Here is an example structure for the JSON output:
|
|
{
|
|
"spec": "The corresponding specification string as described above. You should create the rules based on the competition information instead of copying the requirements."
|
|
}
|
|
|
|
data_loader_coder:
|
|
system: |-
|
|
You are a Python data scientist working on a new project. This project will be used to analyze data and build models to predict future outcomes, and this project codes will be written by GPT.
|
|
Your task is described below:
|
|
{{ task_desc }}
|
|
You should follow the provided specifications to complete this task.
|
|
You need to write the corresponding data loading code based on the information provided in the user's Data Folder Description, rather than relying on any suggestions that might exist in the spec.
|
|
|
|
Notice, the data files are stored in the data folder located at `/kaggle/input/`, and the data folder is structured as described in the Data Folder Description. Please don't load the data from the current directory.
|
|
|
|
Please response the code in the following json format. Here is an example structure for the JSON output:
|
|
{
|
|
"code": "The Python code as a string."
|
|
}
|
|
|
|
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
|
-----------Here is the relevant information for this task-----------
|
|
{% endif %}
|
|
{% if queried_similar_successful_knowledge|length != 0 %}
|
|
--------------Successful Implementations for Similar Models:--------------
|
|
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{loop.index}}:=====
|
|
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
|
=====Code:=====
|
|
{{ similar_successful_knowledge.implementation.all_codes }}
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
{% if queried_former_failed_knowledge|length != 0 %}
|
|
--------------Previous Failed Attempts:--------------
|
|
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
|
=====Code:=====
|
|
{{ former_failed_knowledge.implementation.all_codes }}
|
|
=====Feedback:=====
|
|
{{ former_failed_knowledge.feedback }}
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
user: |-
|
|
---------Competition Information---------
|
|
{{ competition_info }}
|
|
|
|
---------Data Loader Specification---------
|
|
{{ data_loader_spec }}
|
|
|
|
---------Data Folder Description---------(All path are relative to the data folder)
|
|
{{ folder_spec }}
|
|
|
|
{% if latest_code %}
|
|
---------Former code---------
|
|
{{ latest_code }}
|
|
{% if latest_code_feedback is not none %}
|
|
---------Feedback to former code---------
|
|
{{ latest_code_feedback }}
|
|
{% endif %}
|
|
The former code has some errors, you should write the correct code based on the former code. Avoid writing the same code to former code.
|
|
{% endif %}
|
|
|
|
You should strictly follow the function interface specifications provided by the specification to implement the function.
|
|
|
|
|
|
data_loader_eval:
|
|
system: |-
|
|
You are data scientist writing some data loader code for a Kaggle-style machine learning competition project.
|
|
The main code generation task is as follows:
|
|
{{task_desc}}
|
|
|
|
The data loader code is in a file named "load_data.py":
|
|
```python
|
|
{{code}}
|
|
```
|
|
|
|
You are testing the data_loader with the following code
|
|
```python
|
|
{{test_code}}
|
|
```
|
|
|
|
{% if workflow_stdout is not none %}
|
|
Your feature engineering code is also part of the whole workflow, the user also tested the whole workflow and provided you the stdout.
|
|
The whole workflow code is:
|
|
{{workflow_code}}
|
|
Please consider both stdout and approve the code when both the feature engineering test and the whole workflow test pass.
|
|
{% endif %}
|
|
|
|
You'll be given the stdout of your testing scripts.
|
|
Please respond with your feedback in the following JSON format and order
|
|
```json
|
|
{
|
|
"execution": "Describe how well the data loader executed, including any errors or issues encountered. Please keep the error message and tracking information",
|
|
"return_checking": "Detail the checks performed on the data loaded, including data integrity and correctness.",
|
|
"code": "Provide feedback on the code quality, readability, and adherence to specifications. Please also consider the efficiency of the code based on whether it uses multi-threading or GPUs to speed up the process.",
|
|
"final_decision": <true/false>
|
|
}
|
|
```
|
|
user: |-
|
|
Data loader test stdout:
|
|
{{stdout}}
|
|
{% if workflow_stdout is not none %}
|
|
Whole workflow test stdout:
|
|
{{workflow_stdout}}
|
|
{% endif %}
|