NexQuant/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml

spec:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    Currently, you are working on a Kaggle competition project. 
    This project involves analyzing data and building models to beat other competitors, with the code being generated by large language models.

    The runtime environment you are working in includes the following libraries and their respective versions:
    {{ runtime_environment }}

    Your overall task is provided below:
    {{ task_desc }}
    
    Your task is to write five specification texts (in markdown format) for the following tasks, based on the competition information provided
    - Data loading (and preprocessing)
    - Feature Engineering
    - Model Building
    - Ensemble
    - The overall workflow

    The specifications for each step should be tailored to the competition information provided. 
    
    Your specification should consists two parts:
    1. The function definition in code format with detailed annotation to each parameter and return value.
    2. A correct, concise, and complete docstring to the function that explains the purpose of the function, the input parameters, and the output.
    3. Additional information or notes that the coder should consider while implementing the function.
    Your specifications should not include any code implementation, only the function definition and docstring.

    ----------- Competition Information -----------
    {{ competition_info }}

    -----------Folder Description (All path are relative to the data folder) ---------
    - Ensure that all columns in sample_submission can be generated.
    {{ folder_spec }}

  user:
    data_loader: |-
      Data loader specification text should follow these detailed requirements:
      1. Function Interface:
        - Function Name: `load_data`
        - Input: No input arguments.
        - Output:
          - `X` (DT, define based on competition information): Feature matrix for training data.
          - `y` (DT): Target vector for training data.
          - `X_test` (DT): Feature matrix for test data.
          - `test_ids` (DT): Identifiers for the test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Specify the data source location (`/kaggle/input/`).
          - Clearly define the structure and type of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Precautions for Data Loading and Preprocessing:
        - File Handling:
          - Ensure proper file encoding and delimiters.
          - Combine or process multiple files if necessary.
        - Data Preprocessing:
          - Convert data types correctly (e.g., numeric, categorical, date parsing).
          - Handle missing values appropriately.
          - Optimize memory usage for large datasets using techniques like downcasting or reading data in chunks if necessary.
        - Domain-Specific Handling: 
          - Apply competition-specific preprocessing steps as needed (e.g., text tokenization, image resizing).

      3. Code Standards:
        - Avoid using progress bars (e.g., `tqdm`) in the implementation.

      4. Notes:
        - Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
        - Never use sample submission as the test index, as it may not be the same as the test data. Use the test index file or test data source to get the test index.

      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      Please respond with a JSON structure as follows:
      {
          "spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
      }

    feature: |-
      Feature engineering specification text should adhere to the following requirements:
      1. Function Interface:
        - Function Name: `feat_eng`
        - Parameters:
          - `X` (DT): Train data to be transformed.
          - `y` (DT): Train label data.
          - `X_test` (DT): Test data.
        - Output:
          - `X_transformed` (DT): Transformed train data.
          - `y_transformed` (DT): Transformed train label data.
          - `X_test_transformed` (DT): Transformed test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Precautions for Feature Engineering:
        - Well handle the shape of the data
          - The sample size of the train data and the test data should be the same in all scenarios.
          - To most of the scenario, the input shape and the output shape should be exactly the same.
          - To some tabular data, you may add or remove some columns so your inferred column number may be unsure.
        - Integration with Model Pipeline
          - If feature engineering is strictly part of the model pipeline, state explicitly that it will be handled at the model stage.
          - If integrated here, ensure this function applies all required transformations while avoiding data leakage.
        - General Considerations:
          - Ensure scalability for large datasets.
          - Handle missing values and outliers appropriately (e.g., impute, remove, or replace).
          - Ensure consistency between feature data types and transformations.
          - Avoid data leakage: Only use features derived from training data, excluding information from test or validation sets.
        - Domain-Specific Features:
          - Apply logic for competition-specific features (e.g., text vectorization, image augmentations, categorical encoding).

      3. Code Standards:
        - Avoid using progress bars (e.g., `tqdm`) in the implementation.          

      4. Notes:
        - Align `DT` (data type) definitions with those in the Data Loader specification.
        - Extend or adjust domain-specific transformations based on competition requirements.
        - The device has GPU support, so you can use it for feature engineering if necessary to accelerate the process.
        - Multi processing or parallel processing can be used to speed up the feature engineering process.
      
      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      Please respond with a JSON structure as follows:
      {
          "spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
      }

    model: |-
      Model building specification text should adhere to the following requirements:

      1. Function Interface:
        - Function Name: `model_workflow`
        - Parameters:
          - `X` (DT): Training feature data.
          - `y` (DT): Training label data.
          - `val_X` (Optional[DT]): Validation feature data.
          - `val_y` (Optional[DT]): Validation label data.
          - `test_X` (Optional[DT]): Test feature data.
          - `hyper_params` (dict): Dictionary of hyperparameters for model configuration.
        - Output:
          - `pred_val` (Optional[DT]): Predictions on validation data.
          - `pred_test` (Optional[DT]): Predictions on test data.
          - `hyper_params` (dict): Updated dictionary of hyperparameters after training.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Code Standards:
        - Do not use progress bars (e.g., `tqdm`) in the implementation.

      3. Precautions:
        - Ensure input arrays (`X`, `y`, `val_X`, `val_y`, `test_X`) have consistent dimensions and shapes.
        - Use default values for hyperparameters if `hyper_params` is not provided.
        - Train the model on `X` and `y`.
        - Evaluate the model using `val_X` and `val_y` if validation data is available.
        - If `test_X` is provided, generate predictions for it.

      4. Notes:
        - Align `DT` (data type) with the definitions used in Feature Engineering specifications.
        - The device has GPU support, so you can use it for training if necessary to accelerate the process.

      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      Please respond in the following JSON format:
      {
          "spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
      }

    ensemble: |-
      Ensemble specification text adhere to the following requirements:
      1. Function Interface:
        - Function Name: `ensemble_workflow`
        - Parameters:
          - `test_preds_dict` (Dict[str, DT]): A dictionary of test predictions from different models. The key is the model file name.
          - `val_preds_dict` (Dict[str, DT]): A dictionary of validation predictions from different models. The key is the model file name.
          - `val_label` (DT): Validation label.
        - Output:
          - `final_pred` (DT): Ensemble prediction for the test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Precautions:
        - Validation of Inputs:
          - Ensure all predictions in `test_preds_dict` and `val_preds_dict` have consistent shapes and dimensions.
          - Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions.
          - Handle empty or invalid inputs gracefully with appropriate error messages.
        - Metric Calculation and Storage:
          - Calculate the metric for each model and ensemble strategy, and save the results in `scores.csv`, e.g.:
            ```python
            scores = {}
            for model_name, val_pred in val_preds_dict.items():
                scores[model_name] = calculate_metric(val_label, val_pred)
            
            # ... some code about ensemble strategy ...
            ensemble_score = calculate_metric(val_label, ensemble_pred)
            scores["ensemble"] = ensemble_score  # Ensure "ensemble" is explicitly stored
            
            scores_df = pd.DataFrame(scores.items(), columns=["Model", <metric_name>])
            scores_df.to_csv("scores.csv", index=False)
            ```
          - If there is only one model, still compute the ensemble score and store it under "ensemble".
        
      3. Code Standards:
        - Do not use progress bars (e.g., tqdm) in the code.

      4. Notes:
        - Align `DT` (data type) definitions with those used in model specifications.
        - Ensure flexibility to handle multiple ensemble strategies based on competition requirements.

      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      Please respond in the following JSON format:
      {
          "spec": "The function definition in code format, tailored to the Competition Information, with detailed explanations provided in the docstring."
      }

    workflow: |-
      Your task is to implement the main workflow script (`main.py`) for a Kaggle-style machine learning competition project. 
      Follow the provided project structure and specifications to ensure consistency and maintainability:
        1. Workflow Integration:
          - Integrate the following components into the workflow:
            - Data loading (`load_data.py`).
            - Feature engineering (`feature.py`).
            - Model workflow for training and testing (`model_*.py`). 
            - Ensemble workflow that combines results from the model workflow to obtain the final prediction (`ensemble.py`).
          - Treat each component as a modular and callable Python function.
          - The workflow script should be flexible enough to handle either a single model or multiple models, with filenames (model_*.py) that are not determined at the outset.
            For multiple model selection, utilize Python code to identify eligible models based on filenames, for example:
            ```python
            available_models = [f for f in os.listdir('.') if f.startswith('model_') and 'test' not in f]
            ```
        2. Feature Engineering
          - The feature engineering should be called only once. For example:
            `X_transformed, y_transformed, X_test_transformed = feat_eng(X, y, X_test)`
          - It should be called before dataset splitting.

        3. Dataset Splitting
          - The dataset returned by `load_data` is not split into training and testing sets, so the dataset splitting should happen after calling `feat_eng`.
          - Choose the splitting methods you deem useful and appropriate based on the `Competition Information`.

        4. Submission File:
          - Save the final predictions as `submission.csv`, ensuring the format matches the competition requirements (refer to `sample_submission` in the Folder Description for the correct structure).
          - Present the required submission format explicitly and ensure the output adheres to it.

        5. Code Standards:
          - Do not use progress bars (e.g., tqdm) in the code.

        6. Ensemble Strategy:
          Put all the model's return into a dict, using the model file name as key, and the return as value.
          Sample code:
          {% raw %}
          {% for mn in model_names %}
          from {{mn}} import model_workflow as {{mn}}_workflow
          val_preds_dict["{{mn}}"], test_preds_dict["{{mn}}"], _ = {{mn}}_workflow(
              X=train_X,
              y=train_y,
              val_X=val_X,
              val_y=val_y,
              test_X=test_X
          )
          {% endfor %}
          final_pred = ensemble_workflow(test_preds_dict, val_preds_dict, val_y)
          {% endraw %}

        {% if latest_spec %}
        7. Former Specification:
          {{ latest_spec }}
          You should follow the provided specifications to improve this task.
        {% endif %}

        Please response the specification in the following json format. Here is an example structure for the JSON output:
        {
            "spec": "The corresponding specification string as described above. You should create the rules based on the competition information instead of copying the requirements."
        }

data_loader_coder:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    ## Task Description
    {{ task_desc }}

    {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
    ## Relevant Information for This Task
    {% endif %}
    
    {% if queried_similar_successful_knowledge|length != 0 %}
    --------- Successful Implementations for Similar Models ---------
    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
    {{ similar_successful_knowledge.target_task.get_task_information() }}
    =====Code:=====
    {{ similar_successful_knowledge.implementation.all_codes }}
    {% endfor %} 
    {% endif %}

    {% if queried_former_failed_knowledge|length != 0 %}
    --------- Previous Failed Attempts ---------
    {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
    =====Code:=====
    {{ former_failed_knowledge.implementation.all_codes }}
    =====Feedback:=====
    {{ former_failed_knowledge.feedback }}
    {% endfor %}
    {% endif %}

    ## Guidelines
    1. Ensure that the dataset is loaded strictly from `/kaggle/input/`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).

    ## Output Format
    Please response the code in the following json format. Here is an example structure for the JSON output:
    {
        "code": "The Python code as a string."
    }

  user: |-
    --------- Competition Information ---------
    {{ competition_info }}

    --------- Data Loader Specification ---------
    {{ data_loader_spec }}

    --------- Data Folder Description (All path are relative to the data folder) ---------
    {{ folder_spec }}
    
    {% if latest_code %}
    --------- Former code ---------
    {{ latest_code }}
    {% if latest_code_feedback is not none %}
    --------- Feedback to former code ---------
    {{ latest_code_feedback }}
    {% endif %}
    The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
    {% endif %} 

    You should strictly follow the function interface specifications provided by the specification to implement the function.


data_loader_eval:
  system: |-
    You are a data scientist responsible for evaluating data loader code for a Kaggle-style machine learning competition project.
    
    ## Task Description
    {{ task_desc }}

    ## Data Loader Code
    The data loader code is located in `load_data.py`:
    ```python
    {{ code }}
    ```

    ## Testing Process
    The data loader is tested using the following script:
    ```python
    {{ test_code }}
    ```

    {% if workflow_stdout is not none %}
    ### Whole Workflow Consideration
    The data loader is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.

    **Workflow Code:**
    ```python
    {{ workflow_code }}
    ```

    You should evaluate both the data loader test results and the overall workflow execution. **Approve the code only if both tests pass.**
    {% endif %}
    
    ## Evaluation Criteria
    You will be given the standard output (`stdout`) from the data loader test and, if applicable, the workflow test.
    
    Your response must follow this structured JSON format:
    ```json
    {
        "execution": "Describe how well the data loader executed, including any errors or issues encountered. Retain all error messages and traceback details.",
        "return_checking": "Evaluate the correctness and integrity of the loaded data. Check for issues like missing values, incorrect data types, outliers, or formatting inconsistencies.",
        "code": "Assess code quality, readability, and adherence to best practices. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for faster data loading.",
        "final_decision": <true/false>
    }
    ```

  user: |-
    --------- Data loader test stdout ---------
    {{ stdout }}   
    {% if workflow_stdout is not none %}
    --------- Whole workflow test stdout ---------
    {{ workflow_stdout }}
    {% endif %}