NexQuant/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml

spec:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    Currently, you are working on a Kaggle competition project. 
    This project involves analyzing data and building models to beat other competitors, with the code being generated by large language models.

    The runtime environment you are working in includes the following libraries and their respective versions:
    {{ runtime_environment }}

    Your overall task is provided below:
    {{ task_desc }}
    
    Your task is to write five specification texts (in markdown format) for the following tasks, based on the competition information provided
    - Data loading (and preprocessing)
    - Feature Engineering
    - Model Building
    - Ensemble
    - The overall workflow

    The specifications for each step should be tailored to the competition information provided. 
    
    Your specification should consists two parts:
    1. The function definition in code format, including type annotations and a clear, complete docstring that describes the function's purpose, input parameters, return value, and any relevant exceptions.
    2. Additional information or notes that the coder should consider while implementing the function.
    
    Your specifications should include only the function definition and docstring, without any code implementation or inline comments.

    ## Competition Information for This Task
    {{ competition_info }}

    ----------- Folder Description (All path are relative to the data folder) ---------
    - Ensure that all columns in sample_submission can be generated.
    {{ folder_spec }}

  user:
    data_loader: |-
      Data loader specification text should follow these detailed requirements:
      1. Function Interface:
        - Function Name: `load_data`
        - Input: No input arguments.
        - Output:
          - `X` (DT, define based on competition information): Feature matrix for training data.
          - `y` (DT): Target vector for training data.
          - `X_test` (DT): Feature matrix for test data.
          - `test_ids` (DT): Identifiers for the test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Specify the data source location (`{% include "scenarios.data_science.share:scen.input_path" %}`).
          - Clearly define the structure and type of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
      2. Notes:
        - Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
        - Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.

      Responsibilities and notes of an implemented data loader that aligns with the generated specification.
      {% include "scenarios.data_science.share:component_spec.DataLoadSpec" %}

      {% if latest_spec %}
      6. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      ## Output Format
      You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.

    feature: |-
      Feature engineering specification text should adhere to the following requirements:
      1. Function Interface:
        - Function Name: `feat_eng`
        - Parameters:
          - `X` (DT): Train data to be transformed.
          - `y` (DT): Train label data.
          - `X_test` (DT): Test data.
        - Output:
          - `X_transformed` (DT): Transformed train data.
          - `y_transformed` (DT): Transformed train label data.
          - `X_test_transformed` (DT): Transformed test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Precautions for Feature Engineering:
        - Well handle the shape of the data:
          - The sample size of the train data and the test data should be the same in all scenarios.
          - To some tabular or time-series data, you may add or remove some columns so your inferred column number may be unsure.
          - For scenarios where each dimension does not have a special meaning (like image, audio, and so on), the input shape and the output shape should be exactly the same in most cases unless there is a compelling reason to change them.
        - Integration with the Model Pipeline:
          - If feature engineering is deferred to the model pipeline for better overall performance, state explicitly that it will be handled at the model stage.
            - Model-related operations should not be implemented in this step. (e.g., it uses tools combined with models like torch.Dataset with rich data transformation/augmentation)
          - Otherwise, ensure this function applies all required transformations while avoiding data leakage.
        - General Considerations:
          - Ensure scalability for large datasets.
          - Handle missing values and outliers appropriately (e.g., impute, remove, or replace).
          - Ensure consistency between feature data types and transformations.
          - Prevent data leakage: Do not use information derived from the test set when transforming training data.
        - Domain-Specific Features:
          - Apply logic for competition-specific features (e.g., text vectorization, image augmentations, categorical encoding).

      3. Code Standards:
        - Avoid using progress bars (e.g., `tqdm`) in the implementation.          

      4. Notes:
        - Align `DT` (data type) definitions with those in the Data Loader specification.
        - GPU and multiprocessing are available and are encouraged to use for accelerating transformations.
        - Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.
      
      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      ## Output Format
      You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.

    model: |-
      Model building specification text should adhere to the following requirements:

      1. Function Interface:
        - Function Name: `model_workflow`
        - Parameters:
          - `X` (DT): Training feature data.
          - `y` (DT): Training label data.
          - `val_X` (Optional[DT]): Validation feature data.
          - `val_y` (Optional[DT]): Validation label data.
          - `test_X` (Optional[DT]): Test feature data.
          - `hyper_params` (dict): Dictionary of hyperparameters for model configuration.
        - Output:
          - `pred_val` (Optional[DT]): Predictions on validation data.
          - `pred_test` (Optional[DT]): Predictions on test data.
          - `hyper_params` (dict): Updated dictionary of hyperparameters after training.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Code Standards:
        - Do not use progress bars (e.g., `tqdm`) in the implementation.

      3. Precautions:
        - Ensure input arrays (`X`, `y`, `val_X`, `val_y`, `test_X`) have consistent dimensions and shapes.
        - Use default values for hyperparameters if `hyper_params` is not provided.
        - Train the model on `X` and `y`.
        - Evaluate the model using `val_X` and `val_y` if validation data is available.
        - If `test_X` is provided, generate predictions for it.

      4. Notes:
        - Align `DT` (data type) with the definitions used in Feature Engineering specifications.
        - The device has GPU support, so you are encouraged to use it for training if necessary to accelerate the process.
        - Some data transformations/augmentations can be included in this step (e.g., data tools provided by TensorFlow and Torch)

      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      ## Output Format
      You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.

    ensemble: |-
      Ensemble specification text adhere to the following requirements:
      1. Function Interface:
        - Function Name: `ensemble_workflow`
        - Parameters:
          - `test_preds_dict` (Dict[str, DT]): A dictionary of test predictions from different models. The key is the model file name.
          - `val_preds_dict` (Dict[str, DT]): A dictionary of validation predictions from different models. The key is the model file name.
          - `val_label` (DT): Validation label.
        - Output:
          - `final_pred` (DT): Ensemble prediction for the test data.
        - Docstring Requirements:
          - Describe the purpose of the function.
          - Clarify the input parameters and their data types.
          - Define the structure and format of the output.
          - Inferred data shape to each input and output data variables. To uncertain dimension, use -1.

      2. Precautions:
        - Input Validation:
          - Ensure all predictions in `test_preds_dict` and `val_preds_dict` have consistent shapes and dimensions.
          - Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions.
          - Handle empty or invalid inputs gracefully with appropriate error messages.
        - Metric Calculation and Storage:
          - Calculate the metric (mentioned in the evaluation section of the competition information) for each model and ensemble strategy on valid, and save the results in `scores.csv`, e.g.:
            ```python
            scores = {}
            for model_name, val_pred in val_preds_dict.items():
                scores[model_name] = calculate_metric(val_label, val_pred)
            
            ...
            some code about ensemble strategy
            ...
            ensemble_val_pred = ...

            ensemble_score = calculate_metric(val_label, ensemble_val_pred)
            scores["ensemble"] = ensemble_score  # Ensure "ensemble" is explicitly stored
            
            scores_df = pd.DataFrame(scores.items(), columns=["Model", <metric_name>])
            scores_df.to_csv("scores.csv", index=False)
            ```
          - Even if only one model is present, compute the ensemble score and store it under `"ensemble"`.
        
      3. Code Standards:
        - Do not use progress bars (e.g., tqdm) in the code.

      4. Notes:
        - Align `DT` (data type) definitions with those used in model specifications.
        - Ensure flexibility to handle multiple ensemble strategies based on competition requirements.
        - Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.

      {% if latest_spec %}
      5. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      ## Output Format
      You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.

    workflow: |-
      {% include "scenarios.data_science.share:component_spec.Workflow" %}

      {% if latest_spec %}
      7. Former Specification:
        {{ latest_spec }}
        You should follow the provided specifications to improve this task.
      {% endif %}

      ## Output Format
      You should return the specification in markdown format directly.
      You should create the rules based on the competition information instead of copying the requirements.

data_loader_coder:
  system: |-
    You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
    Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.

    ## Task Description
    {{ task_desc }}

    {% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
    ## Relevant Information for This Task
    {% endif %}
    
    {% if queried_similar_successful_knowledge|length != 0 %}
    --------- Successful Implementation Examples for Similar Task ---------
    ====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Example {{ loop.index }}:=====
    {{ similar_successful_knowledge.target_task.get_task_information() }}
    =====Code:=====
    {{ similar_successful_knowledge.implementation.all_codes }}
    {% endfor %} 
    {% endif %}

    {% if queried_former_failed_knowledge|length != 0 %}
    --------- Previous Failed Attempts ---------
    {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
    =====Code:=====
    {{ former_failed_knowledge.implementation.all_codes }}
    =====Feedback:=====
    {{ former_failed_knowledge.feedback }}
    {% endfor %}
    {% endif %}

    ## Guidelines
    1. Ensure that the dataset is loaded strictly from `{% include "scenarios.data_science.share:scen.input_path" %}`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).
    2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
    3. You should use the following cache decorator to cache the results of the function:
    ```python
    from joblib import Memory
    memory = Memory(location='{% include "scenarios.data_science.share:scen.cache_path" %}', verbose=0)
    @memory.cache```
    {% include "scenarios.data_science.share:guidelines.coding" %}
    
    ## Exploratory Data Analysis (EDA) part(Required):
    - Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better.
    - The EDA part should include but not limited in the following information in plain text:
      - The shape of the data.
      - The first 5 rows of the data.
      - The data types of each column.
      - The number of missing values in each column.
      - The number of unique values in each column.
      - The distribution of the target variable.
      - Any other information that you think is important for the following steps.
    - The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema: 
      === Start of EDA part ===
      { You EDA output content }
      === End of EDA part ===
      User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
    - An evaluation agent will help to check whether the EDA part is added correctly.
    - During the EDA part, you should try to avoid any irrelevant information sending to the standard output.

    ## Output Format
    {% if out_spec %}
    {{ out_spec }}
    {% else %}
    Please response the code in the following json format. Here is an example structure for the JSON output:
    {
        "code": "The Python code as a string."
    }
    {% endif %}

  user: |-
    --------- Competition Information ---------
    {{ competition_info }}

    --------- Code Specification ---------
    {{ code_spec }}

    --------- Data Folder Description (All path are relative to the data folder, i.e. "{% include "scenarios.data_science.share:scen.input_path" %}") ---------
    {{ folder_spec }}
    
    {% if latest_code %}
    --------- Former code ---------
    {{ latest_code }}
    {% if latest_code_feedback is not none %}
    --------- Feedback to former code ---------
    {{ latest_code_feedback }}
    {% endif %}
    The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
    {% endif %} 

    You should strictly follow the code specifications provided by the specification to implement the function.


data_loader_eval:
  system: |-
    You are a data scientist responsible for evaluating data loader code for a Kaggle-style machine learning competition project.
    
    ## Task Description
    {{ task_desc }}

    ## Data Loader Code
    The data loader code is located in `load_data.py`:
    ```python
    {{ code }}
    ```

    ## Testing Process
    The data loader is tested using the following script:
    ```python
    {{ test_code }}
    ```

    {% if workflow_stdout is not none %}
    ### Whole Workflow Consideration
    The data loader is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.

    **Workflow Code:**
    {{ workflow_code }}

    You should evaluate both the data loader test results and the overall workflow execution. **Approve the code only if both tests pass.**
    {% endif %}
    
    ## Evaluation Criteria
    You will be given the standard output (`stdout`) from the data loader test and, if applicable, the workflow test.

    ## Exploratory Data Analysis (EDA) Part evaluation
    - The code has also generated some EDA output to help understand the data better. 
    - The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema: 
      === Start of EDA part ===
      { You EDA output content }
      === End of EDA part ===
      User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
    - The EDA part should include but not limited in the following information in plain text:
      - The shape of the data.
      - The first 5 rows of the data.
      - The data types of each column.
      - The number of missing values in each column.
      - The number of unique values in each column.
      - The distribution of the target variable.
      - Any other information that you think is important for the following steps.
    You will be given the EDA output, your job is to check whether the output contains the required and sufficient information. If no EDA output is provided, you should consider it as a failure. Put this evaluation result in the return_checking part.
    
    Your response must follow this structured JSON format:
    ```json
    {
        "execution": "Describe how well the data loader executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
        "return_checking": "Evaluate the correctness and integrity of the loaded data. Check for issues like missing values, incorrect data types, outliers, or formatting inconsistencies.",
        "code": "Assess code quality, readability, and adherence to best practices. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for faster data loading.",
        "final_decision": <true/false>
    }
    ```

  user: |-
    --------- Data loader test stdout ---------
    {{ stdout }}   
    --------- Data loader EDA stdout ---------
    {% if eda_output is not none %}
    {{ eda_output }}
    {% else %}
    No EDA output is provided.
    {% endif %}
    {% if workflow_stdout is not none %}
    --------- Whole workflow test stdout ---------
    {{ workflow_stdout }}
    {% endif %}