mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 15:37:44 +00:00
docs: update data science docs (#1015)
* update data science docs part1 * update data science docs part2 * update data science docs part3 * update data science docs part4 * update data science docs part5 * format with isort * update data science docs part6 * add check environment scripts * format with isort * format with isort * merge env_check to health_check * format with isort * format with black * optimize code * use boolean cli options * replace fire with typer * replace fire with typer * optimizing parameter and variable naming
This commit is contained in:
Linlang
@@ -114,6 +114,8 @@ Additionally, you can take a closer look at the examples in our **[🖥️ Live
|
||||
|
||||
# ⚡ Quick start
|
||||
|
||||
### RD-Agent currently only supports Linux.
|
||||
|
||||
You can try above demos by running the following command:
|
||||
|
||||
### 🐳 Docker installation.
|
||||
@@ -153,7 +155,7 @@ More details can be found in the [development setup](https://rdagent.readthedocs
|
||||
- whether the docker installation was successful.
|
||||
- whether the default port used by the [rdagent ui](https://github.com/microsoft/RD-Agent?tab=readme-ov-file#%EF%B8%8F-monitor-the-application-results) is occupied.
|
||||
```sh
|
||||
rdagent health_check
|
||||
rdagent health_check --no-check-env
|
||||
```
|
||||
|
||||
|
||||
@@ -224,6 +226,12 @@ More details can be found in the [development setup](https://rdagent.readthedocs
|
||||
|
||||
|
||||
|
||||
- If your environment configuration is complete, please execute the following commands to check if your configuration is valid. This step is necessary.
|
||||
|
||||
```bash
|
||||
rdagent health_check
|
||||
```
|
||||
|
||||
### 🚀 Run the Application
|
||||
|
||||
The **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)** is implemented by the following commands(each item represents one demo, you can select the one you prefer):
|
||||
@@ -263,44 +271,70 @@ The **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)** is implemented b
|
||||
rdagent general_model "https://arxiv.org/pdf/2210.09789"
|
||||
```
|
||||
|
||||
- Run the **Automated Medical Prediction Model Evolution**: Medical self-loop model proposal and implementation application
|
||||
|
||||
```bash
|
||||
# Generally, you can run the data science program with the following command:
|
||||
rdagent data_science --competition <your competition name>
|
||||
|
||||
# Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example:
|
||||
|
||||
# 1. Download the dataset, extract it to the target folder.
|
||||
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
|
||||
unzip arf-12-hours-prediction-task.zip -d ./git_ignore_folder/ds_data/
|
||||
|
||||
# 2. Configure environment variables in the `.env` file
|
||||
dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
|
||||
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
|
||||
dotenv set DS_IF_USING_MLE_DATA False
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM False
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
|
||||
|
||||
# 3. run the application
|
||||
rdagent data_science --competition arf-12-hours-prediction-task
|
||||
```
|
||||
|
||||
**NOTE:** For more information about the dataset, please refer to the [documentation](https://rdagent.readthedocs.io/en/latest/scens/data_science.html).
|
||||
|
||||
- Run the **Automated Kaggle Model Tuning & Feature Engineering**: self-loop model proposal and feature engineering implementation application <br />
|
||||
> Using **sf-crime** *(San Francisco Crime Classification)* as an example. <br />
|
||||
> Using **tabular-playground-series-dec-2021** as an example. <br />
|
||||
> 1. Register and login on the [Kaggle](https://www.kaggle.com/) website. <br />
|
||||
> 2. Configuring the Kaggle API. <br />
|
||||
> (1) Click on the avatar (usually in the top right corner of the page) -> `Settings` -> `Create New Token`, A file called `kaggle.json` will be downloaded. <br />
|
||||
> (2) Move `kaggle.json` to `~/.config/kaggle/` <br />
|
||||
> (3) Modify the permissions of the kaggle.json file. Reference command: `chmod 600 ~/.config/kaggle/kaggle.json` <br />
|
||||
> 3. Join the competition: Click `Join the competition` -> `I Understand and Accept` at the bottom of the [competition details page](https://www.kaggle.com/competitions/sf-crime/data).
|
||||
> 3. Join the competition: Click `Join the competition` -> `I Understand and Accept` at the bottom of the [competition details page](https://www.kaggle.com/competitions/tabular-playground-series-dec-2021/data).
|
||||
```bash
|
||||
# Generally, you can run the Kaggle competition program with the following command:
|
||||
rdagent data_science --competition <your competition name>
|
||||
|
||||
# Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example:
|
||||
|
||||
# 1. Install chromedriver.
|
||||
|
||||
# 2. Add the competition description file path to the `.env` file.
|
||||
mkdir -p ./git_ignore_folder/kaggle_data
|
||||
dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/kaggle_data"
|
||||
# 1. Configure environment variables in the `.env` file
|
||||
mkdir -p ./git_ignore_folder/ds_data
|
||||
dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
|
||||
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
|
||||
dotenv set DS_IF_USING_MLE_DATA True
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM True
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
|
||||
|
||||
# 3. run the application
|
||||
rdagent data_science --competition sf-crime
|
||||
# 2. run the application
|
||||
rdagent data_science --competition tabular-playground-series-dec-2021
|
||||
```
|
||||
|
||||
### 🖥️ Monitor the Application Results
|
||||
- You can run the following command for our demo program to see the run logs.
|
||||
|
||||
```sh
|
||||
rdagent ui --port 19899 --log_dir <your log folder like "log/">
|
||||
rdagent ui --port 19899 --log_dir <your log folder like "log/"> --data_science <True or False>
|
||||
```
|
||||
|
||||
**Note:** Although port 19899 is not commonly used, but before you run this demo, you need to check if port 19899 is occupied. If it is, please change it to another port that is not occupied.
|
||||
- About the `data_science` parameter: If you want to see the logs of the data science scenario, set the `data_science` parameter to `True`; otherwise set it to `False`.
|
||||
|
||||
- Although port 19899 is not commonly used, but before you run this demo, you need to check if port 19899 is occupied. If it is, please change it to another port that is not occupied.
|
||||
|
||||
You can check if a port is occupied by running the following command.
|
||||
|
||||
```sh
|
||||
rdagent health_check
|
||||
rdagent health_check --no-check-env --no-check-docker
|
||||
```
|
||||
|
||||
# 🏭 Scenarios
|
||||
|
||||
+460
-110
@@ -8,10 +8,59 @@ Data Science Agent
|
||||
------------------------------------------------------------------------------------------
|
||||
The Data Science Agent is an agent that can automatically perform feature engineering and model tuning. It can be used to solve various data science problems, such as image classification, time series forecasting, and text classification.
|
||||
|
||||
🧭 Example Guide
|
||||
🌟 Introduction
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In this scenario, our automated system proposes hypothesis, choose action, implements code, conducts validation, and utilizes feedback in a continuous, iterative process.
|
||||
|
||||
The goal is to automatically optimize performance metrics within the validation set or Kaggle Leaderboard, ultimately discovering the most efficient features and models through autonomous research and development.
|
||||
|
||||
Here's an enhanced outline of the steps:
|
||||
|
||||
**Step 1 : Hypothesis Generation 🔍**
|
||||
|
||||
- Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification.
|
||||
|
||||
**Step 2 : Experiment Creation ✨**
|
||||
|
||||
- Transform the hypothesis into a task.
|
||||
- Choose a specific action within feature engineering or model tuning.
|
||||
- Develop, define, and implement a new feature or model, including its name, description, and formulation.
|
||||
|
||||
**Step 3 : Model/Feature Implementation 👨💻**
|
||||
|
||||
- Implement the model code based on the detailed description.
|
||||
- Evolve the model iteratively as a developer would, ensuring accuracy and efficiency.
|
||||
|
||||
**Step 4 : Validation on Test Set or Kaggle 📉**
|
||||
|
||||
- Validate the newly developed model using the test set or Kaggle dataset.
|
||||
- Assess the model's effectiveness and performance based on the validation results.
|
||||
|
||||
**Step 5: Feedback Analysis 🔍**
|
||||
|
||||
- Analyze validation results to assess performance.
|
||||
- Use insights to refine hypotheses and enhance the model.
|
||||
|
||||
**Step 6: Hypothesis Refinement ♻️**
|
||||
|
||||
- Adjust hypotheses based on validation feedback.
|
||||
- Iterate the process to continuously improve the model.
|
||||
|
||||
📖 Data Science Background
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- 🔧 **Set up RD-Agent Environment**
|
||||
In the evolving landscape of artificial intelligence, **Data Science** represents a powerful paradigm where machines engage in autonomous exploration, hypothesis testing, and model development across diverse domains — from healthcare and finance to logistics and research.
|
||||
|
||||
The **Data Science** Agent stands as a central engine in this transformation, enabling users to automate the entire machine learning workflow: from hypothesis generation to code implementation, validation, and refinement — all guided by performance feedback.
|
||||
|
||||
By leveraging the **Data Science** Agent, researchers and developers can accelerate experimentation cycles. Whether fine-tuning custom models or competing in high-stakes benchmarks like Kaggle, the Data Science Agent unlocks new frontiers in intelligent, self-directed discovery.
|
||||
|
||||
🧭 Example Guide - Customized dataset
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
🔧 **Set up RD-Agent Environment**
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the `documentation <../installation_and_configuration.html>`_.
|
||||
|
||||
@@ -24,161 +73,462 @@ The Data Science Agent is an agent that can automatically perform feature engine
|
||||
dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
|
||||
|
||||
- 📥 **Prepare Competition Data**
|
||||
📥 **Prepare Customized datasets**
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- Data Science competition data typically consists of three components: a competition description file (in Markdown format), the competition dataset, and evaluation scripts. For reference, an example of a custom user-defined dataset is provided in ``rdagent/scenarios/data_science/example``.
|
||||
- A data science competition dataset usually consists of two parts: ``competition dataset`` and ``evaluation dataset``. (We provide `a sample <https://github.com/microsoft/RD-Agent/tree/main/rdagent/scenarios/data_science/example>`_ of a customized dataset named: `arf-12-hours-prediction-task as a reference`.)
|
||||
|
||||
- The ``competition dataset`` contains **training data**, **test data**, **description files**, **formatted submission files**, **data sampling codes**.
|
||||
|
||||
- The ``evaluation dataset`` contains **standard answer file**, **data checking codes**, and **Code for calculation of scores**.
|
||||
|
||||
- **Correct directory structure (Here is an example of competition data with id custom_data)**
|
||||
- We use the ``arf-12-hours-prediction-task`` data as a sample to introduce the preparation workflow for the competition dataset.
|
||||
|
||||
- Create a ``ds_data/source_data/arf-12-hours-prediction-task`` folder, which will be used to store your raw dataset.
|
||||
|
||||
- The raw files for the competition ``arf-12-hours-prediction-task`` have two files: ``ARF_12h.csv`` and ``X.npz``.
|
||||
|
||||
- Create a ``ds_data/source_data/arf-12-hours-prediction-task/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)
|
||||
|
||||
- The following shows the preprocessing code for the raw data of ``arf-12-hours-prediction-task``.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/arf-12-hours-prediction-task/prepare.py
|
||||
:language: python
|
||||
:caption: ds_data/source_data/arf-12-hours-prediction-task/prepare.py
|
||||
:linenos:
|
||||
|
||||
- At the end of program execution, the ``ds_data`` folder structure will look like this:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
└── eval
|
||||
| └── custom_data
|
||||
| └── grade.py
|
||||
| └── valid.py
|
||||
| └── test.csv
|
||||
└── custom_data
|
||||
└── train.csv
|
||||
└── test.csv
|
||||
└── sample_submission.csv
|
||||
└── description.md
|
||||
└── sample.py
|
||||
|
||||
- ``ds_data/custom_data/train.csv:`` Necessary training data in csv or parquet format, or training images.
|
||||
├── arf-12-hours-prediction-task
|
||||
│ ├── train
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ ├── test
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ └── sample_submission.csv
|
||||
├── eval
|
||||
│ └── arf-12-hours-prediction-task
|
||||
│ └── submission_test.csv
|
||||
└── source_data
|
||||
└── arf-12-hours-prediction-task
|
||||
├── ARF_12h.csv
|
||||
├── prepare.py
|
||||
└── X.npz
|
||||
|
||||
- ``ds_data/custom_data/description.md:`` (Optional) Competition description file.
|
||||
- Create a ``ds_data/arf-12-hours-prediction-task/description.md`` file to describe your competition, Objective, dataset, and other information.
|
||||
|
||||
- ``ds_data/custom_data/sample_submission.csv:`` (Optional) Competition sample submission file.
|
||||
- The following shows the description file for ``arf-12-hours-prediction-task``
|
||||
|
||||
- ``ds_data/custom_data/sample.py:`` (Optional) Sample code for generating debug data from the competition dataset. If not provided, R&D-Agent will use its default sampling logic. For details, see the ``create_debug_data`` function in ``rdagent/scenarios/data_science/debug/data.py``.
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/description.md
|
||||
:language: markdown
|
||||
:caption: ds_data/arf-12-hours-prediction-task/description.md
|
||||
:linenos:
|
||||
|
||||
- ``ds_data/eval/custom_data/grade.py:`` (Optional) Competition grade script, in order to calculate the score for the submission.
|
||||
- Create a ``ds_data/arf-12-hours-prediction-task/sample.py`` file to construct the debugging sample data.
|
||||
|
||||
- ``ds_data/eval/custom_data/valid.py:`` (Optional) Competition validation script, in order to check if the submission format is correct.
|
||||
- The following shows the script for constructing the debugging sample data based on the ``arf-12-hours-prediction-task`` dataset implementation.
|
||||
|
||||
- ``ds_data/eval/custom_data/submission_test.csv:`` (Optional) Competition test label file.
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/sample.py
|
||||
:language: markdown
|
||||
:caption: ds_data/arf-12-hours-prediction-task/sample.py
|
||||
:linenos:
|
||||
|
||||
- 🔧 **Set up Environment for Custom User-defined Dataset**
|
||||
- Create a ``ds_data/eval/arf-12-hours-prediction-task/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.
|
||||
|
||||
- The following shows a script that checks the validity of a submission based on the ``arf-12-hours-prediction-task`` data.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/valid.py
|
||||
:language: markdown
|
||||
:caption: ds_data/eval/arf-12-hours-prediction-task/valid.py
|
||||
:linenos:
|
||||
|
||||
- Create a ``ds_data/eval/arf-12-hours-prediction-task/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.
|
||||
|
||||
- The following shows a grading script based on the ``arf-12-hours-prediction-task`` data implementation.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/grade.py
|
||||
:language: markdown
|
||||
:caption: ds_data/eval/arf-12-hours-prediction-task/grade.py
|
||||
:linenos:
|
||||
|
||||
- At this point, you have created a complete dataset. The correct structure of the dataset should look like this.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
├── arf-12-hours-prediction-task
|
||||
│ ├── train
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ ├── test
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ ├── description.md
|
||||
│ ├── sample_submission.csv
|
||||
│ └── sample.py
|
||||
├── eval
|
||||
│ └── arf-12-hours-prediction-task
|
||||
│ ├── grade.py
|
||||
│ ├── submission_test.csv
|
||||
│ └── valid.py
|
||||
└── source_data
|
||||
└── arf-12-hours-prediction-task
|
||||
├── ARF_12h.csv
|
||||
├── prepare.py
|
||||
└── X.npz
|
||||
|
||||
- The above shows the complete dataset creation workflow, some of the files are not required, in practice you can customize the dataset according to your own needs.
|
||||
|
||||
- If we don't need the test set scores, then we can choose not to generate **formatted submission files** and **standard answer file** in the prepare code, and we don't need to write **data checking codes** and **Code for calculation of scores**.
|
||||
|
||||
- **Data sampling code** can also be created according to the actual need, if you do not provide **data sampling code**, RD-Agent will be handed over to the LLM sampling at runtime.
|
||||
|
||||
- In the default sampling method (``create_debug_data``), the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can adjust the sampling ratio by adjusting these two parameters.
|
||||
|
||||
- If you have customized data sampling code, you need to set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` (default is True) in the ``.env`` file before running, so that the program will use the customized sampling code when running, and you can just execute this line of code in the command line:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM False
|
||||
|
||||
- In addition, we provide a data sampling method in `rdagent.scenarios.data_science.debug.data.create_debug_data <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/data_science/debug/data.py#L605>`_, in this method, the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can use this method by the following two ways.
|
||||
|
||||
- You can set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` file so that when the program runs, it will use the sampling code provided by RD-Agent.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM False
|
||||
|
||||
- If you think that the parameters in the receipt sampling method provided by RD-Agent are not suitable, you can customize the parameters in the following command and run it, and set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` so that the program will use the sampling data you provided when running.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python rdagent/app/data_science/debug.py --dataset_path <dataset path> --competition <competiton_name> --min_frac <sampling ratio> --min_num <minimum number of sampling>
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM False
|
||||
|
||||
- If you don't need the scores from the test set and leave the data sampling to the LLM, or if you use the sampling method provided by the RD-Agent, you only need to prepare a minimal dataset. The structure of the simplest dataset should be as shown below.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
├── arf-12-hours-prediction-task
|
||||
│ ├── train
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ ├── test
|
||||
│ │ ├── ARF_12h.csv
|
||||
│ │ └── X.npz
|
||||
│ └── description.md
|
||||
└── source_data
|
||||
└── arf-12-hours-prediction-task
|
||||
├── ARF_12h.csv
|
||||
├── prepare.py
|
||||
└── X.npz
|
||||
|
||||
- We have prepared a dataset based on the above description for your reference. You can download it with the following command.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
|
||||
|
||||
⚙️ **Set up Environment for Customized datasets**
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
|
||||
dotenv set DS_LOCAL_DATA_PATH rdagent/scenarios/data_science/example
|
||||
dotenv set DS_IF_USING_MLE_DATA False
|
||||
dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
|
||||
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
|
||||
dotenv set DS_CODER_COSTEER_ENV_TYPE docker
|
||||
|
||||
- 🚀 **Run the Application**
|
||||
🚀 **Run the Application**
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- You can directly run the application by using the following command:
|
||||
- 🌏 You can directly run the application by using the following command:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
rdagent data_science --competition <Competition ID>
|
||||
|
||||
- Then, you can run the test set score corresponding to each round of the loop.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
|
||||
|
||||
Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
|
||||
|
||||
- 📥 **Visualize the R&D Process**
|
||||
|
||||
- We provide a web UI to visualize the log. You just need to run:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
streamlit run rdagent/log/ui/dsapp.py
|
||||
|
||||
- Then you can input the log path and visualize the R&D process.
|
||||
|
||||
🔍 MLE-bench Guide: Running ML Engineering via MLE-bench
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- 📝 **MLE-bench Overview**
|
||||
|
||||
- MLE-bench is a comprehensive benchmark designed to evaluate the ML engineering capabilities of AI systems using real-world scenarios. The dataset comprises 75 Kaggle competitions. Since Kaggle does not provide held-out test sets for these competitions, the benchmark includes preparation scripts that split the publicly available training data into new training and test sets, and grading scripts are provided for each competition to accurately evaluate submission scores.
|
||||
|
||||
- 🔧 **Set up Environment for MLE-bench**
|
||||
|
||||
- Running R&D-Agent on MLE-bench is designed for full automation. There is no need for manual downloads and data preparation. Simply set the environment variable ``DS_IF_USING_MLE_DATA`` to True.
|
||||
|
||||
- At runtime, R&D-Agent will automatically build the Docker image specified at ``rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile``. This image is responsible for downloading the required datasets and grading files for MLE-bench.
|
||||
|
||||
- Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
|
||||
dotenv set DS_IF_USING_MLE_DATA True
|
||||
|
||||
- 🔨 **Configuring the Kaggle API**
|
||||
|
||||
- Downloading Kaggle competition data requires the Kaggle API. You can set up the Kaggle API by following these steps:
|
||||
|
||||
- Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
|
||||
|
||||
- Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
|
||||
|
||||
- Move ``kaggle.json`` to ``~/.config/kaggle/``
|
||||
|
||||
- Modify the permissions of the ``kaggle.json`` file.
|
||||
- The following shows the command to run based on the ``arf-12-hours-prediction-task`` data
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
chmod 600 ~/.config/kaggle/kaggle.json
|
||||
rdagent data_science --competition arf-12-hours-prediction-task
|
||||
|
||||
- For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
|
||||
- 📈 Visualize the R&D Process
|
||||
|
||||
- We provide a web UI to visualize the log. You just need to run:
|
||||
|
||||
- 🔩 **Setting the Environment Variables for MLE-bench**
|
||||
.. code-block:: sh
|
||||
|
||||
- In addition to auto-downloading the benchmark data, you must also configure the runtime environment for executing the competition code.
|
||||
- Use the environment variable ``DS_CODER_COSTEER_ENV_TYPE`` to select the execution mode:
|
||||
|
||||
• When set to docker (the default), RD-Agent utilizes the official Kaggle Docker image (``gcr.io/kaggle-gpu-images/python:latest``) to ensure that all required packages are available.
|
||||
• If you prefer to use a custom Docker setup, you can modify the configuration using ``DS_DOCKER_IMAGE`` or ``DS_DOCKERFILE_FOLDER_PATH``.
|
||||
• Alternatively, if your competition work only demands basic libraries, you may set ``DS_CODER_COSTEER_ENV_TYPE`` to conda. In this mode, you must create a local conda environment named “kaggle” and pre-install the necessary packages. RD-Agent will execute the competition code within this “kaggle” conda environment.
|
||||
rdagent ui --port <custom port> --log_dir <your log folder like "log/"> --data_science True
|
||||
|
||||
- Then you can input the log path and visualize the R&D process.
|
||||
|
||||
- 🧪 Scoring the test results
|
||||
|
||||
- Finally, shutdown the program, and get the test set scores with this command.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
# Configure the runtime environment: choice between 'docker' (default) or 'conda'
|
||||
dotenv set DS_CODER_COSTEER_ENV_TYPE docker
|
||||
dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
|
||||
|
||||
- 🚀 **Run the Application**
|
||||
Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
|
||||
|
||||
🕹️ Kaggle Agent
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
📖 Background
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
In the landscape of data science competitions, Kaggle serves as the ultimate arena where data enthusiasts harness the power of algorithms to tackle real-world challenges.
|
||||
The Kaggle Agent stands as a pivotal tool, empowering participants to seamlessly integrate cutting-edge models and datasets, transforming raw data into actionable insights.
|
||||
|
||||
By utilizing the **Kaggle Agent**, data scientists can craft innovative solutions that not only uncover hidden patterns but also drive significant advancements in predictive accuracy and model robustness.
|
||||
|
||||
🧭 Example Guide - Kaggle Dataset
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
🛠️ Preparing For The Competition
|
||||
""""""""""""""""""""""""""""""""""
|
||||
|
||||
- 🔨 **Configuring the Kaggle API**
|
||||
|
||||
- Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
|
||||
- Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
|
||||
- Move ``kaggle.json`` to ``~/.config/kaggle/``
|
||||
- Modify the permissions of the ``kaggle.json`` file.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
chmod 600 ~/.config/kaggle/kaggle.json
|
||||
|
||||
- For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
|
||||
|
||||
- 🔩 **Setting the Environment variables at .env file**
|
||||
|
||||
- Determine the path where the data will be stored and add it to the ``.env`` file.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
mkdir -p <your local directory>/ds_data
|
||||
dotenv set KG_LOCAL_DATA_PATH <your local directory>/ds_data
|
||||
|
||||
- 🗳️ **Join the competition**
|
||||
|
||||
- If your Kaggle API account has not joined a competition, you will need to join the competition before running the program.
|
||||
|
||||
- At the bottom of the competition details page, you can find the ``Join the competition`` button, click on it and select ``I Understand and Accept`` to join the competition.
|
||||
|
||||
- In the **Competition List Available** below, you can jump to the competition details page.
|
||||
|
||||
📥 Preparing Competition DataDataset && Set up RD-Agent Environment
|
||||
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
- As a subset of data science, kaggle's dataset still follows the data science format. Based on this, the kaggle dataset can be divided into two categories depending on whether or not it is supported by the **MLE-Bench**.
|
||||
|
||||
- What is **MLE-Bench**?
|
||||
|
||||
- **MLE-Bench** is a comprehensive benchmark designed to evaluate the **machine learning engineering** capabilities of AI systems using real-world scenarios. The dataset includes multiple Kaggle competitions. Since Kaggle does not provide reserved test sets for these competitions, the benchmark includes preparation scripts for splitting publicly available training data into new training and test sets, and scoring scripts for each competition to accurately evaluate submission scores.
|
||||
|
||||
- I'm running a competition Is **MLE-Bench** supported?
|
||||
|
||||
- You can see all the competitions supported by **MLE-Bench** `here <https://github.com/openai/mle-bench/tree/main/mlebench/competitions>`_.
|
||||
|
||||
- Prepare datasets for **MLE-Bench** supported competitions.
|
||||
|
||||
- If you agree with the **MLE-Bench** standard, then you don't need to prepare the dataset, you just need to configure your ``.env`` file to automate the download of the dataset.
|
||||
|
||||
- Configure environment variables, add ``DS_IF_USING_MLE_DATA`` to environment variables, and set it to ``True``.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_IF_USING_MLE_DATA True
|
||||
|
||||
- Configure environment variables, add ``DS_SAMPLE_DATA_BY_LLM`` to environment variables, and set it to ``True``.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM True
|
||||
|
||||
- Configure environment variables, add ``DS_SCEN`` to environment variables, and set it to ``rdagent.scenarios.data_science.scen.KaggleScen``.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
|
||||
|
||||
- At this point, you are ready to start running your competition, which will automatically download the data, and the LLM will automatically extract the minimum dataset.
|
||||
|
||||
- After running the program the structure of the ds_data folder should look like this (Using the ``tabular-playground-series-dec-2021`` contest as an example).
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
├── tabular-playground-series-dec-2021
|
||||
│ ├── description.md
|
||||
│ ├── sample_submission.csv
|
||||
│ ├── test.csv
|
||||
│ └── train.csv
|
||||
└── zip_files
|
||||
└── tabular-playground-series-dec-2021
|
||||
└── tabular-playground-series-dec-2021.zip
|
||||
|
||||
- The ``ds_data/zip_files`` folder contains a zip file of the raw competition data downloaded from kaggle website.
|
||||
|
||||
- At runtime, RD-Agent will automatically build the Docker image specified at `rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile>`_. This image is responsible for downloading the required datasets and grading files for MLE-Bench.
|
||||
|
||||
Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
|
||||
|
||||
- Prepare datasets for competitions that are not supported by **MLE-Bench**.
|
||||
|
||||
- As a subset of data science, we can follow the format and steps of data science dataset to prepare kaggle dataset. Below we will describe the workflow for preparing a kaggle dataset using the competition ``playground-series-s4e9`` as an example.
|
||||
|
||||
- Create a ``ds_data/source_data/playground-series-s4e9`` folder, which will be used to store your raw dataset.
|
||||
|
||||
- The raw files for the competition ``playground-series-s4e9`` have two files: ``train.csv``, ``test.csv``, ``sample_submission.csv``, and there are two ways to get the raw data:
|
||||
|
||||
- You can find the raw data required for the competition on the `official kaggle website <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_.
|
||||
|
||||
- Or you can use the command line to download the raw data for the competition, the download command is as follows.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
kaggle competitions download -c playground-series-s4e9
|
||||
|
||||
- Create a ``ds_data/source_data/playground-series-s4e9/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)
|
||||
|
||||
- The following shows the preprocessing code for the raw data of ``playground-series-s4e9``.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/playground-series-s4e9/prepare.py
|
||||
:language: python
|
||||
:caption: ds_data/source_data/playground-series-s4e9/prepare.py
|
||||
:linenos:
|
||||
|
||||
- At the end of program execution, the ``ds_data`` folder structure will look like this:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
├── playground-series-s4e9
|
||||
│ ├── train.csv
|
||||
│ ├── test.csv
|
||||
│ └── sample_submission.csv
|
||||
├── eval
|
||||
│ └── playground-series-s4e9
|
||||
│ └── submission_test.csv
|
||||
└── source_data
|
||||
└── playground-series-s4e9
|
||||
├── prepare.py
|
||||
├── sample_submission.csv
|
||||
├── test.csv
|
||||
└── train.csv
|
||||
|
||||
- Create a ``ds_data/playground-series-s4e9/description.md`` file to describe your competition, dataset description, and other information. We can find the `competition description information <https://www.kaggle.com/competitions/playground-series-s4e9/overview>`_ and the `dataset description information <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_ from the Kaggle website.
|
||||
|
||||
- The following shows the description file for ``playground-series-s4e9``
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/playground-series-s4e9/description.md
|
||||
:language: markdown
|
||||
:caption: ds_data/playground-series-s4e9/description.md
|
||||
:linenos:
|
||||
|
||||
- Create a ``ds_data/eval/playground-series-s4e9/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.
|
||||
|
||||
- The following shows a script that checks the validity of a submission based on the ``playground-series-s4e9`` data.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/valid.py
|
||||
:language: markdown
|
||||
:caption: ds_data/eval/playground-series-s4e9/valid.py
|
||||
:linenos:
|
||||
|
||||
- Create a ``ds_data/eval/playground-series-s4e9/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.
|
||||
|
||||
- The following shows a grading script based on the ``playground-series-s4e9`` data implementation.
|
||||
|
||||
.. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/grade.py
|
||||
:language: markdown
|
||||
:caption: ds_data/eval/playground-series-s4e9/grade.py
|
||||
:linenos:
|
||||
|
||||
- In this example we don't create a ``ds_data/eval/playground-series-s4e9/sample.py``, we use the sample method provided by RD-Agent by default.
|
||||
|
||||
- At this point, you have created a complete dataset. The correct structure of the dataset should look like this.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
ds_data
|
||||
├── playground-series-s4e9
|
||||
│ ├── train.csv
|
||||
│ ├── test.csv
|
||||
│ ├── description.md
|
||||
│ └── sample_submission.csv
|
||||
├── eval
|
||||
│ └── playground-series-s4e9
|
||||
│ ├── grade.py
|
||||
│ ├── submission_test.csv
|
||||
│ └── valid.py
|
||||
└── source_data
|
||||
└── playground-series-s4e9
|
||||
├── prepare.py
|
||||
├── sample_submission.csv
|
||||
├── test.csv
|
||||
└── train.csv
|
||||
|
||||
- We have prepared a dataset based on the above description for your reference. You can download it with the following command.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip
|
||||
|
||||
- Next, we need to configure the environment for the ``playground-series-s4e9`` contest. You can do this by executing the following command at the command line.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
dotenv set DS_IF_USING_MLE_DATA False
|
||||
dotenv set DS_SAMPLE_DATA_BY_LLM False
|
||||
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
|
||||
|
||||
🚀 **Run the Application**
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
|
||||
- 🌏 You can directly run the application by using the following command:
|
||||
|
||||
- You can directly run the application by using the following command:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
rdagent data_science --competition <Competition ID>
|
||||
|
||||
- 📥 **Visualize the R&D Process**
|
||||
- The following shows the command to run based on the ``playground-series-s4e9`` data
|
||||
|
||||
- We provide a web UI to visualize the log. You just need to run:
|
||||
.. code-block:: sh
|
||||
|
||||
rdagent data_science --competition playground-series-s4e9
|
||||
|
||||
- 📈 Visualize the R&D Process
|
||||
|
||||
- We provide a web UI to visualize the log. You just need to run:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
rdagent ui --port <custom port> --log_dir <your log folder like "log/"> --data_science True
|
||||
|
||||
- Then you can input the log path and visualize the R&D process.
|
||||
|
||||
- 🧪 Scoring the test results
|
||||
|
||||
- Finally, shutdown the program, and get the test set scores with this command.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
streamlit run rdagent/log/ui/dsapp.py
|
||||
dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
|
||||
|
||||
- Then you can input the log path and visualize the R&D process.
|
||||
- If you have configured the full output in ``ds_data/eval/playground-series-s4e9/grade.py``, or if you are running a competition that receives **MLE-Bench** support, you can also summarize the scores by running the following command.
|
||||
|
||||
- **Additional Guidance**
|
||||
|
||||
- **Combine different LLM Models at R&D Stage**
|
||||
|
||||
- You can combine different LLM models at the R&D stage.
|
||||
|
||||
- By default, when you set environment variable ``CHAT_MODEL``, it covers both R&D stages. When customizing the model for the development stage, you can set:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
# This example sets the model to "o3-mini". For some models, the reasoning effort shoule be set to "None".
|
||||
dotenv set LITELLM_CHAT_MODEL_MAP '{"coding":{"model":"o3-mini","reasoning_effort":"high"},"running":{"model":"o3-mini","reasoning_effort":"high"}}'
|
||||
|
||||
|
||||
|
||||
rdagent grade_summary --log_folder=<url_to_log>
|
||||
|
||||
Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
|
||||
|
||||
+21
-18
@@ -6,6 +6,8 @@ This will
|
||||
- autoamtically load dotenv
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv(".env")
|
||||
@@ -15,7 +17,7 @@ load_dotenv(".env")
|
||||
import subprocess
|
||||
from importlib.resources import path as rpath
|
||||
|
||||
import fire
|
||||
import typer
|
||||
|
||||
from rdagent.app.data_science.loop import main as data_science
|
||||
from rdagent.app.general_model.general_model import (
|
||||
@@ -27,7 +29,9 @@ from rdagent.app.qlib_rd_loop.model import main as fin_model
|
||||
from rdagent.app.qlib_rd_loop.quant import main as fin_quant
|
||||
from rdagent.app.utils.health_check import health_check
|
||||
from rdagent.app.utils.info import collect_info
|
||||
from rdagent.log.mle_summary import grade_summary
|
||||
from rdagent.log.mle_summary import grade_summary as grade_summary
|
||||
|
||||
app = typer.Typer()
|
||||
|
||||
|
||||
def ui(port=19899, log_dir="", debug=False, data_science=False):
|
||||
@@ -57,19 +61,18 @@ def server_ui(port=19899):
|
||||
subprocess.run(["python", "rdagent/log/server/app.py", f"--port={port}"])
|
||||
|
||||
|
||||
def app():
|
||||
fire.Fire(
|
||||
{
|
||||
"fin_factor": fin_factor,
|
||||
"fin_factor_report": fin_factor_report,
|
||||
"fin_model": fin_model,
|
||||
"fin_quant": fin_quant,
|
||||
"general_model": general_model,
|
||||
"ui": ui,
|
||||
"health_check": health_check,
|
||||
"collect_info": collect_info,
|
||||
"data_science": data_science,
|
||||
"grade_summary": grade_summary,
|
||||
"server_ui": server_ui,
|
||||
}
|
||||
)
|
||||
app.command(name="fin_factor")(fin_factor)
|
||||
app.command(name="fin_model")(fin_model)
|
||||
app.command(name="fin_quant")(fin_quant)
|
||||
app.command(name="fin_factor_report")(fin_factor_report)
|
||||
app.command(name="general_model")(general_model)
|
||||
app.command(name="data_science")(data_science)
|
||||
app.command(name="grade_summary")(grade_summary)
|
||||
app.command(name="ui")(ui)
|
||||
app.command(name="server_ui")(server_ui)
|
||||
app.command(name="health_check")(health_check)
|
||||
app.command(name="collect_info")(collect_info)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app()
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import fire
|
||||
import typer
|
||||
from typing_extensions import Annotated
|
||||
|
||||
from rdagent.app.data_science.conf import DS_RD_SETTING
|
||||
from rdagent.core.utils import import_class
|
||||
@@ -10,14 +13,15 @@ from rdagent.scenarios.data_science.loop import DataScienceRDLoop
|
||||
|
||||
|
||||
def main(
|
||||
path: str | None = None,
|
||||
checkout: bool | str | Path = True,
|
||||
step_n: int | None = None,
|
||||
loop_n: int | None = None,
|
||||
timeout: str | None = None,
|
||||
path: Optional[str] = None,
|
||||
checkout: Annotated[bool, typer.Option("--checkout/--no-checkout", "-c/-C")] = True,
|
||||
checkout_path: Optional[str] = None,
|
||||
step_n: Optional[int] = None,
|
||||
loop_n: Optional[int] = None,
|
||||
timeout: Optional[str] = None,
|
||||
competition="bms-molecular-translation",
|
||||
replace_timer=True,
|
||||
exp_gen_cls: str | None = None,
|
||||
exp_gen_cls: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
|
||||
@@ -26,12 +30,11 @@ def main(
|
||||
path :
|
||||
A path like `$LOG_PATH/__session__/1/0_propose`. This indicates that we restore the state after finishing step 0 in loop 1.
|
||||
checkout :
|
||||
Used only when a path is provided.
|
||||
Can be True, False, or a path.
|
||||
Default is True.
|
||||
Used to control the log session path. Boolean type, default is True.
|
||||
- If True, the new loop will use the existing folder and clear logs for sessions after the one corresponding to the given path.
|
||||
- If False, the new loop will use the existing folder but keep the logs for sessions after the one corresponding to the given path.
|
||||
- If a path (or a str like Path) is provided, the new loop will be saved to that path, leaving the original path unchanged.
|
||||
checkout_path:
|
||||
If a checkout_path (or a str like Path) is provided, the new loop will be saved to that path, leaving the original path unchanged.
|
||||
step_n :
|
||||
Number of steps to run; if None, the process will run indefinitely until an error or KeyboardInterrupt occurs.
|
||||
loop_n :
|
||||
@@ -52,6 +55,9 @@ def main(
|
||||
dotenv run -- python rdagent/app/data_science/loop.py [--competition titanic] $LOG_PATH/__session__/1/0_propose --step_n 1 # `step_n` is an optional parameter
|
||||
rdagent kaggle --competition playground-series-s4e8 # This command is recommended.
|
||||
"""
|
||||
if not checkout_path is None:
|
||||
checkout = Path(checkout_path)
|
||||
|
||||
if competition is not None:
|
||||
DS_RD_SETTING.competition = competition
|
||||
|
||||
|
||||
@@ -4,9 +4,11 @@ Factor workflow with session control
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, Optional
|
||||
|
||||
import fire
|
||||
import typer
|
||||
from typing_extensions import Annotated
|
||||
|
||||
from rdagent.app.qlib_rd_loop.conf import FACTOR_PROP_SETTING
|
||||
from rdagent.components.workflow.rd_loop import RDLoop
|
||||
@@ -27,11 +29,12 @@ class FactorRDLoop(RDLoop):
|
||||
|
||||
|
||||
def main(
|
||||
path: str | None = None,
|
||||
step_n: int | None = None,
|
||||
loop_n: int | None = None,
|
||||
path: Optional[str] = None,
|
||||
step_n: Optional[int] = None,
|
||||
loop_n: Optional[int] = None,
|
||||
all_duration: str | None = None,
|
||||
checkout: bool | str | Path = True,
|
||||
checkout: Annotated[bool, typer.Option("--checkout/--no-checkout", "-c/-C")] = True,
|
||||
checkout_path: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
Auto R&D Evolving loop for fintech factors.
|
||||
@@ -43,6 +46,9 @@ def main(
|
||||
dotenv run -- python rdagent/app/qlib_rd_loop/factor.py $LOG_PATH/__session__/1/0_propose --step_n 1 # `step_n` is a optional paramter
|
||||
|
||||
"""
|
||||
if not checkout_path is None:
|
||||
checkout = Path(checkout_path)
|
||||
|
||||
if path is None:
|
||||
model_loop = FactorRDLoop(FACTOR_PROP_SETTING)
|
||||
else:
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
import os
|
||||
import socket
|
||||
|
||||
import docker
|
||||
import fire
|
||||
import litellm
|
||||
import typer
|
||||
from litellm import completion, embedding
|
||||
from litellm.utils import ModelResponse
|
||||
from typing_extensions import Annotated
|
||||
|
||||
from rdagent.log import rdagent_logger as logger
|
||||
from rdagent.utils.env import cleanup_container
|
||||
@@ -43,10 +50,121 @@ def check_and_list_free_ports(start_port=19899, max_ports=10) -> None:
|
||||
logger.info(f"Port 19899 is not occupied, you can run the `rdagent ui` command")
|
||||
|
||||
|
||||
def health_check():
|
||||
def test_chat(chat_model, chat_api_key, chat_api_base):
|
||||
logger.info(f"🧪 Testing chat model: {chat_model}")
|
||||
try:
|
||||
if chat_api_base is None:
|
||||
response: ModelResponse = completion(
|
||||
model=chat_model,
|
||||
api_key=chat_api_key,
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello!"},
|
||||
],
|
||||
)
|
||||
else:
|
||||
response: ModelResponse = completion(
|
||||
model=chat_model,
|
||||
api_key=chat_api_key,
|
||||
api_base=chat_api_base,
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello!"},
|
||||
],
|
||||
)
|
||||
logger.info(f"✅ Chat test passed.")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"❌ Chat test failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_embedding(embedding_model, embedding_api_key, embedding_api_base):
|
||||
logger.info(f"🧪 Testing embedding model: {embedding_model}")
|
||||
try:
|
||||
response = embedding(
|
||||
model=embedding_model,
|
||||
api_key=embedding_api_key,
|
||||
api_base=embedding_api_base,
|
||||
input="Hello world!",
|
||||
)
|
||||
logger.info("✅ Embedding test passed.")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"❌ Embedding test failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def env_check():
|
||||
if "BACKEND" not in os.environ:
|
||||
logger.warning(
|
||||
f"We did not find BACKEND in your configuration, please add it to your .env file. "
|
||||
f"You can run a command like this: `dotenv set BACKEND rdagent.oai.backend.LiteLLMAPIBackend`"
|
||||
)
|
||||
|
||||
if "DEEPSEEK_API_KEY" in os.environ:
|
||||
chat_api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
chat_model = os.getenv("CHAT_MODEL")
|
||||
embedding_model = os.getenv("EMBEDDING_MODEL")
|
||||
embedding_api_key = os.getenv("LITELLM_PROXY_API_KEY")
|
||||
embedding_api_base = os.getenv("LITELLM_PROXY_API_BASE")
|
||||
if "DEEPSEEK_API_BASE" in os.environ:
|
||||
chat_api_base = os.getenv("DEEPSEEK_API_BASE")
|
||||
elif "OPENAI_API_BASE" in os.environ:
|
||||
chat_api_base = os.getenv("OPENAI_API_BASE")
|
||||
else:
|
||||
chat_api_base = None
|
||||
elif "OPENAI_API_KEY" in os.environ:
|
||||
chat_api_key = os.getenv("OPENAI_API_KEY")
|
||||
chat_api_base = os.getenv("OPENAI_API_BASE")
|
||||
chat_model = os.getenv("CHAT_MODEL")
|
||||
embedding_model = os.getenv("EMBEDDING_MODEL")
|
||||
embedding_api_key = chat_api_key
|
||||
embedding_api_base = chat_api_base
|
||||
else:
|
||||
logger.error("No valid configuration was found, please check your .env file.")
|
||||
|
||||
logger.info("🚀 Starting test...\n")
|
||||
result_embedding = test_embedding(
|
||||
embedding_model=embedding_model, embedding_api_key=embedding_api_key, embedding_api_base=embedding_api_base
|
||||
)
|
||||
result_chat = test_chat(chat_model=chat_model, chat_api_key=chat_api_key, chat_api_base=chat_api_base)
|
||||
|
||||
if result_chat and result_embedding:
|
||||
logger.info("✅ All tests completed.")
|
||||
else:
|
||||
logger.error(" One or more tests failed. Please check credentials or model support.")
|
||||
|
||||
|
||||
def health_check(
|
||||
check_env: Annotated[bool, typer.Option("--check-env/--no-check-env", "-e/-E")] = True,
|
||||
check_docker: Annotated[bool, typer.Option("--check-docker/--no-check-docker", "-d/-D")] = True,
|
||||
check_ports: Annotated[bool, typer.Option("--check-ports/--no-check-ports", "-p/-P")] = True,
|
||||
):
|
||||
"""
|
||||
Check that docker is installed correctly,
|
||||
and that the ports used in the sample README are not occupied.
|
||||
Run the RD-Agent health check:
|
||||
- Check if Docker is available
|
||||
- Check that the default ports are not occupied
|
||||
- (Optional) Check that the API Key and model are configured correctly.
|
||||
|
||||
Args:
|
||||
check_env (bool): Whether to check API Key and model configuration.
|
||||
check_docker (bool): Checks if Docker is installed and running.
|
||||
check_ports (bool): Whether to check if the default port (19899) is occupied.
|
||||
"""
|
||||
check_docker()
|
||||
check_and_list_free_ports()
|
||||
check_any = False
|
||||
|
||||
if check_env:
|
||||
check_any = True
|
||||
env_check()
|
||||
if check_docker:
|
||||
check_any = True
|
||||
check_docker()
|
||||
if check_ports:
|
||||
check_any = True
|
||||
check_and_list_free_ports()
|
||||
|
||||
if not check_any:
|
||||
logger.warning("⚠️ All health check items are disabled. Please enable at least one check.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
typer.run(health_check)
|
||||
|
||||
@@ -242,7 +242,7 @@ def summarize_folder(log_folder: Path, hours: int | None = None) -> None:
|
||||
# }
|
||||
|
||||
|
||||
def grade_summary(log_folder: str | Path) -> None:
|
||||
def grade_summary(log_folder: str) -> None:
|
||||
"""
|
||||
Generate test scores for log traces in the log folder and save the summary.
|
||||
"""
|
||||
|
||||
@@ -405,7 +405,7 @@ class DefaultSampler(DataSampler):
|
||||
if (
|
||||
str(fp.name) in sample_used_file_names
|
||||
or str(fp.stem) in sample_used_file_names
|
||||
or fp in sample_used_file_paths
|
||||
or fp in sample_used_file_names
|
||||
):
|
||||
used_files.append(fp)
|
||||
else:
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
R\&D-Agent Data Science Pipeline supports automated R\&D optimization for competitions hosted on the Kaggle platform, as well as **custom user-defined datasets**.
|
||||
|
||||
Specifically, you need to prepare files in a structure similar to the provided example. Here, we use the `arf-12-hour-prediction-task` dataset as an illustration.
|
||||
Specifically, you need to prepare files in a structure similar to the provided example. Here, we use the `arf-12-hours-prediction-task` dataset as an illustration.
|
||||
|
||||
## arf-12-hour-prediction-task Introduction
|
||||
## arf-12-hours-prediction-task Introduction
|
||||
|
||||
> Acute Respiratory Failure (ARF) is a life-threatening condition that often develops rapidly in critically ill patients. Accurate early prediction of ARF is essential in Intensive Care Units (ICUs) to enable timely clinical interventions and effective resource allocation. In this task, you are required to build a machine learning model that predicts whether a patient will develop ARF within the next **12 hours**, using multivariate clinical time-series data.
|
||||
>
|
||||
@@ -12,28 +12,54 @@ Specifically, you need to prepare files in a structure similar to the provided e
|
||||
|
||||
## Example Folder Structure
|
||||
|
||||
* `arf-12-hour-prediction-task` (Task Name)
|
||||
* `source_data` (**required**)
|
||||
|
||||
* `train` (**required**)
|
||||
* `arf-12-hours-prediction-task` (Task Name, **required**)
|
||||
|
||||
* `X.npz` (features)
|
||||
* `ARF_12h.csv` (labels)
|
||||
* `test` (**required**)
|
||||
* `prepare.py` Used for data preprocessing to split the raw data into: *training data*, *test data*, *formatted submission file*, and *standard answer file*.
|
||||
|
||||
* `playground-series-s4e9` (Task Name, **required**)
|
||||
|
||||
* `prepare.py` (**required**): Used for data preprocessing to split the raw data into: *training data*, *test data*, *formatted submission file*, and *standard answer file*.
|
||||
|
||||
NOTE: Due to the large size of the raw data, we do not show the raw data in this project, if you want to see the raw data, you can download the full dataset through the link at the bottom.
|
||||
|
||||
* `arf-12-hours-prediction-task` (Task Name)
|
||||
|
||||
* `description.md` (**required**): A detailed description of the task, including sections such as *Task Description*, *Objective*, *Data Description*, *Data usage Notes*, *Modeling*, *Evaluation* and *Submission Format*.
|
||||
|
||||
* `X.npz` (features)
|
||||
* `ARF_12h.csv` (labels to be predicted)
|
||||
* `description.md` (**required**): A detailed description of the task, including sections such as Task Description, Objective, Data Description, Modeling, and Submission Format.
|
||||
* `sample.py` (**optional**): A Python script to sample the dataset for debugging purposes. If not provided, a default sampling logic in R\&D-Agent will be used. Refer to the `create_debug_data` function in `rdagent/scenarios/data_science/debug/data.py`.
|
||||
|
||||
* `eval` (**required**)
|
||||
* `playground-series-s4e9` (Task Name)
|
||||
|
||||
* `arf-12-hour-prediction-task` (Task Name, **required**)
|
||||
* `description.md` (**required**): A detailed description of the task, including sections such as *Task Description*, *Goal*, *Evaluation*, *Data Description*, and *Submission Format*.
|
||||
|
||||
* `eval` (**optional**)
|
||||
|
||||
* `arf-12-hours-prediction-task` (Task Name, **optional**)
|
||||
|
||||
* `grade.py`: Calculates the task score on the test dataset.
|
||||
* `submission_test.csv`: Corresponding labels from the previously provided `test/ARF_12h.csv`.
|
||||
* `valid.py`: Checks the validity of the generated `submission.csv` file.
|
||||
|
||||
The complete dataset folder for `arf-12-hour-prediction-task` can be downloaded from:
|
||||
[https://github.com/SunsetWolf/rdagent\_resource/releases/download/med\_data/rdagent\_datas\_science\_customData\_example.zip](https://github.com/SunsetWolf/rdagent_resource/releases/download/med_data/rdagent_datas_science_customData_example.zip).
|
||||
* `playground-series-s4e9` (Task Name, **optional**)
|
||||
|
||||
The original dataset is sourced from PhysioNet. You can apply for an account at [PhysioNet](https://physionet.org/) and then request access to the FIDDLE preprocessed data: [FIDDLE Dataset](https://physionet.org/content/mimic-eicu-fiddle-feature/1.0.0/).
|
||||
* `grade.py`: Calculates the task score on the test dataset.
|
||||
* `valid.py`: Checks the validity of the generated `submission.csv` file.
|
||||
|
||||
NOTE: You don't need to create the `eval` folder if you are ignoring test set scores.
|
||||
|
||||
---
|
||||
|
||||
The complete dataset folder for `arf-12-hours-prediction-task` can be downloaded from [here](https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip).
|
||||
|
||||
The raw dataset for `arf-12-hours-prediction-task` comes from PhysioNet. You can apply for an account at [PhysioNet](https://physionet.org/) and then request access to the FIDDLE preprocessed data: [FIDDLE Dataset](https://physionet.org/content/mimic-eicu-fiddle-feature/1.0.0/).
|
||||
|
||||
---
|
||||
|
||||
The complete dataset folder for `playground-series-s4e9` can be downloaded from [here](https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip).
|
||||
|
||||
The raw dataset for `playground-series-s4e9` comes from Kaggle. You can apply for an account at [Kaggle](https://www.kaggle.com/) and then request access to the [competition dataset](https://www.kaggle.com/competitions/playground-series-s4e9/data).
|
||||
|
||||
---
|
||||
|
||||
**NOTE:** For more information about the dataset, please refer to the [documentation](https://rdagent.readthedocs.io/en/latest/scens/data_science.html).
|
||||
|
||||
@@ -1,62 +0,0 @@
|
||||
# ARF 12-Hour Prediction Task
|
||||
|
||||
## Overview
|
||||
|
||||
### Description
|
||||
|
||||
Acute Respiratory Failure (ARF) is a life-threatening condition that often develops rapidly in critically ill patients. Accurate early prediction of ARF is crucial in intensive care units (ICUs) to enable timely clinical interventions and resource allocation. In this task, you are asked to build a machine learning model that predicts whether a patient will develop ARF within the next **12 hours**, based on multivariate clinical time series data.
|
||||
|
||||
The dataset is extracted from electronic health records (EHRs) and preprocessed using the **FIDDLE** pipeline to generate structured temporal features for each patient.
|
||||
|
||||
### Objective
|
||||
|
||||
Your goal is to develop a binary classification model that takes a 12-hour time series as input and outputs the probability of ARF onset in the following 12 hours.
|
||||
|
||||
---
|
||||
|
||||
## Data Description
|
||||
|
||||
The dataset is divided into two directories:
|
||||
|
||||
* `train/`
|
||||
|
||||
* `ARF_12h.csv`: `ID` & `ARF_ONSET_HOUR` & Binary labels (`ARF_LABEL`) for training samples.
|
||||
* `X.npz`: 3D sparse array of shape `(N, T, D)`:
|
||||
|
||||
* `N`: number of training samples
|
||||
* `T`: time steps (one per hour)
|
||||
* `D`: number of features
|
||||
|
||||
* `test/`
|
||||
|
||||
* `ARF_12h.csv`: `ID` & `ARF_ONSET_HOUR`.
|
||||
* `X.npz`: Test feature set in the same format as training data.
|
||||
|
||||
The `.npz` files store sparse matrices and are loaded using the [`sparse`](https://github.com/pydata/sparse) library. Each matrix is converted to dense format before model input. (DO NOT USE scipy.sparse)
|
||||
e.g.
|
||||
```
|
||||
import sparse
|
||||
X = sparse.load_npz("<url>").todense()
|
||||
```
|
||||
Then, you can use `X.transpose(0, 2, 1)` to transpose the shape of X from (N, T, D) to (N, D, T)
|
||||
|
||||
---
|
||||
|
||||
## Modeling
|
||||
|
||||
Each sample is a multivariate time series representing 12 hours of clinical observations. Your model should learn temporal and cross-feature dependencies to estimate the likelihood of ARF.
|
||||
|
||||
* **Output**: Probability score ∈ \[0, 1]
|
||||
* **Loss Function**: `CrossEntropyLoss` or equivalent
|
||||
* **Evaluation Metric**: **AUROC** (Area Under the Receiver Operating Characteristic Curve)
|
||||
|
||||
---
|
||||
|
||||
## Submission Format
|
||||
```
|
||||
ID,ARF_LABEL
|
||||
0,0.473
|
||||
1,0.652
|
||||
2,0.129
|
||||
...
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
# Competition name: ARF 12-Hour Prediction Task
|
||||
|
||||
## Overview
|
||||
|
||||
### Description
|
||||
|
||||
Acute Respiratory Failure (ARF) is a life-threatening condition that often develops rapidly in critically ill patients. Accurate early prediction of ARF is crucial in intensive care units (ICUs) to enable timely clinical interventions and resource allocation. In this task, you are asked to build a machine learning model that predicts whether a patient will develop ARF within the next **12 hours**, based on multivariate clinical time series data.
|
||||
|
||||
The dataset is extracted from electronic health records (EHRs) and preprocessed using the **FIDDLE** pipeline to generate structured temporal features for each patient.
|
||||
|
||||
### Objective
|
||||
|
||||
**Your Goal** is to develop a binary classification model that takes a 12-hour time series as input and predicts whether ARF will occur (1) or not (0) in the following 12 hours.
|
||||
|
||||
---
|
||||
|
||||
## Data Description
|
||||
|
||||
1. train/ARF_12h.csv: A CSV file containing the ICU stay ID, the hour of ARF onset, and the binary label indicating whether ARF will occur in the next 12 hours.
|
||||
|
||||
* Columns: ID, ARF_ONSET_HOUR, ARF_LABEL
|
||||
|
||||
2. train/X.npz: N × T × D sparse tensor containing time-dependent features.
|
||||
|
||||
* N: Number of samples (number of ICU stays)
|
||||
* T: Time step (12 hours of records per sample)
|
||||
* D: Dynamic feature dimension (how many features per hour)
|
||||
|
||||
3. test/ARF_12h.csv: Ground truth labels (used for evaluation only).
|
||||
|
||||
4. test/X.npz: Test feature set in the same format as training data.
|
||||
|
||||
---
|
||||
|
||||
## Data usage Notes
|
||||
|
||||
To load the features, you need python and the sparse package.
|
||||
|
||||
import sparse
|
||||
|
||||
X = sparse.load_npz("<url>/X.npz").todense()
|
||||
|
||||
|
||||
To load the labels, use pandas or an alternative csv reader.
|
||||
|
||||
import pandas as pd
|
||||
|
||||
df = pd.read_csv("<url>/ARF_12h.csv")
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Modeling
|
||||
|
||||
Each sample is a 12-hour multivariate time series of ICU patient observations, represented as a tensor of shape (12, D).
|
||||
The goal is to predict whether the patient will develop ARF (1) or not (0) in the following 12 hours.
|
||||
|
||||
* **Input**: 12 × D matrix of clinical features
|
||||
* **Output**: Binary prediction: 0 (no ARF) or 1 (ARF onset)
|
||||
* **Loss Function**: BCEWithLogitsLoss, CrossEntropyLoss or equivalent
|
||||
* **Evaluation Metric**: **AUROC** (Area Under the Receiver Operating Characteristic Curve)
|
||||
|
||||
Note: Although the output is binary, AUROC evaluates the ranking quality of predicted scores. Therefore, your model should output a confidence score during training, which is then thresholded to produce 0 or 1 for final submission.
|
||||
|
||||
---
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Area Under the Receiver Operating Characteristic curve (AUROC)
|
||||
|
||||
The submissions are scored according to the area under the receiver operating characteristic curve. AUROC is defined as:
|
||||
|
||||
$$
|
||||
\text{AUROC} = \frac{1}{|P| \cdot |N|} \sum_{i \in P} \sum_{j \in N} \left[ \mathbb{1}(s_i > s_j) + \frac{1}{2} \cdot \mathbb{1}(s_i = s_j) \right]
|
||||
$$
|
||||
|
||||
AUROC reflects the model's ability to rank positive samples higher than negative ones. A score of 1.0 means perfect discrimination, and 0.5 means random guessing.
|
||||
|
||||
### Submission Format
|
||||
|
||||
For each `ID'' in the ARF_12h.csv file of the test dataset, you must predict whether ARF will occur (label = 1) or not (label = 0) in the following 12 hours(ARF_LABEL), based on the X.npz (sparse tensor, time-varying feature). The file should contain the following format:
|
||||
|
||||
ID,ARF_LABEL
|
||||
246505,0
|
||||
291335,0
|
||||
286713,0
|
||||
etc.
|
||||
|
||||
|
||||
Note: Although the submission is binary, AUROC evaluates the ranking quality of your model. It is recommended to output probabilities during training and apply a threshold (e.g., 0.5) to convert to binary labels for submission.
|
||||
|
||||
---
|
||||
+27
@@ -19,6 +19,7 @@ def sample_and_copy_subfolder(
|
||||
feature_path = input_dir / "X.npz"
|
||||
label_path = input_dir / "ARF_12h.csv"
|
||||
|
||||
# Load sparse features and label
|
||||
X_sparse = sparse.load_npz(feature_path)
|
||||
df_label = pd.read_csv(label_path)
|
||||
|
||||
@@ -35,6 +36,29 @@ def sample_and_copy_subfolder(
|
||||
|
||||
print(f"[INFO] Sampled {n_keep} of {N} from {input_dir.name}")
|
||||
|
||||
# Copy additional files
|
||||
for f in input_dir.glob("*"):
|
||||
if f.name not in {"X.npz", "ARF_12h.csv"} and f.is_file():
|
||||
shutil.copy(f, output_dir / f.name)
|
||||
print(f"[COPY] Extra file: {f.name}")
|
||||
|
||||
|
||||
def copy_other_file(source: Path, target: Path):
|
||||
for item in source.iterdir():
|
||||
if item.name in {"train", "test"}:
|
||||
continue
|
||||
|
||||
relative_path = item.relative_to(source)
|
||||
target_path = target / relative_path
|
||||
|
||||
if item.is_dir():
|
||||
shutil.copytree(item, target_path, dirs_exist_ok=True)
|
||||
print(f"[COPY DIR] {item} -> {target_path}")
|
||||
elif item.is_file():
|
||||
target_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(item, target_path)
|
||||
print(f"[COPY FILE] {item} -> {target_path}")
|
||||
|
||||
|
||||
def create_debug_data(
|
||||
dataset_path: str,
|
||||
@@ -56,6 +80,9 @@ def create_debug_data(
|
||||
min_num=min_num,
|
||||
seed=42 if sub == "train" else 123,
|
||||
)
|
||||
print(dataset_root.resolve())
|
||||
print(output_root.resolve())
|
||||
copy_other_file(source=dataset_root, target=output_root)
|
||||
|
||||
print(f"\n[INFO] Sampling complete → Output in: {output_root}")
|
||||
|
||||
|
|
-8
@@ -4,14 +4,6 @@ import pandas as pd
|
||||
from sklearn.metrics import roc_auc_score
|
||||
|
||||
|
||||
class InvalidSubmissionError(Exception):
|
||||
"""
|
||||
A custom exception for when the agent submission cannot be graded.
|
||||
"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
def prepare_for_auroc_metric(submission: pd.DataFrame, answers: pd.DataFrame, id_col: str, target_col: str) -> dict:
|
||||
|
||||
# Answers checks
|
||||
@@ -0,0 +1,87 @@
|
||||
import datetime
|
||||
import json
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from sklearn.metrics import mean_absolute_error, mean_squared_error, roc_auc_score
|
||||
|
||||
|
||||
class InvalidSubmissionError(Exception):
|
||||
"""
|
||||
A custom exception for when the agent submission cannot be graded.
|
||||
"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
def prepare_for_metric(submission: pd.DataFrame, answers: pd.DataFrame) -> dict:
|
||||
|
||||
if "id" not in submission.columns or "id" not in answers.columns:
|
||||
raise InvalidSubmissionError("Both submission and answers DataFrames must contain an 'id' column.")
|
||||
|
||||
if "price" not in submission.columns:
|
||||
raise InvalidSubmissionError("Submission DataFrame must contain 'price' columns.")
|
||||
|
||||
assert "price" in answers.columns, "Answers DataFrame must contain 'price' columns."
|
||||
|
||||
if len(submission) != len(answers):
|
||||
raise InvalidSubmissionError("Submission must be the same length as the answers.")
|
||||
|
||||
answers_sorted = answers.sort_values("id")
|
||||
submission_sorted = submission.sort_values("id")
|
||||
|
||||
if (submission_sorted["id"].values != answers_sorted["id"].values).any():
|
||||
raise InvalidSubmissionError("Submission and answers have mismatched 'id' columns")
|
||||
|
||||
y_true = answers_sorted[["price"]].to_numpy()
|
||||
y_score = submission_sorted[["price"]].to_numpy()
|
||||
|
||||
return {"y_true": y_true, "y_score": y_score}
|
||||
|
||||
|
||||
def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
|
||||
metric_inputs = prepare_for_metric(submission, answers)
|
||||
return np.sqrt(mean_squared_error(metric_inputs["y_true"], metric_inputs["y_score"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
submission_path = "submission.csv"
|
||||
gt_submission_path = "submission_test.csv"
|
||||
submission = pd.read_csv(submission_path)
|
||||
answers = pd.read_csv(gt_submission_path)
|
||||
score = grade(submission=submission, answers=answers)
|
||||
|
||||
# This `thresholds` can be customized according to the leaderboard page of the Kaggle website and your own needs.
|
||||
# Refs: https://www.kaggle.com/competitions/playground-series-s4e9/leaderboard
|
||||
thresholds = {
|
||||
"gold": 62917.05988,
|
||||
"silver": 62945.91714,
|
||||
"bronze": 62958.13747,
|
||||
"median": 63028.69429,
|
||||
}
|
||||
|
||||
# The output must be in json format. To configure the full output,
|
||||
# you can run the command `rdagent grade_summary --log_folder` to summarize the scores at the end of the program.
|
||||
# If you don't need it, you can just provide the `competition_id`` and `score``.
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"competition_id": "arf-12-hours-prediction-task",
|
||||
"score": score,
|
||||
"gold_threshold": thresholds["gold"],
|
||||
"silver_threshold": thresholds["silver"],
|
||||
"bronze_threshold": thresholds["bronze"],
|
||||
"median_threshold": thresholds["median"],
|
||||
"any_medal": bool(score >= thresholds["bronze"]),
|
||||
"gold_medal": bool(score >= thresholds["gold"]),
|
||||
"silver_medal": bool(score >= thresholds["silver"]),
|
||||
"bronze_medal": bool(score >= thresholds["bronze"]),
|
||||
"above_median": bool(score >= thresholds["median"]),
|
||||
"submission_exists": True,
|
||||
"valid_submission": True,
|
||||
"is_lower_better": False,
|
||||
"created_at": str(datetime.datetime.now().isoformat()),
|
||||
"submission_path": submission_path,
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,21 @@
|
||||
from pathlib import Path
|
||||
|
||||
# Check if our submission file exists
|
||||
assert Path("submission.csv").exists(), "Error: submission.csv not found"
|
||||
|
||||
submission_lines = Path("submission.csv").read_text().splitlines() # 自动生成的
|
||||
test_lines = Path("submission_test.csv").read_text().splitlines() # test.csv
|
||||
|
||||
is_valid = len(submission_lines) == len(test_lines)
|
||||
|
||||
if is_valid:
|
||||
message = "submission.csv and submission_test.csv have the same number of lines."
|
||||
else:
|
||||
message = (
|
||||
f"submission.csv has {len(submission_lines)} lines, while submission_test.csv has {len(test_lines)} lines."
|
||||
)
|
||||
|
||||
print(message)
|
||||
|
||||
if not is_valid:
|
||||
raise AssertionError("Submission is invalid")
|
||||
@@ -0,0 +1,68 @@
|
||||
# Competition name: playground-series-s4e9
|
||||
|
||||
## Overview
|
||||
|
||||
**Welcome to the 2024 Kaggle Playground Series!** We plan to continue in the spirit of previous playgrounds, providing interesting and approachable datasets for our community to practice their machine learning skills, and anticipate a competition each month.
|
||||
|
||||
**Your Goal:** The goal of this competition is to predict the price of used cars based on various attributes.
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Root Mean Squared Error (RMSE)
|
||||
|
||||
Submissions are scored on the root mean squared error. RMSE is defined as:
|
||||
|
||||
$$
|
||||
\mathrm{RMSE} = \left( \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2 \right)^{\frac{1}{2}}
|
||||
$$
|
||||
|
||||
where $\hat{y}_i$ is the predicted value and $y_i$ is the original value for each instance $i$.
|
||||
|
||||
### Submission File
|
||||
|
||||
For each `id` in the test set, you must predict the `price` of the car. The file should contain a header and have the following format:
|
||||
|
||||
```
|
||||
id,price
|
||||
188533,43878.016
|
||||
188534,43878.016
|
||||
188535,43878.016
|
||||
etc.
|
||||
```
|
||||
|
||||
## Timeline
|
||||
- **Start Date** - September 1, 2024
|
||||
- **Entry Deadline** - Same as the Final Submission Deadline
|
||||
- **Team Merger Deadline** - Same as the Final Submission Deadline
|
||||
- **Final Submission Deadline** - September 30, 2024
|
||||
|
||||
All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
|
||||
|
||||
## About the Tabular Playground Series
|
||||
|
||||
The goal of the Tabular Playground Series is to provide the Kaggle community with a variety of fairly light-weight challenges that can be used to learn and sharpen skills in different aspects of machine learning and data science. The duration of each competition will generally only last a few weeks, and may have longer or shorter durations depending on the challenge. The challenges will generally use fairly light-weight datasets that are synthetically generated from real-world data, and will provide an opportunity to quickly iterate through various model and feature engineering ideas, create visualizations, etc.
|
||||
|
||||
### Synthetically-Generated Datasets
|
||||
|
||||
Using synthetic data for Playground competitions allows us to strike a balance between having real-world data (with named features) and ensuring test labels are not publicly available. This allows us to host competitions with more interesting datasets than in the past. While there are still challenges with synthetic data generation, the state-of-the-art is much better now than when we started the Tabular Playground Series two years ago, and that goal is to produce datasets that have far fewer artifacts. Please feel free to give us feedback on the datasets for the different competitions so that we can continue to improve!
|
||||
|
||||
## Prizes
|
||||
- 1st Place - Choice of Kaggle merchandise
|
||||
- 2nd Place - Choice of Kaggle merchandise
|
||||
- 3rd Place - Choice of Kaggle merchandise
|
||||
|
||||
**Please note**: In order to encourage more participation from beginners, Kaggle merchandise will only be awarded once per person in this series. If a person has previously won, we'll skip to the next team.
|
||||
|
||||
## Citation
|
||||
|
||||
Walter Reade and Ashley Chow. Regression of Used Car Prices. https://kaggle.com/competitions/playground-series-s4e9, 2024. Kaggle.
|
||||
|
||||
## Dataset Description
|
||||
|
||||
The dataset for this competition (both train and test) was generated from a deep learning model trained on the [Used Car Price Prediction Dataset](https://www.kaggle.com/datasets/taeefnajib/used-car-price-prediction-dataset). Feature distributions are close to, but not exactly the same, as the original. Feel free to use the original dataset as part of this competition, both to explore differences as well as to see whether incorporating the original in training improves model performance.
|
||||
|
||||
## Files
|
||||
|
||||
- **train.csv** - the training dataset; `price` is the continuous target
|
||||
- **test.csv** - the test dataset; your objective is to predict the value of `price` for each row
|
||||
- **sample_submission.csv** - a sample submission file in the correct format
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
import random
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import sparse
|
||||
|
||||
CURRENT_DIR = Path(__file__).resolve().parent
|
||||
ROOT_DIR = CURRENT_DIR.parent.parent
|
||||
|
||||
raw_feature_path = CURRENT_DIR / "X.npz"
|
||||
raw_label_path = CURRENT_DIR / "ARF_12h.csv"
|
||||
|
||||
public = ROOT_DIR / "arf-12-hour-prediction-task"
|
||||
private = ROOT_DIR / "eval" / "arf-12-hour-prediction-task"
|
||||
|
||||
if not (public / "test").exists():
|
||||
(public / "test").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not (public / "train").exists():
|
||||
(public / "train").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not private.exists():
|
||||
private.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
SEED = 42
|
||||
random.seed(SEED)
|
||||
np.random.seed(SEED)
|
||||
|
||||
X_sparse = sparse.load_npz(raw_feature_path) # COO matrix, shape: [N, D, T]
|
||||
df_label = pd.read_csv(raw_label_path) # Contains column 'ARF_LABEL'
|
||||
N = X_sparse.shape[0]
|
||||
|
||||
indices = np.arange(N)
|
||||
np.random.shuffle(indices)
|
||||
split = int(0.7 * N)
|
||||
train_idx, test_idx = indices[:split], indices[split:]
|
||||
|
||||
X_train = X_sparse[train_idx]
|
||||
X_test = X_sparse[test_idx]
|
||||
|
||||
df_train = df_label.iloc[train_idx].reset_index(drop=True)
|
||||
df_test = df_label.iloc[test_idx].reset_index(drop=True)
|
||||
|
||||
submission_df = df_test.copy()
|
||||
submission_df["ARF_LABEL"] = 0
|
||||
submission_df.drop(submission_df.columns.difference(["ID", "ARF_LABEL"]), axis=1, inplace=True)
|
||||
submission_df.to_csv(public / "sample_submission.csv", index=False)
|
||||
|
||||
df_test.to_csv(private / "submission_test.csv", index=False)
|
||||
|
||||
df_test.drop(["ARF_LABEL"], axis=1, inplace=True)
|
||||
df_test.to_csv(public / "test" / "ARF_12h.csv", index=False)
|
||||
sparse.save_npz(public / "test" / "X.npz", X_test)
|
||||
|
||||
sparse.save_npz(public / "train" / "X.npz", X_train)
|
||||
df_train.to_csv(public / "train" / "ARF_12h.csv", index=False)
|
||||
|
||||
assert (
|
||||
X_train.shape[0] == df_train.shape[0]
|
||||
), f"Mismatch: X_train rows ({X_train.shape[0]}) != df_train rows ({df_train.shape[0]})"
|
||||
assert (
|
||||
X_test.shape[0] == df_test.shape[0]
|
||||
), f"Mismatch: X_test rows ({X_test.shape[0]}) != df_test rows ({df_test.shape[0]})"
|
||||
assert df_test.shape[1] == 2, "Public test set should have 2 columns"
|
||||
assert df_train.shape[1] == 3, "Public train set should have 3 columns"
|
||||
assert len(df_train) + len(df_test) == len(
|
||||
df_label
|
||||
), "Length of new_train and new_test should equal length of old_train"
|
||||
@@ -0,0 +1,42 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from sklearn.model_selection import train_test_split
|
||||
|
||||
|
||||
def prepare(raw: Path, public: Path, private: Path):
|
||||
|
||||
# Create train and test splits from train set
|
||||
old_train = pd.read_csv(raw / "train.csv")
|
||||
new_train, new_test = train_test_split(old_train, test_size=0.1, random_state=0)
|
||||
|
||||
# Create sample submission
|
||||
sample_submission = new_test.copy()
|
||||
sample_submission["price"] = 43878.016
|
||||
sample_submission.drop(sample_submission.columns.difference(["id", "price"]), axis=1, inplace=True)
|
||||
sample_submission.to_csv(public / "sample_submission.csv", index=False)
|
||||
|
||||
# Create private files
|
||||
new_test.to_csv(private / "submission_test.csv", index=False)
|
||||
|
||||
# Create public files visible to agents
|
||||
new_train.to_csv(public / "train.csv", index=False)
|
||||
new_test.drop(["price"], axis=1, inplace=True)
|
||||
new_test.to_csv(public / "test.csv", index=False)
|
||||
|
||||
# Checks
|
||||
assert new_test.shape[1] == 12, "Public test set should have 12 columns"
|
||||
assert new_train.shape[1] == 13, "Public train set should have 13 columns"
|
||||
assert len(new_train) + len(new_test) == len(
|
||||
old_train
|
||||
), "Length of new_train and new_test should equal length of old_train"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
competitions = "playground-series-s4e9"
|
||||
raw = Path(__file__).resolve().parent
|
||||
prepare(
|
||||
raw=raw,
|
||||
public=raw.parent.parent / competitions,
|
||||
private=raw.parent.parent / "eval" / competitions,
|
||||
)
|
||||
Reference in New Issue
Block a user