From 08fc2dcef364627e5475d69c423da2d8cdf159d6 Mon Sep 17 00:00:00 2001 From: Linlang <30293408+SunsetWolf@users.noreply.github.com> Date: Thu, 17 Apr 2025 22:02:49 +0800 Subject: [PATCH] chore: modify kaggle docs & Adding ds_loop at program entry (#786) * modify kaggle docs * optimise code based on comments * Update docs/scens/kaggle_agent.rst * fix docs build error --------- Co-authored-by: you-n-g --- README.md | 14 +++---- docs/scens/kaggle_agent.rst | 82 +++++++++++-------------------------- rdagent/app/cli.py | 4 +- 3 files changed, 31 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index e7631b4a..8a8e1b64 100644 --- a/README.md +++ b/README.md @@ -200,23 +200,21 @@ The **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)** is implemented b ```bash # Generally, you can run the Kaggle competition program with the following command: rdagent kaggle --competition + + # Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example: - # Specifically, you will need to first prepare some competition description files and configure the competition description file path, which you can follow for this specific example: - - # 1. Prepare the competition description files - wget https://github.com/SunsetWolf/rdagent_resource/releases/download/kaggle_data/kaggle_data.zip - unzip kaggle_data.zip -d git_ignore_folder/kaggle_data + # 1. Install chromedriver. # 2. Add the competition description file path to the `.env` file. + mkdir -p ./git_ignore_folder/kaggle_data dotenv set KG_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/kaggle_data" # 3. run the application rdagent kaggle --competition sf-crime ``` > **Description of the above example:**
- > - Kaggle competition data, contains two parts: competition description file (json file) and competition dataset (zip file). We prepare the competition description file for you, the competition dataset will be downloaded automatically when you run the program, as in the example.
- > - If you want to download the competition description file automatically, you need to install chromedriver, The instructions for installing chromedriver can be found in the [documentation](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html#example-guide).
- > - The **Competition List Available** can be found [here](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html#competition-list-available).
+ > - Kaggle competition data is roughly divided into three sections: competition description file (json file) and complete dataset for the competition and simplified dataset for the competition.
+ > - The Kaggle competition data will be downloaded automatically, the download process depends on `chromedriver`, installation instructions can be found in the [documentation](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html#example-guide).
### 🖥️ Monitor the Application Results - You can run the following command for our demo program to see the run logs. diff --git a/docs/scens/kaggle_agent.rst b/docs/scens/kaggle_agent.rst index 9c53ee4c..364dd714 100644 --- a/docs/scens/kaggle_agent.rst +++ b/docs/scens/kaggle_agent.rst @@ -1,8 +1,8 @@ .. _kaggle_agent: -======================= -Kaggle Agent -======================= +=========================== +Data Science Agent - Kaggle +=========================== **🤖 Automated Feature Engineering & Model Tuning Evolution** ------------------------------------------------------------------------------------------ @@ -87,6 +87,7 @@ Here's an enhanced outline of the steps: .. code-block:: sh + mkdir -p /kaggle_data dotenv set KG_LOCAL_DATA_PATH /kaggle_data - 📥 **Download Competition Data** @@ -95,33 +96,22 @@ Here's an enhanced outline of the steps: - **How to get the competition description file** - - *Manual Download (General User Suggestions):* + - The competition description file is downloaded automatically when the programme is run, and the download process relies on ``chromedriver``, which can be installed as follows: - - Download the competition description file prepared in advance, and extract it to the specified directory. + .. code-block:: sh - .. code-block:: sh + # install chrome + wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb + sudo apt install ./google-chrome-stable_current_amd64.deb + google-chrome --version - wget https://github.com/SunsetWolf/rdagent_resource/releases/download/kaggle_data/kaggle_data.zip - unzip kaggle_data.zip -d /kaggle_data - - - *Automatic Download (Developer Suggestions):* - - - Alternatively, you can choose to download the competition description file automatically when you run the program, but it requires ``chromedriver`` to be installed, as follows: - - .. code-block:: sh - - # install chrome - wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb - sudo apt install ./google-chrome-stable_current_amd64.deb - google-chrome --version - - # install chromedriver - wget "https://storage.googleapis.com/chrome-for-testing-public/$(google-chrome --version | grep -oP '\d+\.\d+\.\d+\.\d+')/linux64/chromedriver-linux64.zip" - unzip chromedriver-linux64.zip - cd chromedriver-linux64 - sudo mv chromedriver /usr/local/bin - sudo chmod +x /usr/local/bin/chromedriver - chromedriver --version + # install chromedriver + wget "https://storage.googleapis.com/chrome-for-testing-public/$(google-chrome --version | grep -oP '\d+\.\d+\.\d+\.\d+')/linux64/chromedriver-linux64.zip" + unzip chromedriver-linux64.zip + cd chromedriver-linux64 + sudo mv chromedriver /usr/local/bin + sudo chmod +x /usr/local/bin/chromedriver + chromedriver --version - **How to get the competition dataset** @@ -134,6 +124,9 @@ Here's an enhanced outline of the steps: kaggle_data └── zip_files | └── sf-crime.zip + ├── sample + | └── sf-crime + | └── ... ├── sf-crime.json └── sf-crime └── ... @@ -142,7 +135,9 @@ Here's an enhanced outline of the steps: - ``kaggle_data/sf-crime.json:`` Competition description file. - - ``kaggle_data/sf-crime:`` The target folder for unzipping the competition dataset. + - ``kaggle_data/sf-crime:`` The target folder for unzipping the competition dataset. Complete dataset. + + - ``kaggle_data/sample/sf-crime:`` Simplified dataset based on the complete dataset. Used to quickly verify that the code works. - 🗳️ **Join the competition** @@ -170,37 +165,6 @@ Here's an enhanced outline of the steps: - Else: You can download the prediction results from the UI interface and submit them manually. For more details, refer to the :doc:`UI guide <../ui>`. -📋 Competition List Available -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| **index** | **Competition Name** | **Task** | **Modal** | **ID** | -+===========+===================================+==================+===========+=========================================================================================================+ -| 01 | Media Campaign Cost Dataset | Regression | Tabular | `playground-series-s3e11 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 02 | Wild Blueberry Yield Dataset | Regression | Tabular | `playground-series-s3e14 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 03 | Crab Age Dataset | Regression | Tabular | `playground-series-s3e16 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 04 | Flood Prediction Dataset | Regression | Tabular | `playground-series-s4e5 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 05 | Used Car Prices Dataset | Regression | Tabular | `playground-series-s4e9 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 06 | Cirrhosis Outcomes Dataset | Multi-Class | Tabular | `playground-series-s3e26 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 07 | San Francisco Crime Classification| Multi-Class | Tabular | `sf-crime `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 08 | Poisonous Mushrooms Dataset | Classification | Tabular | `playground-series-s4e8 `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 09 | Spaceship Titanic | Classification | Tabular | `spaceship-titanic `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 10 | Forest Cover Type Prediction | Classification | Tabular | `forest-cover-type-prediction `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| 11 | Digit Recognizer | Classification | Image | `digit-recognizer `_ | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ -| To be continued ... | -+-----------+-----------------------------------+------------------+-----------+---------------------------------------------------------------------------------------------------------+ - 🎨 Customize one template for a new competition diff --git a/rdagent/app/cli.py b/rdagent/app/cli.py index 75569f1b..83bc24aa 100644 --- a/rdagent/app/cli.py +++ b/rdagent/app/cli.py @@ -18,10 +18,10 @@ from importlib.resources import path as rpath import fire from rdagent.app.data_mining.model import main as med_model +from rdagent.app.data_science.loop import main as kaggle from rdagent.app.general_model.general_model import ( extract_models_and_implement as general_model, ) -from rdagent.app.kaggle.loop import main as kaggle_main from rdagent.app.qlib_rd_loop.factor import main as fin_factor from rdagent.app.qlib_rd_loop.factor_from_report import main as fin_factor_report from rdagent.app.qlib_rd_loop.model import main as fin_model @@ -55,6 +55,6 @@ def app(): "ui": ui, "health_check": health_check, "collect_info": collect_info, - "kaggle": kaggle_main, + "kaggle": kaggle, } )