docs: update document structure. Add PR content (#171)

* Add main

* update docs

* rename and reorder

* update documents structure

* remove useless

* Update docker

docs/index.rst

@@ -10,11 +10,12 @@ Welcome to RDAgent's documentation!
    :maxdepth: 3
    :caption: Doctree:
 
-   demo_and_introduction
+   introduction
    installation_and_configuration
    scens/catalog
    project_framework_introduction
-   research/research
+   ui
+   research/catalog
    development
    api_reference
    policy

docs/installation_and_configuration.rst

@@ -5,10 +5,13 @@ Installation and Configuration
 Installation
 ============
 
-For different scenarios
-- for purely users:
+**Install RDAgent**: For different scenarios
+
+- for purely users: please use ``pip install rdagent`` to install RDAgent
 - for dev users: `See development <development.html>`_
 
+**Install Docker**: RDAgent is designed for research and development, acting like a human researcher and developer. It can write and run code in various environments, primarily using Docker for code execution. This keeps the remaining dependencies simple. Users must ensure Docker is installed before attempting most scenarios. Please refer to the `official 🐳Docker page <https://docs.docker.com/engine/install/>`_ for installation instructions.
+
 Configuration
 =============
 

docs/introduction.rst

@@ -0,0 +1,18 @@
+=========================
+Introduction
+=========================
+
+
+
+In modern industry, research and development (R&D) is crucial for the enhancement of industrial productivity, especially in the AI era, where the core aspects of R&D are mainly focused on data and models. We are committed to automate these high-value generic R&D processes through our open source R&D automation tool RDAgent, which let AI drive data-driven AI.
+
+.. image:: _static/scen.jpg
+   :alt: Our focused scenario
+
+
+Our RDAgent is designed to automate the most critical industrial R&D processes, focusing first on data-driven scenarios, to greatly boost the development productivity of models and data. 
+
+Methodologically, we propose an autonomous agent framework that consists of two key parts: (R)esearch stands for actively exploring by proposing new ideas, and (D)evelopment stands for realizing these ideas. The effectiveness of these two components will ultimately get feedbacks through practice, and both research and development capabilities can continuously learn and grow in the process.
+
+
+For a quick start, visit `our GitHub home page <https://github.com/microsoft/RD-Agent>`_ ⚡. If you've already checked it out and want more details, please keep reading.

docs/project_framework_introduction.rst

@@ -12,11 +12,15 @@ Framework & Components
 
 The image above shows the overall framework of RDAgent.
 
+In a data mining expert's daily research and development process, they propose a hypothesis (e.g., a model structure like RNN can capture patterns in time-series data), design experiments (e.g., finance data contains time-series and we can verify the hypothesis in this scenario), implement the experiment as code (e.g., Pytorch model structure), and then execute the code to get feedback (e.g., metrics, loss curve, etc.). The experts learn from the feedback and improve in the next iteration.
+
+We have established a basic method framework that continuously proposes hypotheses, verifies them, and gets feedback from the real world. This is the first scientific research automation framework that supports linking with real-world verification.
+
 
 .. image:: https://github.com/user-attachments/assets/60cc2712-c32a-4492-a137-8aec59cdc66e
     :alt: Class Level Figure
 
-For those interested in the detailed code, the figure above illustrates the main classes and aligns them with the workflow.
+The figure above shows the main classes and how they fit into the workflow for those interested in the detailed code.
 
 
 Detailed Design

docs/research/benchmark.rst

@@ -97,3 +97,22 @@ A png file will be saved to the designated path as shown below.
 
 .. image:: ../_static/benchmark.png
 
+
+Related Paper
+-------------
+
+- `Towards Data-Centric Automatic R&D <https://arxiv.org/abs/2404.11276>`_:
+  We have developed a comprehensive benchmark called RD2Bench to assess data and model R&D capabilities. This benchmark includes a series of tasks that outline the features or structures of models. These tasks are used to evaluate the ability of LLM-Agents to implement them.
+
+.. code-block:: bibtex
+
+    @misc{chen2024datacentric,
+        title={Towards Data-Centric Automatic R&D},
+        author={Haotian Chen and Xinjie Shen and Zeqi Ye and Wenjun Feng and Haoxue Wang and Xiao Yang and Xu Yang and Weiqing Liu and Jiang Bian},
+        year={2024},
+        eprint={2404.11276},
+        archivePrefix={arXiv},
+        primaryClass={cs.AI}
+    }
+
+.. image:: https://github.com/user-attachments/assets/494f55d3-de9e-4e73-ba3d-a787e8f9e841

docs/research/catalog.rst

@@ -0,0 +1,34 @@
+===========
+Research
+===========
+
+To achieve the good effects and improve R&D capabilities, we face multiple challenges, the most important of which is the continuous evolution capability. Existing large language models (LLMs) find it difficult to continue growing their capabilities after training is completed. Moreover, the training process of LLMs focuses more on general knowledge, and the lack of depth in more specialized knowledge becomes an obstacle to solving professional R&D problems within the industry. This specialized knowledge needs to be learned and acquired from in-depth industry practice.
+
+
+Our RD-Agent, on the other hand, can continuously acquire in-depth domain knowledge through deep exploration during the R&D phase, allowing its R&D capabilities to keep growing.
+
+To address these key challenges and achieve industrial value, a series of research work needs to be completed.
+
+
+.. list-table:: Research Areas and Descriptions
+   :header-rows: 1
+
+   * - Research Area
+     - Description
+   * - :doc:`Benchmark <benchmark>`
+     - Benchmark the R&D abilities
+   * - Research
+     - Idea proposal: Explore new ideas or refine existing ones
+   * - :doc:`Development <dev>`
+     - Ability to realize ideas: Implement and execute ideas
+
+
+
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Doctree:
+   :hidden:
+
+   benchmark
+   dev

docs/research/dev.rst

@@ -0,0 +1,25 @@
+==============================
+Development
+==============================
+
+
+Related Paper
+-------------
+
+- `Collaborative Evolving Strategy for Automatic Data-Centric Development <https://arxiv.org/abs/2407.18690>`_
+  Co-STEER is a method to tackle data-centric development (AD2) tasks and highlight its main challenges, which need expert-like implementation (i.e., learning domain knowledge from practice) and task scheduling capability (e.g., starting with easier tasks for better overall efficiency), areas that previous work has largely overlooked. Our Co-STEER agent enhances its domain knowledge through our evolving strategy and improves both its scheduling and implementation skills by gathering and using domain-specific practical experience. With a better schedule, implementation becomes faster. At the same time, as implementation feedback becomes more detailed, scheduling accuracy improves. These two capabilities grow together through practical feedback, enabling a collaborative evolution process.
+
+.. code-block:: bibtex
+
+    @misc{yang2024collaborative,
+        title={Collaborative Evolving Strategy for Automatic Data-Centric Development},
+        author={Xu Yang and Haotian Chen and Wenjun Feng and Haoxue Wang and Zeqi Ye and Xinjie Shen and Xiao Yang and Shizhao Sun and Weiqing Liu and Jiang Bian},
+        year={2024},
+        eprint={2407.18690},
+        archivePrefix={arXiv},
+        primaryClass={cs.AI}
+    }
+
+.. image:: https://github.com/user-attachments/assets/75d9769b-0edd-4caf-9d45-57d1e577054b
+   :alt: Collaborative Evolving Strategy for Automatic Data-Centric Development
+

docs/research/research.rst

@@ -1,15 +0,0 @@
-===========
-Research
-===========
-
-
-
-.. TODO: xiao will add this
-
-
-.. toctree::
-   :maxdepth: 1
-   :caption: Doctree:
-   :hidden:
-
-   benchmark.rst

docs/ui.rst

@@ -1,11 +1,12 @@
-=========================
-Demo and Introduction
-=========================
+==============
+User Interface
+==============
+
 
 Introduction
 ============
 
-RD-Agent will generate some logs during the R&D process. These logs are very useful for debugging and understanding the R&D process. However, just viewing the terminal log is not intuitive enough. RD-Agent provides a web app to visualize the R&D process. You can easily view the R&D process and understand the R&D process better.
+RD-Agent will generate some logs during the R&D process. These logs are very useful for debugging and understanding the R&D process. However, just viewing the terminal log is not intuitive enough. RD-Agent provides a web app as UI to visualize the R&D process. You can easily view the R&D process and understand the R&D process better.
 
 A Quick Demo
 ============
@@ -40,4 +41,4 @@ Use Web App
     - All Loops: Show complete scenario execution process.
     - Next Loop: Show one success **R&D Loop**.
     - One Evolving: Show one **evolving** step of **development** part.
-    - refresh logs: clear shown logs.
+    - refresh logs: clear shown logs.