mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 15:37:44 +00:00
Document in High-level (#93)
* Add init information * Adjust order and move content * GPU support * edit docs * README structure * Update Dockerfile * Update rdagent/utils/env.py * Update read the docs * Update framework
This commit is contained in:
you-n-g
@@ -1,124 +1,89 @@
|
||||
# Project
|
||||
TODO: Add badges.
|
||||
|
||||
> This repo has been populated by an initial template to help get you started. Please
|
||||
> make sure to update the content to build a great experience for community-building.
|
||||
|
||||
As the maintainer of this project, please make a few updates:
|
||||
|
||||
- Improving this README.MD file to provide a great experience
|
||||
- Updating SUPPORT.MD with content about this project's support experience
|
||||
- Understanding the security reporting process in SECURITY.MD
|
||||
- Remove this section from the README
|
||||
# News
|
||||
| 🗞️News | 📝Description |
|
||||
| -- | ------ |
|
||||
| First release | RDAgent are release on Github |
|
||||
|
||||
|
||||
# Motivation
|
||||
R&D Agent: Focusing on automating the most core and valuable part of the industrial R&D process.
|
||||
# Introduction
|
||||
|
||||
Core method: Evolving;
|
||||
|
||||
|
||||
RDAgent aims to automate the most critical and valuable aspects of the industrial R&D process, and we begins with focusing on the data-driven scenarios to streamline the development of models and data.
|
||||
Methodologically, we have identified a framework with two key components: 'R' for proposing new ideas and 'D' for implementing them.
|
||||
We believe that the automatic evolution of R&D will lead to solutions of significant industrial value.
|
||||
|
||||
|
||||
<!-- Tag Cloud -->
|
||||
R&D is a very general scenario. The advent of RDAgent can be your
|
||||
- [🎥Automatic Quant Factory]()
|
||||
- Data mining copilot: iteratively proposing [🎥data]() & [models]() and implementing them by gaining knowledge from data.
|
||||
- Research copilot: Auto read [🎥research papers]()/[🎥reports]() and implement model structures or building datasets.
|
||||
- ...
|
||||
|
||||
You can click the [🎥link]() above to view the demo. More methods and scenarios are being added to the project to empower your R&D processes and boost productivity.
|
||||
|
||||
We have a quick 🎥demo for one use case of RDAgent.
|
||||
- TODO: Demo
|
||||
|
||||
|
||||
# ⚡Quick start
|
||||
You can try our demo by running the following command:
|
||||
|
||||
```bash
|
||||
# TODO:
|
||||
# prepare environment
|
||||
# installation
|
||||
# App entrance
|
||||
```
|
||||
|
||||
The [🎥demo]() is implemented by the above commands.
|
||||
|
||||
# Scenarios
|
||||
|
||||
## Data-driven R&D
|
||||
TODO: importance justification
|
||||
We have applied RD-Agent to multiple valuable data-driven industrial scenarios..
|
||||
|
||||
### 🎯 Goal
|
||||
|
||||
In this project, we are aiming to build a Data-Centric R\&D Agent that can
|
||||
## 🎯 Goal: Agent for Data-driven R&D
|
||||
|
||||
+ Read real-world material (reports, papers, etc.) and extract key formulas, descriptions of interested features, factors and models.
|
||||
|
||||
+ Implement the extracted formulas, features, factors and models in runnable codes.
|
||||
In this project, we are aiming to build a Agent to automate Data-Driven R\&D that can
|
||||
+ 📄Read real-world material (reports, papers, etc.) and **extract** key formulas, descriptions of interested **features** and **models**, which are the key components of data-driven R&D .
|
||||
+ 🛠️**Implement** the extracted formulas, features, factors and models in runnable codes.
|
||||
+ Due the limited ability for LLM in implementing in once, evolving the agent to be able to extend abilities by learn from feedback and knowledge and improve the agent's ability to implement more complex models.
|
||||
|
||||
+ Further propose new ideas based on current knowledge and observations.
|
||||
+ 💡Propose **new ideas** based on current knowledge and observations.
|
||||
|
||||
|
||||
|
||||
### 🛣️ Brief Roadmap
|
||||
In this section, we will briefly introduce the roadmap/technical type of this project.
|
||||
## 📈 Scenarios Matrix
|
||||
Here is our supported scenarios
|
||||
|
||||
1. Backbone LLM: We use GPT series as main backbone of the agent. `.env` file uis used to config settings (such as APIkey, APIEndpoint and etc) in the environment variables way. Check this [Readme](src/scripts/benchmark/README.md) for environment set up.
|
||||
| Scenario/Target | Model Implementation | Data Building |
|
||||
| -- | -- | -- |
|
||||
| 💹Finance | Iteratively Proposing Ideas & Evolving | Auto reports reading & implementation <br/> Iteratively Proposing Ideas & Evolving |
|
||||
| 🩺Medical | Iteratively Proposing Ideas & Evolving | - |
|
||||
| General | Auto paper reading & implementation | - |
|
||||
|
||||
2. KnowledgeGraph based evolving: We do not do any further pertain or fine-tune on the LLM model. Instead, we modify prompts like RAG, but use knowledge graph query information to evolve the agent's ability to implement more complex models.
|
||||
+ Typically, we build a knowledge consisted with `Error`, `Component`(you can think of it as a numeric operation or function), `Trail` and etc. We add nodes of these types to the knowledge graph with relationship while the agent tries to implement a model. For each attempts, the agent will query the knowledge graph to get the information of current status as prompt input. The agent will also update the knowledge graph with the new information after the attempt.
|
||||
Different scenarios vary in entrance and configuration. Please check the detailed setup tutorial in the scenarios documents.
|
||||
|
||||
# ⚙️Framework
|
||||
|
||||
|
||||
## Code Refinement
|
||||
Example: code standard, design. Lint
|
||||
|
||||
|
||||
# 🔧 Development
|
||||
- Set up the development environment.
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
- Run linting and formatting.
|
||||
|
||||
```bash
|
||||
make lint
|
||||
```
|
||||
|
||||
### 📚 Code Structure
|
||||
1. Backbone/APIBackend of LLm are encapsulated in [src/finco/llm.py](src/finco/llm.py). All chat completion request are managed by this file.
|
||||
|
||||
2. All frequently modified codes under tense development are included in the [src/scripts](src/scripts) folder.
|
||||
+ The most important task is to improve the agent's performance in the benchmark of factor implementation.
|
||||
|
||||
3. Currently, factor implementation is the main task. We define basic class of factor implementation in [src/scripts/factor_implementation/share_modules] and implementation strategies in [src/scripts/factor_implementation/baselines].
|
||||
|
||||
|
||||
### 🔮 Future Code Structure
|
||||
|
||||
Currently, the code structure is unstable and will frequently change for quick updates. The code will be refactored before a standard release. Please try to align with the following principles when developing to minimize the effort required for future refactoring.
|
||||
|
||||
```
|
||||
📂 src
|
||||
➥ 📂 <project name>: avoid namespace
|
||||
➥ 📁 core
|
||||
➥ 📁 component A
|
||||
➥ 📁 component B
|
||||
➥ 📁 component C
|
||||
➥ 📂 app
|
||||
➥ 📁 scenario1
|
||||
➥ 📁 scenario2
|
||||
➥ 📁 scripts
|
||||
```
|
||||
|
||||
| Folder Name | Description |
|
||||
|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| 📁 core | The core framework of the system. All classes should be abstract and usually can't be used directly. |
|
||||
| 📁 component X | Useful components that can be used by others(e.g. scenario). Many subclasses of core classes are located here. |
|
||||
| 📁 app | Applications for specific scenarios (usually built based on components). Removing any of them does not affect the system's completeness or other scenarios. |
|
||||
| 📁 scripts | Quick and dirty things. These are candidates for core, components, and apps. |
|
||||
- TODOs:
|
||||
- Framework introdution
|
||||
- Research problems.
|
||||
|
||||
|
||||
|
||||
# Configuration:
|
||||
# 📃Paper list
|
||||
|
||||
You can manually source the `.env` file in your shell before running the Python script:
|
||||
Most of the workflow are controlled by the environment variables.
|
||||
```sh
|
||||
# Export each variable in the .env file; Please note that it is different from `source .env` without export
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
# Run the Python script
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
# Naming convention
|
||||
|
||||
## File naming convention
|
||||
|
||||
| Name | Description |
|
||||
| -- | -- |
|
||||
| `conf.py` | The configuration for the module & app & project |
|
||||
|
||||
<!-- TODO: renaming files -->
|
||||
TODO: under review. Please check the.
|
||||
|
||||
|
||||
# Contributing
|
||||
|
||||
More documents can be found in the [📚readthedocs](). TODO: add link
|
||||
|
||||
## Guidance
|
||||
This project welcomes contributions and suggestions.
|
||||
You can find issues in the issues list or simply running `grep -r "TODO:"`.
|
||||
@@ -126,24 +91,7 @@ You can find issues in the issues list or simply running `grep -r "TODO:"`.
|
||||
Making contributions is not a hard thing. Solving an issue(maybe just answering a question raised in issues list ), fixing/issuing a bug, improving the documents and even fixing a typo are important contributions to RDAgent.
|
||||
|
||||
|
||||
## Policy
|
||||
<img src="https://img.shields.io/github/contributors-anon/microsoft/RD-Agent"/>
|
||||
|
||||
This project welcomes contributions and suggestions. Most contributions require you to agree to a
|
||||
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
|
||||
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
|
||||
<a href="https://github.com/microsoft/RD-Agent/graphs/contributors"><img src="https://contrib.rocks/image?repo=microsoft/RD-Agent&max=240&columns=18" /></a>
|
||||
|
||||
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
|
||||
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
|
||||
provided by the bot. You will only need to do this once across all repos using our CLA.
|
||||
|
||||
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
|
||||
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
|
||||
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
|
||||
|
||||
# Trademarks
|
||||
|
||||
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
|
||||
trademarks or logos is subject to and must follow
|
||||
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
|
||||
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
|
||||
Any use of third-party trademarks or logos are subject to those third-party's policies.
|
||||
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
+10
-13
@@ -1,18 +1,15 @@
|
||||
=========================
|
||||
API reference
|
||||
=========================
|
||||
=============
|
||||
API Reference
|
||||
=============
|
||||
|
||||
title1
|
||||
=========================
|
||||
Here you can find all ``RDAgent``'s interfaces.
|
||||
|
||||
content1
|
||||
|
||||
title2
|
||||
=========================
|
||||
RD Loop
|
||||
=======
|
||||
|
||||
content2
|
||||
Research
|
||||
--------
|
||||
|
||||
title3
|
||||
=========================
|
||||
|
||||
content3
|
||||
.. automodule:: rdagent.core.proposal
|
||||
:members:
|
||||
|
||||
+3
-1
@@ -15,7 +15,9 @@ author = "Microsoft"
|
||||
# -- General configuration ---------------------------------------------------
|
||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
||||
|
||||
extensions = []
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
]
|
||||
|
||||
autodoc_member_order = "bysource"
|
||||
|
||||
|
||||
@@ -2,17 +2,17 @@
|
||||
Demo and Introduction
|
||||
=========================
|
||||
|
||||
title1
|
||||
=========================
|
||||
Introduction
|
||||
============
|
||||
|
||||
content1
|
||||
TODO: copy the content in the README.md
|
||||
|
||||
title2
|
||||
=========================
|
||||
A Quick Demo
|
||||
============
|
||||
|
||||
content2
|
||||
TODO:
|
||||
- copy the demo content in the README.md
|
||||
- Quick start for the demo.
|
||||
|
||||
title3
|
||||
=========================
|
||||
|
||||
content3
|
||||
TODO: link to more demos
|
||||
|
||||
+59
-7
@@ -2,17 +2,69 @@
|
||||
For Development
|
||||
=========================
|
||||
|
||||
title1
|
||||
🔧Prepare for development
|
||||
=========================
|
||||
|
||||
content1
|
||||
- Set up the development environment.
|
||||
|
||||
title2
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
- Run linting and formatting.
|
||||
|
||||
```bash
|
||||
make lint
|
||||
```
|
||||
|
||||
|
||||
Code Structure
|
||||
=========================
|
||||
|
||||
content2
|
||||
.. code-block:: text
|
||||
|
||||
title3
|
||||
=========================
|
||||
📂 src
|
||||
➥ 📂 <project name>: avoid namespace conflict
|
||||
➥ 📁 core
|
||||
➥ 📁 components/A
|
||||
➥ 📁 components/B
|
||||
➥ 📁 components/C
|
||||
➥ 📁 scenarios/X
|
||||
➥ 📁 scenarios/Y
|
||||
➥ 📂 app
|
||||
➥ 📁 scripts
|
||||
|
||||
content3
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Folder Name
|
||||
- Description
|
||||
* - 📁 core
|
||||
- The core framework of the system. All classes should be abstract and usually can't be used directly.
|
||||
* - 📁 component/A
|
||||
- Useful components that can be used by others (e.g., scenarios). Many subclasses of core classes are located here.
|
||||
* - 📁 scenarios/X
|
||||
- Concrete features for specific scenarios (usually built based on components or core). These modules are often unreusable across scenarios.
|
||||
* - 📁 app
|
||||
- Applications for specific scenarios (usually built based on components or scenarios). Removing any of them does not affect the system's completeness or other scenarios.
|
||||
* - 📁 scripts
|
||||
- Quick and dirty things. These are candidates for core, components, scenarios, and apps.
|
||||
|
||||
|
||||
|
||||
Conventions
|
||||
===========
|
||||
|
||||
|
||||
File Naming Convention
|
||||
----------------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Name
|
||||
- Description
|
||||
* - `conf.py`
|
||||
- The configuration for the module, app, and project.
|
||||
|
||||
<!-- TODO: renaming files -->
|
||||
|
||||
+7
-5
@@ -7,16 +7,18 @@ Welcome to RDAgent's documentation!
|
||||
===================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:maxdepth: 3
|
||||
:caption: Doctree:
|
||||
|
||||
demo_and_introduction
|
||||
installation
|
||||
scenarios_and_quickstart
|
||||
project_framework_introduction
|
||||
development
|
||||
api_reference
|
||||
demo_and_introduction
|
||||
project_framework_introduction
|
||||
scenarios_and_quickstart
|
||||
test_dropdown/test1
|
||||
policy
|
||||
|
||||
.. test_dropdown/test1
|
||||
|
||||
|
||||
Indices and tables
|
||||
|
||||
+8
-11
@@ -2,17 +2,14 @@
|
||||
Installation
|
||||
=========================
|
||||
|
||||
title1
|
||||
=========================
|
||||
Installation
|
||||
============
|
||||
|
||||
content1
|
||||
For different scenarios
|
||||
- for purely users:
|
||||
- for dev users: link to development
|
||||
|
||||
title2
|
||||
=========================
|
||||
Configuration
|
||||
=============
|
||||
|
||||
content2
|
||||
|
||||
title3
|
||||
=========================
|
||||
|
||||
content3
|
||||
Quick configuration
|
||||
|
||||
@@ -1,18 +1,27 @@
|
||||
===============================
|
||||
Project framework introduction
|
||||
Framework Design & Components
|
||||
===============================
|
||||
|
||||
title1
|
||||
Framework & Components
|
||||
=========================
|
||||
|
||||
content1
|
||||
- TODO: Components & Feature Level
|
||||
|
||||
title2
|
||||
- Class Level Figure
|
||||
|
||||
Detailed Design
|
||||
=========================
|
||||
|
||||
content2
|
||||
|
||||
title3
|
||||
=========================
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
You can manually source the `.env` file in your shell before running the Python script:
|
||||
Most of the workflow are controlled by the environment variables.
|
||||
```sh
|
||||
# Export each variable in the .env file; Please note that it is different from `source .env` without export
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
# Run the Python script
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
content3
|
||||
@@ -2,17 +2,32 @@
|
||||
Scenarios and Quick Start
|
||||
=========================
|
||||
|
||||
title1
|
||||
Scenario lists
|
||||
=========================
|
||||
|
||||
content1
|
||||
TODO: Copy the content in the README.md
|
||||
|
||||
title2
|
||||
|
||||
Scnarios' demo & quick start
|
||||
=========================
|
||||
|
||||
content2
|
||||
Scen1
|
||||
-----
|
||||
|
||||
title3
|
||||
=========================
|
||||
Scen1 Intro
|
||||
~~~~~~~~~~~
|
||||
|
||||
Scen1 Demo
|
||||
~~~~~~~~~~
|
||||
|
||||
Scen1 Quick Start
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Usage of modules
|
||||
================
|
||||
TODO: Show some examples:
|
||||
|
||||
content3
|
||||
@@ -174,7 +174,7 @@ class DockerEnv(Env[DockerConf]):
|
||||
try:
|
||||
client.containers.run(self.conf.image, "nvidia-smi", **gpu_kwargs)
|
||||
logger.info("GPU Devices are available.")
|
||||
except docker.errors.APIError as e:
|
||||
except docker.errors.APIError:
|
||||
return {}
|
||||
return gpu_kwargs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user