Commit Graph

76 Commits

Author SHA1 Message Date
Linlang 80d4f32c8f feat: add collect info (#233)
* add collect info

* fix isort error

* optimize code

* fix black error

* Update rdagent/app/cli.py

* Modify the code according to the comments

* fix isort error

* add docker info

* docs: update contributors (#230)

* fix: package dependency. (#234)

* fix package

* fix lint

* docs: Update development.rst (#235)

* feat: add cross validation for kaggle scenario (#236)

* update cross validation for kaggle scenario

* CI Issues

* delete useless file

* CI issues

* docs: Update README.md (#245)

* docs: refine the README (#244)

* init a scenario for kaggle feature engineering

* update the readme

* Delete rdagent/app/kaggle_feature/conf.py

* update some pictures

* Delete rdagent/app/kaggle_feature/model.py

* change a photo

* add pics to docs

* update the readme

* update the README

* for a try

* for another try

* change the style of the pictures in readme

* fix a small bug

* update the readme and the docs

* update the docs

* fix a typo

* change a website url

* change some styles

* fix a typo

* change the size of the logo

* change two urls

* Update README.md

* Update README.md

* Update README.md

* Update README.md

---------

Co-authored-by: Linlang <Lv.Linlang@hotmail.com>

* move the component to other files

* last container

* optimize rdagent info

* format with isort

* format with black

* format_with_black

* fix pip error

* format with isort

* change requirements

* fix pip error

* fix_pip_error

* fix pip error

* format with black

* fix pip error

---------

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Haotian Chen <113661982+Hytn@users.noreply.github.com>
Co-authored-by: Haoran Pan <167847254+TPLin22@users.noreply.github.com>
Co-authored-by: Way2Learn <118058822+Xisen-Wang@users.noreply.github.com>
Co-authored-by: WinstonLiyt <104308117+WinstonLiyt@users.noreply.github.com>
Co-authored-by: Young <afe.young@gmail.com>
2024-09-07 12:37:08 +08:00
Linlang 9dd35307a1 test: add test import (#242)
* add test import

* format with isort

* format with black

* merge main

* fix pytest error

* fix pytest error

* fix pytest error

* fix pytest error

* format with black

* fix pytest error

* fix pytest error

* fix pytest error

* fix pytest error

* fix pytest error

* format with isort

* Exclude entrance

* Add offline test

* auto-lint

* update coverage rate

---------

Co-authored-by: Young <afe.young@gmail.com>
2024-09-06 17:18:52 +08:00
WinstonLiyt 0dc9247153 docs: refine the README (#244)
* init a scenario for kaggle feature engineering

* update the readme

* Delete rdagent/app/kaggle_feature/conf.py

* update some pictures

* Delete rdagent/app/kaggle_feature/model.py

* change a photo

* add pics to docs

* update the readme

* update the README

* for a try

* for another try

* change the style of the pictures in readme

* fix a small bug

* update the readme and the docs

* update the docs

* fix a typo

* change a website url

* change some styles

* fix a typo

* change the size of the logo

* change two urls

* Update README.md

* Update README.md

* Update README.md

* Update README.md

---------

Co-authored-by: Linlang <Lv.Linlang@hotmail.com>
2024-09-06 15:39:41 +08:00
WinstonLiyt 96424bd1ff feat: Added loop code for Kaggle scene. (#211)
* fuse all code into one commit

* remove container auto

* change remove method

* add kaggle env start

* change kaggle api

* change structure

* add crawler

* add requirements

* refeact the code

* delete mistaken codes

* merge docker settings and crawler

* add chrome install README for crawler usage

* Connect scen with Kaggle to download data

* Reformat some files to pass CI.

* fix some ci errors

* fix a ci error

* fix a ci error

---------

Co-authored-by: Bowen Xian <xianbowen@outlook.com>
2024-08-19 10:58:28 +08:00
XianBW 51d28db28b feat: Demo display effect and usage (#162)
Demo changes
2024-08-09 20:40:16 +08:00
XianBW 9da4a76b7e feat: change ui entry (#197)
change ui start cmd
2024-08-09 17:23:45 +08:00
WinstonLiyt f9cd22e05c docs: Updated documentation for all the major scenarios. (#190)
* Fixed some bugs introduced during refactoring.

* update data_agent_fin_doc

* Updated documentation for the four major scenarios

* feat: remove pdfs and enable online pdf readings (#183)

* remove pdfs and enable online pdf readings

* update doc format

* use url as key

* feat: add entry for rdagent. (#187)

* Add entries

* update entry for rdagent

* lint

* fix typo

* docs: Demo links (#188)

add demo links

* fix: Fix a fail href in readme (#189)

* fix a ci bug

* doc

* feat: remove pdfs and enable online pdf readings (#183)

* remove pdfs and enable online pdf readings

* update doc format

* use url as key

* feat: add entry for rdagent. (#187)

* Add entries

* update entry for rdagent

* lint

* fix typo

* doc

* Updated documentation for med_model scenarios.

* fix a ci bug

---------

Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
Co-authored-by: SH-Src <suhan.c@outlook.com>
2024-08-09 13:27:33 +08:00
Xu Yang 007b632497 fix: fix quick start problem (#191)
* all update code

* fix a small bug
2024-08-09 12:58:34 +08:00
you-n-g 5c061d452f feat: add entry for rdagent. (#187)
* Add entries

* update entry for rdagent

* lint

* fix typo
2024-08-08 17:10:37 +08:00
Xu Yang 8c83317528 feat: remove pdfs and enable online pdf readings (#183)
* remove pdfs and enable online pdf readings

* update doc format

* use url as key
2024-08-08 15:20:39 +08:00
Xisen Wang 087c1c4b66 doc: Update General Model Scenario Documentation (#173)
* Init todo

* Evaluation & dataset

* Generate new data

* dataset generation

* add the result

* Analysis

* Factor update

* Updates

* Reformat analysis.py

* CI fix

* Revised General Model Doc & Layout

* Update model_copilot_general.rst

* Update eval.py

* Update eval.py

* Update analysis.py

* Update eval_method.py

* Update model_copilot_general.rst to add configurations

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Taozhi Wang <taozhi.mark.wang@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: cyncyw <47289405+taozhiwang@users.noreply.github.com>
Co-authored-by: WinstonLiyt <104308117+WinstonLiyt@users.noreply.github.com>
2024-08-06 17:44:10 +08:00
WinstonLiyt db68a28c1b fix: Fixed some bugs introduced during refactoring. (#167)
* Fixed some bugs introduced during refactoring.

* fix a minor bug

* build factor source data (price and volumns) from qlib if no source data is provided by the user (#168)

* Fixed some bugs introduced during refactoring.

* fix a small bug

* fix a small bug

---------

Co-authored-by: Xu Yang <peteryang@vip.qq.com>
2024-08-05 21:48:24 +08:00
Xu Yang 0933eb2948 build factor source data (price and volumns) from qlib if no source data is provided by the user (#168) 2024-08-05 21:01:59 +08:00
Xu Yang 987cfc723e fix: first round app folder cleaning (#166)
* first round app folder cleaning

* fix CI
2024-08-05 18:11:23 +08:00
WinstonLiyt 420dbbe584 feat: Added QlibFactorFromReportScenario and improved the report-factor loop. (#161)
* Optimize factor hypothesis prompt

* Optimize the factor feedback prompt.

* Improve the prompts in feedback(factor).

* change some prompts

* Added QlibFactorFromReportScenario and improved the report-factor loop.

* reformat

* reformat

* reformat

* reformat
2024-08-05 14:12:05 +08:00
Xu Yang 4eca5fd8c0 use mp to execute all feature and fix a small bug in config (#157) 2024-08-03 10:17:01 +08:00
Xu Yang 38e97dbd81 some small refinement (#156) 2024-08-02 19:06:21 +08:00
XianBW 4ea5fefad8 feat: streamlit webapp demo for different scenarios (#135)
add streamlit webapp demo & docs
2024-08-02 18:37:15 +08:00
WinstonLiyt 317ad89599 fix: fix some small bugs in report-factor loop (#152)
* Init todo

* update all code

* update

* Extract factors from financial reports loop finished

* Fix two small bugs.

* Delete rdagent/app/qlib_rd_loop/run_script.sh

* Minor mod

* Delete rdagent/app/qlib_rd_loop/nohup.out

* Fix a small bug in file reading.

* some updates

* Update the detailed process and prompt of factor loop.

* Evaluation & dataset

* Optimize the prompt for generating hypotheses and feedback in the factor loop.

* Generate new data

* dataset generation

* Performed further optimizations on the factor loop and report extraction loop, added log handling for both processes, and implemented a screenshot feature for report extraction.

* Update rdagent/components/coder/factor_coder/CoSTEER/evaluators.py

* Update package.txt for fitz.

* add the result

* Performed further optimizations on the factor loop and report extraction loop, added log handling for both processes, and implemented a screenshot feature for report extraction. (#100) (#102)

- Performed further optimizations on the factor loop and report extraction loop.
- Added log handling for both processes.
- Implemented a screenshot feature for report extraction.

* Analysis

* Optimized log output.

* Factor update

* A draft of the "Quick Start" section for README

* Add scenario descriptions.

* Updates

* Adjust content

* Enable logging of backtesting in Qlib and store rich-text descriptions in Trace. Support one-step debugging for factor extraction.

* Reformat analysis.py

* CI fix

* Refactor

* remove useless code

* fix bugs (#111)

* Fix two small bugs.

* Fix a merge bug.

* Fix two small bugs.

* fix some bugs.

* Fix some format bugs.

* Restore a file.

* Fix a format bug.

* draft renew of evaluators

* fix a small bug.

* fix a small bug

* Support Factor Report Loop

* Update framework for extracting factors from research reports.

* Refactor report-based factor extraction and fix minor bugs.

* fix a small bug of log.

* change some prompts

* improve factor_runner

* fix a small bug

* change some prompts

* cancel some comments

* cancel some comments and fix some bugs

* fix some bugs in factor from reports loop

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Taozhi Wang <taozhi.mark.wang@gmail.com>
Co-authored-by: Suhan Cui <51844791+SH-Src@users.noreply.github.com>
2024-08-02 17:14:12 +08:00
Xisen Wang 84d957a615 feat: Uploaded Documentation, Updated Prompts & Some Code for model demo (#144)
* Added three new keys on hypothesis reasoning

* Updated two scenario rich text

* Uploaded Documentation & Further Improved Demo of Models

* Add docs

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-08-02 16:22:06 +08:00
WinstonLiyt c6bf4e5015 fix: Comprehensive update to factor extraction. (#143)
* Init todo

* update all code

* update

* Extract factors from financial reports loop finished

* Fix two small bugs.

* Delete rdagent/app/qlib_rd_loop/run_script.sh

* Minor mod

* Delete rdagent/app/qlib_rd_loop/nohup.out

* Fix a small bug in file reading.

* some updates

* Update the detailed process and prompt of factor loop.

* Evaluation & dataset

* Optimize the prompt for generating hypotheses and feedback in the factor loop.

* Generate new data

* dataset generation

* Performed further optimizations on the factor loop and report extraction loop, added log handling for both processes, and implemented a screenshot feature for report extraction.

* Update rdagent/components/coder/factor_coder/CoSTEER/evaluators.py

* Update package.txt for fitz.

* add the result

* Performed further optimizations on the factor loop and report extraction loop, added log handling for both processes, and implemented a screenshot feature for report extraction. (#100) (#102)

- Performed further optimizations on the factor loop and report extraction loop.
- Added log handling for both processes.
- Implemented a screenshot feature for report extraction.

* Analysis

* Optimized log output.

* Factor update

* A draft of the "Quick Start" section for README

* Add scenario descriptions.

* Updates

* Adjust content

* Enable logging of backtesting in Qlib and store rich-text descriptions in Trace. Support one-step debugging for factor extraction.

* Reformat analysis.py

* CI fix

* Refactor

* remove useless code

* fix bugs (#111)

* Fix two small bugs.

* Fix a merge bug.

* Fix two small bugs.

* fix some bugs.

* Fix some format bugs.

* Restore a file.

* Fix a format bug.

* draft renew of evaluators

* fix a small bug.

* fix a small bug

* Support Factor Report Loop

* Update framework for extracting factors from research reports.

* Refactor report-based factor extraction and fix minor bugs.

* fix a small bug of log.

* change some prompts

* improve factor_runner

* fix a small bug

* change some prompts

* cancel some comments

* cancel some comments and fix some bugs

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Taozhi Wang <taozhi.mark.wang@gmail.com>
Co-authored-by: Suhan Cui <51844791+SH-Src@users.noreply.github.com>
2024-08-02 15:04:49 +08:00
Xisen Wang 0accbe258b Added three new keys on hypothesis reasoning (#138)
* Added three new keys on hypothesis reasoning

* Updated two scenario rich text
2024-08-01 10:53:37 +08:00
Xu Yang 345f91b708 refine the graph model interface (#136) 2024-07-30 19:38:37 +08:00
cyncyw 6e8f912968 Openai api & eval debug (#124)
* Openai api & eval debug


---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-07-30 18:06:48 +08:00
XianBW dbbec2ffaf New Structure Demo (#120)
better demo
---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Taozhi Wang <taozhi.mark.wang@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: cyncyw <47289405+taozhiwang@users.noreply.github.com>
2024-07-30 17:23:05 +08:00
Xisen Wang 1b4e98a752 Checking general demo (#126)
* Init todo

* Evaluation & dataset

* Generate new data

* dataset generation

* add the result

* Analysis

* Factor update

* Updates

* Reformat analysis.py

* CI fix

* Further Optimised Model Workflow by Incorporating Feedbacks on Exp Task Card

* Rebasing To build the extraction & implementation demo

* Revised for clean code

* Revised further to show "Knowledge"

* Revised to make model_research_copilot better

* Further Optimised Model Workflow by Incorporating Feedbacks on Exp Task Card

* Rebasing To build the extraction & implementation demo

* Revised for clean code

* Revised further to show "Knowledge"

* Revised to make model_research_copilot better

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Taozhi Wang <taozhi.mark.wang@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: cyncyw <47289405+taozhiwang@users.noreply.github.com>
2024-07-30 15:47:21 +08:00
WinstonLiyt 7ec13c4087 Fix some minor bugs caused by version changes. (#128)
Fix some minor bugs caused by version changes.
2024-07-29 18:02:17 +08:00
you-n-g 45c9726133 Support special step customization (#123)
* Support special step customization

* lint
2024-07-26 14:29:19 +08:00
Linlang faa2fb03ad CI checks that can be automatically repaired (#119)
* fix isort & black & toml-sort & sphinx error

* fix ci error

* fix ci error

* add comments

* Update Makefile

* change sphinx build command

* add auto-lint

* add black args

* format with black

* Auto Linting document

* fix ci error

---------

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Young <afe.young@gmail.com>
2024-07-26 12:12:16 +08:00
WinstonLiyt a4fa000862 Fix two small bugs. (#121)
* Fix two small bugs.
2024-07-26 11:00:21 +08:00
cyncyw a82222d077 Benchmark (#114)
* Init todo

* Evaluation & dataset

* Generate new data

* dataset generation

* add the result

* Analysis

* Factor update

* Updates

* Reformat analysis.py

* CI fix

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-07-25 18:22:57 +08:00
WinstonLiyt 25fb571a0f A draft of the "Quick Start" section for your README. Enable logging of backtesting in Qlib and store rich-text descriptions in Trace. Support one-step debugging for factor extraction. (#105)
- A draft of the "Quick Start" section for your README. 
- Enable logging of backtesting in Qlib and store rich-text descriptions in Trace. 
- Support one-step debugging for factor extraction.
2024-07-25 08:43:13 +00:00
Linlang 571b5304cb fix mypy error (#91)
* fix mypy error

* fix mypy error

* fix ruff error

* change command

* delete python 3.8&3.9 from CI

* change command

* Some modifications according to the comments

* Add literal type

* Update .github/workflows/ci.yml

* Some modifications according to the comments

* fix ruff error

* fix meta dict

* Fix type

* Some modifications according to the comments

* merge latest code

* Some modifications according to the comments

* Some modifications according to the comments

* fix ci error

* fix ruff error

* Update Makefile

* Update Makefile

---------

Co-authored-by: Ubuntu <debug@debug.qjtqi00gqezu1eqs55bqdrf51f.px.internal.cloudapp.net>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2024-07-25 15:20:04 +08:00
WinstonLiyt 068292f749 Optimized log output. (#104)
Optimized log output.
2024-07-24 17:06:52 +08:00
you-n-g c707a40073 Data mining (#103)
* scen

* scen2

* app

* fix

* Simplify workflow

* We can share more code in new scenarios

* rename model to rd loop

* Optimize data path

* Update rdagent/app/data_mining/model.py

* Add TODO

* Support GPU

* gpu

---------

Co-authored-by: SH-Src <suhan.c@outlook.com>
2024-07-24 16:56:27 +08:00
WinstonLiyt 226a0470ac Performed further optimizations on the factor loop and report extraction loop, added log handling for both processes, and implemented a screenshot feature for report extraction. (#100)
- Performed further optimizations on the factor loop and report extraction loop.
- Added log handling for both processes.
- Implemented a screenshot feature for report extraction.
2024-07-24 12:15:56 +08:00
you-n-g 40d108687c Support run one step for debugging (#99) 2024-07-23 19:00:59 +08:00
Xisen Wang 44169507b2 Merge pull request #97 from microsoft/recover-branch-xisen-3
Further Optimisation For Model
2024-07-23 17:19:17 +08:00
you-n-g 4f484e009a Workflow Support Loading and saving sessions (#98)
* Successfully logging the trace

* Start debugging & Add policy file

* Support loading sessions

* Add docs

* Add tqdm
2024-07-23 16:37:41 +08:00
Xisen-Wang 81fe304127 Further Optimised Kwargs & Confs 2024-07-23 06:32:26 +00:00
WinstonLiyt 7220d33668 Update the detailed process and prompt of factor loop. (#96)
Update the detailed process and prompt of factor loop.
2024-07-22 18:18:08 +08:00
WinstonLiyt 761df330de Fix a small bug in file reading. 2024-07-22 14:40:14 +08:00
WinstonLiyt c17244a317 Extract factors from financial reports loop finished. (#90)
- Extract factors from financial reports loop finished.

- Fix some small bugs.
2024-07-20 12:31:40 +08:00
Xisen Wang a59ab982c9 Merge pull request #86 from microsoft/recover-branch-xisen-3
Model Optimisation Update 

After optimization, the rd agent loop for model in the qlib scenario should now work.
2024-07-19 15:05:24 +08:00
Linlang 276b462f40 Fix ruff error1 (#81)
* fix_ruff_error1

* fix_ruff_error

* fix ruff error

* fix ruff error

* pass model.py

* rename exception class

* rename exception class

* rename func name generate_feedback

* remove prepare args

* optimize code

* optimize code

* fix code error
2024-07-18 22:36:04 +08:00
you-n-g c0e9a5637d Update rdagent/app/qlib_rd_loop/RDAgent.py 2024-07-18 18:33:31 +08:00
Xisen-Wang 82c00d8507 Revised all prompts 2024-07-18 09:42:55 +00:00
Xisen-Wang ee486e9d2d Trying 2024-07-18 09:42:55 +00:00
Xu Yang a9cb50f3c4 add_logger_to_factor_and_fix_a_small_bug_in_factor_runner (#85) 2024-07-18 17:00:41 +08:00
you-n-g 9f89511be6 Model run with logger (#79) 2024-07-17 17:29:39 +08:00