* feat: refactor CoSTEER classes to use DSCoSTEER and update max seconds handling
* remove useless line
* enable time_ratio_limit_to_enable_hyperparameter_tuning
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry
* feat: add async LoopBase with parallel workers and step semaphores
* fix: replace pickle with dill and run blocking tasks via joblib wrapper
* feat: add log format settings, dynamic parallelism & pickle-based snapshot
* fix: default step semaphore to 1 and avoid subprocess when single worker
* merge bowen's changes
* merge tim's changes
* refactor: extract component task mapping, add conditional logger setup
* lint
* refactor: add type hints and safer remain_time metric logging in workflow
* lint
* fix: allow BadRequestError to be pickled via custom copyreg reducer
* fix: stop loop when LoopTerminationError is raised in LoopBase
* lint
* refactor: make log tag context-local using ContextVar for thread safety
* feat: add subproc_step flag and helper to decide subprocess execution
* fix: use ./cache path and normalize relative volume bind paths
* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow
* fix: avoid chmod on cache and input dirs in Env timeout wrapper
* fix: skip chmod on 'cache' and 'input' dirs using find -prune
* fix: restrict chmod to immediate mount dirs excluding cache/input
* fix: chmod cache and input dirs alongside their contents after entry run
* fix: guard chmod with directory checks for cache and input
* fix: prefix mount_path in chmod command for cache/input dirs
* fix: drop quotes from find exclude patterns to ensure chmod executes
* fix: skip chmod on cache/input directories to avoid warning spam
* feat: support string volume mappings and poll subprocess stdout/stderr
* support remove symbolic link
* test: use dynamic home path and code volume in LocalEnv local_simple
* fix: skip trace and progress update when loop step is withdrawn
* refactor: add clean_workspace util and non-destructive workspace backup
* fix: preserve symlinks when backing up workspace with copytree
* fix: prevent AttributeError when _pbar not yet initialized in LoopBase
* perf: replace shutil.copytree with rsync for faster workspace backup
* fix: cast log directory Path to str in tar command of data science loop
* fix: use portable 'cp -r -P' instead of rsync for workspace backup
* fix: add retry and logging to workspace backup for robustness
* refactor: extract backup_folder helper and reuse in DataScienceRDLoop
* fix: propagate backup errors & default _pbar getattr to avoid error
* fix the division by zero bug
* refactor: execute RD loops via asyncio.run and add necessary imports
* lint
* lint
* lint
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* fix model input shape bug and costeer_model bug
* fix a bug
* fix a bug in docker result extraction
* a system-level optimization
* add a filter of stdout
* update
* add stdout to model
* model training_hyperparameters update
* quant scenario
* update some quant settings
* llm choose action
* Thompson Sampling Bandit for action choosing
* refine both scens
* add trace messages for quant scen
* fix some bugs
* fix some bugs
* update
* update
* update
* fix
* fix
* fix
* update for merge
* fix ci
* fix some bugs
* fix ci
* fix ci
* fix ci
* fix ci
* refactor
* default qlib4rdagent local env downloading
* fix ci
* fix ci
* fix a bug
* fix ci
* fix: align all prompts on template (#908)
* use template to render all prompts
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* add fin_quant in cli
* fix a bug
* fix ci
* fix some bugs
* refactor
* remove the columns in hypothesis if no value generated in this column
* fix a bug
* fix ci
* fix conda env
* add qlib gitignore
* remove existed qlib folder & install torch in qlib conda
* fix workspace ui in feedback
* align model config in coder and runner in docker or conda
* fix CI
* fix CI
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* Use ExtendedBaseSettings to replace BaseSettings
* update a more general way to pass the default setting
* update all code
* fix CI
* fix CI
* fix qlib scenario
* fix CI
* fix CI
* fix CI & add data science interfaces
* remove redundant code
* abandon costeer knowledge base v1
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
* initial version
* test requirements
* fix bugs
* fix bugs
* add annotation
* fix ruff error
* fix CI
* fix CI
* fix CI
* fix CI
* change random usage
* move cache_seed_gen to core/utils.py
* fix CI
* change cache_seed_gen name
---------
Co-authored-by: Young <afe.young@gmail.com>
* simplify RDAgent conf
* add unified cacher(untested)
* fix small bugs
* fix a bug
* fix a small bug in runner
* use hash_key = None to skip cache
* fix CI
* in factor execution, ignore cache when raise exception
* add file locker to avoid mp calling
* fix CI
* use function __module__ name as folder in cache
* Add a table & factor debug
* Congfig setting
* Update env example and configuration list.
Also change api priority
* Add a TODO for the rst
* CI: shorter line
* Update docs/installation_and_configuration.rst
* Update docs/installation_and_configuration.rst
* Update links & standard config
* Add TODO
* Fix bug
* Update rdagent/oai/llm_utils.py
---------
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* ignore result csv file
* fix app scripts
* rename taskgenerator to developer and generate to develop
* fix a config bug in coder
* fix a small bug in factor coder evaluators
* remove a single logger in factor coder evaluators
* fix a small bug in model coder main.py
* rename Implementation to Workspace
* move the prepare the inject_code into FBWorkspace to align all the behavior
* fix a small bug in model feedback
* remove debug lines for multi processing and simplify evaluators multi proc
* add a copy function to workspace to freeze the workspace && add config prefix to speed up debugging
* make hypothesisgen a abc class
* use Qlib***Experiment
* fix a small bug
* rename Imp to Ws
* rename sub_implementations to sub_workspace_list
* fix a bug in feedback not presented as content in prompts
* move proposal pys to proposal folder
* reformat the folder
* align factor and model qlib workspace and use template to handle the workspace
* add a filter to evoagent to filter out false evo
* align multi_proc_n into RDAGENT seeting
* handle when runner gets empty experiment
* fix logger merge remaining problems
* fix black and isort automatically
* remove ruff comment in log.py
* change log framework and fix llm_utils.py's logs
* Some thoughts for logging
* fix SingletonMeta's definition, maintain an instance dict for each class that inherits it
* adjust log codes directory, add some tag for factor implementation logging
* Update rdagent/core/conf.py
* fix factor task app & log
* fix log import
* Streamlet framework
* fix log tag to path logic
* Add todos
* Add example in docstring
* add log tag for llm_utils.py
* Capture lost content
---------
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* refine CI script
* refine all the code to higher quality
* refine the script to factor extraction and implementation
* add task loader interface
* add a task loader interface && move pdf analysis to pdf task loader
* change the name to global variables
---------
Co-authored-by: xuyang1 <xuyang1@microsoft.com>
* reformat factor implement process
* move some code to more reasonable place
* fix the bug
* add test function in factor_extract_and_implement.py
* change select factor number to ratio , add some factor implement setting and fix some bug while using knowledgebase
* change evoagent
* add abstract class EvoAgent
* add benchmark workflow
* fix some bug in llm_utils
* run wenjun's code
* fix the knowledgebase instance check
---------
Co-authored-by: xuyang1 <xuyang1@microsoft.com>
Tim