* refactor: split workflow into pkg, add WorkflowTracker & wait_retry
* feat: add async LoopBase with parallel workers and step semaphores
* fix: replace pickle with dill and run blocking tasks via joblib wrapper
* feat: add log format settings, dynamic parallelism & pickle-based snapshot
* fix: default step semaphore to 1 and avoid subprocess when single worker
* merge bowen's changes
* merge tim's changes
* refactor: extract component task mapping, add conditional logger setup
* lint
* refactor: add type hints and safer remain_time metric logging in workflow
* lint
* fix: allow BadRequestError to be pickled via custom copyreg reducer
* fix: stop loop when LoopTerminationError is raised in LoopBase
* lint
* refactor: make log tag context-local using ContextVar for thread safety
* feat: add subproc_step flag and helper to decide subprocess execution
* fix: use ./cache path and normalize relative volume bind paths
* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow
* fix: avoid chmod on cache and input dirs in Env timeout wrapper
* fix: skip chmod on 'cache' and 'input' dirs using find -prune
* fix: restrict chmod to immediate mount dirs excluding cache/input
* fix: chmod cache and input dirs alongside their contents after entry run
* fix: guard chmod with directory checks for cache and input
* fix: prefix mount_path in chmod command for cache/input dirs
* fix: drop quotes from find exclude patterns to ensure chmod executes
* fix: skip chmod on cache/input directories to avoid warning spam
* feat: support string volume mappings and poll subprocess stdout/stderr
* support remove symbolic link
* test: use dynamic home path and code volume in LocalEnv local_simple
* fix: skip trace and progress update when loop step is withdrawn
* refactor: add clean_workspace util and non-destructive workspace backup
* fix: preserve symlinks when backing up workspace with copytree
* fix: prevent AttributeError when _pbar not yet initialized in LoopBase
* perf: replace shutil.copytree with rsync for faster workspace backup
* fix: cast log directory Path to str in tar command of data science loop
* fix: use portable 'cp -r -P' instead of rsync for workspace backup
* fix: add retry and logging to workspace backup for robustness
* refactor: extract backup_folder helper and reuse in DataScienceRDLoop
* fix: propagate backup errors & default _pbar getattr to avoid error
* fix the division by zero bug
* refactor: execute RD loops via asyncio.run and add necessary imports
* lint
* lint
* lint
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* use conda to run kaggle and mlebench code
* refactor: Simplify environment configuration and execution logic
* add setting to use local env in ds
* refine dockerfile
* fix: Move MLEBDockerEnv initialization inside conditionals & fix condaenv
* refactor: reformat code for better readability and consistency
* feat: add conda env to all envs.
* fix: fix bugs when run loop
* refactor: Simplify DockerEnv configuration in mle_summary.py
* fix image bug
* style: reformat code for better readability and consistency
* change commit
* feat: Add entrypoint script for sing_docker scenario in rdagent
* refactor: add Any type hints and comments for clarity in env.py
* feat: Create log directory if it doesn't exist in entrypoint script
* feat: Add debug mode and list root directory in entrypoint script
* fix: Remove specific branch checkout in Dockerfile for RD-Agent
* fix: Add competition argument to loop.py script execution
* fix: Correct directory navigation and dependency installation in entrypoint.sh
* fix: Correct user ownership assignment in entrypoint script
* refactor: Comment out redundant log copying to RD_OUTPUT_DIR
* fix: Unset LOG_TRACE_PATH to prevent log contamination in entrypoint.sh
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* refactor: Add run_ret_code method and update run method to use it
* feat: Add kwargs support to run methods and test for run_ret_code
* fix: preserve exit code after chmod in DockerEnv entry command
* chore: Change file permissions from 755 to 644 in env_tpl directory
* refactor: Return execution code and update evaluator logic
* lint
* refactor: Use MappingProxyType for running_extra_volume in DockerEnv methods
* lint
* File structure for supporting litellm
* more litellm support
* feat: Add CachedAPIBackend class and dynamic API backend retrieval function
* fix: update benchmark folder path and add default values for architecture and hyperparameters
* feat: add LiteLLMAPIBackend and DeprecBackend ; changed structure of the project ; with bus
* fix : deprec_backend
* feat: Add LiteLLMAPIBackend class and related features; update configuration and test cases.
* feat: Enhance LiteLLMAPIBackend with encoder support and dynamic argument handling;Enhance log Colors
* lint
* fix lint...
* fix: Lint
* fix:make auto-lint
* fix:test oai
* fix:redundant _abckend.py
* fix: Optimize LiteLLMAPIBackend on token counting functiona, and clean up unused code;add test on this function
* feat: Add LiteLLMSettings class and update model settings usage
* fix: Update LiteLLMSettings environment variable prefix and model configurations
* fix : gitignore
* test: Consolidate and relocate test files for litellm backend and oai
* fix : lint
* fix: lint
* auto lint
* lint
* LINT
* lint
* chore: remove deprecated backend configuration comments
* refactor: Remove unused functions and imports from deprec.py and llm_utils.py
* refactor: Move md5_hash function from deprec.py to llm_utils.py
* chore: Remove extra newline and add missing import in deprec.py
* lint
* refactor: Move md5_hash function to utils module
* lint
* lint
* lint
---------
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Yihua Chen <v-yihuachen@microsoft.com>
* refine ds modal for more cases: eval and es
* update model template
* prompts for model and ensemble
* fix a bug
* fix a bug
* init: ds workflow evovingstrategy
* Adding ensemble (#505)
* Initial Draft
* Updating logic for init
* Revising
* Successful Testing
* Updating to use the latest & right class
* bug: bug-fixing for testing
* data science loop changes
* data science loop base
* ds loop feedback
* fix
* remove measure_time because it's duplicated (in LoopBase)
* add the knowledge query for data_loader & feature
* edit ds workflow evaluator
* data_loader bug fix
* stop evolving when all tasks completed
* llm app change
* fix break all complete strategy
* Adding queried knowledge (#508)
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
* fix loop bug
* ds workflow evaluator; test; refine prompts
* workflow spec
* fix ci
* feature task changes
* ds loop change
* fix a bug in feat
* add query knowledge for model and workflow
* llm_debug info(for show) using pickle instead of json
* remove NextLoopException
* loop change
* coder raise CoderError when all sub_tasks failed
* rename code_dict to file_dict in FBWorkspace
* add CoSTEER unittest
* now show self.version in Task.get_task_information(), simplify CoSTEER sub tasks definition
* remove some properties in ModelTask, add model_type in it.
* fix llm app bug
* llm web app bug fix
* ds loop bug fix
* fix: give component code to feature&ens eval
* loop catch error bug
* rename load_from_raw_data to load_data
* feat: Add debug data creation functionality for data science scenarios
* support local folder (#511)
* support local folder
* remove unnecessary random
* KaggleScen Subclass
* small fix
* use template for style description
* update default scen to kaggle
* update sample data script
* make sure frac < 1
* fix a bug
* feature spec changes
* fix
* changeimport order
* clear unnecessary std outputs
* fix a typo
* create sample folder after unzip kaggle data
* feature/model test script update
* Align the data types across modules.
* fix a bug in model eval
* show line number
* move sample entry point to app
* spec & model prompt changes
* Refine the competition specification to address the data type problem and the coherence issue.
* fix some bugs
* add file filter in FBworkspace.code property
* support non-binary prediction
* avoid too much warnings
* fix a bug in ensemble module
* filtered the knowledge query in all modules
* delete RAG in idea proposal
* refine the code in ensemble
* show exp workspace in llm_st
* exp_gen bug fix
* feedback bug fix
* use `feature` instead of `feat01`
* Trace & method of judging if exp is completed change
* fix a bug in package calling and execute ci
* fix code
* bug fix
* bug fix
* fix a bug
* fix some bugs
* fix a bug
* refactor: Enhance error handling and feedback in data science loop
* support different use_azure on chat and embedding models
* multi-model proposal logic
* fix a small syntax error
* loopBase and some changes
* ensemble scores change
* fbworkspace.code -> .all_codes
* use all model codes in workflow coder
* check scores.csv's keys(model_names)
* model name changes
* add a todo in ensemble test
* sota_exp changes
* give model info in exp gen
* add runner time limit
* config using debug data or not in evals
* exp to feedback base
* add feature code when writing model task
* small problem
* copying during sampling
* update
* refactor: Simplify code handling and improve workspace management
* model part output fix
* print model's execution time
* bug fix
* ensemble test fix
* ens small change
* ens_test bug fix
* Refine partial expansion logic to display only a few subfolders when their structure is uniform, improving readability in nested directories.
* several update on prompts
* sample subfolders
* Filter the stdout after code execution to remove irrelevant information e.g. progress bars, whitespace characters, excessive line breaks.
* Add some more prompts and comments
* several update on the first init rounds
* model timeout as error
* fix pattern of getting model codes in workspace
* small bux fix on model prompts
* remove get_code_with_key since we have regex pattern
* fix: Correct tqdm progress bar update logic in LoopBase class
* feat: Add diff generation and enhance feedback mechanism in data science loop
* update some fix to model and workflow prompts
* refine the logic of progress bar filter
* add last_successful_exp in exp_gen
* fix a one line bug
* add a hint in prompt
* fix data sample for bms
* fix data sample for bms
* hypothesis small fix
* crawler readme update
* fix component gen
* fix bug
* annotation change
* load description.md if it exists
* refactor: Simplify SOTA description handling in feedback and prompts
* refactor: Use shared templates for feedback and experiment descriptions
* change webapp for model codes changes
* update proposal
* add timeout message for docker run output
* fix
* refine the code in docker time processing
* use .shape instead of len() when do shape eval
* won't change size during iteration
* support bson sample
* sample support jsonl and bson
* add former_code to coder prompts
* a little speed us in debug data creating
* filter progress bar when eval ens and main
* avoid costeer makes no change to former code
* fix several log error
* add timeout judge threshold
* fix some bugs in the evaluation of component output shapes
* File structure for supporting litellm (#517)
Co-authored-by: Young <afe.young@gmail.com>
* ignore submission and show processing
* ignore submission and show processing
* add efficiency notice
* refactor: Enhance error message with detailed feedback summary
* refactor: Simplify component handling in DSExpGen class
* refactor: Update code structure and add docstring for clarity
* reserve one sample to each label in data sampling
* add Evaluation info
* refine costeer code to avoid giving same code twice
* use raw_description as plain text
* add a prompt hint to avoid same dict key
* model task name bug in first model exp gen
* fix a typo
* add some debug info in costeer tests
* task init change
* enhance data sampling
* refine the code in data_loader
* more reasonable loop
* fix a bug in data folder description
* add error msg & traceback to execution feedback
* fix llm error msg detection
* add task information to costeer eval & add cache to docker run(use zipfile to store the whole workspace)
* fix CI first round
* fix CI second round
* use txt to store test script to avoid pytest
* remove zipfile in requirements
* add azure.identity to requirements
* ignore debug web page
* component test changes
* remove redundent task_desc in model coder
* feat: Add APE module and prompts for automated prompt engineering
* fix: Update .gitignore and improve text formatting in eval.py
* refactor: Update print output and improve code comments and imports
* style: Fix string formatting and import order in ape.py and fmt.py
* exclude ape
* add a data folder notice
* reduce unnecessary output to stdout
* refine the code of describe_data_folder
* fix ci
* style: streamlit style update (#522)
* streamlit style update
* fix import
* fix format
* fix llm_st loop progress bar
* debugapp small change
* fix model str
* refine some prompts
* fix model str
* fix CI
* refine the logic associated with the data_folder
* fix ci
* small change
* set filter_progress_bar as default in execute
* model proposal with workflow
* add submission check in workflow eval
* fix bug
* small change
* fix CI
* fix CI
* refactor: Move generate_diff to utils and update DSExpGen logic
* more reasonable prompt describing metric direction
* fix a minor jinja2 bug
* quick fix exp_gen bugs
* fix the following bug
* fix
* fix some bugs
* remove workflow from model
* add pending_tasks_list in data science to enable coding model and workflow
* refine the code for handling JSON-formatted data descriptions
* assert with information
* ensure correct csv file name
* add logging to help record the output
* log competition
* add log tag for debug llm app
* test: Test ds refactor ll (#523)
* fix bugs to former scenario
* fix a bug because coding in rdloop changed
* fix the bug when feedback gets no hypothesis
* fix trace structure
* change all trace hist when merging hypothesis to experiments
* ignore some error in ruff
* fix kaggle scenario bugs
* refine one line
* another bug
* another small bug
* fix ui bugs
* chage kaggle train.py path
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* fix CI
* Update rdagent/app/data_science/loop.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add samplecsv into spec prompts
* fix CI
---------
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: yuanteli <1957922024@qq.com>
Co-authored-by: Xisen Wang <118058822+xisen-w@users.noreply.github.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
Co-authored-by: Tim <illking@foxmail.com>
Co-authored-by: 炼金术师华华 <37462254+YeewahChan@users.noreply.github.com>
Co-authored-by: Linlang <30293408+SunsetWolf@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* init trail
* Add spec info
* auto unzip mlebench prepared data for out scenario
* successfully run example
* successfully run main
* simplify load traing
* extract load_from_raw_data
* split the fies(still buggy)
It should stop on ~20 epoch and reach the end
* some changes
* Fix bug to run example
* (success) until feature
* refine model and ensemble
* add metrics in ens.py
* update README & spec.md
* ens change
* fix ens bug
* Delete rdagent/scenarios/kaggle/tpl_ex/aerial-cactus-identification/train.py
* add template_path in KG_conf
* fix test kaggle
* CI
* make test_import not check kaggle template codes
---------
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
* copy init version
* feat: new-york-city-taxi-fare-prediction_template
* add move to linear model
* Add more details about docker
* auto lint
* auto lint with new black
* initial version
* test requirements
* fix bugs
* fix bugs
* add annotation
* fix ruff error
* fix CI
* fix CI
* fix CI
* fix CI
* change random usage
* move cache_seed_gen to core/utils.py
* fix CI
* change cache_seed_gen name
---------
Co-authored-by: Young <afe.young@gmail.com>
* fix mypy error
* fix mypy error
* fix ruff error
* change command
* delete python 3.8&3.9 from CI
* change command
* Some modifications according to the comments
* Add literal type
* Update .github/workflows/ci.yml
* Some modifications according to the comments
* fix ruff error
* fix meta dict
* Fix type
* Some modifications according to the comments
* merge latest code
* Some modifications according to the comments
* Some modifications according to the comments
* fix ci error
* fix ruff error
* Update Makefile
* Update Makefile
---------
Co-authored-by: Ubuntu <debug@debug.qjtqi00gqezu1eqs55bqdrf51f.px.internal.cloudapp.net>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
* ignore result csv file
* fix app scripts
* rename taskgenerator to developer and generate to develop
* fix a config bug in coder
* fix a small bug in factor coder evaluators
* remove a single logger in factor coder evaluators
* fix a small bug in model coder main.py
* rename Implementation to Workspace
* move the prepare the inject_code into FBWorkspace to align all the behavior
* fix a small bug in model feedback
* remove debug lines for multi processing and simplify evaluators multi proc
* add a copy function to workspace to freeze the workspace && add config prefix to speed up debugging
* make hypothesisgen a abc class
* use Qlib***Experiment
* fix a small bug
* rename Imp to Ws
* rename sub_implementations to sub_workspace_list
* fix a bug in feedback not presented as content in prompts
* move proposal pys to proposal folder
* reformat the folder
* align factor and model qlib workspace and use template to handle the workspace
* add a filter to evoagent to filter out false evo
* align multi_proc_n into RDAGENT seeting
* handle when runner gets empty experiment
* fix logger merge remaining problems
* fix black and isort automatically
* use CoSTEER as component name
* rename factorimplementation to avoid confusion
* rename modelimplementation
* align benchmark and evolving evaluators
* add scenario to evaluator init function
* rename all factorimplementationknowledge in CoSTEER
* remove all scenario related information in component
* remove useless code
---------
Co-authored-by: xuyang1 <xuyang1@microsoft.com>
* Initial framework for docker env
* Update test name
* add features
* Download Qlib data with extra_volume
* fix pytest error
* Fix the parameters
---------
Co-authored-by: Young <afe.young@gmail.com>
* Commit init framework
* Co-authored-by: Yuante Li (FESCO Adecco Human Resources) <v-yuanteli@microsoft.com>
Co-authored-by: XianBW <XianBW@users.noreply.github.com>
* add an import
* refine the whole framework
* benchmark related framework
* fix black and isort errors
* move requirements to folder
* fix black again
---------
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: xuyang1 <xuyang1@microsoft.com>
you-n-g