* update all code
* update all code
* dump knowledge base
* rename the tag
* add timer to RD-Agent
* fix CI
* fix CI
* use batch embedding
* fix a small bug
* fix prompt bug
* feat: add pause resume to handle K8S cluster pause (#804)
* add resume to cluster running
* fix non-pickle problem
* fix a small bug
* fix a small bug
* avoid shutil move error
* refine the logic
* move knowledge base out of session
* avoid mistake information to pipeline coding
* avoid load and dump in steps
* archive the right folder
* small improvement
* avoid restart when timer is already started
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* refactor: use dynamic input path and update template loader
* fix: update include syntax for data source in prompts.yaml
* add customization path
* docs: update prompts for ensemble scoring and metric direction
* chore: remove obsolete data_science/share.yaml file
* style: Simplify language and improve clarity in prompts and share.yaml
* style: Update prompt wording for clarity in raw_data_loader
* style: Simplify conditional logic in task_gen system prompt
* refactor: Update prompts and proposal for component output format handling
* fix: Correct grammar and add clarification in prompts.yaml
* feat: Include coding guidelines in data science component prompts
* lint
* feat: add DocDev for auto-generating workspace documentation
* fix: update markdown instructions in tpl.yaml
* feat: add enable_doc_dev flag and conditionally call DocDev
* refactor: update T import and prompt keys for DocDev
* fix: update include path for MarkdownOut template
* fix: update file search, README injection, and brief prompt text
* docs: update prompt to only introduce models
* lint
* feat: add model dump flag and multi-evaluator support
* tmp code
* refactor: update evaluator feedback and FBWorkspace types
* feat: add get_clear_ws_cmd and CPU count in Docker environment
* feat: Add model dump check level and enhance evaluator functionality
fix data type bug
* fix: Ensure required files exist before model dump evaluation
* refactor: streamline prompt and file checks in model dump evaluation
* fix: add assertions and reorder file reads in model dump evaluator
* feat: remove EDA part from evaluation output
* docs: update dump_model guidelines and eval prompt to include template
* style: reformat multiline dicts and lists in conf and eval files
* fix: add DOTALL flag to EDA removal regex
* rebase selection code
* bug-free run: checkpoint selection and dynamic EDA loading
* add prototypes of various selectors, to imp. and test later
* fix EDA write bug
* move selector to from proposal.py tp seletc.py
* auto lint
* fix line-too-long typos
* aligh the design of "selection", rm extra instance check
* make auto-lint
* add non-trival selector: SOTAjump
* cache function
* fix test
* bin cache
* fix test
* fix test
* fix test
* cache for different source
* cache for localenv
* remove unnecessary log
* reformat
* remove unrelated modify
* init commit
* limit problem numbers
* ensemble lower case
* add runtime and spec to coder
* submission check notice
* sub EDA in sample execution
* avoid lightgbm
* add time limit to scenario
* rephrase the submission check
* give positive feedback when facing warning in check
* ENABLE FEEDBACK
* fix feedback bug
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* docs: Update prompts and descriptions for data science components
* chore: Remove outdated comments from conf.py
* feat: Add metric_name attribute to DataScienceScen class
* style: Update description in prompts.yaml and reorder metric_name init
* docs: Update prompts.yaml with feature engineering guidelines
* update metric_name
* fix some bugs
* add an evaluation in workflow
* add an evalution in runner
* fix ci
* test change
* fix CI
---------
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: yuanteli <1957922024@qq.com>
* feat: Add spec_enabled configuration for data science settings
* make spec alternative
* change spec logic in exp_gen
* remove some general texts
* align
---------
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: yuanteli <1957922024@qq.com>
* prune model task
* add component_description
* add model removal logic to component, hypo, and task gen
* fix ci
* adjust coder to meet the requirement of model removal
* fix and refine the logic of model removal
* add model removal logic in model_eval
* fix ci
* fix ci
* prune some unnecessary codes
* use conda to run kaggle and mlebench code
* refactor: Simplify environment configuration and execution logic
* add setting to use local env in ds
* refine dockerfile
* fix: Move MLEBDockerEnv initialization inside conditionals & fix condaenv
* refactor: reformat code for better readability and consistency
* feat: add conda env to all envs.
* fix: fix bugs when run loop
* refactor: Simplify DockerEnv configuration in mle_summary.py
* fix image bug
* style: reformat code for better readability and consistency
* change commit
* feat: Add entrypoint script for sing_docker scenario in rdagent
* refactor: add Any type hints and comments for clarity in env.py
* feat: Create log directory if it doesn't exist in entrypoint script
* feat: Add debug mode and list root directory in entrypoint script
* fix: Remove specific branch checkout in Dockerfile for RD-Agent
* fix: Add competition argument to loop.py script execution
* fix: Correct directory navigation and dependency installation in entrypoint.sh
* fix: Correct user ownership assignment in entrypoint script
* refactor: Comment out redundant log copying to RD_OUTPUT_DIR
* fix: Unset LOG_TRACE_PATH to prevent log contamination in entrypoint.sh
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* move cache auto continue and retry to all api backend
* add type checker to json mode output
* fix CI
* feat: Add json_mode handling and streaming support in chat completion function
* lint
* fix a bug when returning a dict which value could contain int or bool
* remove litellm
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Young <afe.young@gmail.com>
Roland Minrui