* remove state.times in old ui
* remove "r" tag
* remove "d" tag
* remove "ef" tag
* remove "init" tag
* fix CI
* remove old tag in app UI
* fix bugs
* fix CI
* some updates
* filter tags
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry
* feat: add async LoopBase with parallel workers and step semaphores
* fix: replace pickle with dill and run blocking tasks via joblib wrapper
* feat: add log format settings, dynamic parallelism & pickle-based snapshot
* fix: default step semaphore to 1 and avoid subprocess when single worker
* merge bowen's changes
* merge tim's changes
* refactor: extract component task mapping, add conditional logger setup
* lint
* refactor: add type hints and safer remain_time metric logging in workflow
* lint
* fix: allow BadRequestError to be pickled via custom copyreg reducer
* fix: stop loop when LoopTerminationError is raised in LoopBase
* lint
* refactor: make log tag context-local using ContextVar for thread safety
* feat: add subproc_step flag and helper to decide subprocess execution
* fix: use ./cache path and normalize relative volume bind paths
* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow
* fix: avoid chmod on cache and input dirs in Env timeout wrapper
* fix: skip chmod on 'cache' and 'input' dirs using find -prune
* fix: restrict chmod to immediate mount dirs excluding cache/input
* fix: chmod cache and input dirs alongside their contents after entry run
* fix: guard chmod with directory checks for cache and input
* fix: prefix mount_path in chmod command for cache/input dirs
* fix: drop quotes from find exclude patterns to ensure chmod executes
* fix: skip chmod on cache/input directories to avoid warning spam
* feat: support string volume mappings and poll subprocess stdout/stderr
* support remove symbolic link
* test: use dynamic home path and code volume in LocalEnv local_simple
* fix: skip trace and progress update when loop step is withdrawn
* refactor: add clean_workspace util and non-destructive workspace backup
* fix: preserve symlinks when backing up workspace with copytree
* fix: prevent AttributeError when _pbar not yet initialized in LoopBase
* perf: replace shutil.copytree with rsync for faster workspace backup
* fix: cast log directory Path to str in tar command of data science loop
* fix: use portable 'cp -r -P' instead of rsync for workspace backup
* fix: add retry and logging to workspace backup for robustness
* refactor: extract backup_folder helper and reuse in DataScienceRDLoop
* fix: propagate backup errors & default _pbar getattr to avoid error
* fix the division by zero bug
* refactor: execute RD loops via asyncio.run and add necessary imports
* lint
* lint
* lint
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* fix model input shape bug and costeer_model bug
* fix a bug
* fix a bug in docker result extraction
* a system-level optimization
* add a filter of stdout
* update
* add stdout to model
* model training_hyperparameters update
* quant scenario
* update some quant settings
* llm choose action
* Thompson Sampling Bandit for action choosing
* refine both scens
* add trace messages for quant scen
* fix some bugs
* fix some bugs
* update
* update
* update
* fix
* fix
* fix
* update for merge
* fix ci
* fix some bugs
* fix ci
* fix ci
* fix ci
* fix ci
* refactor
* default qlib4rdagent local env downloading
* fix ci
* fix ci
* fix a bug
* fix ci
* fix: align all prompts on template (#908)
* use template to render all prompts
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* add fin_quant in cli
* fix a bug
* fix ci
* fix some bugs
* refactor
* remove the columns in hypothesis if no value generated in this column
* fix a bug
* fix ci
* fix conda env
* add qlib gitignore
* remove existed qlib folder & install torch in qlib conda
* fix workspace ui in feedback
* align model config in coder and runner in docker or conda
* fix CI
* fix CI
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* set constrains on max_sota_retrieved, fix logis on identical problem
* fix: only Auto SOTA selector use max_sota_retrieved_num
* set max_sota_retrieved_num=10 by default
* minor update
* auto lint
* provide get metric direction in kaggle_crawler.py
* move utils function
* move statistics logic
* fix CI
* fix CI
* fix CI
* fix CI
* move some tool functions
* fix CI
* move curves win
* add compare tool
* change function name
* fix CI
* custom data
* fix: simplify competition check and log local description file
* no sample data
* feat: add test evaluation module with error handling support
* fix: update eval path to use eval_sub_dir and add valid_check TODO
* refactor: add MLETestEval check to conditionally run grading steps
* avoid blank stdout
* valid in testeval
* rename test.csv to avoid conflict
* Support Disabling sample submission
* refactoring
* fix: remove DS_KAGGLE_DATA and update prompt instructions
* add try for grade
* ignore submission
* fix: remove tee from eval command and warn about pipeline exit code detection
* optional to use raw description
* support old data
* add execution result to stdout
* add metric to raw description
* custom data explain
* add debug_path
* rst update
---------
Co-authored-by: Young <afe.young@gmail.com>
* fix: return first index if 'SOTA Exp Score (valid)' is empty
* feat: add get_state_data_range helper for loops and slider bounds
* fix: exclude batch embedding tag from log filtering
* feat: add feedback support in sota_experiment and adjust merge flow
* fix: use fb function for merging experiments
* style: remove extra whitespace and reformat code
* update all code
* update all code
* dump knowledge base
* rename the tag
* add timer to RD-Agent
* fix CI
* fix CI
* use batch embedding
* fix a small bug
* fix prompt bug
* feat: add pause resume to handle K8S cluster pause (#804)
* add resume to cluster running
* fix non-pickle problem
* fix a small bug
* fix a small bug
* avoid shutil move error
* refine the logic
* move knowledge base out of session
* avoid mistake information to pipeline coding
* avoid load and dump in steps
* archive the right folder
* small improvement
* avoid restart when timer is already started
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* ui changes
* add ours vs medal threshold
* add success loop statistic of components
* show times info
* UI updates
* summary selected
* change colors
* fix CI
* add stat hours param for `mle_summary.py --summary` command
* add 24h summary button
* fix CI
* add logger info for dockerEnv/condaEnv running time
* use conda to run kaggle and mlebench code
* refactor: Simplify environment configuration and execution logic
* add setting to use local env in ds
* refine dockerfile
* fix: Move MLEBDockerEnv initialization inside conditionals & fix condaenv
* refactor: reformat code for better readability and consistency
* feat: add conda env to all envs.
* fix: fix bugs when run loop
* refactor: Simplify DockerEnv configuration in mle_summary.py
* fix image bug
* style: reformat code for better readability and consistency
* change commit
* feat: Add entrypoint script for sing_docker scenario in rdagent
* refactor: add Any type hints and comments for clarity in env.py
* feat: Create log directory if it doesn't exist in entrypoint script
* feat: Add debug mode and list root directory in entrypoint script
* fix: Remove specific branch checkout in Dockerfile for RD-Agent
* fix: Add competition argument to loop.py script execution
* fix: Correct directory navigation and dependency installation in entrypoint.sh
* fix: Correct user ownership assignment in entrypoint script
* refactor: Comment out redundant log copying to RD_OUTPUT_DIR
* fix: Unset LOG_TRACE_PATH to prevent log contamination in entrypoint.sh
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Yuante Li