* fix: update fallback criterion
* fix: ensure evo_fb is initialized and used correctly in fallback logic
* refactor: rename use_new_evo to should_use_new_evo for clarity
* feat: add option to enable hyperparameter tuning only in first eval loop
* fix: use total_seconds() for accurate time calculations in evolution and tracking
* refine prompts and add additional package info
* refine prompts to be specific for GBDT models
* minor refine prompts
* use include to replace duplicate info
* refine prompts
* refactor: import DSTrace from base and remove exp_gen __init__
* lint
---------
Co-authored-by: Young <afe.young@gmail.com>
* change DSCoSTEER_eval prompts
* fallback to better exp only
* fix fallback
* fix and reformat
* fix bug when base_fb is None
* add reasoning to hyperparameter evaluation
* feat: add acceptable assessment in exp_feedback (#1159)
* add time
* refine eval prompt and make the logic of tuning check more clear
* some refinement
* fix CI
* fix a small bug, only consider score in runner
* refine comment
* simplify compare function
---------
Co-authored-by: jingyuanlm <842442862@qq.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Jensen Lee <91518020+Jensen246@users.noreply.github.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* feat: refactor CoSTEER classes to use DSCoSTEER and update max seconds handling
* remove useless line
* enable time_ratio_limit_to_enable_hyperparameter_tuning
* add scores.csv metric name in both task_gen and coder
* a little fix to column names
* small fix
* avoid sample submission read in task_gen
* avoid sample_submission reading in coding
* code change summary bug fix
* little update
* little refinement to eval
* refine coder and runner eval prompts
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* add prev loops to runner history
* fix evolving history
* fix bug on initializing feedback without final decision
* reformat
* refine
* add comments
* fix ci
* a little refinement
* fix CI
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* feat: add time ratio limit for hyperparameter tuning in Kaggle settings and update evaluator logic
* add recommend time limit
* remove 25% hard line
* small fix
* feat: introduce max_seconds_multiplier for timeout management across components
* avoid using assert in test
* hot fix a very small bug
* runner multiply twice
* revert feedback change
* add a switch to longer timeout
* change refine prompt for full code
* fix: fix the logic of running
* refine prompt
* fix some bugs
* fix
* add two guidelines
* refactor the code
* make costeer evaluator more logical
* refine eval prompt
* make costeer eval prompt markdown
* update code diff prompt
* correct pipeline
* feat: add apply_patch utility and update ret.py with patch functionality (#1071)
* restore to the right version
* fix the docstring
* fix extract_output fcn
* add inplace parameter to apply patch
* remove enable_runner_iteration and make the eval prompt same as main
* refine runner eval prompt based on main
* Update rdagent/scenarios/data_science/dev/runner/prompts.yaml
* add wait_retry
* refactor: move enable_runner_code_diff to DSRunnerCoSTEERSettings as diff_mode
* reformat and remove enable_runner_code_diff
---------
Co-authored-by: yuanteli <1957922024@qq.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Jensen Lee <91518020+Jensen246@users.noreply.github.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Qizheng Li <jenssenlee@163.com>
* fix: refine prompts formatting and logic across data science scenarios
* Make runner evaluator more detailed
* feat: switch to system_debugger prompt and print early stopping stats
* check sample submission & add package constraint
* add trace.log into clear
* change default
* simplify
* clear CI workspace before running
* move to CI
* use sudo to clean workspace
* move prepare out of global var
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* init commit
* remove the 5-fold spec from prompts
* refine the hyperparameter specification
* do not sample data
* a small spelling issue
* refine prompt to avoid submission cheating
* do not sample data
* simplify code
* refine the coder evaluator prompt
* refine wording
* remove runtime from proposal
* refine wording
* refine prompt
* add gpu info in runtime_info.py
* modify the spec
* add router and add refinement exp gen
* fix prompt bug
* use rule-based logic for router
* complete the prompt
* fix circular import bug
* fix bug
* make refine_decision optional
* update pipeline prompts: (1) add scenary: in an iterative cooding loop and use sample datasets (2)add some generation tops in coding (3)add evaluation guidelines in evaluation (4)polish the json schema and description
* fix a small bug
* fix a small bug
* rdagent/scenarios/data_science/loop.py back to the original version
* refactor: replace _get_exp_gen with default_exp_gen for exp generation
* import
* refactor: make the __init__ back to main
* fix small bugs
* fix bugs for proposal_version
* move refine into runner
* check early stop
* EDA improvement & coder classes number
* fix CI
* slightly refine the prompt
* remove rule_base_eval and remove useless prompt
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: amstrongzyf <amstrongzyf@126.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Young <afe.young@gmail.com>
* commit all code
* fix feedback bug
* add debug mode in ds
* update prompt
* prioritize performance than time lit
* use run instead
* store running time
* move all data running into running phase
* fix a bug
* use sample data as default
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* start to work on multi-trace + async
* init ver of async-multi-tarce, to test
* add eng-ver log
* complete version of async+ mul-trace
* debug
* fix bug on DS_RD_SETTING.get()
* update
* fix bug + simplif the usage of async in multi-trace
* fix mini bug of arg_name
* Move local_selection into class Experiment & clean the code
* add coder version
* merge cooder and feedback prompts
* align v2 and v3 proposal prompts
* fix a small bug
* fix a bug
* fix another bug
* support both function calling and json mode in v2 proposal
* fix minor bug
* reformat
* remove proposal v3
* fix a small bug in json mode
* fix CI
* remove tmp file
* remove v3 check
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* rebase selection code
* bug-free run: checkpoint selection and dynamic EDA loading
* add prototypes of various selectors, to imp. and test later
* fix EDA write bug
* imp SOTA-Jump policy
* fix small bug
* allow to set different selector by .env
* add always-win selector
* add init length for AlwaysWinCKPSelector
* add back_jump selector
* auto lint
* add sota_exp_to_submit attribute; change the name of ckp_selector and sota-selector
* fix bug
* auto lint
* working on auto sota selector
* add subtrace counter
* fix bug, remove unuse selector
* add auto sota selector
* auto lint
* fix bug
* fix small logic bug
* add logging
* add inject_diverse feat
* auto lint
* capable to None-select
* feat: add hypothesis_gen config and ExpGen2TraceAndMerge functionality
* refactor: use dynamic import for experiment generator instantiation
* feat: add BestValidSelector for improved SOTA experiment selection
* runnable twin-trace version
* fix logic error of trace-merge
* auto lint
* use import_class to set selector,
* auto-lint
---------
Co-authored-by: Young <afe.young@gmail.com>
* custom data
* fix: simplify competition check and log local description file
* no sample data
* feat: add test evaluation module with error handling support
* fix: update eval path to use eval_sub_dir and add valid_check TODO
* refactor: add MLETestEval check to conditionally run grading steps
* avoid blank stdout
* valid in testeval
* rename test.csv to avoid conflict
* Support Disabling sample submission
* refactoring
* fix: remove DS_KAGGLE_DATA and update prompt instructions
* add try for grade
* ignore submission
* fix: remove tee from eval command and warn about pipeline exit code detection
* optional to use raw description
* support old data
* add execution result to stdout
* add metric to raw description
* custom data explain
* add debug_path
* rst update
---------
Co-authored-by: Young <afe.young@gmail.com>
Star dust