* add simple rag mcp
* add rag_agent in expGen v2
* add conf config for research rag
* fix CI
* refactor: move context7 and rag config files to new conf modules
* make rag agent general
* fix CI
---------
Co-authored-by: Young <afe.young@gmail.com>
* feat: add interactor classes and user interaction handling for experiments
* update code
* use fragment retry mechanism instead of rerun()
* fix a bug
* integrate user instructions into proposal and coder
* fix CI
* fix CI
* feat: add approval option for user instructions submission
* feat: enhance user instructions handling in Task and DSExperiment classes
* fix CI
* add user instructions into hypothesis rewrite
* add interface to command line
---------
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
* add prev loops to runner history
* fix evolving history
* fix bug on initializing feedback without final decision
* reformat
* refine
* add comments
* fix ci
* a little refinement
* fix CI
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* feat: add time ratio limit for hyperparameter tuning in Kaggle settings and update evaluator logic
* add recommend time limit
* remove 25% hard line
* small fix
* feat: introduce max_seconds_multiplier for timeout management across components
* avoid using assert in test
* hot fix a very small bug
* runner multiply twice
* revert feedback change
* add a switch to longer timeout
* refactor: move exp_gen selection files into select directory
* fix selector value in data_science conf
* fix selector template name
* fix CI
---------
Co-authored-by: Young <afe.young@gmail.com>
* check sample submission & add package constraint
* add trace.log into clear
* change default
* simplify
* clear CI workspace before running
* move to CI
* use sudo to clean workspace
* move prepare out of global var
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* init commit
* remove the 5-fold spec from prompts
* refine the hyperparameter specification
* do not sample data
* a small spelling issue
* refine prompt to avoid submission cheating
* do not sample data
* simplify code
* refine the coder evaluator prompt
* refine wording
* remove runtime from proposal
* refine wording
* refine prompt
* add gpu info in runtime_info.py
* modify the spec
* add router and add refinement exp gen
* fix prompt bug
* use rule-based logic for router
* complete the prompt
* fix circular import bug
* fix bug
* make refine_decision optional
* update pipeline prompts: (1) add scenary: in an iterative cooding loop and use sample datasets (2)add some generation tops in coding (3)add evaluation guidelines in evaluation (4)polish the json schema and description
* fix a small bug
* fix a small bug
* rdagent/scenarios/data_science/loop.py back to the original version
* refactor: replace _get_exp_gen with default_exp_gen for exp generation
* import
* refactor: make the __init__ back to main
* fix small bugs
* fix bugs for proposal_version
* move refine into runner
* check early stop
* EDA improvement & coder classes number
* fix CI
* slightly refine the prompt
* remove rule_base_eval and remove useless prompt
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: amstrongzyf <amstrongzyf@126.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Young <afe.young@gmail.com>
* commit all code
* fix feedback bug
* add debug mode in ds
* update prompt
* prioritize performance than time lit
* use run instead
* store running time
* move all data running into running phase
* fix a bug
* use sample data as default
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* add custom data setting for the data science scene
* fix ci?
* fix ci
* add custom data as an example
* fix ci
* add package
* fix test_import ci error
* fix the logic of kb-inject, allow different verion
* set more flexiable proposal-version change for multi-tarce
* auto-lint
* fix the divede-zero-bug in a trival way
* keep the dump imp. first, update in next version
* use get_sub_trace_count() to get trace_num_count
* fix the conern case bug of divide-zero
* update corner case
* fix the bug
* auto-lint
* fis the bug
* fix the logic bug in max_sota_filter
* fix bug of old version of self.exp_gen.gen
* update the reset_exp_gen_version
* use get_parent_exps to replace all collect_all_ancestors
* auto lint
* fix the bug of reset_exp_gen_version
* fix bug: update V3's old hypothesis_rank
* trival patch on gap of V3 & V2
* make dump patch to unify proposal_V3's dentify_problems
* auto-lint
* fix the bug of sub_trace_count
* fix the problems weights bug
* refactor: remove DSExpGen
* update problems weights calculation
* update problems weights calculation
* remove the selection parameter from exp_gen
* v2 support draft
* v3 also support decomposition
* make the identify_problems an independent function
* fix minor bug
* reformat
* rename exp_num to weighted_exp_num
* add the set_current_selection before the exp_gen when merging
* reformat
* fix wrong selection
* refactor: drop selection arg from ExpGen.gen and DS merge generators
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* set constrains on max_sota_retrieved, fix logis on identical problem
* fix: only Auto SOTA selector use max_sota_retrieved_num
* set max_sota_retrieved_num=10 by default
* minor update
* auto lint
* rebase selection code
* bug-free run: checkpoint selection and dynamic EDA loading
* add prototypes of various selectors, to imp. and test later
* fix EDA write bug
* imp SOTA-Jump policy
* fix small bug
* allow to set different selector by .env
* add always-win selector
* add init length for AlwaysWinCKPSelector
* add back_jump selector
* auto lint
* add sota_exp_to_submit attribute; change the name of ckp_selector and sota-selector
* fix bug
* auto lint
* working on auto sota selector
* add subtrace counter
* fix bug, remove unuse selector
* add auto sota selector
* auto lint
* fix bug
* fix small logic bug
* add logging
* add inject_diverse feat
* auto lint
* capable to None-select
* feat: add hypothesis_gen config and ExpGen2TraceAndMerge functionality
* refactor: use dynamic import for experiment generator instantiation
* feat: add BestValidSelector for improved SOTA experiment selection
* runnable twin-trace version
* fix logic error of trace-merge
* auto lint
* use import_class to set selector,
* auto-lint
---------
Co-authored-by: Young <afe.young@gmail.com>
* custom data
* fix: simplify competition check and log local description file
* no sample data
* feat: add test evaluation module with error handling support
* fix: update eval path to use eval_sub_dir and add valid_check TODO
* refactor: add MLETestEval check to conditionally run grading steps
* avoid blank stdout
* valid in testeval
* rename test.csv to avoid conflict
* Support Disabling sample submission
* refactoring
* fix: remove DS_KAGGLE_DATA and update prompt instructions
* add try for grade
* ignore submission
* fix: remove tee from eval command and warn about pipeline exit code detection
* optional to use raw description
* support old data
* add execution result to stdout
* add metric to raw description
* custom data explain
* add debug_path
* rst update
---------
Co-authored-by: Young <afe.young@gmail.com>
* update all code
* update all code
* dump knowledge base
* rename the tag
* add timer to RD-Agent
* fix CI
* fix CI
* use batch embedding
* fix a small bug
* fix prompt bug
* feat: add pause resume to handle K8S cluster pause (#804)
* add resume to cluster running
* fix non-pickle problem
* fix a small bug
* fix a small bug
* avoid shutil move error
* refine the logic
* move knowledge base out of session
* avoid mistake information to pipeline coding
* avoid load and dump in steps
* archive the right folder
* small improvement
* avoid restart when timer is already started
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* feat: add model dump flag and multi-evaluator support
* tmp code
* refactor: update evaluator feedback and FBWorkspace types
* feat: add get_clear_ws_cmd and CPU count in Docker environment
* feat: Add model dump check level and enhance evaluator functionality
fix data type bug
* fix: Ensure required files exist before model dump evaluation
* refactor: streamline prompt and file checks in model dump evaluation
* fix: add assertions and reorder file reads in model dump evaluator
* feat: remove EDA part from evaluation output
* docs: update dump_model guidelines and eval prompt to include template
* style: reformat multiline dicts and lists in conf and eval files
* fix: add DOTALL flag to EDA removal regex
* init commit
* limit problem numbers
* ensemble lower case
* add runtime and spec to coder
* submission check notice
* sub EDA in sample execution
* avoid lightgbm
* add time limit to scenario
* rephrase the submission check
* give positive feedback when facing warning in check
* ENABLE FEEDBACK
* fix feedback bug
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* first framework commit
* idea proposal v2
Co-authored-by: Roland Minrui <RolandMinrui@users.noreply.github.com>
* fix a small bug in v1
* fix a small bug
* add problem to DShypothesis
* use exp gen as unified interface
* merge yuante's code into pr
* fix a small bug in draft
* update all minrui's code
* small update
* fix small bug & remove useless code
* fix return type
* fix CI
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Roland Minrui <RolandMinrui@users.noreply.github.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* docs: Update prompts and descriptions for data science components
* chore: Remove outdated comments from conf.py
* feat: Add metric_name attribute to DataScienceScen class
* style: Update description in prompts.yaml and reorder metric_name init
* docs: Update prompts.yaml with feature engineering guidelines
* feat: Add spec_enabled configuration for data science settings
* make spec alternative
* change spec logic in exp_gen
* remove some general texts
* align
---------
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: yuanteli <1957922024@qq.com>
* refine ds modal for more cases: eval and es
* update model template
* prompts for model and ensemble
* fix a bug
* fix a bug
* init: ds workflow evovingstrategy
* Adding ensemble (#505)
* Initial Draft
* Updating logic for init
* Revising
* Successful Testing
* Updating to use the latest & right class
* bug: bug-fixing for testing
* data science loop changes
* data science loop base
* ds loop feedback
* fix
* remove measure_time because it's duplicated (in LoopBase)
* add the knowledge query for data_loader & feature
* edit ds workflow evaluator
* data_loader bug fix
* stop evolving when all tasks completed
* llm app change
* fix break all complete strategy
* Adding queried knowledge (#508)
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
* fix loop bug
* ds workflow evaluator; test; refine prompts
* workflow spec
* fix ci
* feature task changes
* ds loop change
* fix a bug in feat
* add query knowledge for model and workflow
* llm_debug info(for show) using pickle instead of json
* remove NextLoopException
* loop change
* coder raise CoderError when all sub_tasks failed
* rename code_dict to file_dict in FBWorkspace
* add CoSTEER unittest
* now show self.version in Task.get_task_information(), simplify CoSTEER sub tasks definition
* remove some properties in ModelTask, add model_type in it.
* fix llm app bug
* llm web app bug fix
* ds loop bug fix
* fix: give component code to feature&ens eval
* loop catch error bug
* rename load_from_raw_data to load_data
* feat: Add debug data creation functionality for data science scenarios
* support local folder (#511)
* support local folder
* remove unnecessary random
* KaggleScen Subclass
* small fix
* use template for style description
* update default scen to kaggle
* update sample data script
* make sure frac < 1
* fix a bug
* feature spec changes
* fix
* changeimport order
* clear unnecessary std outputs
* fix a typo
* create sample folder after unzip kaggle data
* feature/model test script update
* Align the data types across modules.
* fix a bug in model eval
* show line number
* move sample entry point to app
* spec & model prompt changes
* Refine the competition specification to address the data type problem and the coherence issue.
* fix some bugs
* add file filter in FBworkspace.code property
* support non-binary prediction
* avoid too much warnings
* fix a bug in ensemble module
* filtered the knowledge query in all modules
* delete RAG in idea proposal
* refine the code in ensemble
* show exp workspace in llm_st
* exp_gen bug fix
* feedback bug fix
* use `feature` instead of `feat01`
* Trace & method of judging if exp is completed change
* fix a bug in package calling and execute ci
* fix code
* bug fix
* bug fix
* fix a bug
* fix some bugs
* fix a bug
* refactor: Enhance error handling and feedback in data science loop
* support different use_azure on chat and embedding models
* multi-model proposal logic
* fix a small syntax error
* loopBase and some changes
* ensemble scores change
* fbworkspace.code -> .all_codes
* use all model codes in workflow coder
* check scores.csv's keys(model_names)
* model name changes
* add a todo in ensemble test
* sota_exp changes
* give model info in exp gen
* add runner time limit
* config using debug data or not in evals
* exp to feedback base
* add feature code when writing model task
* small problem
* copying during sampling
* update
* refactor: Simplify code handling and improve workspace management
* model part output fix
* print model's execution time
* bug fix
* ensemble test fix
* ens small change
* ens_test bug fix
* Refine partial expansion logic to display only a few subfolders when their structure is uniform, improving readability in nested directories.
* several update on prompts
* sample subfolders
* Filter the stdout after code execution to remove irrelevant information e.g. progress bars, whitespace characters, excessive line breaks.
* Add some more prompts and comments
* several update on the first init rounds
* model timeout as error
* fix pattern of getting model codes in workspace
* small bux fix on model prompts
* remove get_code_with_key since we have regex pattern
* fix: Correct tqdm progress bar update logic in LoopBase class
* feat: Add diff generation and enhance feedback mechanism in data science loop
* update some fix to model and workflow prompts
* refine the logic of progress bar filter
* add last_successful_exp in exp_gen
* fix a one line bug
* add a hint in prompt
* fix data sample for bms
* fix data sample for bms
* hypothesis small fix
* crawler readme update
* fix component gen
* fix bug
* annotation change
* load description.md if it exists
* refactor: Simplify SOTA description handling in feedback and prompts
* refactor: Use shared templates for feedback and experiment descriptions
* change webapp for model codes changes
* update proposal
* add timeout message for docker run output
* fix
* refine the code in docker time processing
* use .shape instead of len() when do shape eval
* won't change size during iteration
* support bson sample
* sample support jsonl and bson
* add former_code to coder prompts
* a little speed us in debug data creating
* filter progress bar when eval ens and main
* avoid costeer makes no change to former code
* fix several log error
* add timeout judge threshold
* fix some bugs in the evaluation of component output shapes
* File structure for supporting litellm (#517)
Co-authored-by: Young <afe.young@gmail.com>
* ignore submission and show processing
* ignore submission and show processing
* add efficiency notice
* refactor: Enhance error message with detailed feedback summary
* refactor: Simplify component handling in DSExpGen class
* refactor: Update code structure and add docstring for clarity
* reserve one sample to each label in data sampling
* add Evaluation info
* refine costeer code to avoid giving same code twice
* use raw_description as plain text
* add a prompt hint to avoid same dict key
* model task name bug in first model exp gen
* fix a typo
* add some debug info in costeer tests
* task init change
* enhance data sampling
* refine the code in data_loader
* more reasonable loop
* fix a bug in data folder description
* add error msg & traceback to execution feedback
* fix llm error msg detection
* add task information to costeer eval & add cache to docker run(use zipfile to store the whole workspace)
* fix CI first round
* fix CI second round
* use txt to store test script to avoid pytest
* remove zipfile in requirements
* add azure.identity to requirements
* ignore debug web page
* component test changes
* remove redundent task_desc in model coder
* feat: Add APE module and prompts for automated prompt engineering
* fix: Update .gitignore and improve text formatting in eval.py
* refactor: Update print output and improve code comments and imports
* style: Fix string formatting and import order in ape.py and fmt.py
* exclude ape
* add a data folder notice
* reduce unnecessary output to stdout
* refine the code of describe_data_folder
* fix ci
* style: streamlit style update (#522)
* streamlit style update
* fix import
* fix format
* fix llm_st loop progress bar
* debugapp small change
* fix model str
* refine some prompts
* fix model str
* fix CI
* refine the logic associated with the data_folder
* fix ci
* small change
* set filter_progress_bar as default in execute
* model proposal with workflow
* add submission check in workflow eval
* fix bug
* small change
* fix CI
* fix CI
* refactor: Move generate_diff to utils and update DSExpGen logic
* more reasonable prompt describing metric direction
* fix a minor jinja2 bug
* quick fix exp_gen bugs
* fix the following bug
* fix
* fix some bugs
* remove workflow from model
* add pending_tasks_list in data science to enable coding model and workflow
* refine the code for handling JSON-formatted data descriptions
* assert with information
* ensure correct csv file name
* add logging to help record the output
* log competition
* add log tag for debug llm app
* test: Test ds refactor ll (#523)
* fix bugs to former scenario
* fix a bug because coding in rdloop changed
* fix the bug when feedback gets no hypothesis
* fix trace structure
* change all trace hist when merging hypothesis to experiments
* ignore some error in ruff
* fix kaggle scenario bugs
* refine one line
* another bug
* another small bug
* fix ui bugs
* chage kaggle train.py path
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* fix CI
* Update rdagent/app/data_science/loop.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add samplecsv into spec prompts
* fix CI
---------
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: yuanteli <1957922024@qq.com>
Co-authored-by: Xisen Wang <118058822+xisen-w@users.noreply.github.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: XianBW <36835909+XianBW@users.noreply.github.com>
Co-authored-by: Tim <illking@foxmail.com>
Co-authored-by: 炼金术师华华 <37462254+YeewahChan@users.noreply.github.com>
Co-authored-by: Linlang <30293408+SunsetWolf@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
XianBW