* refine prompt
* small update
* fix a small bug
* remove debug config after execution
* fix: only remove <think> at start
* feat: support creating dataset & multi-eval frame (#1302)
* feat: add iterative evolve and evaluation support with partial chain stop
* feat: add FTDataEvaluator and support multiple implement functions in finetune
* feat: data implement for pre-proposal and proposal and add datasets (#1303)
* feat:(1) support for multi layer dataset extraction (2) add category.json for dataset in datasets/
* fix: fix bug for generate category.json
* feat: add get_dataset_folder_desc
* init data proposal and merge qzli/ft
* update data proposal prompts and add max_position_embeddings and resolve confilcts
* remove sample counts in data proposal
* turn data and train to unified hypo_gen
* refine prompts
* remove category.json and add it to dataset_info
* fix jinja problem and proposal done
* lint
* add ai-generated description and raw readme into dataset_info.json
* update prompt for description
* add datasets
* initial fix for proposal of data
* final version for data proposal
* lint
* feat: add stats in dataset_info, and enable data coder (#1306)
* refactor(dataset): add stats into dataset_info.json, and remove dataset from gitignore_folder
* feat: enable data coder and run data process
* feat: Merge data coder (#1307)
* feat: implement finetune data coding, evaluation, and config improvements
* fix: deepspeed config path
* fix: dataset info columns
---------
Co-authored-by: Young <afe.young@gmail.com>
* replace str length with token_limit
* add readme to dataset_info and remove useless blank lines in scenario description
* feat: dataset prepare
* fix: extract prams script name
* feat: add loss&predictions samples to feedback
* remove duplicate envs and and add llm_api_preferences and enhance reasoning token limits
* feat: network for ft_env
* fix: remove gpt-4o, which has low quota
* feat: a simple ui
* feat: merge data and train task type (#1309)
* feat: filter redundant prams of lf
* fix: ui bug caused by removing task_type
* fix: force agent to use high concurrency, and remove redundant prompt
* feat: extract info from llama factory log, and check data exists before download
* fix: add compatibility rules
* feat: llm evaluator for data coder
* feat: openai package in ft docker, and refine prompt
* feat: refine ft ui, add more info
* feat: add raw logs
* refine data coder prompt(for feedback debug)
* feat: select dataset in scen init
* fix: ui for docker log seperately
* feat: sync log through blob
* improve ui, and add llm feedback in Runner&Exp2FB (#1312)
* fix: ui bug to visualize docker log, and lint
* feat: unified docker log for ft env, and some refactor
* fix bugs and improve ui
* feat: save log of evaluator(single feedback)
* feat: add evaluator, set cleanup docker log
* feat: call llm in RunnerEvaluator and Feedback
* fix: extract structured error message in RunnerEvaluator
* feat: feedback improve, and fix some bugs
* feat: feedback improve when runner fails
* small update
* feat(UI): add running info and benchmark metric in loop expander
* feat(UI): add render markdown toggle
* feat: refine prompts and add error type in exp2fb
* feat: add filterd params reason, set default benchmark timeout to infinite, and refine train loss express
* recover dataset deepscaler
* feat: set timeout in .env
* refactor: unifiied ft_env timeout
* feat: debug mode for data coder
* feat: deliver data_stats after generate debug_data
* feat: use gpt-5.1 as judge model, set judge_retry, and refine debug mode prompt
* refine prompt
* refactor: llama factory manager logic, and refine data processing prompt
* feat(DockerEnv): support GPU selection via CUDA_VISIBLE_DEVICES
* feat: set api concurrency via .env
* fix: ft env timeout bug
* feat: enable CondaEnv run
* fix: can't update bin path in first run, and path bug in lf manager
* feat(ui): set log path through .env
* refactor(ui): wrap_lines, remove css
* feat(coder): retry when parse code-block fail
* fix: refine single-fb in ui, and fix path bug(not allow proposal to decide path)
* fix: opencompass CondaEnv torch compatible with vllm
* fix: refine error text in coding
* feat: deepspeed config for CondaEnv
* feat: memory estimator
* fix: deepspeed package for condaenv
* fix: use `client.chat.completions.create()` only
* feat: flash attention for condaenv
* feat: strong and weak models interface
* fix: condaenv package dependency
* use multi round conversation in llm finetune proposal
* refine prompt for data processing
* enable evolving in data coder
* maximize output token size
* fix: refine ui
* fix: optional packages for llama factory
* fix: torch denpendency for b200
* fix: opencompass dependency
* update cot prompts
* skip the sub implement
* skip conda preparation if env exists
* update chemcot datasets
* fix: unify docker to use litellm
* update readme and instructions
* fix: set CUDA_VISIBLE_DEVICES for CondaEnv
* feat: add panorama dataset, refactor dataset interface
* feat: calculate token using tiktoken, and ndarray bug
* fix: download subtasks of chemcotdataset seperately
* feat: customized prepare func for datasets
* feat: update new benchmarks
* add datasets package
* docs: readme for llm finetune
* feat: download raw data directly, with post-process function
* feat: analyze raw dataset
* suppress litellm debug info
* feat(ui): summary page
* feat: run multi-jobs
* feat: improve ui
* feat: add path and checkout options to LLM finetune loop entrypoint
* feat: add FinanceIQ_ppl benchmark with auto-download and dataset desc rendering
* refactor: remove unused imports and dead code, fix session folder logging
* feat: enable tablebench and tableInstruct dataset
* refine dataset readme, and coder prompt
* refine proposal and coder prompt
* fix: ui path (default log path)
* feat: add automatic LoRA model merging for benchmarking with vLLM
* refactor: reorganize finetune benchmark and merge modules under benchmark dir
* refactor: modularize benchmark config and error extraction for finetune scenario
* fix: update benchmark import paths and disable env cache for device info
* refactor docke&conda env and fix import bugs
* modify init python file
* feat: add FinanceIQ dataset split utility and integrate with pipeline
* feat: set weak and strong model by env, distribute workload across models
* feat: sample dataset and rm params for tensorboard, wandb
* update script to run jobs
* refine proposal prompt, remove specific dataset name
* fix(ui): auto switch log folder
* fix: estimate the processed full data after sample
* feat: filter raw data more aggressively, and lower data_eval standard
* feat: sync workspace to blob
* feat: rdkit for chemcotbench
* update qwen2.5&llama3.1 context
* fix: force failure on validation error and remove try/except in validator
* feat: unified error sample extraction (with test scripts)
* feat: set conda cache with .env
* feat: skip data eval if data pass in last evo
* fix: rm redundant param
* fix ui bug
* refactor: centralize assign_code_list_to_evo in MultiProcessEvolvingStrategy
* feat: add test_params.yaml generation and workspace cleanup improvements for finetune
* refactor: replace get_clear_ws_cmd with clear_workspace and update prompts for hard check criteria
* add bioprobench dataset
* fix: handle commas in training config extraction and refactor prompt includes
* bioprobench description
* add bioprobench readme
* feat: merge lora adapter for blackwell gpu
* feat: support for multi benchmarks in one job
* change dfficult aware content for training
* update difficulty-aware and logging principles
* fix: resolve variable name conflict in FTRunnerEvaluator
* set job id accuracy to minute
* feat(ui): display one selected metric per benchmark
* feat: store sota exp, and fix ws_ckp bug
* fix: truncate data.json in feedback
* fix: opencompass data for conda env
* fix: save only the last model
* feat: set log path and ws path
* fix: set overwrite_cache to avoid lock contention(through injecting params)
* feat: redirect stdout to file in localenv
* add pickle cache to dataset desc
* fix CI
* fix: remove redundant wrapper
* feat: set python_unbuffered
* move redirect stdout to env run
* fix a small bug
* move model folder
* feat(ui): display benchmark baseline
* fix: enrich scenario and benchmark description
* fix: rewrite runner eval to accept easier
* feat: compare with baseline when no SOTA
* update tablebench readme
* fix: switch back to single benchmark (for baseline)
* feat(ui): add ws path in ui
* refactor: update SOTA tracking to use DAG traversal and parent selection
* fix: prioritize local_selection in trace and refactor sibling retrieval logic
* refactor: unify error handling in feedback generation and update workspace injection
* feat: add skip_loop_error_stepname to control error skip step in LoopBase
* fix: set local_selection to NEW_ROOT for experiments without parent
* feat: set different ports for jobs
* feat: set different ports for jobs
* feat: add upper data size limit for LLM fine-tuning and update related prompts
* fix: replace get_truncated_stdout() with stdout for consistent output handling
* refactor: remove data.json from cache and workspace logic, focus on script-based reuse
* fix: rm target_scenario
* feat: add selective cache extraction and custom cache key for data processing
* fix(ui): bug when displaying tablebench
* fix: filter config in dataset_info.json
* feat: add test set, set valid set
* feat(ui): update test score, and set color for final decision
* feat: add test score for baseline and update ui
* fix: use [-100:] as test range
* feat: update data_stats in runner
* feat: wait for opencompass init when run multi jobs
* fix: adjust test&valid split
* feat: force to generate COT(with <think> token), and add answer format in scenarios.json
* feat: improve ui
* fix: unify benchmark volume mounts and set extra_volumes for conda env
* fix(ui): number color
* fix: update GPU memory handling to use total memory in GB and streamline code
* fix: set use_cot_postprocessor
* feat: add env_dict to config classes and merge env vars in Env run
* fix: let coder obey proposal
* fix(ui): direction bug and update chemcot core metirc
* fix: set consistent benchmark mount points and env vars for docker and conda
* fix: addintional target for LoRA
* feat: workspace dir log for benchmark running
* fix: tableInstruct path bug and update benchmark description
* feat: timeout for whole job
* fix: align FinanceIQ import to opencompass
* feat: use llm_judge for FinanceIQ
* feat: switch to turn on <think> or not
* feat: using scripts to redirect stdout, and run in different windows
* feat: sync litellm log
* fix: gpu memory format
* fix: escape special characters in benchmark desc
* fix: set data processing timeout to 1h
* feat: set valid_loss and save_best_model
* fix: inject timeout and stage
* fix: loss history extract logic
* feat: inject output dir
* feat: inject eval batch size
* feat: inject save_total_limit
* feat: update data prompt
* fix: escape shell special characters
* fix: tablebench visualization UI
* fix: move implementation validation to coder, and ignore injected params
* feat: README for FinanceIQ dataset
* fix: bioprobench desc error
* fix: remove task alignment when coder eval
* fix: FinanceIQ now extracts last capital as answer
* fix: stdout contains binary data
* feat: recover estimate full output and set eval setting automatically
* fix(ui): precision for summary table
* fix(ui): import error
* feat: try to use lora
* fix(api): fix litellm bug for code block
* fix: refine prompts to give agent more decision space
* chore(ci): fix mypy typing issues
* chore(ci): format code with black
* chore(ci): fix ruff lint violations
* chore(ci): sort imports with isort
* chore(ci): format code with black
* test: temporarily skip extract_parameters imports due to numpy pin
* fix: compatibility issues for qlib scenarios on finetune branch
* fix(fin_factor): skip to fb for coder error
* fix(loop): default skip to feedback step on skip_loop_error
When skip_loop_error exception happens and skip_loop_error_stepname is not
explicitly set, default to jumping to 'feedback' step if it exists,
otherwise fall back to the last step (record).
This prevents KeyError when record step tries to access feedback data that
doesn't exist because we skipped the feedback phase.
Also removed redundant skip_loop_error_stepname from finetune loop since
it's now the default behavior.
* add 'skip to record' to DS scenario like other scenarios
* fix 2 scenarios bug about rd_loop class
* fix: lint(mypy, ruff, black) error
* fix: mypy lint error
* fix data science scenario bug
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Qizheng Li <jenssenlee@163.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: amstrongzyf <201840057@smail.nju.edu.cn>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: amstrongzyf <amstrongzyf@126.com>
Co-authored-by: chelsea97 <zhuowbrown@gmail.com>
Co-authored-by: SunsetWolf <Lv.Linlang@hotmail.com>
* feat: add interactor classes and user interaction handling for experiments
* update code
* use fragment retry mechanism instead of rerun()
* fix a bug
* integrate user instructions into proposal and coder
* fix CI
* fix CI
* feat: add approval option for user instructions submission
* feat: enhance user instructions handling in Task and DSExperiment classes
* fix CI
* add user instructions into hypothesis rewrite
* add interface to command line
---------
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
* refactor: use get_truncated_stdout for consistent stdout handling across modules
* lint
* feat: add dump_stdout_type to DSRunnerCoSTEERSettings and use in eval
* fix: avoid circular import by moving DSRunnerEvaluator import inside method
* refine prompts and add additional package info
* refine prompts to be specific for GBDT models
* minor refine prompts
* use include to replace duplicate info
* refine prompts
* refactor: import DSTrace from base and remove exp_gen __init__
* lint
---------
Co-authored-by: Young <afe.young@gmail.com>
* feat: refactor CoSTEER classes to use DSCoSTEER and update max seconds handling
* remove useless line
* enable time_ratio_limit_to_enable_hyperparameter_tuning
* add scores.csv metric name in both task_gen and coder
* a little fix to column names
* small fix
* avoid sample submission read in task_gen
* avoid sample_submission reading in coding
* code change summary bug fix
* little update
* little refinement to eval
* refine coder and runner eval prompts
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
* feat: introduce max_seconds_multiplier for timeout management across components
* avoid using assert in test
* hot fix a very small bug
* runner multiply twice
* revert feedback change
* add a switch to longer timeout
* check sample submission & add package constraint
* add trace.log into clear
* change default
* simplify
* clear CI workspace before running
* move to CI
* use sudo to clean workspace
* move prepare out of global var
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* init commit
* remove the 5-fold spec from prompts
* refine the hyperparameter specification
* do not sample data
* a small spelling issue
* refine prompt to avoid submission cheating
* do not sample data
* simplify code
* refine the coder evaluator prompt
* refine wording
* remove runtime from proposal
* refine wording
* refine prompt
* add gpu info in runtime_info.py
* modify the spec
* add router and add refinement exp gen
* fix prompt bug
* use rule-based logic for router
* complete the prompt
* fix circular import bug
* fix bug
* make refine_decision optional
* update pipeline prompts: (1) add scenary: in an iterative cooding loop and use sample datasets (2)add some generation tops in coding (3)add evaluation guidelines in evaluation (4)polish the json schema and description
* fix a small bug
* fix a small bug
* rdagent/scenarios/data_science/loop.py back to the original version
* refactor: replace _get_exp_gen with default_exp_gen for exp generation
* import
* refactor: make the __init__ back to main
* fix small bugs
* fix bugs for proposal_version
* move refine into runner
* check early stop
* EDA improvement & coder classes number
* fix CI
* slightly refine the prompt
* remove rule_base_eval and remove useless prompt
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: TPLin22 <tplin2@163.com>
Co-authored-by: amstrongzyf <amstrongzyf@126.com>
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
Co-authored-by: Young <afe.young@gmail.com>
* commit all code
* fix feedback bug
* add debug mode in ds
* update prompt
* prioritize performance than time lit
* use run instead
* store running time
* move all data running into running phase
* fix a bug
* use sample data as default
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* add coder version
* merge cooder and feedback prompts
* align v2 and v3 proposal prompts
* fix a small bug
* fix a bug
* fix another bug
* support both function calling and json mode in v2 proposal
* fix minor bug
* reformat
* remove proposal v3
* fix a small bug in json mode
* fix CI
* remove tmp file
* remove v3 check
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* feat: replace hard-coded cache paths with dynamic cache_path config
* style: reorder wait_retry import and format chmod list
* refactor: pass workspace_path to chmod command and use DockerConf check
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry
* feat: add async LoopBase with parallel workers and step semaphores
* fix: replace pickle with dill and run blocking tasks via joblib wrapper
* feat: add log format settings, dynamic parallelism & pickle-based snapshot
* fix: default step semaphore to 1 and avoid subprocess when single worker
* merge bowen's changes
* merge tim's changes
* refactor: extract component task mapping, add conditional logger setup
* lint
* refactor: add type hints and safer remain_time metric logging in workflow
* lint
* fix: allow BadRequestError to be pickled via custom copyreg reducer
* fix: stop loop when LoopTerminationError is raised in LoopBase
* lint
* refactor: make log tag context-local using ContextVar for thread safety
* feat: add subproc_step flag and helper to decide subprocess execution
* fix: use ./cache path and normalize relative volume bind paths
* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow
* fix: avoid chmod on cache and input dirs in Env timeout wrapper
* fix: skip chmod on 'cache' and 'input' dirs using find -prune
* fix: restrict chmod to immediate mount dirs excluding cache/input
* fix: chmod cache and input dirs alongside their contents after entry run
* fix: guard chmod with directory checks for cache and input
* fix: prefix mount_path in chmod command for cache/input dirs
* fix: drop quotes from find exclude patterns to ensure chmod executes
* fix: skip chmod on cache/input directories to avoid warning spam
* feat: support string volume mappings and poll subprocess stdout/stderr
* support remove symbolic link
* test: use dynamic home path and code volume in LocalEnv local_simple
* fix: skip trace and progress update when loop step is withdrawn
* refactor: add clean_workspace util and non-destructive workspace backup
* fix: preserve symlinks when backing up workspace with copytree
* fix: prevent AttributeError when _pbar not yet initialized in LoopBase
* perf: replace shutil.copytree with rsync for faster workspace backup
* fix: cast log directory Path to str in tar command of data science loop
* fix: use portable 'cp -r -P' instead of rsync for workspace backup
* fix: add retry and logging to workspace backup for robustness
* refactor: extract backup_folder helper and reuse in DataScienceRDLoop
* fix: propagate backup errors & default _pbar getattr to avoid error
* fix the division by zero bug
* refactor: execute RD loops via asyncio.run and add necessary imports
* lint
* lint
* lint
---------
Co-authored-by: Xu <v-xuminrui@microsoft.com>
* fix model input shape bug and costeer_model bug
* fix a bug
* fix a bug in docker result extraction
* a system-level optimization
* add a filter of stdout
* update
* add stdout to model
* model training_hyperparameters update
* quant scenario
* update some quant settings
* llm choose action
* Thompson Sampling Bandit for action choosing
* refine both scens
* add trace messages for quant scen
* fix some bugs
* fix some bugs
* update
* update
* update
* fix
* fix
* fix
* update for merge
* fix ci
* fix some bugs
* fix ci
* fix ci
* fix ci
* fix ci
* refactor
* default qlib4rdagent local env downloading
* fix ci
* fix ci
* fix a bug
* fix ci
* fix: align all prompts on template (#908)
* use template to render all prompts
* fix CI
---------
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* add fin_quant in cli
* fix a bug
* fix ci
* fix some bugs
* refactor
* remove the columns in hypothesis if no value generated in this column
* fix a bug
* fix ci
* fix conda env
* add qlib gitignore
* remove existed qlib folder & install torch in qlib conda
* fix workspace ui in feedback
* align model config in coder and runner in docker or conda
* fix CI
* fix CI
---------
Co-authored-by: Xu Yang <peteryang@vip.qq.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
* rebase selection code
* bug-free run: checkpoint selection and dynamic EDA loading
* add prototypes of various selectors, to imp. and test later
* fix EDA write bug
* imp SOTA-Jump policy
* fix small bug
* allow to set different selector by .env
* add always-win selector
* add init length for AlwaysWinCKPSelector
* add back_jump selector
* auto lint
* add sota_exp_to_submit attribute; change the name of ckp_selector and sota-selector
* fix bug
* auto lint
* working on auto sota selector
* add subtrace counter
* fix bug, remove unuse selector
* add auto sota selector
* auto lint
* fix bug
* fix small logic bug
* add logging
* add inject_diverse feat
* auto lint
* capable to None-select
* feat: add hypothesis_gen config and ExpGen2TraceAndMerge functionality
* refactor: use dynamic import for experiment generator instantiation
* feat: add BestValidSelector for improved SOTA experiment selection
* runnable twin-trace version
* fix logic error of trace-merge
* auto lint
* use import_class to set selector,
* auto-lint
---------
Co-authored-by: Young <afe.young@gmail.com>
* custom data
* fix: simplify competition check and log local description file
* no sample data
* feat: add test evaluation module with error handling support
* fix: update eval path to use eval_sub_dir and add valid_check TODO
* refactor: add MLETestEval check to conditionally run grading steps
* avoid blank stdout
* valid in testeval
* rename test.csv to avoid conflict
* Support Disabling sample submission
* refactoring
* fix: remove DS_KAGGLE_DATA and update prompt instructions
* add try for grade
* ignore submission
* fix: remove tee from eval command and warn about pipeline exit code detection
* optional to use raw description
* support old data
* add execution result to stdout
* add metric to raw description
* custom data explain
* add debug_path
* rst update
---------
Co-authored-by: Young <afe.young@gmail.com>
TPTBusiness