Commit Graph

661 Commits

Author SHA1 Message Date
XianBW c335ebc25c chore: ui server updates (#951)
* ui server update

* update

* fix bugs

* updates

* use randomname

* fix

* fix bugs

* change time interval in debug server

* some change

* fix CI

* time interval change

* updates

* some changes

* fix

* fix curves

* one IP, one pointers to show msgs return progress

* enable /control

* fix bugs

* fix CI

* fix isort
2025-07-04 12:32:14 +08:00
XianBW ef6b5866c7 feat: show first evo round codes diff (#1009)
* log settings

* add first diff in evolving

* save dict settings

* load settings in data science ui
2025-07-03 19:21:14 +08:00
you-n-g deb01ac5c1 docs: add README for exp_gen folder structure proposal (#1010)
* docs: add README for exp_gen folder structure proposal

* docs: update README with tool strategy and adjust folder structure

* docs: rename 'Tool strategy' section to 'Tools' in exp_gen README
2025-07-03 11:31:51 +08:00
you-n-g 856fbde807 refactor: replace run_ret_code with run and adjust helper methods (#1013) 2025-07-03 11:24:05 +08:00
Yuante Li fb8f2b561e fix: fix a minor bug in DS eval (#1012) 2025-07-02 23:31:29 +08:00
Tim 0077b2caa2 chore: sample data map (#1005)
* chore: map data sampler
2025-07-02 15:11:57 +08:00
Yuante Li 667af3e1ea feat: added running time statistics for the DS scenario experiment (#1007)
* added running time statistics for the DS scenario experiment

* update execute_ret_code to return running_time

* fix

* fix

* update describe

* add EnvResult

* update corresponding calls

* add RunningInfo class

* fix

* fix

* fix

* fix ci

* rename function name

* fix ci

* fix

* refine running_time logic

* fix ci
2025-07-02 15:11:18 +08:00
XianBW cbab4830c3 add code diff in evolving window (#1008) 2025-06-30 20:10:34 +08:00
you-n-g cd237292ea feat: add hide_base_name option and update data folder prompts (#1004)
* feat: add hide_base_name option and update data folder prompts

* rename folder
2025-06-29 21:22:16 +08:00
you-n-g 66e0e8ffd8 feat: add code change summary (#1000)
* feat: add code change summary and dict_get_with_warning util

* feat: support code_change_summary in feedback classes

* lint

* feat: validate response_format using BaseModel and warn unknown formats
2025-06-29 13:40:15 +08:00
amstrongzyf 6c007ee7fb refactor: convert data describe functions to class-based implementation and add output example (#999)
* feat: Enhance data folder description for clarity and robustness

* fix bug

* fix present bugs

* delete useless files

* add output example and refactor the hole util.py

* fix bug for file tree

* add corner case example

* delete useless file
2025-06-28 21:23:40 +08:00
Linlang e3278bfe84 fix: fix mount (#1001)
* docs: document extra_volumes dict format in DockerConf

* feat: accept dict values in extra_volumes to specify bind and mode

* fix: skip invalid PDF reports to prevent infinite loop

* from break to raise self.LoopTerminationError

* format with black

---------

Co-authored-by: Young <afe.young@gmail.com>
2025-06-28 20:01:14 +08:00
xuangu-fang cd0aab3c31 fix: handle the bug of wrong dag_parant index (#996)
* fix the dag_parant_index bug caused by the wrong calling order of

* auto-lint
2025-06-27 17:01:29 +08:00
Yuante Li 7e77a44e11 fix: improve log folder sorting and selection UX (#993)
* fix ui

* update
2025-06-27 11:38:05 +08:00
you-n-g defefd37f5 fix: add async to direct_exp_gen avoid infinite loop (#992)
* refactor: convert direct_exp_gen to async and enforce parallel limit

* fix bug

* change coroutine function position

* fix fin_quant's direct_exp_gen

* format with isort

---------

Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: SunsetWolf <Lv.Linlang@hotmail.com>
2025-06-26 22:10:52 +08:00
xuangu-fang a439d9ef2e feat: async mechanism for multi-trace (#981)
* start to work on multi-trace + async

* init ver of async-multi-tarce, to test

* add eng-ver log

* complete version of async+ mul-trace

* debug

* fix bug on         DS_RD_SETTING.get()

* update

* fix bug + simplif the usage of async in multi-trace

* fix mini bug of arg_name

* Move local_selection into class Experiment & clean the code
2025-06-26 15:49:47 +08:00
you-n-g ea59efc139 fix: refine prompt (#987)
* refactor: rename failed_exp_and_feedback_list to include _after_sota suffix

* refactor: merge prompts_v3 into prompts_v2 and update references
2025-06-26 12:08:31 +08:00
Xu Yang c5fbff82b7 support tar in dataset & fix a small bug in loop (#986)
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-25 15:32:37 +08:00
XianBW 4ec3561c60 fix LLM log bug (#985) 2025-06-25 10:38:20 +08:00
Tim 2aaa20cb0c chore: merge with more trace (#982)
* merge support more traces
* use feedback from all traces
2025-06-24 10:47:45 +08:00
you-n-g b60f5fc506 fix: refine details (#979)
* feat: add parquet preview and extract common DataFrame preview logic

* refactor: improve error messages, prompts, regex, and session loading

* lint
2025-06-23 10:38:27 +08:00
Yuante Li a4206e99b2 fix: fix a bug and update the docs (#978) 2025-06-20 10:58:14 +08:00
Copilot f03e6a9812 fix: docker container cleanup to prevent accumulation and system slowdown (#975)
* Initial plan for issue

* Fix Docker container cleanup issue by using try-finally block

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Fix additional Docker container leaks in health_check and GPU test functions

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Remove temporary test files and finalize Docker container cleanup fix

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Refactor container cleanup code to reduce duplication as requested in review feedback

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Refactor container cleanup to use shared function and always stop before remove

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* fix CI

* Fix mypy type checking errors for Docker container cleanup

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* fix CI

* Remove unnecessary _cleanup_container wrapper method in DockerEnv class

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-19 18:32:50 +08:00
Xu Yang 89a39b9396 fix: merge datascience v3 and v2 (#974)
* add coder version

* merge cooder and feedback prompts

* align v2 and v3 proposal prompts

* fix a small bug

* fix a bug

* fix another bug

* support both function calling and json mode in v2 proposal

* fix minor bug

* reformat

* remove proposal v3

* fix a small bug in json mode

* fix CI

* remove tmp file

* remove v3 check

---------

Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-19 13:25:10 +08:00
Linlang 32a29a7479 fix: main bug (#938)
* feat: parameterize cache paths with USER to avoid conflicts

* guide for missing training_hyperparameters

* guidance for  KeyError: 'concise_reason'

* fixed three bugs in the test

* fix general_model task bug

* fixed some bugs in the med_model scenario

* delete comments

* format with black

* fix mypy error

* fix ruff error

* fix isort error

* sync code

* revert cache_path code

* revert cache_path code

* delete data mining scenario

* fix factor report loop

* fix LiteLLMAPIBackend log_llm_chat_content setting

* refine fin factor report scenario

* remove unused LogColors

* fix UI

* remove medical scenario docs

* change **kaggle** to **data_science**

* remove default dataset_path in create_debug_data

* remove KAGGLE_SETTINGS in kaggle_crawler

* limit litellm versions

* reformat with black

* change README

* fix_data_science_docs

* make hypothesis observations string

* Hiding old versions of kaggle docs

* hidding kaggle agent docs

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: yuanteli <1957922024@qq.com>
2025-06-18 14:35:45 +08:00
Yuante Li 4b838d3f45 refactor: refactor RD-Agent(Q) configuration files (#972)
* refactor rdagent(q) conf files

* fix

* fix ci
2025-06-18 12:28:29 +08:00
Tim 8d1a182292 fix: use simple stdout and stderr (#966)
* use simple stdout and stderr
* add live_output config in LocalConf
2025-06-18 11:54:27 +08:00
Tim 76c8e2d041 fix: get_metric_direction for aerial-cactus-identification (#970) 2025-06-17 18:06:38 +08:00
Yuante Li f73f387c89 docs: add readme for custom data in R&D Agent DS scenario (#968) 2025-06-17 16:29:07 +08:00
Tim b63b79f26f chore: break when loop_n runs out (#964)
* raise loop termination in execute_loop

* add SENTINEL
2025-06-17 15:46:38 +08:00
Yuante Li 4dd69471d7 refactor: add custom data setting for data science scene (#967)
* add custom data setting for the data science scene

* fix ci?

* fix ci

* add custom data as an example

* fix ci

* add package

* fix test_import ci error
2025-06-17 14:55:11 +08:00
XianBW 90b0765561 fix: log info (#965)
* fix log caller_info

* make env info beauty
2025-06-16 19:52:44 +08:00
amstrongzyf 9165d64464 docs: update explanation for separate config use in litellm (#958)
* docs: update explanation for separate config use in litellm

* docs: update default backend to `rdagent.oai.backend.LiteLLMAPIBackend`

* docs: update .rst format

* Update installation_and_configuration.rst
2025-06-13 18:32:39 +08:00
XianBW 3bb4276595 chore: remove redundant tag in old scenarios (#917)
* remove state.times in old ui

* remove "r" tag

* remove "d" tag

* remove "ef" tag

* remove "init" tag

* fix CI

* remove old tag in app UI

* fix bugs

* fix CI

* some updates

* filter tags
2025-06-13 17:49:58 +08:00
you-n-g 0de9c95c17 fix: add missing semicolon after chmod in env shell command (#955) 2025-06-12 21:02:07 +08:00
you-n-g 6505410268 feat: replace hard-coded cache paths with dynamic cache_path config (#952)
* feat: replace hard-coded cache paths with dynamic cache_path config

* style: reorder wait_retry import and format chmod list

* refactor: pass workspace_path to chmod command and use DockerConf check
2025-06-12 17:44:31 +08:00
Xu Yang fb0528e4b4 print out the qlib data generation error message to help user find the problem (#954)
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-12 17:34:51 +08:00
you-n-g 09be71d586 feat: parallel loop running based on asyncio (#932)
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry

* feat: add async LoopBase with parallel workers and step semaphores

* fix: replace pickle with dill and run blocking tasks via joblib wrapper

* feat: add log format settings, dynamic parallelism & pickle-based snapshot

* fix: default step semaphore to 1 and avoid subprocess when single worker

* merge bowen's changes

* merge tim's changes

* refactor: extract component task mapping, add conditional logger setup

* lint

* refactor: add type hints and safer remain_time metric logging in workflow

* lint

* fix: allow BadRequestError to be pickled via custom copyreg reducer

* fix: stop loop when LoopTerminationError is raised in LoopBase

* lint

* refactor: make log tag context-local using ContextVar for thread safety

* feat: add subproc_step flag and helper to decide subprocess execution

* fix: use ./cache path and normalize relative volume bind paths

* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow

* fix: avoid chmod on cache and input dirs in Env timeout wrapper

* fix: skip chmod on 'cache' and 'input' dirs using find -prune

* fix: restrict chmod to immediate mount dirs excluding cache/input

* fix: chmod cache and input dirs alongside their contents after entry run

* fix: guard chmod with directory checks for cache and input

* fix: prefix mount_path in chmod command for cache/input dirs

* fix: drop quotes from find exclude patterns to ensure chmod executes

* fix: skip chmod on cache/input directories to avoid warning spam

* feat: support string volume mappings and poll subprocess stdout/stderr

* support remove symbolic link

* test: use dynamic home path and code volume in LocalEnv local_simple

* fix: skip trace and progress update when loop step is withdrawn

* refactor: add clean_workspace util and non-destructive workspace backup

* fix: preserve symlinks when backing up workspace with copytree

* fix: prevent AttributeError when _pbar not yet initialized in LoopBase

* perf: replace shutil.copytree with rsync for faster workspace backup

* fix: cast log directory Path to str in tar command of data science loop

* fix: use portable 'cp -r -P' instead of rsync for workspace backup

* fix: add retry and logging to workspace backup for robustness

* refactor: extract backup_folder helper and reuse in DataScienceRDLoop

* fix: propagate backup errors & default _pbar getattr to avoid error

* fix the division by zero bug

* refactor: execute RD loops via asyncio.run and add necessary imports

* lint

* lint

* lint

---------

Co-authored-by: Xu <v-xuminrui@microsoft.com>
2025-06-12 11:44:14 +08:00
xuangu-fang 235fcd308a feat: enable to set different version of idea-proposal for multi traces (#895)
* fix the logic of kb-inject, allow different verion

* set more flexiable proposal-version change for multi-tarce

* auto-lint

* fix the divede-zero-bug in a trival way

* keep the dump imp. first, update in next version

* use get_sub_trace_count() to get trace_num_count

* fix the conern case bug of divide-zero

* update corner case

* fix the bug

* auto-lint

* fis the bug

* fix the logic bug in max_sota_filter

* fix bug of old version of self.exp_gen.gen

* update the reset_exp_gen_version

* use get_parent_exps to replace all  collect_all_ancestors

* auto lint

* fix the bug of reset_exp_gen_version

* fix bug: update V3's old hypothesis_rank

* trival patch on gap of V3 & V2

* make dump patch to unify proposal_V3's dentify_problems

* auto-lint

* fix the bug of sub_trace_count
2025-06-11 23:13:30 +08:00
Tim a2e434448a fix: 'DSProposalV2ExpGen' object has no attribute 'COMPONENT_TASK_MAP… (#950)
* fix: 'DSProposalV2ExpGen' object has no attribute 'COMPONENT_TASK_MAPPING'
* chore: add get_component function
2025-06-09 21:49:46 +08:00
Jensen Lee eca0d88e0a fix: filter system metadata dirs and init missing DSTrace attribute (#946)
* fix: filter system metadata dirs and init missing DSTrace attribute

* style:pre-commit
2025-06-07 19:44:23 +08:00
Tim 60f3c99556 chore: continue to read output (#945) 2025-06-07 13:22:56 +08:00
Linlang c8f1c5364a chore: add a rdagent server with UI & logger storage refinement(#553)
* change_log_object

* lint code

* delete comments

* change_log_object

* change_log_object

* fix import test error

* update code

* update code

* fix bugs

* skip mypy error

* skip mypy error

* skip mypy error

* Start the flask server before running the demo.

* achieve front and back interaction

* fix github-advanced-security comments

* fix github-advanced-security comments

* tmp ignore

* fix CI

* move some logic

* change format

* adjust logic

* log2json changes

* tmp

* fix

* fix bug

* refine log2json between 5 scenarios

* fix

* refine codes

* fix logic

* use localhost

* add loop & all_duration param for old scenario startup

* merge control logic

* add README for server ui api

* update README

* reuse code in logger

* add loop_n and all_duration param

* fix upload

* ui server now use port in setting

* fix port setting

* fix port setting

* fix mypy check

* refine logger and log storage

* fix ruff error

* fix CI

* refine logger, loop, storage

* bind one FileStorage with one logger

* not truncate log storage

* refine LoopBase.load(), use `checkout` instead of `output_path` and `do_truncate`

* clear session folder when loading loop to run

* move component info init step to ExpGen Class

* Update rdagent/utils/workflow.py

* move truncate_session function to LoopBase class

* add checkout param for other scenarios

* fix bug

* move WebStorage to UI

* change web_storage name

* add randomname to requirements

* add typer

* fix requirements

---------

Co-authored-by: WinstonLiyte <1957922024@qq.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2025-06-06 18:52:19 +08:00
Haoran Pan 3490719259 feat: update prompt to improve json respond format of some LLM models (#928)
* update prompt to improve json respond format of some LLM models

* fix

* fix

---------

Co-authored-by: WinstonLiyte <1957922024@qq.com>
2025-06-06 11:49:40 +08:00
Tim 6ce765f05d fix: conda error information (#941)
* test with conda error

* reformat
2025-06-06 11:45:48 +08:00
XianBW 83dd0de98a change qlib install method (#937) 2025-06-05 11:40:46 +08:00
XianBW 3e5c587e13 refine debug window logic (#936) 2025-06-05 11:29:37 +08:00
XianBW d0d050eee4 fix detect scenario logic (#935) 2025-06-05 11:19:09 +08:00
XianBW 2ffeb9e3e2 some update (#934) 2025-06-05 10:56:54 +08:00
Tim 7518ab66fa fix: ds trace (#929)
* fix: remove_ansi_codes
* chore: rename function
2025-06-04 18:30:29 +08:00