Commit Graph

777 Commits

Author SHA1 Message Date
you-n-g 5c2f015757 chore: release 0.6.1
Release-As: 0.6.1
2025-06-28 20:06:57 +08:00
Linlang e3278bfe84 fix: fix mount (#1001)
* docs: document extra_volumes dict format in DockerConf

* feat: accept dict values in extra_volumes to specify bind and mode

* fix: skip invalid PDF reports to prevent infinite loop

* from break to raise self.LoopTerminationError

* format with black

---------

Co-authored-by: Young <afe.young@gmail.com>
2025-06-28 20:01:14 +08:00
xuangu-fang cd0aab3c31 fix: handle the bug of wrong dag_parant index (#996)
* fix the dag_parant_index bug caused by the wrong calling order of

* auto-lint
2025-06-27 17:01:29 +08:00
Yuante Li 7e77a44e11 fix: improve log folder sorting and selection UX (#993)
* fix ui

* update
2025-06-27 11:38:05 +08:00
you-n-g ecb4223c6e chore: release 0.6.0
Release-As: 0.6.0
2025-06-26 22:32:15 +08:00
you-n-g defefd37f5 fix: add async to direct_exp_gen avoid infinite loop (#992)
* refactor: convert direct_exp_gen to async and enforce parallel limit

* fix bug

* change coroutine function position

* fix fin_quant's direct_exp_gen

* format with isort

---------

Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: SunsetWolf <Lv.Linlang@hotmail.com>
2025-06-26 22:10:52 +08:00
xuangu-fang a439d9ef2e feat: async mechanism for multi-trace (#981)
* start to work on multi-trace + async

* init ver of async-multi-tarce, to test

* add eng-ver log

* complete version of async+ mul-trace

* debug

* fix bug on         DS_RD_SETTING.get()

* update

* fix bug + simplif the usage of async in multi-trace

* fix mini bug of arg_name

* Move local_selection into class Experiment & clean the code
2025-06-26 15:49:47 +08:00
you-n-g ea59efc139 fix: refine prompt (#987)
* refactor: rename failed_exp_and_feedback_list to include _after_sota suffix

* refactor: merge prompts_v3 into prompts_v2 and update references
2025-06-26 12:08:31 +08:00
Xu Yang c5fbff82b7 support tar in dataset & fix a small bug in loop (#986)
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-25 15:32:37 +08:00
XianBW 4ec3561c60 fix LLM log bug (#985) 2025-06-25 10:38:20 +08:00
Tim 2aaa20cb0c chore: merge with more trace (#982)
* merge support more traces
* use feedback from all traces
2025-06-24 10:47:45 +08:00
you-n-g b60f5fc506 fix: refine details (#979)
* feat: add parquet preview and extract common DataFrame preview logic

* refactor: improve error messages, prompts, regex, and session loading

* lint
2025-06-23 10:38:27 +08:00
Yuante Li a4206e99b2 fix: fix a bug and update the docs (#978) 2025-06-20 10:58:14 +08:00
Copilot f03e6a9812 fix: docker container cleanup to prevent accumulation and system slowdown (#975)
* Initial plan for issue

* Fix Docker container cleanup issue by using try-finally block

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Fix additional Docker container leaks in health_check and GPU test functions

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Remove temporary test files and finalize Docker container cleanup fix

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Refactor container cleanup code to reduce duplication as requested in review feedback

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* Refactor container cleanup to use shared function and always stop before remove

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* fix CI

* Fix mypy type checking errors for Docker container cleanup

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

* fix CI

* Remove unnecessary _cleanup_container wrapper method in DockerEnv class

Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: peteryang1 <25981102+peteryang1@users.noreply.github.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-19 18:32:50 +08:00
Xu Yang 89a39b9396 fix: merge datascience v3 and v2 (#974)
* add coder version

* merge cooder and feedback prompts

* align v2 and v3 proposal prompts

* fix a small bug

* fix a bug

* fix another bug

* support both function calling and json mode in v2 proposal

* fix minor bug

* reformat

* remove proposal v3

* fix a small bug in json mode

* fix CI

* remove tmp file

* remove v3 check

---------

Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-19 13:25:10 +08:00
you-n-g c8d611ba9b chore: release 0.5.0
Release-As: 0.5.0
2025-06-18 15:08:11 +08:00
Yuante Li a30ad77afb docs: update readme for v0.5.0 release (#973) 2025-06-18 14:49:24 +08:00
Linlang 32a29a7479 fix: main bug (#938)
* feat: parameterize cache paths with USER to avoid conflicts

* guide for missing training_hyperparameters

* guidance for  KeyError: 'concise_reason'

* fixed three bugs in the test

* fix general_model task bug

* fixed some bugs in the med_model scenario

* delete comments

* format with black

* fix mypy error

* fix ruff error

* fix isort error

* sync code

* revert cache_path code

* revert cache_path code

* delete data mining scenario

* fix factor report loop

* fix LiteLLMAPIBackend log_llm_chat_content setting

* refine fin factor report scenario

* remove unused LogColors

* fix UI

* remove medical scenario docs

* change **kaggle** to **data_science**

* remove default dataset_path in create_debug_data

* remove KAGGLE_SETTINGS in kaggle_crawler

* limit litellm versions

* reformat with black

* change README

* fix_data_science_docs

* make hypothesis observations string

* Hiding old versions of kaggle docs

* hidding kaggle agent docs

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: yuanteli <1957922024@qq.com>
2025-06-18 14:35:45 +08:00
Yuante Li 4b838d3f45 refactor: refactor RD-Agent(Q) configuration files (#972)
* refactor rdagent(q) conf files

* fix

* fix ci
2025-06-18 12:28:29 +08:00
Tim 8d1a182292 fix: use simple stdout and stderr (#966)
* use simple stdout and stderr
* add live_output config in LocalConf
2025-06-18 11:54:27 +08:00
Yuante Li c1a0b7fd17 docs: update the documentation for custom dataset (#969) 2025-06-18 10:58:28 +08:00
Tim 76c8e2d041 fix: get_metric_direction for aerial-cactus-identification (#970) 2025-06-17 18:06:38 +08:00
Yuante Li f73f387c89 docs: add readme for custom data in R&D Agent DS scenario (#968) 2025-06-17 16:29:07 +08:00
Tim b63b79f26f chore: break when loop_n runs out (#964)
* raise loop termination in execute_loop

* add SENTINEL
2025-06-17 15:46:38 +08:00
Yuante Li 4dd69471d7 refactor: add custom data setting for data science scene (#967)
* add custom data setting for the data science scene

* fix ci?

* fix ci

* add custom data as an example

* fix ci

* add package

* fix test_import ci error
2025-06-17 14:55:11 +08:00
XianBW 90b0765561 fix: log info (#965)
* fix log caller_info

* make env info beauty
2025-06-16 19:52:44 +08:00
amstrongzyf 9165d64464 docs: update explanation for separate config use in litellm (#958)
* docs: update explanation for separate config use in litellm

* docs: update default backend to `rdagent.oai.backend.LiteLLMAPIBackend`

* docs: update .rst format

* Update installation_and_configuration.rst
2025-06-13 18:32:39 +08:00
XianBW 3bb4276595 chore: remove redundant tag in old scenarios (#917)
* remove state.times in old ui

* remove "r" tag

* remove "d" tag

* remove "ef" tag

* remove "init" tag

* fix CI

* remove old tag in app UI

* fix bugs

* fix CI

* some updates

* filter tags
2025-06-13 17:49:58 +08:00
you-n-g 0de9c95c17 fix: add missing semicolon after chmod in env shell command (#955) 2025-06-12 21:02:07 +08:00
you-n-g 6505410268 feat: replace hard-coded cache paths with dynamic cache_path config (#952)
* feat: replace hard-coded cache paths with dynamic cache_path config

* style: reorder wait_retry import and format chmod list

* refactor: pass workspace_path to chmod command and use DockerConf check
2025-06-12 17:44:31 +08:00
Xu Yang fb0528e4b4 print out the qlib data generation error message to help user find the problem (#954)
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-12 17:34:51 +08:00
you-n-g 09be71d586 feat: parallel loop running based on asyncio (#932)
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry

* feat: add async LoopBase with parallel workers and step semaphores

* fix: replace pickle with dill and run blocking tasks via joblib wrapper

* feat: add log format settings, dynamic parallelism & pickle-based snapshot

* fix: default step semaphore to 1 and avoid subprocess when single worker

* merge bowen's changes

* merge tim's changes

* refactor: extract component task mapping, add conditional logger setup

* lint

* refactor: add type hints and safer remain_time metric logging in workflow

* lint

* fix: allow BadRequestError to be pickled via custom copyreg reducer

* fix: stop loop when LoopTerminationError is raised in LoopBase

* lint

* refactor: make log tag context-local using ContextVar for thread safety

* feat: add subproc_step flag and helper to decide subprocess execution

* fix: use ./cache path and normalize relative volume bind paths

* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow

* fix: avoid chmod on cache and input dirs in Env timeout wrapper

* fix: skip chmod on 'cache' and 'input' dirs using find -prune

* fix: restrict chmod to immediate mount dirs excluding cache/input

* fix: chmod cache and input dirs alongside their contents after entry run

* fix: guard chmod with directory checks for cache and input

* fix: prefix mount_path in chmod command for cache/input dirs

* fix: drop quotes from find exclude patterns to ensure chmod executes

* fix: skip chmod on cache/input directories to avoid warning spam

* feat: support string volume mappings and poll subprocess stdout/stderr

* support remove symbolic link

* test: use dynamic home path and code volume in LocalEnv local_simple

* fix: skip trace and progress update when loop step is withdrawn

* refactor: add clean_workspace util and non-destructive workspace backup

* fix: preserve symlinks when backing up workspace with copytree

* fix: prevent AttributeError when _pbar not yet initialized in LoopBase

* perf: replace shutil.copytree with rsync for faster workspace backup

* fix: cast log directory Path to str in tar command of data science loop

* fix: use portable 'cp -r -P' instead of rsync for workspace backup

* fix: add retry and logging to workspace backup for robustness

* refactor: extract backup_folder helper and reuse in DataScienceRDLoop

* fix: propagate backup errors & default _pbar getattr to avoid error

* fix the division by zero bug

* refactor: execute RD loops via asyncio.run and add necessary imports

* lint

* lint

* lint

---------

Co-authored-by: Xu <v-xuminrui@microsoft.com>
2025-06-12 11:44:14 +08:00
xuangu-fang 235fcd308a feat: enable to set different version of idea-proposal for multi traces (#895)
* fix the logic of kb-inject, allow different verion

* set more flexiable proposal-version change for multi-tarce

* auto-lint

* fix the divede-zero-bug in a trival way

* keep the dump imp. first, update in next version

* use get_sub_trace_count() to get trace_num_count

* fix the conern case bug of divide-zero

* update corner case

* fix the bug

* auto-lint

* fis the bug

* fix the logic bug in max_sota_filter

* fix bug of old version of self.exp_gen.gen

* update the reset_exp_gen_version

* use get_parent_exps to replace all  collect_all_ancestors

* auto lint

* fix the bug of reset_exp_gen_version

* fix bug: update V3's old hypothesis_rank

* trival patch on gap of V3 & V2

* make dump patch to unify proposal_V3's dentify_problems

* auto-lint

* fix the bug of sub_trace_count
2025-06-11 23:13:30 +08:00
Haoran Pan b6c0e891d3 docs: update llm setting guidance and "REASONING_THINK_RM" description (#943)
* update llm setting guidance and "REASONING_THINK_RM" description

* remove deprecated backend in readme
2025-06-11 15:45:22 +08:00
Tim a2e434448a fix: 'DSProposalV2ExpGen' object has no attribute 'COMPONENT_TASK_MAP… (#950)
* fix: 'DSProposalV2ExpGen' object has no attribute 'COMPONENT_TASK_MAPPING'
* chore: add get_component function
2025-06-09 21:49:46 +08:00
Jensen Lee eca0d88e0a fix: filter system metadata dirs and init missing DSTrace attribute (#946)
* fix: filter system metadata dirs and init missing DSTrace attribute

* style:pre-commit
2025-06-07 19:44:23 +08:00
you-n-g 76d87d241b docs: Update README.md (#948) 2025-06-07 15:24:25 +08:00
Yuante Li 9240a71c54 docs: update document for RD-Agent(Q) (#940) 2025-06-07 14:41:02 +08:00
Tim 60f3c99556 chore: continue to read output (#945) 2025-06-07 13:22:56 +08:00
Linlang c8f1c5364a chore: add a rdagent server with UI & logger storage refinement(#553)
* change_log_object

* lint code

* delete comments

* change_log_object

* change_log_object

* fix import test error

* update code

* update code

* fix bugs

* skip mypy error

* skip mypy error

* skip mypy error

* Start the flask server before running the demo.

* achieve front and back interaction

* fix github-advanced-security comments

* fix github-advanced-security comments

* tmp ignore

* fix CI

* move some logic

* change format

* adjust logic

* log2json changes

* tmp

* fix

* fix bug

* refine log2json between 5 scenarios

* fix

* refine codes

* fix logic

* use localhost

* add loop & all_duration param for old scenario startup

* merge control logic

* add README for server ui api

* update README

* reuse code in logger

* add loop_n and all_duration param

* fix upload

* ui server now use port in setting

* fix port setting

* fix port setting

* fix mypy check

* refine logger and log storage

* fix ruff error

* fix CI

* refine logger, loop, storage

* bind one FileStorage with one logger

* not truncate log storage

* refine LoopBase.load(), use `checkout` instead of `output_path` and `do_truncate`

* clear session folder when loading loop to run

* move component info init step to ExpGen Class

* Update rdagent/utils/workflow.py

* move truncate_session function to LoopBase class

* add checkout param for other scenarios

* fix bug

* move WebStorage to UI

* change web_storage name

* add randomname to requirements

* add typer

* fix requirements

---------

Co-authored-by: WinstonLiyte <1957922024@qq.com>
Co-authored-by: Bowen Xian <xianbowen@outlook.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2025-06-06 18:52:19 +08:00
you-n-g fc0fcc3700 chore: update load add & (#944) 2025-06-06 16:58:09 +08:00
Haoran Pan 3490719259 feat: update prompt to improve json respond format of some LLM models (#928)
* update prompt to improve json respond format of some LLM models

* fix

* fix

---------

Co-authored-by: WinstonLiyte <1957922024@qq.com>
2025-06-06 11:49:40 +08:00
Tim 6ce765f05d fix: conda error information (#941)
* test with conda error

* reformat
2025-06-06 11:45:48 +08:00
XianBW 83dd0de98a change qlib install method (#937) 2025-06-05 11:40:46 +08:00
XianBW 3e5c587e13 refine debug window logic (#936) 2025-06-05 11:29:37 +08:00
XianBW d0d050eee4 fix detect scenario logic (#935) 2025-06-05 11:19:09 +08:00
XianBW 2ffeb9e3e2 some update (#934) 2025-06-05 10:56:54 +08:00
Tim 7518ab66fa fix: ds trace (#929)
* fix: remove_ansi_codes
* chore: rename function
2025-06-04 18:30:29 +08:00
Tim bb71c18080 feat: merge selectively (#888)
* chore: avoid incorporate changes
best as sota
merge hypothesis
fix: max_retrieve_num after decision
chore: select last experiments and feedbacks
* add the set_current_selection before the exp_gen when merging
add trace.NEW_ROOT
fix: no scen_prob_multiplier
fix: use regex with timeout
chore: hypothesis_rank with selected_idx
chore: define is_parent in proposal
chore: rename collect_all_ancestors to get_parent_exps
---------

Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: Xu <v-xuminrui@microsoft.com>
Co-authored-by: Xu Yang <xuyang1@microsoft.com>
2025-06-04 14:46:47 +08:00
you-n-g 4aae5b6854 feat: log reaching max time limit before breaking CoSTEER evolution (#921) 2025-06-03 12:51:56 +08:00