Commit Graph

176 Commits

Author SHA1 Message Date
TPTBusiness 669263db37 fix: add missing debug() method to RDAgentLog 2026-05-06 21:25:59 +02:00
TPTBusiness 732361bb90 fix(security): resolve path-injection, B701, B101, B112 Bandit alerts
- Path injection (B614): centralized safe_resolve_path in core/utils.py,
  refactored 6 UI modules to use it with safe_root validation
- B701: added explicit autoescape=select_autoescape() to Jinja2
  Environment() calls in 3 files
- B101: replaced assert statements with proper if/raise patterns in
  12+ files (partial)
- B112: added logger.warning() to bare except:continue blocks in
  5 files
2026-05-01 13:42:59 +02:00
TPTBusiness 23b2518c74 fix(security): resolve path-injection and add nosec for safe temp paths (B108, py/path-injection)
- ds_trace.py: resolve() user-provided save path and use Path.name for filenames
  to prevent directory traversal in the local workspace save UI
- rl/finetune UI data_loaders: nosec B614 where paths are already validated
  against safe_root via realpath() before use
- Temp paths (/tmp/sample, /tmp/full, /tmp/mock/*, /tmp/predix_loop.pid,
  /tmp/autorl_output): nosec B108 — fixed Docker volume mount points or
  single-process admin files, not user-writable attack surface

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 19:26:38 +02:00
TPTBusiness 2126062edf fix(security): nosec for B608/B701 false positives in UI and template code
B608: Bandit flags any f-string containing "select" as potential SQL
injection. All four cases (app.py, ds_trace.py, llm_st.py, merge.py)
are Streamlit UI labels or log messages — not database queries.

B701: Jinja2 autoescape=False warnings in coder.py and utils.py are
false positives — these render Python code and plain-text templates,
not HTML. Enabling autoescape would corrupt the rendered code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 09:35:09 +02:00
TPTBusiness 95b14fcfc8 chore(logging): size-based rotation and cap LLM call content
- Switch log rotation from midnight-only ("00:00") to size-based:
  per-command logs: 50 MB, all.log: 100 MB (with gz compression)
- Shorten retention from 30/60 days to 7 days
- Cap llm_calls.jsonl entries to 500 chars per field to prevent
  GB-scale files from long-running loops

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 07:19:27 +02:00
TPTBusiness f24f678713 feat(logging): write complete LLM prompts and responses to daily JSONL log
Add log_llm_call() to daily_log.py that appends every LLM interaction
(system prompt, user prompt, response, duration_ms) as a JSON object to
logs/YYYY-MM-DD/llm_calls.jsonl. Call it from both chat completion paths
in base.py so all LLM activity — factor generation, strategy generation,
feedback, proposals — is captured in a human-readable, grep/jq-friendly
format alongside the existing binary pickle logs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 21:46:28 +02:00
TPTBusiness 2cec08bc91 feat: add daily log rotation, llama health wait, factor auto-fixer, and README updates
- Add rdagent/log/daily_log.py: daily-rotating structured logs per command
  (fin_quant, strategies, evaluate, parallel) with loguru; all.log combined sink
- predix.py: route TeeWriter output to logs/YYYY-MM-DD/ instead of root dir;
  wrap quant() and evaluate() in daily_log.session() for start/stop/duration tracking
- rdagent/app/cli.py: fin_quant_cli waits for llama.cpp /health endpoint before
  starting pipeline (up to 300 s); daily_log integration for fin_quant,
  generate_strategies, eval_all, parallel commands
- scripts/predix_gen_strategies_real_bt.py: daily_log integration with
  per-strategy ACCEPTED/REJECTED entries and summary on completion
- rdagent/components/coder/factor_coder/auto_fixer.py: new module that patches
  common LLM-generated factor issues (min_periods, inf/NaN, groupby.transform,
  MultiIndex corrections)
- rdagent/components/coder/factor_coder/prompts.yaml: add critical rules for
  EURUSD 1-min intraday factors (min_periods, inf handling, groupby, date range)
- README.md: document --reasoning off and --n-gpu-layers 28 for llama-server;
  explain VRAM constraints when Ollama is running alongside llama.cpp
- .bandit.yml: suppress B615 (HuggingFace unsafe download) for RL benchmark files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 07:20:08 +02:00
TPTBusiness 7dff60cf2e fix: Display litellm messages as info instead of warnings 2026-04-06 19:45:04 +02:00
TPTBusiness 30c0a9166e chore: initial Predix state (RD-Agent fork + EURUSD setup) 2026-03-22 21:20:02 +01:00
XianBW 14395488b9 feat: add a web UI server (#1345)
* update rdagent cmd

* fix log error message

* use multiProcessing.Process instead of subprocess.Popen

* add traces to gitignore

* add user interactor in RDLoop (finance scenarios)

* add interactor (feedback, hypothesis) for quant scens

* fix the test_end in qlib conf

* add features init config, general instruction to qlib scenarios

* set base features for based exp

* fix bug when combine factors

* move traces folder to git_ignore_folder

* fix bug in features init

* fix quant interact bug

* fix logger warning error

* bug fixes

* modify rdagent logger, now it can set file output

* adjust cli functions and fix logger bug

* fix server port transport problem

* update server_ui in cli

* add web code

* fix CI problem

* black fix

* update web ui README

* update README

* update readme
2026-03-18 14:04:52 +08:00
Linlang 6196ba31f2 fix: preserve null end_time when rendering dataset segments template (#1326)
* fix: preserve null end_time when rendering dataset segments template

* deps(qlib): bump qlib revision to 2fb9380

* fix: lint error
2026-02-13 10:50:46 +08:00
Linlang 32ecf92afc fix: handle mixed str and dict types in code_list (#1279)
* fix: handle mixed str and dict types in code_list

* fix: handle missing token_costs entry for loop 0 in summarize_win
2025-11-03 21:52:41 +08:00
you-n-g 35a7ae5e1f feat: show the summarized final difference between the final workspace and the base workspace (#1281) 2025-11-03 17:59:46 +08:00
Xu Yang 6e09dc6d69 feat: add user interaction in data science scenario (#1251)
* feat: add interactor classes and user interaction handling for experiments

* update code

* use fragment retry mechanism instead of rerun()

* fix a bug

* integrate user instructions into proposal and coder

* fix CI

* fix CI

* feat: add approval option for user instructions submission

* feat: enhance user instructions handling in Task and DSExperiment classes

* fix CI

* add user instructions into hypothesis rewrite

* add interface to command line

---------

Co-authored-by: Bowen Xian <xianbowen@outlook.com>
2025-09-18 11:10:12 +08:00
XianBW beab473b40 fix: summary page bug (#1219)
* fix summary bug

* fix CI
2025-09-02 17:06:54 +08:00
you-n-g 70fd91cd05 feat: ui, support disable cache (#1217)
* fix: (drop) add cache_enable for functions in ui.utils

* fix: (drop) handle no logging, rename sota_exp to last_sota_exp for clarity in running_win
2025-09-01 15:58:05 +08:00
XianBW 215db05656 chore: ui updates (#1209)
* show a workspace path string for easier copy, only percent existing columns when percent summary dataframe

* catch summary exception

* fix a bug

* fix ci
2025-08-29 15:05:15 +08:00
XianBW 70fdfc89b0 fix ui bug (#1207) 2025-08-28 14:18:58 +08:00
XianBW 3df94e920c chore: refine get_summary_df and sota exp stat in log utils (#1201)
* add best valid selector to get sota exp stat

* remove unused in summary & refine get_summary_df

* fix hypothesis table bug in trace
2025-08-26 18:31:21 +08:00
XianBW 04878f9e70 fix: kaggle competition metric direction (#1195)
* fix leaderboard and competition direction

* ui fix
2025-08-20 16:40:53 +08:00
Tim 3260c0c3ff chore: refine env command (#1193) 2025-08-18 18:16:16 +08:00
XianBW 2f8261f82b fix: ui bug (#1192)
* small bug

* fix ci
2025-08-18 17:48:49 +08:00
XianBW 2c0702a4ac fix ui bugs (#1190) 2025-08-18 10:56:51 +08:00
you-n-g 481e226844 refactor: use UI_SETTING.amlt_path for combined log folder path (#1184) 2025-08-14 21:51:05 +08:00
Tim e0f713c7d3 chore: add merge statistics (#1176)
* chore: add stat for merge
* use best selector
* use trace.hist
2025-08-13 22:59:34 +08:00
XianBW 02069767e1 fix exec_time in summary because of parallel (#1171) 2025-08-08 17:14:02 +08:00
XianBW 6f1a8c63e0 add auto sota selector llm log in trace (#1169) 2025-08-08 16:26:48 +08:00
XianBW 7f90253c4c add parent & root node in trace page (#1164) 2025-08-07 15:02:46 +08:00
XianBW d771b26a26 chore: show timeline and change load_times (#1161)
* show timeline, deperacate old load_times

* fix bug
2025-08-06 18:42:17 +08:00
Tim 35c209b092 feat: enable finetune llm (#1055)
* feat: start with previous workspace

* feat: finetune llm

* add PrevModelLoadEvaluator

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: v-jianwan <v-jianwan@microsoft.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>
2025-08-06 12:24:44 +08:00
XianBW c2bde6dcc3 save more time info (#1151) 2025-08-04 17:28:25 +08:00
XianBW c985e34531 change something for ui compatiblity (#1149) 2025-08-04 12:40:35 +08:00
XianBW 52bf46318f chore: overwrite sota exp stat in summary page (#1146)
* overwrite sota exp stat in summary page

* remove unused import
2025-08-01 18:55:26 +08:00
XianBW d0832eb5ff bug fix (#1145) 2025-08-01 18:07:19 +08:00
XianBW 38aa6eb046 use popover instead of expander (#1138) 2025-07-31 18:14:38 +08:00
XianBW 0ccb6e6d56 log time duration when call LLM and show timeout ratio of coding & running in trace UI (#1128) 2025-07-29 17:54:29 +08:00
XianBW 4cee886c80 fix current_selection maybe empty (#1125) 2025-07-29 11:56:59 +08:00
XianBW e19fe4b233 fix find last SOTA diff (#1124) 2025-07-29 10:56:41 +08:00
XianBW 15fa03d415 fix sota_exp is None (#1121) 2025-07-25 11:53:26 +08:00
XianBW 5025448e84 add lite curves figure (#1119) 2025-07-24 20:11:09 +08:00
XianBW 59a2d7f91d fix curve (#1118) 2025-07-24 19:15:01 +08:00
XianBW e92cd99450 chore: some ui features change (#1117)
* add some todos

* fix bug

* fix leaderboard download function

* summary stat fix

* show medal

* show SOTA Loop

* fix CI
2025-07-24 18:50:29 +08:00
Tim 9f7a4280dc chore: show merge loop in figure (#1105) 2025-07-22 17:02:56 +08:00
Tim 1f142cc5d8 chore: update figure (#1102)
* chore: update figure

* add record id
2025-07-21 18:53:07 +08:00
XianBW c326aa52b9 chore: add response format in chatbot (#1099)
* add response format in chatbot

* fix CI
2025-07-21 12:19:05 +08:00
you-n-g 810af369e0 feat: add loop ID mapping to trace nodes and update UI labels (#1098)
* Add loop index

* feat: add loop ID mapping to trace nodes and update UI labels

* lint

* doc lint
2025-07-21 11:02:37 +08:00
Linlang 81d0ac7185 docs: update data science docs (#1015)
* update data science docs part1

* update data science docs part2

* update data science docs part3

* update data science docs part4

* update data science docs part5

* format with isort

* update data science docs part6

* add check environment scripts

* format with isort

* format with isort

* merge env_check to health_check

* format with isort

* format with black

* optimize code

* use boolean cli options

* replace fire with typer

* replace fire with typer

* optimizing parameter and variable naming
2025-07-19 18:35:24 +08:00
XianBW 3b57264d2c fix ui bugs (#1090) 2025-07-18 16:01:39 +08:00
XianBW 221fea1e15 chore: add a simple chatbot in LLM_LOG window in DataScience UI (#1088)
* refine time show

* updates

* add chatbot in llm_log_win

* use newest litellm
2025-07-17 19:29:25 +08:00
you-n-g 5496168d07 feat: add enable_cache toggle for UI data caching (#1075) 2025-07-15 22:26:06 +08:00