chore(master): release 1.3.8 (#42)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

.github/workflows/release.yml

@@ -12,7 +12,7 @@ jobs:
   release-please:
     runs-on: ubuntu-latest
     steps:
-      - uses: googleapis/release-please-action@v4
+      - uses: googleapis/release-please-action@v5
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           config-file: release-please-config.json

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.3.5"
+  ".": "1.3.8"
 }

CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## [1.3.8](https://github.com/TPTBusiness/Predix/compare/v1.3.7...v1.3.8) (2026-04-30)
+
+
+### Bug Fixes
+
+* **deps:** relax aiohttp constraint to >=3.13.4 for litellm compatibility ([34ab192](https://github.com/TPTBusiness/Predix/commit/34ab1923a887089eb36e5cbad6cb8df16f0333ca))
+* **qlib:** correct indentation in except blocks in quant_proposal and factor_runner ([8143451](https://github.com/TPTBusiness/Predix/commit/8143451e8c0ead01c4d86d19669268c7bfb15fac))
+* **security:** replace eval() with ast.literal_eval in finetune validator (B307) ([0508caf](https://github.com/TPTBusiness/Predix/commit/0508caf9140d210b823fefefa28ee535ec85a0ae))
+* **security:** replace shell=True subprocess calls with list args in env.py (B602) ([2012d5a](https://github.com/TPTBusiness/Predix/commit/2012d5ae4e77cc2f1ab9a48beaaac5a74695d083))
+* **security:** resolve path-injection and add nosec for safe temp paths (B108, py/path-injection) ([6727480](https://github.com/TPTBusiness/Predix/commit/67274803bd1d14e5d1df9a063f46b2edb8501a2b))
+
+## [1.3.7](https://github.com/TPTBusiness/Predix/compare/v1.3.6...v1.3.7) (2026-04-30)
+
+
+### Bug Fixes
+
+* **security:** nosec for B608/B701 false positives in UI and template code ([5eb5d7e](https://github.com/TPTBusiness/Predix/commit/5eb5d7e8fdbe90e0dced83fef4e09f5a33e96b2b))
+* **security:** replace eval() with ast.literal_eval and add request timeouts (B307, B113) ([3301ada](https://github.com/TPTBusiness/Predix/commit/3301ada697ca7d3afa1a188d2a76a87ae98b4529))
+* **security:** replace shell=True subprocess calls with list args (B602) ([13c08f4](https://github.com/TPTBusiness/Predix/commit/13c08f4ce6813eb7c314087921ec8c0f40074bd7))
+
+## [1.3.6](https://github.com/TPTBusiness/Predix/compare/v1.3.5...v1.3.6) (2026-04-30)
+
+
+### Bug Fixes
+
+* **security:** real fix for B110 (logging in factor_proposal.py [#746](https://github.com/TPTBusiness/Predix/issues/746)) ([16624e0](https://github.com/TPTBusiness/Predix/commit/16624e0bd966ae4d24c4a3eb42bbc31c11da3136))
+* **security:** real fix for B110 (logging in factor_runner.py [#744](https://github.com/TPTBusiness/Predix/issues/744)) ([88cf0fb](https://github.com/TPTBusiness/Predix/commit/88cf0fb8828b11c97f2f3ae2881a4900b020c6f0))
+* **security:** real fix for B110 (logging in quant_proposal.py [#741](https://github.com/TPTBusiness/Predix/issues/741)) ([7cf2a64](https://github.com/TPTBusiness/Predix/commit/7cf2a644f553b054bd4b0607ea51e5372e68d90a))
+* **security:** real fix for B110 (logging in quant_proposal.py [#741](https://github.com/TPTBusiness/Predix/issues/741)) ([ef985f8](https://github.com/TPTBusiness/Predix/commit/ef985f86035d8dca707c60137e6508349a0c4ae6))
+* **security:** real fix for B404/B603 (sys.executable in factor_runner.py [#745](https://github.com/TPTBusiness/Predix/issues/745)) ([819655a](https://github.com/TPTBusiness/Predix/commit/819655aaa3efa76596d60501d0e8ca365df3e5e2))
+* **security:** revert broken read_pickle encoding arg in kaggle template (B301) ([3574907](https://github.com/TPTBusiness/Predix/commit/35749073c91e69f63ddaad61dae3f2b799327e63))
+* **security:** validate SQL identifiers in _add_column_if_not_exists (B608) ([e10dfa2](https://github.com/TPTBusiness/Predix/commit/e10dfa2576038e911f83595d3b466c261bc0cd54))
+* **security:** whitelist-validate metric column in get_top_factors (B608) ([e50519f](https://github.com/TPTBusiness/Predix/commit/e50519fe066e68aec2f19b83df4f643c3c22053d))
+
 ## [1.3.5](https://github.com/TPTBusiness/Predix/compare/v1.3.4...v1.3.5) (2026-04-27)
 
 

rdagent/app/cli.py

@@ -1251,7 +1251,7 @@ def start_loop_cli(
     script_dir = str(Path(__file__).parent.parent.parent.parent)
     generator = f"python {script_dir}/scripts/predix_smart_strategy_gen.py"
     logfile = f"{script_dir}/results/logs/generator_loop.log"
-    pidfile = "/tmp/predix_loop.pid"
+    pidfile = "/tmp/predix_loop.pid"  # nosec B108 — administrative PID file, single-process daemon
 
     os.makedirs(f"{script_dir}/results/logs", exist_ok=True)
 

rdagent/app/finetune/llm/ui/data_loader.py

@@ -93,7 +93,7 @@ def get_valid_sessions(log_folder: Path, safe_root: Path | None = None) -> list[
         folder_real = os.path.realpath(str(log_folder.expanduser()))
         if not (folder_real == root_real or folder_real.startswith(root_real + os.sep)):
             return []
-        log_folder = Path(folder_real)
+        log_folder = Path(folder_real)  # nosec B614 — path validated against safe_root via realpath above
 
     if not log_folder.exists():
         return []
@@ -379,7 +379,7 @@ def load_ft_session(log_path: Path, safe_root: Path | None = None) -> Session:
         path_real = os.path.realpath(str(log_path.expanduser()))
         if not (path_real == root_real or path_real.startswith(root_real + os.sep)):
             return Session()
-        log_path = Path(path_real)
+        log_path = Path(path_real)  # nosec B614 — path validated against safe_root via realpath above
 
     session = Session()
     storage = FileStorage(log_path)

rdagent/app/rl/ui/app.py

@@ -82,7 +82,7 @@ def get_job_options(base_path: Path, safe_root: Path | None = None) -> list[str]
 
     # Security fix: Validate base_path to prevent path traversal
     try:
-        base_path_resolved = base_path.expanduser().resolve()
+        base_path_resolved = base_path.expanduser().resolve()  # nosec B614 — validated against safe_root below via relative_to()
 
         if safe_root is not None:
             safe_root_resolved = safe_root.expanduser().resolve()

rdagent/app/rl/ui/data_loader.py

@@ -80,7 +80,7 @@ def get_valid_sessions(log_folder: Path, safe_root: Path | None = None) -> list[
         folder_real = os.path.realpath(str(log_folder.expanduser()))
         if not (folder_real == root_real or folder_real.startswith(root_real + os.sep)):
             return []
-        log_folder = Path(folder_real)
+        log_folder = Path(folder_real)  # nosec B614 — path validated against safe_root via realpath above
 
     if not log_folder.exists():
         return []
@@ -251,7 +251,7 @@ def load_session(log_path: Path, safe_root: Path | None = None) -> Session:
         path_real = os.path.realpath(str(log_path.expanduser()))
         if not (path_real == root_real or path_real.startswith(root_real + os.sep)):
             return Session()
-        log_path = Path(path_real)
+        log_path = Path(path_real)  # nosec B614 — path validated against safe_root via realpath above
 
     session = Session()
 

rdagent/app/utils/info.py

@@ -54,11 +54,11 @@ def rdagent_info():
     current_version = importlib.metadata.version("rdagent")
     logger.info(f"RD-Agent version: {current_version}")
     api_url = f"https://api.github.com/repos/microsoft/RD-Agent/contents/requirements.txt?ref=main"
-    response = requests.get(api_url)
+    response = requests.get(api_url, timeout=30)
     if response.status_code == 200:
         files = response.json()
         file_url = files["download_url"]
-        file_response = requests.get(file_url)
+        file_response = requests.get(file_url, timeout=30)
         if file_response.status_code == 200:
             all_file_contents = file_response.text.split("\n")
         else:

rdagent/components/backtesting/results_db.py

@@ -71,6 +71,9 @@ class ResultsDatabase:
 
         self.conn.commit()
 
+    _ALLOWED_TABLES = frozenset({"factors", "backtest_runs", "loop_results"})
+    _ALLOWED_COL_TYPES = frozenset({"REAL", "TEXT", "INTEGER", "BLOB"})
+
     def _add_column_if_not_exists(self, table: str, column: str, col_type: str) -> None:
         """
         Add a column to a table if it doesn't already exist.
@@ -78,20 +81,24 @@ class ResultsDatabase:
         Parameters
         ----------
         table : str
-            Table name
+            Table name (must be in _ALLOWED_TABLES)
         column : str
-            Column name to add
+            Column name to add (alphanumeric + underscore only)
         col_type : str
-            SQL column type (e.g., 'REAL', 'TEXT')
+            SQL column type (must be in _ALLOWED_COL_TYPES)
         """
+        if table not in self._ALLOWED_TABLES:
+            raise ValueError(f"Unknown table: {table!r}")
+        if not column.replace("_", "").isalnum():
+            raise ValueError(f"Invalid column name: {column!r}")
+        if col_type not in self._ALLOWED_COL_TYPES:
+            raise ValueError(f"Invalid column type: {col_type!r}")
+
         c = self.conn.cursor()
-        try:
-            # Try to query the column - if it fails, it doesn't exist
-            # nosec B608: Internal schema migration, column names are controlled
-            c.execute(f"SELECT {column} FROM {table} LIMIT 1")  # nosec B608
-        except sqlite3.OperationalError:
-            # Column doesn't exist, add it
-            c.execute(f"ALTER TABLE {table} ADD COLUMN {column} {col_type}")  # nosec B608
+        c.execute("SELECT name FROM pragma_table_info(?)", (table,))
+        existing = {row[0] for row in c.fetchall()}
+        if column not in existing:
+            c.execute(f"ALTER TABLE {table} ADD COLUMN {column} {col_type}")
     
     def add_factor(self, name: str, type: str = "unknown") -> int:
         c = self.conn.cursor()
@@ -183,16 +190,18 @@ class ResultsDatabase:
         pd.DataFrame
             DataFrame with factor names and metrics
         """
-        # Map shorthand to full column name
+        _ALLOWED_METRICS = frozenset({
+            'sharpe', 'ic', 'annual_return', 'max_drawdown',
+            'win_rate', 'information_ratio', 'volatility',
+        })
         metric_map = {
-            'sharpe': 'sharpe',
-            'ic': 'ic',
-            'return': 'annual_return',
-            'drawdown': 'max_drawdown',
-            'win_rate': 'win_rate',
+            'sharpe': 'sharpe', 'ic': 'ic', 'return': 'annual_return',
+            'drawdown': 'max_drawdown', 'win_rate': 'win_rate',
             'information_ratio': 'information_ratio',
         }
         col = metric_map.get(metric, metric)
+        if col not in _ALLOWED_METRICS:
+            raise ValueError(f"Unknown metric: {metric!r}")
 
         return pd.read_sql_query(
             f"""SELECT factor_name, ic, sharpe, annual_return, max_drawdown,
@@ -201,7 +210,7 @@ class ResultsDatabase:
                 JOIN factors ON factor_id = factors.id
                 WHERE {col} IS NOT NULL
                 ORDER BY {col} DESC
-                LIMIT ?""",
+                LIMIT ?""",  # nosec B608 — col is validated against _ALLOWED_METRICS above
             self.conn,
             params=[limit]
         )

rdagent/components/coder/factor_coder/factor.py

@@ -161,8 +161,7 @@ class FactorFBWorkspace(FBWorkspace):
 
             try:
                 subprocess.check_output(
-                    f"{FACTOR_COSTEER_SETTINGS.python_bin} {execution_code_path}",
-                    shell=True,
+                    [FACTOR_COSTEER_SETTINGS.python_bin, str(execution_code_path)],
                     cwd=self.workspace_path,
                     stderr=subprocess.STDOUT,
                     timeout=FACTOR_COSTEER_SETTINGS.file_based_execution_timeout,

rdagent/components/coder/finetune/unified_validator.py

@@ -6,6 +6,7 @@ Two-step validation:
 2. Micro-batch testing - Runtime validation with small dataset
 """
 
+import ast
 import json
 import re
 import time
@@ -229,7 +230,7 @@ class LLMConfigValidator:
             final_metrics = re.search(r"\{'train_runtime':[^}]+\}", stdout)
             if final_metrics:
                 try:
-                    metrics = eval(final_metrics.group(0))  # Safe: only numbers and strings
+                    metrics = ast.literal_eval(final_metrics.group(0))
                     result["final_metrics"] = {
                         "train_loss": metrics.get("train_loss"),
                         "train_runtime": metrics.get("train_runtime"),

rdagent/log/daily_log.py

@@ -27,6 +27,7 @@ Usage:
 from __future__ import annotations
 
 import json as _json
+import logging
 import sys
 import threading
 from contextlib import contextmanager
@@ -36,21 +37,24 @@ from typing import Any
 
 from loguru import logger as _root
 
-# ── paths ─────────────────────────────────────────────────────────────────────
+# ── paths ─────────────────────────────────────────────────────────────────────────────────
 LOGS_ROOT: Path = Path(__file__).parent.parent.parent / "logs"
 
-# ── format ────────────────────────────────────────────────────────────────────
+# ── format ────────────────────────────────────────────────────────────────────────────────
 _FILE_FMT = (
     "{time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | {extra[cmd]: <18} | {message}"
 )
 
-# ── internal state ─────────────────────────────────────────────────────────────
+# ── internal state ─────────────────────────────────────────────────────────────────────────────
 _registered: set[str] = set()   # command keys that already have a file sink
 _all_added: bool = False         # whether the combined all.log sink is active
 _llm_log_lock = threading.Lock()  # guards concurrent writes to llm_calls.jsonl
 
+# Maximum characters stored per field in llm_calls.jsonl to prevent GB-scale files.
+_LLM_CALL_MAX_CHARS = 500
 
-# ── helpers ───────────────────────────────────────────────────────────────────
+
+# ── helpers ────────────────────────────────────────────────────────────────────────────────
 
 def _today_dir() -> Path:
     d = LOGS_ROOT / datetime.now().strftime("%Y-%m-%d")
@@ -79,7 +83,7 @@ def _banner(log, title: str, meta: dict[str, Any]) -> None:
     log.info(sep)
 
 
-# ── public API ────────────────────────────────────────────────────────────────
+# ── public API ──────────────────────────────────────────────────────────────────────────────
 
 def log_llm_call(
     system: str | None,
@@ -88,16 +92,19 @@ def log_llm_call(
     start_time: Any = None,
     end_time: Any = None,
 ) -> None:
-    """Append one complete LLM call to logs/YYYY-MM-DD/llm_calls.jsonl.
+    """Append one LLM call summary to logs/YYYY-MM-DD/llm_calls.jsonl.
+
+    Prompt/response content is capped at _LLM_CALL_MAX_CHARS to prevent
+    GB-scale log files from long-running loops.
 
     Each line is a self-contained JSON object so the file is grep/jq-friendly:
       jq 'select(.duration_ms > 5000)' logs/2026-04-17/llm_calls.jsonl
     """
     entry: dict[str, Any] = {
         "ts": datetime.now().isoformat(timespec="milliseconds"),
-        "system": system or "",
-        "user": user,
-        "response": response,
+        "system": (system or "")[:_LLM_CALL_MAX_CHARS],
+        "user": user[:_LLM_CALL_MAX_CHARS],
+        "response": response[:_LLM_CALL_MAX_CHARS],
     }
     if start_time is not None and end_time is not None:
         try:
@@ -130,13 +137,13 @@ def setup(command: str, **context: Any):
     key = command.lower()
 
     if key not in _registered:
-        # Per-command rotating file
         _root.add(
             str(log_dir / f"{key}.log"),
             format=_FILE_FMT,
             filter=lambda r, k=key: r["extra"].get("cmd", "").lower() == k,
-            rotation="00:00",      # new file at midnight
-            retention="30 days",
+            rotation="50 MB",
+            compression="gz",
+            retention="7 days",
             encoding="utf-8",
             enqueue=True,
             backtrace=False,
@@ -145,13 +152,13 @@ def setup(command: str, **context: Any):
         _registered.add(key)
 
     if not _all_added:
-        # Combined log — all commands
         _root.add(
             str(log_dir / "all.log"),
             format=_FILE_FMT,
             filter=lambda r: "cmd" in r["extra"],
-            rotation="00:00",
-            retention="60 days",
+            rotation="100 MB",
+            compression="gz",
+            retention="7 days",
             encoding="utf-8",
             enqueue=True,
             backtrace=False,

rdagent/log/ui/llm_st.py

@@ -0,0 +1,306 @@
+import argparse
+import json
+import pickle  # nosec
+import re
+import time
+from pathlib import Path
+
+import streamlit as st
+from streamlit import session_state
+
+from rdagent.log.ui.conf import UI_SETTING
+from rdagent.log.utils import extract_evoid, extract_loopid_func_name
+
+st.set_page_config(layout="wide", page_title="debug_llm", page_icon="🎓", initial_sidebar_state="expanded")
+
+# 获取 log_path 参数
+parser = argparse.ArgumentParser(description="RD-Agent Streamlit App")
+parser.add_argument("--log_dir", type=str, help="Path to the log directory")
+args = parser.parse_args()
+
+
+def get_folders_sorted(log_path):
+    """缓存并返回排序后的文件夹列表，并加入进度打印"""
+    with st.spinner("正在加载文件夹列表..."):
+        folders = sorted(
+            (folder for folder in log_path.iterdir() if folder.is_dir() and list(folder.iterdir())),
+            key=lambda folder: folder.stat().st_mtime,
+            reverse=True,
+        )
+        st.write(f"找到 {len(folders)} 个文件夹")
+    return [folder.name for folder in folders]
+
+
+if UI_SETTING.enable_cache:
+    get_folders_sorted = st.cache_data(get_folders_sorted)
+
+
+# 设置主日志路径
+main_log_path = Path(args.log_dir) if args.log_dir else Path("./log")
+if not main_log_path.exists():
+    st.error(f"Log dir {main_log_path} does not exist!")
+    st.stop()
+
+if "data" not in session_state:
+    session_state.data = []
+if "log_path" not in session_state:
+    session_state.log_path = None
+
+tlist = []
+
+
+def load_data():
+    """加载数据到 session_state 并显示进度"""
+    log_file = main_log_path / session_state.log_path / "debug_llm.pkl"
+    try:
+        with st.spinner(f"正在加载数据文件 {log_file}..."):
+            start_time = time.time()
+            with open(log_file, "rb") as f:
+                session_state.data = pickle.load(f, encoding="utf-8")  # nosec
+            st.success(f"数据加载完成！耗时 {time.time() - start_time:.2f} 秒")
+            st.session_state["current_loop"] = 1
+    except Exception as e:
+        session_state.data = [{"error": str(e)}]
+        st.error(f"加载数据失败: {e}")
+
+
+# UI - Sidebar
+with st.sidebar:
+    st.markdown(":blue[**Log Path**]")
+    manually = st.toggle("Manual Input")
+    if manually:
+        st.text_input("log path", key="log_path", label_visibility="collapsed")
+    else:
+        folders = get_folders_sorted(main_log_path)
+        st.selectbox(f"**Select from {main_log_path.absolute()}**", folders, key="log_path")  # nosec B608 — not SQL, Bandit false positive on "Select" in UI label
+
+    if st.button("Refresh Data"):
+        load_data()
+        st.rerun()
+
+
+# Helper functions
+def show_text(text, lang=None):
+    """显示文本代码块"""
+    if lang:
+        st.code(text, language=lang, wrap_lines=True)
+    elif "\n" in text:
+        st.code(text, language="python", wrap_lines=True)
+    else:
+        st.code(text, language="html", wrap_lines=True)
+
+
+def highlight_prompts_uri(uri):
+    """高亮 URI 的格式"""
+    parts = uri.split(":")
+    return f"**{parts[0]}:**:green[**{parts[1]}**]"
+
+
+# Display Data
+progress_text = st.empty()
+progress_bar = st.progress(0)
+
+# 每页展示一个 Loop
+LOOPS_PER_PAGE = 1
+
+# 获取所有的 Loop ID
+loop_groups = {}
+for i, d in enumerate(session_state.data):
+    tag = d["tag"]
+    loop_id, _ = extract_loopid_func_name(tag)
+    if loop_id:
+        if loop_id not in loop_groups:
+            loop_groups[loop_id] = []
+        loop_groups[loop_id].append(d)
+
+# 按 Loop ID 排序
+sorted_loop_ids = sorted(loop_groups.keys(), key=int)  # 假设 Loop ID 是数字
+total_loops = len(sorted_loop_ids)
+total_pages = total_loops  # 每页展示一个 Loop
+
+
+# simple display
+# FIXME: Delete this simple UI if trace have tag(evo_id & loop_id)
+# with st.sidebar:
+#     start = int(st.text_input("start", 0))
+#     end = int(st.text_input("end", 100))
+# for m in session_state.data[start:end]:
+#     if "tpl" in m["tag"]:
+#         obj = m["obj"]
+#         uri = obj["uri"]
+#         tpl = obj["template"]
+#         cxt = obj["context"]
+#         rd = obj["rendered"]
+#         with st.expander(highlight_prompts_uri(uri), expanded=False, icon="⚙️"):
+#             t1, t2, t3 = st.tabs([":green[**Rendered**]", ":blue[**Template**]", ":orange[**Context**]"])
+#             with t1:
+#                 show_text(rd)
+#             with t2:
+#                 show_text(tpl, lang="django")
+#             with t3:
+#                 st.json(cxt)
+#     if "llm" in m["tag"]:
+#         obj = m["obj"]
+#         system = obj.get("system", None)
+#         user = obj["user"]
+#         resp = obj["resp"]
+#         with st.expander(f"**LLM**", expanded=False, icon="🤖"):
+#             t1, t2, t3 = st.tabs([":green[**Response**]", ":blue[**User**]", ":orange[**System**]"])
+#             with t1:
+#                 try:
+#                     rdict = json.loads(resp)
+#                     if "code" in rdict:
+#                         code = rdict["code"]
+#                         st.markdown(":red[**Code in response dict:**]")
+#                         st.code(code, language="python", wrap_lines=True, line_numbers=True)
+#                         rdict.pop("code")
+#                     elif "spec" in rdict:
+#                         spec = rdict["spec"]
+#                         st.markdown(":red[**Spec in response dict:**]")
+#                         st.markdown(spec)
+#                         rdict.pop("spec")
+#                     else:
+#                         # show model codes
+#                         showed_keys = []
+#                         for k, v in rdict.items():
+#                             if k.startswith("model_") and k.endswith(".py"):
+#                                 st.markdown(f":red[**{k}**]")
+#                                 st.code(v, language="python", wrap_lines=True, line_numbers=True)
+#                                 showed_keys.append(k)
+#                         for k in showed_keys:
+#                             rdict.pop(k)
+#                     st.write(":red[**Other parts (except for the code or spec) in response dict:**]")
+#                     st.json(rdict)
+#                 except:
+#                     st.json(resp)
+#             with t2:
+#                 show_text(user)
+#             with t3:
+#                 show_text(system or "No system prompt available")
+
+
+if total_pages:
+    # 初始化 current_loop
+    if "current_loop" not in st.session_state:
+        st.session_state["current_loop"] = 1
+
+    # Loop 导航按钮
+    col1, col2, col3, col4, col5 = st.sidebar.columns([1.2, 1, 2, 1, 1.2])
+
+    with col1:
+        if st.button("|<"):  # 首页
+            st.session_state["current_loop"] = 1
+    with col2:
+        if st.button("<") and st.session_state["current_loop"] > 1:  # 上一页
+            st.session_state["current_loop"] -= 1
+    with col3:
+        # 下拉列表显示所有 Loop
+        st.session_state["current_loop"] = st.selectbox(
+            "选择 Loop",
+            options=list(range(1, total_loops + 1)),
+            index=st.session_state["current_loop"] - 1,  # 默认选中当前 Loop
+            label_visibility="collapsed",  # 隐藏标签
+        )
+    with col4:
+        if st.button("\>") and st.session_state["current_loop"] < total_loops:  # 下一页
+            st.session_state["current_loop"] += 1
+    with col5:
+        if st.button("\>|"):  # 最后一页
+            st.session_state["current_loop"] = total_loops
+
+    # 获取当前 Loop
+    current_loop = st.session_state["current_loop"]
+
+    # 渲染当前 Loop 数据
+    loop_id = sorted_loop_ids[current_loop - 1]
+    progress_text = st.empty()
+    progress_text.text(f"正在处理 Loop {loop_id}...")
+    progress_bar.progress(current_loop / total_loops, text=f"Loop :green[**{current_loop}**] / {total_loops}")
+
+    # 渲染 Loop Header
+    loop_anchor = f"Loop_{loop_id}"
+    if loop_anchor not in tlist:
+        tlist.append(loop_anchor)
+        st.header(loop_anchor, anchor=loop_anchor, divider="blue")
+
+    # 渲染当前 Loop 的所有数据
+    loop_data = loop_groups[loop_id]
+    for d in loop_data:
+        tag = d["tag"]
+        obj = d["obj"]
+        _, func_name = extract_loopid_func_name(tag)
+        evo_id = extract_evoid(tag)
+
+        func_anchor = f"loop_{loop_id}.{func_name}"
+        if func_anchor not in tlist:
+            tlist.append(func_anchor)
+            st.header(f"in *{func_name}*", anchor=func_anchor, divider="green")
+
+        evo_anchor = f"loop_{loop_id}.evo_step_{evo_id}"
+        if evo_id and evo_anchor not in tlist:
+            tlist.append(evo_anchor)
+            st.subheader(f"evo_step_{evo_id}", anchor=evo_anchor, divider="orange")
+
+        # 根据 tag 渲染内容
+        if "debug_exp_gen" in tag:
+            with st.expander(
+                f"Exp in :violet[**{obj.experiment_workspace.workspace_path}**]", expanded=False, icon="🧩"
+            ):
+                st.write(obj)
+        elif "debug_tpl" in tag:
+            uri = obj["uri"]
+            tpl = obj["template"]
+            cxt = obj["context"]
+            rd = obj["rendered"]
+            with st.expander(highlight_prompts_uri(uri), expanded=False, icon="⚙️"):
+                t1, t2, t3 = st.tabs([":green[**Rendered**]", ":blue[**Template**]", ":orange[**Context**]"])
+                with t1:
+                    show_text(rd)
+                with t2:
+                    show_text(tpl, lang="django")
+                with t3:
+                    st.json(cxt)
+        elif "debug_llm" in tag:
+            system = obj.get("system", None)
+            user = obj["user"]
+            resp = obj["resp"]
+            with st.expander(f"**LLM**", expanded=False, icon="🤖"):
+                t1, t2, t3 = st.tabs([":green[**Response**]", ":blue[**User**]", ":orange[**System**]"])
+                with t1:
+                    try:
+                        rdict = json.loads(resp)
+                        if "code" in rdict:
+                            code = rdict["code"]
+                            st.markdown(":red[**Code in response dict:**]")
+                            st.code(code, language="python", wrap_lines=True, line_numbers=True)
+                            rdict.pop("code")
+                        elif "spec" in rdict:
+                            spec = rdict["spec"]
+                            st.markdown(":red[**Spec in response dict:**]")
+                            st.markdown(spec)
+                            rdict.pop("spec")
+                        else:
+                            # show model codes
+                            showed_keys = []
+                            for k, v in rdict.items():
+                                if k.startswith("model_") and k.endswith(".py"):
+                                    st.markdown(f":red[**{k}**]")
+                                    st.code(v, language="python", wrap_lines=True, line_numbers=True)
+                                    showed_keys.append(k)
+                            for k in showed_keys:
+                                rdict.pop(k)
+                        st.write(":red[**Other parts (except for the code or spec) in response dict:**]")
+                        st.json(rdict)
+                    except:
+                        st.json(resp)
+                with t2:
+                    show_text(user)
+                with t3:
+                    show_text(system or "No system prompt available")
+
+    progress_text.text("当前 Loop 数据处理完成！")
+
+    # Sidebar TOC
+    with st.sidebar:
+        toc = "\n".join([f"- [{t}](#{t})" if t.startswith("L") else f"  - [{t.split('.')[1]}](#{t})" for t in tlist])
+        st.markdown(toc, unsafe_allow_html=True)

rdagent/scenarios/data_science/proposal/exp_gen/merge.py

@@ -182,7 +182,7 @@ class ExpGen2Hypothesis(DSProposalV2ExpGen):
 
         success_fb_list = list(set(trace_fbs))
         logger.info(
-            f"Merge Hypothesis: select {len(success_fb_list)} from {len(trace_fbs)} SOTA experiments found in {len(leaves)} traces"
+            f"Merge Hypothesis: select {len(success_fb_list)} from {len(trace_fbs)} SOTA experiments found in {len(leaves)} traces"  # nosec B608 — not SQL, Bandit false positive on "select" in log message
         )
 
         if len(success_fb_list) > 0:

rdagent/scenarios/data_science/proposal/exp_gen/select/submit.py

@@ -1,3 +1,4 @@
+import ast
 import json
 import os
 import pickle
@@ -292,7 +293,7 @@ class ValidationSelector(SOTAexpSelector):
         Sorts all valid experiments by score and returns the top N.
         """
 
-        mock_folder = f"/tmp/mock/{self.competition}"
+        mock_folder = f"/tmp/mock/{self.competition}"  # nosec B108 — Docker volume mount point derived from internal competition name
 
         try:
             data_py_code, grade_py_code = self._prepare_validation_scripts(
@@ -539,7 +540,7 @@ def process_experiment(
 
         # Run main script
         env = get_ds_env(
-            extra_volumes={f"/tmp/mock/{competition}/{input_folder}": input_folder},
+            extra_volumes={f"/tmp/mock/{competition}/{input_folder}": input_folder},  # nosec B108 — Docker volume mount point derived from internal competition name
             running_timeout_period=DS_RD_SETTING.full_timeout,
         )
         result = ws.run(env=env, entry="python main.py")
@@ -587,8 +588,8 @@ def _parsing_score(grade_stdout: str) -> Optional[float]:
         except:
             pass
         try:
-            # Priority 2: Eval dict
-            return float(eval(json_str)["score"])
+            # Priority 2: safe literal eval for Python-style dicts
+            return float(ast.literal_eval(json_str)["score"])
         except:
             pass
         try:

rdagent/scenarios/kaggle/developer/coder.py

@@ -38,7 +38,7 @@ class KGModelFeatureSelectionCoder(Developer[KGModelExperiment]):
         assert target_model_type in KG_SELECT_MAPPING
         if len(exp.experiment_workspace.data_description) == 1:
             code = (
-                Environment(undefined=StrictUndefined)
+                Environment(undefined=StrictUndefined)  # nosec B701 — renders Python code templates, not HTML; autoescape would corrupt code
                 .from_string(DEFAULT_SELECTION_CODE)
                 .render(feature_index_list=None)
             )
@@ -62,7 +62,7 @@ class KGModelFeatureSelectionCoder(Developer[KGModelExperiment]):
             chosen_index_to_list_index = [i - 1 for i in chosen_index]
 
             code = (
-                Environment(undefined=StrictUndefined)
+                Environment(undefined=StrictUndefined)  # nosec B701 — renders Python code templates, not HTML; autoescape would corrupt code
                 .from_string(DEFAULT_SELECTION_CODE)
                 .render(feature_index_list=chosen_index_to_list_index)
             )

rdagent/scenarios/kaggle/experiment/spaceship-titanic_template/fea_share_preprocess.py

@@ -79,12 +79,12 @@ def preprocess_script():
     This method applies the preprocessing steps to the training, validation, and test datasets.
     """
     if os.path.exists("/kaggle/input/X_train.pkl"):
-        X_train = pd.read_pickle("/kaggle/input/X_train.pkl")
-        X_valid = pd.read_pickle("/kaggle/input/X_valid.pkl")
-        y_train = pd.read_pickle("/kaggle/input/y_train.pkl")
-        y_valid = pd.read_pickle("/kaggle/input/y_valid.pkl")
-        X_test = pd.read_pickle("/kaggle/input/X_test.pkl")
-        others = pd.read_pickle("/kaggle/input/others.pkl")
+        X_train = pd.read_pickle("/kaggle/input/X_train.pkl")  # nosec B301 — trusted Kaggle input
+        X_valid = pd.read_pickle("/kaggle/input/X_valid.pkl")  # nosec B301
+        y_train = pd.read_pickle("/kaggle/input/y_train.pkl")  # nosec B301
+        y_valid = pd.read_pickle("/kaggle/input/y_valid.pkl")  # nosec B301
+        X_test = pd.read_pickle("/kaggle/input/X_test.pkl")  # nosec B301
+        others = pd.read_pickle("/kaggle/input/others.pkl")  # nosec B301
         y_train = pd.Series(y_train).reset_index(drop=True)
         y_valid = pd.Series(y_valid).reset_index(drop=True)
 

rdagent/scenarios/qlib/developer/factor_runner.py

@@ -1,4 +1,6 @@
+import sys
 import os
+import logging
 from pathlib import Path
 """
 Qlib Factor Runner - Executes factor backtests in Docker.
@@ -965,7 +967,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
                 shutil.copy(str(full_data), str(tmp / "intraday_pv.h5"))
 
                 ret = subprocess.run(
-                    ["python", "factor.py"],
+                    ["sys.executable", "factor.py"],
                     cwd=str(tmp),
                     capture_output=True,
                     timeout=300,
@@ -1000,7 +1002,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
             series.to_frame().to_parquet(str(parquet_path))
 
         except Exception:
-            pass
+            logging.debug("Error in save_factor_values_to_parquet", exc_info=True)
 
     def _log_result_warnings(self, factor_name: str, result, metrics: dict) -> None:
         """

rdagent/scenarios/qlib/experiment/utils.py

@@ -67,7 +67,7 @@ def get_file_desc(p: Path, variable_list=[]) -> str:
     """
     p = Path(p)
 
-    JJ_TPL = Environment(undefined=StrictUndefined).from_string("""
+    JJ_TPL = Environment(undefined=StrictUndefined).from_string("""  # nosec B701 — renders plain text description, not HTML; autoescape not applicable
 # {{file_name}}
 
 ## File Type

rdagent/scenarios/qlib/proposal/factor_proposal.py

@@ -1,3 +1,4 @@
+import logging
 import json
 import os
 from typing import List, Tuple
@@ -38,7 +39,7 @@ def _build_compressed_history(trace: Trace, max_history: int) -> str:
                 if exp.result is not None and "IC" in exp.result.index:
                     ic_str = f" IC={exp.result.loc['IC']:.4f}"
             except Exception:
-                pass
+                logging.debug("Exception caught", exc_info=True)
             decision = "PASS" if fb.decision else "FAIL"
             obs = (fb.observations or "")[:120].replace("\n", " ")
             lines.append(f"- [{decision}]{ic_str} {', '.join(names) or 'unknown'}: {obs}")

rdagent/scenarios/qlib/proposal/quant_proposal.py

@@ -1,3 +1,4 @@
+import logging
 import json
 import os
 import random
@@ -175,7 +176,7 @@ class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
                                 ic_val = exp.result.loc["IC"] if "IC" in exp.result.index else ""
                                 ic_str = f" IC={ic_val:.4f}" if ic_val != "" else ""
                         except Exception:
-                            pass
+                            logging.debug("Error getting IC", exc_info=True)
                         decision_str = "PASS" if fb.decision else "FAIL"
                         obs_short = (fb.observations or "")[:120].replace("\n", " ")
                         summary_lines.append(f"- [{decision_str}]{ic_str} {names_str}: {obs_short}")

rdagent/scenarios/rl/autorl_bench/agents/example_agent/train.py

@@ -71,7 +71,7 @@ def submit_for_grading(grading_url: str, model_path: str) -> dict | None:
 def main():
     MODEL_PATH = os.environ.get("MODEL_PATH")
     DATA_PATH = os.environ.get("DATA_PATH")
-    OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/tmp/autorl_output")
+    OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/tmp/autorl_output")  # nosec B108 — Docker container output dir, configurable via env var
     GRADING_SERVER_URL = os.environ.get("GRADING_SERVER_URL", "")
     TRAIN_RATIO = float(os.environ.get("TRAIN_RATIO", "0.05"))
     NUM_EPOCHS = int(os.environ.get("NUM_EPOCHS", "3"))

rdagent/scenarios/rl/autorl_bench/requirements.txt

@@ -9,7 +9,7 @@ peft>=0.18.1
 
 # Evaluation
 opencompass==0.5.1
-setuptools<75  # uv venv doesn't include, opencompass depends on pkg_resources
+setuptools>=78.1.1  # Security fix: GHSA-8g6x-3r52-4m6c (path traversal in PackageIndex.download, arbitrary file write/RCE)
 
 # Inference acceleration (optional, TRL supports 0.10.2-0.12.0)
 # Security: Version >=0.14.0 fixes CVE-2026-22807 (RCE via auto_map dynamic module loading)

rdagent/utils/env.py

@@ -614,7 +614,7 @@ class LocalEnv(Env[ASpecificLocalConf]):
         if self.conf.extra_volumes is not None:
             for lp, rp in self.conf.extra_volumes.items():
                 volumes[lp] = rp["bind"] if isinstance(rp, dict) else rp
-            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"
+            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"  # nosec B108 — fixed Docker volume mount point, not a user-writable temp file
             Path(cache_path).mkdir(parents=True, exist_ok=True)
             volumes[cache_path] = T("scenarios.data_science.share:scen.cache_path").r()
         for lp, rp in running_extra_volume.items():
@@ -678,7 +678,7 @@ class LocalEnv(Env[ASpecificLocalConf]):
             cwd = Path(local_path).resolve() if local_path else None
             env = {k: str(v) if isinstance(v, int) else v for k, v in env.items()}
 
-            process = subprocess.Popen(
+            process = subprocess.Popen(  # nosec B602 — entry is an internal command string set by LocalEnvConf, not user input
                 entry,
                 cwd=cwd,
                 env={**os.environ, **env},
@@ -761,12 +761,15 @@ class CondaConf(LocalConf):
         to ensure bin_path is set correctly even if the conda env was just created.
         """
         conda_path_result = subprocess.run(
-            f"conda run -n {self.conda_env_name} --no-capture-output env | grep '^PATH='",
+            ["conda", "run", "-n", self.conda_env_name, "--no-capture-output", "env"],
             capture_output=True,
             text=True,
-            shell=True,
         )
-        self.bin_path = conda_path_result.stdout.strip().split("=")[1] if conda_path_result.returncode == 0 else ""
+        if conda_path_result.returncode == 0:
+            path_lines = [l for l in conda_path_result.stdout.splitlines() if l.startswith("PATH=")]
+            self.bin_path = path_lines[0].split("=", 1)[1] if path_lines else ""
+        else:
+            self.bin_path = ""
 
 
 class MLECondaConf(CondaConf):
@@ -850,24 +853,22 @@ class QlibCondaEnv(LocalEnv[QlibCondaConf]):
     def prepare(self) -> None:
         """Prepare the conda environment if not already created."""
         try:
-            envs = subprocess.run("conda env list", capture_output=True, text=True, shell=True)
+            envs = subprocess.run(["conda", "env", "list"], capture_output=True, text=True)
             if self.conf.conda_env_name not in envs.stdout:
                 print(f"[yellow]Conda env '{self.conf.conda_env_name}' not found, creating...[/yellow]")
                 subprocess.check_call(
-                    f"conda create -y -n {self.conf.conda_env_name} python=3.10",
-                    shell=True,
+                    ["conda", "create", "-y", "-n", self.conf.conda_env_name, "python=3.10"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install --upgrade pip cython",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install", "--upgrade", "pip", "cython"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install git+https://github.com/microsoft/qlib.git@2fb9380b342556ddb50a4b24e4fe8655d548b2b8",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                     "git+https://github.com/microsoft/qlib.git@2fb9380b342556ddb50a4b24e4fe8655d548b2b8"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install catboost xgboost tables torch",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                     "catboost", "xgboost", "tables", "torch"],
                 )
 
         except Exception as e:
@@ -888,10 +889,9 @@ def _sync_conda_cache_with_real_envs() -> None:
     """Ensure the prepared cache includes environments that already exist on disk."""
     try:
         result = subprocess.run(
-            "conda env list",
+            ["conda", "env", "list"],
             capture_output=True,
             text=True,
-            shell=True,
             check=False,
         )
     except Exception as exc:  # pragma: no cover - best-effort helper
@@ -924,14 +924,15 @@ def _prepare_conda_env(env_name: str, requirements_file: Path, python_version: s
         python_version: Python version for the environment
     """
     # 1. Create conda environment if not exists
-    result = subprocess.run(f"conda env list | grep -q '^{env_name} '", shell=True)
-    if result.returncode != 0:
+    env_list = subprocess.run(["conda", "env", "list"], capture_output=True, text=True, check=False)
+    env_exists = any(line.split()[0] == env_name for line in env_list.stdout.splitlines() if line and not line.startswith("#"))
+    if not env_exists:
         print(f"[yellow]Creating conda env '{env_name}' (Python {python_version})...[/yellow]")
-        subprocess.check_call(f"conda create -y -n {env_name} python={python_version}", shell=True)
-        subprocess.check_call(f"conda run -n {env_name} pip install --upgrade pip", shell=True)
+        subprocess.check_call(["conda", "create", "-y", "-n", env_name, f"python={python_version}"])
+        subprocess.check_call(["conda", "run", "-n", env_name, "pip", "install", "--upgrade", "pip"])
 
     print(f"[yellow]Installing dependencies from {requirements_file.name}...[/yellow]")
-    subprocess.check_call(f"conda run -n {env_name} pip install -r {requirements_file}", shell=True)
+    subprocess.check_call(["conda", "run", "-n", env_name, "pip", "install", "-r", str(requirements_file)])
     print(f"[green]Conda env '{env_name}' ready[/green]")
 
     _CONDA_ENV_PREPARED.add(env_name)
@@ -971,8 +972,8 @@ class FTCondaEnv(LocalEnv[FTCondaConf]):
             # Note: flash-attn>=2.8 is required for B200 (sm_100) support
             print("[yellow]Installing flash-attn (compiling, may take a few minutes)...[/yellow]")
             subprocess.check_call(
-                f"conda run -n {self.conf.conda_env_name} pip install 'flash-attn>=2.8' --no-build-isolation --no-cache-dir",
-                shell=True,
+                ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                 "flash-attn>=2.8", "--no-build-isolation", "--no-cache-dir"],
             )
 
             # Re-update bin_path after prepare() in case the conda env was just created
@@ -1442,7 +1443,7 @@ class DockerEnv(Env[DockerConf]):
         if self.conf.extra_volumes is not None:
             for lp, rp in self.conf.extra_volumes.items():
                 volumes[lp] = rp if isinstance(rp, dict) else {"bind": rp, "mode": self.conf.extra_volume_mode}
-            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"
+            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"  # nosec B108 — fixed Docker volume mount point, not a user-writable temp file
             Path(cache_path).mkdir(parents=True, exist_ok=True)
             volumes[cache_path] = {
                 "bind": T("scenarios.data_science.share:scen.cache_path").r(),

requirements.txt

@@ -9,8 +9,8 @@ psutil
 fire
 fuzzywuzzy
 openai
-litellm>=1.73  # to support `from litellm import get_valid_models`
-aiohttp>=3.13.4  # CVE-2026-22815, CVE-2026-34515, CVE-2026-34516, CVE-2026-34525
+litellm>=1.83.14  # to support `from litellm import get_valid_models`
+aiohttp>=3.13.4  # CVE-2026-22815, CVE-2026-34515, CVE-2026-34516, CVE-2026-34525; >=3.13.4 due to litellm==1.83.14 exact pin
 azure.identity
 pyarrow
 rich
@@ -98,8 +98,8 @@ optuna>=3.5.0
 beautifulsoup4>=4.12.0
 
 # ML Training Pipeline
-lightgbm>=3.3.0
-scipy>=1.9.0
+lightgbm>=3.3.5
+scipy>=1.15.3
 
 # RL Trading (optional - system works without these)
 # Install for full RL training: pip install stable-baselines3[extra] gymnasium

requirements/rl.txt

@@ -8,7 +8,7 @@
 # Only install if you want to use full PPO/A2C/SAC training.
 
 # Core RL library
-stable-baselines3[extra]>=2.0.0
+stable-baselines3[extra]>=2.8.0
 
 # Gymnasium environment (OpenAI Gym successor)
 gymnasium>=0.29.0