chore(master): release 1.3.8 (#42)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

.github/workflows/release.yml

@@ -12,7 +12,7 @@ jobs:
   release-please:
     runs-on: ubuntu-latest
     steps:
-      - uses: googleapis/release-please-action@v4
+      - uses: googleapis/release-please-action@v5
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           config-file: release-please-config.json

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.3.3"
+  ".": "1.3.8"
 }

CHANGELOG.md

@@ -1,5 +1,77 @@
 # Changelog
 
+## [1.3.8](https://github.com/TPTBusiness/Predix/compare/v1.3.7...v1.3.8) (2026-04-30)
+
+
+### Bug Fixes
+
+* **deps:** relax aiohttp constraint to >=3.13.4 for litellm compatibility ([34ab192](https://github.com/TPTBusiness/Predix/commit/34ab1923a887089eb36e5cbad6cb8df16f0333ca))
+* **qlib:** correct indentation in except blocks in quant_proposal and factor_runner ([8143451](https://github.com/TPTBusiness/Predix/commit/8143451e8c0ead01c4d86d19669268c7bfb15fac))
+* **security:** replace eval() with ast.literal_eval in finetune validator (B307) ([0508caf](https://github.com/TPTBusiness/Predix/commit/0508caf9140d210b823fefefa28ee535ec85a0ae))
+* **security:** replace shell=True subprocess calls with list args in env.py (B602) ([2012d5a](https://github.com/TPTBusiness/Predix/commit/2012d5ae4e77cc2f1ab9a48beaaac5a74695d083))
+* **security:** resolve path-injection and add nosec for safe temp paths (B108, py/path-injection) ([6727480](https://github.com/TPTBusiness/Predix/commit/67274803bd1d14e5d1df9a063f46b2edb8501a2b))
+
+## [1.3.7](https://github.com/TPTBusiness/Predix/compare/v1.3.6...v1.3.7) (2026-04-30)
+
+
+### Bug Fixes
+
+* **security:** nosec for B608/B701 false positives in UI and template code ([5eb5d7e](https://github.com/TPTBusiness/Predix/commit/5eb5d7e8fdbe90e0dced83fef4e09f5a33e96b2b))
+* **security:** replace eval() with ast.literal_eval and add request timeouts (B307, B113) ([3301ada](https://github.com/TPTBusiness/Predix/commit/3301ada697ca7d3afa1a188d2a76a87ae98b4529))
+* **security:** replace shell=True subprocess calls with list args (B602) ([13c08f4](https://github.com/TPTBusiness/Predix/commit/13c08f4ce6813eb7c314087921ec8c0f40074bd7))
+
+## [1.3.6](https://github.com/TPTBusiness/Predix/compare/v1.3.5...v1.3.6) (2026-04-30)
+
+
+### Bug Fixes
+
+* **security:** real fix for B110 (logging in factor_proposal.py [#746](https://github.com/TPTBusiness/Predix/issues/746)) ([16624e0](https://github.com/TPTBusiness/Predix/commit/16624e0bd966ae4d24c4a3eb42bbc31c11da3136))
+* **security:** real fix for B110 (logging in factor_runner.py [#744](https://github.com/TPTBusiness/Predix/issues/744)) ([88cf0fb](https://github.com/TPTBusiness/Predix/commit/88cf0fb8828b11c97f2f3ae2881a4900b020c6f0))
+* **security:** real fix for B110 (logging in quant_proposal.py [#741](https://github.com/TPTBusiness/Predix/issues/741)) ([7cf2a64](https://github.com/TPTBusiness/Predix/commit/7cf2a644f553b054bd4b0607ea51e5372e68d90a))
+* **security:** real fix for B110 (logging in quant_proposal.py [#741](https://github.com/TPTBusiness/Predix/issues/741)) ([ef985f8](https://github.com/TPTBusiness/Predix/commit/ef985f86035d8dca707c60137e6508349a0c4ae6))
+* **security:** real fix for B404/B603 (sys.executable in factor_runner.py [#745](https://github.com/TPTBusiness/Predix/issues/745)) ([819655a](https://github.com/TPTBusiness/Predix/commit/819655aaa3efa76596d60501d0e8ca365df3e5e2))
+* **security:** revert broken read_pickle encoding arg in kaggle template (B301) ([3574907](https://github.com/TPTBusiness/Predix/commit/35749073c91e69f63ddaad61dae3f2b799327e63))
+* **security:** validate SQL identifiers in _add_column_if_not_exists (B608) ([e10dfa2](https://github.com/TPTBusiness/Predix/commit/e10dfa2576038e911f83595d3b466c261bc0cd54))
+* **security:** whitelist-validate metric column in get_top_factors (B608) ([e50519f](https://github.com/TPTBusiness/Predix/commit/e50519fe066e68aec2f19b83df4f643c3c22053d))
+
+## [1.3.5](https://github.com/TPTBusiness/Predix/compare/v1.3.4...v1.3.5) (2026-04-27)
+
+
+### Bug Fixes
+
+* **auto-fixer:** add five new factor code fixes for groupby/apply errors ([449c8fd](https://github.com/TPTBusiness/Predix/commit/449c8fd70a327e604dcca122e4a134f0cca918e4))
+* **auto-fixer:** add four new factor code fixes for common runtime errors ([40484f6](https://github.com/TPTBusiness/Predix/commit/40484f6d300425da481f1edd325da4acbc06ec7d))
+* **auto-fixer:** add groupby([level=N,'date']) SyntaxError fix ([ca77c00](https://github.com/TPTBusiness/Predix/commit/ca77c005bea4abdd8854c1de2b0e8d03b7742161))
+* **auto-fixer:** disable _fix_min_periods for intraday data ([77b0740](https://github.com/TPTBusiness/Predix/commit/77b0740f059349df7e769a378af728aa33b2070e))
+* **auto-fixer:** fix chained groupby(level=N).groupby('date') pattern ([7d5fe32](https://github.com/TPTBusiness/Predix/commit/7d5fe32b31a19ce8b04bd8f5a430720fdb748f7a))
+* **auto-fixer:** fix df.loc[instrument] DateParseError on MultiIndex frames ([b7860ea](https://github.com/TPTBusiness/Predix/commit/b7860eafc0ad26384947ce0510ecf4e9f3425807))
+* **auto-fixer:** fix df['instrument'] KeyError on MultiIndex frames ([aad6bd1](https://github.com/TPTBusiness/Predix/commit/aad6bd1c7c720b3d486e0cf248337f32394773b1))
+* **auto-fixer:** fix two assignment-target bugs in instrument column fixers ([421eedf](https://github.com/TPTBusiness/Predix/commit/421eedffed4b883c24397dc5581c019a3985277f))
+* **auto-fixer:** preserve date dimension in groupby(['instrument','date']) fix ([b58fdd8](https://github.com/TPTBusiness/Predix/commit/b58fdd8be43720b5d4363e0f8de9a01591d4d2dc))
+* **auto-fixer:** remove ddof from rolling() args, not only from std()/var() ([b0fc328](https://github.com/TPTBusiness/Predix/commit/b0fc328d0d4a041c65d8eeb32cb3f2bb86568406))
+* **auto-fixer:** strip spurious .reset_index() after .transform() calls ([8708aae](https://github.com/TPTBusiness/Predix/commit/8708aae6e08728cda1875c775a76dc92e43576f3))
+* **loop:** prevent step_idx advance on unhandled exceptions + fix consecutive assistant messages ([5ec4ad1](https://github.com/TPTBusiness/Predix/commit/5ec4ad1b96b5b99ef42bea7bb828cb1ef709a688))
+
+## [1.3.4](https://github.com/TPTBusiness/Predix/compare/v1.3.3...v1.3.4) (2026-04-27)
+
+
+### Bug Fixes
+
+* **auto-fixer:** add five new factor code fixes for groupby/apply errors ([449c8fd](https://github.com/TPTBusiness/Predix/commit/449c8fd70a327e604dcca122e4a134f0cca918e4))
+* **auto-fixer:** add four new factor code fixes for common runtime errors ([40484f6](https://github.com/TPTBusiness/Predix/commit/40484f6d300425da481f1edd325da4acbc06ec7d))
+* **auto-fixer:** add groupby([level=N,'date']) SyntaxError fix ([ca77c00](https://github.com/TPTBusiness/Predix/commit/ca77c005bea4abdd8854c1de2b0e8d03b7742161))
+* **auto-fixer:** disable _fix_min_periods for intraday data ([77b0740](https://github.com/TPTBusiness/Predix/commit/77b0740f059349df7e769a378af728aa33b2070e))
+* **auto-fixer:** fix chained groupby(level=N).groupby('date') pattern ([7d5fe32](https://github.com/TPTBusiness/Predix/commit/7d5fe32b31a19ce8b04bd8f5a430720fdb748f7a))
+* **auto-fixer:** fix df.loc[instrument] DateParseError on MultiIndex frames ([b7860ea](https://github.com/TPTBusiness/Predix/commit/b7860eafc0ad26384947ce0510ecf4e9f3425807))
+* **auto-fixer:** fix df['instrument'] KeyError on MultiIndex frames ([aad6bd1](https://github.com/TPTBusiness/Predix/commit/aad6bd1c7c720b3d486e0cf248337f32394773b1))
+* **auto-fixer:** preserve date dimension in groupby(['instrument','date']) fix ([b58fdd8](https://github.com/TPTBusiness/Predix/commit/b58fdd8be43720b5d4363e0f8de9a01591d4d2dc))
+* **auto-fixer:** remove ddof from rolling() args, not only from std()/var() ([b0fc328](https://github.com/TPTBusiness/Predix/commit/b0fc328d0d4a041c65d8eeb32cb3f2bb86568406))
+* **backtest:** replace broken MC permutation test with binomial win-rate test ([c38d894](https://github.com/TPTBusiness/Predix/commit/c38d89478f586825bfca5715a96ca70ccd8791a3))
+* **factors:** detect and correct look-ahead bias in daily-constant factors ([eb490a4](https://github.com/TPTBusiness/Predix/commit/eb490a461b66cbd815ae53ac5205115754712432))
+* **factors:** extend look-ahead rules to session factors and add intraday-factor guidance ([c24c100](https://github.com/TPTBusiness/Predix/commit/c24c100442d6487686c0578de0b32d240fcbf215))
+* **loop:** compress old experiment history in proposal prompt to reduce context size ([4bf90a9](https://github.com/TPTBusiness/Predix/commit/4bf90a905ba8b2aba2a818191c19998088cccaaf))
+* **loop:** prevent step_idx advance on unhandled exceptions + fix consecutive assistant messages ([5ec4ad1](https://github.com/TPTBusiness/Predix/commit/5ec4ad1b96b5b99ef42bea7bb828cb1ef709a688))
+
 ## [1.3.3](https://github.com/TPTBusiness/Predix/compare/v1.3.2...v1.3.3) (2026-04-25)
 
 

rdagent/app/cli.py

@@ -1251,7 +1251,7 @@ def start_loop_cli(
     script_dir = str(Path(__file__).parent.parent.parent.parent)
     generator = f"python {script_dir}/scripts/predix_smart_strategy_gen.py"
     logfile = f"{script_dir}/results/logs/generator_loop.log"
-    pidfile = "/tmp/predix_loop.pid"
+    pidfile = "/tmp/predix_loop.pid"  # nosec B108 — administrative PID file, single-process daemon
 
     os.makedirs(f"{script_dir}/results/logs", exist_ok=True)
 

rdagent/app/finetune/llm/ui/data_loader.py

@@ -93,7 +93,7 @@ def get_valid_sessions(log_folder: Path, safe_root: Path | None = None) -> list[
         folder_real = os.path.realpath(str(log_folder.expanduser()))
         if not (folder_real == root_real or folder_real.startswith(root_real + os.sep)):
             return []
-        log_folder = Path(folder_real)
+        log_folder = Path(folder_real)  # nosec B614 — path validated against safe_root via realpath above
 
     if not log_folder.exists():
         return []
@@ -379,7 +379,7 @@ def load_ft_session(log_path: Path, safe_root: Path | None = None) -> Session:
         path_real = os.path.realpath(str(log_path.expanduser()))
         if not (path_real == root_real or path_real.startswith(root_real + os.sep)):
             return Session()
-        log_path = Path(path_real)
+        log_path = Path(path_real)  # nosec B614 — path validated against safe_root via realpath above
 
     session = Session()
     storage = FileStorage(log_path)

rdagent/app/rl/ui/app.py

@@ -82,7 +82,7 @@ def get_job_options(base_path: Path, safe_root: Path | None = None) -> list[str]
 
     # Security fix: Validate base_path to prevent path traversal
     try:
-        base_path_resolved = base_path.expanduser().resolve()
+        base_path_resolved = base_path.expanduser().resolve()  # nosec B614 — validated against safe_root below via relative_to()
 
         if safe_root is not None:
             safe_root_resolved = safe_root.expanduser().resolve()

rdagent/app/rl/ui/data_loader.py

@@ -80,7 +80,7 @@ def get_valid_sessions(log_folder: Path, safe_root: Path | None = None) -> list[
         folder_real = os.path.realpath(str(log_folder.expanduser()))
         if not (folder_real == root_real or folder_real.startswith(root_real + os.sep)):
             return []
-        log_folder = Path(folder_real)
+        log_folder = Path(folder_real)  # nosec B614 — path validated against safe_root via realpath above
 
     if not log_folder.exists():
         return []
@@ -251,7 +251,7 @@ def load_session(log_path: Path, safe_root: Path | None = None) -> Session:
         path_real = os.path.realpath(str(log_path.expanduser()))
         if not (path_real == root_real or path_real.startswith(root_real + os.sep)):
             return Session()
-        log_path = Path(path_real)
+        log_path = Path(path_real)  # nosec B614 — path validated against safe_root via realpath above
 
     session = Session()
 

rdagent/app/utils/info.py

@@ -54,11 +54,11 @@ def rdagent_info():
     current_version = importlib.metadata.version("rdagent")
     logger.info(f"RD-Agent version: {current_version}")
     api_url = f"https://api.github.com/repos/microsoft/RD-Agent/contents/requirements.txt?ref=main"
-    response = requests.get(api_url)
+    response = requests.get(api_url, timeout=30)
     if response.status_code == 200:
         files = response.json()
         file_url = files["download_url"]
-        file_response = requests.get(file_url)
+        file_response = requests.get(file_url, timeout=30)
         if file_response.status_code == 200:
             all_file_contents = file_response.text.split("\n")
         else:

rdagent/components/backtesting/results_db.py

@@ -71,6 +71,9 @@ class ResultsDatabase:
 
         self.conn.commit()
 
+    _ALLOWED_TABLES = frozenset({"factors", "backtest_runs", "loop_results"})
+    _ALLOWED_COL_TYPES = frozenset({"REAL", "TEXT", "INTEGER", "BLOB"})
+
     def _add_column_if_not_exists(self, table: str, column: str, col_type: str) -> None:
         """
         Add a column to a table if it doesn't already exist.
@@ -78,20 +81,24 @@ class ResultsDatabase:
         Parameters
         ----------
         table : str
-            Table name
+            Table name (must be in _ALLOWED_TABLES)
         column : str
-            Column name to add
+            Column name to add (alphanumeric + underscore only)
         col_type : str
-            SQL column type (e.g., 'REAL', 'TEXT')
+            SQL column type (must be in _ALLOWED_COL_TYPES)
         """
+        if table not in self._ALLOWED_TABLES:
+            raise ValueError(f"Unknown table: {table!r}")
+        if not column.replace("_", "").isalnum():
+            raise ValueError(f"Invalid column name: {column!r}")
+        if col_type not in self._ALLOWED_COL_TYPES:
+            raise ValueError(f"Invalid column type: {col_type!r}")
+
         c = self.conn.cursor()
-        try:
-            # Try to query the column - if it fails, it doesn't exist
-            # nosec B608: Internal schema migration, column names are controlled
-            c.execute(f"SELECT {column} FROM {table} LIMIT 1")  # nosec B608
-        except sqlite3.OperationalError:
-            # Column doesn't exist, add it
-            c.execute(f"ALTER TABLE {table} ADD COLUMN {column} {col_type}")  # nosec B608
+        c.execute("SELECT name FROM pragma_table_info(?)", (table,))
+        existing = {row[0] for row in c.fetchall()}
+        if column not in existing:
+            c.execute(f"ALTER TABLE {table} ADD COLUMN {column} {col_type}")
     
     def add_factor(self, name: str, type: str = "unknown") -> int:
         c = self.conn.cursor()
@@ -183,16 +190,18 @@ class ResultsDatabase:
         pd.DataFrame
             DataFrame with factor names and metrics
         """
-        # Map shorthand to full column name
+        _ALLOWED_METRICS = frozenset({
+            'sharpe', 'ic', 'annual_return', 'max_drawdown',
+            'win_rate', 'information_ratio', 'volatility',
+        })
         metric_map = {
-            'sharpe': 'sharpe',
-            'ic': 'ic',
-            'return': 'annual_return',
-            'drawdown': 'max_drawdown',
-            'win_rate': 'win_rate',
+            'sharpe': 'sharpe', 'ic': 'ic', 'return': 'annual_return',
+            'drawdown': 'max_drawdown', 'win_rate': 'win_rate',
             'information_ratio': 'information_ratio',
         }
         col = metric_map.get(metric, metric)
+        if col not in _ALLOWED_METRICS:
+            raise ValueError(f"Unknown metric: {metric!r}")
 
         return pd.read_sql_query(
             f"""SELECT factor_name, ic, sharpe, annual_return, max_drawdown,
@@ -201,7 +210,7 @@ class ResultsDatabase:
                 JOIN factors ON factor_id = factors.id
                 WHERE {col} IS NOT NULL
                 ORDER BY {col} DESC
-                LIMIT ?""",
+                LIMIT ?""",  # nosec B608 — col is validated against _ALLOWED_METRICS above
             self.conn,
             params=[limit]
         )

rdagent/components/coder/factor_coder/auto_fixer.py

@@ -51,13 +51,23 @@ class FactorAutoFixer:
         self.fixes_applied = []
         fixed_code = code
 
-        # Apply fixes in order - groupby fixes MUST come before min_periods fixes
+        # Apply fixes in order
+        # NOTE: _fix_min_periods is intentionally excluded — it increased min_periods to
+        # match window size, which causes all-NaN output for intraday data with 96 bars/day
+        # (window=240 > 96 means zero valid bars per day). The LLM sets its own min_periods.
         fix_methods = [
-            self._fix_groupby_apply_to_transform,  # First: fix groupby patterns
-            self._fix_min_periods,                  # Second: fix min_periods in resulting rolling calls
-            self._fix_inf_nan_handling,             # Third: add inf/nan handling
-            self._fix_data_range_processing,        # Fourth: ensure full data range
-            self._fix_multiindex_groupby,           # Fifth: ensure groupby on MultiIndex
+            self._fix_instrument_column_access,     # First: fix df['instrument'] on MultiIndex
+            self._fix_instrument_loc_multiindex,    # Second: fix df.loc[instrument_var] on MultiIndex
+            self._fix_zero_volume_proxy,            # Third: replace zero $volume with range proxy
+            self._fix_reset_index_groupby,          # Fourth: fix groupby(level=N) after reset_index()
+            self._fix_groupby_mixed_levels,         # Fifth: fix groupby(level=[int, str])
+            self._fix_groupby_column_on_multiindex, # Sixth: fix groupby(['instrument','date']) on MultiIndex
+            self._fix_chained_groupby,              # Seventh: fix groupby(level=N).groupby('date') chain
+            self._fix_rolling_ddof,                 # Eighth: remove unsupported ddof kwarg
+            self._fix_groupby_apply_to_transform,   # Ninth: fix groupby patterns
+            self._fix_inf_nan_handling,             # Tenth: add inf/nan handling
+            self._fix_data_range_processing,        # Eleventh: ensure full data range
+            self._fix_multiindex_groupby,           # Twelfth: ensure groupby on MultiIndex
         ]
 
         for fix_method in fix_methods:
@@ -75,6 +85,352 @@ class FactorAutoFixer:
 
         return fixed_code
 
+    def _fix_instrument_column_access(self, code: str) -> str:
+        """
+        Fix: df['instrument'] raises KeyError on a MultiIndex DataFrame because
+        'instrument' is an index level (level 1), not a column.
+
+        Replace df['instrument'] with df.index.get_level_values('instrument')
+        but only when the DataFrame has a MultiIndex (not after reset_index which
+        would have promoted it to a real column).
+
+        Also fixes df.reset_index()['instrument'] correctly since after reset_index
+        the column exists.
+        """
+        fixed_code = code
+
+        # Skip if already fixed or if reset_index() is being used before the access
+        # We only fix bare df['instrument'] where df is the original MultiIndex frame.
+        # Heuristic: if the assignment lhs or context shows reset_index, leave it alone.
+
+        # Pattern: <varname>['instrument'] where varname is NOT a reset_index result
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        def _replace_instrument_access(m: re.Match) -> str:
+            var = m.group(1)
+            if var in reset_vars:
+                return m.group(0)  # leave reset_index vars alone — column exists
+            self.fixes_applied.append(f"instrument_column: {var}['instrument'] → get_level_values(1)")
+            return f"{var}.index.get_level_values(1)"
+
+        # Exclude assignment targets: var['instrument'] = ... must not become
+        # var.index.get_level_values(1) = ... (SyntaxError: cannot assign to function call)
+        fixed_code = re.sub(r"(\w+)\['instrument'\](?!\s*=)", _replace_instrument_access, fixed_code)
+
+        return fixed_code
+
+    def _fix_instrument_loc_multiindex(self, code: str) -> str:
+        """
+        Fix: df.loc[instrument_var] raises DateParseError on a (datetime, instrument)
+        MultiIndex because pandas tries to match the instrument string against the
+        datetime level (level 0).
+
+        Pattern detected: for-loops iterating over get_level_values('instrument') or
+        get_level_values(1) where the loop variable is then used as df.loc[loop_var].
+
+        Replacement: df.loc[instrument_var] → df.xs(instrument_var, level=1)
+        """
+        fixed_code = code
+
+        # Find variables iterated from get_level_values('instrument') or get_level_values(1)
+        inst_vars = set(
+            re.findall(
+                r"for\s+(\w+)\s+in\s+.+?\.get_level_values\s*\(\s*(?:1|['\"]instrument['\"])\s*\)[^:\n]*:",
+                code,
+            )
+        )
+
+        if not inst_vars:
+            return fixed_code
+
+        for var in inst_vars:
+            # Replace DF.loc[var] (read) with DF.xs(var, level=1)
+            # Exclude write-back patterns (DF.loc[var] = ...) — leave those as-is
+            def _make_replacer(v: str):
+                def _replace(m: re.Match) -> str:
+                    df_var = m.group(1)
+                    self.fixes_applied.append(
+                        f"instrument_loc: {df_var}.loc[{v}] → {df_var}.xs({v}, level=1)"
+                    )
+                    return f"{df_var}.xs({v}, level=1)"
+
+                return _replace
+
+            # Only match when NOT followed by ' =' (assignment)
+            fixed_code = re.sub(
+                rf"(\w+)\.loc\[\s*{re.escape(var)}\s*\](?!\s*=)",
+                _make_replacer(var),
+                fixed_code,
+            )
+
+        return fixed_code
+
+    def _fix_zero_volume_proxy(self, code: str) -> str:
+        """
+        Fix: $volume is always 0 in our EUR/USD dataset (FX has no real volume).
+        Any factor using $volume (VWAP, volume-weighted returns, etc.) produces
+        all-NaN output because 0*price=0 and sum(0)/sum(0)=NaN.
+
+        Insert a guard right after pd.read_hdf() that replaces zero volume with
+        the intraday price-range proxy ($high - $low) so volume-weighted factors
+        produce meaningful signals.
+        """
+        if "'$volume'" not in code and '"$volume"' not in code:
+            return code
+
+        # Already patched
+        if "volume proxy" in code:
+            return code
+
+        lines = code.splitlines()
+        insert_after = -1
+        df_var = "df"
+        indent = "    "
+
+        for i, line in enumerate(lines):
+            if "read_hdf(" in line:
+                m = re.match(r"(\s*)(\w+)\s*=\s*", line)
+                if m:
+                    indent = m.group(1)
+                    df_var = m.group(2)
+                else:
+                    m2 = re.match(r"(\s*)", line)
+                    indent = m2.group(1) if m2 else "    "
+                insert_after = i
+                break
+
+        if insert_after == -1:
+            return code
+
+        proxy_lines = [
+            f"{indent}# volume proxy: $volume is always 0 in FX data — use price-range as proxy",
+            f"{indent}if ({df_var}['$volume'] == 0).all():",
+            f"{indent}    {df_var}['$volume'] = {df_var}['$high'] - {df_var}['$low']",
+        ]
+        lines = lines[: insert_after + 1] + proxy_lines + lines[insert_after + 1 :]
+        self.fixes_applied.append("volume_proxy: replaced zero $volume with ($high - $low)")
+        return "\n".join(lines)
+
+    def _fix_reset_index_groupby(self, code: str) -> str:
+        """
+        Fix: groupby(level=N) on a variable created by .reset_index() fails because
+        reset_index() converts the MultiIndex into regular columns, leaving a plain
+        RangeIndex.  Replace groupby(level=N) on such variables with
+        groupby('instrument').
+
+        Detected pattern:
+            varname = <anything>.reset_index(...)
+            ...
+            varname.groupby(level=0|1)
+        """
+        fixed_code = code
+
+        # Find all variables assigned via reset_index()
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        for var in reset_vars:
+            # Replace var.groupby(level=N) with var.groupby('instrument')
+            pattern = rf'{re.escape(var)}\.groupby\(level\s*=\s*\d+\)'
+            if re.search(pattern, fixed_code):
+                fixed_code = re.sub(pattern, f"{var}.groupby('instrument')", fixed_code)
+                self.fixes_applied.append(f"reset_index_groupby: {var}.groupby(level=N) → groupby('instrument')")
+
+        return fixed_code
+
+    def _fix_groupby_mixed_levels(self, code: str) -> str:
+        """
+        Fix: groupby(level=[int, 'str']) raises AssertionError because string level
+        names don't exist on an unnamed MultiIndex.  Keep only integer levels.
+
+        Pattern:  .groupby(level=[0, 'date'])  →  .groupby(level=0)
+                  .groupby(level=[1, 'date'])  →  .groupby(level=1)
+        """
+        fixed_code = code
+
+        def _keep_int_levels(m):
+            inner = m.group(1)
+            ints = re.findall(r'\b(\d+)\b', inner)
+            if not ints:
+                return m.group(0)
+            replacement = f'.groupby(level={ints[0]})' if len(ints) == 1 else f'.groupby(level=[{", ".join(ints)}])'
+            self.fixes_applied.append(f"mixed_levels: groupby(level=[...,str]) → {replacement}")
+            return replacement
+
+        fixed_code = re.sub(r'\.groupby\(level=\[([^\]]+)\]\)', _keep_int_levels, fixed_code)
+        return fixed_code
+
+    def _fix_groupby_column_on_multiindex(self, code: str) -> str:
+        """
+        Fix: groupby(['instrument', 'date']) on a MultiIndex (datetime, instrument)
+        DataFrame fails with KeyError because those are index levels, not columns.
+
+        Correct replacement preserves BOTH dimensions so intraday calculations reset
+        per day:
+            var.groupby(['instrument', 'date'])
+            → var.groupby([var.index.get_level_values(1), var.index.get_level_values(0).normalize()])
+
+        Single-column groupby(['instrument']) is correctly replaced with groupby(level=1).
+        Note: do NOT convert groupby('instrument') → groupby(level=1) here — that would
+        undo the reset_index_groupby fix which correctly emits groupby('instrument').
+        """
+        fixed_code = code
+
+        # Variables created via reset_index() have a plain RangeIndex — applying
+        # get_level_values() on them would raise AttributeError. Skip those.
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        def _replace_two_col_groupby(m: re.Match, order: str) -> str:
+            var = m.group(1)
+            if var in reset_vars:
+                return m.group(0)  # leave reset_index vars alone — RangeIndex, not MultiIndex
+            if order == "instrument_date":
+                repl = (
+                    f"{var}.groupby([{var}.index.get_level_values(1), "
+                    f"{var}.index.get_level_values(0).normalize()])"
+                )
+            else:  # date_instrument
+                repl = (
+                    f"{var}.groupby([{var}.index.get_level_values(0).normalize(), "
+                    f"{var}.index.get_level_values(1)])"
+                )
+            self.fixes_applied.append(f"multiindex_groupby: {m.group(0)[:60]} → two-level")
+            return repl
+
+        # groupby(['instrument', 'date']) — capture variable name before .groupby
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(\[\'instrument\',\s*\'date\'\]\)',
+            lambda m: _replace_two_col_groupby(m, "instrument_date"),
+            fixed_code,
+        )
+        # groupby(['date', 'instrument'])
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(\[\'date\',\s*\'instrument\'\]\)',
+            lambda m: _replace_two_col_groupby(m, "date_instrument"),
+            fixed_code,
+        )
+        # single: groupby(['instrument']) → groupby(level=1), but not on reset_index vars
+        def _replace_single_instrument_groupby(m: re.Match) -> str:
+            # Look backwards to find the variable name
+            prefix = fixed_code[: m.start()]
+            var_match = re.search(r'(\w+)\s*$', prefix)
+            var = var_match.group(1) if var_match else ''
+            if var in reset_vars:
+                return m.group(0)
+            self.fixes_applied.append("multiindex_groupby: groupby(['instrument']) → groupby(level=1)")
+            return ".groupby(level=1)"
+
+        if re.search(r"\.groupby\(\['instrument'\]\)", fixed_code):
+            fixed_code = re.sub(r"\.groupby\(\['instrument'\]\)", _replace_single_instrument_groupby, fixed_code)
+
+        # groupby(level=['instrument', 'date']) — uses level= keyword with string names.
+        # 'date' is NOT a valid level name in our (datetime, instrument) MultiIndex;
+        # replace with get_level_values to normalize datetime to daily timestamps.
+        fixed_code = re.sub(
+            r"(\w+)\.groupby\(level=\['instrument',\s*'date'\]\)",
+            lambda m: (
+                self.fixes_applied.append(
+                    f"multiindex_groupby: {m.group(0)[:60]} → two-level get_level_values"
+                )
+                or f"{m.group(1)}.groupby([{m.group(1)}.index.get_level_values(1), "
+                f"{m.group(1)}.index.get_level_values(0).normalize()])"
+            ),
+            fixed_code,
+        )
+        # groupby(level=['date', 'instrument'])
+        fixed_code = re.sub(
+            r"(\w+)\.groupby\(level=\['date',\s*'instrument'\]\)",
+            lambda m: (
+                self.fixes_applied.append(
+                    f"multiindex_groupby: {m.group(0)[:60]} → two-level get_level_values"
+                )
+                or f"{m.group(1)}.groupby([{m.group(1)}.index.get_level_values(0).normalize(), "
+                f"{m.group(1)}.index.get_level_values(1)])"
+            ),
+            fixed_code,
+        )
+        # single: groupby(level=['instrument']) → groupby(level=1)
+        fixed_code = re.sub(
+            r"\.groupby\(level=\['instrument'\]\)",
+            lambda m: (self.fixes_applied.append("multiindex_groupby: groupby(level=['instrument']) → level=1") or ".groupby(level=1)"),
+            fixed_code,
+        )
+
+        return fixed_code
+
+    def _fix_chained_groupby(self, code: str) -> str:
+        """
+        Fix two broken patterns the LLM generates when trying to group by (instrument, date):
+
+        Pattern A — chained groupby (runtime AttributeError):
+            var.groupby(level=1).groupby('date')
+            → var.groupby([var.index.get_level_values(1),
+                           var.index.get_level_values(0).normalize()])
+
+        Pattern B — keyword arg inside list (SyntaxError):
+            var.groupby([level=1, 'date'])
+            → same two-level replacement
+        """
+        fixed_code = code
+
+        def _two_level(var: str, tag: str) -> str:
+            self.fixes_applied.append(f"chained_groupby: {tag} → two-level")
+            return (
+                f"{var}.groupby([{var}.index.get_level_values(1), "
+                f"{var}.index.get_level_values(0).normalize()])"
+            )
+
+        # Pattern A: var.groupby(level=N).groupby('date')
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(level=\d+\)\.groupby\(["\']date["\']\)',
+            lambda m: _two_level(m.group(1), m.group(0)[:60]),
+            fixed_code,
+        )
+
+        # Pattern B: .groupby([level=N, 'date']) — SyntaxError in Python.
+        # The variable before .groupby may be complex (e.g. df[mask]) so we don't
+        # try to capture it; we use df as the index reference (always correct since
+        # all filtered frames share df's MultiIndex structure).
+        def _two_level_df(tag: str) -> str:
+            self.fixes_applied.append(f"chained_groupby: {tag} → two-level")
+            return ".groupby([df.index.get_level_values(1), df.index.get_level_values(0).normalize()])"
+
+        fixed_code = re.sub(
+            r'\.groupby\(\[\s*level\s*=\s*\d+\s*,\s*["\']?date["\']?\s*\]\)',
+            lambda m: _two_level_df(m.group(0)[:60]),
+            fixed_code,
+        )
+        # Also handle reversed order: ['date', level=N]
+        fixed_code = re.sub(
+            r'\.groupby\(\[\s*["\']?date["\']?\s*,\s*level\s*=\s*\d+\s*\]\)',
+            lambda m: _two_level_df(m.group(0)[:60]),
+            fixed_code,
+        )
+
+        return fixed_code
+
+    def _fix_rolling_ddof(self, code: str) -> str:
+        """
+        Fix: pandas rolling() does not accept a ddof kwarg — raises TypeError.
+        Remove ddof from both rolling(..., ddof=N) and rolling(...).std(ddof=N).
+        """
+        fixed_code = code
+
+        # Form 1: ddof inside rolling() — .rolling(window=N, min_periods=M, ddof=K)
+        def _strip_ddof_from_rolling(m):
+            inner = re.sub(r',?\s*ddof\s*=\s*\d+', '', m.group(1))
+            inner = inner.strip(', ')
+            self.fixes_applied.append("rolling_ddof: removed ddof from rolling()")
+            return f'.rolling({inner})'
+
+        fixed_code = re.sub(r'\.rolling\(([^)]*ddof\s*=\s*\d+[^)]*)\)', _strip_ddof_from_rolling, fixed_code)
+
+        # Form 2: ddof inside .std() / .var() — .std(ddof=N)
+        if re.search(r'\.(std|var)\([^)]*ddof\s*=\s*\d+', fixed_code):
+            fixed_code = re.sub(r'\.(std|var)\([^)]*ddof\s*=\s*\d+[^)]*\)', r'.\1()', fixed_code)
+            self.fixes_applied.append("rolling_ddof: removed ddof from std()/var()")
+
+        return fixed_code
+
     def _fix_min_periods(self, code: str) -> str:
         """
         Fix: Ensure min_periods matches window size in rolling calculations.
@@ -325,6 +681,45 @@ class FactorAutoFixer:
             fixed_code = fixed_code.replace(old_code, new_code)
             self.fixes_applied.append(f"groupby: fixed rolling correlation (window={window}) with reset_index")
 
+        # === GENERAL FIX: DF.groupby(level=N)['col'].apply(lambda x: EXPR) ===
+        # apply() on a grouped Series returns a MultiIndex result (extra level prepended),
+        # causing index shape mismatch when assigned back to df['col'].
+        # Replace with transform() which preserves the original index.
+        col_apply_pattern = re.compile(
+            r"(\w+)\.groupby\(level=(\d+)\)\['([^']+)'\]\.apply\((\s*lambda\s+\w+\s*:.*?)\)",
+            re.DOTALL,
+        )
+        for m in list(col_apply_pattern.finditer(fixed_code)):
+            full = m.group(0)
+            df_var = m.group(1)
+            level = m.group(2)
+            col = m.group(3)
+            lam = m.group(4).strip()
+            new_expr = f"{df_var}.groupby(level={level})['{col}'].transform({lam})"
+            fixed_code = fixed_code.replace(full, new_expr, 1)
+            self.fixes_applied.append(
+                f"groupby: {df_var}.groupby(level={level})['{col}'].apply() → transform()"
+            )
+
+        # === FIX: .transform(...).reset_index(level=N, drop=True) ===
+        # transform() already returns the same index as the input — adding reset_index()
+        # after it drops an index level and causes ValueError on assignment back to df['col'].
+        # Detected line-by-line: if a line contains both .transform( and .reset_index(level=
+        reset_suffix = re.compile(r'\s*\.reset_index\s*\(\s*level\s*=[^,)]+,\s*drop\s*=\s*True\s*\)\s*$')
+        new_lines = []
+        changed = False
+        for line in fixed_code.splitlines():
+            if '.transform(' in line and '.reset_index(' in line:
+                cleaned = reset_suffix.sub('', line)
+                if cleaned != line:
+                    new_lines.append(cleaned)
+                    changed = True
+                    continue
+            new_lines.append(line)
+        if changed:
+            fixed_code = '\n'.join(new_lines)
+            self.fixes_applied.append("groupby: removed spurious .reset_index() after .transform()")
+
         # Pattern: Simple groupby().apply() with rolling().method()
         # df.groupby(level=N).apply(lambda x: x['col'].rolling(...).method())
         apply_pattern = r"df\.groupby\(level=(\d+)\)\.apply\(\s*lambda\s+x:\s+x\['([^']+)'\]\.rolling\([^)]+\)\.(\w+)\([^)]*\)\s*\)"

rdagent/components/coder/factor_coder/factor.py

@@ -161,8 +161,7 @@ class FactorFBWorkspace(FBWorkspace):
 
             try:
                 subprocess.check_output(
-                    f"{FACTOR_COSTEER_SETTINGS.python_bin} {execution_code_path}",
-                    shell=True,
+                    [FACTOR_COSTEER_SETTINGS.python_bin, str(execution_code_path)],
                     cwd=self.workspace_path,
                     stderr=subprocess.STDOUT,
                     timeout=FACTOR_COSTEER_SETTINGS.file_based_execution_timeout,

rdagent/components/coder/finetune/unified_validator.py

@@ -6,6 +6,7 @@ Two-step validation:
 2. Micro-batch testing - Runtime validation with small dataset
 """
 
+import ast
 import json
 import re
 import time
@@ -229,7 +230,7 @@ class LLMConfigValidator:
             final_metrics = re.search(r"\{'train_runtime':[^}]+\}", stdout)
             if final_metrics:
                 try:
-                    metrics = eval(final_metrics.group(0))  # Safe: only numbers and strings
+                    metrics = ast.literal_eval(final_metrics.group(0))
                     result["final_metrics"] = {
                         "train_loss": metrics.get("train_loss"),
                         "train_runtime": metrics.get("train_runtime"),

rdagent/log/daily_log.py

@@ -27,6 +27,7 @@ Usage:
 from __future__ import annotations
 
 import json as _json
+import logging
 import sys
 import threading
 from contextlib import contextmanager
@@ -36,21 +37,24 @@ from typing import Any
 
 from loguru import logger as _root
 
-# ── paths ─────────────────────────────────────────────────────────────────────
+# ── paths ─────────────────────────────────────────────────────────────────────────────────
 LOGS_ROOT: Path = Path(__file__).parent.parent.parent / "logs"
 
-# ── format ────────────────────────────────────────────────────────────────────
+# ── format ────────────────────────────────────────────────────────────────────────────────
 _FILE_FMT = (
     "{time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | {extra[cmd]: <18} | {message}"
 )
 
-# ── internal state ─────────────────────────────────────────────────────────────
+# ── internal state ─────────────────────────────────────────────────────────────────────────────
 _registered: set[str] = set()   # command keys that already have a file sink
 _all_added: bool = False         # whether the combined all.log sink is active
 _llm_log_lock = threading.Lock()  # guards concurrent writes to llm_calls.jsonl
 
+# Maximum characters stored per field in llm_calls.jsonl to prevent GB-scale files.
+_LLM_CALL_MAX_CHARS = 500
 
-# ── helpers ───────────────────────────────────────────────────────────────────
+
+# ── helpers ────────────────────────────────────────────────────────────────────────────────
 
 def _today_dir() -> Path:
     d = LOGS_ROOT / datetime.now().strftime("%Y-%m-%d")
@@ -79,7 +83,7 @@ def _banner(log, title: str, meta: dict[str, Any]) -> None:
     log.info(sep)
 
 
-# ── public API ────────────────────────────────────────────────────────────────
+# ── public API ──────────────────────────────────────────────────────────────────────────────
 
 def log_llm_call(
     system: str | None,
@@ -88,16 +92,19 @@ def log_llm_call(
     start_time: Any = None,
     end_time: Any = None,
 ) -> None:
-    """Append one complete LLM call to logs/YYYY-MM-DD/llm_calls.jsonl.
+    """Append one LLM call summary to logs/YYYY-MM-DD/llm_calls.jsonl.
+
+    Prompt/response content is capped at _LLM_CALL_MAX_CHARS to prevent
+    GB-scale log files from long-running loops.
 
     Each line is a self-contained JSON object so the file is grep/jq-friendly:
       jq 'select(.duration_ms > 5000)' logs/2026-04-17/llm_calls.jsonl
     """
     entry: dict[str, Any] = {
         "ts": datetime.now().isoformat(timespec="milliseconds"),
-        "system": system or "",
-        "user": user,
-        "response": response,
+        "system": (system or "")[:_LLM_CALL_MAX_CHARS],
+        "user": user[:_LLM_CALL_MAX_CHARS],
+        "response": response[:_LLM_CALL_MAX_CHARS],
     }
     if start_time is not None and end_time is not None:
         try:
@@ -130,13 +137,13 @@ def setup(command: str, **context: Any):
     key = command.lower()
 
     if key not in _registered:
-        # Per-command rotating file
         _root.add(
             str(log_dir / f"{key}.log"),
             format=_FILE_FMT,
             filter=lambda r, k=key: r["extra"].get("cmd", "").lower() == k,
-            rotation="00:00",      # new file at midnight
-            retention="30 days",
+            rotation="50 MB",
+            compression="gz",
+            retention="7 days",
             encoding="utf-8",
             enqueue=True,
             backtrace=False,
@@ -145,13 +152,13 @@ def setup(command: str, **context: Any):
         _registered.add(key)
 
     if not _all_added:
-        # Combined log — all commands
         _root.add(
             str(log_dir / "all.log"),
             format=_FILE_FMT,
             filter=lambda r: "cmd" in r["extra"],
-            rotation="00:00",
-            retention="60 days",
+            rotation="100 MB",
+            compression="gz",
+            retention="7 days",
             encoding="utf-8",
             enqueue=True,
             backtrace=False,

rdagent/log/ui/llm_st.py

@@ -0,0 +1,306 @@
+import argparse
+import json
+import pickle  # nosec
+import re
+import time
+from pathlib import Path
+
+import streamlit as st
+from streamlit import session_state
+
+from rdagent.log.ui.conf import UI_SETTING
+from rdagent.log.utils import extract_evoid, extract_loopid_func_name
+
+st.set_page_config(layout="wide", page_title="debug_llm", page_icon="🎓", initial_sidebar_state="expanded")
+
+# 获取 log_path 参数
+parser = argparse.ArgumentParser(description="RD-Agent Streamlit App")
+parser.add_argument("--log_dir", type=str, help="Path to the log directory")
+args = parser.parse_args()
+
+
+def get_folders_sorted(log_path):
+    """缓存并返回排序后的文件夹列表，并加入进度打印"""
+    with st.spinner("正在加载文件夹列表..."):
+        folders = sorted(
+            (folder for folder in log_path.iterdir() if folder.is_dir() and list(folder.iterdir())),
+            key=lambda folder: folder.stat().st_mtime,
+            reverse=True,
+        )
+        st.write(f"找到 {len(folders)} 个文件夹")
+    return [folder.name for folder in folders]
+
+
+if UI_SETTING.enable_cache:
+    get_folders_sorted = st.cache_data(get_folders_sorted)
+
+
+# 设置主日志路径
+main_log_path = Path(args.log_dir) if args.log_dir else Path("./log")
+if not main_log_path.exists():
+    st.error(f"Log dir {main_log_path} does not exist!")
+    st.stop()
+
+if "data" not in session_state:
+    session_state.data = []
+if "log_path" not in session_state:
+    session_state.log_path = None
+
+tlist = []
+
+
+def load_data():
+    """加载数据到 session_state 并显示进度"""
+    log_file = main_log_path / session_state.log_path / "debug_llm.pkl"
+    try:
+        with st.spinner(f"正在加载数据文件 {log_file}..."):
+            start_time = time.time()
+            with open(log_file, "rb") as f:
+                session_state.data = pickle.load(f, encoding="utf-8")  # nosec
+            st.success(f"数据加载完成！耗时 {time.time() - start_time:.2f} 秒")
+            st.session_state["current_loop"] = 1
+    except Exception as e:
+        session_state.data = [{"error": str(e)}]
+        st.error(f"加载数据失败: {e}")
+
+
+# UI - Sidebar
+with st.sidebar:
+    st.markdown(":blue[**Log Path**]")
+    manually = st.toggle("Manual Input")
+    if manually:
+        st.text_input("log path", key="log_path", label_visibility="collapsed")
+    else:
+        folders = get_folders_sorted(main_log_path)
+        st.selectbox(f"**Select from {main_log_path.absolute()}**", folders, key="log_path")  # nosec B608 — not SQL, Bandit false positive on "Select" in UI label
+
+    if st.button("Refresh Data"):
+        load_data()
+        st.rerun()
+
+
+# Helper functions
+def show_text(text, lang=None):
+    """显示文本代码块"""
+    if lang:
+        st.code(text, language=lang, wrap_lines=True)
+    elif "\n" in text:
+        st.code(text, language="python", wrap_lines=True)
+    else:
+        st.code(text, language="html", wrap_lines=True)
+
+
+def highlight_prompts_uri(uri):
+    """高亮 URI 的格式"""
+    parts = uri.split(":")
+    return f"**{parts[0]}:**:green[**{parts[1]}**]"
+
+
+# Display Data
+progress_text = st.empty()
+progress_bar = st.progress(0)
+
+# 每页展示一个 Loop
+LOOPS_PER_PAGE = 1
+
+# 获取所有的 Loop ID
+loop_groups = {}
+for i, d in enumerate(session_state.data):
+    tag = d["tag"]
+    loop_id, _ = extract_loopid_func_name(tag)
+    if loop_id:
+        if loop_id not in loop_groups:
+            loop_groups[loop_id] = []
+        loop_groups[loop_id].append(d)
+
+# 按 Loop ID 排序
+sorted_loop_ids = sorted(loop_groups.keys(), key=int)  # 假设 Loop ID 是数字
+total_loops = len(sorted_loop_ids)
+total_pages = total_loops  # 每页展示一个 Loop
+
+
+# simple display
+# FIXME: Delete this simple UI if trace have tag(evo_id & loop_id)
+# with st.sidebar:
+#     start = int(st.text_input("start", 0))
+#     end = int(st.text_input("end", 100))
+# for m in session_state.data[start:end]:
+#     if "tpl" in m["tag"]:
+#         obj = m["obj"]
+#         uri = obj["uri"]
+#         tpl = obj["template"]
+#         cxt = obj["context"]
+#         rd = obj["rendered"]
+#         with st.expander(highlight_prompts_uri(uri), expanded=False, icon="⚙️"):
+#             t1, t2, t3 = st.tabs([":green[**Rendered**]", ":blue[**Template**]", ":orange[**Context**]"])
+#             with t1:
+#                 show_text(rd)
+#             with t2:
+#                 show_text(tpl, lang="django")
+#             with t3:
+#                 st.json(cxt)
+#     if "llm" in m["tag"]:
+#         obj = m["obj"]
+#         system = obj.get("system", None)
+#         user = obj["user"]
+#         resp = obj["resp"]
+#         with st.expander(f"**LLM**", expanded=False, icon="🤖"):
+#             t1, t2, t3 = st.tabs([":green[**Response**]", ":blue[**User**]", ":orange[**System**]"])
+#             with t1:
+#                 try:
+#                     rdict = json.loads(resp)
+#                     if "code" in rdict:
+#                         code = rdict["code"]
+#                         st.markdown(":red[**Code in response dict:**]")
+#                         st.code(code, language="python", wrap_lines=True, line_numbers=True)
+#                         rdict.pop("code")
+#                     elif "spec" in rdict:
+#                         spec = rdict["spec"]
+#                         st.markdown(":red[**Spec in response dict:**]")
+#                         st.markdown(spec)
+#                         rdict.pop("spec")
+#                     else:
+#                         # show model codes
+#                         showed_keys = []
+#                         for k, v in rdict.items():
+#                             if k.startswith("model_") and k.endswith(".py"):
+#                                 st.markdown(f":red[**{k}**]")
+#                                 st.code(v, language="python", wrap_lines=True, line_numbers=True)
+#                                 showed_keys.append(k)
+#                         for k in showed_keys:
+#                             rdict.pop(k)
+#                     st.write(":red[**Other parts (except for the code or spec) in response dict:**]")
+#                     st.json(rdict)
+#                 except:
+#                     st.json(resp)
+#             with t2:
+#                 show_text(user)
+#             with t3:
+#                 show_text(system or "No system prompt available")
+
+
+if total_pages:
+    # 初始化 current_loop
+    if "current_loop" not in st.session_state:
+        st.session_state["current_loop"] = 1
+
+    # Loop 导航按钮
+    col1, col2, col3, col4, col5 = st.sidebar.columns([1.2, 1, 2, 1, 1.2])
+
+    with col1:
+        if st.button("|<"):  # 首页
+            st.session_state["current_loop"] = 1
+    with col2:
+        if st.button("<") and st.session_state["current_loop"] > 1:  # 上一页
+            st.session_state["current_loop"] -= 1
+    with col3:
+        # 下拉列表显示所有 Loop
+        st.session_state["current_loop"] = st.selectbox(
+            "选择 Loop",
+            options=list(range(1, total_loops + 1)),
+            index=st.session_state["current_loop"] - 1,  # 默认选中当前 Loop
+            label_visibility="collapsed",  # 隐藏标签
+        )
+    with col4:
+        if st.button("\>") and st.session_state["current_loop"] < total_loops:  # 下一页
+            st.session_state["current_loop"] += 1
+    with col5:
+        if st.button("\>|"):  # 最后一页
+            st.session_state["current_loop"] = total_loops
+
+    # 获取当前 Loop
+    current_loop = st.session_state["current_loop"]
+
+    # 渲染当前 Loop 数据
+    loop_id = sorted_loop_ids[current_loop - 1]
+    progress_text = st.empty()
+    progress_text.text(f"正在处理 Loop {loop_id}...")
+    progress_bar.progress(current_loop / total_loops, text=f"Loop :green[**{current_loop}**] / {total_loops}")
+
+    # 渲染 Loop Header
+    loop_anchor = f"Loop_{loop_id}"
+    if loop_anchor not in tlist:
+        tlist.append(loop_anchor)
+        st.header(loop_anchor, anchor=loop_anchor, divider="blue")
+
+    # 渲染当前 Loop 的所有数据
+    loop_data = loop_groups[loop_id]
+    for d in loop_data:
+        tag = d["tag"]
+        obj = d["obj"]
+        _, func_name = extract_loopid_func_name(tag)
+        evo_id = extract_evoid(tag)
+
+        func_anchor = f"loop_{loop_id}.{func_name}"
+        if func_anchor not in tlist:
+            tlist.append(func_anchor)
+            st.header(f"in *{func_name}*", anchor=func_anchor, divider="green")
+
+        evo_anchor = f"loop_{loop_id}.evo_step_{evo_id}"
+        if evo_id and evo_anchor not in tlist:
+            tlist.append(evo_anchor)
+            st.subheader(f"evo_step_{evo_id}", anchor=evo_anchor, divider="orange")
+
+        # 根据 tag 渲染内容
+        if "debug_exp_gen" in tag:
+            with st.expander(
+                f"Exp in :violet[**{obj.experiment_workspace.workspace_path}**]", expanded=False, icon="🧩"
+            ):
+                st.write(obj)
+        elif "debug_tpl" in tag:
+            uri = obj["uri"]
+            tpl = obj["template"]
+            cxt = obj["context"]
+            rd = obj["rendered"]
+            with st.expander(highlight_prompts_uri(uri), expanded=False, icon="⚙️"):
+                t1, t2, t3 = st.tabs([":green[**Rendered**]", ":blue[**Template**]", ":orange[**Context**]"])
+                with t1:
+                    show_text(rd)
+                with t2:
+                    show_text(tpl, lang="django")
+                with t3:
+                    st.json(cxt)
+        elif "debug_llm" in tag:
+            system = obj.get("system", None)
+            user = obj["user"]
+            resp = obj["resp"]
+            with st.expander(f"**LLM**", expanded=False, icon="🤖"):
+                t1, t2, t3 = st.tabs([":green[**Response**]", ":blue[**User**]", ":orange[**System**]"])
+                with t1:
+                    try:
+                        rdict = json.loads(resp)
+                        if "code" in rdict:
+                            code = rdict["code"]
+                            st.markdown(":red[**Code in response dict:**]")
+                            st.code(code, language="python", wrap_lines=True, line_numbers=True)
+                            rdict.pop("code")
+                        elif "spec" in rdict:
+                            spec = rdict["spec"]
+                            st.markdown(":red[**Spec in response dict:**]")
+                            st.markdown(spec)
+                            rdict.pop("spec")
+                        else:
+                            # show model codes
+                            showed_keys = []
+                            for k, v in rdict.items():
+                                if k.startswith("model_") and k.endswith(".py"):
+                                    st.markdown(f":red[**{k}**]")
+                                    st.code(v, language="python", wrap_lines=True, line_numbers=True)
+                                    showed_keys.append(k)
+                            for k in showed_keys:
+                                rdict.pop(k)
+                        st.write(":red[**Other parts (except for the code or spec) in response dict:**]")
+                        st.json(rdict)
+                    except:
+                        st.json(resp)
+                with t2:
+                    show_text(user)
+                with t3:
+                    show_text(system or "No system prompt available")
+
+    progress_text.text("当前 Loop 数据处理完成！")
+
+    # Sidebar TOC
+    with st.sidebar:
+        toc = "\n".join([f"- [{t}](#{t})" if t.startswith("L") else f"  - [{t.split('.')[1]}](#{t})" for t in tlist])
+        st.markdown(toc, unsafe_allow_html=True)

rdagent/oai/backend/base.py

@@ -720,7 +720,13 @@ class APIBackend(ABC):
 
             if finish_reason is None or finish_reason != "length":
                 break  # we get a full response now.
-            new_messages.append({"role": "assistant", "content": response})
+            # Merge into the previous assistant message if there already is one at the end.
+            # Appending a second consecutive assistant message causes llama-server to return 400
+            # ("Cannot have 2 or more assistant messages at the end of the list").
+            if new_messages and new_messages[-1]["role"] == "assistant":
+                new_messages[-1]["content"] += response
+            else:
+                new_messages.append({"role": "assistant", "content": response})
         else:
             raise RuntimeError(f"Failed to continue the conversation after {try_n} retries.")
 

rdagent/scenarios/data_science/proposal/exp_gen/merge.py

@@ -182,7 +182,7 @@ class ExpGen2Hypothesis(DSProposalV2ExpGen):
 
         success_fb_list = list(set(trace_fbs))
         logger.info(
-            f"Merge Hypothesis: select {len(success_fb_list)} from {len(trace_fbs)} SOTA experiments found in {len(leaves)} traces"
+            f"Merge Hypothesis: select {len(success_fb_list)} from {len(trace_fbs)} SOTA experiments found in {len(leaves)} traces"  # nosec B608 — not SQL, Bandit false positive on "select" in log message
         )
 
         if len(success_fb_list) > 0:

rdagent/scenarios/data_science/proposal/exp_gen/select/submit.py

@@ -1,3 +1,4 @@
+import ast
 import json
 import os
 import pickle
@@ -292,7 +293,7 @@ class ValidationSelector(SOTAexpSelector):
         Sorts all valid experiments by score and returns the top N.
         """
 
-        mock_folder = f"/tmp/mock/{self.competition}"
+        mock_folder = f"/tmp/mock/{self.competition}"  # nosec B108 — Docker volume mount point derived from internal competition name
 
         try:
             data_py_code, grade_py_code = self._prepare_validation_scripts(
@@ -539,7 +540,7 @@ def process_experiment(
 
         # Run main script
         env = get_ds_env(
-            extra_volumes={f"/tmp/mock/{competition}/{input_folder}": input_folder},
+            extra_volumes={f"/tmp/mock/{competition}/{input_folder}": input_folder},  # nosec B108 — Docker volume mount point derived from internal competition name
             running_timeout_period=DS_RD_SETTING.full_timeout,
         )
         result = ws.run(env=env, entry="python main.py")
@@ -587,8 +588,8 @@ def _parsing_score(grade_stdout: str) -> Optional[float]:
         except:
             pass
         try:
-            # Priority 2: Eval dict
-            return float(eval(json_str)["score"])
+            # Priority 2: safe literal eval for Python-style dicts
+            return float(ast.literal_eval(json_str)["score"])
         except:
             pass
         try:

rdagent/scenarios/kaggle/developer/coder.py

@@ -38,7 +38,7 @@ class KGModelFeatureSelectionCoder(Developer[KGModelExperiment]):
         assert target_model_type in KG_SELECT_MAPPING
         if len(exp.experiment_workspace.data_description) == 1:
             code = (
-                Environment(undefined=StrictUndefined)
+                Environment(undefined=StrictUndefined)  # nosec B701 — renders Python code templates, not HTML; autoescape would corrupt code
                 .from_string(DEFAULT_SELECTION_CODE)
                 .render(feature_index_list=None)
             )
@@ -62,7 +62,7 @@ class KGModelFeatureSelectionCoder(Developer[KGModelExperiment]):
             chosen_index_to_list_index = [i - 1 for i in chosen_index]
 
             code = (
-                Environment(undefined=StrictUndefined)
+                Environment(undefined=StrictUndefined)  # nosec B701 — renders Python code templates, not HTML; autoescape would corrupt code
                 .from_string(DEFAULT_SELECTION_CODE)
                 .render(feature_index_list=chosen_index_to_list_index)
             )

rdagent/scenarios/kaggle/experiment/spaceship-titanic_template/fea_share_preprocess.py

@@ -79,12 +79,12 @@ def preprocess_script():
     This method applies the preprocessing steps to the training, validation, and test datasets.
     """
     if os.path.exists("/kaggle/input/X_train.pkl"):
-        X_train = pd.read_pickle("/kaggle/input/X_train.pkl")
-        X_valid = pd.read_pickle("/kaggle/input/X_valid.pkl")
-        y_train = pd.read_pickle("/kaggle/input/y_train.pkl")
-        y_valid = pd.read_pickle("/kaggle/input/y_valid.pkl")
-        X_test = pd.read_pickle("/kaggle/input/X_test.pkl")
-        others = pd.read_pickle("/kaggle/input/others.pkl")
+        X_train = pd.read_pickle("/kaggle/input/X_train.pkl")  # nosec B301 — trusted Kaggle input
+        X_valid = pd.read_pickle("/kaggle/input/X_valid.pkl")  # nosec B301
+        y_train = pd.read_pickle("/kaggle/input/y_train.pkl")  # nosec B301
+        y_valid = pd.read_pickle("/kaggle/input/y_valid.pkl")  # nosec B301
+        X_test = pd.read_pickle("/kaggle/input/X_test.pkl")  # nosec B301
+        others = pd.read_pickle("/kaggle/input/others.pkl")  # nosec B301
         y_train = pd.Series(y_train).reset_index(drop=True)
         y_valid = pd.Series(y_valid).reset_index(drop=True)
 

rdagent/scenarios/qlib/developer/factor_runner.py

@@ -1,4 +1,6 @@
+import sys
 import os
+import logging
 from pathlib import Path
 """
 Qlib Factor Runner - Executes factor backtests in Docker.
@@ -965,7 +967,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
                 shutil.copy(str(full_data), str(tmp / "intraday_pv.h5"))
 
                 ret = subprocess.run(
-                    ["python", "factor.py"],
+                    ["sys.executable", "factor.py"],
                     cwd=str(tmp),
                     capture_output=True,
                     timeout=300,
@@ -1000,7 +1002,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
             series.to_frame().to_parquet(str(parquet_path))
 
         except Exception:
-            pass
+            logging.debug("Error in save_factor_values_to_parquet", exc_info=True)
 
     def _log_result_warnings(self, factor_name: str, result, metrics: dict) -> None:
         """

rdagent/scenarios/qlib/experiment/utils.py

@@ -67,7 +67,7 @@ def get_file_desc(p: Path, variable_list=[]) -> str:
     """
     p = Path(p)
 
-    JJ_TPL = Environment(undefined=StrictUndefined).from_string("""
+    JJ_TPL = Environment(undefined=StrictUndefined).from_string("""  # nosec B701 — renders plain text description, not HTML; autoescape not applicable
 # {{file_name}}
 
 ## File Type

rdagent/scenarios/qlib/proposal/factor_proposal.py

@@ -1,4 +1,6 @@
+import logging
 import json
+import os
 from typing import List, Tuple
 
 from rdagent.components.coder.factor_coder.factor import FactorExperiment, FactorTask
@@ -9,6 +11,47 @@ from rdagent.scenarios.qlib.experiment.model_experiment import QlibModelExperime
 from rdagent.scenarios.qlib.experiment.quant_experiment import QlibQuantScenario
 from rdagent.utils.agent.tpl import T
 
+
+def _build_compressed_history(trace: Trace, max_history: int) -> str:
+    """Return hypothesis_and_feedback string with only `max_history` entries.
+
+    Older entries beyond the last 2 are compressed to one bullet line each.
+    """
+    if len(trace.hist) == 0:
+        return "No previous hypothesis and feedback available since it's the first round."
+
+    FULL_DETAIL = 2
+    old_hist = trace.hist[:-FULL_DETAIL] if len(trace.hist) > FULL_DETAIL else []
+    recent_hist = trace.hist[-FULL_DETAIL:] if len(trace.hist) > FULL_DETAIL else trace.hist
+
+    parts = []
+    if old_hist:
+        lines = ["## Earlier experiments (summarized):"]
+        for exp, fb in old_hist:
+            names = []
+            for task in exp.sub_tasks:
+                if task is not None and hasattr(task, "factor_name"):
+                    names.append(task.factor_name)
+                elif task is not None and hasattr(task, "model_type"):
+                    names.append(getattr(task, "model_type", "model"))
+            ic_str = ""
+            try:
+                if exp.result is not None and "IC" in exp.result.index:
+                    ic_str = f" IC={exp.result.loc['IC']:.4f}"
+            except Exception:
+                logging.debug("Exception caught", exc_info=True)
+            decision = "PASS" if fb.decision else "FAIL"
+            obs = (fb.observations or "")[:120].replace("\n", " ")
+            lines.append(f"- [{decision}]{ic_str} {', '.join(names) or 'unknown'}: {obs}")
+        parts.append("\n".join(lines))
+
+    if recent_hist:
+        rt = Trace(trace.scen)
+        rt.hist = recent_hist
+        parts.append(T("scenarios.qlib.prompts:hypothesis_and_feedback").r(trace=rt))
+
+    return "\n\n".join(parts)
+
 QlibFactorHypothesis = Hypothesis
 
 
@@ -17,13 +60,10 @@ class QlibFactorHypothesisGen(FactorHypothesisGen):
         super().__init__(scen)
 
     def prepare_context(self, trace: Trace) -> Tuple[dict, bool]:
-        hypothesis_and_feedback = (
-            T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
-                trace=trace,
-            )
-            if len(trace.hist) > 0
-            else "No previous hypothesis and feedback available since it's the first round."
-        )
+        max_h = int(os.environ.get("QLIB_QUANT_MAX_FACTOR_HISTORY", "20"))
+        limited = Trace(trace.scen)
+        limited.hist = trace.hist[-max_h:] if len(trace.hist) > max_h else trace.hist
+        hypothesis_and_feedback = _build_compressed_history(limited, max_h)
         last_hypothesis_and_feedback = (
             T("scenarios.qlib.prompts:last_hypothesis_and_feedback").r(
                 experiment=trace.hist[-1][0], feedback=trace.hist[-1][1]
@@ -70,15 +110,15 @@ class QlibFactorHypothesis2Experiment(FactorHypothesis2Experiment):
         if len(trace.hist) == 0:
             hypothesis_and_feedback = "No previous hypothesis and feedback available since it's the first round."
         else:
+            max_h = int(os.environ.get("QLIB_QUANT_MAX_FACTOR_HISTORY", "20"))
+            factor_hist = [
+                e for e in trace.hist
+                if not hasattr(e[0].hypothesis, "action") or e[0].hypothesis.action == "factor"
+            ][-max_h:]
             specific_trace = Trace(trace.scen)
-            for i in range(len(trace.hist) - 1, -1, -1):
-                if not hasattr(trace.hist[i][0].hypothesis, "action") or trace.hist[i][0].hypothesis.action == "factor":
-                    specific_trace.hist.insert(0, trace.hist[i])
-            if len(specific_trace.hist) > 0:
-                specific_trace.hist.reverse()
-                hypothesis_and_feedback = T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
-                    trace=specific_trace,
-                )
+            specific_trace.hist = factor_hist
+            if specific_trace.hist:
+                hypothesis_and_feedback = _build_compressed_history(specific_trace, max_h)
             else:
                 hypothesis_and_feedback = "No previous hypothesis and feedback available."
 

rdagent/scenarios/qlib/proposal/quant_proposal.py

@@ -1,3 +1,4 @@
+import logging
 import json
 import os
 import random
@@ -175,7 +176,7 @@ class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
                                 ic_val = exp.result.loc["IC"] if "IC" in exp.result.index else ""
                                 ic_str = f" IC={ic_val:.4f}" if ic_val != "" else ""
                         except Exception:
-                            pass
+                            logging.debug("Error getting IC", exc_info=True)
                         decision_str = "PASS" if fb.decision else "FAIL"
                         obs_short = (fb.observations or "")[:120].replace("\n", " ")
                         summary_lines.append(f"- [{decision_str}]{ic_str} {names_str}: {obs_short}")

rdagent/scenarios/rl/autorl_bench/agents/example_agent/train.py

@@ -71,7 +71,7 @@ def submit_for_grading(grading_url: str, model_path: str) -> dict | None:
 def main():
     MODEL_PATH = os.environ.get("MODEL_PATH")
     DATA_PATH = os.environ.get("DATA_PATH")
-    OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/tmp/autorl_output")
+    OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/tmp/autorl_output")  # nosec B108 — Docker container output dir, configurable via env var
     GRADING_SERVER_URL = os.environ.get("GRADING_SERVER_URL", "")
     TRAIN_RATIO = float(os.environ.get("TRAIN_RATIO", "0.05"))
     NUM_EPOCHS = int(os.environ.get("NUM_EPOCHS", "3"))

rdagent/scenarios/rl/autorl_bench/requirements.txt

@@ -9,7 +9,7 @@ peft>=0.18.1
 
 # Evaluation
 opencompass==0.5.1
-setuptools<75  # uv venv doesn't include, opencompass depends on pkg_resources
+setuptools>=78.1.1  # Security fix: GHSA-8g6x-3r52-4m6c (path traversal in PackageIndex.download, arbitrary file write/RCE)
 
 # Inference acceleration (optional, TRL supports 0.10.2-0.12.0)
 # Security: Version >=0.14.0 fixes CVE-2026-22807 (RCE via auto_map dynamic module loading)

rdagent/utils/env.py

@@ -614,7 +614,7 @@ class LocalEnv(Env[ASpecificLocalConf]):
         if self.conf.extra_volumes is not None:
             for lp, rp in self.conf.extra_volumes.items():
                 volumes[lp] = rp["bind"] if isinstance(rp, dict) else rp
-            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"
+            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"  # nosec B108 — fixed Docker volume mount point, not a user-writable temp file
             Path(cache_path).mkdir(parents=True, exist_ok=True)
             volumes[cache_path] = T("scenarios.data_science.share:scen.cache_path").r()
         for lp, rp in running_extra_volume.items():
@@ -678,7 +678,7 @@ class LocalEnv(Env[ASpecificLocalConf]):
             cwd = Path(local_path).resolve() if local_path else None
             env = {k: str(v) if isinstance(v, int) else v for k, v in env.items()}
 
-            process = subprocess.Popen(
+            process = subprocess.Popen(  # nosec B602 — entry is an internal command string set by LocalEnvConf, not user input
                 entry,
                 cwd=cwd,
                 env={**os.environ, **env},
@@ -761,12 +761,15 @@ class CondaConf(LocalConf):
         to ensure bin_path is set correctly even if the conda env was just created.
         """
         conda_path_result = subprocess.run(
-            f"conda run -n {self.conda_env_name} --no-capture-output env | grep '^PATH='",
+            ["conda", "run", "-n", self.conda_env_name, "--no-capture-output", "env"],
             capture_output=True,
             text=True,
-            shell=True,
         )
-        self.bin_path = conda_path_result.stdout.strip().split("=")[1] if conda_path_result.returncode == 0 else ""
+        if conda_path_result.returncode == 0:
+            path_lines = [l for l in conda_path_result.stdout.splitlines() if l.startswith("PATH=")]
+            self.bin_path = path_lines[0].split("=", 1)[1] if path_lines else ""
+        else:
+            self.bin_path = ""
 
 
 class MLECondaConf(CondaConf):
@@ -850,24 +853,22 @@ class QlibCondaEnv(LocalEnv[QlibCondaConf]):
     def prepare(self) -> None:
         """Prepare the conda environment if not already created."""
         try:
-            envs = subprocess.run("conda env list", capture_output=True, text=True, shell=True)
+            envs = subprocess.run(["conda", "env", "list"], capture_output=True, text=True)
             if self.conf.conda_env_name not in envs.stdout:
                 print(f"[yellow]Conda env '{self.conf.conda_env_name}' not found, creating...[/yellow]")
                 subprocess.check_call(
-                    f"conda create -y -n {self.conf.conda_env_name} python=3.10",
-                    shell=True,
+                    ["conda", "create", "-y", "-n", self.conf.conda_env_name, "python=3.10"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install --upgrade pip cython",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install", "--upgrade", "pip", "cython"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install git+https://github.com/microsoft/qlib.git@2fb9380b342556ddb50a4b24e4fe8655d548b2b8",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                     "git+https://github.com/microsoft/qlib.git@2fb9380b342556ddb50a4b24e4fe8655d548b2b8"],
                 )
                 subprocess.check_call(
-                    f"conda run -n {self.conf.conda_env_name} pip install catboost xgboost tables torch",
-                    shell=True,
+                    ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                     "catboost", "xgboost", "tables", "torch"],
                 )
 
         except Exception as e:
@@ -888,10 +889,9 @@ def _sync_conda_cache_with_real_envs() -> None:
     """Ensure the prepared cache includes environments that already exist on disk."""
     try:
         result = subprocess.run(
-            "conda env list",
+            ["conda", "env", "list"],
             capture_output=True,
             text=True,
-            shell=True,
             check=False,
         )
     except Exception as exc:  # pragma: no cover - best-effort helper
@@ -924,14 +924,15 @@ def _prepare_conda_env(env_name: str, requirements_file: Path, python_version: s
         python_version: Python version for the environment
     """
     # 1. Create conda environment if not exists
-    result = subprocess.run(f"conda env list | grep -q '^{env_name} '", shell=True)
-    if result.returncode != 0:
+    env_list = subprocess.run(["conda", "env", "list"], capture_output=True, text=True, check=False)
+    env_exists = any(line.split()[0] == env_name for line in env_list.stdout.splitlines() if line and not line.startswith("#"))
+    if not env_exists:
         print(f"[yellow]Creating conda env '{env_name}' (Python {python_version})...[/yellow]")
-        subprocess.check_call(f"conda create -y -n {env_name} python={python_version}", shell=True)
-        subprocess.check_call(f"conda run -n {env_name} pip install --upgrade pip", shell=True)
+        subprocess.check_call(["conda", "create", "-y", "-n", env_name, f"python={python_version}"])
+        subprocess.check_call(["conda", "run", "-n", env_name, "pip", "install", "--upgrade", "pip"])
 
     print(f"[yellow]Installing dependencies from {requirements_file.name}...[/yellow]")
-    subprocess.check_call(f"conda run -n {env_name} pip install -r {requirements_file}", shell=True)
+    subprocess.check_call(["conda", "run", "-n", env_name, "pip", "install", "-r", str(requirements_file)])
     print(f"[green]Conda env '{env_name}' ready[/green]")
 
     _CONDA_ENV_PREPARED.add(env_name)
@@ -971,8 +972,8 @@ class FTCondaEnv(LocalEnv[FTCondaConf]):
             # Note: flash-attn>=2.8 is required for B200 (sm_100) support
             print("[yellow]Installing flash-attn (compiling, may take a few minutes)...[/yellow]")
             subprocess.check_call(
-                f"conda run -n {self.conf.conda_env_name} pip install 'flash-attn>=2.8' --no-build-isolation --no-cache-dir",
-                shell=True,
+                ["conda", "run", "-n", self.conf.conda_env_name, "pip", "install",
+                 "flash-attn>=2.8", "--no-build-isolation", "--no-cache-dir"],
             )
 
             # Re-update bin_path after prepare() in case the conda env was just created
@@ -1442,7 +1443,7 @@ class DockerEnv(Env[DockerConf]):
         if self.conf.extra_volumes is not None:
             for lp, rp in self.conf.extra_volumes.items():
                 volumes[lp] = rp if isinstance(rp, dict) else {"bind": rp, "mode": self.conf.extra_volume_mode}
-            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"
+            cache_path = "/tmp/sample" if "/sample/" in "".join(self.conf.extra_volumes.keys()) else "/tmp/full"  # nosec B108 — fixed Docker volume mount point, not a user-writable temp file
             Path(cache_path).mkdir(parents=True, exist_ok=True)
             volumes[cache_path] = {
                 "bind": T("scenarios.data_science.share:scen.cache_path").r(),

rdagent/utils/workflow/loop.py

@@ -270,6 +270,11 @@ class LoopBase:
                         msg = "We have reset the loop instance, stop all the routines and resume."
                         raise self.LoopResumeError(msg) from e
                     else:
+                        # Do NOT advance step_idx for unhandled exceptions (e.g. LoopResumeError
+                        # propagating from _propose). Keeping step_idx at the current step lets
+                        # kickoff_loop retry step 0 on the next resume instead of permanently
+                        # corrupting the loop with a missing direct_exp_gen result.
+                        step_forward = False
                         raise  # re-raise unhandled exceptions
                 finally:
                     # No matter the execution succeed or not, we have to finish the following steps

requirements.txt

@@ -9,8 +9,8 @@ psutil
 fire
 fuzzywuzzy
 openai
-litellm>=1.73  # to support `from litellm import get_valid_models`
-aiohttp>=3.13.4  # CVE-2026-22815, CVE-2026-34515, CVE-2026-34516, CVE-2026-34525
+litellm>=1.83.14  # to support `from litellm import get_valid_models`
+aiohttp>=3.13.4  # CVE-2026-22815, CVE-2026-34515, CVE-2026-34516, CVE-2026-34525; >=3.13.4 due to litellm==1.83.14 exact pin
 azure.identity
 pyarrow
 rich
@@ -98,8 +98,8 @@ optuna>=3.5.0
 beautifulsoup4>=4.12.0
 
 # ML Training Pipeline
-lightgbm>=3.3.0
-scipy>=1.9.0
+lightgbm>=3.3.5
+scipy>=1.15.3
 
 # RL Trading (optional - system works without these)
 # Install for full RL training: pip install stable-baselines3[extra] gymnasium

requirements/rl.txt

@@ -8,7 +8,7 @@
 # Only install if you want to use full PPO/A2C/SAC training.
 
 # Core RL library
-stable-baselines3[extra]>=2.0.0
+stable-baselines3[extra]>=2.8.0
 
 # Gymnasium environment (OpenAI Gym successor)
 gymnasium>=0.29.0

test/qlib/test_auto_fixer.py

@@ -0,0 +1,249 @@
+"""Tests for FactorAutoFixer — the pre-execution code patcher."""
+
+import pytest
+
+from rdagent.components.coder.factor_coder.auto_fixer import FactorAutoFixer
+
+
+@pytest.fixture()
+def fixer():
+    return FactorAutoFixer()
+
+
+class TestResetIndexGroupby:
+    def test_replaces_level_groupby_on_reset_var(self, fixer):
+        code = "df_r = df.reset_index()\ndf_r['x'] = df_r.groupby(level=1)['$close'].mean()"
+        result = fixer.fix(code)
+        assert "groupby('instrument')" in result
+
+    def test_does_not_touch_normal_multiindex_groupby(self, fixer):
+        code = "df['x'] = df.groupby(level=1)['$close'].mean()"
+        result = fixer.fix(code)
+        assert "groupby(level=1)" in result
+
+
+class TestGroupbyMixedLevels:
+    def test_strips_string_from_mixed_list(self, fixer):
+        result = fixer.fix("df.groupby(level=[1, 'date']).apply(fn)")
+        assert "groupby(level=1)" in result
+
+    def test_multiple_ints_kept(self, fixer):
+        result = fixer.fix("df.groupby(level=[0, 1, 'x']).apply(fn)")
+        assert "groupby(level=[0, 1])" in result
+
+
+class TestGroupbyColumnOnMultiindex:
+    def test_instrument_date_becomes_two_level(self, fixer):
+        code = "df['v'] = df.groupby(['instrument', 'date'])['$volume'].cumsum()"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "normalize()" in result
+        assert "level=1)" not in result.split("get_level_values")[0]
+
+    def test_date_instrument_becomes_two_level(self, fixer):
+        code = "df['v'] = df.groupby(['date', 'instrument'])['$volume'].cumsum()"
+        result = fixer.fix(code)
+        assert "get_level_values(0).normalize()" in result
+        assert "get_level_values(1)" in result
+
+    def test_single_instrument_becomes_level1(self, fixer):
+        result = fixer.fix("df.groupby(['instrument'])['x'].mean()")
+        assert "groupby(level=1)" in result
+
+    def test_reset_index_not_double_fixed(self, fixer):
+        # After reset_index fix emits groupby('instrument'), this fixer must NOT
+        # convert that to groupby(level=1).
+        code = "df_r = df.reset_index()\ndf_r['x'] = df_r.groupby(level=1)['p'].mean()"
+        result = fixer.fix(code)
+        assert "groupby('instrument')" in result
+
+
+class TestChainedGroupby:
+    def test_chained_groupby_level_then_date(self, fixer):
+        code = "df.groupby(level=1).groupby('date')['price_volume'].transform('cumsum')"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "get_level_values(0).normalize()" in result
+        assert ".groupby('date')" not in result
+
+    def test_chained_groupby_with_double_quotes(self, fixer):
+        code = 'df.groupby(level=0).groupby("date")["col"].sum()'
+        result = fixer.fix(code)
+        assert "get_level_values" in result
+        assert '.groupby("date")' not in result
+
+    def test_list_with_level_keyword_syntax_error(self, fixer):
+        # groupby([level=1, 'date']) is a SyntaxError — must be fixed before execution
+        code = "asian_vol = df[mask].groupby([level=1, 'date'])['log_return'].std()"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "normalize()" in result
+        assert "level=1," not in result
+
+    def test_list_with_level_keyword_reversed(self, fixer):
+        code = "df.groupby(['date', level=1])['x'].mean()"
+        result = fixer.fix(code)
+        assert "get_level_values" in result
+        assert "level=1" not in result
+
+
+class TestMinPeriodsNotTouched:
+    def test_small_min_periods_preserved(self, fixer):
+        # _fix_min_periods is disabled — LLM-set min_periods must not be changed.
+        # window=60, min_periods=1 should stay as-is (was wrongly raised to 60 before).
+        result = fixer.fix("df.groupby(level=1)['x'].transform(lambda x: x.rolling(window=60, min_periods=1).mean())")
+        assert "min_periods=1" in result
+
+    def test_large_window_min_periods_preserved(self, fixer):
+        # window=240 > 96 bars/day: if min_periods were set to 240 the output would be
+        # all-NaN for intraday data. Verify we leave it untouched.
+        result = fixer.fix("df['x'] = df.groupby(level=1)['y'].transform(lambda x: x.rolling(240, min_periods=10).std())")
+        assert "min_periods=10" in result
+
+
+class TestInstrumentColumnAccess:
+    def test_instrument_column_replaced(self, fixer):
+        code = "df['group_key'] = df['instrument'] + '_' + df['day_id'].astype(str)"
+        result = fixer.fix(code)
+        assert "df.index.get_level_values(1)" in result
+        assert "df['instrument']" not in result
+
+    def test_reset_index_var_not_touched(self, fixer):
+        # After reset_index, 'instrument' IS a real column — must not be replaced
+        code = "df_r = df.reset_index()\nval = df_r['instrument'].unique()"
+        result = fixer.fix(code)
+        assert "df_r['instrument']" in result
+        assert "get_level_values" not in result
+
+    def test_groupby_after_instrument_fix(self, fixer):
+        # Combined: df['instrument'] in a groupby context
+        code = "df['key'] = df['instrument']\nout = df.groupby(df['key'])[['$close']].mean()"
+        result = fixer.fix(code)
+        assert "df['instrument']" not in result
+
+    def test_assignment_target_not_touched(self, fixer):
+        # df['instrument'] = <expr> is an assignment — must NOT be converted to
+        # df.index.get_level_values(1) = <expr> (SyntaxError)
+        code = "df['instrument'] = df.index.get_level_values('instrument')"
+        result = fixer.fix(code)
+        assert "df['instrument'] =" in result
+
+
+class TestInstrumentLocMultiindex:
+    def test_loc_replaced_with_xs(self, fixer):
+        code = (
+            "for instrument in df.index.get_level_values('instrument').unique():\n"
+            "    inst_df = df.loc[instrument].copy()\n"
+        )
+        result = fixer.fix(code)
+        assert "df.xs(instrument, level=1)" in result
+        assert "df.loc[instrument]" not in result
+
+    def test_loc_replaced_with_level1_int(self, fixer):
+        code = (
+            "for inst in df.index.get_level_values(1).unique():\n"
+            "    data = df.loc[inst]\n"
+        )
+        result = fixer.fix(code)
+        assert "df.xs(inst, level=1)" in result
+
+    def test_loc_assignment_not_touched(self, fixer):
+        # Write-back df.loc[instrument] = ... must not be changed
+        code = (
+            "for instrument in df.index.get_level_values('instrument').unique():\n"
+            "    df.loc[instrument] = modified\n"
+        )
+        result = fixer.fix(code)
+        assert "df.loc[instrument] = modified" in result
+
+    def test_non_instrument_loop_not_touched(self, fixer):
+        # for-loop not related to instrument levels must not be changed
+        code = "for date in dates:\n    sub = df.loc[date]\n"
+        result = fixer.fix(code)
+        assert "df.loc[date]" in result
+
+
+class TestGroupbyLevelStringNames:
+    def test_level_instrument_date_replaced(self, fixer):
+        code = "df.groupby(level=['instrument', 'date'])['col'].transform('sum')"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "get_level_values(0).normalize()" in result
+        assert "level=['instrument', 'date']" not in result
+
+    def test_level_date_instrument_replaced(self, fixer):
+        code = "data.groupby(level=['date', 'instrument'])['x'].mean()"
+        result = fixer.fix(code)
+        assert "get_level_values(0).normalize()" in result
+        assert "get_level_values(1)" in result
+
+    def test_level_instrument_single_replaced(self, fixer):
+        code = "df.groupby(level=['instrument'])['vol'].sum()"
+        result = fixer.fix(code)
+        assert "groupby(level=1)" in result
+        assert "level=['instrument']" not in result
+
+
+class TestGroupbyApplyToTransform:
+    def test_col_apply_lambda_replaced(self, fixer):
+        code = "df_overlap.groupby(level=1)['$close'].apply(lambda x: np.log(x / x.shift(1)))"
+        result = fixer.fix(code)
+        assert ".transform(" in result
+        assert ".apply(" not in result
+
+    def test_col_apply_lambda_preserves_lambda_body(self, fixer):
+        code = "series.groupby(level=1)['ret'].apply(lambda x: x.cumsum())"
+        result = fixer.fix(code)
+        assert "lambda x: x.cumsum()" in result
+        assert ".transform(" in result
+
+    def test_transform_reset_index_stripped(self, fixer):
+        # .transform() already preserves index — .reset_index() after it is wrong
+        code = "df['v'] = df.groupby(level=1)['x'].transform(lambda x: x.rolling(20).mean()).reset_index(level=0, drop=True)"
+        result = fixer.fix(code)
+        assert ".reset_index(level=0, drop=True)" not in result
+        assert ".transform(" in result
+
+
+class TestZeroVolumeProxy:
+    def test_injects_proxy_when_volume_used(self, fixer):
+        code = (
+            "def calc():\n"
+            "    df = pd.read_hdf('data.h5', key='data')\n"
+            "    df['pv'] = df['$close'] * df['$volume']\n"
+            "    return df[['pv']]\n"
+        )
+        result = fixer.fix(code)
+        assert "volume proxy" in result
+        assert "df['$volume'] = df['$high'] - df['$low']" in result
+        # Proxy must come right after read_hdf line
+        lines = result.splitlines()
+        hdf_idx = next(i for i, l in enumerate(lines) if "read_hdf" in l)
+        assert "volume proxy" in lines[hdf_idx + 1]
+
+    def test_no_injection_when_volume_absent(self, fixer):
+        code = "df = pd.read_hdf('data.h5', key='data')\ndf['x'] = df['$close'].pct_change()\n"
+        result = fixer.fix(code)
+        assert "volume proxy" not in result
+
+    def test_no_double_injection(self, fixer):
+        code = (
+            "def calc():\n"
+            "    df = pd.read_hdf('data.h5', key='data')\n"
+            "    # volume proxy: $volume is always 0 in FX data — use price-range as proxy\n"
+            "    if (df['$volume'] == 0).all():\n"
+            "        df['$volume'] = df['$high'] - df['$low']\n"
+            "    df['pv'] = df['$close'] * df['$volume']\n"
+        )
+        result = fixer.fix(code)
+        assert result.count("volume proxy") == 1
+
+
+class TestRollingDdof:
+    def test_removes_ddof_from_rolling_args(self, fixer):
+        result = fixer.fix("df.rolling(20, min_periods=1, ddof=1).std()")
+        assert "ddof" not in result
+
+    def test_removes_ddof_from_std_args(self, fixer):
+        result = fixer.fix("df.rolling(20).std(ddof=1)")
+        assert "ddof" not in result