chore(master): release 1.3.5 (#38)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.3.3"
+  ".": "1.3.5"
 }

CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## [1.3.5](https://github.com/TPTBusiness/Predix/compare/v1.3.4...v1.3.5) (2026-04-27)
+
+
+### Bug Fixes
+
+* **auto-fixer:** add five new factor code fixes for groupby/apply errors ([449c8fd](https://github.com/TPTBusiness/Predix/commit/449c8fd70a327e604dcca122e4a134f0cca918e4))
+* **auto-fixer:** add four new factor code fixes for common runtime errors ([40484f6](https://github.com/TPTBusiness/Predix/commit/40484f6d300425da481f1edd325da4acbc06ec7d))
+* **auto-fixer:** add groupby([level=N,'date']) SyntaxError fix ([ca77c00](https://github.com/TPTBusiness/Predix/commit/ca77c005bea4abdd8854c1de2b0e8d03b7742161))
+* **auto-fixer:** disable _fix_min_periods for intraday data ([77b0740](https://github.com/TPTBusiness/Predix/commit/77b0740f059349df7e769a378af728aa33b2070e))
+* **auto-fixer:** fix chained groupby(level=N).groupby('date') pattern ([7d5fe32](https://github.com/TPTBusiness/Predix/commit/7d5fe32b31a19ce8b04bd8f5a430720fdb748f7a))
+* **auto-fixer:** fix df.loc[instrument] DateParseError on MultiIndex frames ([b7860ea](https://github.com/TPTBusiness/Predix/commit/b7860eafc0ad26384947ce0510ecf4e9f3425807))
+* **auto-fixer:** fix df['instrument'] KeyError on MultiIndex frames ([aad6bd1](https://github.com/TPTBusiness/Predix/commit/aad6bd1c7c720b3d486e0cf248337f32394773b1))
+* **auto-fixer:** fix two assignment-target bugs in instrument column fixers ([421eedf](https://github.com/TPTBusiness/Predix/commit/421eedffed4b883c24397dc5581c019a3985277f))
+* **auto-fixer:** preserve date dimension in groupby(['instrument','date']) fix ([b58fdd8](https://github.com/TPTBusiness/Predix/commit/b58fdd8be43720b5d4363e0f8de9a01591d4d2dc))
+* **auto-fixer:** remove ddof from rolling() args, not only from std()/var() ([b0fc328](https://github.com/TPTBusiness/Predix/commit/b0fc328d0d4a041c65d8eeb32cb3f2bb86568406))
+* **auto-fixer:** strip spurious .reset_index() after .transform() calls ([8708aae](https://github.com/TPTBusiness/Predix/commit/8708aae6e08728cda1875c775a76dc92e43576f3))
+* **loop:** prevent step_idx advance on unhandled exceptions + fix consecutive assistant messages ([5ec4ad1](https://github.com/TPTBusiness/Predix/commit/5ec4ad1b96b5b99ef42bea7bb828cb1ef709a688))
+
+## [1.3.4](https://github.com/TPTBusiness/Predix/compare/v1.3.3...v1.3.4) (2026-04-27)
+
+
+### Bug Fixes
+
+* **auto-fixer:** add five new factor code fixes for groupby/apply errors ([449c8fd](https://github.com/TPTBusiness/Predix/commit/449c8fd70a327e604dcca122e4a134f0cca918e4))
+* **auto-fixer:** add four new factor code fixes for common runtime errors ([40484f6](https://github.com/TPTBusiness/Predix/commit/40484f6d300425da481f1edd325da4acbc06ec7d))
+* **auto-fixer:** add groupby([level=N,'date']) SyntaxError fix ([ca77c00](https://github.com/TPTBusiness/Predix/commit/ca77c005bea4abdd8854c1de2b0e8d03b7742161))
+* **auto-fixer:** disable _fix_min_periods for intraday data ([77b0740](https://github.com/TPTBusiness/Predix/commit/77b0740f059349df7e769a378af728aa33b2070e))
+* **auto-fixer:** fix chained groupby(level=N).groupby('date') pattern ([7d5fe32](https://github.com/TPTBusiness/Predix/commit/7d5fe32b31a19ce8b04bd8f5a430720fdb748f7a))
+* **auto-fixer:** fix df.loc[instrument] DateParseError on MultiIndex frames ([b7860ea](https://github.com/TPTBusiness/Predix/commit/b7860eafc0ad26384947ce0510ecf4e9f3425807))
+* **auto-fixer:** fix df['instrument'] KeyError on MultiIndex frames ([aad6bd1](https://github.com/TPTBusiness/Predix/commit/aad6bd1c7c720b3d486e0cf248337f32394773b1))
+* **auto-fixer:** preserve date dimension in groupby(['instrument','date']) fix ([b58fdd8](https://github.com/TPTBusiness/Predix/commit/b58fdd8be43720b5d4363e0f8de9a01591d4d2dc))
+* **auto-fixer:** remove ddof from rolling() args, not only from std()/var() ([b0fc328](https://github.com/TPTBusiness/Predix/commit/b0fc328d0d4a041c65d8eeb32cb3f2bb86568406))
+* **backtest:** replace broken MC permutation test with binomial win-rate test ([c38d894](https://github.com/TPTBusiness/Predix/commit/c38d89478f586825bfca5715a96ca70ccd8791a3))
+* **factors:** detect and correct look-ahead bias in daily-constant factors ([eb490a4](https://github.com/TPTBusiness/Predix/commit/eb490a461b66cbd815ae53ac5205115754712432))
+* **factors:** extend look-ahead rules to session factors and add intraday-factor guidance ([c24c100](https://github.com/TPTBusiness/Predix/commit/c24c100442d6487686c0578de0b32d240fcbf215))
+* **loop:** compress old experiment history in proposal prompt to reduce context size ([4bf90a9](https://github.com/TPTBusiness/Predix/commit/4bf90a905ba8b2aba2a818191c19998088cccaaf))
+* **loop:** prevent step_idx advance on unhandled exceptions + fix consecutive assistant messages ([5ec4ad1](https://github.com/TPTBusiness/Predix/commit/5ec4ad1b96b5b99ef42bea7bb828cb1ef709a688))
+
 ## [1.3.3](https://github.com/TPTBusiness/Predix/compare/v1.3.2...v1.3.3) (2026-04-25)
 
 

rdagent/components/coder/factor_coder/auto_fixer.py

@@ -51,13 +51,23 @@ class FactorAutoFixer:
         self.fixes_applied = []
         fixed_code = code
 
-        # Apply fixes in order - groupby fixes MUST come before min_periods fixes
+        # Apply fixes in order
+        # NOTE: _fix_min_periods is intentionally excluded — it increased min_periods to
+        # match window size, which causes all-NaN output for intraday data with 96 bars/day
+        # (window=240 > 96 means zero valid bars per day). The LLM sets its own min_periods.
         fix_methods = [
-            self._fix_groupby_apply_to_transform,  # First: fix groupby patterns
-            self._fix_min_periods,                  # Second: fix min_periods in resulting rolling calls
-            self._fix_inf_nan_handling,             # Third: add inf/nan handling
-            self._fix_data_range_processing,        # Fourth: ensure full data range
-            self._fix_multiindex_groupby,           # Fifth: ensure groupby on MultiIndex
+            self._fix_instrument_column_access,     # First: fix df['instrument'] on MultiIndex
+            self._fix_instrument_loc_multiindex,    # Second: fix df.loc[instrument_var] on MultiIndex
+            self._fix_zero_volume_proxy,            # Third: replace zero $volume with range proxy
+            self._fix_reset_index_groupby,          # Fourth: fix groupby(level=N) after reset_index()
+            self._fix_groupby_mixed_levels,         # Fifth: fix groupby(level=[int, str])
+            self._fix_groupby_column_on_multiindex, # Sixth: fix groupby(['instrument','date']) on MultiIndex
+            self._fix_chained_groupby,              # Seventh: fix groupby(level=N).groupby('date') chain
+            self._fix_rolling_ddof,                 # Eighth: remove unsupported ddof kwarg
+            self._fix_groupby_apply_to_transform,   # Ninth: fix groupby patterns
+            self._fix_inf_nan_handling,             # Tenth: add inf/nan handling
+            self._fix_data_range_processing,        # Eleventh: ensure full data range
+            self._fix_multiindex_groupby,           # Twelfth: ensure groupby on MultiIndex
         ]
 
         for fix_method in fix_methods:
@@ -75,6 +85,352 @@ class FactorAutoFixer:
 
         return fixed_code
 
+    def _fix_instrument_column_access(self, code: str) -> str:
+        """
+        Fix: df['instrument'] raises KeyError on a MultiIndex DataFrame because
+        'instrument' is an index level (level 1), not a column.
+
+        Replace df['instrument'] with df.index.get_level_values('instrument')
+        but only when the DataFrame has a MultiIndex (not after reset_index which
+        would have promoted it to a real column).
+
+        Also fixes df.reset_index()['instrument'] correctly since after reset_index
+        the column exists.
+        """
+        fixed_code = code
+
+        # Skip if already fixed or if reset_index() is being used before the access
+        # We only fix bare df['instrument'] where df is the original MultiIndex frame.
+        # Heuristic: if the assignment lhs or context shows reset_index, leave it alone.
+
+        # Pattern: <varname>['instrument'] where varname is NOT a reset_index result
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        def _replace_instrument_access(m: re.Match) -> str:
+            var = m.group(1)
+            if var in reset_vars:
+                return m.group(0)  # leave reset_index vars alone — column exists
+            self.fixes_applied.append(f"instrument_column: {var}['instrument'] → get_level_values(1)")
+            return f"{var}.index.get_level_values(1)"
+
+        # Exclude assignment targets: var['instrument'] = ... must not become
+        # var.index.get_level_values(1) = ... (SyntaxError: cannot assign to function call)
+        fixed_code = re.sub(r"(\w+)\['instrument'\](?!\s*=)", _replace_instrument_access, fixed_code)
+
+        return fixed_code
+
+    def _fix_instrument_loc_multiindex(self, code: str) -> str:
+        """
+        Fix: df.loc[instrument_var] raises DateParseError on a (datetime, instrument)
+        MultiIndex because pandas tries to match the instrument string against the
+        datetime level (level 0).
+
+        Pattern detected: for-loops iterating over get_level_values('instrument') or
+        get_level_values(1) where the loop variable is then used as df.loc[loop_var].
+
+        Replacement: df.loc[instrument_var] → df.xs(instrument_var, level=1)
+        """
+        fixed_code = code
+
+        # Find variables iterated from get_level_values('instrument') or get_level_values(1)
+        inst_vars = set(
+            re.findall(
+                r"for\s+(\w+)\s+in\s+.+?\.get_level_values\s*\(\s*(?:1|['\"]instrument['\"])\s*\)[^:\n]*:",
+                code,
+            )
+        )
+
+        if not inst_vars:
+            return fixed_code
+
+        for var in inst_vars:
+            # Replace DF.loc[var] (read) with DF.xs(var, level=1)
+            # Exclude write-back patterns (DF.loc[var] = ...) — leave those as-is
+            def _make_replacer(v: str):
+                def _replace(m: re.Match) -> str:
+                    df_var = m.group(1)
+                    self.fixes_applied.append(
+                        f"instrument_loc: {df_var}.loc[{v}] → {df_var}.xs({v}, level=1)"
+                    )
+                    return f"{df_var}.xs({v}, level=1)"
+
+                return _replace
+
+            # Only match when NOT followed by ' =' (assignment)
+            fixed_code = re.sub(
+                rf"(\w+)\.loc\[\s*{re.escape(var)}\s*\](?!\s*=)",
+                _make_replacer(var),
+                fixed_code,
+            )
+
+        return fixed_code
+
+    def _fix_zero_volume_proxy(self, code: str) -> str:
+        """
+        Fix: $volume is always 0 in our EUR/USD dataset (FX has no real volume).
+        Any factor using $volume (VWAP, volume-weighted returns, etc.) produces
+        all-NaN output because 0*price=0 and sum(0)/sum(0)=NaN.
+
+        Insert a guard right after pd.read_hdf() that replaces zero volume with
+        the intraday price-range proxy ($high - $low) so volume-weighted factors
+        produce meaningful signals.
+        """
+        if "'$volume'" not in code and '"$volume"' not in code:
+            return code
+
+        # Already patched
+        if "volume proxy" in code:
+            return code
+
+        lines = code.splitlines()
+        insert_after = -1
+        df_var = "df"
+        indent = "    "
+
+        for i, line in enumerate(lines):
+            if "read_hdf(" in line:
+                m = re.match(r"(\s*)(\w+)\s*=\s*", line)
+                if m:
+                    indent = m.group(1)
+                    df_var = m.group(2)
+                else:
+                    m2 = re.match(r"(\s*)", line)
+                    indent = m2.group(1) if m2 else "    "
+                insert_after = i
+                break
+
+        if insert_after == -1:
+            return code
+
+        proxy_lines = [
+            f"{indent}# volume proxy: $volume is always 0 in FX data — use price-range as proxy",
+            f"{indent}if ({df_var}['$volume'] == 0).all():",
+            f"{indent}    {df_var}['$volume'] = {df_var}['$high'] - {df_var}['$low']",
+        ]
+        lines = lines[: insert_after + 1] + proxy_lines + lines[insert_after + 1 :]
+        self.fixes_applied.append("volume_proxy: replaced zero $volume with ($high - $low)")
+        return "\n".join(lines)
+
+    def _fix_reset_index_groupby(self, code: str) -> str:
+        """
+        Fix: groupby(level=N) on a variable created by .reset_index() fails because
+        reset_index() converts the MultiIndex into regular columns, leaving a plain
+        RangeIndex.  Replace groupby(level=N) on such variables with
+        groupby('instrument').
+
+        Detected pattern:
+            varname = <anything>.reset_index(...)
+            ...
+            varname.groupby(level=0|1)
+        """
+        fixed_code = code
+
+        # Find all variables assigned via reset_index()
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        for var in reset_vars:
+            # Replace var.groupby(level=N) with var.groupby('instrument')
+            pattern = rf'{re.escape(var)}\.groupby\(level\s*=\s*\d+\)'
+            if re.search(pattern, fixed_code):
+                fixed_code = re.sub(pattern, f"{var}.groupby('instrument')", fixed_code)
+                self.fixes_applied.append(f"reset_index_groupby: {var}.groupby(level=N) → groupby('instrument')")
+
+        return fixed_code
+
+    def _fix_groupby_mixed_levels(self, code: str) -> str:
+        """
+        Fix: groupby(level=[int, 'str']) raises AssertionError because string level
+        names don't exist on an unnamed MultiIndex.  Keep only integer levels.
+
+        Pattern:  .groupby(level=[0, 'date'])  →  .groupby(level=0)
+                  .groupby(level=[1, 'date'])  →  .groupby(level=1)
+        """
+        fixed_code = code
+
+        def _keep_int_levels(m):
+            inner = m.group(1)
+            ints = re.findall(r'\b(\d+)\b', inner)
+            if not ints:
+                return m.group(0)
+            replacement = f'.groupby(level={ints[0]})' if len(ints) == 1 else f'.groupby(level=[{", ".join(ints)}])'
+            self.fixes_applied.append(f"mixed_levels: groupby(level=[...,str]) → {replacement}")
+            return replacement
+
+        fixed_code = re.sub(r'\.groupby\(level=\[([^\]]+)\]\)', _keep_int_levels, fixed_code)
+        return fixed_code
+
+    def _fix_groupby_column_on_multiindex(self, code: str) -> str:
+        """
+        Fix: groupby(['instrument', 'date']) on a MultiIndex (datetime, instrument)
+        DataFrame fails with KeyError because those are index levels, not columns.
+
+        Correct replacement preserves BOTH dimensions so intraday calculations reset
+        per day:
+            var.groupby(['instrument', 'date'])
+            → var.groupby([var.index.get_level_values(1), var.index.get_level_values(0).normalize()])
+
+        Single-column groupby(['instrument']) is correctly replaced with groupby(level=1).
+        Note: do NOT convert groupby('instrument') → groupby(level=1) here — that would
+        undo the reset_index_groupby fix which correctly emits groupby('instrument').
+        """
+        fixed_code = code
+
+        # Variables created via reset_index() have a plain RangeIndex — applying
+        # get_level_values() on them would raise AttributeError. Skip those.
+        reset_vars = set(re.findall(r'(\w+)\s*=\s*\w[^=\n]*\.reset_index\(', fixed_code))
+
+        def _replace_two_col_groupby(m: re.Match, order: str) -> str:
+            var = m.group(1)
+            if var in reset_vars:
+                return m.group(0)  # leave reset_index vars alone — RangeIndex, not MultiIndex
+            if order == "instrument_date":
+                repl = (
+                    f"{var}.groupby([{var}.index.get_level_values(1), "
+                    f"{var}.index.get_level_values(0).normalize()])"
+                )
+            else:  # date_instrument
+                repl = (
+                    f"{var}.groupby([{var}.index.get_level_values(0).normalize(), "
+                    f"{var}.index.get_level_values(1)])"
+                )
+            self.fixes_applied.append(f"multiindex_groupby: {m.group(0)[:60]} → two-level")
+            return repl
+
+        # groupby(['instrument', 'date']) — capture variable name before .groupby
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(\[\'instrument\',\s*\'date\'\]\)',
+            lambda m: _replace_two_col_groupby(m, "instrument_date"),
+            fixed_code,
+        )
+        # groupby(['date', 'instrument'])
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(\[\'date\',\s*\'instrument\'\]\)',
+            lambda m: _replace_two_col_groupby(m, "date_instrument"),
+            fixed_code,
+        )
+        # single: groupby(['instrument']) → groupby(level=1), but not on reset_index vars
+        def _replace_single_instrument_groupby(m: re.Match) -> str:
+            # Look backwards to find the variable name
+            prefix = fixed_code[: m.start()]
+            var_match = re.search(r'(\w+)\s*$', prefix)
+            var = var_match.group(1) if var_match else ''
+            if var in reset_vars:
+                return m.group(0)
+            self.fixes_applied.append("multiindex_groupby: groupby(['instrument']) → groupby(level=1)")
+            return ".groupby(level=1)"
+
+        if re.search(r"\.groupby\(\['instrument'\]\)", fixed_code):
+            fixed_code = re.sub(r"\.groupby\(\['instrument'\]\)", _replace_single_instrument_groupby, fixed_code)
+
+        # groupby(level=['instrument', 'date']) — uses level= keyword with string names.
+        # 'date' is NOT a valid level name in our (datetime, instrument) MultiIndex;
+        # replace with get_level_values to normalize datetime to daily timestamps.
+        fixed_code = re.sub(
+            r"(\w+)\.groupby\(level=\['instrument',\s*'date'\]\)",
+            lambda m: (
+                self.fixes_applied.append(
+                    f"multiindex_groupby: {m.group(0)[:60]} → two-level get_level_values"
+                )
+                or f"{m.group(1)}.groupby([{m.group(1)}.index.get_level_values(1), "
+                f"{m.group(1)}.index.get_level_values(0).normalize()])"
+            ),
+            fixed_code,
+        )
+        # groupby(level=['date', 'instrument'])
+        fixed_code = re.sub(
+            r"(\w+)\.groupby\(level=\['date',\s*'instrument'\]\)",
+            lambda m: (
+                self.fixes_applied.append(
+                    f"multiindex_groupby: {m.group(0)[:60]} → two-level get_level_values"
+                )
+                or f"{m.group(1)}.groupby([{m.group(1)}.index.get_level_values(0).normalize(), "
+                f"{m.group(1)}.index.get_level_values(1)])"
+            ),
+            fixed_code,
+        )
+        # single: groupby(level=['instrument']) → groupby(level=1)
+        fixed_code = re.sub(
+            r"\.groupby\(level=\['instrument'\]\)",
+            lambda m: (self.fixes_applied.append("multiindex_groupby: groupby(level=['instrument']) → level=1") or ".groupby(level=1)"),
+            fixed_code,
+        )
+
+        return fixed_code
+
+    def _fix_chained_groupby(self, code: str) -> str:
+        """
+        Fix two broken patterns the LLM generates when trying to group by (instrument, date):
+
+        Pattern A — chained groupby (runtime AttributeError):
+            var.groupby(level=1).groupby('date')
+            → var.groupby([var.index.get_level_values(1),
+                           var.index.get_level_values(0).normalize()])
+
+        Pattern B — keyword arg inside list (SyntaxError):
+            var.groupby([level=1, 'date'])
+            → same two-level replacement
+        """
+        fixed_code = code
+
+        def _two_level(var: str, tag: str) -> str:
+            self.fixes_applied.append(f"chained_groupby: {tag} → two-level")
+            return (
+                f"{var}.groupby([{var}.index.get_level_values(1), "
+                f"{var}.index.get_level_values(0).normalize()])"
+            )
+
+        # Pattern A: var.groupby(level=N).groupby('date')
+        fixed_code = re.sub(
+            r'(\w+)\.groupby\(level=\d+\)\.groupby\(["\']date["\']\)',
+            lambda m: _two_level(m.group(1), m.group(0)[:60]),
+            fixed_code,
+        )
+
+        # Pattern B: .groupby([level=N, 'date']) — SyntaxError in Python.
+        # The variable before .groupby may be complex (e.g. df[mask]) so we don't
+        # try to capture it; we use df as the index reference (always correct since
+        # all filtered frames share df's MultiIndex structure).
+        def _two_level_df(tag: str) -> str:
+            self.fixes_applied.append(f"chained_groupby: {tag} → two-level")
+            return ".groupby([df.index.get_level_values(1), df.index.get_level_values(0).normalize()])"
+
+        fixed_code = re.sub(
+            r'\.groupby\(\[\s*level\s*=\s*\d+\s*,\s*["\']?date["\']?\s*\]\)',
+            lambda m: _two_level_df(m.group(0)[:60]),
+            fixed_code,
+        )
+        # Also handle reversed order: ['date', level=N]
+        fixed_code = re.sub(
+            r'\.groupby\(\[\s*["\']?date["\']?\s*,\s*level\s*=\s*\d+\s*\]\)',
+            lambda m: _two_level_df(m.group(0)[:60]),
+            fixed_code,
+        )
+
+        return fixed_code
+
+    def _fix_rolling_ddof(self, code: str) -> str:
+        """
+        Fix: pandas rolling() does not accept a ddof kwarg — raises TypeError.
+        Remove ddof from both rolling(..., ddof=N) and rolling(...).std(ddof=N).
+        """
+        fixed_code = code
+
+        # Form 1: ddof inside rolling() — .rolling(window=N, min_periods=M, ddof=K)
+        def _strip_ddof_from_rolling(m):
+            inner = re.sub(r',?\s*ddof\s*=\s*\d+', '', m.group(1))
+            inner = inner.strip(', ')
+            self.fixes_applied.append("rolling_ddof: removed ddof from rolling()")
+            return f'.rolling({inner})'
+
+        fixed_code = re.sub(r'\.rolling\(([^)]*ddof\s*=\s*\d+[^)]*)\)', _strip_ddof_from_rolling, fixed_code)
+
+        # Form 2: ddof inside .std() / .var() — .std(ddof=N)
+        if re.search(r'\.(std|var)\([^)]*ddof\s*=\s*\d+', fixed_code):
+            fixed_code = re.sub(r'\.(std|var)\([^)]*ddof\s*=\s*\d+[^)]*\)', r'.\1()', fixed_code)
+            self.fixes_applied.append("rolling_ddof: removed ddof from std()/var()")
+
+        return fixed_code
+
     def _fix_min_periods(self, code: str) -> str:
         """
         Fix: Ensure min_periods matches window size in rolling calculations.
@@ -325,6 +681,45 @@ class FactorAutoFixer:
             fixed_code = fixed_code.replace(old_code, new_code)
             self.fixes_applied.append(f"groupby: fixed rolling correlation (window={window}) with reset_index")
 
+        # === GENERAL FIX: DF.groupby(level=N)['col'].apply(lambda x: EXPR) ===
+        # apply() on a grouped Series returns a MultiIndex result (extra level prepended),
+        # causing index shape mismatch when assigned back to df['col'].
+        # Replace with transform() which preserves the original index.
+        col_apply_pattern = re.compile(
+            r"(\w+)\.groupby\(level=(\d+)\)\['([^']+)'\]\.apply\((\s*lambda\s+\w+\s*:.*?)\)",
+            re.DOTALL,
+        )
+        for m in list(col_apply_pattern.finditer(fixed_code)):
+            full = m.group(0)
+            df_var = m.group(1)
+            level = m.group(2)
+            col = m.group(3)
+            lam = m.group(4).strip()
+            new_expr = f"{df_var}.groupby(level={level})['{col}'].transform({lam})"
+            fixed_code = fixed_code.replace(full, new_expr, 1)
+            self.fixes_applied.append(
+                f"groupby: {df_var}.groupby(level={level})['{col}'].apply() → transform()"
+            )
+
+        # === FIX: .transform(...).reset_index(level=N, drop=True) ===
+        # transform() already returns the same index as the input — adding reset_index()
+        # after it drops an index level and causes ValueError on assignment back to df['col'].
+        # Detected line-by-line: if a line contains both .transform( and .reset_index(level=
+        reset_suffix = re.compile(r'\s*\.reset_index\s*\(\s*level\s*=[^,)]+,\s*drop\s*=\s*True\s*\)\s*$')
+        new_lines = []
+        changed = False
+        for line in fixed_code.splitlines():
+            if '.transform(' in line and '.reset_index(' in line:
+                cleaned = reset_suffix.sub('', line)
+                if cleaned != line:
+                    new_lines.append(cleaned)
+                    changed = True
+                    continue
+            new_lines.append(line)
+        if changed:
+            fixed_code = '\n'.join(new_lines)
+            self.fixes_applied.append("groupby: removed spurious .reset_index() after .transform()")
+
         # Pattern: Simple groupby().apply() with rolling().method()
         # df.groupby(level=N).apply(lambda x: x['col'].rolling(...).method())
         apply_pattern = r"df\.groupby\(level=(\d+)\)\.apply\(\s*lambda\s+x:\s+x\['([^']+)'\]\.rolling\([^)]+\)\.(\w+)\([^)]*\)\s*\)"

rdagent/oai/backend/base.py

@@ -720,7 +720,13 @@ class APIBackend(ABC):
 
             if finish_reason is None or finish_reason != "length":
                 break  # we get a full response now.
-            new_messages.append({"role": "assistant", "content": response})
+            # Merge into the previous assistant message if there already is one at the end.
+            # Appending a second consecutive assistant message causes llama-server to return 400
+            # ("Cannot have 2 or more assistant messages at the end of the list").
+            if new_messages and new_messages[-1]["role"] == "assistant":
+                new_messages[-1]["content"] += response
+            else:
+                new_messages.append({"role": "assistant", "content": response})
         else:
             raise RuntimeError(f"Failed to continue the conversation after {try_n} retries.")
 

rdagent/scenarios/qlib/proposal/factor_proposal.py

@@ -1,4 +1,5 @@
 import json
+import os
 from typing import List, Tuple
 
 from rdagent.components.coder.factor_coder.factor import FactorExperiment, FactorTask
@@ -9,6 +10,47 @@ from rdagent.scenarios.qlib.experiment.model_experiment import QlibModelExperime
 from rdagent.scenarios.qlib.experiment.quant_experiment import QlibQuantScenario
 from rdagent.utils.agent.tpl import T
 
+
+def _build_compressed_history(trace: Trace, max_history: int) -> str:
+    """Return hypothesis_and_feedback string with only `max_history` entries.
+
+    Older entries beyond the last 2 are compressed to one bullet line each.
+    """
+    if len(trace.hist) == 0:
+        return "No previous hypothesis and feedback available since it's the first round."
+
+    FULL_DETAIL = 2
+    old_hist = trace.hist[:-FULL_DETAIL] if len(trace.hist) > FULL_DETAIL else []
+    recent_hist = trace.hist[-FULL_DETAIL:] if len(trace.hist) > FULL_DETAIL else trace.hist
+
+    parts = []
+    if old_hist:
+        lines = ["## Earlier experiments (summarized):"]
+        for exp, fb in old_hist:
+            names = []
+            for task in exp.sub_tasks:
+                if task is not None and hasattr(task, "factor_name"):
+                    names.append(task.factor_name)
+                elif task is not None and hasattr(task, "model_type"):
+                    names.append(getattr(task, "model_type", "model"))
+            ic_str = ""
+            try:
+                if exp.result is not None and "IC" in exp.result.index:
+                    ic_str = f" IC={exp.result.loc['IC']:.4f}"
+            except Exception:
+                pass
+            decision = "PASS" if fb.decision else "FAIL"
+            obs = (fb.observations or "")[:120].replace("\n", " ")
+            lines.append(f"- [{decision}]{ic_str} {', '.join(names) or 'unknown'}: {obs}")
+        parts.append("\n".join(lines))
+
+    if recent_hist:
+        rt = Trace(trace.scen)
+        rt.hist = recent_hist
+        parts.append(T("scenarios.qlib.prompts:hypothesis_and_feedback").r(trace=rt))
+
+    return "\n\n".join(parts)
+
 QlibFactorHypothesis = Hypothesis
 
 
@@ -17,13 +59,10 @@ class QlibFactorHypothesisGen(FactorHypothesisGen):
         super().__init__(scen)
 
     def prepare_context(self, trace: Trace) -> Tuple[dict, bool]:
-        hypothesis_and_feedback = (
-            T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
-                trace=trace,
-            )
-            if len(trace.hist) > 0
-            else "No previous hypothesis and feedback available since it's the first round."
-        )
+        max_h = int(os.environ.get("QLIB_QUANT_MAX_FACTOR_HISTORY", "20"))
+        limited = Trace(trace.scen)
+        limited.hist = trace.hist[-max_h:] if len(trace.hist) > max_h else trace.hist
+        hypothesis_and_feedback = _build_compressed_history(limited, max_h)
         last_hypothesis_and_feedback = (
             T("scenarios.qlib.prompts:last_hypothesis_and_feedback").r(
                 experiment=trace.hist[-1][0], feedback=trace.hist[-1][1]
@@ -70,15 +109,15 @@ class QlibFactorHypothesis2Experiment(FactorHypothesis2Experiment):
         if len(trace.hist) == 0:
             hypothesis_and_feedback = "No previous hypothesis and feedback available since it's the first round."
         else:
+            max_h = int(os.environ.get("QLIB_QUANT_MAX_FACTOR_HISTORY", "20"))
+            factor_hist = [
+                e for e in trace.hist
+                if not hasattr(e[0].hypothesis, "action") or e[0].hypothesis.action == "factor"
+            ][-max_h:]
             specific_trace = Trace(trace.scen)
-            for i in range(len(trace.hist) - 1, -1, -1):
-                if not hasattr(trace.hist[i][0].hypothesis, "action") or trace.hist[i][0].hypothesis.action == "factor":
-                    specific_trace.hist.insert(0, trace.hist[i])
-            if len(specific_trace.hist) > 0:
-                specific_trace.hist.reverse()
-                hypothesis_and_feedback = T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
-                    trace=specific_trace,
-                )
+            specific_trace.hist = factor_hist
+            if specific_trace.hist:
+                hypothesis_and_feedback = _build_compressed_history(specific_trace, max_h)
             else:
                 hypothesis_and_feedback = "No previous hypothesis and feedback available."
 

rdagent/utils/workflow/loop.py

@@ -270,6 +270,11 @@ class LoopBase:
                         msg = "We have reset the loop instance, stop all the routines and resume."
                         raise self.LoopResumeError(msg) from e
                     else:
+                        # Do NOT advance step_idx for unhandled exceptions (e.g. LoopResumeError
+                        # propagating from _propose). Keeping step_idx at the current step lets
+                        # kickoff_loop retry step 0 on the next resume instead of permanently
+                        # corrupting the loop with a missing direct_exp_gen result.
+                        step_forward = False
                         raise  # re-raise unhandled exceptions
                 finally:
                     # No matter the execution succeed or not, we have to finish the following steps

test/qlib/test_auto_fixer.py

@@ -0,0 +1,249 @@
+"""Tests for FactorAutoFixer — the pre-execution code patcher."""
+
+import pytest
+
+from rdagent.components.coder.factor_coder.auto_fixer import FactorAutoFixer
+
+
+@pytest.fixture()
+def fixer():
+    return FactorAutoFixer()
+
+
+class TestResetIndexGroupby:
+    def test_replaces_level_groupby_on_reset_var(self, fixer):
+        code = "df_r = df.reset_index()\ndf_r['x'] = df_r.groupby(level=1)['$close'].mean()"
+        result = fixer.fix(code)
+        assert "groupby('instrument')" in result
+
+    def test_does_not_touch_normal_multiindex_groupby(self, fixer):
+        code = "df['x'] = df.groupby(level=1)['$close'].mean()"
+        result = fixer.fix(code)
+        assert "groupby(level=1)" in result
+
+
+class TestGroupbyMixedLevels:
+    def test_strips_string_from_mixed_list(self, fixer):
+        result = fixer.fix("df.groupby(level=[1, 'date']).apply(fn)")
+        assert "groupby(level=1)" in result
+
+    def test_multiple_ints_kept(self, fixer):
+        result = fixer.fix("df.groupby(level=[0, 1, 'x']).apply(fn)")
+        assert "groupby(level=[0, 1])" in result
+
+
+class TestGroupbyColumnOnMultiindex:
+    def test_instrument_date_becomes_two_level(self, fixer):
+        code = "df['v'] = df.groupby(['instrument', 'date'])['$volume'].cumsum()"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "normalize()" in result
+        assert "level=1)" not in result.split("get_level_values")[0]
+
+    def test_date_instrument_becomes_two_level(self, fixer):
+        code = "df['v'] = df.groupby(['date', 'instrument'])['$volume'].cumsum()"
+        result = fixer.fix(code)
+        assert "get_level_values(0).normalize()" in result
+        assert "get_level_values(1)" in result
+
+    def test_single_instrument_becomes_level1(self, fixer):
+        result = fixer.fix("df.groupby(['instrument'])['x'].mean()")
+        assert "groupby(level=1)" in result
+
+    def test_reset_index_not_double_fixed(self, fixer):
+        # After reset_index fix emits groupby('instrument'), this fixer must NOT
+        # convert that to groupby(level=1).
+        code = "df_r = df.reset_index()\ndf_r['x'] = df_r.groupby(level=1)['p'].mean()"
+        result = fixer.fix(code)
+        assert "groupby('instrument')" in result
+
+
+class TestChainedGroupby:
+    def test_chained_groupby_level_then_date(self, fixer):
+        code = "df.groupby(level=1).groupby('date')['price_volume'].transform('cumsum')"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "get_level_values(0).normalize()" in result
+        assert ".groupby('date')" not in result
+
+    def test_chained_groupby_with_double_quotes(self, fixer):
+        code = 'df.groupby(level=0).groupby("date")["col"].sum()'
+        result = fixer.fix(code)
+        assert "get_level_values" in result
+        assert '.groupby("date")' not in result
+
+    def test_list_with_level_keyword_syntax_error(self, fixer):
+        # groupby([level=1, 'date']) is a SyntaxError — must be fixed before execution
+        code = "asian_vol = df[mask].groupby([level=1, 'date'])['log_return'].std()"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "normalize()" in result
+        assert "level=1," not in result
+
+    def test_list_with_level_keyword_reversed(self, fixer):
+        code = "df.groupby(['date', level=1])['x'].mean()"
+        result = fixer.fix(code)
+        assert "get_level_values" in result
+        assert "level=1" not in result
+
+
+class TestMinPeriodsNotTouched:
+    def test_small_min_periods_preserved(self, fixer):
+        # _fix_min_periods is disabled — LLM-set min_periods must not be changed.
+        # window=60, min_periods=1 should stay as-is (was wrongly raised to 60 before).
+        result = fixer.fix("df.groupby(level=1)['x'].transform(lambda x: x.rolling(window=60, min_periods=1).mean())")
+        assert "min_periods=1" in result
+
+    def test_large_window_min_periods_preserved(self, fixer):
+        # window=240 > 96 bars/day: if min_periods were set to 240 the output would be
+        # all-NaN for intraday data. Verify we leave it untouched.
+        result = fixer.fix("df['x'] = df.groupby(level=1)['y'].transform(lambda x: x.rolling(240, min_periods=10).std())")
+        assert "min_periods=10" in result
+
+
+class TestInstrumentColumnAccess:
+    def test_instrument_column_replaced(self, fixer):
+        code = "df['group_key'] = df['instrument'] + '_' + df['day_id'].astype(str)"
+        result = fixer.fix(code)
+        assert "df.index.get_level_values(1)" in result
+        assert "df['instrument']" not in result
+
+    def test_reset_index_var_not_touched(self, fixer):
+        # After reset_index, 'instrument' IS a real column — must not be replaced
+        code = "df_r = df.reset_index()\nval = df_r['instrument'].unique()"
+        result = fixer.fix(code)
+        assert "df_r['instrument']" in result
+        assert "get_level_values" not in result
+
+    def test_groupby_after_instrument_fix(self, fixer):
+        # Combined: df['instrument'] in a groupby context
+        code = "df['key'] = df['instrument']\nout = df.groupby(df['key'])[['$close']].mean()"
+        result = fixer.fix(code)
+        assert "df['instrument']" not in result
+
+    def test_assignment_target_not_touched(self, fixer):
+        # df['instrument'] = <expr> is an assignment — must NOT be converted to
+        # df.index.get_level_values(1) = <expr> (SyntaxError)
+        code = "df['instrument'] = df.index.get_level_values('instrument')"
+        result = fixer.fix(code)
+        assert "df['instrument'] =" in result
+
+
+class TestInstrumentLocMultiindex:
+    def test_loc_replaced_with_xs(self, fixer):
+        code = (
+            "for instrument in df.index.get_level_values('instrument').unique():\n"
+            "    inst_df = df.loc[instrument].copy()\n"
+        )
+        result = fixer.fix(code)
+        assert "df.xs(instrument, level=1)" in result
+        assert "df.loc[instrument]" not in result
+
+    def test_loc_replaced_with_level1_int(self, fixer):
+        code = (
+            "for inst in df.index.get_level_values(1).unique():\n"
+            "    data = df.loc[inst]\n"
+        )
+        result = fixer.fix(code)
+        assert "df.xs(inst, level=1)" in result
+
+    def test_loc_assignment_not_touched(self, fixer):
+        # Write-back df.loc[instrument] = ... must not be changed
+        code = (
+            "for instrument in df.index.get_level_values('instrument').unique():\n"
+            "    df.loc[instrument] = modified\n"
+        )
+        result = fixer.fix(code)
+        assert "df.loc[instrument] = modified" in result
+
+    def test_non_instrument_loop_not_touched(self, fixer):
+        # for-loop not related to instrument levels must not be changed
+        code = "for date in dates:\n    sub = df.loc[date]\n"
+        result = fixer.fix(code)
+        assert "df.loc[date]" in result
+
+
+class TestGroupbyLevelStringNames:
+    def test_level_instrument_date_replaced(self, fixer):
+        code = "df.groupby(level=['instrument', 'date'])['col'].transform('sum')"
+        result = fixer.fix(code)
+        assert "get_level_values(1)" in result
+        assert "get_level_values(0).normalize()" in result
+        assert "level=['instrument', 'date']" not in result
+
+    def test_level_date_instrument_replaced(self, fixer):
+        code = "data.groupby(level=['date', 'instrument'])['x'].mean()"
+        result = fixer.fix(code)
+        assert "get_level_values(0).normalize()" in result
+        assert "get_level_values(1)" in result
+
+    def test_level_instrument_single_replaced(self, fixer):
+        code = "df.groupby(level=['instrument'])['vol'].sum()"
+        result = fixer.fix(code)
+        assert "groupby(level=1)" in result
+        assert "level=['instrument']" not in result
+
+
+class TestGroupbyApplyToTransform:
+    def test_col_apply_lambda_replaced(self, fixer):
+        code = "df_overlap.groupby(level=1)['$close'].apply(lambda x: np.log(x / x.shift(1)))"
+        result = fixer.fix(code)
+        assert ".transform(" in result
+        assert ".apply(" not in result
+
+    def test_col_apply_lambda_preserves_lambda_body(self, fixer):
+        code = "series.groupby(level=1)['ret'].apply(lambda x: x.cumsum())"
+        result = fixer.fix(code)
+        assert "lambda x: x.cumsum()" in result
+        assert ".transform(" in result
+
+    def test_transform_reset_index_stripped(self, fixer):
+        # .transform() already preserves index — .reset_index() after it is wrong
+        code = "df['v'] = df.groupby(level=1)['x'].transform(lambda x: x.rolling(20).mean()).reset_index(level=0, drop=True)"
+        result = fixer.fix(code)
+        assert ".reset_index(level=0, drop=True)" not in result
+        assert ".transform(" in result
+
+
+class TestZeroVolumeProxy:
+    def test_injects_proxy_when_volume_used(self, fixer):
+        code = (
+            "def calc():\n"
+            "    df = pd.read_hdf('data.h5', key='data')\n"
+            "    df['pv'] = df['$close'] * df['$volume']\n"
+            "    return df[['pv']]\n"
+        )
+        result = fixer.fix(code)
+        assert "volume proxy" in result
+        assert "df['$volume'] = df['$high'] - df['$low']" in result
+        # Proxy must come right after read_hdf line
+        lines = result.splitlines()
+        hdf_idx = next(i for i, l in enumerate(lines) if "read_hdf" in l)
+        assert "volume proxy" in lines[hdf_idx + 1]
+
+    def test_no_injection_when_volume_absent(self, fixer):
+        code = "df = pd.read_hdf('data.h5', key='data')\ndf['x'] = df['$close'].pct_change()\n"
+        result = fixer.fix(code)
+        assert "volume proxy" not in result
+
+    def test_no_double_injection(self, fixer):
+        code = (
+            "def calc():\n"
+            "    df = pd.read_hdf('data.h5', key='data')\n"
+            "    # volume proxy: $volume is always 0 in FX data — use price-range as proxy\n"
+            "    if (df['$volume'] == 0).all():\n"
+            "        df['$volume'] = df['$high'] - df['$low']\n"
+            "    df['pv'] = df['$close'] * df['$volume']\n"
+        )
+        result = fixer.fix(code)
+        assert result.count("volume proxy") == 1
+
+
+class TestRollingDdof:
+    def test_removes_ddof_from_rolling_args(self, fixer):
+        result = fixer.fix("df.rolling(20, min_periods=1, ddof=1).std()")
+        assert "ddof" not in result
+
+    def test_removes_ddof_from_std_args(self, fixer):
+        result = fixer.fix("df.rolling(20).std(ddof=1)")
+        assert "ddof" not in result