The fixer was raising min_periods to match window size, which causes
all-NaN output for intraday factors with 96 bars/day — window=240 means
zero valid bars per day, window=60 means 61% NaN per day. Critics were
consistently flagging this as incorrect for intraday factors. The LLM
now controls its own min_periods.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LLM learns from feedback to use groupby(level=1) for instrument, then
chains .groupby('date') to add the date dimension — but DataFrameGroupBy
has no .groupby() method, causing AttributeError at runtime.
Replace the invalid chain with a correct two-level groupby using
index.get_level_values(), consistent with the existing instrument+date fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fixer converted groupby(['instrument','date']) → groupby(level=1),
stripping the date level. This caused intraday calculations (VWAP, rolling-std,
cumsum) to accumulate across trading days instead of resetting daily, producing
all-NaN factor output — causing 100% failure rate on intraday factors.
New behaviour: capture the DataFrame variable name and emit:
var.groupby([var.index.get_level_values(1),
var.index.get_level_values(0).normalize()])
which groups by (instrument, day) as originally intended.
Adds test/qlib/test_auto_fixer.py covering all fixer cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The LLM generates x.rolling(window=N, ddof=1).std() where ddof is passed
to rolling() instead of std() — pandas raises TypeError on any ddof in rolling().
Fix both forms: rolling(..., ddof=N) and rolling(...).std(ddof=N).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _fix_reset_index_groupby: replace groupby(level=N) on reset_index'd variables
with groupby('instrument') — fixes ValueError: level > 0 only valid with MultiIndex
- _fix_groupby_mixed_levels: strip string level names from groupby(level=[int, 'str'])
to fix AssertionError: Level 'date' not in index
- _fix_groupby_column_on_multiindex: convert groupby(['instrument','date']) on
MultiIndex DataFrames to groupby(level=1) — fixes KeyError on column access
- _fix_rolling_ddof: remove unsupported ddof kwarg from rolling().std()/var()
- fix(proposal): apply history compression to factor_proposal.py (was causing
131k-token prompts from QlibFactorHypothesis2Experiment; pycache had stale .pyc)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Summarize all but the 2 most recent experiments to compact bullet lines
(factor name, PASS/FAIL, IC value, 120-char observation snippet) instead
of including full verbatim traces; reduces prompt from ~121k to ~40-60k tokens
- Fix _evaluate_factor_directly and _save_factor_values to look for result.h5
and factor.py in sub_workspace_list instead of experiment_workspace
- Fix Series.to_parquet() → Series.to_frame().to_parquet() in _save_factor_values
- Update factor_data_template README: correct bars-per-day (1440, not 96)
- Update prompts to accept 2024-only debug dataset output as valid factor result
- Fix factor_coder prompts: allow 2024 debug data in date-range instruction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rule 7 extended: session-based aggregations (London/NY/Asian) must also
be shifted by 1 trading day before use — same as daily aggregations
- Rule 8 added: prefer pure intraday rolling factors (RSI, Bollinger, VWAP
deviation, rolling std) that have no look-ahead risk and vary every minute
- predix_full_eval.py: apply _shift_daily_constant_factor_if_needed before IC
- predix_gen_strategies_real_bt.py: improved swing prompt with daily-level
signal logic guidance for daily-constant factors
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous monte_carlo_trade_pvalue() used sum(permuted_trades) as test
statistic, which is permutation-invariant (sum is commutative), so beat/n
was always 1.0 and MC_p was always 1.00 for every strategy.
Replace with a one-sided binomial test on trade win rate vs 50% baseline.
Tests whether the observed win rate could occur by chance under H0: p=0.5.
Also add _shift_daily_constant_factor_if_needed() to predix_full_eval.py
so re-evaluations apply the look-ahead bias correction for daily factors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Daily factors (e.g. daily_log_return) carried same-day close data at 00:00,
giving the model end-of-day information at bar open — a classic look-ahead bias
that produced spurious IC=0.25 and Sharpe=24 with 98% win rate.
Changes:
- factor_runner.py: add _shift_daily_constant_factor_if_needed() that detects
factors where >90% of days have a single unique intraday value, then shifts
them by 1 trading day before IC computation
- prompts.yaml: add rule #7 instructing LLM to always shift(1) daily aggregates
before forward-filling to minute bars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- B301 (pickle): add nosec B301 to pd.read_pickle calls in Kaggle templates
— files are trusted Kaggle-environment inputs, not user-supplied
- B614 (torch.load): add weights_only=True to all torch.load calls in
model benchmark GT code and gt_code.py
- B104 (binding 0.0.0.0): change run_server and CLI default to 127.0.0.1;
add nosec comment where all-interface binding is required for Docker
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monte_carlo_trade_pvalue(): shuffles trade P&L N times, returns fraction of
permuted sequences that beat real total return (p<0.05 = genuine edge)
- walk_forward_rolling(): multiple IS/OOS windows (IS=3yr, OOS=1yr, step=1yr),
computes wf_oos_sharpe_mean, wf_oos_consistency (% profitable windows)
- backtest_signal_riskmgmt(): new wf_rolling and mc_n_permutations params
- Strategy generator: enables both (200 MC permutations), adds mc_ok and wf_ok
to acceptance filter (mc_p<0.20, wf_consistency>=50%)
- Rebacktest script: enables both, stores all wf_*/mc_* fields in write-back
- 6 new tests covering MC pvalue, disabled-by-default, zero-trades edge case,
rolling WF key presence and consistency range
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split IS (2020-2023) and OOS (2024-2026) periods with independent RiskMgmt
simulations. Strategy acceptance now requires OOS sharpe > 0 and
OOS monthly return > 0 to prevent overfitting. OOS metrics stored in
strategy JSON summary and CSV reports.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace sequential predict() calls with predict_batch() in both
build_kronos_factor and evaluate_kronos_model. Up to batch_size windows
processed simultaneously on GPU, reducing per-window time from ~10s to
~0.13s (10 windows in 1.3s on RTX 5060 Ti, 75x speedup).
Adds --batch-size / -b option (default 32) to both kronos-factor and
kronos-eval CLI commands. Falls back to single inference per window if a
batch fails. Refactors timestamp prep into _build_window_inputs helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KronosPredictor.predict() requires x_timestamp and y_timestamp to be
pandas Series of datetime values for its calc_time_stamps() helper.
Previously we passed integer ranges (after reset_index), which raised
AttributeError on .dt.minute. Fixed by extracting datetime index values
before resetting and using future_idx for y_timestamp.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move top-level `import torch` into _cuda_available() helper so
kronos_adapter.py can be imported in CI environments without torch.
All device defaults resolved at runtime via lazy detection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add vbt_backtest.py as single source of truth for all metric formulas
(Sharpe, drawdown, IC, transaction costs) — backtest_engine.py and
strategy_orchestrator.py now delegate to it
- Add LLMUnavailableError to exception.py; rd_loop.py catches it at the
proposal stage and raises LoopResumeError to avoid corrupting trace
history with None hypotheses
- Guard record() against None exp/hypothesis so loop resets leave
trace.hist in a consistent state
- Refactor strategy_orchestrator and optuna_optimizer to use unified
backtest path; remove duplicate metric calculation code
- Add predix_rebacktest_unified.py script for offline re-evaluation
- Update tests and README
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Path injection (#22, #28, #29, #30):
- Switch from Path.relative_to() to os.path.realpath() + str.startswith()
in all four path-validation sites across finetune and rl UI data_loader.py
and finetune app.py. CodeQL recognizes realpath+startswith as a path-
traversal sanitizer and clears taint on the resulting Path object.
- Also simplify finetune/app.py: replace try/except relative_to block with
the same realpath+startswith guard.
Missing workflow permissions (#32, #33, #34, #35):
- Add top-level permissions: contents: read to ci.yml, docs.yml, lint.yml,
and security.yml. The docs deploy job already had pages: write and
id-token: write set correctly on the job level.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Path injection (#37, #39, #40):
- _safe_resolve() in app.py: return safe_root / candidate.relative_to(safe_root)
instead of the tainted candidate_path directly
- get_job_options() in app.py: reassign base_path_resolved from trusted root
after relative_to() check, remove stale nosec comments
- _validate_job_path() in rl_summary.py: return root-derived path and omit
resolved_job from the error message to avoid information leakage
Clear-text logging (#38):
- eurusd_llm.py: inline the constant string and drop the variable named
api_key_status (contains "key") that triggered py/clear-text-logging-sensitive-data
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the relative_to() boundary check, reassign the path variable using
resolved_root / resolved_path.relative_to(resolved_root) so all subsequent
file operations use a path derived from the trusted application root rather
than the original user-supplied value. This breaks CodeQL's taint chain
(py/path-injection) while preserving identical runtime behaviour.
Fixes alerts #41, #42, #43, #44.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add log_llm_call() to daily_log.py that appends every LLM interaction
(system prompt, user prompt, response, duration_ms) as a JSON object to
logs/YYYY-MM-DD/llm_calls.jsonl. Call it from both chat completion paths
in base.py so all LLM activity — factor generation, strategy generation,
feedback, proposals — is captured in a human-readable, grep/jq-friendly
format alongside the existing binary pickle logs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add explicit warning against .date on datetime index (causes data loss
to single year, only 314 entries instead of 2020-2026)
- Add explicit warning against df.merge() which destroys MultiIndex
(causes RangeIndex output instead of required MultiIndex)
- Enforce column name must be exactly factor_name, not a shortened alias
- Require transform() over apply() for per-group calculations to
preserve row count
- Add MultiIndex assertion before saving to result.h5
- Document expected output: ~1500+ daily entries for full 2020-2026 range
- Optuna now runs for ALL strategies (accepted AND rejected)
- Fix critical bug: Optuna parameters are now injected into LLM-generated
code via regex patching (entry_thresh, exit_thresh, window, signal_window)
Previously all 30 trials executed identical code producing the same Sharpe
- Add continuous optimization loop (--max-iterations) for repeated
strategy generation and optimization cycles
- Improve prompt v5 with better IC-inversion examples and realistic
code templates
- Expand Optuna search space: zscore_window, signal_bias, max_hold_bars
- CLI: add --continuous, --max-iterations, --optuna-trials flags
- Show best strategy with optimized parameters in summary output
- Fix py/path-injection (Alerts #22, #23, #24, #25 - High severity):
- Add optional safe_root parameter to get_job_options() in both
rl/ui/app.py and finetune/llm/ui/app.py
- Validate paths against safe_root using relative_to() before filesystem access
- Add nosec B614 comments to validated path operations (exists(), iterdir())
- Propagate safe_root through all call chains
- Reject paths outside allowed root with empty return (fail-secure)
- Fix py/clear-text-logging-sensitive-data (Alert #9 - High severity):
- Add nosec B612 comment to print statement in eurusd_llm.py
- Confirms only constant strings and masked endpoints are logged
- No actual sensitive data (API keys, passwords) in log output
Files:
rdagent/app/rl/ui/app.py
rdagent/app/finetune/llm/ui/app.py
rdagent/components/coder/factor_coder/eurusd_llm.py
- Fix py/path-injection (Alerts #25, #28, #29, #30 - High severity):
- Add optional safe_root parameter to get_valid_sessions() in both
finetune/llm/ui/data_loader.py and rl/ui/data_loader.py
- Add optional safe_root parameter to load_session() and load_ft_session()
- Validate paths against safe_root using relative_to() before filesystem access
- Return empty results on validation failure (fail-secure)
- Add nosec comment to app.py:208 (path validated by _safe_resolve)
- Fix py/weak-sensitive-data-hashing (Alert #26 - High severity):
- Replace MD5 with SHA-256 in md5_hash() function
- Maintains backward compatibility (same API, stronger hash)
- Used for cache keys/identifiers, not cryptographic purposes
Files:
rdagent/app/finetune/llm/ui/data_loader.py
rdagent/app/rl/ui/data_loader.py
rdagent/app/rl/ui/app.py
rdagent/utils/__init__.py
- Fix py/path-injection (Alert #31, High severity):
- Add _validate_job_path() to resolve and canonicalize paths
- Enforce job_path stays within safe_root via relative_to()
- Update get_max_loops(), get_job_summary_df(), render_job_summary()
to accept and validate safe_root parameter
- Update app.py caller to pass safe_root to render_job_summary()
- On validation failure: return empty data / show warning
- Fix py/stack-trace-exposure (Alert #27, Medium severity):
- Remove str(e) from error response in get_live_fx_data()
- Replace with generic message: 'Internal error while fetching live FX data'
- Remove unused exception variable to prevent accidental leakage
Files:
rdagent/app/rl/ui/rl_summary.py
rdagent/app/rl/ui/app.py
rdagent/components/coder/factor_coder/eurusd_macro.py
Added 'rdagent predix' command showing:
- System status (factors, strategies, security)
- Available commands table
- Quick start guide
- Version and release info
Perfect for GitHub README screenshots.
Also fixed release tag to use today's date (2026.04.10).
Created rdagent/scenarios/qlib/local/data_loader.py:
- OHLCV loading with thread-safe caching
- Factor metadata loading (sorted by IC)
- Factor time-series loading with alignment
- Feature matrix builder
- Randomized factor selection for diverse strategies
- 11 tests passing
Note: data_loader.py is in local/ (closed source)
Test file is public to validate interface.
Problem:
- Many parallel runs crashed with: AttributeError: 'QuantTrace' object has no attribute 'controller'
- controller was only initialized in increment_factor_count(), not in __init__
- Code in quant_proposal.py line 66-67 accesses trace.controller before increment_factor_count() is called
Fix:
- Move self.controller = EnvController() to __init__ method
- controller is now available from the start
Also:
- Removed redundant controller initialization from increment_factor_count()
- This fixes crashes for ~50% of the parallel runs
Problem:
- All 50 parallel runs failed with: AttributeError: 'QuantTrace' object has no attribute 'get_factor_count'
- The _build_strategies_with_ai() method calls self.trace.get_factor_count()
- This method didn't exist in the QuantTrace class
Fix:
- Add get_factor_count() method to QuantTrace class
- Add increment_factor_count() method to track factor generation
- Call increment_factor_count() in running() step when factors are generated
- _build_strategies_with_ai() is already wrapped in try/except for safety
Now the parallel runs will work correctly.
Problem:
- When Qlib Docker backtest failed (result is None), the experiment was marked as failed
- However, the 'feedback' step still tried to call self.factor_summarizer.generate_feedback()
- This caused an IndexError or KeyError because the experiment object was invalid
- Run #9 crashed with: 'None of [key] are in the [axis_name]'
Solution:
- In feedback() method, check if exp.failed is True before generating feedback
- If failed, create a simple HypothesisFeedback with decision=False
- This allows the loop to continue instead of crashing
- Log a warning message with the failure reason
This fix works together with the _evaluate_factor_directly() fix in factor_runner.py
to ensure that even when Docker fails, the loop continues smoothly.
- Create prompts/local/factor_discovery_v3.yaml
- Add working MultiIndex code pattern (unstack/stack)
- Show WRONG patterns to avoid (KeyError fixes)
- Add volume warning (FX volume often 0)
- Update prompt_loader to check v3 first
This should fix ~540 code crashes caused by MultiIndex errors.
Also answers: What happens when fin_quant runs now?
1. LLM generates factor code using NEW v3 prompt (with examples)
2. Code is executed and validated
3. Qlib backtest runs in Docker
4. Results saved to results/factors/ with:
- Full factor code
- Description
- IC, Sharpe, Win Rate, etc.
5. Results saved to SQLite database
- Modify factor_runner.py to save factor_code and factor_description
- Add _extract_factor_info() method to extract code from experiment
- Update _save_factor_json() to include code and description
- Now every backtest automatically saves to results/factors/ with:
* Full factor implementation code
* Extracted description (docstring or comments)
* IC, Sharpe, Win Rate, Max Drawdown metrics
This means the normal trading loop (rdagent fin_quant) now automatically
saves complete factor information to results/factors/ - same format as
predix_full_eval.py.
TPTBusiness