Compare commits

...

20 Commits

Author SHA1 Message Date
github-actions[bot] ff556fd228 chore(master): release 1.4.2 (#48)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-03 11:02:32 +02:00
TPTBusiness b23e145341 test: add 129 tests for critical untested code (core, CoSTEER, factor_coder, model_coder, qlib pipeline) 2026-05-03 10:59:24 +02:00
TPTBusiness 7d3765d0c0 fix: add missing sys import and fix undefined acc_rate in factor eval 2026-05-03 10:19:59 +02:00
github-actions[bot] d53f4bbbeb chore(master): release 1.4.1 (#47)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-03 09:39:56 +02:00
TPTBusiness 5bf84ff835 fix: 15 bug fixes across orchestrator, runner, backtest, and infrastructure
Critical:
- strategy_orchestrator: fix IndentationError that prevented import (line 764)
- factor_runner: fix literal 'sys.executable' string → variable (line 966)

High (path bugs causing wrong directories):
- backtest_engine: fix results_path depth (3→4 .parent hops)
- results_db: fix factors_dir/failed_dir depth (3→4 .parent hops)
- factor_runner: eliminate run_id variable shadowing (parallel_run_id/db_run_id)
- model_runner: fix DB connection leak on add_backtest exception
- optuna_optimizer: fix imported logger shadowed by module-level reassignment

Medium:
- env: handle non-UTF-8 Docker build output with errors='replace'
- env: guard conda env list parsing against empty lines
- factor_runner: add check=False + stderr logging for full-data subprocess
- strategy_orchestrator: log exec() exceptions at ERROR level with traceback
- strategy_orchestrator: warn on unreplaced {{template}} variables in prompts

Low:
- factor_runner: guard IC_max.index access against scalar (AttributeError)
- predix_parallel: close log file handle on Popen failure
- predix_rebacktest_strategies: replace 4 bare except: with except Exception:
2026-05-03 09:37:00 +02:00
TPTBusiness 571c902c2d test: add regression tests for background task path and env bugs
- Verify parallel runner project_root is repo root, not scripts/
- Verify .env loading from correct path
- Verify API key distribution (single key, multi-key comma-separated)
- Verify CLI project_root depth (3 .parent hops, not 4)
- Verify start_loop uses sys.executable and child_proc, not pkill
- Verify parallel_cli does not hardcode model=local
- Verify all referenced scripts exist at resolved paths
2026-05-03 08:57:56 +02:00
TPTBusiness 0a47a667e4 fix: correct project root paths and subprocess handling in parallel runner and CLI
- predix_parallel.py: fix project_root from scripts/ to repo root (parent.parent)
- predix_parallel.py: fix .env loading path and API key distribution logic
- cli.py: fix project_root depth from 4 to 3 .parent hops (7 locations)
- cli.py start_loop: use sys.executable instead of hardcoded python
- cli.py start_loop: replace broad pkill with targeted child process management
- cli.py parallel: remove hardcoded model=local
2026-05-03 08:49:18 +02:00
TPTBusiness 8a945b7ce0 fix: also catch ValueError in mean_variance for dimension mismatch 2026-05-03 00:39:22 +02:00
TPTBusiness 8f27854898 test: add direct unit tests for _apply_ftmo_mask, safe_resolve_path, import_class, and _add_column_if_not_exists 2026-05-03 00:35:57 +02:00
TPTBusiness dcd4697b75 fix: filter NaN in max(), remove redundant ternary, handle non-finite vbt results 2026-05-03 00:25:58 +02:00
TPTBusiness bc15434e02 fix: fix type annotation, remove unused parameter, improve import_class errors 2026-05-03 00:22:16 +02:00
TPTBusiness d44dcb7111 fix: close log file handle, fix FTMO equity double-count, remove bare except 2026-05-03 00:17:02 +02:00
TPTBusiness 15084f593c fix: resolve dead code, shell injection risk, mutable defaults, and other bugs
- strategy_orchestrator.py: remove unreachable dead 'if not factor_values' after early return
- strategy_orchestrator.py: eliminate duplicate OHLVC load in evaluate_strategy
- env.py: escape single-quotes in Docker entry to prevent shell injection (CWE-78)
- env.py: replace mutable default args with None pattern in DockerEnv subclasses
- factor_runner.py: move pandarallel.initialize() from import-time to lazy init
2026-05-02 23:21:38 +02:00
TPTBusiness 20428f7d91 fix: resolve unbound variable, logger shadowing, withdraw_loop edge case, and other bugs in main scripts
- quant.py: guard against empty orch_factors, move strategy_name before try block
- quant_proposal.py: fix __init__ return type Tuple[dict,bool] -> None
- strategy_orchestrator.py: remove dead rdagent_logger import shadowed by getLogger
- factor.py: replace unusual 'not x is None' with idiomatic 'x is not None'
- workflow/loop.py: withdraw_loop(0) raises RuntimeError instead of looking for folder -1
- workflow/tracking.py: replace crash-prone AssertionError with logger.warning + skip
- factor_from_report.py: fix misleading comment about loop_n/step_n dual use
2026-05-02 22:56:29 +02:00
github-actions[bot] 7d97d84100 chore(master): release 1.4.0 (#46)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-01 15:59:06 +02:00
TPTBusiness 9bc525a264 feat(optimizer): add max_positions parameter to Optuna search space
Add max_positions (1-5) as an optimizable hyperparameter across all
three Optuna search stages (coarse, fine, very fine). The parameter
scales effective position size as min(position_size_pct × max_positions,
1.0), allowing the optimizer to discover pyramiding strategies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 15:58:01 +02:00
github-actions[bot] 7cb6531c2a chore(master): release 1.3.11 (#45)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-01 13:58:55 +02:00
TPTBusiness 44c8af572e fix(ci): lazy import logger in predix.py and cli.py to avoid ImportError in test env
Wrapped  in try/except
ImportError with standard logging fallback. The rdagent.log
module chain fails when predix.py is imported as a module
in the CI test environment (kronos CLI tests).
2026-05-01 13:58:16 +02:00
github-actions[bot] b53749df7d chore(master): release 1.3.10 (#44)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-01 13:50:38 +02:00
TPTBusiness 3a1a3d5f77 fix(security): replace remaining assert statements with proper error handling
Replaced 53 assert statements across 22 files with proper
if/raise patterns (TypeError, ValueError, AssertionError)
to resolve Bandit B101 alerts.
2026-05-01 13:49:58 +02:00
54 changed files with 3408 additions and 766 deletions
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "1.3.9"
".": "1.4.2"
}
+42
View File
@@ -1,5 +1,47 @@
# Changelog
## [1.4.2](https://github.com/TPTBusiness/Predix/compare/v1.4.1...v1.4.2) (2026-05-03)
### Bug Fixes
* add missing sys import and fix undefined acc_rate in factor eval ([c45f990](https://github.com/TPTBusiness/Predix/commit/c45f9908ee321400f0a19c57f1482e4cd1394a50))
## [1.4.1](https://github.com/TPTBusiness/Predix/compare/v1.4.0...v1.4.1) (2026-05-03)
### Bug Fixes
* 15 bug fixes across orchestrator, runner, backtest, and infrastructure ([163687d](https://github.com/TPTBusiness/Predix/commit/163687d7e1c278a085d7052a3f958a3edb501e77))
* also catch ValueError in mean_variance for dimension mismatch ([ed73b72](https://github.com/TPTBusiness/Predix/commit/ed73b7253f7dc6459ee30dd81a1ce1194e46e9af))
* close log file handle, fix FTMO equity double-count, remove bare except ([76219a5](https://github.com/TPTBusiness/Predix/commit/76219a53efddaafc2b8bd48a0f76c1d4325e6ea5))
* correct project root paths and subprocess handling in parallel runner and CLI ([9735e3a](https://github.com/TPTBusiness/Predix/commit/9735e3a4d8f01e7b16fb9b185a002396a915cea4))
* filter NaN in max(), remove redundant ternary, handle non-finite vbt results ([f89fbb3](https://github.com/TPTBusiness/Predix/commit/f89fbb3421faf6ccdc8e68a911fd9db2c166120f))
* fix type annotation, remove unused parameter, improve import_class errors ([8b6ab73](https://github.com/TPTBusiness/Predix/commit/8b6ab735c05629bf6b76ddc2fd8b15617600cad7))
* resolve dead code, shell injection risk, mutable defaults, and other bugs ([afff262](https://github.com/TPTBusiness/Predix/commit/afff26287f7c4df7ddfde4e816d280fe845e11eb))
* resolve unbound variable, logger shadowing, withdraw_loop edge case, and other bugs in main scripts ([748cf9b](https://github.com/TPTBusiness/Predix/commit/748cf9b214a3e8447f1289fc4cf1e92ad6cc2f1a))
## [1.4.0](https://github.com/TPTBusiness/Predix/compare/v1.3.11...v1.4.0) (2026-05-01)
### Features
* **optimizer:** add max_positions parameter to Optuna search space ([fdb4be3](https://github.com/TPTBusiness/Predix/commit/fdb4be3b3ebd93325e7821f4251148424184a40d))
## [1.3.11](https://github.com/TPTBusiness/Predix/compare/v1.3.10...v1.3.11) (2026-05-01)
### Bug Fixes
* **ci:** lazy import logger in predix.py and cli.py to avoid ImportError in test env ([60763e8](https://github.com/TPTBusiness/Predix/commit/60763e8eae34f41865ba8e5e65bdfde13b564b4b))
## [1.3.10](https://github.com/TPTBusiness/Predix/compare/v1.3.9...v1.3.10) (2026-05-01)
### Bug Fixes
* **security:** replace remaining assert statements with proper error handling ([928533d](https://github.com/TPTBusiness/Predix/commit/928533d9a81bd5062f07458fbf94d3c7fe347775))
## [1.3.9](https://github.com/TPTBusiness/Predix/compare/v1.3.8...v1.3.9) (2026-05-01)
+112 -96
View File
@@ -13,12 +13,18 @@ import sys
from pathlib import Path
from dotenv import load_dotenv
load_dotenv(Path(__file__).parent / ".env")
import typer
from rich.console import Console
from rdagent.utils.env import logger
try:
from rdagent.utils.env import logger
except ImportError:
import logging
logger = logging.getLogger(__name__)
app = typer.Typer(help="Predix - AI Quantitative Trading Agent")
console = Console()
@@ -109,7 +115,7 @@ def _ensure_kronos_factor_in_pool(con) -> None:
color = "green" if abs(ic) > 0.01 else "yellow"
con.print(
f" [bold {color}]Kronos Factor ready:[/bold {color}] IC={ic:.4f}, "
f"Hit-Rate={hit_rate:.1%} — added to strategy pool"
f"Hit-Rate={hit_rate:.1%} — added to strategy pool",
)
except Exception as e:
@@ -196,9 +202,9 @@ def quant(
predix health - Check system health and configuration
"""
import subprocess
import sys
import threading
import time
import sys
# ---- Parallel Run Isolation ----
# When run_id > 0, isolate all outputs (logs, results, workspace)
@@ -221,10 +227,9 @@ def quant(
console.print(f" [dim]Log: {log_file}[/dim]")
console.print(f" [dim]Results: results/runs/run{run_id}/[/dim]")
console.print(f" [dim]Workspace: {workspace_dir.name}/[/dim]")
else:
# Single run mode: default log file
if log_file is None:
log_file = "fin_quant.log"
# Single run mode: default log file
elif log_file is None:
log_file = "fin_quant.log"
# ---- Log File Setup (daily-rotated) ----
from datetime import datetime as _dt
@@ -232,10 +237,14 @@ def quant(
_daily_dir = Path(__file__).parent / "logs" / _today
_daily_dir.mkdir(parents=True, exist_ok=True)
_log_f = None
_orig_stdout = sys.stdout
_orig_stderr = sys.stderr
if log_file.lower() != "none":
log_path = _daily_dir / log_file
# Open log file for appending (raw stdout/stderr capture)
log_f = open(log_path, "a", encoding="utf-8")
_log_f = open(log_path, "a", encoding="utf-8")
# Redirect stdout and stderr to both console and log file
class TeeWriter:
@@ -247,18 +256,18 @@ def quant(
try:
s.write(data)
s.flush()
except:
except Exception:
pass
def flush(self):
for s in self._streams:
try:
s.flush()
except:
except Exception:
pass
sys.stdout = TeeWriter(sys.__stdout__, log_f)
sys.stderr = TeeWriter(sys.__stderr__, log_f)
sys.stdout = TeeWriter(_orig_stdout, _log_f)
sys.stderr = TeeWriter(_orig_stderr, _log_f)
console.print(f"\n[dim]📝 Logging to: logs/{_today}/{log_file}[/dim]")
else:
@@ -271,7 +280,7 @@ def quant(
if not api_key:
console.print("\n[bold red]❌ OPENROUTER_API_KEY not set in .env[/bold red]")
console.print("[yellow]Add your API key to .env:[/yellow]")
console.print(' OPENROUTER_API_KEY=sk-or-your-key-here')
console.print(" OPENROUTER_API_KEY=sk-or-your-key-here")
raise typer.Exit(code=1)
# Setup both API keys for load balancing
@@ -284,7 +293,7 @@ def quant(
os.environ["LITELLM_PARALLEL_CALLS"] = "2"
console.print(f"\n[bold blue]🌐 Using OpenRouter (2 API Keys):[/bold blue] [cyan]{os.environ['CHAT_MODEL']}[/cyan]")
console.print(f" [dim]Keys: {api_key[:15]}*** + {api_key_2[:15]}***[/dim]")
console.print(f" [dim]Parallel: 2 concurrent requests[/dim]")
console.print(" [dim]Parallel: 2 concurrent requests[/dim]")
else:
os.environ["OPENAI_API_KEY"] = api_key
console.print(f"\n[bold blue]🌐 Using OpenRouter:[/bold blue] [cyan]{os.environ['CHAT_MODEL']}[/cyan]")
@@ -302,7 +311,7 @@ def quant(
# ---- Dashboards ----
if dashboard:
def start_web_dashboard():
console.print(f"\n[bold green]🚀 Web Dashboard: http://localhost:5000[/bold green]")
console.print("\n[bold green]🚀 Web Dashboard: http://localhost:5000[/bold green]")
subprocess.run(
["python", "web/dashboard_api.py"],
cwd=str(Path(__file__).parent),
@@ -327,7 +336,7 @@ def quant(
from rdagent.app.qlib_rd_loop.quant import main as fin_quant
from rdagent.log.daily_log import session as _daily_session
console.print(f"\n[bold cyan]📊 Starting EURUSD Trading Loop...[/bold cyan]\n")
console.print("\n[bold cyan]📊 Starting EURUSD Trading Loop...[/bold cyan]\n")
_ctx = {"model": model}
if run_id:
@@ -337,11 +346,17 @@ def quant(
if step_n:
_ctx["steps"] = step_n
with _daily_session("fin_quant", **_ctx):
fin_quant(
step_n=step_n,
loop_n=loop_n,
)
try:
with _daily_session("fin_quant", **_ctx):
fin_quant(
step_n=step_n,
loop_n=loop_n,
)
finally:
if _log_f is not None:
sys.stdout = _orig_stdout
sys.stderr = _orig_stderr
_log_f.close()
@app.command()
@@ -410,8 +425,8 @@ def evaluate(
predix portfolio - Select a diversified portfolio of uncorrelated factors
predix quant - Generate new factors via LLM trading loop
"""
from rich.panel import Panel
from rdagent.log.daily_log import session as _daily_session
from rich.panel import Panel
console.print(Panel(
"[bold cyan]📊 Predix Factor Evaluator[/bold cyan]\n"
@@ -491,11 +506,12 @@ def top(
predix portfolio - Select diversified portfolio from top factors
predix build-strategies - Combine factors into trading strategies
"""
import json
import glob as glob_module
import json
import numpy as np
from rich.table import Table
from rich.panel import Panel
from rich.table import Table
factors_dir = Path(__file__).parent / "results" / "factors"
if not factors_dir.exists():
@@ -560,9 +576,11 @@ def top(
console.print(table)
# Summary
valid_ic = [r.get("ic") for r in results if r.get("ic") is not None]
valid_sharpe = [r.get("sharpe") for r in results if r.get("sharpe") is not None]
# Summary — filter None, NaN, and non-numeric values
valid_ic = [v for v in (r.get("ic") for r in results)
if isinstance(v, (int, float)) and v is not None and not np.isnan(v)]
valid_sharpe = [v for v in (r.get("sharpe") for r in results)
if isinstance(v, (int, float)) and v is not None and not np.isnan(v)]
# Filter extreme outliers for average
valid_sharpe_filtered = [s for s in valid_sharpe if abs(s or 0) < 1e6]
@@ -637,16 +655,16 @@ def portfolio(
predix top - View top factors before portfolio selection
predix build-strategies - Build strategies from selected factors
"""
import json
import glob as glob_module
import json
import shutil
import subprocess
import tempfile
import shutil
import numpy as np
import pandas as pd
from rich.table import Table
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn, TimeElapsedColumn
from rich.progress import BarColumn, Progress, SpinnerColumn, TaskProgressColumn, TextColumn, TimeElapsedColumn
from rich.table import Table
factors_dir = Path(__file__).parent / "results" / "factors"
if not factors_dir.exists():
@@ -678,12 +696,12 @@ def portfolio(
# 2. Evaluate candidates to get time-series values for correlation
# We need to run the factor code to get the series of values.
# We do this sequentially to avoid OOM.
# Locate data file
data_file = Path(__file__).parent / "git_ignore_folder" / "factor_implementation_source_data" / "intraday_pv.h5"
if not data_file.exists():
data_file = Path(__file__).parent / "git_ignore_folder" / "factor_implementation_source_data_debug" / "intraday_pv.h5"
if not data_file.exists():
console.print("[red]Source data file (intraday_pv.h5) not found.[/red]")
return
@@ -700,11 +718,11 @@ def portfolio(
console=console,
) as progress:
task = progress.add_task(f"Computing values for {len(candidates)} factors...", total=len(candidates))
for cand in candidates:
fname = cand.get("factor_name", "unknown")
fcode = cand.get("factor_code", "")
if not fcode:
errors.append((fname, "No code in JSON"))
progress.advance(task)
@@ -720,10 +738,10 @@ def portfolio(
# If symlink fails, copy the file
import shutil
shutil.copy(str(data_file), str(tmp_path / "intraday_pv.h5"))
# Write code
(tmp_path / "factor.py").write_text(fcode)
try:
# Run factor
result = subprocess.run(
@@ -731,16 +749,16 @@ def portfolio(
cwd=tmp_path,
capture_output=True,
text=True,
timeout=120 # 2 min timeout per factor
timeout=120, # 2 min timeout per factor
)
# Read result
res_file = tmp_path / "result.h5"
if res_file.exists():
df = pd.read_hdf(str(res_file), key="data")
# Get the series (first column)
series = df.iloc[:, 0]
# Count non-NaN values
non_nan = series.count()
if non_nan < 1000:
@@ -760,7 +778,7 @@ def portfolio(
except Exception as e:
errors.append((fname, str(e)[:100]))
progress.update(task, description=f"{fname} ❌ (Error)")
progress.advance(task)
# Show summary of errors
@@ -774,7 +792,7 @@ def portfolio(
if len(factor_series) < 3:
console.print("[red]Not enough valid factor series to build portfolio (need at least 3).[/red]")
console.print("[yellow]Tip: Factors might be producing mostly NaN values or failing execution.[/yellow]")
# Fallback: Show top factors by IC without diversification
console.print("\n[dim]Showing top factors by IC instead:[/dim]")
table = Table(
@@ -792,50 +810,49 @@ def portfolio(
str(i),
cand.get("factor_name", "unknown")[:38],
f"{cand.get('ic', 0):.6f}",
f"{cand.get('sharpe', 0):.4f}" if cand.get('sharpe') else "N/A",
f"{cand.get('sharpe', 0):.4f}" if cand.get("sharpe") else "N/A",
)
console.print(table)
return
# 3. Build Correlation Matrix
console.print(f"\n[dim]Building correlation matrix from {len(factor_series)} factors...[/dim]")
# Align indices and drop NaN
combined = pd.DataFrame(factor_series).dropna()
if combined.empty or len(combined) < 100:
console.print("[red]Not enough valid overlapping data to compute correlation.[/red]")
console.print("[dim]This means the factors produce values at different times or have too many NaN values.[/dim]")
return
corr_matrix = combined.corr().fillna(0)
ic_map = {cand['factor_name']: cand.get('ic', 0) for cand in candidates}
ic_map = {cand["factor_name"]: cand.get("ic", 0) for cand in candidates}
# 4. Greedy Selection
selected = []
remaining = list(corr_matrix.columns)
# Sort remaining by IC to prioritize high IC factors
remaining.sort(key=lambda x: abs(ic_map.get(x, 0)), reverse=True)
for factor in remaining:
if len(selected) >= target:
break
# If it's the first one, just take it
if not selected:
selected.append(factor)
continue
# Check correlation with already selected
# We want max(|corr|) < max_corr
max_c = 0
for sel in selected:
c = abs(corr_matrix.loc[factor, sel])
if c > max_c:
max_c = c
max_c = max(max_c, c)
if max_c < max_corr:
selected.append(factor)
@@ -853,23 +870,23 @@ def portfolio(
for i, fname in enumerate(selected, 1):
# Find original data for display
data = next((c for c in candidates if c['factor_name'] == fname), {})
ic = data.get('ic')
sharpe = data.get('sharpe')
data = next((c for c in candidates if c["factor_name"] == fname), {})
ic = data.get("ic")
sharpe = data.get("sharpe")
# Calculate max corr with other selected factors
max_c_val = 0
for s in selected:
if s != fname:
val = abs(corr_matrix.loc[fname, s])
if val > max_c_val: max_c_val = val
max_c_val = max(max_c_val, val)
table.add_row(
str(i),
fname[:38],
f"{ic:.6f}" if ic is not None else "N/A",
f"{sharpe:.4f}" if sharpe is not None else "N/A",
f"{max_c_val:.4f}" if max_c_val > 0 else "-"
f"{max_c_val:.4f}" if max_c_val > 0 else "-",
)
console.print(table)
@@ -879,20 +896,20 @@ def portfolio(
"selected_factors": selected,
"max_correlation": max_corr,
"pool_size": top,
"timestamp": pd.Timestamp.now().isoformat()
"timestamp": pd.Timestamp.now().isoformat(),
}
out_dir = Path(__file__).parent / "results" / "portfolio"
out_dir.mkdir(parents=True, exist_ok=True)
out_file = out_dir / "selected_factors.json"
with open(out_file, "w") as f:
json.dump(portfolio_data, f, indent=2)
console.print(Panel(
f"[bold]Portfolio saved to results/portfolio/selected_factors.json[/bold]\n"
f"Selected {len(selected)} unique factors from {top} candidates.",
border_style="green"
border_style="green",
))
@@ -938,13 +955,12 @@ def portfolio_simple(
predix top - View top factors before portfolio selection
predix build-strategies - Build strategies from selected factors
"""
import json
import glob as glob_module
import re
import numpy as np
import json
import pandas as pd
from rich.table import Table
from rich.panel import Panel
from rich.table import Table
factors_dir = Path(__file__).parent / "results" / "factors"
if not factors_dir.exists():
@@ -988,14 +1004,14 @@ def portfolio_simple(
for cand in candidates:
fname = cand.get("factor_name", "").lower()
assigned = False
# Check each category's keywords
for cat, keywords in categories.items():
if any(kw in fname for kw in keywords):
categorized[cat].append(cand)
assigned = True
break
if not assigned:
categorized["other"].append(cand)
@@ -1006,7 +1022,7 @@ def portfolio_simple(
best = categorized[cat][0] # Already sorted by IC
selected.append({
"factor": best,
"category": cat.capitalize() if cat != "other" else "Other"
"category": cat.capitalize() if cat != "other" else "Other",
})
# 5. Display Results
@@ -1029,7 +1045,7 @@ def portfolio_simple(
cand.get("factor_name", "unknown")[:38],
cat,
f"{cand.get('ic', 0):.6f}",
f"{cand.get('sharpe', 0):.4f}" if cand.get('sharpe') else "N/A",
f"{cand.get('sharpe', 0):.4f}" if cand.get("sharpe") else "N/A",
)
console.print(table)
@@ -1039,7 +1055,7 @@ def portfolio_simple(
"selected_factors": [item["factor"]["factor_name"] for item in selected],
"categories": {item["category"]: item["factor"]["factor_name"] for item in selected},
"method": "simple_keyword_categorization",
"timestamp": str(pd.Timestamp.now().isoformat())
"timestamp": str(pd.Timestamp.now().isoformat()),
}
out_dir = Path(__file__).parent / "results" / "portfolio"
@@ -1052,7 +1068,7 @@ def portfolio_simple(
console.print(Panel(
f"[bold]Simple Portfolio saved to results/portfolio/portfolio_simple.json[/bold]\n"
f"Selected {len(selected)} factors across {len([c for c in categorized if categorized[c]])} categories.",
border_style="green"
border_style="green",
))
@@ -1115,12 +1131,10 @@ def build_strategies(
predix portfolio - Select diversified factors before combining
predix top - View top factors before building strategies
"""
import pandas as pd
import numpy as np
from rich.table import Table
from rich.panel import Panel
from rdagent.scenarios.qlib.developer.strategy_builder import StrategyBuilder
from rich.panel import Panel
from rich.table import Table
console.print(Panel(
"[bold cyan]🏗️ Predix Strategy Builder[/bold cyan]\n"
@@ -1276,9 +1290,10 @@ def build_strategies_ai(
predix quant - Generate new alpha factors via LLM trading loop
predix evaluate - Evaluate factors before strategy building
"""
from rich.panel import Panel
from pathlib import Path
from rich.panel import Panel
console.print(Panel(
"[bold cyan]🧠 StrategyCoSTEER - AI Strategy Builder[/bold cyan]\n"
"Generating trading strategies from existing factors\n"
@@ -1304,7 +1319,7 @@ def build_strategies_ai(
# Setup LLM environment (same as quant command)
api_key = os.getenv("OPENROUTER_API_KEY") or os.getenv("OPENAI_API_KEY", "")
api_key_2 = os.getenv("OPENROUTER_API_KEY_2", "")
if api_key and not api_key.startswith("sk-or-"):
# OPENROUTER_API_KEY not set, try to use what we have
api_key = os.getenv("OPENROUTER_API_KEY", api_key)
@@ -1331,8 +1346,8 @@ def build_strategies_ai(
return
# Load evaluated factors
import json
import glob as glob_module
import json
factors = []
for f in glob_module.glob(str(factors_dir / "*.json")):
@@ -1416,15 +1431,15 @@ def build_strategies_ai(
for i, r in enumerate(results, 1):
# Monthly return: use real backtest if available, else estimate
rb = r.get('real_backtest', {})
if isinstance(rb, dict) and rb.get('status') == 'success':
monthly_pct = rb.get('monthly_return_pct', r.get('monthly_return_pct', 0))
n_trades = rb.get('n_trades', '-')
real_ic = rb.get('ic', 0)
rb = r.get("real_backtest", {})
if isinstance(rb, dict) and rb.get("status") == "success":
monthly_pct = rb.get("monthly_return_pct", r.get("monthly_return_pct", 0))
n_trades = rb.get("n_trades", "-")
real_ic = rb.get("ic", 0)
else:
monthly_pct = r.get('monthly_return_pct', r.get('real_monthly_return', 0))
n_trades = '-'
real_ic = rb.get('ic', 0) if isinstance(rb, dict) else 0
monthly_pct = r.get("monthly_return_pct", r.get("real_monthly_return", 0))
n_trades = "-"
real_ic = rb.get("ic", 0) if isinstance(rb, dict) else 0
table.add_row(
str(i),
@@ -1517,7 +1532,7 @@ def status():
# Process check
result = subprocess.run(
["pgrep", "-f", "fin_quant"],
capture_output=True, text=True
capture_output=True, text=True,
)
if result.returncode == 0:
console.print("[bold green]✅ Trading Loop: RUNNING[/bold green]")
@@ -1535,7 +1550,7 @@ def status():
factors = c.fetchone()[0]
conn.close()
console.print(f"\n📊 Results:")
console.print("\n📊 Results:")
console.print(f" Backtest runs: {runs}")
console.print(f" Factors: {factors}")
@@ -1612,6 +1627,7 @@ def best(
$ predix best -n 50 --export /tmp/top.json
"""
import json
from rich.table import Table
items = _load_strategies()
@@ -1728,7 +1744,7 @@ def kronos_factor(
console.print("Run data conversion first — see README Data Setup section.")
raise typer.Exit(1)
console.print(f"[bold]Kronos Factor Generator[/bold]")
console.print("[bold]Kronos Factor Generator[/bold]")
console.print(f" Context: [cyan]{context}[/cyan] bars | Pred: [cyan]{pred}[/cyan] bars | Device: [cyan]{_device}[/cyan]")
from rdagent.components.coder.kronos_adapter import build_kronos_factor
@@ -1811,7 +1827,7 @@ def kronos_eval(
console.print(f"[red]ERROR: Data not found at {data_path}[/red]")
raise typer.Exit(1)
console.print(f"[bold]Kronos Model Evaluator[/bold] (alongside LightGBM)")
console.print("[bold]Kronos Model Evaluator[/bold] (alongside LightGBM)")
console.print(f" Context: [cyan]{context}[/cyan] bars | Pred: [cyan]{pred}[/cyan] bars | Device: [cyan]{_device}[/cyan]")
console.print(" Running evaluation...")
@@ -1826,12 +1842,12 @@ def kronos_eval(
batch_size=batch_size,
)
console.print(f"\n[bold]Kronos-mini Results[/bold]")
console.print("\n[bold]Kronos-mini Results[/bold]")
console.print(f" Predictions: [cyan]{metrics['n_predictions']}[/cyan]")
console.print(f" IC (mean): [{'green' if metrics['IC_mean'] > 0.02 else 'yellow'}]{metrics['IC_mean']:.4f}[/]")
console.print(f" IC IR: [{'green' if metrics['IC_IR'] > 0.5 else 'yellow'}]{metrics['IC_IR']:.4f}[/] (>0.5 = strong signal)")
console.print(f" Hit Rate: [{'green' if metrics['hit_rate'] > 0.52 else 'yellow'}]{metrics['hit_rate']:.2%}[/] (>50% = directionally useful)")
console.print(f"\n[dim]Reference: LightGBM baseline IC typically 0.010.05 on 1-min EUR/USD[/dim]")
console.print("\n[dim]Reference: LightGBM baseline IC typically 0.010.05 on 1-min EUR/USD[/dim]")
import json as _json
out_dir = Path("results/kronos")
+122 -108
View File
@@ -21,13 +21,17 @@ load_dotenv(".env")
import subprocess
from importlib.resources import path as rpath
from typing import Dict, Optional
from typing import Annotated
import typer
from rich.console import Console
from typing_extensions import Annotated
from rdagent.utils.env import logger
try:
from rdagent.utils.env import logger
except ImportError:
import logging
logger = logging.getLogger(__name__)
from rdagent.app.data_science.loop import main as data_science
from rdagent.app.finetune.llm.loop import main as llm_finetune
@@ -141,10 +145,10 @@ def ds_user_interact(port=19900):
@app.command(name="fin_factor")
def fin_factor_cli(
path: Optional[str] = None,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
all_duration: Optional[str] = None,
path: str | None = None,
step_n: int | None = None,
loop_n: int | None = None,
all_duration: str | None = None,
checkout: CheckoutOption = True,
):
fin_factor(path=path, step_n=step_n, loop_n=loop_n, all_duration=all_duration, checkout=checkout)
@@ -152,10 +156,10 @@ def fin_factor_cli(
@app.command(name="fin_model")
def fin_model_cli(
path: Optional[str] = None,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
all_duration: Optional[str] = None,
path: str | None = None,
step_n: int | None = None,
loop_n: int | None = None,
all_duration: str | None = None,
checkout: CheckoutOption = True,
):
fin_model(path=path, step_n=step_n, loop_n=loop_n, all_duration=all_duration, checkout=checkout)
@@ -163,10 +167,10 @@ def fin_model_cli(
@app.command(name="fin_quant")
def fin_quant_cli(
path: Optional[str] = None,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
all_duration: Optional[str] = None,
path: str | None = None,
step_n: int | None = None,
loop_n: int | None = None,
all_duration: str | None = None,
checkout: CheckoutOption = True,
with_dashboard: bool = typer.Option(False, "--with-dashboard/-d", help="Start web dashboard automatically"),
with_cli_dashboard: bool = typer.Option(False, "--cli-dashboard/-c", help="Show beautiful CLI dashboard"),
@@ -226,7 +230,7 @@ def fin_quant_cli(
if not api_key:
console.print("\n[bold red]❌ OPENROUTER_API_KEY not set in .env[/bold red]")
console.print("[yellow]Add your API key to .env and retry:[/yellow]")
console.print(' OPENROUTER_API_KEY=sk-or-your-key-here')
console.print(" OPENROUTER_API_KEY=sk-or-your-key-here")
raise typer.Exit(code=1)
os.environ["OPENAI_API_KEY"] = api_key
@@ -245,8 +249,8 @@ def fin_quant_cli(
console.print(f" [dim]Base URL: {os.environ['OPENAI_API_BASE']}[/dim]")
# Wait until the llama.cpp server is fully loaded before starting the pipeline
import urllib.request
import urllib.error
import urllib.request
base_url = os.environ["OPENAI_API_BASE"].removesuffix("/v1").rstrip("/")
health_url = f"{base_url}/health"
@@ -280,7 +284,7 @@ def fin_quant_cli(
subprocess.run(
["python", "web/dashboard_api.py"],
cwd=str(Path(__file__).parent.parent.parent),
env={**os.environ, "FLASK_ENV": "development"}
env={**os.environ, "FLASK_ENV": "development"},
)
dashboard_thread = threading.Thread(target=start_web_dashboard, daemon=True)
@@ -322,9 +326,9 @@ def fin_quant_cli(
@app.command(name="fin_factor_report")
def fin_factor_report_cli(
report_folder: Optional[str] = None,
path: Optional[str] = None,
all_duration: Optional[str] = None,
report_folder: str | None = None,
path: str | None = None,
all_duration: str | None = None,
checkout: CheckoutOption = True,
):
fin_factor_report(report_folder=report_folder, path=path, all_duration=all_duration, checkout=checkout)
@@ -337,12 +341,12 @@ def general_model_cli(report_file_path: str):
@app.command(name="data_science")
def data_science_cli(
path: Optional[str] = None,
path: str | None = None,
checkout: CheckoutOption = True,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
timeout: Optional[str] = None,
competition: Optional[str] = None,
step_n: int | None = None,
loop_n: int | None = None,
timeout: str | None = None,
competition: str | None = None,
):
data_science(
path=path,
@@ -356,16 +360,16 @@ def data_science_cli(
@app.command(name="llm_finetune")
def llm_finetune_cli(
path: Optional[str] = None,
path: str | None = None,
checkout: CheckoutOption = True,
benchmark: Optional[str] = None,
benchmark_description: Optional[str] = None,
dataset: Optional[str] = None,
base_model: Optional[str] = None,
upper_data_size_limit: Optional[int] = None,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
timeout: Optional[str] = None,
benchmark: str | None = None,
benchmark_description: str | None = None,
dataset: str | None = None,
base_model: str | None = None,
upper_data_size_limit: int | None = None,
step_n: int | None = None,
loop_n: int | None = None,
timeout: str | None = None,
):
llm_finetune(
path=path,
@@ -431,6 +435,7 @@ def rl_trading_cli(
rdagent rl_trading --mode backtest --no-with-protections
"""
from pathlib import Path
import yaml
console = Console()
@@ -442,18 +447,18 @@ def rl_trading_cli(
with open(config_path) as f:
config = yaml.safe_load(f) or {}
console.print(f"\n[bold blue]🤖 RL Trading Agent[/bold blue]")
console.print("\n[bold blue]🤖 RL Trading Agent[/bold blue]")
console.print(f"Mode: [cyan]{mode}[/cyan]")
console.print(f"Algorithm: [cyan]{algorithm.upper()}[/cyan]")
console.print(f"Protections: {'[green]Enabled[/green]' if with_protections else '[red]Disabled[/red]'}")
try:
from rdagent.components.coder.rl import RLTradingAgent, RLCosteer, TradingEnv
from rdagent.components.coder.rl import RLCosteer, RLTradingAgent, TradingEnv
except ImportError as e:
console.print(f"[bold red]Error: RL components not available.[/bold red]")
console.print("[bold red]Error: RL components not available.[/bold red]")
console.print(f"Details: {e}")
console.print(f"\n[yellow]Install RL dependencies:[/yellow]")
console.print(f" pip install stable-baselines3 gymnasium")
console.print("\n[yellow]Install RL dependencies:[/yellow]")
console.print(" pip install stable-baselines3 gymnasium")
raise typer.Exit(code=1)
if mode == "train":
@@ -469,8 +474,8 @@ def rl_trading_cli(
console.print("[dim]Loading market data...[/dim]")
# TODO: Load actual data from config
# For now, create mock environment
import numpy as np
import gymnasium as gym
import numpy as np
# Create simple mock environment for demonstration
class MockTradingEnv(gym.Env):
@@ -506,7 +511,7 @@ def rl_trading_cli(
model_path_out.parent.mkdir(parents=True, exist_ok=True)
agent.save(model_path_out)
console.print(f"\n[bold green]✅ Training complete![/bold green]")
console.print("\n[bold green]✅ Training complete![/bold green]")
console.print(f"Model saved to: [cyan]{model_path_out}[/cyan]")
console.print(f"Algorithm: {result['algorithm']}")
console.print(f"Timesteps: {result['total_timesteps']:,}")
@@ -532,9 +537,9 @@ def rl_trading_cli(
agent = RLTradingAgent(algorithm=algorithm.upper())
# Run backtest
from rdagent.components.backtesting import FactorBacktester
import pandas as pd
import numpy as np
import pandas as pd
from rdagent.components.backtesting import FactorBacktester
backtester = FactorBacktester()
@@ -543,8 +548,8 @@ def rl_trading_cli(
n_steps = 500
mock_prices = pd.Series(100 + np.cumsum(np.random.randn(n_steps) * 0.5))
mock_indicators = pd.DataFrame({
'rsi': np.random.uniform(30, 70, n_steps),
'macd': np.random.randn(n_steps) * 0.1,
"rsi": np.random.uniform(30, 70, n_steps),
"macd": np.random.randn(n_steps) * 0.1,
})
console.print("[yellow]Running backtest...[/yellow]")
@@ -555,7 +560,7 @@ def rl_trading_cli(
enable_protections=with_protections,
)
console.print(f"\n[bold green]✅ Backtest complete![/bold green]")
console.print("\n[bold green]✅ Backtest complete![/bold green]")
console.print(f" Final Equity: [green]${metrics.get('final_equity', 0):,.2f}[/green]")
console.print(f" Sharpe Ratio: {metrics.get('sharpe_ratio', 0):.3f}")
console.print(f" Max Drawdown: {metrics.get('max_drawdown', 0):.2%}")
@@ -629,7 +634,7 @@ def generate_strategies_cli(
rdagent generate_strategies -n 3 -i 10 --optuna-trials 50 # Deep optimization
"""
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeRemainingColumn
from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn, TimeRemainingColumn
from rich.table import Table
console = Console()
@@ -648,7 +653,7 @@ def generate_strategies_cli(
raise typer.Exit(code=1)
console.print(f"\n[bold blue]{'='*60}[/bold blue]")
console.print(f"[bold blue] PREDIX Strategy Generator[/bold blue]")
console.print("[bold blue] PREDIX Strategy Generator[/bold blue]")
console.print(f"[bold blue]{'='*60}[/bold blue]")
console.print(f" Strategies: [cyan]{count}[/cyan]")
console.print(f" Workers: [cyan]{workers}[/cyan]")
@@ -675,12 +680,12 @@ def generate_strategies_cli(
_slog = _dlog.setup("strategies", **_strat_ctx)
try:
from rdagent.components.coder.strategy_orchestrator import StrategyOrchestrator
import pandas as pd
from rdagent.components.coder.strategy_orchestrator import StrategyOrchestrator
all_results = []
best_strategy = None
best_sharpe = float('-inf')
best_sharpe = float("-inf")
# CONTINUOUS OPTIMIZATION LOOP
for iteration in range(1, max_iterations + 1):
@@ -729,7 +734,7 @@ def generate_strategies_cli(
# Track best strategy
for r in results:
sharpe = r.get("sharpe_ratio", float('-inf'))
sharpe = r.get("sharpe_ratio", float("-inf"))
if sharpe > best_sharpe:
best_sharpe = sharpe
best_strategy = r
@@ -749,7 +754,7 @@ def generate_strategies_cli(
rejected = [r for r in results if r.get("status") == "rejected"]
console.print(f"\n[bold green]{'='*60}[/bold green]")
console.print(f"[bold green] Strategy Generation Summary[/bold green]")
console.print("[bold green] Strategy Generation Summary[/bold green]")
console.print(f"[bold green]{'='*60}[/bold green]")
table = Table(show_header=True, header_style="bold magenta", show_lines=True)
@@ -778,7 +783,7 @@ def generate_strategies_cli(
# Show best strategy details
if best_strategy:
console.print(f"\n[bold gold1]{'='*60}[/bold gold1]")
console.print(f"[bold gold1] BEST STRATEGY[/bold gold1]")
console.print("[bold gold1] BEST STRATEGY[/bold gold1]")
console.print(f"[bold gold1]{'='*60}[/bold gold1]")
console.print(f" Name: [cyan]{best_strategy.get('strategy_name', 'Unknown')}[/cyan]")
console.print(f" Sharpe: [green]{best_strategy.get('sharpe_ratio', 0):.4f}[/green]")
@@ -786,13 +791,13 @@ def generate_strategies_cli(
console.print(f" Max DD: [yellow]{best_strategy.get('max_drawdown', 0):.2%}[/yellow]")
console.print(f" Win Rate: [cyan]{best_strategy.get('win_rate', 0):.2%}[/cyan]")
if best_strategy.get("best_params"):
console.print(f"\n [bold]Optimized Parameters:[/bold]")
console.print("\n [bold]Optimized Parameters:[/bold]")
for param, val in best_strategy["best_params"].items():
console.print(f" {param}: [cyan]{val}[/cyan]")
console.print(f"[bold gold1]{'='*60}[/bold gold1]")
if accepted:
console.print(f"\n[bold]Accepted Strategies:[/bold]")
console.print("\n[bold]Accepted Strategies:[/bold]")
acc_table = Table(show_header=True, header_style="bold cyan")
acc_table.add_column("#", width=4)
acc_table.add_column("Strategy", width=30)
@@ -815,13 +820,13 @@ def generate_strategies_cli(
)
console.print(acc_table)
console.print(f"\n[bold green]Strategies saved to:[/bold green] [cyan]results/strategies_new/[/cyan]")
console.print("\n[bold green]Strategies saved to:[/bold green] [cyan]results/strategies_new/[/cyan]")
console.print(f"[bold blue]{'='*60}[/bold blue]\n")
_slog.success(f"Generated {len(all_results)} strategies ({len([r for r in all_results if r.get('status')=='accepted'])} accepted)")
except ImportError as e:
_slog.error(f"Strategy components not available: {e}")
console.print(f"[bold red]Error: Strategy components not available.[/bold red]")
console.print("[bold red]Error: Strategy components not available.[/bold red]")
console.print(f"Details: {e}")
raise typer.Exit(code=1)
except Exception as e:
@@ -857,17 +862,18 @@ def optimize_portfolio_cli(
raise typer.Exit(code=1)
console.print(f"\n[bold blue]{'='*60}[/bold blue]")
console.print(f"[bold blue] PREDIX Portfolio Optimizer[/bold blue]")
console.print("[bold blue] PREDIX Portfolio Optimizer[/bold blue]")
console.print(f"[bold blue]{'='*60}[/bold blue]")
console.print(f" Top N: [cyan]{top_n}[/cyan]")
console.print(f" Method: [cyan]{method}[/cyan]")
console.print(f"[bold blue]{'='*60}[/bold blue]\n")
try:
from rdagent.components.backtesting.risk_management import PortfolioOptimizer
import json
from pathlib import Path
from rdagent.components.backtesting.risk_management import PortfolioOptimizer
project_root = Path(__file__).parent.parent.parent
strategies_dir = project_root / "results" / "strategies_new"
@@ -1004,14 +1010,15 @@ def strategies_report_cli(
rdagent strategies_report -s path/to/strategy.json # Single strategy
rdagent strategies_report -o custom/reports/ # Custom output dir
"""
from pathlib import Path
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn
from pathlib import Path
console = Console()
console.print(f"\n[bold blue]{'='*60}[/bold blue]")
console.print(f"[bold blue] PREDIX Strategy Report Generator[/bold blue]")
console.print("[bold blue] PREDIX Strategy Report Generator[/bold blue]")
console.print(f"[bold blue]{'='*60}[/bold blue]\n")
project_root = Path(__file__).parent.parent.parent
@@ -1063,20 +1070,20 @@ def strategies_report_cli(
progress.update(task, completed=1)
console.print(f"\n[bold green]{'='*60}[/bold green]")
console.print(f"[bold green] Report Generation Complete[/bold green]")
console.print("[bold green] Report Generation Complete[/bold green]")
console.print(f"[bold green]{'='*60}[/bold green]")
console.print(f" Reports generated: [cyan]{reports_generated}/{len(strategy_files)}[/cyan]")
console.print(f" Output directory: [cyan]{output_dir_path}[/cyan]")
console.print(f"[bold green]{'='*60}[/bold green]\n")
def _generate_single_strategy_report(strategy_file: Path, output_dir: Path) -> Dict:
def _generate_single_strategy_report(strategy_file: Path, output_dir: Path) -> dict:
"""Generate a report for a single strategy."""
import json
import matplotlib
matplotlib.use("Agg") # Non-interactive backend
import matplotlib.pyplot as plt
import seaborn as sns
with open(strategy_file, encoding="utf-8") as f:
strategy = json.load(f)
@@ -1151,7 +1158,7 @@ if __name__ == "__main__":
@app.command(name="start_llama")
def start_llama_cli(
model: str = typer.Option(
None, "--model", "-m", help="Path to model file"
None, "--model", "-m", help="Path to model file",
),
port: int = typer.Option(8081, "--port", "-p", help="Server port"),
gpu_layers: int = typer.Option(30, "--gpu-layers", "-g", help="GPU layers"),
@@ -1173,8 +1180,6 @@ def start_llama_cli(
rdagent start_llama --gpu-layers 40 --ctx-size 4096
rdagent start_llama --reasoning
"""
import subprocess
import sys
import os
model_path = model or os.getenv(
@@ -1211,7 +1216,7 @@ def start_llama_cli(
if not reasoning:
cmd.extend(["--reasoning", "off"])
print(f"🚀 Starting llama.cpp server...")
print("🚀 Starting llama.cpp server...")
print(f" Model: {Path(model_path).name}")
print(f" Port: {port}")
print(f" GPU Layers: {gpu_layers}")
@@ -1244,15 +1249,14 @@ def start_loop_cli(
rdagent start_loop
rdagent start_loop --target 5 --max-wait 3600
"""
import subprocess
import signal
import sys
import os
from datetime import datetime
import signal
import subprocess
import time
from datetime import datetime
script_dir = str(Path(__file__).parent.parent.parent.parent)
generator = f"python {script_dir}/scripts/predix_smart_strategy_gen.py"
script_dir = str(Path(__file__).parent.parent.parent)
generator = [sys.executable, f"{script_dir}/scripts/predix_smart_strategy_gen.py"]
logfile = f"{script_dir}/results/logs/generator_loop.log"
pidfile = "/tmp/predix_loop.pid" # nosec B108 — administrative PID file, single-process daemon
@@ -1265,12 +1269,19 @@ def start_loop_cli(
with open(logfile, "a") as f:
f.write(line + "\n")
child_proc = None # track current child PID for targeted cleanup
def cleanup(signum=None, frame=None):
log("Received termination signal. Cleaning up...")
try:
subprocess.run(["pkill", "-f", "predix_smart_strategy_gen.py"], capture_output=True)
except Exception:
pass
if child_proc is not None:
try:
child_proc.terminate()
child_proc.wait(timeout=10)
except Exception:
try:
child_proc.kill()
except Exception:
pass
try:
os.remove(pidfile)
except FileNotFoundError:
@@ -1316,26 +1327,32 @@ def start_loop_cli(
strat_count = len(list(strat_dir.glob("*.json"))) if strat_dir.exists() else 0
log(f"📁 Existing strategies: {strat_count}")
# Kill stale processes
try:
subprocess.run(["pkill", "-9", "-f", "predix_smart_strategy_gen.py"], capture_output=True)
except Exception:
pass
time.sleep(2)
# Kill stale child from previous iteration
if child_proc is not None:
try:
child_proc.terminate()
child_proc.wait(timeout=10)
except subprocess.TimeoutExpired:
child_proc.kill()
child_proc.wait()
except Exception:
pass
child_proc = None
time.sleep(2)
# Start generator
log("🤖 Starting generator...")
proc = subprocess.Popen(
generator.split(),
child_proc = subprocess.Popen(
generator,
cwd=script_dir,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
log(f" PID: {proc.pid}")
log(f" PID: {child_proc.pid}")
# Monitor progress
elapsed = 0
while proc.poll() is None:
while child_proc.poll() is None:
time.sleep(30)
elapsed += 30
@@ -1344,11 +1361,12 @@ def start_loop_cli(
if elapsed >= max_wait:
log(f" ⏰ Timeout after {elapsed}s. Killing...")
proc.kill()
child_proc.kill()
break
# Check results
exit_code = proc.wait()
exit_code = child_proc.wait()
child_proc = None
if exit_code == 0:
log("✅ Generator completed successfully")
elif exit_code == -9:
@@ -1389,24 +1407,24 @@ def parallel_cli(
rdagent parallel -n 10 -k 2
"""
import subprocess
import sys
from pathlib import Path
from rdagent.log import daily_log as _dlog
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_parallel.py"
if not script.exists():
typer.echo(f"❌ Script not found: {script}")
raise typer.Exit(code=1)
cmd = [sys.executable, str(script), "--runs", str(runs), "--api-keys", str(api_keys), "-m", "local"]
cmd = [sys.executable, str(script), "--runs", str(runs), "--api-keys", str(api_keys)]
_plog = _dlog.setup("parallel", runs=runs, api_keys=api_keys, model="local")
typer.echo(f"🚀 Starting {runs} parallel runs...")
typer.echo(f" Script: {script}")
typer.echo(f" API Keys: {api_keys}")
typer.echo(f" Model: local (llama.cpp)")
typer.echo(" Model: local (llama.cpp)")
try:
result = subprocess.run(cmd, cwd=str(project_root))
@@ -1440,11 +1458,11 @@ def eval_all_cli(
rdagent eval_all -n 500 -p 8
"""
import subprocess
import sys
from pathlib import Path
from rdagent.log import daily_log as _dlog
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_full_eval.py"
if not script.exists():
@@ -1495,10 +1513,9 @@ def batch_backtest_cli(
rdagent batch_backtest --all
"""
import subprocess
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_batch_backtest.py"
if not script.exists():
@@ -1548,10 +1565,9 @@ def simple_eval_cli(
rdagent simple_eval --all
"""
import subprocess
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_simple_eval.py"
if not script.exists():
@@ -1581,7 +1597,7 @@ def simple_eval_cli(
@app.command(name="rebacktest")
def rebacktest_cli(
strategies_dir: str = typer.Option(
None, "--strategies-dir", "-d", help="Directory containing strategy JSON files"
None, "--strategies-dir", "-d", help="Directory containing strategy JSON files",
),
):
"""
@@ -1595,10 +1611,9 @@ def rebacktest_cli(
rdagent rebacktest -d results/strategies_new/
"""
import subprocess
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_rebacktest_strategies.py"
if not script.exists():
@@ -1623,10 +1638,10 @@ def rebacktest_cli(
@app.command(name="report")
def report_cli(
strategy_path: str = typer.Option(
None, "--strategy", "-s", help="Path to single strategy JSON (default: all strategies)"
None, "--strategy", "-s", help="Path to single strategy JSON (default: all strategies)",
),
output: str = typer.Option(
None, "--output", "-o", help="Output directory (default: results/strategy_reports/)"
None, "--output", "-o", help="Output directory (default: results/strategy_reports/)",
),
):
"""
@@ -1649,10 +1664,9 @@ def report_cli(
rdagent report -o custom/reports/
"""
import subprocess
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent.parent
project_root = Path(__file__).parent.parent.parent
script = project_root / "scripts" / "predix_strategy_report.py"
if not script.exists():
+2 -3
View File
@@ -201,6 +201,5 @@ class DataScienceBasePropSetting(KaggleBasePropSetting):
DS_RD_SETTING = DataScienceBasePropSetting()
# enable_cross_trace_diversity and llm_select_hypothesis should not be true at the same time
assert not (
DS_RD_SETTING.enable_cross_trace_diversity and DS_RD_SETTING.llm_select_hypothesis
), "enable_cross_trace_diversity and llm_select_hypothesis cannot be true at the same time"
if DS_RD_SETTING.enable_cross_trace_diversity and DS_RD_SETTING.llm_select_hypothesis:
raise ValueError("enable_cross_trace_diversity and llm_select_hypothesis cannot be true at the same time")
+8 -9
View File
@@ -58,18 +58,18 @@ def main(
if user_target_scenario:
FT_RD_SETTING.user_target_scenario = user_target_scenario
assert (
FT_RD_SETTING.user_target_scenario is None
), "user_target_scenario is not yet supported, please specify via benchmark and benchmark_description"
if FT_RD_SETTING.user_target_scenario is not None:
raise ValueError("user_target_scenario is not yet supported, please specify via benchmark and benchmark_description")
if upper_data_size_limit:
FT_RD_SETTING.upper_data_size_limit = upper_data_size_limit
logger.info(f"Set upper_data_size_limit to {FT_RD_SETTING.upper_data_size_limit}")
if benchmark and benchmark_description:
FT_RD_SETTING.target_benchmark = benchmark
FT_RD_SETTING.benchmark_description = benchmark_description
assert FT_RD_SETTING.user_target_scenario or (
FT_RD_SETTING.target_benchmark and FT_RD_SETTING.benchmark_description
), "Either user_target_scenario or target_benchmark must be specified for LLM fine-tuning."
if not (
FT_RD_SETTING.user_target_scenario or (FT_RD_SETTING.target_benchmark and FT_RD_SETTING.benchmark_description)
):
raise ValueError("Either user_target_scenario or target_benchmark must be specified for LLM fine-tuning.")
# Update configuration with provided parameters
if dataset:
@@ -82,9 +82,8 @@ def main(
model_target = FT_RD_SETTING.base_model if FT_RD_SETTING.base_model else "auto selected model"
# Temporary assertion until auto-selection is implemented
assert (
FT_RD_SETTING.base_model is not None
), "Base model auto selection not yet supported, please specify via --base-model"
if FT_RD_SETTING.base_model is None:
raise ValueError("Base model auto selection not yet supported, please specify via --base-model")
logger.info(f"Starting LLM fine-tuning on dataset='{data_set_target}' with model='{model_target}'")
+8 -9
View File
@@ -4,10 +4,9 @@ Factor workflow with session control
import asyncio
from pathlib import Path
from typing import Any, Optional
from typing import Any
import fire
from rdagent.app.qlib_rd_loop.conf import FACTOR_PROP_SETTING
from rdagent.components.workflow.rd_loop import RDLoop
from rdagent.core.exception import CoderError, FactorEmptyError
@@ -21,20 +20,20 @@ class FactorRDLoop(RDLoop):
def running(self, prev_out: dict[str, Any]):
exp = self.runner.develop(prev_out["coding"])
if exp is None:
logger.error(f"Factor extraction failed.")
logger.error("Factor extraction failed.")
raise FactorEmptyError("Factor extraction failed.")
logger.log_object(exp, tag="runner result")
return exp
def main(
path: Optional[str] = None,
step_n: Optional[int] = None,
loop_n: Optional[int] = None,
path: str | None = None,
step_n: int | None = None,
loop_n: int | None = None,
all_duration: str | None = None,
checkout: bool = True,
checkout_path: Optional[str] = None,
base_features_path: Optional[str] = None,
checkout_path: str | None = None,
base_features_path: str | None = None,
**kwargs,
):
"""
@@ -47,7 +46,7 @@ def main(
dotenv run -- python rdagent/app/qlib_rd_loop/factor.py $LOG_PATH/__session__/1/0_propose --step_n 1 # `step_n` is a optional paramter
"""
if not checkout_path is None:
if checkout_path is not None:
checkout = Path(checkout_path)
if path is None:
@@ -1,10 +1,9 @@
import asyncio
import json
from pathlib import Path
from typing import Any, Dict, Tuple
from typing import Any
import fire
from rdagent.app.qlib_rd_loop.conf import FACTOR_FROM_REPORT_PROP_SETTING
from rdagent.app.qlib_rd_loop.factor import FactorRDLoop
from rdagent.components.document_reader.document_reader import (
@@ -12,7 +11,7 @@ from rdagent.components.document_reader.document_reader import (
load_and_process_pdfs_by_langchain,
)
from rdagent.core.conf import RD_AGENT_SETTINGS
from rdagent.core.proposal import Hypothesis, HypothesisFeedback
from rdagent.core.proposal import Hypothesis
from rdagent.log import rdagent_logger as logger
from rdagent.oai.llm_utils import APIBackend
from rdagent.scenarios.qlib.experiment.factor_experiment import QlibFactorExperiment
@@ -36,14 +35,14 @@ def generate_hypothesis(factor_result: dict, report_content: str) -> str:
"""
system_prompt = T(".prompts:hypothesis_generation.system").r()
user_prompt = T(".prompts:hypothesis_generation.user").r(
factor_descriptions=json.dumps(factor_result), report_content=report_content
factor_descriptions=json.dumps(factor_result), report_content=report_content,
)
response = APIBackend().build_messages_and_create_chat_completion(
user_prompt=user_prompt,
system_prompt=system_prompt,
json_mode=True,
json_target_type=Dict[str, str],
json_target_type=dict[str, str],
)
response_json = json.loads(response)
@@ -99,7 +98,7 @@ class FactorReportLoop(FactorRDLoop, metaclass=LoopMeta):
super().__init__(PROP_SETTING=FACTOR_FROM_REPORT_PROP_SETTING)
if report_folder is None:
self.judge_pdf_data_items = json.load(
open(FACTOR_FROM_REPORT_PROP_SETTING.report_result_json_file_path, "r")
open(FACTOR_FROM_REPORT_PROP_SETTING.report_result_json_file_path),
)
else:
self.judge_pdf_data_items = [i for i in Path(report_folder).rglob("*.pdf")]
@@ -118,7 +117,7 @@ class FactorReportLoop(FactorRDLoop, metaclass=LoopMeta):
if exp is None:
self.shift_report += 1
self.loop_n -= 1
if self.loop_n < 0: # NOTE: on every step, we self.loop_n -= 1 at first.
if self.loop_n < 0: # loop_n is decremented above when reports are empty; prevents infinite skipping
raise self.LoopTerminationError("Reach stop criterion and stop loop")
continue
exp.based_experiments = [QlibFactorExperiment(sub_tasks=[], hypothesis=exp.hypothesis)] + [
+34 -32
View File
@@ -8,7 +8,6 @@ from pathlib import Path
from typing import Any
import fire
from rdagent.app.qlib_rd_loop.conf import QUANT_PROP_SETTING
from rdagent.components.workflow.conf import BasePropSetting
from rdagent.components.workflow.rd_loop import RDLoop
@@ -44,11 +43,11 @@ class QuantRDLoop(RDLoop):
logger.log_object(self.hypothesis_gen, tag="quant hypothesis generator")
self.factor_hypothesis2experiment: Hypothesis2Experiment = import_class(
PROP_SETTING.factor_hypothesis2experiment
PROP_SETTING.factor_hypothesis2experiment,
)()
logger.log_object(self.factor_hypothesis2experiment, tag="factor hypothesis2experiment")
self.model_hypothesis2experiment: Hypothesis2Experiment = import_class(
PROP_SETTING.model_hypothesis2experiment
PROP_SETTING.model_hypothesis2experiment,
)()
logger.log_object(self.model_hypothesis2experiment, tag="model hypothesis2experiment")
@@ -78,7 +77,8 @@ class QuantRDLoop(RDLoop):
while True:
if self.get_unfinished_loop_cnt(self.loop_idx) < RD_AGENT_SETTINGS.get_max_parallel():
hypo = self._propose()
assert hypo.action in ["factor", "model"]
if hypo.action not in ["factor", "model"]:
raise ValueError(f"hypo.action must be 'factor' or 'model', got {hypo.action!r}")
if hypo.action == "factor":
exp = self.factor_hypothesis2experiment.convert(hypo, self.trace)
else:
@@ -132,7 +132,6 @@ class QuantRDLoop(RDLoop):
"""
import json
from datetime import datetime
from pathlib import Path
try:
project_root = Path(__file__).parent.parent.parent.parent
@@ -195,11 +194,11 @@ class QuantRDLoop(RDLoop):
if prev_out["direct_exp_gen"]["propose"].action == "factor":
exp = self.factor_runner.develop(prev_out["coding"])
if exp is None:
logger.error(f"Factor extraction failed.")
logger.error("Factor extraction failed.")
raise FactorEmptyError("Factor extraction failed.")
# Increment factor count for tracking
if hasattr(self, 'trace') and hasattr(self.trace, 'increment_factor_count'):
if hasattr(self, "trace") and hasattr(self.trace, "increment_factor_count"):
self.trace.increment_factor_count()
# Handle failed experiments gracefully (don't break the loop)
@@ -210,7 +209,7 @@ class QuantRDLoop(RDLoop):
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
logger.warning(
f"Factor '{factor_name}' failed evaluation: {reason}. "
f"Continuing with next factor."
f"Continuing with next factor.",
)
# Return exp anyway - loop will continue
elif prev_out["direct_exp_gen"]["propose"].action == "model":
@@ -219,7 +218,7 @@ class QuantRDLoop(RDLoop):
return exp
def feedback(self, prev_out: dict[str, Any]):
e = prev_out.get(self.EXCEPTION_KEY, None)
e = prev_out.get(self.EXCEPTION_KEY)
if e is not None:
feedback = HypothesisFeedback(
observations=str(e),
@@ -245,11 +244,10 @@ class QuantRDLoop(RDLoop):
reason=reason,
decision=False,
)
else:
if prev_out["direct_exp_gen"]["propose"].action == "factor":
feedback = self.factor_summarizer.generate_feedback(prev_out["running"], self.trace)
elif prev_out["direct_exp_gen"]["propose"].action == "model":
feedback = self.model_summarizer.generate_feedback(prev_out["running"], self.trace)
elif prev_out["direct_exp_gen"]["propose"].action == "factor":
feedback = self.factor_summarizer.generate_feedback(prev_out["running"], self.trace)
elif prev_out["direct_exp_gen"]["propose"].action == "model":
feedback = self.model_summarizer.generate_feedback(prev_out["running"], self.trace)
# NOTE: DB save is handled by factor_runner.py _save_result_to_database()
# which runs immediately after Docker execution. No duplicate save needed here.
@@ -258,20 +256,20 @@ class QuantRDLoop(RDLoop):
factor_count = self.trace.get_factor_count()
# Check for auto-strategies trigger
auto_strategies = getattr(self, '_auto_strategies', False)
auto_threshold = getattr(self, '_auto_strategies_threshold', 500)
auto_strategies = getattr(self, "_auto_strategies", False)
auto_threshold = getattr(self, "_auto_strategies_threshold", 500)
if auto_strategies and factor_count > 0 and factor_count % auto_threshold == 0:
logger.info(
f"Auto-strategy trigger: {factor_count} factors evaluated. "
f"Suggesting strategy generation now..."
f"Suggesting strategy generation now...",
)
self._build_strategies_with_ai()
elif factor_count > 0 and factor_count % 50 == 0 and not auto_strategies:
# Standard periodic suggestion (every 50 factors)
logger.info(
f"Periodic check: {factor_count} factors evaluated. "
f"Consider running 'rdagent generate_strategies' for AI strategy generation."
f"Consider running 'rdagent generate_strategies' for AI strategy generation.",
)
feedback = self._interact_feedback(feedback)
@@ -292,10 +290,11 @@ class QuantRDLoop(RDLoop):
- Optuna hyperparameter optimization
"""
try:
from rdagent.components.coder.strategy_orchestrator import StrategyOrchestrator
from pathlib import Path
import yaml
from rdagent.components.coder.strategy_orchestrator import StrategyOrchestrator
# Load improved prompt
project_root = Path(__file__).parent.parent.parent.parent
prompt_path = project_root / "prompts" / "strategy_generation_v2.yaml"
@@ -335,44 +334,47 @@ class QuantRDLoop(RDLoop):
logger.info(f"StrategyOrchestrator: Building strategies from {len(top_factors)} top factors...")
logger.info(f" - Using improved prompt: {improved_prompt is not None}")
logger.info(f" - Optuna optimization: enabled (20 trials)")
logger.info(f" - Real OHLCV backtest: enabled")
logger.info(" - Optuna optimization: enabled (20 trials)")
logger.info(" - Real OHLCV backtest: enabled")
# Initialize orchestrator with Optuna
orchestrator = StrategyOrchestrator(
top_factors=20,
trading_style='swing',
trading_style="swing",
min_sharpe=0.5,
max_drawdown=-0.20,
min_win_rate=0.40,
use_optuna=True,
optuna_trials=20,
)
# Override with improved prompt if available
if improved_prompt:
orchestrator.strategy_prompt = improved_prompt.get('strategy_generation', {})
orchestrator.strategy_prompt = improved_prompt.get("strategy_generation", {})
# Generate 3 strategies per cycle
n_strategies = 3
logger.info(f"Generating {n_strategies} strategies...")
# Load top factors for generation
orch_factors = orchestrator.load_top_factors()
if len(orch_factors) < 2:
logger.warning(f"Not enough factors for strategy generation (need >= 2, got {len(orch_factors)}). Skipping.")
return
for i in range(n_strategies):
strategy_name = f"auto_gen_v{i+1}"
try:
# Select random factor combination
import random
n_factors = random.randint(2, min(5, len(orch_factors)))
factor_subset = random.sample(orch_factors, n_factors)
strategy_name = f"auto_gen_v{i+1}"
code = orchestrator.generate_strategy_code(factor_subset, strategy_name)
if code:
result = orchestrator.evaluate_strategy(code, strategy_name, factor_subset)
if result.get("status") == "accepted":
logger.info(f"✅ Strategy {strategy_name} accepted!")
logger.info(f" Sharpe: {result.get('sharpe_ratio', 0):.2f}")
@@ -430,7 +432,7 @@ def main(
quant_loop._auto_strategies = True
quant_loop._auto_strategies_threshold = auto_strategies_threshold
logger.info(
f"Auto-strategies enabled. Will trigger after {auto_strategies_threshold} factors."
f"Auto-strategies enabled. Will trigger after {auto_strategies_threshold} factors.",
)
else:
quant_loop._auto_strategies = False
@@ -72,7 +72,7 @@ class BacktestMetrics:
class FactorBacktester:
def __init__(self):
self.metrics = BacktestMetrics()
self.results_path = Path(__file__).parent.parent.parent / "results" / "backtests"
self.results_path = Path(__file__).parent.parent.parent.parent / "results" / "backtests"
self.results_path.mkdir(parents=True, exist_ok=True)
def run_backtest(
@@ -222,7 +222,7 @@ class FactorBacktester:
# Calculate return for this step
if step > 0:
prev_price = float(price_values[step - 1]) if step > 0 else current_price
prev_price = float(price_values[step - 1])
if prev_price > 0:
step_return = (current_price - prev_price) / prev_price * position
returns_history.append(step_return)
+3 -3
View File
@@ -166,7 +166,7 @@ class ResultsDatabase:
self.conn.commit()
return c.lastrowid
def add_loop(self, loop_idx: int, success: int, fail: int, best_ic: float = None, status: str = "completed") -> int:
def add_loop(self, loop_idx: int, success: int, fail: int, best_ic: float | None = None, status: str = "completed") -> int:
c = self.conn.cursor()
rate = success / (success + fail) if (success + fail) > 0 else 0
c.execute("""INSERT INTO loop_results (loop_index, factors_success, factors_fail, success_rate, best_ic, status)
@@ -330,13 +330,13 @@ class ResultsDatabase:
worst_drawdown = all_results['max_drawdown'].min() if total_runs > 0 and all_results['max_drawdown'].notna().any() else None
# Scan factors directory for JSON files
factors_dir = Path(__file__).parent.parent.parent / "results" / "factors"
factors_dir = Path(__file__).parent.parent.parent.parent / "results" / "factors"
json_factor_files = 0
if factors_dir.exists():
json_factor_files = len(list(factors_dir.glob("*.json")))
# Scan failed runs
failed_dir = Path(__file__).parent.parent.parent / "results" / "failed_runs"
failed_dir = Path(__file__).parent.parent.parent.parent / "results" / "failed_runs"
failed_runs_file = failed_dir / "failed_runs.json"
failed_runs_count = 0
failed_runs_data = []
@@ -1,21 +1,19 @@
"""
Predix Risk Management - Korrelation, Portfolio-Optimierung
"""
import numpy as np
import pandas as pd
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime
import json
class CorrelationAnalyzer:
def __init__(self, lookback: int = 60):
self.lookback = lookback
def calculate_matrix(self, returns: pd.DataFrame) -> pd.DataFrame:
return returns.dropna().corr()
def find_uncorrelated(self, corr: pd.DataFrame, threshold: float = 0.3) -> List[str]:
def find_uncorrelated(self, corr: pd.DataFrame, threshold: float = 0.3) -> list[str]:
result = []
for f in corr.columns:
others = [x for x in corr.columns if x != f]
@@ -28,9 +26,9 @@ class PortfolioOptimizer:
try:
w = np.linalg.inv(cov.values) @ exp_ret.values
return w / np.sum(w)
except:
except (np.linalg.LinAlgError, ValueError):
return np.ones(len(exp_ret)) / len(exp_ret)
def risk_parity(self, cov: pd.DataFrame, max_iter: int = 100) -> np.ndarray:
n = cov.shape[0]
w = np.ones(n) / n
@@ -53,36 +51,36 @@ class AdvancedRiskManager:
self.max_dd = max_dd
self.corr_analyzer = CorrelationAnalyzer()
self.optimizer = PortfolioOptimizer()
def check_limits(self, weights: np.ndarray, vol: float, dd: float) -> Dict[str, bool]:
def check_limits(self, weights: np.ndarray, vol: float, dd: float) -> dict[str, bool]:
return {
'position_limit': np.max(np.abs(weights)) <= self.max_pos,
'leverage_limit': np.sum(np.abs(weights)) <= self.max_lev,
'drawdown_limit': abs(dd) <= self.max_dd,
"position_limit": np.max(np.abs(weights)) <= self.max_pos,
"leverage_limit": np.sum(np.abs(weights)) <= self.max_lev,
"drawdown_limit": abs(dd) <= self.max_dd,
}
if __name__ == "__main__":
print("=== Risk Test ===")
np.random.seed(42)
n, names = 252, ['Mom', 'MeanRev', 'Vol', 'Volu', 'ML']
n, names = 252, ["Mom", "MeanRev", "Vol", "Volu", "ML"]
ret = pd.DataFrame(np.random.randn(n, 5), columns=names)
corr = CorrelationAnalyzer().calculate_matrix(ret)
print("Korrelationsmatrix:")
print(corr.round(2))
opt = PortfolioOptimizer()
exp_ret = pd.Series([0.1, 0.08, 0.06, 0.07, 0.12], index=names)
cov = ret.cov() * 252
mv = opt.mean_variance(exp_ret, cov)
print("\nMean-Variance:")
for n, w in zip(names, mv): print(f" {n}: {w:.2%}")
rp = opt.risk_parity(cov)
print("\nRisk Parity:")
for n, w in zip(names, rp): print(f" {n}: {w:.2%}")
rm = AdvancedRiskManager()
checks = rm.check_limits(mv, 0.15, -0.08)
print(f"\nLimits OK: {all(checks.values())}")
+22 -26
View File
@@ -19,7 +19,7 @@ Design goals
"""
from __future__ import annotations
from typing import Any, Dict, Optional
from typing import Any
import numpy as np
import pandas as pd
@@ -67,9 +67,8 @@ def _cross_check_with_vbt(
close: pd.Series,
position: pd.Series,
txn_cost: float,
manual_total_return: float,
freq: str,
) -> Optional[float]:
) -> float | None:
"""Run a vectorbt simulation and return its total_return for comparison."""
if not VBT_AVAILABLE:
return None
@@ -84,7 +83,8 @@ def _cross_check_with_vbt(
init_cash=10_000.0,
freq=freq,
)
return float(pf.total_return())
tr = float(pf.total_return())
return tr if np.isfinite(tr) else None
except Exception:
return None
@@ -95,9 +95,9 @@ def backtest_signal(
txn_cost_bps: float = DEFAULT_TXN_COST_BPS,
freq: str = "1min",
bars_per_year: int = DEFAULT_BARS_PER_YEAR,
forward_returns: Optional[pd.Series] = None,
forward_returns: pd.Series | None = None,
cross_check: bool = False,
) -> Dict[str, Any]:
) -> dict[str, Any]:
"""
Run a single-asset backtest from a position signal.
@@ -204,7 +204,7 @@ def backtest_signal(
calmar = ann_return_arith / abs(max_dd) if max_dd < 0 else 0.0
trade_pnl = _compute_trade_pnl(position, strategy_returns)
n_trades = int(len(trade_pnl))
n_trades = len(trade_pnl)
n_position_changes = int((position.diff().fillna(0) != 0).sum())
if n_trades > 0:
@@ -216,7 +216,7 @@ def backtest_signal(
win_rate = 0.0
profit_factor = 0.0
ic: Optional[float] = None
ic: float | None = None
if forward_returns is not None:
fwd = pd.to_numeric(forward_returns, errors="coerce")
common = signal.index.intersection(fwd.dropna().index)
@@ -227,7 +227,7 @@ def backtest_signal(
ic_val = float(s.corr(f))
ic = ic_val if np.isfinite(ic_val) else None
result: Dict[str, Any] = {
result: dict[str, Any] = {
"status": "success",
"sharpe": sharpe,
"sortino": sortino,
@@ -244,7 +244,7 @@ def backtest_signal(
"volatility": volatility,
"n_trades": n_trades,
"n_position_changes": n_position_changes,
"n_bars": int(len(strategy_returns)),
"n_bars": len(strategy_returns),
"n_months": float(n_months),
"signal_long": int((signal > 0).sum()),
"signal_short": int((signal < 0).sum()),
@@ -264,7 +264,6 @@ def backtest_signal(
close=close,
position=position,
txn_cost=txn_cost,
manual_total_return=total_return,
freq=freq,
)
@@ -293,7 +292,7 @@ def _apply_ftmo_mask(
daily_breaches = 0
total_breached = False
total_breach_ts: Optional[pd.Timestamp] = None
total_breach_ts: pd.Timestamp | None = None
current_day = None
day_start_eq = FTMO_INITIAL_CAPITAL
@@ -308,11 +307,8 @@ def _apply_ftmo_mask(
pos_i = float(signal.at[ts]) * leverage
ret_i = float(bar_ret.get(ts, 0.0))
cost_i = abs(pos_i - pos_prev) * txn_cost
ret_net = pos_prev * ret_i - cost_i
equity = equity * (1.0 + ret_net / FTMO_INITIAL_CAPITAL * FTMO_INITIAL_CAPITAL / equity
if equity > 0 else 1.0)
# Simpler: track as fraction
equity += FTMO_INITIAL_CAPITAL * ret_net
ret_frac = pos_prev * ret_i - cost_i
equity *= 1.0 + ret_frac if equity > 0 else 1.0
pos_prev = pos_i
if total_breached:
@@ -399,7 +395,7 @@ def walk_forward_rolling(
is_years: int = WF_IS_YEARS,
oos_years: int = WF_OOS_YEARS,
step_years: int = WF_STEP_YEARS,
) -> Dict[str, Any]:
) -> dict[str, Any]:
"""
Rolling walk-forward validation: multiple IS/OOS windows shifted by ``step_years``.
@@ -433,7 +429,7 @@ def walk_forward_rolling(
yr += step_years
continue
window: Dict[str, Any] = {
window: dict[str, Any] = {
"is_start": str(is_start.date()),
"is_end": str(is_end.date()),
"oos_start": str(is_end.date()),
@@ -475,11 +471,11 @@ def backtest_signal_ftmo(
stop_pips: float = FTMO_STOP_PIPS,
max_leverage: float = FTMO_MAX_LEVERAGE,
bars_per_year: int = DEFAULT_BARS_PER_YEAR,
forward_returns: Optional[pd.Series] = None,
oos_start: Optional[str] = OOS_START_DEFAULT,
forward_returns: pd.Series | None = None,
oos_start: str | None = OOS_START_DEFAULT,
wf_rolling: bool = False,
mc_n_permutations: int = 0,
) -> Dict[str, Any]:
) -> dict[str, Any]:
"""
FTMO-compliant backtest of a strategy signal on EUR/USD.
@@ -547,7 +543,7 @@ def backtest_signal_ftmo(
is_mask = close.index < oos_ts
oos_mask = close.index >= oos_ts
def _split_bt(mask: "pd.Series[bool]", prefix: str) -> None:
def _split_bt(mask: pd.Series[bool], prefix: str) -> None:
if mask.sum() < 100:
return
close_s = close.loc[mask]
@@ -602,7 +598,7 @@ def backtest_from_forward_returns(
forward_returns: pd.Series,
txn_cost_bps: float = DEFAULT_TXN_COST_BPS,
bars_per_year: int = DEFAULT_BARS_PER_YEAR,
) -> Dict[str, Any]:
) -> dict[str, Any]:
"""
Backtest a factor using sign(factor) as signal against forward returns.
@@ -640,7 +636,7 @@ def backtest_from_forward_returns(
ic = ic_val if np.isfinite(ic_val) else 0.0
trade_pnl = _compute_trade_pnl(position, strategy_returns)
n_trades = int(len(trade_pnl))
n_trades = len(trade_pnl)
win_rate = float((trade_pnl > 0).mean()) if n_trades > 0 else 0.0
ann_return = float(strategy_returns.mean() * bars_per_year)
@@ -656,7 +652,7 @@ def backtest_from_forward_returns(
"win_rate": win_rate,
"n_trades": n_trades,
"ic": ic,
"n_bars": int(len(strategy_returns)),
"n_bars": len(strategy_returns),
"txn_cost_bps": txn_cost_bps,
"bars_per_year": bars_per_year,
}
+14 -7
View File
@@ -75,8 +75,10 @@ class CoSTEER(Developer[Experiment]):
def _get_last_fb(self) -> CoSTEERMultiFeedback:
fb = self.evolve_agent.evolving_trace[-1].feedback
assert fb is not None, "feedback is None"
assert isinstance(fb, CoSTEERMultiFeedback), "feedback must be of type CoSTEERMultiFeedback"
if fb is None:
raise AssertionError("feedback is None")
if not isinstance(fb, CoSTEERMultiFeedback):
raise TypeError("feedback must be of type CoSTEERMultiFeedback")
return fb
def should_use_new_evo(self, base_fb: CoSTEERMultiFeedback | None, new_fb: CoSTEERMultiFeedback) -> bool:
@@ -121,7 +123,8 @@ class CoSTEER(Developer[Experiment]):
for evo_exp in self.evolve_agent.multistep_evolve(evo_exp, self.evaluator):
iteration_count += 1
assert isinstance(evo_exp, Experiment) # multiple inheritance
if not isinstance(evo_exp, Experiment):
raise TypeError("evo_exp must be an instance of Experiment")
evo_fb = self._get_last_fb()
update_fallback = self.should_use_new_evo(
base_fb=fallback_evo_fb,
@@ -154,7 +157,8 @@ class CoSTEER(Developer[Experiment]):
evo_exp = fallback_evo_exp
evo_exp.recover_ws_ckp()
evo_fb = fallback_evo_fb
assert evo_fb is not None # multistep_evolve should run at least once
if evo_fb is None:
raise AssertionError("multistep_evolve should run at least once")
evo_exp = self._exp_postprocess_by_feedback(evo_exp, evo_fb)
except CoderError as e:
e.caused_by_timeout = reached_max_seconds
@@ -264,9 +268,12 @@ class CoSTEER(Developer[Experiment]):
- Raise Error if it failed to handle the develop task
-
"""
assert isinstance(evo, Experiment)
assert isinstance(feedback, CoSTEERMultiFeedback)
assert len(evo.sub_workspace_list) == len(feedback)
if not isinstance(evo, Experiment):
raise TypeError("evo must be an instance of Experiment")
if not isinstance(feedback, CoSTEERMultiFeedback):
raise TypeError("feedback must be an instance of CoSTEERMultiFeedback")
if len(evo.sub_workspace_list) != len(feedback):
raise ValueError("Length of sub_workspace_list must match length of feedback")
# FIXME: when whould the feedback be None?
failed_feedbacks = [
@@ -122,7 +122,8 @@ class MultiProcessEvolvingStrategy(EvolvingStrategy):
last_feedback = None
if len(evolving_trace) > 0:
last_feedback = evolving_trace[-1].feedback
assert isinstance(last_feedback, CoSTEERMultiFeedback)
if not isinstance(last_feedback, CoSTEERMultiFeedback):
raise TypeError("last_feedback must be of type CoSTEERMultiFeedback")
# 1.找出需要evolve的task
to_be_finished_task_index: list[int] = []
@@ -1028,7 +1028,8 @@ class CoSTEERKnowledgeBaseV2(EvolvingKnowledgeBase):
"""
node_count = len(nodes)
assert node_count >= 2, "nodes length must >=2"
if node_count < 2:
raise ValueError("nodes length must >=2")
intersection_node_list = []
if output_intersection_origin:
origin_list = []
@@ -328,12 +328,13 @@ class FactorEqualValueRatioEvaluator(FactorEvaluator):
"The source dataframe is None. Please check the implementation.",
-1,
)
acc_rate = -1
try:
close_values = gen_df.sub(gt_df).abs().lt(1e-6)
result_int = close_values.astype(int)
pos_num = result_int.sum().sum()
acc_rate = pos_num / close_values.size
except:
except Exception:
close_values = gen_df
if close_values.all().iloc[0]:
return (
@@ -58,10 +58,12 @@ class ModelCodeEvaluator(CoSTEEREvaluator):
model_execution_feedback: str = "",
model_value_feedback: str = "",
):
assert isinstance(target_task, ModelTask)
assert isinstance(implementation, ModelFBWorkspace)
if gt_implementation is not None:
assert isinstance(gt_implementation, ModelFBWorkspace)
if not isinstance(target_task, ModelTask):
raise TypeError("target_task must be of type ModelTask")
if not isinstance(implementation, ModelFBWorkspace):
raise TypeError("implementation must be of type ModelFBWorkspace")
if gt_implementation is not None and not isinstance(gt_implementation, ModelFBWorkspace):
raise TypeError("gt_implementation must be of type ModelFBWorkspace")
model_task_information = target_task.get_task_information()
code = implementation.all_codes
@@ -113,10 +115,12 @@ class ModelFinalEvaluator(CoSTEEREvaluator):
model_value_feedback: str,
model_code_feedback: str,
):
assert isinstance(target_task, ModelTask)
assert isinstance(implementation, ModelFBWorkspace)
if gt_implementation is not None:
assert isinstance(gt_implementation, ModelFBWorkspace)
if not isinstance(target_task, ModelTask):
raise TypeError("target_task must be of type ModelTask")
if not isinstance(implementation, ModelFBWorkspace):
raise TypeError("implementation must be of type ModelFBWorkspace")
if gt_implementation is not None and not isinstance(gt_implementation, ModelFBWorkspace):
raise TypeError("gt_implementation must be of type ModelFBWorkspace")
system_prompt = T(".prompts:evaluator_final_feedback.system").r(
scenario=(
+15 -1
View File
@@ -23,7 +23,7 @@ import pandas as pd
from rdagent.log import rdagent_logger as logger
logger = logging.getLogger(__name__)
_optuna_logger = logging.getLogger(__name__)
try:
import optuna
@@ -292,6 +292,7 @@ class OptunaOptimizer:
"volatility_lookback": trial.suggest_int("volatility_lookback", 5, 500, step=5),
"signal_bias": trial.suggest_float("signal_bias", -1.0, 1.0, step=0.05),
"max_hold_bars": trial.suggest_int("max_hold_bars", 5, 1000, step=5),
"max_positions": trial.suggest_int("max_positions", 1, 5, step=1),
}
# Parameters that are allowed to be negative (not clamped to 0).
@@ -308,6 +309,7 @@ class OptunaOptimizer:
"volatility_lookback": 1.0,
"signal_bias": -1.0,
"max_hold_bars": 1.0,
"max_positions": 1.0,
}
def _suggest_bounded(
@@ -357,6 +359,7 @@ class OptunaOptimizer:
"volatility_lookback": (center.get("volatility_lookback", 100), 30),
"signal_bias": (center.get("signal_bias", 0.0), 0.2),
"max_hold_bars": (center.get("max_hold_bars", 100), 50),
"max_positions": (center.get("max_positions", 1), 2),
}
return {key: self._suggest_bounded(trial, key, c, hw) for key, (c, hw) in ranges.items()}
@@ -388,6 +391,7 @@ class OptunaOptimizer:
"volatility_lookback": (center.get("volatility_lookback", 100), 10),
"signal_bias": (center.get("signal_bias", 0.0), 0.07),
"max_hold_bars": (center.get("max_hold_bars", 100), 17),
"max_positions": (center.get("max_positions", 1), 1),
}
return {key: self._suggest_bounded(trial, key, c, hw) for key, (c, hw) in ranges.items()}
@@ -467,6 +471,9 @@ class OptunaOptimizer:
# Max holding periods (in bars)
"max_hold_bars": trial.suggest_int("max_hold_bars", 10, 500, step=10),
# Max concurrent positions (1 = no pyramiding, 2-5 = scale-in)
"max_positions": trial.suggest_int("max_positions", 1, 5, step=1),
}
return params
@@ -597,6 +604,13 @@ class OptunaOptimizer:
if signal_bias != 0.0:
signal = (signal.astype(float) + signal_bias).round().astype(int).clip(-1, 1)
# Apply max_positions: scale signal by position_size_pct and cap exposure
max_positions = int(params.get("max_positions", 1))
position_size_pct = float(params.get("position_size_pct", 1.0))
# Each "position" is position_size_pct of equity; total exposure capped at max_positions × size
effective_size = min(position_size_pct * max_positions, 1.0)
signal = (signal.astype(float) * effective_size).clip(-1.0, 1.0)
# Build a synthetic close from the factor-mean so we can route
# through the same unified engine as every other backtest path.
# Backtest formulas must match the orchestrator's real-OHLCV path.
+138 -137
View File
@@ -26,22 +26,18 @@ import traceback
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional
from typing import Any
import numpy as np
import pandas as pd
import requests
from rdagent.components.prompt_loader import load_prompt
from rdagent.components.coder.optuna_optimizer import OptunaOptimizer
from rdagent.components.prompt_loader import load_prompt
# OHLCV data path
OHLCV_PATH = Path(os.getenv(
'PREDIX_OHLCV_PATH',
'/home/nico/Predix/git_ignore_folder/factor_implementation_source_data/intraday_pv.h5'
"PREDIX_OHLCV_PATH",
"/home/nico/Predix/git_ignore_folder/factor_implementation_source_data/intraday_pv.h5",
))
from rdagent.log import rdagent_logger as logger
logger = logging.getLogger(__name__)
@@ -60,7 +56,7 @@ class StrategyOrchestrator:
min_sharpe: float = 0.3,
max_drawdown: float = -0.30,
min_win_rate: float = 0.40,
results_dir: Optional[str] = None,
results_dir: str | None = None,
use_optuna: bool = True,
optuna_trials: int = 20,
continuous_optimization: bool = True,
@@ -118,7 +114,7 @@ class StrategyOrchestrator:
logger.info(
f"StrategyOrchestrator initialized: style={self.trading_style}, "
f"top_factors={self.top_factors}, min_sharpe={self.min_sharpe}"
f"top_factors={self.top_factors}, min_sharpe={self.min_sharpe}",
)
def load_ohlcv_close(self) -> pd.Series:
@@ -126,31 +122,31 @@ class StrategyOrchestrator:
if not OHLCV_PATH.exists():
logger.warning(f"OHLCV data not found: {OHLCV_PATH}")
return None
try:
ohlcv = pd.read_hdf(str(OHLCV_PATH), key='data')
if '$close' in ohlcv.columns:
close = ohlcv['$close'].dropna()
elif 'close' in ohlcv.columns:
close = ohlcv['close'].dropna()
ohlcv = pd.read_hdf(str(OHLCV_PATH), key="data")
if "$close" in ohlcv.columns:
close = ohlcv["$close"].dropna()
elif "close" in ohlcv.columns:
close = ohlcv["close"].dropna()
else:
close = ohlcv.select_dtypes(include=[np.number]).iloc[:, 0].dropna()
# Handle MultiIndex
if isinstance(close.index, pd.MultiIndex):
try:
close = close.xs('EURUSD', level='instrument')
close = close.xs("EURUSD", level="instrument")
except KeyError:
idx = close.index.get_level_values('instrument') == 'EURUSD'
idx = close.index.get_level_values("instrument") == "EURUSD"
close = close[idx]
close.index = close.index.droplevel('instrument')
close.index = close.index.droplevel("instrument")
return close
except Exception as e:
logger.warning(f"Failed to load OHLCV data: {e}")
return None
def load_top_factors(self) -> List[Dict[str, Any]]:
def load_top_factors(self) -> list[dict[str, Any]]:
"""
Load top evaluated factors from JSON files.
@@ -177,7 +173,7 @@ class StrategyOrchestrator:
# Sort by absolute IC and take top N
factors.sort(key=lambda x: abs(x.get("ic", 0) or 0), reverse=True)
# Filter to only include factors that have parquet files
factors_with_files = []
for f in factors:
@@ -188,16 +184,16 @@ class StrategyOrchestrator:
factors_with_files.append(f)
else:
logger.debug(f"Skipping {fname} - no parquet file")
# Select diverse factor TYPES, not just top IC
# This ensures we get momentum, volatility, session, volume, etc.
type_keywords = {
"momentum": [], "trend": [], "volatility": [], "volume": [],
"session": [], "london": [], "range": [], "vwap": [],
"return": [], "ofi": [], "spread": [], "close": [],
"divergence": [], "other": []
"divergence": [], "other": [],
}
for f in factors_with_files:
name = f.get("factor_name", "").lower()
matched = False
@@ -208,18 +204,18 @@ class StrategyOrchestrator:
break
if not matched:
type_keywords["other"].append(f)
# Select best from each type (ensures diversity)
selected = []
already_names = set()
# Priority order: momentum, divergence, volatility, session, volume, etc.
priority_types = ["momentum", "divergence", "volatility", "session",
"london", "range", "vwap", "volume", "ofi", "spread",
priority_types = ["momentum", "divergence", "volatility", "session",
"london", "range", "vwap", "volume", "ofi", "spread",
"return", "trend", "close", "other"]
per_type = max(2, self.top_factors // len(priority_types))
for kw in priority_types:
for f in sorted(type_keywords[kw], key=lambda x: abs(x.get("ic", 0)), reverse=True):
if f["factor_name"] not in already_names:
@@ -227,13 +223,13 @@ class StrategyOrchestrator:
already_names.add(f["factor_name"])
if len([s for s in selected if s["factor_name"] in [x["factor_name"] for x in type_keywords[kw]]]) >= per_type:
break
# Fill remaining with highest IC not yet selected
if len(selected) < self.top_factors:
remaining = [f for f in factors_with_files if f["factor_name"] not in already_names]
remaining.sort(key=lambda x: abs(x.get("ic", 0)), reverse=True)
selected.extend(remaining[:self.top_factors - len(selected)])
# Log diversity
type_counts = {}
for f in selected:
@@ -246,12 +242,12 @@ class StrategyOrchestrator:
break
if not matched:
type_counts["other"] = type_counts.get("other", 0) + 1
logger.info(f"Selected {len(selected)} diverse factors: {type_counts}")
return selected[:self.top_factors]
def load_factor_values(self, factor_name: str) -> Optional[pd.Series]:
def load_factor_values(self, factor_name: str) -> pd.Series | None:
"""
Load factor time-series values from parquet file.
@@ -273,32 +269,32 @@ class StrategyOrchestrator:
try:
df = pd.read_parquet(str(parquet_path))
# Handle empty DataFrame
if df.empty or len(df.columns) == 0:
logger.warning(f"Empty parquet file: {parquet_path}")
return None
# Handle MultiIndex (datetime, instrument)
if isinstance(df.index, pd.MultiIndex):
# Get the factor column name (should be the only column)
factor_col = df.columns[0]
# Extract EURUSD series
try:
series = df.xs('EURUSD', level='instrument')[factor_col]
series = df.xs("EURUSD", level="instrument")[factor_col]
except KeyError:
# Try alternative extraction
df_reset = df.reset_index()
if 'instrument' in df_reset.columns:
df_eur = df_reset[df_reset['instrument'] == 'EURUSD'].set_index('datetime')
if "instrument" in df_reset.columns:
df_eur = df_reset[df_reset["instrument"] == "EURUSD"].set_index("datetime")
series = df_eur[factor_col] if factor_col in df_eur.columns else df_eur.iloc[:, -1]
else:
series = df.iloc[:, 0]
else:
series = df.iloc[:, 0]
# Ensure numeric
series = pd.to_numeric(series, errors='coerce')
series = pd.to_numeric(series, errors="coerce")
series.name = factor_name
return series
except Exception as e:
@@ -307,10 +303,10 @@ class StrategyOrchestrator:
def generate_strategy_code(
self,
factors: List[Dict[str, Any]],
factors: list[dict[str, Any]],
strategy_name: str,
max_retries: int = 3,
) -> Optional[str]:
) -> str | None:
"""
Generate strategy code using LLM from factor combinations.
@@ -341,6 +337,12 @@ class StrategyOrchestrator:
user_prompt = user_prompt.replace("{{ additional_context }}", f"Strategy name: {strategy_name}")
user_prompt = user_prompt.replace("{{ trading_style }}", self.trading_style)
user_prompt = user_prompt.replace("{{ min_sharpe }}", str(self.min_sharpe))
if "{{" in user_prompt:
unreplaced = [w for w in user_prompt.split() if "{{" in w]
logger.warning(
f"Unreplaced template variables in prompt for '{strategy_name}': {unreplaced}"
)
user_prompt = user_prompt.replace("{{ max_drawdown }}", str(self.max_drawdown))
system_prompt = self.strategy_prompt.get("system", "")
else:
@@ -371,12 +373,12 @@ class StrategyOrchestrator:
last_error = f"Attempt {attempt}: LLM returned empty or invalid code"
logger.warning(f"LLM attempt {attempt}/{max_retries} failed: {last_error}")
except Exception as e:
last_error = f"Attempt {attempt}: {str(e)}"
last_error = f"Attempt {attempt}: {e!s}"
logger.warning(f"LLM attempt {attempt}/{max_retries} failed with exception: {e}")
logger.warning(
f"LLM strategy generation failed after {max_retries} attempts. "
f"Last error: {last_error}"
f"Last error: {last_error}",
)
# Fallback: generate template code programmatically
@@ -385,10 +387,10 @@ class StrategyOrchestrator:
def _generate_with_llm(
self,
context: Dict[str, Any],
context: dict[str, Any],
attempt: int = 1,
feedback: Optional[str] = None,
) -> Optional[str]:
feedback: str | None = None,
) -> str | None:
"""
Generate strategy code using LLM with APIBackend (same as Factor Coder).
@@ -406,7 +408,6 @@ class StrategyOrchestrator:
str or None
Validated Python strategy code, or None if invalid
"""
import json as json_module
# Build user message with optional feedback
user_content = context.get("user_prompt", "")
@@ -446,14 +447,13 @@ class StrategyOrchestrator:
if self._validate_python_code(code):
logger.info(f"[DEBUG] Valid Python code extracted ({len(code)} chars)")
return code
else:
logger.warning(f"JSON 'code' field contains invalid Python (attempt {attempt}). Preview: {code[:200]}")
logger.warning(f"JSON 'code' field contains invalid Python (attempt {attempt}). Preview: {code[:200]}")
else:
logger.warning(f"JSON parsed but no valid 'code' field found (attempt {attempt}). Keys: {list(json_data.keys())}")
# === STEP 2: Fallback - Extract Python code block directly (like Factor Coder) ===
import re
code_block_match = re.search(r'```python\s*\n(.*?)\n```', content, re.DOTALL)
code_block_match = re.search(r"```python\s*\n(.*?)\n```", content, re.DOTALL)
if code_block_match:
code = code_block_match.group(1).strip()
if code and self._validate_python_code(code):
@@ -463,7 +463,7 @@ class StrategyOrchestrator:
logger.warning(f"All extraction methods failed (attempt {attempt}). Response preview: {response[:200]}")
return None
def _extract_json(self, content: str) -> Optional[Dict[str, Any]]:
def _extract_json(self, content: str) -> dict[str, Any] | None:
"""
Extract JSON object from LLM response content.
@@ -491,7 +491,7 @@ class StrategyOrchestrator:
pass
# Strategy 2: Find ```json ... ``` blocks
json_block_match = re.search(r'```json\s*\n(.*?)\n```', content, re.DOTALL)
json_block_match = re.search(r"```json\s*\n(.*?)\n```", content, re.DOTALL)
if json_block_match:
try:
return json_module.loads(json_block_match.group(1))
@@ -499,7 +499,7 @@ class StrategyOrchestrator:
pass
# Strategy 3: Find ```python ... ``` blocks (Qwen often puts JSON in python blocks)
python_block_match = re.search(r'```python\s*\n(.*?)\n```', content, re.DOTALL)
python_block_match = re.search(r"```python\s*\n(.*?)\n```", content, re.DOTALL)
if python_block_match:
block = python_block_match.group(1).strip()
if block.startswith("{") and block.endswith("}"):
@@ -519,13 +519,13 @@ class StrategyOrchestrator:
# Try to fix common JSON issues (trailing commas, unescaped newlines)
try:
# Remove trailing commas before } or ]
json_str_fixed = re.sub(r',\s*([}\]])', r'\1', json_str)
json_str_fixed = re.sub(r",\s*([}\]])", r"\1", json_str)
return json_module.loads(json_str_fixed)
except json_module.JSONDecodeError:
pass
# Strategy 5: Find ``` ... ``` blocks (any language tag)
code_block_match = re.search(r'```\w*\s*\n(.*?)\n```', content, re.DOTALL)
code_block_match = re.search(r"```\w*\s*\n(.*?)\n```", content, re.DOTALL)
if code_block_match:
block = code_block_match.group(1).strip()
if block.startswith("{") and block.endswith("}"):
@@ -536,7 +536,7 @@ class StrategyOrchestrator:
return None
def _extract_code_from_json(self, json_data: Dict[str, Any]) -> Optional[str]:
def _extract_code_from_json(self, json_data: dict[str, Any]) -> str | None:
"""
Extract Python code from parsed JSON data.
@@ -559,7 +559,7 @@ class StrategyOrchestrator:
return None
def _extract_code_from_raw(self, content: str) -> Optional[str]:
def _extract_code_from_raw(self, content: str) -> str | None:
"""
Extract Python code from raw (non-JSON) LLM response.
@@ -579,12 +579,12 @@ class StrategyOrchestrator:
code = content.strip()
# Try to find ```python blocks
python_match = re.search(r'```python\s*\n(.*?)\n```', code, re.DOTALL)
python_match = re.search(r"```python\s*\n(.*?)\n```", code, re.DOTALL)
if python_match:
code = python_match.group(1)
else:
# Try generic ``` blocks
block_match = re.search(r'```\s*\n(.*?)\n```', code, re.DOTALL)
block_match = re.search(r"```\s*\n(.*?)\n```", code, re.DOTALL)
if block_match:
code = block_match.group(1)
@@ -636,7 +636,7 @@ class StrategyOrchestrator:
# Remove non-ASCII characters (emojis, etc.)
code = code.encode("ascii", "ignore").decode("ascii").strip()
return code if code else None
return code or None
def _validate_python_code(self, code: str) -> bool:
"""
@@ -663,14 +663,14 @@ class StrategyOrchestrator:
logger.debug(f"Python syntax error: {e}")
return False
def _generate_fallback_code(self, context: Dict[str, Any]) -> str:
def _generate_fallback_code(self, context: dict[str, Any]) -> str:
"""Generate fallback strategy code programmatically."""
factor_names = context["factor_names"]
style_config = "daytrading" if context["trading_style"] == "daytrading" else "swing"
# Build factor assignment code
factor_assignments = "\n ".join(
[f'"{name}": factors["{name}"]' for name in factor_names if name != "timestamp"]
[f'"{name}": factors["{name}"]' for name in factor_names if name != "timestamp"],
)
code = f'''"""
@@ -717,8 +717,8 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
return code
def evaluate_strategy(
self, strategy_code: str, strategy_name: str, factors: List[Dict[str, Any]]
) -> Dict[str, Any]:
self, strategy_code: str, strategy_name: str, factors: list[dict[str, Any]],
) -> dict[str, Any]:
"""
Evaluate a strategy by executing its code and calculating metrics.
@@ -755,23 +755,20 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
}
# Align factor values with common index
if not factor_values:
df_factors = pd.DataFrame()
else:
# Find common index across all series
common_idx = None
for name, s in factor_values.items():
if common_idx is None:
common_idx = s.index
else:
common_idx = common_idx.intersection(s.index)
if common_idx is not None and len(common_idx) > 100:
df_factors = pd.DataFrame({
name: s.reindex(common_idx) for name, s in factor_values.items()
}).dropna()
# Find common index across all series
common_idx = None
for name, s in factor_values.items():
if common_idx is None:
common_idx = s.index
else:
df_factors = pd.DataFrame()
common_idx = common_idx.intersection(s.index)
if common_idx is not None and len(common_idx) > 100:
df_factors = pd.DataFrame({
name: s.reindex(common_idx) for name, s in factor_values.items()
}).dropna()
else:
df_factors = pd.DataFrame()
if len(df_factors) < 100:
return {
@@ -783,8 +780,8 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
# Convert all factor columns to numeric
for col in df_factors.columns:
df_factors[col] = pd.to_numeric(df_factors[col], errors='coerce')
df_factors[col] = pd.to_numeric(df_factors[col], errors="coerce")
# Forward-fill daily factors to match OHLCV 1-min index
# Many factors are daily (1 value per day), need to ffill to 1-min
# FIX 6: Track ffill ratio for data quality monitoring
@@ -799,11 +796,11 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
f"[DEBUG] {strategy_name}: data quality: "
f"original_rows={original_len}, "
f"ffill_rows={len(df_factors) - original_len}, "
f"ffill_ratio={ffill_ratio:.2%}"
f"ffill_ratio={ffill_ratio:.2%}",
)
df_factors = df_factors.dropna()
if len(df_factors) < 1000:
return {
"strategy_name": strategy_name,
@@ -811,54 +808,58 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
"reason": f"Insufficient numeric data after conversion ({len(df_factors)} rows)",
"factors_used": factor_names,
}
# close is already loaded above for ffill, reuse it
# Reindex close to match factor index
if close is not None:
close = close.reindex(df_factors.index)
# Execute strategy code with factor data and close prices
local_vars = {"factors": df_factors}
if close is not None:
local_vars["close"] = close
try:
exec(strategy_code, {"np": np, "pd": pd, "numpy": np}, local_vars)
except Exception as e:
import traceback
logger.error(
f"Strategy code execution failed for '{strategy_name}': {e}\n"
f"{traceback.format_exc()[-2000:]}"
)
return {
"strategy_name": strategy_name,
"status": "rejected",
"reason": f"Code execution error: {str(e)}",
"reason": f"Code execution error: {e!s}",
"factors_used": factor_names,
}
if "signal" not in local_vars:
signal = local_vars.get("signal")
if signal is None or (isinstance(signal, pd.Series) and signal.empty):
return {
"strategy_name": strategy_name,
"status": "rejected",
"reason": "Strategy did not produce 'signal' variable",
"reason": "Strategy did not produce valid 'signal' variable",
"factors_used": factor_names,
}
signal = local_vars["signal"]
logger.info(
f"[DEBUG] {strategy_name}: signal stats: "
f"len={len(signal)}, "
f"long={int((signal > 0).sum())}, "
f"short={int((signal < 0).sum())}, "
f"flat={int((signal == 0).sum())}, "
f"unique={signal.nunique()}"
f"unique={signal.nunique()}",
)
# Delegate all metric computation to the single source of truth.
# Same formulas as every other backtest path in the repo.
from rdagent.components.backtesting.vbt_backtest import (
backtest_signal_ftmo,
DEFAULT_TXN_COST_BPS,
backtest_signal_ftmo,
)
close = self.load_ohlcv_close()
# Reuse the already-loaded close from above; create a synthetic proxy if unavailable
if close is None:
logger.warning("OHLCV data unavailable, using factor-mean proxy")
proxy = df_factors.mean(axis=1).astype(float)
@@ -890,7 +891,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
logger.info(
f"[DEBUG] {strategy_name}: bt stats: "
f"sharpe={sharpe:.4f} dd={max_dd:.4f} wr={win_rate:.4f} "
f"trades={num_real_trades} total_ret={bt['total_return']:.4%}"
f"trades={num_real_trades} total_ret={bt['total_return']:.4%}",
)
metrics = {
@@ -920,7 +921,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
f"[DEBUG] {strategy_name}: rejection breakdown: "
f"sharpe={sharpe:.4f} (need>={self.min_sharpe}), "
f"dd={max_dd:.4f} (need>={self.max_drawdown}), "
f"wr={win_rate:.4f} (need>={self.min_win_rate})"
f"wr={win_rate:.4f} (need>={self.min_win_rate})",
)
metrics["reason"] = self._get_rejection_reason(sharpe, max_dd, win_rate)
@@ -932,7 +933,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
return {
"strategy_name": strategy_name,
"status": "rejected",
"reason": f"Evaluation error: {str(e)}",
"reason": f"Evaluation error: {e!s}",
"factors_used": [],
}
@@ -951,7 +952,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
reasons.append(f"Win Rate {win_rate:.2%} < {self.min_win_rate:.2%}")
return "; ".join(reasons) if reasons else "Unknown"
def _generate_strategy_name(self, factors: List[Dict[str, Any]], idx: int) -> str:
def _generate_strategy_name(self, factors: list[dict[str, Any]], idx: int) -> str:
"""Generate a strategy name from its factors."""
# Extract key words from factor names
words = []
@@ -962,7 +963,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
for p in parts:
# Extract capitalized words
cap_words = [w for w in p.split() if w[0:1].isupper()]
words.extend(cap_words if cap_words else [p])
words.extend(cap_words or [p])
# Take up to 3 unique words
unique_words = list(dict.fromkeys(words))[:3]
@@ -975,7 +976,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
count: int = 10,
workers: int = 2, # Reduced from 4 to 2 to avoid LLM server overload
progress_callback=None,
) -> List[Dict[str, Any]]:
) -> list[dict[str, Any]]:
"""
Generate and evaluate trading strategies.
@@ -1028,7 +1029,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
logger.info(
f"Strategy ACCEPTED: {result['strategy_name']} | "
f"Sharpe={result['sharpe_ratio']:.2f} | "
f"DD={result['max_drawdown']:.2%}"
f"DD={result['max_drawdown']:.2%}",
)
else:
# Also save rejected strategies for debugging
@@ -1036,7 +1037,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
logger.warning(
f"Strategy REJECTED: {result['strategy_name']} - {result.get('reason', 'unknown')} | "
f"Sharpe={result.get('sharpe_ratio', 'N/A')} | "
f"DD={result.get('max_drawdown', 'N/A')}"
f"DD={result.get('max_drawdown', 'N/A')}",
)
if progress_callback:
@@ -1052,12 +1053,12 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
logger.info(
f"Strategy generation complete: {strategies_accepted}/{strategies_generated} accepted "
f"({strategies_accepted/max(strategies_generated,1)*100:.1f}%)"
f"({strategies_accepted/max(strategies_generated,1)*100:.1f}%)",
)
return results
def _generate_strategy_configs(self, factors: List[Dict], count: int) -> List[List[Dict]]:
def _generate_strategy_configs(self, factors: list[dict], count: int) -> list[list[dict]]:
"""
Generate strategy configurations from factor combinations.
@@ -1085,7 +1086,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
np.random.shuffle(configs)
return configs[: count * 2] # Generate extras
def _generate_and_evaluate_single(self, idx: int, factors: List[Dict]) -> Dict[str, Any]:
def _generate_and_evaluate_single(self, idx: int, factors: list[dict]) -> dict[str, Any]:
"""Generate and evaluate a single strategy."""
strategy_name = self._generate_strategy_name(factors, idx + 1)
@@ -1108,7 +1109,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
# by finding optimal entry/exit thresholds, signal smoothing, etc.
if self.use_optuna:
initial_status = result.get("status", "rejected")
initial_sharpe = result.get("sharpe_ratio", float('-inf'))
initial_sharpe = result.get("sharpe_ratio", float("-inf"))
logger.info(f"Running Optuna optimization for {strategy_name} (initial: {initial_status}, Sharpe={initial_sharpe:.4f})...")
optimizer = OptunaOptimizer(n_trials=self.optuna_trials)
@@ -1117,7 +1118,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
if factor_values is not None:
optimized = optimizer.optimize_strategy(result, factor_values)
optimized_sharpe = optimized.get("sharpe_ratio", float('-inf'))
optimized_sharpe = optimized.get("sharpe_ratio", float("-inf"))
optimized_status = optimized.get("status", "rejected")
best_params = optimized.get("best_params", {})
@@ -1126,14 +1127,14 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
improvement = optimized_sharpe - initial_sharpe
logger.info(
f"Optuna {'RESCUED' if optimized_status == 'accepted' and initial_status == 'rejected' else 'improved'} "
f"{strategy_name}: Sharpe {initial_sharpe:.4f}{optimized_sharpe:.4f} (+{improvement:.4f})"
f"{strategy_name}: Sharpe {initial_sharpe:.4f}{optimized_sharpe:.4f} (+{improvement:.4f})",
)
# Re-evaluate with best parameters to get comparable metrics
if best_params:
patched_code = self._patch_strategy_code(code, best_params)
re_eval = self._evaluate_with_patched_code(patched_code, strategy_name, factors)
if re_eval.get("sharpe_ratio", float('-inf')) > initial_sharpe:
if re_eval.get("sharpe_ratio", float("-inf")) > initial_sharpe:
result.update(re_eval)
result["code"] = patched_code
result["best_params"] = best_params
@@ -1142,7 +1143,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
result.pop("reason", None)
logger.info(
f"Re-evaluated {strategy_name} with best params: "
f"Sharpe {initial_sharpe:.4f}{re_eval.get('sharpe_ratio', 0):.4f}"
f"Sharpe {initial_sharpe:.4f}{re_eval.get('sharpe_ratio', 0):.4f}",
)
else:
result.update(optimized)
@@ -1162,7 +1163,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
return result
def _prepare_factor_values(self, factors: List[Dict]) -> Optional[pd.DataFrame]:
def _prepare_factor_values(self, factors: list[dict]) -> pd.DataFrame | None:
"""Prepare factor values DataFrame for Optuna optimization."""
factor_values = {}
for f in factors:
@@ -1181,7 +1182,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
return df.dropna()
return None
def _patch_strategy_code(self, code: str, params: Dict[str, Any]) -> str:
def _patch_strategy_code(self, code: str, params: dict[str, Any]) -> str:
"""Patch strategy code with Optuna's best parameters."""
import re
patched = code
@@ -1192,26 +1193,26 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
signal_window = params.get("signal_window", 3)
param_patterns = [
(r'entry_thresh\s*=\s*[\d.]+', f'entry_thresh = {entry_thresh}'),
(r'exit_thresh\s*=\s*[\d.]+', f'exit_thresh = {exit_thresh}'),
(r'window\s*=\s*\d+', f'window = {zscore_window}'),
(r'signal_window\s*=\s*\d+', f'signal_window = {signal_window}'),
(r"entry_thresh\s*=\s*[\d.]+", f"entry_thresh = {entry_thresh}"),
(r"exit_thresh\s*=\s*[\d.]+", f"exit_thresh = {exit_thresh}"),
(r"window\s*=\s*\d+", f"window = {zscore_window}"),
(r"signal_window\s*=\s*\d+", f"signal_window = {signal_window}"),
]
for pattern, replacement in param_patterns:
patched = re.sub(pattern, replacement, patched)
# Patch .rolling(N) calls for common window sizes
rolling_pattern = r'\.rolling\((\d+)\)'
rolling_pattern = r"\.rolling\((\d+)\)"
def replace_rolling(match):
val = int(match.group(1))
if val in (20, 30, 50, 100, 200):
return f'.rolling({zscore_window})'
return f".rolling({zscore_window})"
return match.group(0)
patched = re.sub(rolling_pattern, replace_rolling, patched)
return patched
def _evaluate_with_patched_code(self, patched_code: str, strategy_name: str, factors: List[Dict]) -> Dict[str, Any]:
def _evaluate_with_patched_code(self, patched_code: str, strategy_name: str, factors: list[dict]) -> dict[str, Any]:
"""Re-evaluate strategy with patched parameters using full OHLCV backtest."""
try:
factor_names = [f["factor_name"] for f in factors if f["factor_name"] != "timestamp"]
@@ -1222,7 +1223,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
factor_values[fname] = series
if not factor_values:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
common_idx = None
for name, s in factor_values.items():
@@ -1232,14 +1233,14 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
common_idx = common_idx.intersection(s.index)
if common_idx is None or len(common_idx) < 100:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
df_factors = pd.DataFrame({
name: s.reindex(common_idx) for name, s in factor_values.items()
}).dropna()
for col in df_factors.columns:
df_factors[col] = pd.to_numeric(df_factors[col], errors='coerce')
df_factors[col] = pd.to_numeric(df_factors[col], errors="coerce")
close = self.load_ohlcv_close()
if close is not None:
@@ -1247,7 +1248,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
df_factors = df_factors.dropna()
if len(df_factors) < 1000:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
if close is not None:
close = close.reindex(df_factors.index)
@@ -1256,21 +1257,21 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
try:
exec(patched_code, {"np": np, "pd": pd, "numpy": np}, local_vars)
except Exception:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
if "signal" not in local_vars:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
signal = local_vars["signal"]
from rdagent.components.backtesting.vbt_backtest import (
backtest_signal_ftmo,
DEFAULT_TXN_COST_BPS,
backtest_signal_ftmo,
)
close_for_bt = close.reindex(signal.index).ffill() if close is not None else None
if close_for_bt is None:
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
bt = backtest_signal_ftmo(
close=close_for_bt,
@@ -1278,7 +1279,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
txn_cost_bps=float(os.getenv("TXN_COST_BPS", DEFAULT_TXN_COST_BPS)),
)
if bt.get("status") != "success":
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
sharpe = bt["sharpe"]
max_dd = bt["max_drawdown"]
@@ -1307,9 +1308,9 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
except Exception as e:
logger.debug(f"Re-evaluation failed for {strategy_name}: {e}")
return {"sharpe_ratio": float('-inf'), "status": "rejected"}
return {"sharpe_ratio": float("-inf"), "status": "rejected"}
def _save_strategy(self, result: Dict[str, Any]) -> None:
def _save_strategy(self, result: dict[str, Any]) -> None:
"""Save accepted strategy to JSON file."""
timestamp = int(time.time())
safe_name = result["strategy_name"].replace("/", "_").replace(" ", "_")[:60]
@@ -1325,7 +1326,7 @@ signal = signal.rolling(window=3, min_periods=1).mean().round().astype(int)
logger.info(f"Saved strategy to {filepath}")
def get_strategy_summary(self, results: List[Dict[str, Any]]) -> Dict[str, Any]:
def get_strategy_summary(self, results: list[dict[str, Any]]) -> dict[str, Any]:
"""
Generate summary statistics from strategy generation results.
@@ -85,13 +85,16 @@ def load_and_process_one_pdf_by_azure_document_intelligence(
def load_and_process_pdfs_by_azure_document_intelligence(path: Path) -> dict[str, str]:
assert RD_AGENT_SETTINGS.azure_document_intelligence_key is not None
assert RD_AGENT_SETTINGS.azure_document_intelligence_endpoint is not None
if RD_AGENT_SETTINGS.azure_document_intelligence_key is None:
raise AssertionError("azure_document_intelligence_key must be set")
if RD_AGENT_SETTINGS.azure_document_intelligence_endpoint is None:
raise AssertionError("azure_document_intelligence_endpoint must be set")
content_dict = {}
ab_path = path.resolve()
if ab_path.is_file():
assert ".pdf" in ab_path.suffixes, "The file must be a PDF file."
if ".pdf" not in ab_path.suffixes:
raise ValueError("The file must be a PDF file.")
proc = load_and_process_one_pdf_by_azure_document_intelligence
content_dict[str(ab_path)] = proc(
ab_path,
+2 -1
View File
@@ -87,7 +87,8 @@ class ModelWsLoader(WsLoader[ModelTask, ModelFBWorkspace]):
self.path = Path(path)
def load(self, task: ModelTask) -> ModelFBWorkspace:
assert task.name is not None
if task.name is None:
raise AssertionError("task.name should not be None")
mti = ModelFBWorkspace(task)
mti.prepare()
with open(self.path / f"{task.name}.py", "r") as f:
+17 -3
View File
@@ -83,10 +83,24 @@ def import_class(class_path: str) -> Any:
Returns
-------
class of `class_path`
Raises
------
ImportError
If module or class cannot be found.
"""
module_path, class_name = class_path.rsplit(".", 1)
module = importlib.import_module(module_path)
return getattr(module, class_name)
try:
module_path, class_name = class_path.rsplit(".", 1)
except ValueError:
raise ImportError(f"Invalid class path: {class_path!r}")
try:
module = importlib.import_module(module_path)
except ModuleNotFoundError as e:
raise ImportError(f"Module not found: {module_path!r}") from e
try:
return getattr(module, class_name)
except AttributeError as e:
raise ImportError(f"Class not found: {class_name!r} in {module_path!r}") from e
class CacheSeedGen:
+2 -1
View File
@@ -541,7 +541,8 @@ class APIBackend(ABC):
**kwargs,
) -> str | list[list[float]]:
"""This function to share operation between embedding and chat completion"""
assert not (chat_completion and embedding), "chat_completion and embedding cannot be True at the same time"
if chat_completion and embedding:
raise ValueError("chat_completion and embedding cannot be True at the same time")
max_retry = LLM_SETTINGS.max_retry if LLM_SETTINGS.max_retry is not None else max_retry
timeout_count = 0
violation_count = 0
+2 -1
View File
@@ -268,7 +268,8 @@ class JsonReducer(DataReducer):
parent[key] = sampled # type: ignore # parent 是 listkey 是 index, list.__setitem__(key, sampled)
self.sampled_files.extend([self.extract_filename(i) for i in sampled])
break
assert len(self.sampled_files) > 0
if len(self.sampled_files) <= 0:
raise AssertionError("sampled_files must contain at least one file")
return data
def _find_all_lists(
@@ -56,14 +56,17 @@ sparse.save_npz(public / "test" / "X.npz", X_test)
sparse.save_npz(public / "train" / "X.npz", X_train)
df_train.to_csv(public / "train" / "ARF_12h.csv", index=False)
assert (
X_train.shape[0] == df_train.shape[0]
), f"Mismatch: X_train rows ({X_train.shape[0]}) != df_train rows ({df_train.shape[0]})"
assert (
X_test.shape[0] == df_test.shape[0]
), f"Mismatch: X_test rows ({X_test.shape[0]}) != df_test rows ({df_test.shape[0]})"
assert df_test.shape[1] == 2, "Public test set should have 2 columns"
assert df_train.shape[1] == 3, "Public train set should have 3 columns"
assert len(df_train) + len(df_test) == len(
df_label
), "Length of new_train and new_test should equal length of old_train"
if X_train.shape[0] != df_train.shape[0]:
raise ValueError(
f"Mismatch: X_train rows ({X_train.shape[0]}) != df_train rows ({df_train.shape[0]})"
)
if X_test.shape[0] != df_test.shape[0]:
raise ValueError(
f"Mismatch: X_test rows ({X_test.shape[0]}) != df_test rows ({df_test.shape[0]})"
)
if df_test.shape[1] != 2:
raise ValueError("Public test set should have 2 columns")
if df_train.shape[1] != 3:
raise ValueError("Public train set should have 3 columns")
if len(df_train) + len(df_test) != len(df_label):
raise ValueError("Length of new_train and new_test should equal length of old_train")
+2 -1
View File
@@ -320,7 +320,8 @@ class DataScienceRDLoop(RDLoop):
# only clean current workspace without affecting other loops.
for k in "direct_exp_gen", "coding", "running":
if k in prev_out and prev_out[k] is not None:
assert isinstance(prev_out[k], DSExperiment)
if not isinstance(prev_out[k], DSExperiment):
raise TypeError(f"prev_out[{k!r}] must be an instance of DSExperiment")
clean_workspace(prev_out[k].experiment_workspace.workspace_path)
# Backup the workspace (only necessary files are included)
@@ -213,7 +213,8 @@ class DSTrace(Trace[DataScienceScen, KnowledgeBase]):
self, component: COMPONENT, search_list: list[tuple[DSExperiment, ExperimentFeedback]] = []
) -> bool:
for exp, fb in search_list:
assert isinstance(exp.hypothesis, DSHypothesis), "Hypothesis should be DSHypothesis (and not None)"
if not isinstance(exp.hypothesis, DSHypothesis):
raise TypeError("Hypothesis should be DSHypothesis (and not None)")
if exp.hypothesis.component == component and fb:
return True
return False
@@ -377,7 +377,8 @@ class ExpGen2TraceAndMergeV2(ExpGen):
if DS_RD_SETTING.enable_multi_version_exp_gen:
exp_gen_version_list = DS_RD_SETTING.exp_gen_version_list.split(",")
for version in exp_gen_version_list:
assert version in ["v3", "v2", "v1"]
if version not in ["v3", "v2", "v1"]:
raise ValueError(f"version must be 'v1', 'v2', or 'v3', got {version!r}")
if len(trace.hist) == 0:
# set the proposal version for the first sub-trace
@@ -339,7 +339,8 @@ class DSProposalV1ExpGen(ExpGen):
eda_output = sota_exp.experiment_workspace.file_dict.get("EDA.md", None)
scenario_desc = trace.scen.get_scenario_all_desc(eda_output=eda_output)
assert sota_exp is not None, "SOTA experiment is not provided."
if sota_exp is None:
raise ValueError("SOTA experiment is not provided.")
last_exp = trace.last_exp()
# exp_and_feedback = trace.hist[-1]
# last_exp = exp_and_feedback[0]
@@ -445,8 +446,10 @@ class DSProposalV1ExpGen(ExpGen):
json_target_type=dict[str, dict[str, str | dict] | str],
)
)
assert "hypothesis_proposal" in resp_dict, "Hypothesis proposal not provided."
assert "task_design" in resp_dict, "Task design not provided."
if "hypothesis_proposal" not in resp_dict:
raise ValueError("Hypothesis proposal not provided.")
if "task_design" not in resp_dict:
raise ValueError("Task design not provided.")
task_class = component_info["task_class"]
hypothesis_proposal = resp_dict.get("hypothesis_proposal", {})
hypothesis = DSHypothesis(
@@ -1149,8 +1152,10 @@ You help users retrieve relevant knowledge from community discussions and public
)
response_dict = json.loads(response)
assert response_dict.get("component") in HypothesisComponent.__members__, f"Invalid component"
assert response_dict.get("hypothesis") is not None, f"Invalid hypothesis"
if response_dict.get("component") not in HypothesisComponent.__members__:
raise ValueError(f"Invalid component: {response_dict.get('component')}")
if response_dict.get("hypothesis") is None:
raise ValueError("Invalid hypothesis")
return response_dict
# END: for support llm-based hypothesis selection -----
@@ -1253,7 +1258,8 @@ You help users retrieve relevant knowledge from community discussions and public
description=task_desc,
)
assert isinstance(task, PipelineTask), f"Task {task_name} is not a PipelineTask, got {type(task)}"
if not isinstance(task, PipelineTask):
raise TypeError(f"Task {task_name} is not a PipelineTask, got {type(task)}")
# only for llm with response schema.(TODO: support for non-schema llm?)
# If the LLM provides a "packages" field (list[str]), compute runtime environment now and cache it for subsequent prompts in later loops.
if isinstance(task_dict, dict) and "packages" in task_dict and isinstance(task_dict["packages"], list):
@@ -165,7 +165,8 @@ class KGScenario(Scenario):
return data_info
def output_format(self, tag=None) -> str:
assert tag in [None, "feature", "model"]
if tag not in [None, "feature", "model"]:
raise ValueError(f"tag must be None, 'feature', or 'model', got {tag!r}")
feature_output_format = f"""The feature code should output following the format:
{T(".prompts:kg_feature_output_format").r()}"""
model_output_format = f"""The model code should output following the format:\n""" + T(
@@ -180,7 +181,8 @@ class KGScenario(Scenario):
return model_output_format
def interface(self, tag=None) -> str:
assert tag in [None, "feature", "XGBoost", "RandomForest", "LightGBM", "NN"]
if tag not in [None, "feature", "XGBoost", "RandomForest", "LightGBM", "NN"]:
raise ValueError(f"tag must be None, 'feature', 'XGBoost', 'RandomForest', 'LightGBM', or 'NN', got {tag!r}")
feature_interface = f"""The feature code should follow the interface:
{T(".prompts:kg_feature_interface").r()}"""
if tag == "feature":
@@ -195,7 +197,8 @@ class KGScenario(Scenario):
return model_interface
def simulator(self, tag=None) -> str:
assert tag in [None, "feature", "model"]
if tag not in [None, "feature", "model"]:
raise ValueError(f"tag must be None, 'feature', or 'model', got {tag!r}")
kg_feature_simulator = (
"The feature code will be sent to the simulator:\n" + T(".prompts:kg_feature_simulator").r()
+6 -1
View File
@@ -87,7 +87,12 @@ def crawl_descriptions(
content = e.get_attribute("innerHTML")
contents.append(content)
assert len(subtitles) == len(contents) + 1 and subtitles[-1] == "Citation"
if not (len(subtitles) == len(contents) + 1 and subtitles[-1] == "Citation"):
raise AssertionError(
f"Expected len(contents)+1 == len(subtitles) and last subtitle == 'Citation', "
f"got len(subtitles)={len(subtitles)}, len(contents)={len(contents)}, "
f"last subtitle={subtitles[-1]!r}"
)
for i in range(len(subtitles) - 1):
descriptions[subtitles[i]] = contents[i]
@@ -307,7 +307,8 @@ class KGHypothesisGen(FactorAndModelHypothesisGen):
class KGHypothesis2Experiment(FactorAndModelHypothesis2Experiment):
def prepare_context(self, hypothesis: Hypothesis, trace: Trace) -> Tuple[dict, bool]:
scenario = trace.scen.get_scenario_all_desc(filtered_tag="hypothesis_and_experiment")
assert isinstance(hypothesis, KGHypothesis)
if not isinstance(hypothesis, KGHypothesis):
raise TypeError("hypothesis must be an instance of KGHypothesis")
experiment_output_format = (
T("scenarios.kaggle.prompts:feature_experiment_output_format").r()
if hypothesis.action in [KG_ACTION_FEATURE_ENGINEERING, KG_ACTION_FEATURE_PROCESSING]
+146 -142
View File
@@ -1,7 +1,7 @@
import sys
import os
import logging
import os
from pathlib import Path
"""
Qlib Factor Runner - Executes factor backtests in Docker.
@@ -11,15 +11,8 @@ NOTE: The @cache_with_pickle decorator was REMOVED from develop() because:
- Docker-level caching (QlibDockerConf.enable_cache=False) is sufficient
- The pickle cache caused 240+ factor generations but ZERO Docker backtests
"""
from pathlib import Path
from typing import Optional
import pandas as pd
from pandarallel import pandarallel
pandarallel.initialize(verbose=1)
from rdagent.app.qlib_rd_loop.conf import FactorBasePropSetting
from rdagent.components.runner import CachedRunner
from rdagent.core.exception import FactorEmptyError
@@ -74,7 +67,7 @@ def _shift_daily_constant_factor_if_needed(factor_col: "pd.Series", factor_name:
logger.warning(
f"[LookAheadFix] Factor '{factor_name}' is daily-constant "
f"({fraction_constant:.0%} of days). Applying 1-day shift to remove look-ahead bias."
f"({fraction_constant:.0%} of days). Applying 1-day shift to remove look-ahead bias.",
)
# Shift: for each instrument, map daily values forward by 1 trading day
@@ -122,13 +115,13 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
"""
def calculate_information_coefficient(
self, concat_feature: pd.DataFrame, SOTA_feature_column_size: int, new_feature_columns_size: int
self, concat_feature: pd.DataFrame, SOTA_feature_column_size: int, new_feature_columns_size: int,
) -> pd.DataFrame:
res = pd.Series(index=range(SOTA_feature_column_size * new_feature_columns_size))
for col1 in range(SOTA_feature_column_size):
for col2 in range(SOTA_feature_column_size, SOTA_feature_column_size + new_feature_columns_size):
res.loc[col1 * new_feature_columns_size + col2 - SOTA_feature_column_size] = concat_feature.iloc[
:, col1
:, col1,
].corr(concat_feature.iloc[:, col2])
return res
@@ -137,16 +130,21 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# if the IC is larger than a threshold, remove the new_feature column
# return the new_feature
from pandarallel import pandarallel
pandarallel.initialize(verbose=1)
concat_feature = pd.concat([SOTA_feature, new_feature], axis=1)
IC_max = (
concat_feature.groupby("datetime")
.parallel_apply(
lambda x: self.calculate_information_coefficient(x, SOTA_feature.shape[1], new_feature.shape[1])
lambda x: self.calculate_information_coefficient(x, SOTA_feature.shape[1], new_feature.shape[1]),
)
.mean()
)
IC_max.index = pd.MultiIndex.from_product([range(SOTA_feature.shape[1]), range(new_feature.shape[1])])
IC_max = IC_max.unstack().max(axis=0)
if not hasattr(IC_max, "index"):
return new_feature
return new_feature.iloc[:, IC_max[IC_max < 0.99].index]
def develop(self, exp: QlibFactorExperiment) -> QlibFactorExperiment:
@@ -161,7 +159,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
self._ensure_results_dirs()
if exp.based_experiments and exp.based_experiments[-1].result is None:
logger.info(f"Baseline experiment execution ...")
logger.info("Baseline experiment execution ...")
exp.based_experiments[-1] = self.develop(exp.based_experiments[-1])
fbps = FactorBasePropSetting()
@@ -185,11 +183,11 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
base_exp for base_exp in exp.based_experiments if isinstance(base_exp, QlibFactorExperiment)
]
if len(sota_factor_experiments_list) > 1:
logger.info(f"SOTA factor processing ...")
logger.info("SOTA factor processing ...")
SOTA_factor = process_factor_data(sota_factor_experiments_list)
# Process the new factors data
logger.info(f"New factor processing ...")
logger.info("New factor processing ...")
new_factors = process_factor_data(exp)
if new_factors.empty:
@@ -200,7 +198,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
new_factors = self.deduplicate_new_factors(SOTA_factor, new_factors)
if new_factors.empty:
raise FactorEmptyError(
"The factors generated in this round are highly similar to the previous factors. Please change the direction for creating new factors."
"The factors generated in this round are highly similar to the previous factors. Please change the direction for creating new factors.",
)
combined_factors = pd.concat([SOTA_factor, new_factors], axis=1).dropna()
else:
@@ -211,7 +209,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
combined_factors = combined_factors.loc[:, ~combined_factors.columns.duplicated(keep="last")]
new_columns = pd.MultiIndex.from_product([["feature"], combined_factors.columns])
combined_factors.columns = new_columns
logger.info(f"Factor data processing completed.")
logger.info("Factor data processing completed.")
num_features = len(exp.base_features) + len(combined_factors.columns)
@@ -230,10 +228,10 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
sota_model_exp = base_exp
exist_sota_model_exp = True
break
logger.info(f"Experiment execution ...")
logger.info("Experiment execution ...")
if exist_sota_model_exp:
exp.experiment_workspace.inject_files(
**{"model.py": sota_model_exp.sub_workspace_list[0].file_dict["model.py"]}
**{"model.py": sota_model_exp.sub_workspace_list[0].file_dict["model.py"]},
)
sota_training_hyperparameters = sota_model_exp.sub_tasks[0].training_hyperparameters
if sota_training_hyperparameters:
@@ -244,19 +242,19 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
"early_stop": str(sota_training_hyperparameters.get("early_stop", 10)),
"batch_size": str(sota_training_hyperparameters.get("batch_size", 256)),
"weight_decay": str(sota_training_hyperparameters.get("weight_decay", 0.0001)),
}
},
)
sota_model_type = sota_model_exp.sub_tasks[0].model_type
if sota_model_type == "TimeSeries":
env_to_use.update(
{"dataset_cls": "TSDatasetH", "num_features": num_features, "step_len": 20, "num_timesteps": 20}
{"dataset_cls": "TSDatasetH", "num_features": num_features, "step_len": 20, "num_timesteps": 20},
)
elif sota_model_type == "Tabular":
env_to_use.update({"dataset_cls": "DatasetH", "num_features": num_features})
# model + combined factors
result, stdout = exp.experiment_workspace.execute(
qlib_config_name="conf_combined_factors_sota_model.yaml", run_env=env_to_use
qlib_config_name="conf_combined_factors_sota_model.yaml", run_env=env_to_use,
)
else:
# LGBM + combined factors
@@ -265,7 +263,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
run_env=env_to_use,
)
else:
logger.info(f"Experiment execution ...")
logger.info("Experiment execution ...")
if exp.base_feature_codes:
factors = process_factor_data(exp)
factors = factors.sort_index()
@@ -275,7 +273,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
target_path = exp.experiment_workspace.workspace_path / "combined_factors_df.parquet"
# Save the combined factors to the workspace
factors.to_parquet(target_path, engine="pyarrow")
logger.info(f"Factor data processing completed.")
logger.info("Factor data processing completed.")
result, stdout = exp.experiment_workspace.execute(
qlib_config_name="conf_combined_factors.yaml",
run_env=env_to_use,
@@ -288,10 +286,10 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Handle Qlib Docker backtest failure gracefully
if result is None:
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
logger.warning(
f"Qlib Docker backtest returned None for '{factor_name}'. "
f"Attempting direct factor evaluation..."
f"Attempting direct factor evaluation...",
)
# Try to compute metrics directly from the factor's result.h5
@@ -303,7 +301,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
else:
logger.error(
f"Both Qlib Docker backtest and direct evaluation failed for '{factor_name}'. "
f"Skipping this factor and continuing."
f"Skipping this factor and continuing.",
)
# Save failed run info for debugging
self._save_failed_run(exp, stdout, error_type="result_none")
@@ -321,7 +319,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
if validation_result.get("has_issues"):
logger.warning(
f"Result validation warnings for factor '{getattr(exp.hypothesis, 'hypothesis', 'unknown')}': "
f"{validation_result['warnings']}"
f"{validation_result['warnings']}",
)
# Save warning info for debugging
self._save_failed_run(exp, stdout, error_type="validation_warnings", validation=validation_result)
@@ -372,43 +370,43 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
details = {}
factor_name = "unknown"
if hasattr(exp, 'hypothesis') and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
if hasattr(exp, "hypothesis") and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
if isinstance(result, pd.Series):
# Check IC
ic_value = result.get('IC', None)
details['ic_raw'] = ic_value
ic_value = result.get("IC", None)
details["ic_raw"] = ic_value
if ic_value is None or (isinstance(ic_value, float) and (ic_value != ic_value)): # NaN check
warnings.append("IC is None/NaN — factor has no predictive power")
else:
try:
ic_float = float(ic_value)
details['ic'] = ic_float
details["ic"] = ic_float
if abs(ic_float) < 0.001:
warnings.append(
f"IC is near zero ({ic_float:.6f}) — factor may not predict returns"
f"IC is near zero ({ic_float:.6f}) — factor may not predict returns",
)
except (ValueError, TypeError):
warnings.append(f"IC value is not numeric: {ic_value}")
# Check positions (1day.pos)
pos_value = result.get('1day.pos', None)
details['positions_raw'] = pos_value
pos_value = result.get("1day.pos", None)
details["positions_raw"] = pos_value
if pos_value is not None:
try:
pos_float = float(pos_value)
details['positions'] = pos_float
details["positions"] = pos_float
if pos_float == 0:
warnings.append(
"1day.pos == 0 — model opened ZERO positions (stayed neutral). "
"Possible causes: (1) topk too high for single-asset, "
"(2) signal threshold too restrictive, (3) no valid predictions"
"(2) signal threshold too restrictive, (3) no valid predictions",
)
elif pos_float < 10:
warnings.append(
f"1day.pos = {pos_float:.0f} — very few positions opened. "
f"Check signal threshold and topk settings"
f"Check signal threshold and topk settings",
)
except (ValueError, TypeError):
pass # pos might be a string
@@ -416,24 +414,24 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Check if result is essentially empty (all values None or NaN)
non_null_count = result.notna().sum()
total_count = len(result)
details['non_null_metrics'] = int(non_null_count)
details['total_metrics'] = int(total_count)
details["non_null_metrics"] = int(non_null_count)
details["total_metrics"] = int(total_count)
if non_null_count < 3:
warnings.append(
f"Result has only {non_null_count}/{total_count} non-null metrics — "
f"backtest likely produced empty results"
f"backtest likely produced empty results",
)
# Check for key metrics
required_metrics = ['IC', '1day.excess_return_with_cost.shar', '1day.pos']
required_metrics = ["IC", "1day.excess_return_with_cost.shar", "1day.pos"]
for metric_name in required_metrics:
val = result.get(metric_name, None)
details[f'has_{metric_name}'] = val is not None
details[f"has_{metric_name}"] = val is not None
elif isinstance(result, dict):
# Dict-based result validation
ic_value = result.get('IC', result.get('ic', None))
details['ic_raw'] = ic_value
ic_value = result.get("IC", result.get("ic", None))
details["ic_raw"] = ic_value
if ic_value is None:
warnings.append("IC is None — factor has no predictive power")
@@ -443,7 +441,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
"details": details,
}
def _evaluate_factor_directly(self, exp, stdout: str) -> Optional[pd.Series]:
def _evaluate_factor_directly(self, exp, stdout: str) -> pd.Series | None:
"""
Evaluate factor directly from its result.h5 file when Qlib Docker fails.
@@ -475,7 +473,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
workspace_path = None
if exp.sub_workspace_list:
for ws in exp.sub_workspace_list:
if ws is not None and hasattr(ws, 'workspace_path'):
if ws is not None and hasattr(ws, "workspace_path"):
candidate = ws.workspace_path / "result.h5"
if candidate.exists():
workspace_path = ws.workspace_path
@@ -579,7 +577,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
logger.info(
f"Direct evaluation: IC={ic:.6f}, Sharpe={sharpe:.4f}, "
f"AnnRet={annualized_return:.4f}%, WR={win_rate:.2%}"
f"AnnRet={annualized_return:.4f}%, WR={win_rate:.2%}",
)
return result
@@ -588,7 +586,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
return None
def _save_failed_run(self, exp, stdout: str, error_type: str = "unknown",
validation: Optional[dict] = None) -> None:
validation: dict | None = None) -> None:
"""
Save failed run information to results/failed_runs.json for debugging.
@@ -615,20 +613,20 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Get factor name
factor_name = "unknown"
if hasattr(exp, 'hypothesis') and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
if hasattr(exp, "hypothesis") and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
# Build failed run record
failed_record = {
"timestamp": datetime.now().isoformat(),
"factor_name": factor_name,
"error_type": error_type,
"stdout": stdout if stdout else "(empty)",
"stdout": stdout or "(empty)",
"validation": validation,
"experiment_details": {
"base_features": list(getattr(exp, 'base_features', {}).keys()) if hasattr(exp, 'base_features') else [],
"hypothesis": getattr(exp.hypothesis, 'hypothesis', str(getattr(exp, 'hypothesis', 'N/A')))
if hasattr(exp, 'hypothesis') else "N/A",
"base_features": list(getattr(exp, "base_features", {}).keys()) if hasattr(exp, "base_features") else [],
"hypothesis": getattr(exp.hypothesis, "hypothesis", str(getattr(exp, "hypothesis", "N/A")))
if hasattr(exp, "hypothesis") else "N/A",
},
}
@@ -651,11 +649,11 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
failed_file.write_text(
json.dumps(existing_records, indent=2, default=str, ensure_ascii=False),
encoding="utf-8"
encoding="utf-8",
)
logger.info(
f"Failed run saved: {factor_name} (type={error_type}) "
f"{failed_file}"
f"{failed_file}",
)
except Exception as e:
@@ -678,23 +676,23 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
containing metric names like 'IC', '1day.excess_return_with_cost.shar', etc.
"""
try:
import json
import pandas as pd
from pathlib import Path
import pandas as pd
from rdagent.components.backtesting import ResultsDatabase
# Get factor name: prefer hypothesis, fallback to result Series 'factor_name' key
factor_name = "unknown"
if hasattr(exp, 'hypothesis') and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
if factor_name == 'unknown' and isinstance(result, pd.Series) and 'factor_name' in result.index:
factor_name = str(result['factor_name'])
if hasattr(exp, "hypothesis") and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
if factor_name == "unknown" and isinstance(result, pd.Series) and "factor_name" in result.index:
factor_name = str(result["factor_name"])
# Check if already rejected by protection
if getattr(exp, 'rejected_by_protection', False):
if getattr(exp, "rejected_by_protection", False):
logger.info(
f"Factor rejected by protection, skipping DB save: "
f"{getattr(exp, 'protection_reason', 'unknown')}"
f"{getattr(exp, 'protection_reason', 'unknown')}",
)
return
@@ -710,47 +708,47 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Extract metrics from result (pd.Series from qlib_res.csv)
metrics = {}
if isinstance(result, pd.Series):
metrics['ic'] = self._safe_float(result.get('IC', None))
metrics['sharpe_ratio'] = self._safe_float(
result.get('1day.excess_return_with_cost.shar',
result.get('1day.excess_return_with_cost.sharpe', None))
metrics["ic"] = self._safe_float(result.get("IC", None))
metrics["sharpe_ratio"] = self._safe_float(
result.get("1day.excess_return_with_cost.shar",
result.get("1day.excess_return_with_cost.sharpe", None)),
)
metrics['annualized_return'] = self._safe_float(
result.get('1day.excess_return_with_cost.annualized_return', None)
metrics["annualized_return"] = self._safe_float(
result.get("1day.excess_return_with_cost.annualized_return", None),
)
metrics['max_drawdown'] = self._safe_float(
result.get('1day.excess_return_with_cost.max_drawdown', None)
metrics["max_drawdown"] = self._safe_float(
result.get("1day.excess_return_with_cost.max_drawdown", None),
)
metrics['win_rate'] = self._safe_float(result.get('win_rate', None))
metrics['information_ratio'] = self._safe_float(
result.get('1day.excess_return_with_cost.information_ratio', None)
metrics["win_rate"] = self._safe_float(result.get("win_rate", None))
metrics["information_ratio"] = self._safe_float(
result.get("1day.excess_return_with_cost.information_ratio", None),
)
metrics['volatility'] = self._safe_float(
result.get('1day.excess_return_with_cost.std',
result.get('1day.excess_return_with_cost.volatility', None))
metrics["volatility"] = self._safe_float(
result.get("1day.excess_return_with_cost.std",
result.get("1day.excess_return_with_cost.volatility", None)),
)
# Store raw metrics for JSON export
metrics['raw_metrics'] = result.to_dict()
metrics["raw_metrics"] = result.to_dict()
elif isinstance(result, dict):
metrics['ic'] = self._safe_float(result.get('IC', result.get('ic', None)))
metrics['sharpe_ratio'] = self._safe_float(
result.get('sharpe', result.get('sharpe_ratio', None))
metrics["ic"] = self._safe_float(result.get("IC", result.get("ic", None)))
metrics["sharpe_ratio"] = self._safe_float(
result.get("sharpe", result.get("sharpe_ratio", None)),
)
metrics['annualized_return'] = self._safe_float(result.get('annualized_return', None))
metrics['max_drawdown'] = self._safe_float(result.get('max_drawdown', None))
metrics['win_rate'] = self._safe_float(result.get('win_rate', None))
metrics['information_ratio'] = None
metrics['volatility'] = None
metrics['raw_metrics'] = result
metrics["annualized_return"] = self._safe_float(result.get("annualized_return", None))
metrics["max_drawdown"] = self._safe_float(result.get("max_drawdown", None))
metrics["win_rate"] = self._safe_float(result.get("win_rate", None))
metrics["information_ratio"] = None
metrics["volatility"] = None
metrics["raw_metrics"] = result
# Result validation before saving (warnings, not blocking)
self._log_result_warnings(factor_name, result, metrics)
# Only save if we have at least IC or Sharpe
if metrics.get('ic') is None and metrics.get('sharpe_ratio') is None:
if metrics.get("ic") is None and metrics.get("sharpe_ratio") is None:
logger.warning(
f"No valid IC/Sharpe for factor '{factor_name}', skipping DB save. "
f"IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}"
f"IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}",
)
return
@@ -761,19 +759,19 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
db_file = db_path / "backtest_results.db"
# Parallel run isolation: use run-specific subdirectory if PARALLEL_RUN_ID is set
run_id = os.getenv("PARALLEL_RUN_ID", "0")
if run_id != "0":
parallel_run_id = os.getenv("PARALLEL_RUN_ID", "0")
if parallel_run_id != "0":
# For parallel runs, save to isolated results directory
isolated_db_path = project_root / "results" / "runs" / f"run{run_id}" / "db"
isolated_db_path = project_root / "results" / "runs" / f"run{parallel_run_id}" / "db"
isolated_db_path.mkdir(parents=True, exist_ok=True)
db_file = isolated_db_path / "backtest_results.db"
# Save to database
db = ResultsDatabase(db_path=str(db_file))
run_id = db.add_backtest(factor_name=factor_name[:100], metrics=metrics)
db_run_id = db.add_backtest(factor_name=factor_name[:100], metrics=metrics)
logger.info(
f"Factor result saved to DB: {factor_name[:60]} "
f"(IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}, run_id={run_id})"
f"(IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}, run_id={db_run_id})"
)
# Extract factor code and description from experiment
@@ -781,10 +779,10 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Also write a JSON summary to results/factors/ for file-based access
self._save_factor_json(
factor_name, metrics, run_id,
factor_name, metrics, db_run_id,
factor_code=factor_code,
factor_description=factor_description,
exp=exp
exp=exp,
)
# Save factor values as parquet for strategy building
@@ -796,7 +794,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
import traceback
logger.error(
f"Database save failed for factor '{getattr(exp.hypothesis, 'hypothesis', 'unknown')}': {e}\n"
f"Traceback: {traceback.format_exc()}"
f"Traceback: {traceback.format_exc()}",
)
def _save_factor_json(self, factor_name: str, metrics: dict, run_id: int,
@@ -907,14 +905,14 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
factor_description = match.group(1).strip()[:500]
else:
# Try comments
lines = factor_code.split('\n')
lines = factor_code.split("\n")
desc_lines = []
for line in lines[:20]:
stripped = line.strip()
if stripped.startswith('#') and not stripped.startswith('#!'):
if stripped.startswith("#") and not stripped.startswith("#!"):
desc_lines.append(stripped[1:].strip())
if desc_lines:
factor_description = ' '.join(desc_lines)[:500]
factor_description = " ".join(desc_lines)[:500]
return factor_code, factor_description
@@ -926,8 +924,9 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
the complete backtest range (not just the debug 2024 subset).
"""
import os as _os
import subprocess
import shutil
import subprocess
import sys
import tempfile
try:
@@ -935,7 +934,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
workspace_path = None
if exp.sub_workspace_list:
for ws in exp.sub_workspace_list:
if ws is not None and hasattr(ws, 'workspace_path'):
if ws is not None and hasattr(ws, "workspace_path"):
fp = ws.workspace_path / "factor.py"
if fp.exists():
workspace_path = ws.workspace_path
@@ -967,12 +966,17 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
shutil.copy(str(full_data), str(tmp / "intraday_pv.h5"))
ret = subprocess.run(
["sys.executable", "factor.py"],
[sys.executable, "factor.py"],
cwd=str(tmp),
capture_output=True,
timeout=300,
check=False,
)
if ret.returncode != 0:
logger.warning(
f"Full-data factor run failed (exit {ret.returncode}): "
f"{ret.stderr[:500] if ret.stderr else '(no stderr)'}"
)
# Fall back to debug-data result if full-data run fails
result_h5 = workspace_path / "result.h5"
if not result_h5.exists():
@@ -1023,7 +1027,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
warnings_list = []
# Check IC
ic = metrics.get('ic')
ic = metrics.get("ic")
if ic is None:
warnings_list.append("IC is None — factor has no predictive power")
elif abs(ic) < 0.001:
@@ -1031,7 +1035,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
# Check positions (1day.pos) — CRITICAL for EURUSD
if isinstance(result, pd.Series):
pos_value = result.get('1day.pos', None)
pos_value = result.get("1day.pos", None)
if pos_value is not None:
try:
pos_float = float(pos_value)
@@ -1039,23 +1043,23 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
warnings_list.append(
"WARNING: 1day.pos == 0 — ZERO positions opened! "
"Model stayed completely neutral. Check Qlib config: "
"ensure topk=1 and market=eurusd for single-asset trading."
"ensure topk=1 and market=eurusd for single-asset trading.",
)
elif pos_float < 10:
warnings_list.append(
f"Low position count: 1day.pos = {pos_float:.0f}"
f"model traded very rarely"
f"model traded very rarely",
)
except (ValueError, TypeError):
pass
# Check Sharpe
sharpe = metrics.get('sharpe_ratio')
sharpe = metrics.get("sharpe_ratio")
if sharpe is not None and abs(sharpe) < 0.1:
warnings_list.append(f"Sharpe near zero ({sharpe:.4f}) — no risk-adjusted edge")
# Check max drawdown
mdd = metrics.get('max_drawdown')
mdd = metrics.get("max_drawdown")
if mdd is not None and mdd < -0.5:
warnings_list.append(f"Extreme drawdown: {mdd:.2%} — high risk factor")
@@ -1070,7 +1074,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
return None
try:
f = float(value)
if pd.isna(f) or f == float('inf') or f == float('-inf'):
if pd.isna(f) or f == float("inf") or f == float("-inf"):
return None
return f
except (ValueError, TypeError):
@@ -1113,7 +1117,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
if protection_result.should_block:
logger.warning(
f"Factor {factor_name} rejected by protection manager: {protection_result.reason}"
f"Factor {factor_name} rejected by protection manager: {protection_result.reason}",
)
# Mark factor as rejected by protection
exp.rejected_by_protection = True
@@ -1138,8 +1142,8 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
from pathlib import Path
factor_name = "unknown"
if hasattr(exp, 'hypothesis') and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
if hasattr(exp, "hypothesis") and exp.hypothesis is not None:
factor_name = getattr(exp.hypothesis, "hypothesis", "unknown")
# Build log entry
log_entry = {
@@ -1151,42 +1155,42 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
"annualized_return": None,
"max_drawdown": None,
"win_rate": None,
"rejected_by_protection": getattr(exp, 'rejected_by_protection', False),
"protection_reason": getattr(exp, 'protection_reason', None),
"rejected_by_protection": getattr(exp, "rejected_by_protection", False),
"protection_reason": getattr(exp, "protection_reason", None),
}
# Extract metrics if available
if result is not None:
if hasattr(result, 'get'): # pd.Series or dict
ic_val = result.get('IC', result.get('ic', None))
log_entry['ic'] = self._safe_float(ic_val) if ic_val is not None else None
if hasattr(result, "get"): # pd.Series or dict
ic_val = result.get("IC", result.get("ic", None))
log_entry["ic"] = self._safe_float(ic_val) if ic_val is not None else None
sharpe_val = result.get('1day.excess_return_with_cost.shar',
result.get('1day.excess_return_with_cost.sharpe',
result.get('sharpe', None)))
log_entry['sharpe'] = self._safe_float(sharpe_val) if sharpe_val is not None else None
sharpe_val = result.get("1day.excess_return_with_cost.shar",
result.get("1day.excess_return_with_cost.sharpe",
result.get("sharpe", None)))
log_entry["sharpe"] = self._safe_float(sharpe_val) if sharpe_val is not None else None
ann_ret = result.get('1day.excess_return_with_cost.annualized_return',
result.get('annualized_return', None))
log_entry['annualized_return'] = self._safe_float(ann_ret) if ann_ret is not None else None
ann_ret = result.get("1day.excess_return_with_cost.annualized_return",
result.get("annualized_return", None))
log_entry["annualized_return"] = self._safe_float(ann_ret) if ann_ret is not None else None
mdd = result.get('1day.excess_return_with_cost.max_drawdown',
result.get('max_drawdown', None))
log_entry['max_drawdown'] = self._safe_float(mdd) if mdd is not None else None
mdd = result.get("1day.excess_return_with_cost.max_drawdown",
result.get("max_drawdown", None))
log_entry["max_drawdown"] = self._safe_float(mdd) if mdd is not None else None
wr = result.get('win_rate', None)
log_entry['win_rate'] = self._safe_float(wr) if wr is not None else None
wr = result.get("win_rate", None)
log_entry["win_rate"] = self._safe_float(wr) if wr is not None else None
# Determine status
if log_entry['ic'] is not None or log_entry['sharpe'] is not None:
log_entry['status'] = "success"
elif getattr(exp, 'rejected_by_protection', False):
log_entry['status'] = "rejected_protection"
if log_entry["ic"] is not None or log_entry["sharpe"] is not None:
log_entry["status"] = "success"
elif getattr(exp, "rejected_by_protection", False):
log_entry["status"] = "rejected_protection"
else:
log_entry['status'] = "no_valid_metrics"
log_entry["status"] = "no_valid_metrics"
else:
log_entry['status'] = "execution_failed"
log_entry['reason'] = "Result was None"
log_entry["status"] = "execution_failed"
log_entry["reason"] = "Result was None"
# Write to results/logs/
try:
@@ -1209,7 +1213,7 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
logger.info(
f"Run log written for '{factor_name[:50]}': "
f"status={log_entry['status']}, IC={log_entry['ic']}, Sharpe={log_entry['sharpe']}"
f"status={log_entry['status']}, IC={log_entry['ic']}, Sharpe={log_entry['sharpe']}",
)
except Exception as e:
logger.error(f"Failed to write run log: {e}")
@@ -203,12 +203,14 @@ class QlibModelRunner(CachedRunner[QlibModelExperiment]):
# Save to database
db = ResultsDatabase()
run_id = db.add_backtest(factor_name=factor_name[:100], metrics=metrics)
logger.info(
f"Model result saved to DB: {factor_name[:50]} "
f"(IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}, run_id={run_id})"
)
db.close()
try:
run_id = db.add_backtest(factor_name=factor_name[:100], metrics=metrics)
logger.info(
f"Model result saved to DB: {factor_name[:50]} "
f"(IC={metrics.get('ic')}, Sharpe={metrics.get('sharpe_ratio')}, run_id={run_id})"
)
finally:
db.close()
except Exception as e:
logger.warning(f"Database save failed for model {getattr(exp.hypothesis, 'hypothesis', 'unknown')}: {e}")
@@ -56,7 +56,8 @@ class QlibQuantScenario(Scenario):
)
def background(self, tag=None) -> str:
assert tag in [None, "factor", "model"]
if tag not in [None, "factor", "model"]:
raise ValueError(f"tag must be None, 'factor', or 'model', got {tag!r}")
quant_background = "The background of the scenario is as follows:\n" + T(".prompts:qlib_quant_background").r(
runtime_environment=self.get_runtime_environment(),
)
@@ -83,7 +84,8 @@ class QlibQuantScenario(Scenario):
return self._source_data
def output_format(self, tag=None) -> str:
assert tag in [None, "factor", "model"]
if tag not in [None, "factor", "model"]:
raise ValueError(f"tag must be None, 'factor', or 'model', got {tag!r}")
factor_output_format = (
"The factor code should output the following format:\n" + T(".prompts:qlib_factor_output_format").r()
)
@@ -99,7 +101,8 @@ class QlibQuantScenario(Scenario):
return model_output_format
def interface(self, tag=None) -> str:
assert tag in [None, "factor", "model"]
if tag not in [None, "factor", "model"]:
raise ValueError(f"tag must be None, 'factor', or 'model', got {tag!r}")
factor_interface = (
"The factor code should be written in the following interface:\n" + T(".prompts:qlib_factor_interface").r()
)
@@ -115,7 +118,8 @@ class QlibQuantScenario(Scenario):
return model_interface
def simulator(self, tag=None) -> str:
assert tag in [None, "factor", "model"]
if tag not in [None, "factor", "model"]:
raise ValueError(f"tag must be None, 'factor', or 'model', got {tag!r}")
factor_simulator = "The factor code will be sent to the simulator:\n" + T(".prompts:qlib_factor_simulator").r()
model_simulator = "The model code will be sent to the simulator:\n" + T(".prompts:qlib_model_simulator").r()
@@ -185,7 +189,8 @@ class QlibQuantScenario(Scenario):
return common_description(action) + interface(action) + output(action) + simulator(action)
def get_runtime_environment(self, tag: str = None) -> str:
assert tag in [None, "factor", "model"]
if tag not in [None, "factor", "model"]:
raise ValueError(f"tag must be None, 'factor', or 'model', got {tag!r}")
if tag is None or tag == "factor":
# Use factor env to get the runtime environment
@@ -1,8 +1,7 @@
import logging
import json
import logging
import os
import random
from typing import Tuple
from rdagent.app.qlib_rd_loop.conf import QUANT_PROP_SETTING
from rdagent.components.proposal import FactorAndModelHypothesisGen
@@ -42,7 +41,7 @@ class QlibQuantHypothesis(Hypothesis):
action: str,
) -> None:
super().__init__(
hypothesis, reason, concise_reason, concise_observation, concise_justification, concise_knowledge
hypothesis, reason, concise_reason, concise_observation, concise_justification, concise_knowledge,
)
self.action = action
@@ -54,10 +53,10 @@ Reason: {self.reason}
class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
def __init__(self, scen: Scenario) -> Tuple[dict, bool]:
def __init__(self, scen: Scenario) -> None:
super().__init__(scen)
def prepare_context(self, trace: Trace) -> Tuple[dict, bool]:
def prepare_context(self, trace: Trace) -> tuple[dict, bool]:
# ========= Bandit ==========
if QUANT_PROP_SETTING.action_selection == "bandit":
@@ -85,7 +84,7 @@ class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
last_hypothesis_and_feedback = (
T("scenarios.qlib.prompts:last_hypothesis_and_feedback").r(
experiment=trace.hist[-1][0], feedback=trace.hist[-1][1]
experiment=trace.hist[-1][0], feedback=trace.hist[-1][1],
)
if len(trace.hist) > 0
else "No previous hypothesis and feedback available since it's the first round."
@@ -195,7 +194,7 @@ class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
for i in range(len(trace.hist) - 1, -1, -1):
if trace.hist[i][0].hypothesis.action == action:
last_hypothesis_and_feedback = T("scenarios.qlib.prompts:last_hypothesis_and_feedback").r(
experiment=trace.hist[i][0], feedback=trace.hist[i][1]
experiment=trace.hist[i][0], feedback=trace.hist[i][1],
)
break
@@ -204,7 +203,7 @@ class QlibQuantHypothesisGen(FactorAndModelHypothesisGen):
for i in range(len(trace.hist) - 1, -1, -1):
if trace.hist[i][0].hypothesis.action == "model" and trace.hist[i][1].decision is True:
sota_hypothesis_and_feedback = T("scenarios.qlib.prompts:sota_hypothesis_and_feedback").r(
experiment=trace.hist[i][0], feedback=trace.hist[i][1]
experiment=trace.hist[i][0], feedback=trace.hist[i][1],
)
break
+22 -19
View File
@@ -436,13 +436,10 @@ class Env(Generic[ASpecificEnvConf]):
else:
timeout_cmd = f"timeout --kill-after=10 {self.conf.running_timeout_period} {entry}"
entry_add_timeout = (
f"/bin/sh -c '" # start of the sh command
+ f"{timeout_cmd}; entry_exit_code=$?; "
"/bin/sh -c '" # start of the sh command
+ timeout_cmd.replace("'", "'\\''") + "; entry_exit_code=$?; "
+ (
f"{_get_chmod_cmd(self.conf.mount_path)}; "
# We don't have to change the permission of the cache and input folder to remove it
# + f"if [ -d {self.conf.mount_path}/cache ]; then chmod 777 {self.conf.mount_path}/cache; fi; " +
# f"if [ -d {self.conf.mount_path}/input ]; then chmod 777 {self.conf.mount_path}/input; fi; "
if isinstance(self.conf, DockerConf)
else ""
)
@@ -620,7 +617,8 @@ class LocalEnv(Env[ASpecificLocalConf]):
for lp, rp in running_extra_volume.items():
volumes[lp] = rp
assert local_path is not None, "local_path should not be None"
if local_path is None:
raise ValueError("local_path should not be None")
volumes = normalize_volumes(volumes, local_path)
@contextlib.contextmanager
@@ -925,7 +923,11 @@ def _prepare_conda_env(env_name: str, requirements_file: Path, python_version: s
"""
# 1. Create conda environment if not exists
env_list = subprocess.run(["conda", "env", "list"], capture_output=True, text=True, check=False)
env_exists = any(line.split()[0] == env_name for line in env_list.stdout.splitlines() if line and not line.startswith("#"))
env_exists = any(
line.split()[0] == env_name
for line in env_list.stdout.splitlines()
if line and not line.startswith("#") and len(line.split()) > 0
)
if not env_exists:
print(f"[yellow]Creating conda env '{env_name}' (Python {python_version})...[/yellow]")
subprocess.check_call(["conda", "create", "-y", "-n", env_name, f"python={python_version}"])
@@ -1191,7 +1193,7 @@ class DockerEnv(Env[DockerConf]):
with Progress(SpinnerColumn(), TextColumn("{task.description}")) as p:
task = p.add_task("[cyan]Building image...")
for part in resp_stream:
lines = part.decode("utf-8").split("\r\n")
lines = part.decode("utf-8", errors="replace").split("\r\n")
for line in lines:
if line.strip():
status_dict = json.loads(line)
@@ -1472,7 +1474,8 @@ class DockerEnv(Env[DockerConf]):
cpu_count=self.conf.cpu_count, # Set CPU limit
**self._gpu_kwargs(client),
)
assert container is not None # Ensure container was created successfully
if container is None:
raise AssertionError("Docker container was not created successfully")
logs = container.logs(stream=True)
print(Rule("[bold green]Docker Logs Begin[/bold green]", style="dark_orange"))
table = Table(title="Run Info", show_header=False)
@@ -1522,8 +1525,8 @@ class DockerEnv(Env[DockerConf]):
class QTDockerEnv(DockerEnv):
"""Qlib Torch Docker"""
def __init__(self, conf: DockerConf = QlibDockerConf()):
super().__init__(conf)
def __init__(self, conf: DockerConf | None = None):
super().__init__(conf if conf is not None else QlibDockerConf())
def prepare(self, *args, **kwargs) -> None: # type: ignore[no-untyped-def]
"""
@@ -1542,15 +1545,15 @@ class QTDockerEnv(DockerEnv):
class KGDockerEnv(DockerEnv):
"""Kaggle Competition Docker"""
def __init__(self, competition: str | None = None, conf: DockerConf = KGDockerConf()):
super().__init__(conf)
def __init__(self, competition: str | None = None, conf: DockerConf | None = None):
super().__init__(conf if conf is not None else KGDockerConf())
class MLEBDockerEnv(DockerEnv):
"""MLEBench Docker"""
def __init__(self, conf: DockerConf = MLEBDockerConf()):
super().__init__(conf)
def __init__(self, conf: DockerConf | None = None):
super().__init__(conf if conf is not None else MLEBDockerConf())
class FTDockerEnv(DockerEnv):
@@ -1566,8 +1569,8 @@ class FTDockerEnv(DockerEnv):
export FT_DOCKER_save_logs_to_file=false # disable log file
"""
def __init__(self, conf: DockerConf = FTDockerConf()):
super().__init__(conf)
def __init__(self, conf: DockerConf | None = None):
super().__init__(conf if conf is not None else FTDockerConf())
class BenchmarkDockerEnv(DockerEnv):
@@ -1584,5 +1587,5 @@ class BenchmarkDockerEnv(DockerEnv):
export BENCHMARK_DOCKER_terminal_tail_lines=100 # show last 100 lines
"""
def __init__(self, conf: DockerConf = BenchmarkDockerConf()):
super().__init__(conf)
def __init__(self, conf: DockerConf | None = None):
super().__init__(conf if conf is not None else BenchmarkDockerConf())
+23 -24
View File
@@ -15,19 +15,19 @@ import multiprocessing.queues
import os
import pickle
from collections import defaultdict
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable, Optional, Union, cast
from typing import Any, cast
import psutil
from tqdm.auto import tqdm
from rdagent.core.conf import RD_AGENT_SETTINGS
from rdagent.log import rdagent_logger as logger
from rdagent.log.conf import LOG_SETTINGS
from rdagent.log.timer import RD_Agent_TIMER_wrapper, RDAgentTimer
from rdagent.utils.workflow.tracking import WorkflowTracker
from tqdm.auto import tqdm
class LoopMeta(type):
@@ -98,7 +98,7 @@ class LoopBase:
skip_loop_error: tuple[type[BaseException], ...] = () # you can define a list of error that will skip current loop
skip_loop_error_stepname: str | None = None # if skip_loop_error exception happens, what's the next step to work on
withdraw_loop_error: tuple[
type[BaseException], ...
type[BaseException], ...,
] = () # you can define a list of error that will withdraw current loop
EXCEPTION_KEY = "_EXCEPTION"
@@ -129,8 +129,8 @@ class LoopBase:
self.tracker = WorkflowTracker(self) # Initialize tracker with this LoopBase instance
# progress control
self.loop_n: Optional[int] = None # remain loop count
self.step_n: Optional[int] = None # remain step count
self.loop_n: int | None = None # remain loop count
self.step_n: int | None = None # remain step count
self.semaphores: dict[str, asyncio.Semaphore] = {}
@@ -169,7 +169,7 @@ class LoopBase:
self._pbar.close()
del self._pbar
def _check_exit_conditions_on_step(self, loop_id: Optional[int] = None, step_id: Optional[int] = None) -> None:
def _check_exit_conditions_on_step(self, loop_id: int | None = None, step_id: int | None = None) -> None:
"""Check if the loop should continue or terminate.
Raises
@@ -188,8 +188,7 @@ class LoopBase:
if self.timer.is_timeout():
logger.warning("Timeout, exiting the loop.")
raise self.LoopTerminationError("Timer timeout")
else:
logger.info(f"Timer remaining time: {self.timer.remain_time()}")
logger.info(f"Timer remaining time: {self.timer.remain_time()}")
async def _run_step(self, li: int, force_subproc: bool = False) -> None:
"""Execute a single step (next unrun step) in the workflow (async version with force_subproc option).
@@ -217,7 +216,7 @@ class LoopBase:
with logger.tag(f"Loop_{li}.{name}"):
start = datetime.now(timezone.utc)
func: Callable[..., Any] = cast(Callable[..., Any], getattr(self, name))
func: Callable[..., Any] = cast("Callable[..., Any]", getattr(self, name))
next_step_idx = si + 1
step_forward = True
@@ -233,15 +232,14 @@ class LoopBase:
# Using deepcopy is to avoid triggering errors like "RuntimeError: dictionary changed size during iteration"
# GUESS: Some content in self.loop_prev_out[li] may be in the middle of being changed.
result = await curr_loop.run_in_executor(
pool, copy.deepcopy(func), copy.deepcopy(self.loop_prev_out[li])
pool, copy.deepcopy(func), copy.deepcopy(self.loop_prev_out[li]),
)
# auto determine whether to run async or sync
elif asyncio.iscoroutinefunction(func):
result = await func(self.loop_prev_out[li])
else:
# auto determine whether to run async or sync
if asyncio.iscoroutinefunction(func):
result = await func(self.loop_prev_out[li])
else:
# Default: run sync function directly
result = func(self.loop_prev_out[li])
# Default: run sync function directly
result = func(self.loop_prev_out[li])
# Store result in the nested dictionary
self.loop_prev_out[li][name] = result
except Exception as e:
@@ -251,14 +249,13 @@ class LoopBase:
next_step_idx = self.steps.index(self.skip_loop_error_stepname)
if next_step_idx <= si:
raise RuntimeError(
f"Cannot skip backwards or to same step. Current: {si} ({name}), Target: {next_step_idx} ({self.skip_loop_error_stepname})"
f"Cannot skip backwards or to same step. Current: {si} ({name}), Target: {next_step_idx} ({self.skip_loop_error_stepname})",
) from e
# Default: jump to feedback step if exists, otherwise jump to the last step (record)
elif "feedback" in self.steps:
next_step_idx = self.steps.index("feedback")
else:
# Default: jump to feedback step if exists, otherwise jump to the last step (record)
if "feedback" in self.steps:
next_step_idx = self.steps.index("feedback")
else:
next_step_idx = len(self.steps) - 1
next_step_idx = len(self.steps) - 1
self.loop_prev_out[li][name] = None
self.loop_prev_out[li][self.EXCEPTION_KEY] = e
elif isinstance(e, self.withdraw_loop_error):
@@ -409,6 +406,8 @@ class LoopBase:
self.close_pbar()
def withdraw_loop(self, loop_idx: int) -> None:
if loop_idx <= 0:
raise RuntimeError(f"Cannot withdraw loop {loop_idx}: no previous loop exists.")
prev_session_dir = self.session_folder / str(loop_idx - 1)
prev_path = min(
(p for p in prev_session_dir.glob("*_*") if p.is_file()),
@@ -501,7 +500,7 @@ class LoopBase:
session_folder = path.parent.parent
with path.open("rb") as f:
session = cast(LoopBase, pickle.load(f))
session = cast("LoopBase", pickle.load(f))
# set session folder
if checkout:
+8 -7
View File
@@ -9,7 +9,6 @@ import datetime
from typing import TYPE_CHECKING
import pytz
from rdagent.core.conf import RD_AGENT_SETTINGS
from rdagent.log.timer import RD_Agent_TIMER_wrapper
@@ -84,12 +83,14 @@ class WorkflowTracker:
# Log timer status if timer is started
if self.loop_base.timer.started:
remain_time = self.loop_base.timer.remain_time()
assert remain_time is not None
mlflow.log_metric("remain_time", remain_time.total_seconds())
mlflow.log_metric(
"remain_percent",
remain_time / self.loop_base.timer.all_duration * 100,
)
if remain_time is None:
logger.warning("remain_time is None despite timer.started, skipping timer metrics")
else:
mlflow.log_metric("remain_time", remain_time.total_seconds())
mlflow.log_metric(
"remain_percent",
remain_time / self.loop_base.timer.all_duration * 100,
)
# Keep only the log_workflow_state method as it's the primary entry point now
except Exception as e:
+43 -44
View File
@@ -19,19 +19,16 @@ import sys
import time
from datetime import datetime
from pathlib import Path
from typing import Dict, List, Optional
from dotenv import load_dotenv
from rich.console import Console
from rich.live import Live
from rich.markdown import Markdown
from rich.panel import Panel
from rich.table import Table
from rich.markdown import Markdown
from rich.layout import Layout
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
# Load environment variables from .env file
load_dotenv(Path(__file__).parent / ".env")
load_dotenv(Path(__file__).parent.parent / ".env")
console = Console()
@@ -43,12 +40,12 @@ class RunState:
self.run_id = run_id
self.api_key_idx = api_key_idx
self.model = model
self.process: Optional[subprocess.Popen] = None
self.process: subprocess.Popen | None = None
self.status: str = "pending" # pending, running, success, failed, stopped
self.start_time: Optional[datetime] = None
self.end_time: Optional[datetime] = None
self.exit_code: Optional[int] = None
self.error_message: Optional[str] = None
self.start_time: datetime | None = None
self.end_time: datetime | None = None
self.exit_code: int | None = None
self.error_message: str | None = None
self.log_file: str = f"fin_quant_run{run_id}.log"
@property
@@ -105,8 +102,8 @@ class ParallelRunner:
self.num_runs = num_runs
self.num_api_keys = num_api_keys
self.model = model
self.runs: List[RunState] = []
self.project_root = Path(__file__).parent
self.runs: list[RunState] = []
self.project_root = Path(__file__).parent.parent
self._shutdown_requested = False
# Read API keys from environment
@@ -115,10 +112,10 @@ class ParallelRunner:
# Validate we have enough API keys
if self.model == "openrouter" and len(self.api_keys) < num_api_keys:
console.print(
f"[yellow]⚠️ Requested {num_api_keys} API keys, but only {len(self.api_keys)} found in .env[/yellow]"
f"[yellow]⚠️ Requested {num_api_keys} API keys, but only {len(self.api_keys)} found in .env[/yellow]",
)
console.print(
f"[dim]Distributing across {len(self.api_keys)} available key(s)[/dim]"
f"[dim]Distributing across {len(self.api_keys)} available key(s)[/dim]",
)
self.num_api_keys = len(self.api_keys)
@@ -129,7 +126,7 @@ class ParallelRunner:
run_state = RunState(run_id=i, api_key_idx=api_key_idx, model=model)
self.runs.append(run_state)
def _load_api_keys(self) -> List[str]:
def _load_api_keys(self) -> list[str]:
"""Load API keys from environment variables."""
keys = []
@@ -149,7 +146,7 @@ class ParallelRunner:
return keys
def _build_env(self, run_state: RunState) -> Dict[str, str]:
def _build_env(self, run_state: RunState) -> dict[str, str]:
"""
Build isolated environment for a subprocess.
@@ -174,16 +171,14 @@ class ParallelRunner:
env["RD_AGENT_WORKSPACE"] = str(workspace_dir)
# Configure API key for this run
if self.model == "openrouter" and run_state.api_key_idx < len(self.api_keys):
api_key = self.api_keys[run_state.api_key_idx]
env["OPENAI_API_KEY"] = api_key
env["OPENAI_API_BASE"] = "https://openrouter.ai/api/v1"
env["CHAT_MODEL"] = os.getenv("OPENROUTER_MODEL", "openrouter/google/gemma-4-26b-a4b-it:free")
# If we configured multiple API keys AND have enough keys, use load balancing
if self.model == "openrouter":
if self.num_api_keys >= 2 and len(self.api_keys) >= 2:
env["OPENAI_API_KEY"] = f"{self.api_keys[0]},{self.api_keys[1]}"
env["LITELLM_PARALLEL_CALLS"] = "2"
elif run_state.api_key_idx < len(self.api_keys):
env["OPENAI_API_KEY"] = self.api_keys[run_state.api_key_idx]
env["OPENAI_API_BASE"] = "https://openrouter.ai/api/v1"
env["CHAT_MODEL"] = os.getenv("OPENROUTER_MODEL", "openrouter/google/gemma-4-26b-a4b-it:free")
elif self.model == "local":
env["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY", "local")
env["OPENAI_API_BASE"] = os.getenv("OPENAI_API_BASE", "http://localhost:8081/v1")
@@ -191,7 +186,7 @@ class ParallelRunner:
return env
def _build_command(self, run_state: RunState) -> List[str]:
def _build_command(self, run_state: RunState) -> list[str]:
"""
Build the subprocess command to run predix quant.
@@ -235,20 +230,24 @@ class ParallelRunner:
log_path = self.project_root / run_state.log_file
log_f = open(log_path, "a", encoding="utf-8")
# Start subprocess
run_state.process = subprocess.Popen(
cmd,
env=env,
cwd=str(self.project_root),
stdout=log_f,
stderr=subprocess.STDOUT,
)
run_state.status = "running"
run_state.start_time = datetime.now()
try:
# Start subprocess
run_state.process = subprocess.Popen(
cmd,
env=env,
cwd=str(self.project_root),
stdout=log_f,
stderr=subprocess.STDOUT,
)
run_state.status = "running"
run_state.start_time = datetime.now()
except Exception:
log_f.close()
raise
console.print(
f"[dim] ▶️ Run {run_state.run_id} started (PID: {run_state.process.pid}, "
f"API Key: {run_state.api_key_idx + 1}, Model: {run_state.model})[/dim]"
f"API Key: {run_state.api_key_idx + 1}, Model: {run_state.model})[/dim]",
)
def _check_run(self, run_state: RunState) -> None:
@@ -273,14 +272,14 @@ class ParallelRunner:
run_state.status = "success"
console.print(
f"[bold green] ✅ Run {run_state.run_id} completed "
f"({run_state.elapsed})[/bold green]"
f"({run_state.elapsed})[/bold green]",
)
else:
run_state.status = "failed"
run_state.error_message = f"Exit code: {poll_result}"
console.print(
f"[bold red] ❌ Run {run_state.run_id} failed "
f"({run_state.elapsed}, exit code: {poll_result})[/bold red]"
f"({run_state.elapsed}, exit code: {poll_result})[/bold red]",
)
def _stop_run(self, run_state: RunState) -> None:
@@ -386,7 +385,7 @@ class ParallelRunner:
if run.status == "running":
self._stop_run(run)
def run(self) -> Dict[str, int]:
def run(self) -> dict[str, int]:
"""
Execute all parallel runs and show live dashboard.
@@ -400,7 +399,7 @@ class ParallelRunner:
signal.signal(signal.SIGTERM, self._signal_handler)
console.print(f"\n[bold cyan]{'=' * 60}[/bold cyan]")
console.print(f"[bold cyan]🔀 Predix Parallel Runner[/bold cyan]")
console.print("[bold cyan]🔀 Predix Parallel Runner[/bold cyan]")
console.print(f"[bold cyan]{'=' * 60}[/bold cyan]")
console.print(f" Runs: {self.num_runs}")
console.print(f" API Keys: {self.num_api_keys} ({len(self.api_keys)} available)")
@@ -451,7 +450,7 @@ class ParallelRunner:
stopped_count = sum(1 for r in self.runs if r.status == "stopped")
console.print(f"\n[bold cyan]{'=' * 60}[/bold cyan]")
console.print(f"[bold cyan]📊 Parallel Run Summary[/bold cyan]")
console.print("[bold cyan]📊 Parallel Run Summary[/bold cyan]")
console.print(f"[bold cyan]{'=' * 60}[/bold cyan]")
console.print(f" ✅ Success: {success_count}/{self.num_runs}")
console.print(f" ❌ Failed: {failed_count}/{self.num_runs}")
@@ -463,7 +462,7 @@ class ParallelRunner:
if run.start_time and run.end_time:
delta = run.end_time - run.start_time
console.print(
f" Run #{run.run_id}: {run.status} ({delta.total_seconds():.0f}s)"
f" Run #{run.run_id}: {run.status} ({delta.total_seconds():.0f}s)",
)
return {
@@ -478,7 +477,7 @@ def main(
runs: int = 5,
api_keys: int = 2,
model: str = "openrouter",
) -> Dict[str, int]:
) -> dict[str, int]:
"""
Run multiple factor experiments in parallel.
@@ -504,7 +503,7 @@ if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(
description="Predix Parallel Runner - Run multiple factor experiments concurrently"
description="Predix Parallel Runner - Run multiple factor experiments concurrently",
)
parser.add_argument(
"--runs", "-n",
@@ -542,7 +541,7 @@ if __name__ == "__main__":
elif args.runs > 25:
console.print(f"\n[yellow]⚠️ {args.runs} runs - high resource usage expected[/yellow]")
console.print(f" Estimated RAM: ~{args.runs * 0.65:.0f} GB")
console.print(f" Use --force to confirm.\n")
console.print(" Use --force to confirm.\n")
import time
time.sleep(2)
+5 -4
View File
@@ -16,7 +16,8 @@ def load_factors(names, vdir):
if df is not None and len(df.columns) > 0:
dfs[n] = df.iloc[:, 0]
break
except: pass
except Exception:
pass
return dfs
def fix_code(code, available):
@@ -79,7 +80,7 @@ except Exception as e:
if r.returncode != 0:
return None
sig = pd.read_pickle(str(tdp / "s.pkl"))
except:
except Exception:
return None
fwd = df.mean(axis=1).shift(-96).dropna()
@@ -124,7 +125,7 @@ def main(count=None):
d = json.load(open(f))
if isinstance(d, dict) and 'strategy_name' in d:
files.append(f)
except: pass
except Exception: pass
if count: files = files[:count]
print(f"Re-evaluating {len(files)} strategies...\n")
@@ -147,7 +148,7 @@ def main(count=None):
with open(f, 'w') as out: json.dump(data, out, indent=2, ensure_ascii=False)
updated += 1
results.append({'name':data['strategy_name'], **bt})
except:
except Exception:
pass
p.update(task, advance=1)
+140
View File
@@ -16,7 +16,9 @@ import pytest
from rdagent.components.backtesting.vbt_backtest import (
OOS_START_DEFAULT,
_apply_ftmo_mask,
backtest_signal_ftmo,
FTMO_INITIAL_CAPITAL,
FTMO_MAX_DAILY_LOSS,
FTMO_MAX_TOTAL_LOSS,
monte_carlo_trade_pvalue,
@@ -238,3 +240,141 @@ def test_wf_consistency_range(close_6yr):
c = r.get("wf_oos_consistency")
if c is not None:
assert 0.0 <= c <= 1.0
# ---------------------------------------------------------------------------
# Direct _apply_ftmo_mask unit tests
# ---------------------------------------------------------------------------
class TestApplyFtmoMask:
"""Direct unit tests for _apply_ftmo_mask — the core FTMO daily/total loss engine."""
@pytest.fixture
def flat_close(self) -> pd.Series:
n = 3000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
return pd.Series(1.10, index=idx)
def test_returns_compliance_dict(self, flat_close):
signal = _random_signal(flat_close.index)
masked, info = _apply_ftmo_mask(signal, flat_close, leverage=1.0, txn_cost_bps=2.14)
assert "ftmo_daily_breaches" in info
assert "ftmo_total_breached" in info
assert "ftmo_total_breach_ts" in info
assert "ftmo_compliant" in info
def test_flat_market_zero_signal_fully_compliant(self, flat_close):
"""No trades → always compliant."""
signal = pd.Series(0.0, index=flat_close.index)
masked, info = _apply_ftmo_mask(signal, flat_close, leverage=1.0, txn_cost_bps=2.14)
assert info["ftmo_daily_breaches"] == 0
assert info["ftmo_total_breached"] is False
assert info["ftmo_compliant"] is True
# All signals should remain zero
assert (masked == 0).all()
def test_daily_loss_breach_zeroes_rest_of_day(self):
"""When daily loss exceeds 5%, rest of that day's signals are zeroed."""
n = 3000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
# Price drops sharply in first few bars to trigger daily loss
price = pd.Series(1.10, index=idx, dtype=float)
price.iloc[3:20] = 0.00 # crash from 1.10 to 0.00 → massive loss
signal = pd.Series(1.0, index=idx) # always long at 30x leverage
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
assert info["ftmo_daily_breaches"] > 0
# After breach, signals on same day must be zeroed
breach_day = idx[0].date()
same_day_late = (idx[-1] if idx[-1].date() == breach_day else idx[20])
if same_day_late.date() == breach_day:
assert masked.loc[same_day_late] == 0
def test_total_loss_breach_zeroes_all_remaining(self):
"""When total loss exceeds 10%, ALL subsequent signals are zeroed."""
n = 5000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
# Price crashes → max position → total loss limit breached
price = pd.Series(1.10, index=idx, dtype=float)
price.iloc[5:50] = 0.50 # >10% drop with 30x leverage
signal = pd.Series(1.0, index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
assert info["ftmo_total_breached"] is True
assert info["ftmo_total_breach_ts"] is not None
# After breach, ALL later signals must be zero
assert (masked.iloc[100:] == 0).all()
def test_total_breach_respected_across_days(self):
"""Total breach persists across day boundaries — no new trades after breach."""
n = 5000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx, dtype=float)
price.iloc[5:50] = 0.50
signal = pd.Series(1.0, index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
# All signals after breach index must be zero
breach_ts = pd.Timestamp(info["ftmo_total_breach_ts"])
assert (masked.loc[masked.index > breach_ts] == 0).all()
def test_daily_loss_resets_on_new_day(self):
"""Daily loss limit resets at day boundary — new day starts fresh (unless total breached)."""
n = 5000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx, dtype=float)
# Trigger daily breach on day 1 by dropping 1%
price.iloc[5:20] = 1.09 # ~1% drop with 30x → ~30% loss
signal = pd.Series(1.0, index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
assert info["ftmo_daily_breaches"] >= 1
# Day 2 signals should be active again if not total-breached
day2_mask = idx.date > idx[0].date()
if day2_mask.any() and not info["ftmo_total_breached"]:
day2 = idx[day2_mask][0]
assert masked.loc[day2] != 0
def test_compliant_flag_false_after_daily_breach(self):
"""Even one daily breach makes ftmo_compliant=False."""
n = 3000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx, dtype=float)
price.iloc[3:20] = 0.00
signal = pd.Series(1.0, index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
assert info["ftmo_compliant"] is False
def test_compliant_flag_false_after_total_breach(self):
"""Total breach makes ftmo_compliant=False."""
n = 5000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx, dtype=float)
price.iloc[5:50] = 0.50
signal = pd.Series(1.0, index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=30.0, txn_cost_bps=0)
assert info["ftmo_compliant"] is False
def test_transaction_costs_reduce_equity(self):
"""Transaction costs should reduce equity — compliant scenario with fees."""
n = 1000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx, dtype=float)
# Alternating signal → lots of position changes → high costs
signal = pd.Series([1.0 if i % 2 == 0 else -1.0 for i in range(n)], index=idx)
masked, info = _apply_ftmo_mask(signal, price, leverage=1.0, txn_cost_bps=10.0)
# With high costs and flat market, equity should drop
assert "ftmo_daily_breaches" in info
def test_output_mask_has_same_index(self):
n = 2000
idx = pd.date_range("2024-01-01", periods=n, freq="1min")
price = pd.Series(1.10, index=idx)
signal = _random_signal(idx, seed=1)
masked, info = _apply_ftmo_mask(signal, price, leverage=1.0, txn_cost_bps=2.14)
assert len(masked) == len(signal)
assert masked.index.equals(signal.index)
+88
View File
@@ -399,3 +399,91 @@ class TestDatabaseIntegrity:
# Import am Anfang der Datei für die Tests
from rdagent.components.backtesting.results_db import ResultsDatabase
class TestAddColumnIfNotExists:
"""Direct tests for _add_column_if_not_exists migration helper."""
def test_add_new_column_succeeds(self):
"""Adding a new column to an existing table should work."""
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
db._add_column_if_not_exists("backtest_runs", "test_new_col", "REAL")
c = db.conn.cursor()
c.execute("PRAGMA table_info(backtest_runs)")
cols = [row[1] for row in c.fetchall()]
assert "test_new_col" in cols
finally:
db.close()
def test_existing_column_noop(self):
"""Adding an already existing column should succeed (no-op)."""
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
# First call adds, second call should be no-op
db._add_column_if_not_exists("backtest_runs", "ic", "REAL")
db._add_column_if_not_exists("backtest_runs", "ic", "REAL")
c = db.conn.cursor()
c.execute("PRAGMA table_info(backtest_runs)")
cols = [row[1] for row in c.fetchall()]
assert cols.count("ic") == 1 # should exist exactly once
finally:
db.close()
def test_invalid_table_raises(self):
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
with pytest.raises(ValueError, match="Unknown table"):
db._add_column_if_not_exists("nonexistent_table", "col", "REAL")
finally:
db.close()
def test_invalid_column_name_raises(self):
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
with pytest.raises(ValueError, match="Invalid column name"):
db._add_column_if_not_exists("backtest_runs", "bad;column", "REAL")
finally:
db.close()
def test_invalid_column_type_raises(self):
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
with pytest.raises(ValueError, match="Invalid column type"):
db._add_column_if_not_exists("backtest_runs", "col", "INVALID_TYPE")
finally:
db.close()
def test_all_allowed_types_work(self):
"""REAL, TEXT, INTEGER, BLOB should all be valid types."""
with tempfile.TemporaryDirectory() as tmpdir:
import os
db_path = os.path.join(tmpdir, "test.db")
db = ResultsDatabase(db_path=db_path)
try:
for col_type in ("REAL", "TEXT", "INTEGER", "BLOB"):
db._add_column_if_not_exists(
"backtest_runs", f"test_{col_type.lower()}", col_type,
)
c = db.conn.cursor()
c.execute("PRAGMA table_info(backtest_runs)")
cols = {row[1] for row in c.fetchall()}
for col_type in ("REAL", "TEXT", "INTEGER", "BLOB"):
assert f"test_{col_type.lower()}" in cols
finally:
db.close()
+277
View File
@@ -0,0 +1,277 @@
"""
Tests for background task infrastructure (parallel runner, CLI paths, env loading).
Verifies bugs that were previously present:
- predix_parallel.py: project_root pointing to scripts/ instead of repo root
- predix_parallel.py: .env loaded from scripts/ instead of repo root
- predix_parallel.py: API key round-robin overwritten by comma-separated list
- cli.py: project_root depth wrong (4 .parent hops instead of 3)
- cli.py start_loop: hardcoded "python" instead of sys.executable
- cli.py parallel: hardcoded model=local
"""
import os
import sys
from pathlib import Path
from unittest.mock import Mock, patch
import pytest
# ── predix_parallel.py ──────────────────────────────────────────────────
class TestParallelRunnerProjectRoot:
"""Verify ParallelRunner.project_root points to the repo root, not scripts/."""
def test_project_root_is_repo_root(self):
"""Bug: project_root was Path(__file__).parent (= scripts/)."""
from scripts.predix_parallel import ParallelRunner
runner = ParallelRunner(num_runs=1, num_api_keys=1, model="local")
root = runner.project_root
# Must contain predix.py (repo root), NOT be the scripts/ dir
assert (root / "predix.py").exists(), (
f"project_root={root} does not contain predix.py — "
f"likely still pointing to scripts/ instead of repo root"
)
assert root.name != "scripts", (
f"project_root={root} ends with 'scripts/' — should be repo root"
)
def test_build_command_points_to_predix_py(self):
"""Bug: command pointed to scripts/predix.py which doesn't exist."""
from scripts.predix_parallel import ParallelRunner, RunState
runner = ParallelRunner(num_runs=1, num_api_keys=1, model="local")
run = RunState(run_id=1, api_key_idx=0, model="local")
cmd = runner._build_command(run)
predix_path = Path(cmd[1])
assert predix_path.exists(), (
f"Command references {predix_path} which does not exist — "
f"project_root likely still wrong"
)
assert predix_path.name == "predix.py"
assert predix_path.parent.name != "scripts", (
"predix.py should be in repo root, not scripts/"
)
def test_env_loading_from_repo_root(self):
"""Bug: load_dotenv loaded scripts/.env which doesn't exist."""
# load_dotenv is called at module import time, so we just verify
# that after import, the env reflects any .env at repo root.
# The key test: the call should not raise FileNotFoundError.
repo_root = Path(__file__).parent.parent.parent
env_path = repo_root / ".env"
assert env_path.exists(), (
f".env not found at {env_path} — repo root detection may be wrong"
)
class TestParallelRunnerAPIKeys:
"""Verify API key distribution logic."""
def test_single_api_key_no_overwrite(self):
"""Bug: with num_api_keys=1, individual key was set then overwritten."""
from scripts.predix_parallel import ParallelRunner, RunState
with patch.dict(os.environ, {}, clear=True):
os.environ["OPENROUTER_API_KEY"] = "sk-test-key-1"
runner = ParallelRunner(num_runs=2, num_api_keys=1, model="openrouter")
# Reset api_keys since _load_api_keys already ran in __init__
runner.api_keys = ["sk-test-key-1"]
runner.num_api_keys = 1
env = runner._build_env(RunState(run_id=1, api_key_idx=0, model="openrouter"))
assert env["OPENAI_API_KEY"] == "sk-test-key-1", (
"Single key should be assigned directly, not overwritten"
)
assert "LITELLM_PARALLEL_CALLS" not in env, (
"LITELLM_PARALLEL_CALLS should not be set for single key"
)
def test_multi_api_key_comma_separated(self):
"""With 2+ keys, all runs get comma-separated list for load balancing."""
from scripts.predix_parallel import ParallelRunner, RunState
with patch.dict(os.environ, {}, clear=True):
os.environ["OPENROUTER_API_KEY"] = "sk-key-a"
os.environ["OPENROUTER_API_KEY_2"] = "sk-key-b"
runner = ParallelRunner(num_runs=3, num_api_keys=2, model="openrouter")
env = runner._build_env(RunState(run_id=1, api_key_idx=0, model="openrouter"))
assert env["OPENAI_API_KEY"] == "sk-key-a,sk-key-b", (
"Multiple keys should be comma-separated for LiteLLM load balancing"
)
assert env.get("LITELLM_PARALLEL_CALLS") == "2"
def test_round_robin_api_key_index(self):
"""Verify round-robin API key index assignment is computed correctly."""
from scripts.predix_parallel import ParallelRunner
with patch.dict(os.environ, {}, clear=True):
os.environ["OPENROUTER_API_KEY"] = "a"
os.environ["OPENROUTER_API_KEY_2"] = "b"
runner = ParallelRunner(num_runs=5, num_api_keys=2, model="openrouter")
# 5 runs, 2 keys → indices: 0, 1, 0, 1, 0
expected = [0, 1, 0, 1, 0]
actual = [r.api_key_idx for r in runner.runs]
assert actual == expected, f"Round-robin mismatch: {actual} != {expected}"
class TestParallelRunnerLogFileHandling:
"""Verify log files and results go to the right place."""
def test_log_file_paths_in_repo_root(self):
"""Bug: logs went to scripts/fin_quant_runN.log."""
from scripts.predix_parallel import ParallelRunner
runner = ParallelRunner(num_runs=2, num_api_keys=1, model="local")
for run in runner.runs:
log_file = run.log_file
# log_file is relative — should be "fin_quant_runN.log"
assert "scripts" not in log_file, (
f"Log file {log_file} should not be in scripts/"
)
assert log_file.startswith("fin_quant_run"), (
f"Unexpected log file name: {log_file}"
)
# ── cli.py ──────────────────────────────────────────────────────────────
class TestCLIProjectRoot:
"""Verify CLI commands resolve project_root to the actual repo root."""
REPO_ROOT = Path(__file__).parent.parent.parent
def test_cli_project_root_depth(self):
"""Bug: 4x .parent put project_root one level above the repo."""
# The fixed code uses .parent.parent.parent (3 hops) from rdagent/app/cli.py
cli_file = self.REPO_ROOT / "rdagent" / "app" / "cli.py"
assert cli_file.exists(), f"cli.py not found at {cli_file}"
# Simulate what the fixed code does
resolved = cli_file.parent.parent.parent
assert resolved == self.REPO_ROOT, (
f"3 .parent hops from cli.py should yield repo root, got {resolved}"
)
# The bug used 4 hops which would overshoot
buggy = cli_file.parent.parent.parent.parent
assert buggy != self.REPO_ROOT, (
"4 .parent hops should NOT yield repo root "
f"(got {buggy}, expected {self.REPO_ROOT.parent})"
)
assert (buggy / "Predix").exists() or buggy == self.REPO_ROOT.parent, (
f"4 .parent hops overshoots repo root: {buggy}"
)
def test_cli_start_loop_uses_sys_executable(self):
"""Bug: start_loop used hardcoded 'python' instead of sys.executable."""
from rdagent.app.cli import start_loop_cli
import inspect
source = inspect.getsource(start_loop_cli)
# The fixed code uses sys.executable in the generator list
assert "sys.executable" in source, (
"start_loop_cli should use sys.executable, not hardcoded 'python'"
)
# Should NOT contain the old hardcoded pattern
assert 'f"python ' not in source, (
"start_loop_cli should not contain hardcoded 'python' string"
)
def test_cli_parallel_not_hardcoded_model(self):
"""Bug: parallel_cli hardcoded -m local in subprocess command."""
from rdagent.app.cli import parallel_cli
import inspect
source = inspect.getsource(parallel_cli)
# The fixed code no longer passes -m local as a cmd argument
assert '-m", "local"' not in source and '-m", \n "local"' not in source and '"-m", "local"' not in source and '"local"]' not in source, (
"parallel_cli should not hardcode model=local in subprocess command list"
)
def test_cli_scripts_exist_at_resolved_paths(self):
"""Verify scripts referenced by CLI commands exist at the resolved paths."""
from rdagent.app.cli import eval_all_cli, batch_backtest_cli, simple_eval_cli
from rdagent.app.cli import rebacktest_cli, report_cli, parallel_cli
import inspect
# All these commands use Path(__file__).parent.parent.parent as project_root
commands = {
"eval_all": "scripts/predix_full_eval.py",
"batch_backtest": "scripts/predix_batch_backtest.py",
"simple_eval": "scripts/predix_simple_eval.py",
"rebacktest": "scripts/predix_rebacktest_strategies.py",
"report": "scripts/predix_strategy_report.py",
"parallel": "scripts/predix_parallel.py",
}
for cmd_name, script_path in commands.items():
full_path = self.REPO_ROOT / script_path
assert full_path.exists(), (
f"CLI command '{cmd_name}' references {full_path} which does not exist. "
f"project_root depth may be wrong."
)
def test_start_loop_generator_script_exists(self):
"""Bug: wrong project_root meant generator script not found."""
from rdagent.app.cli import start_loop_cli
import inspect
source = inspect.getsource(start_loop_cli)
# The generator should reference scripts/predix_smart_strategy_gen.py
assert "predix_smart_strategy_gen.py" in source, (
"start_loop_cli should reference predix_smart_strategy_gen.py"
)
script = self.REPO_ROOT / "scripts" / "predix_smart_strategy_gen.py"
assert script.exists(), (
f"Generator script not found at {script}"
)
def test_start_loop_uses_child_proc_not_pkill(self):
"""Bug: cleanup used pkill -f which killed all instances system-wide."""
from rdagent.app.cli import start_loop_cli
import inspect
source = inspect.getsource(start_loop_cli)
# Fixed code uses child_proc.terminate() / child_proc.kill()
assert "child_proc" in source, (
"start_loop_cli should use child_proc variable for targeted cleanup"
)
# Should NOT contain the old broad pkill
assert "pkill" not in source, (
"start_loop_cli should not use broad pkill for process management"
)
# ── Integration: full import checks ─────────────────────────────────────
class TestImportsDontCrash:
"""Verify that importing the fixed modules doesn't crash."""
def test_import_parallel_runner(self):
"""ParallelRunner should import without errors."""
from scripts.predix_parallel import ParallelRunner, RunState
runner = ParallelRunner(num_runs=1, num_api_keys=1, model="local")
assert runner.num_runs == 1
assert len(runner.runs) == 1
def test_import_cli_app(self):
"""CLI app should import without errors."""
from rdagent.app.cli import app
assert app is not None
+281
View File
@@ -0,0 +1,281 @@
"""
Tests for bug fixes in strategy_orchestrator, factor_runner, backtest_engine,
results_db, model_runner, optuna_optimizer, env, and related modules.
Verifies:
- strategy_orchestrator.py compiles (IndentationError was fixed)
- factor_runner.py uses sys.executable (variable), not literal string
- backtest_engine.py path depth is 4 (not 3) .parent hops
- results_db.py path depth for factors/failed dirs is 4 (not 3)
- model_runner.py DB connection closed via try/finally
- factor_runner.py variable shadowing eliminated (run_id vs db_run_id)
- optuna_optimizer.py no longer shadows imported logger
- env.py Docker build output handles non-UTF-8 bytes
- env.py conda env list parsing guards against empty lines
- strategy_orchestrator.py exec() exception logged at ERROR level
- strategy_orchestrator.py template validation warns on unreplaced {{...}}
- factor_runner.py IC_max guard against scalar (AttributeError)
- predix_parallel.py handle leak on Popen failure
- predix_rebacktest_strategies.py bare except replaced with except Exception
"""
import ast
import inspect
import os
import sys
from pathlib import Path
from unittest.mock import patch
import pytest
REPO_ROOT = Path(__file__).parent.parent.parent
# ── Fix 1: strategy_orchestrator.py IndentationError ──────────────────────
class TestStrategyOrchestratorSyntax:
def test_file_compiles(self):
"""Bug: IndentationError at line 764 prevented the entire file from importing."""
import py_compile
py_compile.compile(
str(REPO_ROOT / "rdagent/components/coder/strategy_orchestrator.py"),
doraise=True,
)
def test_module_imports(self):
"""Verify StrategyOrchestrator can be imported after syntax fix."""
from rdagent.components.coder.strategy_orchestrator import StrategyOrchestrator
assert StrategyOrchestrator is not None
# ── Fix 2: factor_runner.py literal "sys.executable" ──────────────────────
class TestFactorRunnerSysExecutable:
def test_not_literal_string(self):
"""Bug: ["sys.executable", ...] was a literal string, not the variable."""
source = (REPO_ROOT / "rdagent/scenarios/qlib/developer/factor_runner.py").read_text()
# The fix should NOT contain the quoted literal 'sys.executable'
assert '"sys.executable"' not in source, (
"factor_runner.py still contains literal string 'sys.executable'"
"should be sys.executable (variable)"
)
# Should use sys.executable (variable, part of a list)
assert "sys.executable" in source
def test_subprocess_run_with_check_false(self):
"""Bug: subprocess.run without explicit check=False."""
source = (REPO_ROOT / "rdagent/scenarios/qlib/developer/factor_runner.py").read_text()
# The fix added check=False to the full-data factor run
assert 'check=False' in source, (
"subprocess.run should have explicit check=False for full-data factor run"
)
# ── Fix 3: backtest_engine.py path depth ─────────────────────────────────
class TestBacktestEnginePathDepth:
def test_path_depth_is_4(self):
"""Bug: 3 .parent hops ended at rdagent/ instead of repo root."""
from rdagent.components.backtesting.backtest_engine import FactorBacktester
fb = FactorBacktester()
results = fb.results_path
# results_path should be under the repo root, not under rdagent/
assert REPO_ROOT in results.parents or results.parent == REPO_ROOT / "results", (
f"results_path={results} is not under repo root {REPO_ROOT}. "
"Path depth may still be wrong."
)
# The path should NOT be inside rdagent/
assert "rdagent/results" not in str(results).replace(str(REPO_ROOT), ""), (
f"results_path={results} appears to be inside rdagent/ directory"
)
# ── Fix 4: results_db.py path depth ──────────────────────────────────────
class TestResultsDBPathDepth:
def test_factors_dir_depth(self):
"""Bug: 3 .parent hops from results_db.py ended at rdagent/."""
from rdagent.components.backtesting.results_db import ResultsDatabase
source = inspect.getsource(ResultsDatabase.generate_results_summary)
# After fix, should use .parent.parent.parent.parent (4 hops)
assert ".parent.parent.parent.parent" in source, (
"ResultsDatabase.generate_results_summary should use 4 .parent hops, not 3"
)
# ── Fix 5: model_runner.py DB close ──────────────────────────────────────
class TestModelRunnerDBClose:
def test_try_finally_for_db_close(self):
"""Bug: db.close() was after add_backtest, not in finally block."""
source = (REPO_ROOT / "rdagent/scenarios/qlib/developer/model_runner.py").read_text()
# After fix, db.close() should be in a finally block or try/finally context
assert "finally:" in source, (
"model_runner.py should use try/finally to close DB connection"
)
assert "db.close()" in source
# ── Fix 6: factor_runner.py variable shadowing ───────────────────────────
class TestFactorRunnerShadowing:
def test_no_run_id_shadowing(self):
"""Bug: run_id was reassigned from parallel run ID to DB row ID."""
source = (REPO_ROOT / "rdagent/scenarios/qlib/developer/factor_runner.py").read_text()
# After fix, parallel_run_id and db_run_id are separate variables
assert "parallel_run_id" in source, (
"factor_runner.py should use parallel_run_id for parallel run isolation"
)
assert "db_run_id" in source, (
"factor_runner.py should use db_run_id for DB row ID"
)
# ── Fix 7: optuna_optimizer.py logger shadowing ──────────────────────────
class TestOptunaLoggerShadowing:
def test_logger_not_reassigned(self):
"""Bug: logger was reassigned from rdagent_logger to raw logging.getLogger."""
source = (REPO_ROOT / "rdagent/components/coder/optuna_optimizer.py").read_text()
# After fix, the module-level logger should be rdagent_logger
assert "from rdagent.log import rdagent_logger as logger" in source
# The second assignment should use a different name
assert "_optuna_logger" in source, (
"optuna_optimizer.py should not shadow the rdagent logger"
)
# ── Fix 8: env.py UnicodeDecodeError ─────────────────────────────────────
class TestEnvUnicodeDecode:
def test_decode_with_errors_replace(self):
"""Bug: part.decode('utf-8') could raise UnicodeDecodeError."""
source = (REPO_ROOT / "rdagent/utils/env.py").read_text()
# After fix, decode uses errors="replace"
assert 'decode("utf-8", errors="replace")' in source, (
"env.py should handle non-UTF-8 Docker build output with errors='replace'"
)
# ── Fix 9: env.py conda env list parsing ─────────────────────────────────
class TestEnvCondaParsing:
def test_guard_against_empty_lines(self):
"""Bug: line.split()[0] crashed on empty tokens."""
source = (REPO_ROOT / "rdagent/utils/env.py").read_text()
# After fix, guards against empty split results
assert "len(line.split()) > 0" in source, (
"env.py should guard against empty split results in conda env list parsing"
)
# ── Fix 10: strategy_orchestrator.py exec() logging ──────────────────────
class TestStrategyOrchExecLogging:
def test_error_logging_in_exec_handler(self):
"""Bug: exec() exception was silently swallowed."""
source = (REPO_ROOT / "rdagent/components/coder/strategy_orchestrator.py").read_text()
# After fix, logger.error is called inside the except block
assert "logger.error" in source, (
"strategy_orchestrator.py should log exec() errors at ERROR level"
)
# ── Fix 11: strategy_orchestrator.py template validation ─────────────────
class TestStrategyOrchTemplateValidation:
def test_unreplaced_template_warning(self):
"""Bug: no validation that {{...}} placeholders were replaced."""
source = (REPO_ROOT / "rdagent/components/coder/strategy_orchestrator.py").read_text()
# After fix, warns on unreplaced template variables
assert "Unreplaced template variables" in source, (
"strategy_orchestrator.py should warn on unreplaced {{...}} placeholders"
)
# ── Fix 12: factor_runner.py IC_max guard ────────────────────────────────
class TestFactorRunnerICMaxGuard:
def test_hasattr_guard(self):
"""Bug: IC_max[...].index failed with AttributeError on scalar result."""
source = (REPO_ROOT / "rdagent/scenarios/qlib/developer/factor_runner.py").read_text()
# After fix, guards against scalar IC_max result
assert 'hasattr(IC_max, "index")' in source, (
"factor_runner.py should guard IC_max.index access with hasattr"
)
# ── Fix 13: predix_parallel.py handle leak ───────────────────────────────
class TestParallelRunnerHandleLeak:
def test_log_f_close_on_popen_failure(self):
"""Bug: open() file handle leaked if Popen failed."""
source = (REPO_ROOT / "scripts/predix_parallel.py").read_text()
# After fix, log_f.close() is called before re-raise
assert "log_f.close()" in source, (
"predix_parallel.py should close log file handle on Popen failure"
)
# ── Fix 14: predix_rebacktest_strategies.py bare except ──────────────────
class TestRebacktestBareExcept:
def test_not_bare_except(self):
"""Bug: bare except: pass swallowed all errors including SystemExit."""
source = (REPO_ROOT / "scripts/predix_rebacktest_strategies.py").read_text()
# After fix, should use except Exception, not bare except
assert "except Exception:" in source
assert "except:" not in source, (
"predix_rebacktest_strategies.py should not use bare except:"
)
# ── Integration: import checks ───────────────────────────────────────────
class TestAllImportsDontCrash:
def test_strategy_orchestrator_imports(self):
"""Verify all fixed modules import without errors."""
import rdagent.components.coder.strategy_orchestrator # noqa: F401
def test_factor_runner_imports(self):
import rdagent.scenarios.qlib.developer.factor_runner # noqa: F401
def test_backtest_engine_imports(self):
import rdagent.components.backtesting.backtest_engine # noqa: F401
def test_results_db_imports(self):
import rdagent.components.backtesting.results_db # noqa: F401
def test_model_runner_imports(self):
import rdagent.scenarios.qlib.developer.model_runner # noqa: F401
def test_optuna_optimizer_imports(self):
import rdagent.components.coder.optuna_optimizer # noqa: F401
+243
View File
@@ -0,0 +1,243 @@
"""Tests for rdagent.core — the core framework abstractions."""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import MagicMock
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# Feedback base class
# =============================================================================
class TestFeedback:
def test_default_is_acceptable_returns_true(self):
from rdagent.core.evaluation import Feedback
fb = Feedback()
assert fb.is_acceptable() is True
def test_default_finished_returns_true(self):
from rdagent.core.evaluation import Feedback
fb = Feedback()
assert fb.finished() is True
def test_default_bool_is_true(self):
from rdagent.core.evaluation import Feedback
fb = Feedback()
assert bool(fb) is True
# =============================================================================
# EvoStep dataclass
# =============================================================================
class TestEvoStep:
def test_default_construction(self):
from rdagent.core.evolving_framework import EvoStep
es = EvoStep(evolvable_subjects="mock_evo")
assert es.evolvable_subjects == "mock_evo"
assert es.queried_knowledge is None
assert es.feedback is None
def test_full_construction(self):
from rdagent.core.evolving_framework import EvoStep, QueriedKnowledge
qk = QueriedKnowledge()
es = EvoStep(evolvable_subjects="evo", queried_knowledge=qk, feedback="fb")
assert es.queried_knowledge is qk
assert es.feedback == "fb"
def test_equality_by_reference(self):
from rdagent.core.evolving_framework import EvoStep
es1 = EvoStep(evolvable_subjects="a")
es2 = EvoStep(evolvable_subjects="a")
assert es1 == es2
# =============================================================================
# Scenario base class
# =============================================================================
class TestScenario:
def test_source_data_default_returns_empty_string(self):
from rdagent.core.scenario import Scenario
class MinimalScenario(Scenario):
@property
def background(self) -> str: return "bg"
@property
def rich_style_description(self) -> str: return "rich"
def get_scenario_all_desc(self, **kwargs) -> str: return "all"
def get_runtime_environment(self) -> str: return "env"
scen = MinimalScenario()
assert scen.source_data == ""
def test_source_data_property_calls_get_source_data_desc(self):
from rdagent.core.scenario import Scenario
class FakeScenario(Scenario):
@property
def background(self) -> str: return "bg"
@property
def rich_style_description(self) -> str: return "rich"
def get_scenario_all_desc(self, **kwargs) -> str: return "all"
def get_runtime_environment(self) -> str: return "env"
def get_source_data_desc(self, task=None) -> str: return "custom_data"
scen = FakeScenario()
assert scen.source_data == "custom_data"
# =============================================================================
# EvolvingStrategy base class
# =============================================================================
class TestEvolvingStrategy:
def test_init_stores_scenario(self):
from rdagent.core.evolving_framework import EvolvingStrategy
class MinimalStrategy(EvolvingStrategy):
def evolve_iter(self, evo, queried_knowledge=None, evolving_trace=None):
yield evo
mock_scen = MagicMock()
es = MinimalStrategy(mock_scen)
assert es.scen is mock_scen
# =============================================================================
# IterEvaluator base class
# =============================================================================
class TestIterEvaluator:
def test_evaluate_returns_feedback(self):
from rdagent.core.evaluation import Feedback
from rdagent.core.evolving_framework import IterEvaluator, EvolvableSubjects
class MyFeedback(Feedback):
pass
class MyEvaluator(IterEvaluator):
def evaluate_iter(self):
evo = yield MyFeedback()
yield MyFeedback()
return MyFeedback()
eva = MyEvaluator()
result = eva.evaluate(EvolvableSubjects())
assert isinstance(result, MyFeedback)
def test_evaluate_iter_send_none_stops(self):
"""Sending None mid-iteration triggers StopIteration with final feedback."""
from rdagent.core.evaluation import Feedback
from rdagent.core.evolving_framework import IterEvaluator
class MyEvaluator(IterEvaluator):
def evaluate_iter(self):
yield Feedback() # kick-off (none)
evo_next = yield Feedback() # partial eval
if evo_next is None:
return Feedback() # early return
return Feedback() # normal path
eva = MyEvaluator()
gen = eva.evaluate_iter()
next(gen) # kick-off → first Feedback
gen.send("any") # evo gets "any" → second Feedback (evo_next NOT assigned yet)
with pytest.raises(StopIteration):
gen.send(None) # evo_next = None → return → StopIteration
# =============================================================================
# Developer base class
# =============================================================================
class TestDeveloper:
def test_develop_raises_not_implemented(self):
from rdagent.core.developer import Developer
class MinimalDeveloper(Developer):
def develop(self, exp):
return super().develop(exp)
dev = MinimalDeveloper(MagicMock())
with pytest.raises(NotImplementedError):
dev.develop(MagicMock())
# =============================================================================
# Knowledge / QueriedKnowledge
# =============================================================================
class TestKnowledgeHierarchy:
def test_knowledge_pass_through(self):
from rdagent.core.evolving_framework import Knowledge, QueriedKnowledge
k = Knowledge()
qk = QueriedKnowledge()
assert isinstance(k, Knowledge)
assert isinstance(qk, QueriedKnowledge)
# =============================================================================
# EvolvingAgent (abstract interface)
# =============================================================================
class TestEvolvingAgent:
def test_ragevo_agent_init(self):
from rdagent.core.evolving_agent import RAGEvoAgent
mock_strategy = MagicMock()
mock_rag = MagicMock()
agent = RAGEvoAgent.__new__(RAGEvoAgent)
RAGEvoAgent.__init__(agent, max_loop=5, evolving_strategy=mock_strategy, rag=mock_rag)
assert agent.max_loop == 5
assert agent.evolving_strategy is mock_strategy
assert agent.rag is mock_rag
def test_ragevo_agent_default_knowledge_flags(self):
from rdagent.core.evolving_agent import RAGEvoAgent
agent = RAGEvoAgent.__new__(RAGEvoAgent)
RAGEvoAgent.__init__(agent, max_loop=3, evolving_strategy=MagicMock(), rag=MagicMock())
assert agent.with_knowledge is False
assert agent.knowledge_self_gen is False
assert agent.enable_filelock is False
def test_ragevo_agent_with_knowledge_enabled(self):
from rdagent.core.evolving_agent import RAGEvoAgent
agent = RAGEvoAgent.__new__(RAGEvoAgent)
RAGEvoAgent.__init__(
agent, max_loop=3, evolving_strategy=MagicMock(), rag=MagicMock(),
with_knowledge=True, knowledge_self_gen=True,
enable_filelock=True, filelock_path="/tmp/test.lock",
)
assert agent.with_knowledge is True
assert agent.knowledge_self_gen is True
assert agent.enable_filelock is True
assert agent.filelock_path == "/tmp/test.lock"
# =============================================================================
# EvolvableSubjects clone
# =============================================================================
class TestEvolvableSubjects:
def test_clone_produces_deep_copy(self):
from rdagent.core.evolving_framework import EvolvableSubjects
es = EvolvableSubjects()
clone = es.clone()
assert clone is not es
assert type(clone) is type(es)
+338
View File
@@ -0,0 +1,338 @@
"""Tests for CoSTEER feedback types and EvolvingItem."""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import MagicMock
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# CoSTEERSingleFeedback
# =============================================================================
class TestCoSTEERSingleFeedback:
def test_construction_with_valid_fields(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb = CoSTEERSingleFeedback(
execution="exec ok",
return_checking="return ok",
code="code ok",
final_decision=True,
)
assert fb.execution == "exec ok"
assert fb.return_checking == "return ok"
assert fb.code == "code ok"
assert fb.final_decision is True
def test_bool_returns_final_decision(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb_true = CoSTEERSingleFeedback(execution="x", return_checking="x", code="x", final_decision=True)
fb_false = CoSTEERSingleFeedback(execution="x", return_checking="x", code="x", final_decision=False)
assert bool(fb_true) is True
assert bool(fb_false) is False
def test_default_values(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb = CoSTEERSingleFeedback(execution="x", return_checking=None, code="x")
assert fb.final_decision is None
assert fb.raw_execution == ""
assert fb.source_feedback == {}
def test_val_and_update_init_dict_converts_boolean(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
assert CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z", "final_decision": "true"}
)["final_decision"] is True
assert CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z", "final_decision": "false"}
)["final_decision"] is False
assert CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z", "final_decision": "True"}
)["final_decision"] is True
assert CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z", "final_decision": "False"}
)["final_decision"] is False
def test_val_and_update_init_dict_rejects_non_boolean(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
with pytest.raises(ValueError):
CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z", "final_decision": 42}
)
def test_val_and_update_init_dict_missing_final_decision_raises(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
with pytest.raises(ValueError, match="final_decision"):
CoSTEERSingleFeedback.val_and_update_init_dict(
{"execution": "x", "return_checking": "y", "code": "z"}
)
def test_val_and_update_init_dict_json_dumps_non_string_attrs(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
import json
data = {
"execution": {"key": "val"},
"return_checking": ["list"],
"code": 123,
"final_decision": True,
}
result = CoSTEERSingleFeedback.val_and_update_init_dict(data)
for attr in ("execution", "return_checking", "code"):
# Should have been converted to JSON string
assert isinstance(result[attr], str)
_ = json.loads(result[attr]) # valid JSON
def test_merge_all_true_decisions(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb1 = CoSTEERSingleFeedback(execution="a", return_checking="ra", code="c1", final_decision=True)
fb2 = CoSTEERSingleFeedback(execution="b", return_checking="rb", code="c2", final_decision=True)
merged = CoSTEERSingleFeedback.merge([fb1, fb2])
assert merged.final_decision is True
def test_merge_one_false_decision(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb1 = CoSTEERSingleFeedback(execution="a", return_checking="ra", code="c1", final_decision=True)
fb2 = CoSTEERSingleFeedback(execution="b", return_checking="rb", code="c2", final_decision=False)
merged = CoSTEERSingleFeedback.merge([fb1, fb2])
assert merged.final_decision is False
def test_merge_concatenates_fields(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb1 = CoSTEERSingleFeedback(execution="A", return_checking="RA", code="C1", final_decision=True)
fb2 = CoSTEERSingleFeedback(execution="B", return_checking="RB", code="C2", final_decision=True)
merged = CoSTEERSingleFeedback.merge([fb1, fb2])
assert "A\n\nB" in merged.execution
assert "RA\n\nRB" in merged.return_checking
assert "C1\n\nC2" in merged.code
def test_merge_skips_none_fields(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb1 = CoSTEERSingleFeedback(execution="A", return_checking="RA", code="C1", final_decision=True)
fb2 = CoSTEERSingleFeedback(execution="B", return_checking=None, code="C2", final_decision=True)
merged = CoSTEERSingleFeedback.merge([fb1, fb2])
assert merged.execution == "A\n\nB"
assert merged.return_checking == "RA"
assert merged.code == "C1\n\nC2"
def test_merge_aggregates_source_feedback(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb1 = CoSTEERSingleFeedback(execution="a", return_checking="x", code="c", final_decision=True,
source_feedback={"eval1": True})
fb2 = CoSTEERSingleFeedback(execution="b", return_checking="y", code="d", final_decision=True,
source_feedback={"eval2": False})
merged = CoSTEERSingleFeedback.merge([fb1, fb2])
assert merged.source_feedback == {"eval1": True, "eval2": False}
def test_str_contains_all_sections(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb = CoSTEERSingleFeedback(execution="exec", return_checking="ret", code="code", final_decision=True)
s = str(fb)
assert "Execution" in s
assert "Return Checking" in s
assert "Code" in s
assert "Final Decision" in s
assert "SUCCESS" in s
def test_str_shows_fail_for_false(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
fb = CoSTEERSingleFeedback(execution="exec", return_checking="ret", code="code", final_decision=False)
assert "FAIL" in str(fb)
# =============================================================================
# CoSTEERSingleFeedbackDeprecated
# =============================================================================
class TestCoSTEERSingleFeedbackDeprecated:
def test_property_getters(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
fb = CoSTEERSingleFeedbackDeprecated(
execution_feedback="exec",
code_feedback="code",
value_feedback="val",
shape_feedback="shape",
final_decision=True,
final_feedback="final",
value_generated_flag=True,
final_decision_based_on_gt=True,
)
assert fb.execution == "exec"
assert fb.code == "code"
assert fb.final_decision is True
assert fb.value_generated_flag is True
assert fb.final_decision_based_on_gt is True
def test_return_checking_when_value_generated(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
fb = CoSTEERSingleFeedbackDeprecated(
value_generated_flag=True, value_feedback="vals", shape_feedback="shapes",
)
rc = fb.return_checking
assert "vals" in rc
assert "shapes" in rc
def test_return_checking_when_no_value_generated(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
fb = CoSTEERSingleFeedbackDeprecated(value_generated_flag=False)
assert fb.return_checking is None
def test_setters_work(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
fb = CoSTEERSingleFeedbackDeprecated()
fb.execution = "new_exec"
fb.code = "new_code"
fb.return_checking = "new_rc"
assert fb.execution_feedback == "new_exec"
assert fb.code_feedback == "new_code"
assert fb.value_feedback == "new_rc"
assert fb.shape_feedback == "new_rc"
def test_str_contains_all_sections(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedbackDeprecated
fb = CoSTEERSingleFeedbackDeprecated(
execution_feedback="exec", shape_feedback="shape",
code_feedback="code", value_feedback="val",
final_feedback="final", final_decision=True,
)
s = str(fb)
for keyword in ("Execution", "Shape", "Code", "Value", "Final Decision", "SUCCESS"):
assert keyword in s
# =============================================================================
# CoSTEERMultiFeedback
# =============================================================================
class TestCoSTEERMultiFeedback:
def _make_fb(self, decision=True):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
return CoSTEERSingleFeedback(execution="x", return_checking="x", code="x", final_decision=decision)
def test_getitem(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
fb1, fb2 = self._make_fb(True), self._make_fb(False)
mf = CoSTEERMultiFeedback([fb1, fb2])
assert mf[0].final_decision is True
assert mf[1].final_decision is False
def test_len(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
assert len(CoSTEERMultiFeedback([self._make_fb()])) == 1
assert len(CoSTEERMultiFeedback([])) == 0
def test_append(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
mf = CoSTEERMultiFeedback([])
mf.append(self._make_fb(True))
assert len(mf) == 1
assert mf[0].final_decision is True
def test_iter(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
fbs = [self._make_fb(True), self._make_fb(True)]
mf = CoSTEERMultiFeedback(fbs)
assert list(mf) == fbs
def test_finished_all_true(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
mf = CoSTEERMultiFeedback([self._make_fb(True), self._make_fb(True)])
assert mf.finished() is True
def test_finished_one_false(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
mf = CoSTEERMultiFeedback([self._make_fb(True), self._make_fb(False)])
assert mf.finished() is False
def test_finished_with_none_skips(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
mf = CoSTEERMultiFeedback([self._make_fb(True), None])
assert mf.finished() is True # None = skipped = accepted
def test_bool_all_true(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
assert bool(CoSTEERMultiFeedback([self._make_fb(True), self._make_fb(True)])) is True
def test_bool_one_false(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
assert bool(CoSTEERMultiFeedback([self._make_fb(True), self._make_fb(False)])) is False
def test_is_acceptable_delegates(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
mf = CoSTEERMultiFeedback([self._make_fb(True), self._make_fb(True)])
assert mf.is_acceptable() is True
# =============================================================================
# EvolvingItem
# =============================================================================
class TestEvolvingItem:
def test_construction_without_gt(self):
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
t1 = Task(name="task1")
t2 = Task(name="task2")
ei = EvolvingItem(sub_tasks=[t1, t2])
assert len(ei.sub_tasks) == 2
assert ei.sub_gt_implementations is None
def test_construction_with_matching_gt(self):
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task, FBWorkspace
t1, t2 = Task(name="t1"), Task(name="t2")
ws1, ws2 = FBWorkspace(), FBWorkspace()
ei = EvolvingItem(sub_tasks=[t1, t2], sub_gt_implementations=[ws1, ws2])
assert ei.sub_gt_implementations == [ws1, ws2]
def test_mismatched_gt_length_resets_to_none(self):
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task, FBWorkspace
t1, t2 = Task(name="t1"), Task(name="t2")
ei = EvolvingItem(sub_tasks=[t1, t2], sub_gt_implementations=[FBWorkspace()])
assert ei.sub_gt_implementations is None
def test_from_experiment(self):
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Experiment, Task
exp = Experiment(sub_tasks=[Task(name="x")])
exp.based_experiments = ["base"]
exp.experiment_workspace = "ws"
ei = EvolvingItem.from_experiment(exp)
assert len(ei.sub_tasks) == 1
assert ei.based_experiments == ["base"]
assert ei.experiment_workspace == "ws"
# =============================================================================
# CoSTEERQueriedKnowledge
# =============================================================================
class TestCoSTEERQueriedKnowledge:
def test_default_construction(self):
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
qk = CoSTEERQueriedKnowledge()
assert qk.success_task_to_knowledge_dict == {}
assert qk.failed_task_info_set == set()
def test_with_data(self):
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
qk = CoSTEERQueriedKnowledge(
success_task_to_knowledge_dict={"a": "knowledge_a"},
failed_task_info_set={"fail1", "fail2"},
)
assert qk.success_task_to_knowledge_dict["a"] == "knowledge_a"
assert "fail1" in qk.failed_task_info_set
assert "fail2" in qk.failed_task_info_set
+225
View File
@@ -0,0 +1,225 @@
"""Tests for CoSTEER config, task, and evolve strategy population logic."""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# CoSTEERSettings
# =============================================================================
class TestCoSTEERSettings:
def test_default_values(self):
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
s = CoSTEERSettings()
assert s.max_loop == 1
assert s.fail_task_trial_limit == 5
assert s.v2_query_component_limit == 1
assert s.v2_query_error_limit == 1
assert s.v2_query_former_trace_limit == 3
assert s.v2_add_fail_attempt_to_latest_successful_execution is False
assert s.v2_knowledge_sampler == 1.0
assert s.coder_use_cache is False
assert s.enable_filelock is False
def test_singleton_instance(self):
from rdagent.components.coder.CoSTEER.config import CoSTEER_SETTINGS
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
assert isinstance(CoSTEER_SETTINGS, CoSTEERSettings)
assert CoSTEER_SETTINGS.max_loop == 1
# =============================================================================
# CoSTEERTask
# =============================================================================
class TestCoSTEERTask:
def test_base_code_stored(self):
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
t = CoSTEERTask(name="test", base_code="print(1)")
assert t.base_code == "print(1)"
def test_base_code_none_by_default(self):
from rdagent.components.coder.CoSTEER.task import CoSTEERTask
t = CoSTEERTask(name="test")
assert t.base_code is None
# =============================================================================
# MultiProcessEvolvingStrategy.assign_code_list_to_evo
# =============================================================================
class TestAssignCodeListToEvo:
def test_empty_code_list_noops(self):
from rdagent.components.coder.CoSTEER.evolving_strategy import MultiProcessEvolvingStrategy
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
strat = MultiProcessEvolvingStrategy.__new__(MultiProcessEvolvingStrategy)
MultiProcessEvolvingStrategy.__init__(strat, scen=MagicMock(), settings=MagicMock())
ei = EvolvingItem(sub_tasks=[Task(name="t1")])
ei.experiment_workspace = MagicMock()
result = strat.assign_code_list_to_evo([{}], ei)
assert result is ei
def test_none_entry_is_skipped(self):
from rdagent.components.coder.CoSTEER.evolving_strategy import MultiProcessEvolvingStrategy
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
strat = MultiProcessEvolvingStrategy.__new__(MultiProcessEvolvingStrategy)
MultiProcessEvolvingStrategy.__init__(strat, scen=MagicMock(), settings=MagicMock())
ei = EvolvingItem(sub_tasks=[Task(name="t1")])
ei.experiment_workspace = MagicMock()
result = strat.assign_code_list_to_evo([None], ei)
assert result.sub_workspace_list[0] is None # unchanged
def test_code_injects_files(self):
from rdagent.components.coder.CoSTEER.evolving_strategy import MultiProcessEvolvingStrategy
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
strat = MultiProcessEvolvingStrategy.__new__(MultiProcessEvolvingStrategy)
MultiProcessEvolvingStrategy.__init__(strat, scen=MagicMock(), settings=MagicMock())
ei = EvolvingItem(sub_tasks=[Task(name="t1")])
mock_ws = MagicMock()
ei.experiment_workspace = mock_ws
strat.assign_code_list_to_evo([{"factor.py": "x=1"}], ei)
mock_ws.inject_files.assert_called_once_with(**{"factor.py": "x=1"})
def test_change_summary_extracted(self):
from rdagent.components.coder.CoSTEER.evolving_strategy import MultiProcessEvolvingStrategy
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
strat = MultiProcessEvolvingStrategy.__new__(MultiProcessEvolvingStrategy)
MultiProcessEvolvingStrategy.__init__(strat, scen=MagicMock(), settings=MagicMock())
mock_ws = MagicMock()
ei = EvolvingItem(sub_tasks=[Task(name="t1")])
ei.experiment_workspace = mock_ws
strat.assign_code_list_to_evo([{"__change_summary__": "summary", "factor.py": "x"}], ei)
assert mock_ws.change_summary == "summary"
# change_summary should have been popped from dict
mock_ws.inject_files.assert_called_once_with(**{"factor.py": "x"})
# =============================================================================
# MultiProcessEvolvingStrategy.evolve_iter
# =============================================================================
class TestEvolveIter:
def _make_strat(self):
from rdagent.components.coder.CoSTEER.evolving_strategy import MultiProcessEvolvingStrategy
from rdagent.components.coder.CoSTEER.config import CoSTEERSettings
strat = MultiProcessEvolvingStrategy(
scen=MagicMock(), settings=CoSTEERSettings(), improve_mode=False,
)
return strat
def _make_evo(self, n_tasks=2):
from rdagent.components.coder.CoSTEER.evolvable_subjects import EvolvingItem
from rdagent.core.experiment import Task
tasks = [Task(name=f"task_{i}") for i in range(n_tasks)]
for t in tasks:
t.get_task_information = MagicMock(return_value=f"info_{t.name}")
ei = EvolvingItem(sub_tasks=tasks)
ei.experiment_workspace = MagicMock()
return ei
def test_raises_without_queried_knowledge(self):
strat = self._make_strat()
evo = self._make_evo()
with pytest.raises(ValueError, match="queried_knowledge"):
next(strat.evolve_iter(evo=evo, queried_knowledge=None))
def test_successful_tasks_not_scheduled(self):
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
strat = self._make_strat()
evo = self._make_evo(n_tasks=1)
qk = CoSTEERQueriedKnowledge(
success_task_to_knowledge_dict={
"info_task_0": MagicMock(implementation=MagicMock(file_dict={"f.py": "x"})),
},
)
# evolve_iter is a generator, next() starts it
gen = strat.evolve_iter(evo=evo, queried_knowledge=qk)
# Should yield the evo (populated from success knowledge)
result = next(gen)
# The task was already successful, so no new scheduling
assert result is evo
def test_failed_tasks_skipped(self):
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
strat = self._make_strat()
evo = self._make_evo(n_tasks=1)
qk = CoSTEERQueriedKnowledge(
failed_task_info_set={"info_task_0"},
)
gen = strat.evolve_iter(evo=evo, queried_knowledge=qk)
result = next(gen)
# Task skipped because it's in failed_set
assert result is evo
def test_improve_mode_skips_with_no_last_feedback(self):
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
strat = self._make_strat()
strat.improve_mode = True
evo = self._make_evo(n_tasks=1)
qk = CoSTEERQueriedKnowledge()
gen = strat.evolve_iter(evo=evo, queried_knowledge=qk, evolving_trace=[])
result = next(gen)
# In improve_mode with no last_feedback, task should be skipped
# (code_list[0] should be {} — empty implementation)
assert result is evo
def test_non_improve_mode_schedules_new_tasks(self):
"""Tasks not in success/failed should be scheduled."""
from rdagent.components.coder.CoSTEER.knowledge_management import CoSTEERQueriedKnowledge
strat = self._make_strat()
evo = self._make_evo(n_tasks=1)
qk = CoSTEERQueriedKnowledge()
with patch(
"rdagent.components.coder.CoSTEER.evolving_strategy.multiprocessing_wrapper",
return_value=[{"factor.py": "x=1"}],
):
gen = strat.evolve_iter(evo=evo, queried_knowledge=qk)
result = next(gen)
assert result is evo
# =============================================================================
# CoSTEERMultiEvaluator (partial)
# =============================================================================
class TestCoSTEERMultiEvaluator:
def test_init_with_single_evaluator(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
mock_eval = MagicMock()
eva = CoSTEERMultiEvaluator(single_evaluator=mock_eval, scen=MagicMock())
assert eva.single_evaluator is mock_eval
def test_init_with_evaluator_list(self):
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiEvaluator
mock_list = [MagicMock(), MagicMock()]
eva = CoSTEERMultiEvaluator(single_evaluator=mock_list, scen=MagicMock())
assert eva.single_evaluator is mock_list
+221
View File
@@ -0,0 +1,221 @@
"""Tests for factor_coder — evaluators, task, workspace."""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# FactorTask
# =============================================================================
class TestFactorTask:
def test_construction_fields(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
t = FactorTask(
factor_name="f1",
factor_description="desc",
factor_formulation="formula",
variables={"x": 1},
resource="r1",
)
assert t.factor_name == "f1"
assert t.factor_description == "desc"
assert t.factor_formulation == "formula"
assert t.variables == {"x": 1}
assert t.factor_resources == "r1"
assert t.factor_implementation is False
assert t.base_code is None # from CoSTEERTask
def test_get_task_information(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
t = FactorTask("f1", "desc", "formula", variables={"x": 1})
info = t.get_task_information()
assert "factor_name: f1" in info
assert "factor_description: desc" in info
assert "factor_formulation: formula" in info
assert "variables: {'x': 1}" in info
def test_get_task_brief_information(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
t = FactorTask("f1", "desc", "formula")
info = t.get_task_brief_information()
assert "factor_name: f1" in info
def test_get_task_information_and_implementation_result(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
t = FactorTask("f1", "desc", "formula")
result = t.get_task_information_and_implementation_result()
assert result["factor_name"] == "f1"
assert result["factor_description"] == "desc"
assert "factor_implementation" in result
def test_from_dict(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
d = {
"factor_name": "f2",
"factor_description": "d2",
"factor_formulation": "f2",
"variables": {},
"resource": None,
"factor_implementation": True,
}
t = FactorTask.from_dict(d)
assert t.factor_name == "f2"
assert t.factor_implementation is True
def test_repr(self):
from rdagent.components.coder.factor_coder.factor import FactorTask
t = FactorTask("myfactor", "desc", "formula")
assert "FactorTask" in repr(t)
assert "myfactor" in repr(t)
# =============================================================================
# FactorFBWorkspace
# =============================================================================
class TestFactorFBWorkspace:
def test_init_sets_workspace_path(self):
from rdagent.components.coder.factor_coder.factor import FactorFBWorkspace, FactorTask
t = FactorTask("f1", "desc", "formula")
ws = FactorFBWorkspace(target_task=t)
assert ws.workspace_path is not None
# Directory is created lazily by execute(), not in __init__
assert isinstance(ws.workspace_path, Path)
def test_execute_returns_message_and_dataframe(self):
from rdagent.components.coder.factor_coder.factor import FactorFBWorkspace, FactorTask
t = FactorTask("f1", "desc", "formula")
t.version = 1
ws = FactorFBWorkspace(target_task=t)
# Inject valid factor code
ws.inject_files(**{
"factor.py": (
"import pandas as pd\n"
"import numpy as np\n"
"data = pd.read_hdf('intraday_pv.h5', key='data')\n"
"factor_val = data['$close'].pct_change()\n"
"factor_val = factor_val.to_frame('f1')\n"
"factor_val.to_hdf('result.h5', key='data', mode='w')\n"
),
})
msg, df = ws.execute()
assert isinstance(msg, str)
assert df is not None
def test_execute_succeeds_and_returns_data(self):
from rdagent.components.coder.factor_coder.factor import FactorFBWorkspace, FactorTask
t = FactorTask("fl1", "desc", "formula")
ws = FactorFBWorkspace(target_task=t)
ws.inject_files(**{
"factor.py": (
"import pandas as pd\n"
"data = pd.read_hdf('intraday_pv.h5', key='data')\n"
"factor_val = data['$close'].pct_change().to_frame('fl1')\n"
"factor_val.to_hdf('result.h5', key='data', mode='w')\n"
),
})
msg, df = ws.execute()
assert FactorFBWorkspace.FB_EXEC_SUCCESS in msg
assert FactorFBWorkspace.FB_OUTPUT_FILE_FOUND in msg
assert df is not None
# =============================================================================
# FactorEvaluatorForCoder (partial integration)
# =============================================================================
class TestFactorEvaluatorForCoder:
def test_init_creates_sub_evaluators(self):
from rdagent.components.coder.factor_coder.evaluators import FactorEvaluatorForCoder
mock_scen = MagicMock()
eva = FactorEvaluatorForCoder(scen=mock_scen)
assert eva.value_evaluator is not None
assert eva.code_evaluator is not None
assert eva.final_decision_evaluator is not None
def test_evaluate_with_none_implementation(self):
from rdagent.components.coder.factor_coder.evaluators import FactorEvaluatorForCoder
eva = FactorEvaluatorForCoder(scen=MagicMock())
assert eva.evaluate(target_task=MagicMock(), implementation=None) is None
def test_evaluate_returns_queried_knowledge_if_present(self):
from rdagent.components.coder.factor_coder.evaluators import FactorEvaluatorForCoder
from rdagent.components.coder.factor_coder.factor import FactorTask
eva = FactorEvaluatorForCoder(scen=MagicMock())
t = FactorTask("f1", "desc", "formula")
qk = MagicMock()
qk.success_task_to_knowledge_dict = {"info_f1": MagicMock(feedback="cached_fb")}
t.get_task_information = MagicMock(return_value="info_f1")
qk.failed_task_info_set = set()
fb = eva.evaluate(target_task=t, implementation=MagicMock(), queried_knowledge=qk)
assert fb == "cached_fb" # returned from cache
def test_evaluate_skips_failed_task(self):
from rdagent.components.coder.factor_coder.evaluators import FactorEvaluatorForCoder
from rdagent.components.coder.factor_coder.factor import FactorTask
eva = FactorEvaluatorForCoder(scen=MagicMock())
t = FactorTask("f1", "desc", "formula")
qk = MagicMock()
qk.success_task_to_knowledge_dict = {}
t.get_task_information = MagicMock(return_value="info_f1")
qk.failed_task_info_set = {"info_f1"}
fb = eva.evaluate(target_task=t, implementation=MagicMock(), queried_knowledge=qk)
assert fb.final_decision is False
assert "failed too many times" in fb.execution_feedback
# =============================================================================
# FactorEvaluator (eva_utils) — constructors and identity
# =============================================================================
class TestFactorEvaluatorsInit:
def test_factor_inf_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorInfEvaluator
eva = FactorInfEvaluator()
assert str(eva) == "FactorInfEvaluator"
def test_factor_single_column_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorSingleColumnEvaluator
eva = FactorSingleColumnEvaluator()
assert str(eva) == "FactorSingleColumnEvaluator"
def test_factor_output_format_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorOutputFormatEvaluator
eva = FactorOutputFormatEvaluator()
assert str(eva) == "FactorOutputFormatEvaluator"
def test_factor_missing_values_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorMissingValuesEvaluator
eva = FactorMissingValuesEvaluator()
assert str(eva) == "FactorMissingValuesEvaluator"
def test_factor_correlation_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorCorrelationEvaluator
eva = FactorCorrelationEvaluator(hard_check=True)
assert eva.hard_check is True
assert str(eva) == "FactorCorrelationEvaluator"
def test_factor_value_evaluator_init(self):
from rdagent.components.coder.factor_coder.eva_utils import FactorValueEvaluator
mock_scen = MagicMock()
eva = FactorValueEvaluator(mock_scen)
assert eva.scen is mock_scen
+241
View File
@@ -0,0 +1,241 @@
"""Tests for bugs found in the factor evaluation pipeline."""
import sys
from pathlib import Path
from unittest.mock import MagicMock, patch
import numpy as np
import pandas as pd
import pytest
# Project root
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# Bug 1: Missing `import sys` in _save_factor_values (factor_runner.py:968)
# =============================================================================
class TestSaveFactorValuesMissingSysImport:
"""Verify that _save_factor_values has `import sys` — uses sys.executable at line 968."""
def test_save_factor_values_has_sys_import(self):
import inspect
from rdagent.scenarios.qlib.developer.factor_runner import QlibFactorRunner
source = inspect.getsource(QlibFactorRunner._save_factor_values)
assert "import sys" in source, (
"BUG: _save_factor_values calls sys.executable but does not import sys. "
"This causes a NameError at runtime, silently swallowed by the try/except."
)
def test_save_factor_values_nameerror_when_called(self):
"""Simulate calling _save_factor_values without sys available."""
from rdagent.scenarios.qlib.developer.factor_runner import QlibFactorRunner
runner = QlibFactorRunner.__new__(QlibFactorRunner)
mock_exp = MagicMock()
mock_exp.sub_workspace_list = []
mock_exp.experiment_workspace.workspace_path = None
# This should NOT raise NameError for 'sys' — if it does, the bug is present
try:
runner._save_factor_values("TestFactor", mock_exp)
except NameError as e:
if "sys" in str(e):
pytest.fail(
"BUG CONFIRMED: _save_factor_values raises NameError because "
"'sys' is not imported. The factor values parquet is never saved."
)
raise
# =============================================================================
# Bug 2: `acc_rate` undefined in FactorEqualValueRatioEvaluator (eva_utils.py:335-346)
# =============================================================================
class TestEqualValueRatioAccRateUndefined:
"""Verify FactorEqualValueRatioEvaluator handles shape-mismatch correctly."""
def test_acc_rate_undefined_after_except(self):
"""If gen_df.sub(gt_df) raises, acc_rate should still be defined (default -1)."""
from rdagent.components.coder.factor_coder.eva_utils import FactorEqualValueRatioEvaluator
evaluator = FactorEqualValueRatioEvaluator()
# Simulate the case where _get_df returns None for gt_df, which causes
# gen_df.sub(None) to raise AttributeError. The except clause must not
# reference an undefined acc_rate variable.
gen_df = pd.DataFrame({"x": [1.0, 2.0, 3.0]}, index=[0, 1, 2])
gt_ws = MagicMock()
imp_ws = MagicMock()
gt_ws.execute.return_value = ("", None) # _get_df will set gt_df = None
imp_ws.execute.return_value = ("", gen_df)
# Should NOT raise NameError
try:
result = evaluator.evaluate(imp_ws, gt_ws)
assert isinstance(result, tuple)
assert len(result) == 2
feedback, metric = result
assert metric == -1, f"Expected -1 (fallback), got {metric}"
except NameError as e:
if "acc_rate" in str(e):
pytest.fail(
"BUG CONFIRMED: FactorEqualValueRatioEvaluator references 'acc_rate' "
"which is undefined when gen_df.sub(gt_df) raises an exception."
)
raise
def test_acc_rate_defined_when_shapes_match(self):
"""Normal case: same shapes — acc_rate should be defined and returned."""
from rdagent.components.coder.factor_coder.eva_utils import FactorEqualValueRatioEvaluator
evaluator = FactorEqualValueRatioEvaluator()
gt_ws = MagicMock()
imp_ws = MagicMock()
gen_df = pd.DataFrame({"x": [1.0, 2.0, 3.0]}, index=[0, 1, 2])
gt_df = pd.DataFrame({"y": [1.0, 2.0, 3.0]}, index=[0, 1, 2])
gt_ws.execute.return_value = ("", gt_df)
imp_ws.execute.return_value = ("", gen_df)
result = evaluator.evaluate(imp_ws, gt_ws)
assert isinstance(result, tuple)
assert len(result) == 2
feedback, metric = result
# When values match within tolerance, metric should be a float near 1.0
assert isinstance(metric, float) or isinstance(metric, (int, np.integer))
assert metric >= 0
# =============================================================================
# Bug 3: Annualization factor hardcoded in _evaluate_factor_directly (factor_runner.py:553)
# =============================================================================
class TestAnnualizationFactorInDirectEval:
"""Verify direct evaluation uses correct annualization with forward_return_bars."""
def test_annualization_factor_uses_forward_bars(self):
"""The direct eval method hardcodes 96 instead of using forward_return_bars param."""
import inspect
from rdagent.scenarios.qlib.developer.factor_runner import QlibFactorRunner
source = inspect.getsource(QlibFactorRunner._evaluate_factor_directly)
# Check that the method uses `np.sqrt(252 * 1440 / 96)` which hardcodes 96
# This should ideally be parameterized or at least consistent with the
# forward return calculation at line ~530 which also uses 96.
assert "np.sqrt(252 * 1440 / 96)" in source or "np.sqrt(252*1440/96)" in source, (
"The annualization factor in _evaluate_factor_directly should match "
"the forward_return_bars used for computing forward returns."
)
def test_ann_factor_is_consistent_with_forward_ret(self):
"""Verify both the forward return shift and annualization use 96 bars."""
import inspect
from rdagent.scenarios.qlib.developer.factor_runner import QlibFactorRunner
source = inspect.getsource(QlibFactorRunner._evaluate_factor_directly)
# forward return uses `.shift(-96)` at line ~530
assert '.shift(-96)' in source, "Forward return shift should use 96 bars (1 day)"
# annualization should also use 96
assert '1440 / 96' in source, (
"Annualization factor should use the same number (96) as the forward return shift"
)
# =============================================================================
# Bug 4: _fix_inf_nan_handling inserts code before .dropna() or .to_hdf() in wrong context
# =============================================================================
class TestInfNanHandlingInsertion:
"""Verify inf/nan auto-fixer doesn't insert code in the wrong context."""
def test_no_insertion_before_dropna_when_no_column_found(self):
from rdagent.components.coder.factor_coder.auto_fixer import FactorAutoFixer
fixer = FactorAutoFixer()
# Code where the LAST assignment before .dropna() is NOT a df['col'] = pattern
# but dropna() still exists (e.g., on a temporary variable)
code = (
"def calc():\n"
" df = pd.read_hdf('data.h5', key='data')\n"
" temp = df['$close'].diff()\n"
" temp = temp.dropna()\n"
" df['result'] = temp * 2\n"
" result = df[['result']]\n"
)
result = fixer.fix(code)
# The code should still be valid (no syntax error from misplaced insertion)
import ast
try:
ast.parse(result)
except SyntaxError as e:
pytest.fail(f"Auto-fixer produced invalid Python code: {e}")
def test_inf_handling_inserted_before_result_assignment(self):
from rdagent.components.coder.factor_coder.auto_fixer import FactorAutoFixer
fixer = FactorAutoFixer()
code = (
"def calc():\n"
" df = pd.read_hdf('data.h5', key='data')\n"
" df['myfactor'] = df['$close'] / df['sigma_60bar']\n"
" df['myfactor'] = df['myfactor'] / df['sigma_5bar']\n"
" result = df[['myfactor']]\n"
)
result = fixer.fix(code)
# Should have added inf handling before the result = df[[...]] line
# but not broken syntax
import ast
try:
ast.parse(result)
except SyntaxError as e:
pytest.fail(f"Auto-fixer produced invalid Python code: {e}")
assert "replace([np.inf, -np.inf]" in result
# =============================================================================
# Bug 5: scan_factors reads factor_code twice (predix_full_eval.py:174 + 195)
# =============================================================================
class TestScanFactorsDoubleRead:
"""Verify scan_factors doesn't wastefully read factor file twice."""
def test_factor_code_read_only_when_needed(self):
"""Confirm the scan_factors double-read behavior (line 174+195)."""
import inspect
from scripts import predix_full_eval
source = inspect.getsource(predix_full_eval.scan_factors)
# Count occurrences of `.read_text()`
count = source.count(".read_text()")
# Expected: at least 2 (line 174 in fallback, line 195 in FactorInfo)
# Bug: if factor_name comes from result.h5 (line 168-170), then line 174
# is skipped, but line 195 always reads again — that's one wasted read.
assert count == 2, (
f"scan_factors has {count} read_text() calls. "
"Expected exactly 2 (one for name extraction, one for FactorInfo). "
"Consider caching to avoid double reads."
)
+182
View File
@@ -0,0 +1,182 @@
"""Tests for model_coder — ModelTask, shape/value evaluators, config."""
from __future__ import annotations
import sys
from pathlib import Path
from unittest.mock import MagicMock
import numpy as np
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# ModelTask
# =============================================================================
class TestModelTask:
def test_construction_fields(self):
from rdagent.components.coder.model_coder.model import ModelTask
t = ModelTask(
name="m1",
description="desc",
architecture="LSTM",
hyperparameters={"lr": 0.001},
training_hyperparameters={"epochs": 10},
formulation="y = f(x)",
variables={"x": "feature"},
model_type="TimeSeries",
)
assert t.name == "m1"
assert t.description == "desc"
assert t.architecture == "LSTM"
assert t.hyperparameters == {"lr": 0.001}
assert t.training_hyperparameters == {"epochs": 10}
assert t.formulation == "y = f(x)"
assert t.variables == {"x": "feature"}
assert t.model_type == "TimeSeries"
assert t.base_code is None
def test_get_task_information(self):
from rdagent.components.coder.model_coder.model import ModelTask
t = ModelTask(
name="m1", description="desc", architecture="LSTM",
hyperparameters={}, training_hyperparameters={},
model_type="Tabular",
)
info = t.get_task_information()
assert "name: m1" in info
assert "architecture: LSTM" in info
assert "model_type: Tabular" in info
def test_get_task_information_with_optional_fields(self):
from rdagent.components.coder.model_coder.model import ModelTask
t = ModelTask(
name="m2", description="d2", architecture="GRU",
hyperparameters={}, training_hyperparameters={},
formulation="f1", variables={"v": 1}, model_type="Graph",
)
info = t.get_task_information()
assert "formulation: f1" in info
assert "variables: {'v': 1}" in info
def test_get_task_brief_information(self):
from rdagent.components.coder.model_coder.model import ModelTask
t = ModelTask(
name="m1", description="desc", architecture="LSTM",
hyperparameters={"lr": 0.01}, training_hyperparameters={"epochs": 5},
)
info = t.get_task_brief_information()
assert "name: m1" in info
assert "architecture: LSTM" in info
assert "hyperparameters" in info
def test_from_dict(self):
from rdagent.components.coder.model_coder.model import ModelTask
d = {
"name": "m3", "description": "d3", "architecture": "TCN",
"hyperparameters": {}, "training_hyperparameters": {},
}
t = ModelTask.from_dict(d)
assert t.name == "m3"
def test_repr(self):
from rdagent.components.coder.model_coder.model import ModelTask
t = ModelTask(
name="mymodel", description="d", architecture="LSTM",
hyperparameters={}, training_hyperparameters={},
)
assert "ModelTask" in repr(t)
assert "mymodel" in repr(t)
# =============================================================================
# Shape/Value evaluators (eva_utils)
# =============================================================================
class TestShapeEvaluator:
def test_correct_shape(self):
from rdagent.components.coder.model_coder.eva_utils import shape_evaluator
msg, ok = shape_evaluator(np.ones((32, 10)), target_shape=(32, 10))
assert ok is True
assert "correct" in msg.lower()
def test_incorrect_shape(self):
from rdagent.components.coder.model_coder.eva_utils import shape_evaluator
msg, ok = shape_evaluator(np.ones((32, 5)), target_shape=(32, 10))
assert ok is False
assert "incorrect" in msg.lower()
def test_none_prediction(self):
from rdagent.components.coder.model_coder.eva_utils import shape_evaluator
msg, ok = shape_evaluator(None, target_shape=(32, 10))
assert ok is False
def test_none_target_shape(self):
from rdagent.components.coder.model_coder.eva_utils import shape_evaluator
msg, ok = shape_evaluator(np.ones((3,)), target_shape=None)
assert ok is False
def test_float_array(self):
from rdagent.components.coder.model_coder.eva_utils import shape_evaluator
msg, ok = shape_evaluator(np.array([1.0, 2.0]), target_shape=(2,))
assert ok is True
class TestValueEvaluator:
def test_none_prediction(self):
from rdagent.components.coder.model_coder.eva_utils import value_evaluator
msg, ok = value_evaluator(None, np.ones((3,)))
assert ok is False
def test_none_target(self):
from rdagent.components.coder.model_coder.eva_utils import value_evaluator
msg, ok = value_evaluator(np.ones((3,)), None)
assert ok is False
def test_small_difference_passes(self):
from rdagent.components.coder.model_coder.eva_utils import value_evaluator
msg, ok = value_evaluator(
np.array([1.0, 2.0, 3.0]),
np.array([1.0, 2.0, 3.01]),
)
assert bool(ok) is True # diff < 0.1
def test_large_difference_fails(self):
from rdagent.components.coder.model_coder.eva_utils import value_evaluator
msg, ok = value_evaluator(
np.array([1.0, 2.0]),
np.array([10.0, 20.0]),
)
assert bool(ok) is False # diff > 0.1
# =============================================================================
# ModelCoSTEERSettings
# =============================================================================
class TestModelCoSTEERSettings:
def test_default_env_type(self):
from rdagent.components.coder.model_coder.conf import ModelCoSTEERSettings
s = ModelCoSTEERSettings()
assert s.env_type == "conda"
def test_singleton(self):
from rdagent.components.coder.model_coder.conf import MODEL_COSTEER_SETTINGS
from rdagent.components.coder.model_coder.conf import ModelCoSTEERSettings
assert isinstance(MODEL_COSTEER_SETTINGS, ModelCoSTEERSettings)
def test_get_model_env_runs(self):
from rdagent.components.coder.model_coder.conf import get_model_env
# May succeed (conda available) or fail — either way, test the code path
try:
env = get_model_env()
assert env is not None
except Exception:
pass # expected if docker/conda not available
+186
View File
@@ -0,0 +1,186 @@
"""Tests for qlib pipeline — feedback, bandit, quant_loop_factory."""
from __future__ import annotations
import sys
import tempfile
from pathlib import Path
from unittest.mock import MagicMock, patch
import numpy as np
import pandas as pd
import pytest
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
# =============================================================================
# process_results (feedback.py)
# =============================================================================
class TestProcessResults:
def test_process_results_handles_named_series(self):
"""process_results renames column "0" to "Current Result" — this works
when the Series name is '0' (string), which matches the rename dict."""
from rdagent.scenarios.qlib.developer.feedback import process_results
import pandas as pd
# process_results expects the Series to produce a DataFrame column named "0" (string)
# This happens when the Series has name '0'
current = pd.Series(
{"IC": 0.05, "1day.excess_return_with_cost.annualized_return": 0.12,
"1day.excess_return_with_cost.max_drawdown": -0.08},
name="0",
)
sota = pd.Series(
{"IC": 0.03, "1day.excess_return_with_cost.annualized_return": 0.10,
"1day.excess_return_with_cost.max_drawdown": -0.05},
name="0",
)
result = process_results(current, sota)
assert "IC of Current Result is" in result
assert "of SOTA Result is" in result
def test_raises_on_missing_metrics(self):
from rdagent.scenarios.qlib.developer.feedback import process_results
current = pd.Series({"IC": 0.05})
sota = pd.Series({"IC": 0.03})
with pytest.raises(KeyError):
process_results(current, sota)
# =============================================================================
# bandit.py — Metrics and extract_metrics_from_experiment
# =============================================================================
class TestBanditMetrics:
def test_default_values_are_zero(self):
from rdagent.scenarios.qlib.proposal.bandit import Metrics
m = Metrics()
assert m.ic == 0.0
assert m.sharpe == 0.0
assert m.mdd == 0.0
def test_as_vector_length(self):
from rdagent.scenarios.qlib.proposal.bandit import Metrics
m = Metrics(ic=0.1, sharpe=1.5)
v = m.as_vector()
assert len(v) == 8
assert v[0] == 0.1
assert v[7] == 1.5
def test_mdd_negated_in_vector(self):
from rdagent.scenarios.qlib.proposal.bandit import Metrics
m = Metrics(mdd=0.15)
v = m.as_vector()
assert v[6] == -0.15 # -self.mdd
def test_extract_metrics_from_experiment(self):
from rdagent.scenarios.qlib.proposal.bandit import extract_metrics_from_experiment
mock_exp = MagicMock()
mock_exp.result = {
"IC": 0.04,
"ICIR": 0.5,
"Rank IC": 0.03,
"Rank ICIR": 0.4,
"1day.excess_return_with_cost.annualized_return ": 0.10,
"1day.excess_return_with_cost.information_ratio": 0.6,
"1day.excess_return_with_cost.max_drawdown": -0.12,
}
m = extract_metrics_from_experiment(mock_exp)
assert m.ic == 0.04
assert m.rank_ic == 0.03
assert m.mdd == -0.12
def test_extract_metrics_returns_default_on_error(self):
from rdagent.scenarios.qlib.proposal.bandit import extract_metrics_from_experiment
mock_exp = MagicMock()
mock_exp.result = None # Will cause AttributeError
m = extract_metrics_from_experiment(mock_exp)
assert m.ic == 0.0
assert m.sharpe == 0.0
def test_sharpe_computation(self):
from rdagent.scenarios.qlib.proposal.bandit import extract_metrics_from_experiment
mock_exp = MagicMock()
mock_exp.result = {
"IC": 0.0, "ICIR": 0.0, "Rank IC": 0.0, "Rank ICIR": 0.0,
"1day.excess_return_with_cost.annualized_return ": 0.15,
"1day.excess_return_with_cost.information_ratio": 0.0,
"1day.excess_return_with_cost.max_drawdown": -0.10,
}
m = extract_metrics_from_experiment(mock_exp)
assert m.sharpe == pytest.approx(1.5) # 0.15 / 0.10
# =============================================================================
# LinearThompsonTwoArm
# =============================================================================
class TestLinearThompsonTwoArm:
def test_initialization(self):
from rdagent.scenarios.qlib.proposal.bandit import LinearThompsonTwoArm
bandit = LinearThompsonTwoArm(dim=5)
assert bandit.dim == 5
assert bandit.noise_var == 1.0
assert bandit.mean["factor"].shape == (5,)
assert bandit.mean["model"].shape == (5,)
assert bandit.precision["factor"].shape == (5, 5)
def test_sample_reward_returns_float(self):
from rdagent.scenarios.qlib.proposal.bandit import LinearThompsonTwoArm
bandit = LinearThompsonTwoArm(dim=3)
x = np.ones(3)
reward = bandit.sample_reward("factor", x)
assert isinstance(reward, float)
def test_arms_are_initialized_identically(self):
from rdagent.scenarios.qlib.proposal.bandit import LinearThompsonTwoArm
bandit = LinearThompsonTwoArm(dim=4)
assert np.array_equal(bandit.mean["factor"], bandit.mean["model"])
assert np.array_equal(bandit.precision["factor"], bandit.precision["model"])
# =============================================================================
# quant_loop_factory.py
# =============================================================================
class TestHasLocalComponents:
def test_returns_bool(self):
from rdagent.scenarios.qlib.quant_loop_factory import has_local_components
result = has_local_components()
assert isinstance(result, bool)
def test_returns_false_with_no_local_dir(self, monkeypatch):
from rdagent.scenarios.qlib import quant_loop_factory
monkeypatch.setattr(quant_loop_factory.Path, "exists", lambda self: False)
assert quant_loop_factory.has_local_components() is False
class TestCountValidFactors:
def test_returns_zero_when_no_dir(self):
from rdagent.scenarios.qlib.quant_loop_factory import count_valid_factors
with patch("rdagent.scenarios.qlib.quant_loop_factory.Path.exists", return_value=False):
assert count_valid_factors() == 0
def test_returns_int(self):
from rdagent.scenarios.qlib.quant_loop_factory import count_valid_factors
result = count_valid_factors()
assert isinstance(result, int)
assert result >= 0
class TestAdvancedLoopThreshold:
def test_constant_is_defined(self):
from rdagent.scenarios.qlib.quant_loop_factory import ADVANCED_LOOP_FACTOR_THRESHOLD
assert ADVANCED_LOOP_FACTOR_THRESHOLD == 5000
+74 -1
View File
@@ -1,8 +1,10 @@
import tempfile
import unittest
from pathlib import Path
import pytest
from rdagent.core.utils import SingletonBaseClass
from rdagent.core.utils import SingletonBaseClass, import_class, safe_resolve_path
class A(SingletonBaseClass):
@@ -70,5 +72,76 @@ class MiscTest(unittest.TestCase):
# print(a1.kwargs) # a1 will be changed.
class TestSafeResolvePath:
"""Tests for safe_resolve_path — path traversal prevention."""
def test_inside_root_returns_absolute(self):
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
result = safe_resolve_path(root / "subdir" / "file.txt", safe_root=root)
assert result.is_absolute()
assert str(result).startswith(str(root.resolve()))
def test_no_safe_root_just_resolves(self):
result = safe_resolve_path(Path("/tmp/nonexistent_test"), safe_root=None)
assert result.is_absolute()
def test_path_traversal_raises(self):
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
with pytest.raises(ValueError, match="outside allowed root"):
safe_resolve_path(root / ".." / "etc" / "passwd", safe_root=root)
def test_symlink_outside_root_raises(self):
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
inside = root / "inside"
inside.mkdir()
link = inside / "escape"
link.symlink_to("/etc/passwd")
with pytest.raises(ValueError, match="outside allowed root"):
safe_resolve_path(link, safe_root=root)
def test_root_itself_is_valid(self):
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
result = safe_resolve_path(root, safe_root=root)
assert result == root.resolve()
def test_expanduser_resolves_home(self):
result = safe_resolve_path(Path("~/nonexistent_test"), safe_root=None)
assert str(result).startswith(str(Path.home()))
class TestImportClass:
"""Tests for import_class — dynamic class loading."""
def test_valid_class_import(self):
cls = import_class("pathlib.Path")
assert cls is Path
def test_builtin_class_import(self):
cls = import_class("collections.OrderedDict")
from collections import OrderedDict
assert cls is OrderedDict
def test_invalid_module_raises_import_error(self):
with pytest.raises(ImportError, match="Module not found"):
import_class("nonexistent.module.ClassName")
def test_missing_class_raises_import_error(self):
with pytest.raises(ImportError, match="Class not found"):
import_class("pathlib.NonExistentClass")
def test_invalid_format_raises_import_error(self):
with pytest.raises(ImportError, match="Invalid class path"):
import_class("no_dots_at_all")
def test_pandas_class_import(self):
cls = import_class("pandas.DataFrame")
import pandas as pd
assert cls is pd.DataFrame
if __name__ == "__main__":
unittest.main()