Commit Graph

25 Commits

Author SHA1 Message Date
TPTBusiness 39b49b1724 fix: filter NaN in max(), remove redundant ternary, handle non-finite vbt results 2026-05-03 00:25:58 +02:00
TPTBusiness ca003cd0f2 fix: close log file handle, fix RiskMgmt equity double-count, remove bare except 2026-05-03 00:17:02 +02:00
TPTBusiness 87610d660f fix(ci): lazy import logger in predix.py and cli.py to avoid ImportError in test env
Wrapped  in try/except
ImportError with standard logging fallback. The rdagent.log
module chain fails when predix.py is imported as a module
in the CI test environment (kronos CLI tests).
2026-05-01 13:58:16 +02:00
TPTBusiness 732361bb90 fix(security): resolve path-injection, B701, B101, B112 Bandit alerts
- Path injection (B614): centralized safe_resolve_path in core/utils.py,
  refactored 6 UI modules to use it with safe_root validation
- B701: added explicit autoescape=select_autoescape() to Jinja2
  Environment() calls in 3 files
- B101: replaced assert statements with proper if/raise patterns in
  12+ files (partial)
- B112: added logger.warning() to bare except:continue blocks in
  5 files
2026-05-01 13:42:59 +02:00
TPTBusiness 3f54381052 feat(fin_quant): auto-generate Kronos factor before loop start
Adds _ensure_kronos_factor_in_pool() which runs automatically at the
start of every fin_quant / predix quant invocation. If the Kronos
factor is not yet in results/factors/values/, it generates it
(stride=500, batch=32 GPU) and computes IC via evaluate_kronos_model.
Writes a StrategyOrchestrator-compatible JSON so the factor is
immediately available to strategy generation without manual steps.
Is a no-op when the factor already exists with a valid IC.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 12:18:51 +02:00
TPTBusiness 888e841366 perf(kronos): batch GPU inference via predict_batch — 75x faster
Replace sequential predict() calls with predict_batch() in both
build_kronos_factor and evaluate_kronos_model. Up to batch_size windows
processed simultaneously on GPU, reducing per-window time from ~10s to
~0.13s (10 windows in 1.3s on RTX 5060 Ti, 75x speedup).

Adds --batch-size / -b option (default 32) to both kronos-factor and
kronos-eval CLI commands. Falls back to single inference per window if a
batch fails. Refactors timestamp prep into _build_window_inputs helper.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 12:10:30 +02:00
TPTBusiness 6e52c8a15d fix(kronos): lazy torch import to fix CI ModuleNotFoundError
Move top-level `import torch` into _cuda_available() helper so
kronos_adapter.py can be imported in CI environments without torch.
All device defaults resolved at runtime via lazy detection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 10:00:45 +02:00
TPTBusiness 3f460226f2 feat: add Kronos CLI commands, expand tests, document in README
- predix kronos-factor: generate KronosPredReturn alpha factor via CLI
- predix kronos-eval: evaluate Kronos IC/hit-rate vs LightGBM via CLI
- 19 tests covering adapter, factor builder, model evaluator, CLI (mock-based)
- README: Kronos section in Features + CLI commands table
- Total test suite: 153 passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 09:53:56 +02:00
TPTBusiness 22e638af86 feat: unified backtest engine, LLM error handling, strategy refactor
- Add vbt_backtest.py as single source of truth for all metric formulas
  (Sharpe, drawdown, IC, transaction costs) — backtest_engine.py and
  strategy_orchestrator.py now delegate to it
- Add LLMUnavailableError to exception.py; rd_loop.py catches it at the
  proposal stage and raises LoopResumeError to avoid corrupting trace
  history with None hypotheses
- Guard record() against None exp/hypothesis so loop resets leave
  trace.hist in a consistent state
- Refactor strategy_orchestrator and optuna_optimizer to use unified
  backtest path; remove duplicate metric calculation code
- Add predix_rebacktest_unified.py script for offline re-evaluation
- Update tests and README

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 22:52:07 +02:00
TPTBusiness 2cec08bc91 feat: add daily log rotation, llama health wait, factor auto-fixer, and README updates
- Add rdagent/log/daily_log.py: daily-rotating structured logs per command
  (fin_quant, strategies, evaluate, parallel) with loguru; all.log combined sink
- predix.py: route TeeWriter output to logs/YYYY-MM-DD/ instead of root dir;
  wrap quant() and evaluate() in daily_log.session() for start/stop/duration tracking
- rdagent/app/cli.py: fin_quant_cli waits for llama.cpp /health endpoint before
  starting pipeline (up to 300 s); daily_log integration for fin_quant,
  generate_strategies, eval_all, parallel commands
- scripts/predix_gen_strategies_real_bt.py: daily_log integration with
  per-strategy ACCEPTED/REJECTED entries and summary on completion
- rdagent/components/coder/factor_coder/auto_fixer.py: new module that patches
  common LLM-generated factor issues (min_periods, inf/NaN, groupby.transform,
  MultiIndex corrections)
- rdagent/components/coder/factor_coder/prompts.yaml: add critical rules for
  EURUSD 1-min intraday factors (min_periods, inf handling, groupby, date range)
- README.md: document --reasoning off and --n-gpu-layers 28 for llama-server;
  explain VRAM constraints when Ollama is running alongside llama.cpp
- .bandit.yml: suppress B615 (HuggingFace unsafe download) for RL benchmark files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 07:20:08 +02:00
TPTBusiness b98c9cd572 feat: Add GitHub infrastructure, CI/CD pipelines, and examples
- Add GitHub issue templates (bug, feature, docs)
- Add pull request template with closed-source checklist
- Add CODEOWNERS for code review assignment
- Add CI/CD workflows (ci, lint, security, docs, release)
  - pytest + coverage with Python 3.10/3.11 matrix
  - Ruff + MyPy code quality checks
  - Bandit + safety security scanning
  - Sphinx docs + GitHub Pages deployment
  - Automated PyPI releases on tag push
- Add 6 comprehensive examples + Jupyter quickstart
  - 01_factor_discovery.py (LLM factor generation)
  - 02_factor_evolution.py (factor optimization)
  - 03_strategy_generation.py (IC-weighted combination)
  - 04_backtest_simple.py (strategy backtesting)
  - 05_model_training.py (XGBoost/LSTM training)
  - 06_rl_trading_agent.py (PPO/DQN/A2C agents)
  - notebooks/quickstart.ipynb (interactive tutorial)
- Restructure .gitignore with explicit closed-source sections
- Add CI/coverage/license badges to README
- Complete CLI docstrings for all 9 commands
- Add data_config.yaml for quant loop configuration
2026-04-11 21:40:18 +02:00
TPTBusiness 108a63fd79 feat: Strategy Generator working with local LLM (P0-P4)
Integrated strategy generation into fin_quant loop:
- Fixed LLM code extraction from JSON responses
- Fixed factor loading (MultiIndex parquet handling)
- Fixed return calculation (realistic proxy)
- Fixed max drawdown (NaN/inf handling)
- Added llama.cpp --reasoning off support
- Strategy orchestrator with LLM + evaluation
- Optuna optimizer integration

100% acceptance rate on test run (2/2 strategies).
Total changes: +2006 lines, -129 lines across 8 files.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-04-09 12:55:04 +02:00
TPTBusiness ac9bfc6fb4 fix: Add get_factor_count() to QuantTrace to prevent parallel run crashes
Problem:
- All 50 parallel runs failed with: AttributeError: 'QuantTrace' object has no attribute 'get_factor_count'
- The _build_strategies_with_ai() method calls self.trace.get_factor_count()
- This method didn't exist in the QuantTrace class

Fix:
- Add get_factor_count() method to QuantTrace class
- Add increment_factor_count() method to track factor generation
- Call increment_factor_count() in running() step when factors are generated
- _build_strategies_with_ai() is already wrapped in try/except for safety

Now the parallel runs will work correctly.
2026-04-06 10:52:28 +02:00
TPTBusiness a8c23bd130 feat: Add AI Strategy Builder (StrategyCoSTEER) - Closed Source
Open Source Changes:
- Add prompt loader functions for strategy prompts
- Add factor values persistence (parquet) in factor_runner.py
- Add CLI command: predix build-strategies-ai
- Integrate strategy building into QuantRDLoop (every 50 factors)
- Add StrategyBuilder design documentation

Closed Source Files (NOT committed, in local/):
- strategy_coster.py - Main CoSTEER loop for strategies
- strategy_evaluator.py - Walk-forward backtesting
- strategy_runner.py - Strategy execution
- strategy_discovery_v1.yaml - LLM prompts

Usage:
  predix build-strategies-ai              # Build from top 50 factors
  predix build-strategies-ai -t 100       # Use top 100 factors
  predix build-strategies-ai -l 10        # 10 improvement loops

The system:
1. Loads top factors with time-series values
2. LLM generates strategy hypotheses
3. LLM writes strategy code (entry/exit rules)
4. Backtests strategy with walk-forward validation
5. LLM gets feedback and improves
6. Repeats until profitable strategy found
2026-04-05 19:30:12 +02:00
TPTBusiness 7e8148c002 fix: Import pandas in predix portfolio_simple command 2026-04-05 13:36:13 +02:00
TPTBusiness 971a2535d6 feat: Improve predix portfolio command with robust error handling
Problem:
- Portfolio selection failed with 'Not enough valid overlapping data'
- Factors produce mostly NaN values or fail execution
- No diagnostics shown to user about why factors failed

Solution:
- Add detailed error tracking for each factor (timeout, no result, low values)
- Show summary of skipped factors with reasons
- Increase timeout to 2 minutes per factor
- Add fallback to show top factors by IC if portfolio fails
- Better progress messages showing valid value counts
- Handle symlink failures by copying data file instead
- Minimum 1000 non-NaN values required for factor to be included

Usage:
  predix portfolio          # Select top 10 from top 50
  predix portfolio -n 100   # Select from top 100 candidates
  predix portfolio -c 0.5   # Allow higher correlation (0.5)
2026-04-05 13:26:46 +02:00
TPTBusiness 12c949b0b4 fix: Add missing Panel import in predix evaluate command
Fixed NameError when running 'predix evaluate --all'
2026-04-05 11:49:26 +02:00
TPTBusiness 00a1d48aad feat: Add 'predix top' command + explain factor evaluation results
New CLI commands:
- predix top [-n 20] [-m ic|sharpe]: Show top factors by IC/Sharpe
- predix evaluate [--top N] [--all] [--force]: Evaluate factors

Why 100 new factors mostly failed:
- 56 factors have IC=None (factor values are all NaN or constant)
- LLM generates code that doesn't work with EURUSD 1min data
- Common issues: volume=0 causes division by zero, wrong MultiIndex handling
- Only 346 of 501 factors have valid IC values

Working factors (Top 5):
1. daily_close_open_mom     IC=0.255
2. daily_ret_log_1d         IC=0.255
3. daily_ret_close_1d       IC=0.255
4. daily_close_to_close_ret IC=0.255
5. daily_ret_vol_adj_1d     IC=0.235

Usage:
  predix top                   # Show top 20 by IC
  predix top -n 50             # Show top 50
  predix top -m sharpe         # Sort by Sharpe
  predix evaluate --all        # Evaluate all NEW factors
  predix evaluate --force      # Re-evaluate ALL
2026-04-05 11:47:25 +02:00
TPTBusiness f0814c6bf7 feat: Add 'predix evaluate' command to CLI
Integrated predix_full_eval.py into the main Predix CLI.

New command:
  predix evaluate [--top N] [--all] [--parallel P] [--force]

Features:
- Evaluates factors with full 1min data (2020-2026)
- Computes IC, Sharpe, Max DD, Win Rate
- Automatically skips already evaluated factors
- --force flag to re-evaluate all factors
2026-04-05 11:44:57 +02:00
TPTBusiness 68ea969c32 feat: Add parallel run system with API key distribution
- Add predix_parallel.py: Run multiple factor experiments concurrently
  * python predix_parallel.py --runs 5 --api-keys 2 -m openrouter
  * Round-robin API key distribution across available keys
  * Rich live dashboard with per-run status, elapsed time, exit codes
  * Graceful shutdown (Ctrl+C kills all children cleanly)

- Add --run-id parameter to predix.py for isolated single runs
  * Separate log files: fin_quant_run{N}.log
  * Separate results: results/runs/run{N}/
  * Separate workspace: RD-Agent_workspace_run{N}/
  * Separate databases per run

- Modify CoSTEER and FactorRunner for PARALLEL_RUN_ID isolation
  * _save_intermediate_results uses run-specific directories
  * _save_result_to_database and _write_run_log isolated per run
  * _ensure_results_dirs creates run-specific paths

- Reduce max_loop from 10 to 3 for faster iterations
- Add docs/parallel_runs.md with full documentation

Tests: 103 passed
2026-04-04 09:39:12 +02:00
TPTBusiness 127ee0f44f test: Add CLI model selection and logging tests (10 new tests)
- TestCLIModelSelection: 8 tests for predix.py CLI
  * predix module imports
  * fin_quant --model option
  * predix quant --model and --log-file options
  * OpenRouter API key validation
  * TeeWriter existence check
  * health and status commands

- TestLoggingTeeWriter: 2 tests for TeeWriter
  * Multi-stream writing
  * Broken stream handling

All 103 integration tests pass (93 + 10 new).

Also fix predix.py logging:
- Add --log-file flag (default: fin_quant.log)
- TeeWriter writes to both console AND file
- Works for both local and openrouter backends
2026-04-04 08:38:15 +02:00
TPTBusiness d3ae6df454 feat: Add CLI model selection (local vs OpenRouter)
- Add --model/-m flag to select LLM backend
  * local: llama.cpp on localhost (default)
  * openrouter: Cloud models via OpenRouter API
- Create predix.py as new CLI entry point with model selection
- Add OPENROUTER_API_KEY and OPENROUTER_MODEL to .env
- Add health and status commands to CLI
- Update rdagent/app/cli.py with model selection logic

Usage:
  predix quant                    # Local (default)
  predix quant -m openrouter      # OpenRouter cloud
  predix quant -m local -d        # Local + dashboard

Tests: 93 passed
2026-04-04 08:26:48 +02:00
TPTBusiness 30ee6c6221 chore: Remove unnecessary files for v1.0.0 release
Removed:
- Makefile (RD-Agent specific, not fully functional)
- predix.py (duplicates rdagent CLI)
- QWEN.md (internal dev guide - now in .gitignore)
- TODO.md (internal tracking - now in .gitignore)

Kept:
- pyproject.toml (required for pip install)
- start_loop.sh (useful for 24/7 trading)
- data_config.yaml (central configuration)
- start_loop.sh (24/7 trading)

.gitignore updated to exclude internal docs.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-04-02 21:12:58 +02:00
TPTBusiness b72cca9868 fix: CLI dashboard in separate terminal window
Problem: fin_quant overwrites dashboard output

Solution:
- CLI Dashboard (-c) starts in NEW terminal window
- Web Dashboard (-d) runs parallel in browser
- Both combinable: python predix.py fin_quant -d -c

Supported terminal emulators:
- gnome-terminal
- konsole
- xterm
- tilix

Warning displayed if no terminal is found.
2026-04-02 19:22:22 +02:00
TPTBusiness 757c66cddb feat: predix.py wrapper for dashboard support
Due to Typer CLI caching issues, created a wrapper script
that correctly supports dashboard options.

Usage:
  python predix.py fin_quant              # Normal
  python predix.py fin_quant -d           # Web Dashboard
  python predix.py fin_quant -c           # CLI Dashboard
  python predix.py fin_quant -d -c        # Both
  python predix.py fin_quant --help       # Help

Alternatively still available:
  rdagent fin_quant                       # Original CLI
  ./start_trading.sh                      # Interactive
  ./start_loop.sh                         # Endless loop
2026-04-02 19:21:37 +02:00