diff --git a/QWEN.md b/QWEN.md deleted file mode 100644 index c1a7db26..00000000 --- a/QWEN.md +++ /dev/null @@ -1,1921 +0,0 @@ -# Predix - QWEN.md Context File - -## Project Overview - -**Predix** is an autonomous AI-powered quantitative trading agent for EUR/USD forex markets. Built on the RD-Agent framework, it automates the full research and development cycle for trading strategies. - -### Core Purpose -- Generate trading factors (signals) autonomously using LLMs -- Backtest and validate factors on 1-minute EUR/USD data -- Generate AI strategies with LLM + REAL OHLCV backtest (96-bar forward returns) -- Optimize portfolios using modern portfolio theory -- Target: 1-3% monthly returns with Sharpe > 2.0 - -### Key Technologies -- **Python 3.10/3.11** - Primary language -- **PyTorch** - Deep learning models -- **Qlib** - Backtesting engine -- **LLM (Qwen3.5-35B via OpenRouter)** - Factor/strategy generation -- **Flask** - Web dashboard API -- **SQLite** - Results database -- **Rich/Typer** - CLI interface -- **Matplotlib/Seaborn** - Performance report charts - -### Architecture - -``` -Predix/ -├── rdagent/ # Core agent framework -│ ├── app/ -│ │ └── cli.py # Main CLI entry point (rdagent command) + P4 Commands -│ ├── components/ -│ │ ├── backtesting/ # Backtest engine, metrics, database -│ │ ├── coder/ -│ │ │ ├── factor_coder/ # Factor generation & EURUSD-specific modules -│ │ │ ├── strategy_orchestrator.py # P2: Strategy generation from factors -│ │ │ ├── optuna_optimizer.py # P3: Optuna hyperparameter optimization -│ │ │ └── rl/ # RL Trading Agent -│ │ ├── loader.py # Prompt loader (auto-loads local prompts) -│ │ └── model_loader.py # Model loader (auto-loads local models) -│ └── scenarios/ -│ └── qlib/ # Qlib integration for FX trading -│ └── local/ # Closed source components (NOT in Git!) -│ ├── data_loader.py # OHLCV & factor data loader -│ ├── strategy_worker.py # LLM strategy generation + backtest -│ ├── strategy_coster.py # StrategyCoSTEER (LLM strategy gen) -│ ├── strategy_evaluator.py # Comprehensive strategy metrics -│ ├── strategy_runner.py # Strategy execution & backtesting -│ ├── ml_trainer.py # ML model training on factors -│ ├── feedback_integrator.py # P6: ML feedback into factor loop -│ ├── portfolio_optimizer.py # P7: Mean-variance & risk parity -│ └── strategy_discovery_v1.yaml # LLM prompts for strategy gen -├── predix.py # Main CLI wrapper (predix.py commands) -├── predix_parallel.py # Parallel factor evolution -├── predix_gen_strategies_real_bt.py # AI Strategy Gen + REAL OHLCV Backtest -├── predix_strategy_report.py # Performance report generator (charts + PDF) -├── debug_backtest.py # Debug backtest alignment & IC -├── prompts/ # LLM Prompts -│ ├── standard_prompts.yaml # Standard prompts (in Git) -│ └── local/ # Your improved prompts (NOT in Git!) -├── models/ # ML Models -│ ├── standard/ # Standard models (in Git) -│ └── local/ # Your improved models (NOT in Git!) -├── results/ # Backtest results (NOT in git) -│ ├── factors/ # ~872 evaluated factors -│ │ └── values/ # Factor time-series parquet files (862) -│ ├── strategies_new/ # AI-generated strategies with real backtests -│ └── strategy_reports/ # Performance reports with charts -├── git_ignore_folder/ # OHLCV data (intraday_pv.h5) -└── .env # Environment config (API keys) -``` - -### CLI Commands Reference - -#### Trading Loop -```bash -rdagent fin_quant # Start factor evolution -rdagent fin_quant --loop-n 5 # 5 evolution loops -rdagent fin_quant --with-dashboard # With web dashboard -rdagent fin_quant --cli-dashboard # With CLI Rich dashboard -rdagent fin_quant --auto-strategies # Auto-generate strategies after threshold -rdagent fin_quant --auto-strategies --auto-strategies-threshold 1000 -``` - -#### Strategy Generation (P4 - NEW) -```bash -rdagent generate_strategies # Generate 10 strategies (default) -rdagent generate_strategies -n 20 -w 8 # 20 strategies, 8 workers -rdagent generate_strategies -s daytrading # Day trading style -rdagent generate_strategies --no-optuna # Skip Optuna optimization -rdagent generate_strategies --optuna-trials 50 # 50 Optuna trials per strategy -``` - -#### Portfolio Optimization (P4 - NEW) -```bash -rdagent optimize_portfolio # Mean-variance, top 30 strategies -rdagent optimize_portfolio --method risk_parity # Risk parity weighting -rdagent optimize_portfolio --top-n 20 # Top 20 strategies only -``` - -#### Strategy Reports (P4 - NEW) -```bash -rdagent strategies_report # Reports for ALL strategies -rdagent strategies_report -s path/to/strategy.json # Single strategy -rdagent strategies_report -o custom/reports/ # Custom output directory -``` - -#### Parallel Execution -```bash -python predix_parallel.py --runs 5 --api-keys 1 -m openrouter # 5 parallel runs -python predix_parallel.py --runs 20 --api-keys 2 -m openrouter # 20 runs, 2 keys -``` - -#### AI Strategy Generation (REAL OHLCV Backtest) -```bash -python predix_gen_strategies_real_bt.py # Generate 10 strategies -python predix_gen_strategies_real_bt.py 20 # Generate 20 strategies -python predix_gen_strategies_real_bt.py 5 # Generate 5 (faster test) -``` -Each accepted strategy gets: -- JSON file in `results/strategies_new/` -- Performance report with charts in `results/strategy_reports/` -- Dashboard PNG (equity curve, drawdown, signals, monthly returns) -- Text report with full metrics - -#### Strategy Reports -```bash -python predix_strategy_report.py # Reports for ALL strategies -python predix_strategy_report.py # Report for single strategy -``` - -#### Factor Evaluation -```bash -python predix.py evaluate --all # Evaluate all factors -python predix.py top -n 20 # Top 20 factors by IC -python predix.py portfolio-simple # Portfolio optimization -``` - -#### Debug -```bash -python debug_backtest.py # Debug alignment & IC -``` - ---- - -## 🚀 Live Trading System (cTrader + FTMO) - -### Overview - -Predix includes a **complete live trading system** that executes strategies on cTrader via Open API with FTMO broker. - -**All live trading code is CLOSED SOURCE** and stored in `git_ignore_folder/` (never committed to Git). - -### Architecture - -``` -┌──────────────────────────────────────────────────────────────┐ -│ PREDIX LIVE TRADING │ -├──────────────────────────────────────────────────────────────┤ -│ │ -│ Strategy JSON → Factor Calculator → Signal Generator │ -│ ↓ ↓ ↓ │ -│ results/strategies Live OHLCV Data LONG/SHORT │ -│ _new/*.json (cTrader API) /NEUTRAL │ -│ ↓ │ -│ Risk Manager │ -│ ↓ │ -│ cTrader Orders API │ -│ ↓ │ -│ FTMO Account (Live) │ -│ │ -│ Logging: results/live_trading/ │ -│ - trades_*.json (trade log) │ -│ - trading_*.log (detailed log) │ -└──────────────────────────────────────────────────────────────┘ -``` - -### Files (Closed Source) - -``` -git_ignore_folder/ -├── predix_live_trader.py ← Main live trading script -└── LIVE_TRADING_SETUP.md ← Setup guide - -results/live_trading/ -├── trades_*.json ← Trade log -└── trading_*.log ← Detailed log -``` - -### Prerequisites - -1. **cTrader Account** with FTMO broker -2. **cTrader Open API** credentials: https://developers.ctrader.com/ - - Client ID - - Client Secret - - Broker ID - - Access Token -3. **Python 3.10+** with `requests`, `pandas`, `numpy`, `python-dotenv` - -### Setup cTrader API - -1. **Register Application:** - - Go to https://developers.ctrader.com/ - - Login with your cTrader credentials - - Create new application - - Note down: Client ID, Client Secret, Broker ID - -2. **Generate Access Token:** - - OAuth2 flow or generate in dashboard - - Token expires - refresh as needed - -3. **Configure .env:** -```bash -# Add to .env file: -CTRADE_API_BASE=https://api.ctrader.com -CTRADE_CLIENT_ID=your_client_id -CTRADE_CLIENT_SECRET=your_client_secret -CTRADE_ACCESS_TOKEN=your_access_token -CTRADE_BROKER_ID=your_broker_id - -# Trading parameters -TRADING_SYMBOL=EURUSD -TRADING_TIMEFRAME=M1 -DEFAULT_LOT_SIZE=0.01 -MAX_DAILY_LOSS_PCT=2.0 -MAX_POSITIONS=1 -``` - -### How It Works - -#### 1. **Strategy Loading** -```python -# Loads strategy from JSON -strategy = json.load(open('results/strategies_new/123_MomentumDivergenceZScore.json')) -code = strategy['code'] # Strategy Python code -factors = strategy['factor_names'] # Factor names list -``` - -#### 2. **Factor Calculation** -```python -# Computes factors from live OHLCV -ohlcv = client.get_ohlcv('EURUSD', 'M1', count=1000) -factors_df = compute_factors(ohlcv) -# Calculates: daily_close_return_96, daily_session_momentum_divergence_1d, etc. -``` - -#### 3. **Signal Generation** -```python -# Executes strategy code -exec(strategy_code, {'factors': factors_df}, local_vars) -signal = local_vars['signal'] # 1=LONG, -1=SHORT, 0=NEUTRAL -``` - -#### 4. **Order Execution** -```python -if signal != last_signal and signal != 0: - # Close opposite positions - if signal == 1: close_all_shorts() - if signal == -1: close_all_longs() - - # Place new order - client.place_order( - symbol='EURUSD', - side='LONG' if signal == 1 else 'SHORT', - lot_size=calculate_position_size(), - stop_loss=0.0050, # 50 pips - take_profit=0.0100, # 100 pips - comment='Predix-{strategy_name}' - ) -``` - -#### 5. **Risk Management** -- **Daily Loss Limit:** Stops trading if daily loss > 2% -- **Max Positions:** Only 1 position at a time -- **Position Sizing:** Dynamic based on balance and ATR -- **Stop Loss:** 50 pips automatic -- **Take Profit:** 100 pips automatic - -### Usage - -#### Paper Trading (TEST FIRST!) -```bash -python git_ignore_folder/predix_live_trader.py \ - --strategy results/strategies_new/1775543215_MomentumDivergenceZScore.json \ - --paper -``` - -#### Live Trading (REAL MONEY) -```bash -python git_ignore_folder/predix_live_trader.py \ - --strategy results/strategies_new/1775543215_MomentumDivergenceZScore.json \ - --lot-size 0.01 -``` - -#### Custom Parameters -```bash -python git_ignore_folder/predix_live_trader.py \ - --strategy results/strategies_new/123_MyStrategy.json \ - --lot-size 0.02 \ - --symbol EURUSD \ - --timeframe M5 -``` - -### CLI Options - -| Option | Short | Description | Default | -|--------|-------|-------------|---------| -| `--strategy` | `-s` | Path to strategy JSON | Required | -| `--paper` | `-p` | Paper trading mode | False | -| `--lot-size` | `-l` | Fixed lot size | 0.01 | -| `--symbol` | | Trading symbol | EURUSD | -| `--timeframe` | | Timeframe | M1 | - -### Monitoring - -#### Log Files -```bash -# View trade log -cat results/live_trading/trades_*.json | jq . - -# View detailed log -tail -f results/live_trading/trading_*.log -``` - -#### Trade Log Format -```json -[ - { - "timestamp": "2026-04-07T12:05:30", - "signal": 1, - "side": "LONG", - "lot_size": 0.01, - "result": { "orderId": "12345", "price": 1.08500 } - } -] -``` - -### ⚠️ Critical Warnings - -1. **ALWAYS test in paper mode first** - Never go live without testing -2. **Start small** - Use 0.01 lots initially -3. **Monitor daily** - Check logs every day -4. **FTMO rules** - Respect max drawdown limits (usually 10%) -5. **Token expiry** - Refresh API tokens before they expire -6. **Internet required** - System stops if connection drops -7. **No guarantees** - Past performance ≠ future results - -### Troubleshooting - -| Error | Cause | Solution | -|-------|-------|----------| -| "Connection failed" | Wrong API credentials | Check .env values | -| "No OHLCV data" | cTrader not running | Start cTrader platform | -| "Signal error" | Missing factors | Strategy needs factors not in live data | -| "Order failed" | Insufficient margin | Check FTMO account balance | -| "Daily loss limit" | Hit 2% daily loss | System stopped - wait for next day | - -### cTrader API Endpoints - -The system uses these cTrader Open API endpoints: - -``` -GET /api/accounts # Get account info -GET /api/positions # Get open positions -GET /api/cbars # Get OHLCV data -POST /api/orders # Place order -DELETE /api/positions/{id} # Close position -``` - -### Future Enhancements - -- [ ] Multi-strategy portfolio trading -- [ ] Dynamic stop loss/take profit -- [ ] Trailing stop loss -- [ ] Webhook alerts for trades -- [ ] Telegram notifications -- [ ] Auto-restart on disconnect -- [ ] Backtest with live data sync - -│ └── local/ # Your improved models (NOT in Git!) -│ ├── transformer_factor.py -│ ├── tcn_factor.py -│ ├── patchtst_factor.py -│ └── cnn_lstm_hybrid.py -├── results/ # Backtest results (NOT in git) -│ ├── backtests/ # Individual factor backtests (JSON/CSV) -│ ├── db/ # SQLite database -│ ├── factors/ # Factor analysis -│ ├── runs/ # Run results & risk reports -│ └── logs/ # Backtest logs -├── web/ # Dashboard frontend -│ ├── dashboard_api.py # Flask API backend -│ └── dashboard.html # Web UI -├── .env # Environment config (API keys, etc.) -├── data_config.yaml # EURUSD data configuration -└── requirements.txt # Python dependencies -``` - -### Open Source vs. Closed Source - -**🟢 OPEN SOURCE (Public on GitHub - FULLY WORKING):** -- `rdagent/` - Core framework (ALL components) -- `models/standard/` - Base models (XGBoost, LightGBM) -- `prompts/standard_prompts.yaml` - Base prompts -- `web/` - Dashboards -- `test/` - ALL tests (integration, unit, security) -- `rdagent/components/coder/rl/` - RL Trading System (with fallback) -- `rdagent/components/backtesting/protections/` - Trading Protection System -- `scripts/` - Utility scripts - -**GitHub users get:** -✅ Full working trading system -✅ RL Trading with graceful fallback (no stable-baselines3 needed) -✅ Protection Manager (drawdown, cooldown, stoploss guard) -✅ Backtesting Engine with RL support -✅ CLI commands (`fin_quant`, `rl_trading`, etc.) -✅ Web and CLI dashboards -✅ All 200+ integration tests - -**🔒 CLOSED SOURCE (Local Only - NOT on GitHub):** -- `models/local/` - Your improved models (Transformer, TCN, PatchTST, CNN+LSTM) -- `prompts/local/` - Your improved prompts (v2.0 optimized) -- `rdagent/scenarios/qlib/local/` - Advanced components: - - `strategy_coster.py` - StrategyCoSTEER (LLM strategy generation) - - `strategy_evaluator.py` - Comprehensive strategy metrics - - `strategy_runner.py` - Strategy execution & backtesting - - `strategy_discovery_v1.yaml` - LLM prompts for strategy generation - - Plus: ml_trainer, portfolio_optimizer, quant_loop_advanced, etc. -- `.env` - API keys -- `results/` - Backtest results -- `git_ignore_folder/` - Trading data -- `QWEN.md`, `TODO.md` - Internal docs - -**Protection:** -- `.gitignore` excludes all `local/` directories -- Your competitive edge (alpha) stays private -- Framework is open, but your best models/prompts are closed - -### Open Source Fallback Strategy - -**For users without stable-baselines3:** -The RL system provides graceful degradation: -- ❌ No stable-baselines3 → Uses simple momentum-based fallback -- ✅ Still fully functional: CLI, backtesting, protections work -- ✅ No errors or broken features -- ✅ Clear warning message with installation instructions - -**For users without LLM (llama.cpp):** -- Factor evolution degrades gracefully -- System still works with standard models -- Clear error messages for missing LLM - -**PRINCIPLE:** Every GitHub user MUST be able to run the full system. Missing optional components should never break the project. - -## Building and Running - -### Installation - -```bash -# Clone repository -git clone https://github.com/PredixAI/predix -cd predix - -# Create conda environment -conda create -n predix python=3.10 -conda activate predix - -# Install in editable mode -pip install -e .[test,lint] -``` - -### Configuration - -1. **Create `.env` file:** -```bash -# Local LLM (llama.cpp) -OPENAI_API_KEY=local -OPENAI_API_BASE=http://localhost:8081/v1 -CHAT_MODEL=qwen3.5-35b - -# Embedding (Ollama) -LITELLM_PROXY_API_KEY=local -LITELLM_PROXY_API_BASE=http://localhost:11434/v1 -EMBEDDING_MODEL=nomic-embed-text - -# Paths -QLIB_DATA_DIR=~/.qlib/qlib_data/eurusd_1min_data -``` - -2. **Start LLM server (llama.cpp):** -```bash -~/llama.cpp/build/bin/llama-server \ - --model ~/models/qwen3.5/Qwen3.5-35B-A3B-Q3_K_M.gguf \ - --n-gpu-layers 36 \ - --ctx-size 80000 \ - --port 8081 -``` - -### Running the Trading Loop - -```bash -# Start trading loop (24/7) -./start_loop.sh - -# Or single run -rdagent fin_quant - -# With dashboard -rdagent fin_quant --with-dashboard - -# With CLI dashboard -rdagent fin_quant --cli-dashboard -``` - -### Running the Dashboard - -```bash -# Web dashboard (runs with fin_quant --with-dashboard) -# Access at: http://localhost:5000/dashboard.html - -# Or standalone -python web/dashboard_api.py -``` - -### Testing - -#### Integration Test Suite (ALL Features) - -**Comprehensive test system that validates ALL 13 implemented features:** - -```bash -# Run ALL integration tests (60 tests, ~7.5 seconds) -pytest test/integration/test_all_features.py -v - -# Run with coverage report -pytest test/integration/test_all_features.py --cov=rdagent.components.backtesting -v - -# Run via test runner script -./scripts/run_all_tests.sh - -# Test specific features only -pytest test/integration/test_all_features.py -k "backtest or database" -v - -# Skip slow tests -pytest test/integration/test_all_features.py -m "not slow" -v -``` - -**Tested Features (60 Tests, ALL MUST PASS):** - -| # | Feature | Tests | Status | -|---|---------|-------|--------| -| 1 | Factor Evolution | 5 | ✅ LLM generates trading factors autonomously | -| 2 | Model Evolution | 5 | ✅ ML models auto-improved | -| 3 | Quant Loop (fin_quant) | 4 | ✅ Main 24/7 trading loop | -| 4 | Backtesting Engine | 5 | ✅ IC, Sharpe, Drawdown, Win Rate | -| 5 | Results Database | 5 | ✅ SQLite with queries | -| 6 | Risk Management | 6 | ✅ Correlation, Portfolio Optimization | -| 7 | CLI Dashboard | 4 | ✅ Rich live-progress display | -| 8 | Web Dashboard | 4 | ✅ Flask API + HTML | -| 9 | Health Check | 4 | ✅ Environment validation | -| 10 | Streamlit UI | 3 | ✅ Alternative dashboard | -| 11 | LLM Integration | 5 | ✅ llama.cpp (Qwen3.5-35B) | -| 12 | Embedding | 3 | ✅ Ollama (nomic-embed-text) | -| 13 | Security Scanning | 5 | ✅ Bandit pre-commit hook | - -**⚠️ MANDATORY: These tests run BEFORE every commit and MUST pass!** - -#### Unit Tests - -```bash -# Run all unit tests -pytest test/ - -# Run with coverage -pytest --cov=rdagent --cov-report=html - -# Test backtesting module -python rdagent/components/backtesting/backtest_engine.py -python rdagent/components/backtesting/results_db.py -python rdagent/components/backtesting/risk_management.py -``` - -### Code Quality - -```bash -# Linting -ruff check rdagent/ - -# Type checking -mypy rdagent/ - -# Format -black rdagent/ - -# Pre-commit (install first) -pre-commit install -pre-commit run --all-files -``` - -## Development Conventions - -### Language Policy - -**ALL code comments and documentation MUST be in English.** - -❌ **Wrong (German):** -```python -# Inspiriert von: TradingAgents -# Berechnet den Sharpe Ratio -# Achtung: Division durch Null möglich! -# Hinweis: Diese Funktion ist experimentell -``` - -✅ **Correct (English):** -```python -# Inspired by: TradingAgents -# Calculates the Sharpe ratio -# Warning: Division by zero possible! -# Note: This function is experimental -``` - -**Rationale:** -- International collaboration -- Better searchability -- Professional codebase -- Consistent with commit messages (also English-only) - -**Enforcement:** -- All new code must have English comments -- Existing German comments should be translated when modified -- PRs with German comments will be rejected - -### Code Style - -- **Line length:** 120 characters (configured in pyproject.toml) -- **Type hints:** Required for all public functions -- **Docstrings:** Google style for public APIs -- **Imports:** Sorted automatically with isort - -### Testing Practices -- Unit tests in `test/` directory -- Test files named `test_*.py` -- Use pytest fixtures for common setup -- Mock external APIs (LLM, yfinance) -- Minimum 80% coverage target - -### Commit Conventions -```bash -git commit --author="TPTBusiness " -m "type: description" - -# Types: -# - feat: New feature -# - fix: Bug fix -# - docs: Documentation -# - style: Formatting -# - refactor: Code restructuring -# - test: Tests -# - chore: Maintenance -``` - -### Module Structure -```python -""" -Module Name - Brief description - -Longer description if needed. -""" - -import numpy as np -import pandas as pd -from typing import Dict, List, Optional -from datetime import datetime - -class ClassName: - """Class docstring.""" - - def __init__(self, param: type) -> None: - """Initialize.""" - pass - - def method(self, param: type) -> ReturnType: - """ - Method docstring. - - Parameters - ---------- - param : type - Description - - Returns - ------- - ReturnType - Description - """ - pass -``` - -### Backtesting Module Usage - -```python -from rdagent.components.backtesting import ( - FactorBacktester, - ResultsDatabase, - PortfolioOptimizer, - AdvancedRiskManager -) - -# Run backtest -backtester = FactorBacktester() -metrics = backtester.run_backtest( - factor_values=factor_series, - forward_returns=forward_returns, - factor_name="MyFactor" -) - -# Save to database -db = ResultsDatabase() -db.add_backtest("MyFactor", metrics) - -# Query top factors -top = db.get_top_factors('sharpe_ratio', limit=20) - -# Portfolio optimization -optimizer = PortfolioOptimizer() -weights = optimizer.mean_variance(expected_returns, cov_matrix) - -# Risk management -risk_manager = AdvancedRiskManager() -report = risk_manager.generate_risk_report(returns, weights) -``` - -### Key Metrics - -| Metric | Target | Minimum | -|--------|--------|---------| -| IC (Information Coefficient) | > 0.05 | > 0.02 | -| Sharpe Ratio | > 2.0 | > 1.0 | -| Max Drawdown | < 15% | < 25% | -| Win Rate | > 55% | > 45% | -| Annualized Return | > 10% | > 5% | - -### Important Files - -- `rdagent/app/cli.py` - Main CLI entry point -- `rdagent/components/backtesting/` - Backtest engine -- `rdagent/components/coder/factor_coder/` - Factor generation -- `results/README.md` - Results documentation -- `data_config.yaml` - EURUSD configuration -- `web/dashboard_api.py` - Dashboard API -- `requirements.txt` - Dependencies - -### External Dependencies - -- **llama.cpp** - Local LLM inference (Qwen3.5-35B) -- **Ollama** - Embedding models -- **Qlib** - Backtesting engine -- **yfinance** - Live market data - -### Common Issues - -1. **LLM Connection Errors:** Ensure llama.cpp server is running on port 8081 -2. **Embedding Errors:** Check Ollama is running with nomic-embed-text loaded -3. **Database Lock:** Close all connections before running multiple processes -4. **Memory Issues:** Reduce batch size or context length for LLM - -### Project Status - -- ✅ Factor Generation (110+ factors created) -- ✅ Backtesting Engine (IC, Sharpe, Drawdown, RL support) -- ✅ Results Database (SQLite with queries) -- ✅ Risk Management (Correlation, Portfolio Optimization) -- ✅ Trading Protection System (Drawdown, Cooldown, Stoploss Guard, Low Performance) -- ✅ RL Trading Agent (PPO/A2C/SAC with Gymnasium environment + fallback) -- ✅ Strategy Orchestrator (P2 - LLM factor combination + strategy generation) -- ✅ Optuna Optimizer (P3 - Hyperparameter optimization for strategies) -- ✅ CLI Commands (P4 - generate_strategies, optimize_portfolio, strategies_report) -- ✅ Auto-Strategies Hook (fin_quant --auto-strategies integration) -- ✅ Strategy Worker (LLM strategy generation + FTMO-compliant backtesting) -- ✅ Data Loader (OHLCV + factor data loading with caching) -- ✅ ML Feedback Integrator (P6 - Auto-triggers for ML/strategy/portfolio at factor milestones) -- ✅ Portfolio Optimizer (P7 - Mean-Variance, Risk Parity, IC-Weighted, Correlation Analysis) -- ✅ Integration Tests P6-P8 (27 new tests, full pipeline end-to-end) -- ✅ Dashboards (Web + CLI) -- ✅ CLI Commands (`fin_quant`, `rl_trading`, `generate_strategies`, `optimize_portfolio`, etc.) -- ✅ Integration Tests (247+ tests, run before EVERY commit) -- ✅ Security Scanning (Bandit pre-commit hook) -- ⏳ Live Trading (Paper trading - in development) - -### Next Steps - -1. ✅ Connect RL with Protection Manager (DONE) -2. ✅ Connect RL with Backtesting Engine (DONE) -3. ✅ Add CLI command for RL Trading (DONE) -4. ✅ Ensure GitHub users can run full system (DONE - fallback system) -5. ✅ P2: Strategy Orchestrator (DONE) -6. ✅ P3: Optuna Optimizer (DONE) -7. ✅ P4: CLI Commands (DONE) -8. ✅ P6: Feedback Loop (DONE - MLFeedbackMixin hooks into QuantRDLoop.feedback) -9. ✅ P7: Portfolio Optimizer (DONE - Mean-Variance + Risk Parity + IC-Weighted) -10. ✅ P8: Integration Tests (DONE - 27 tests for P6-P8 + full pipeline) -11. ✅ P9: Documentation (DONE - QWEN.md updated) -12. Backtest all 110 factors -13. Select top 20 by IC/Sharpe -10. Portfolio optimization -8. 4 weeks paper trading -9. Live trading with small capital - ---- - -## Git Commit Guidelines - -### Language Policy - -**ALL commit messages MUST be in English.** - -❌ **Wrong (German):** -```bash -git commit -m "feat: Neue Funktion hinzugefügt" -git commit -m "fix: Fehler behoben" -git commit -m "chore: QWEN.md zu .gitignore hinzugefügt" -``` - -✅ **Correct (English):** -```bash -git commit -m "feat: Add new feature" -git commit -m "fix: Fix bug" -git commit -m "chore: Add QWEN.md to .gitignore" -``` - -### Pre-Commit Checklist - -**BEFORE every commit, you MUST:** - -1. **Run `git status`** and verify: - - Only intended files are staged - - No generated files (.qwen/, results/, *.db, etc.) - - No sensitive data (.env, API keys, etc.) - -2. **Check .gitignore** is working: - ```bash - git status - # Verify .qwen/, results/, *.db are NOT shown - ``` - -3. **Review staged changes:** - ```bash - git diff --staged - # Review what will be committed - ``` - -4. **Run tests** (if applicable): - ```bash - pytest test/backtesting/ -v - # Ensure all tests pass - ``` - -### Commit Message Format - -Use [Conventional Commits](https://www.conventionalcommits.org/): - -``` -: - -[optional body] -``` - -**Types:** -- `feat:` - New feature -- `fix:` - Bug fix -- `test:` - Tests -- `docs:` - Documentation -- `chore:` - Maintenance -- `style:` - Formatting -- `refactor:` - Code restructuring - -**Examples:** -```bash -feat: Add backtesting tests with 98% coverage -fix: Remove .qwen/ from Git tracking -test: Add unit tests for ResultsDatabase -docs: Update QWEN.md with commit guidelines -chore: Add pytest to requirements.txt -``` - -### Protected Files (NEVER commit) - -These files/directories MUST NEVER be committed: - -``` -.qwen/ # AI agent files (generated) -results/ # Backtest results (sensitive data) -*.db # SQLite databases -.env # Environment variables (API keys!) -git_ignore_folder/ # Generated data -*.log # Log files -``` - -If you accidentally commit any of these: - -```bash -# Remove from last commit (keeps files locally) -git reset HEAD~1 - -# Or remove from tracking -git rm -r --cached .qwen/ -git commit -m "chore: Remove .qwen/ from tracking" -``` - -### Fixing Past Commits - -**To fix the last 3-5 commits:** - -```bash -# For last 5 commits -git rebase -i HEAD~5 - -# In the editor, change 'pick' to 'reword' for commits to rename -# Save and close -# Write new English message for each commit -``` - -**To fix older commits (advanced):** - -```bash -# Find the commit hash -git log --oneline - -# Start rebase from that commit -git rebase -i ^ - -# Follow same process as above -``` - -**Current German commits to fix (as of April 2026):** -``` -73140b68 test: Backtesting Tests mit 98.77% Coverage - → test: Add backtesting tests with 98.77% coverage - -5148d17d chore: QWEN.md zu .gitignore hinzugefügt - → chore: Add QWEN.md to .gitignore - -df93e162 feat: Intelligent Embedding Chunking statt Kürzung - → feat: Intelligent embedding chunking instead of truncation - -01aa183a fix: CLI Dashboard in separatem Terminal-Fenster - → fix: CLI dashboard in separate terminal window - -df356978 feat: predix.py Wrapper für Dashboard-Support - → feat: predix.py wrapper for dashboard support - -89d01f5d feat: Beautiful CLI Dashboard + korrigierter Start-Befehl - → feat: Beautiful CLI dashboard + corrected start command - -48e4f44e feat: Auto-Start Dashboard für fin_quant - → feat: Auto-start dashboard for fin_quant - -59122a19 feat: Dashboard + Live-Daten Integration (Phase 4) - → feat: Dashboard + live data integration (Phase 4) - -a0f414ed feat: EURUSD Trading-Verbesserungen (Phase 2 & 3) - → feat: EURUSD trading improvements (Phase 2 & 3) - -e8b962b5 feat: EURUSD Trading-Verbesserungen implementiert (Phase 1) - → feat: Implement EURUSD trading improvements (Phase 1) -``` - -**⚠️ Warning:** Rewriting history changes commit hashes. If you've already pushed: - -```bash -# After rebasing locally -git push --force-with-lease origin master - -# Tell team members to re-clone: -git clone -``` - -### Push Policy - -**BEFORE pushing:** - -1. Verify commit messages are in English -2. Verify no protected files are included -3. Run tests one final time - -```bash -git status -git log -3 --oneline # Verify last 3 commits -pytest test/backtesting/ -v # Quick test -git push origin master -``` - -### Enforcement - -- All PRs will be rejected if commit messages are not in English -- Protected files in commits will be rejected -- Tests must pass before merging - -**Remember:** Consistent English commit messages ensure: -- International collaboration -- Better searchability -- Professional project history - ---- - -## Implementation Guide: Prompts & Models - -### Using the Prompt Loader - -**Auto-Load Prompts (Local First):** - -```python -from rdagent.components.loader import load_prompt - -# Load factor discovery prompt -# Automatically loads from prompts/local/ if exists! -prompt = load_prompt("factor_discovery") - -# Load specific section -system_prompt = load_prompt("factor_discovery", section="system") -user_prompt = load_prompt("factor_discovery", section="user") - -# Force local only (raise error if not found) -prompt = load_prompt("factor_discovery", local_only=True) - -# List available prompts -from rdagent.components.loader import list_available_prompts -available = list_available_prompts() -print(f"Standard: {available['standard']}") -print(f"Local: {available['local']}") -``` - -**Priority:** -1. `prompts/local/factor_discovery_v2.yaml` (loaded first if exists) -2. `prompts/local/factor_discovery.yaml` -3. `prompts/standard_prompts.yaml` (fallback) - ---- - -### Using the Model Loader - -**Auto-Load Models (Local First):** - -```python -from rdagent.components.model_loader import load_model - -# Load XGBoost model -# Automatically loads from models/local/ if exists! -model_factory = load_model("xgboost_factor") - -# Create model instance -model = model_factory(max_depth=8, learning_rate=0.03) - -# Train -model.fit(X_train, y_train, epochs=50, batch_size=64) - -# Predict -predictions = model.predict(X_test) - -# Save/Load -model.save("models/my_model.pth") -model.load("models/my_model.pth") -``` - -**Available Models:** - -| Model | Location | Use Case | -|-------|----------|----------| -| `xgboost_factor` | `models/standard/` | Tabular data, fast training | -| `lightgbm_factor` | `models/standard/` | Large datasets, faster than XGBoost | -| `transformer_factor` | `models/local/` | Time-series, long-range dependencies | -| `tcn_factor` | `models/local/` | Multi-scale patterns | -| `patchtst_factor` | `models/local/` | **SOTA** for time-series forecasting | -| `cnn_lstm_hybrid` | `models/local/` | Complex pattern recognition | - -**Priority:** -1. `models/local/{name}_v2.py` (loaded first if exists) -2. `models/local/{name}.py` -3. `models/standard/{name}.py` (fallback) - ---- - -### Creating Your Improved Prompts - -**Step 1: Create Local Prompt** - -```bash -mkdir -p prompts/local -nano prompts/local/factor_discovery_v3.yaml -``` - -**Step 2: Add Your Improvements** - -```yaml -# prompts/local/factor_discovery_v3.yaml - -factor_discovery: - system: |- - YOUR IMPROVED SYSTEM PROMPT HERE - - Add your proprietary insights: - - Specific EURUSD patterns you've discovered - - Your unique factor formulas - - Custom session filters - - Proprietary risk management rules - - user: |- - YOUR IMPROVED USER PROMPT HERE -``` - -**Step 3: Test** - -```python -from rdagent.components.loader import load_prompt - -# Auto-loads your v3! -prompt = load_prompt("factor_discovery") -``` - ---- - -### Creating Your Improved Models - -**Step 1: Create Local Model** - -```bash -mkdir -p models/local -nano models/local/my_optimized_model.py -``` - -**Step 2: Implement Model** - -```python -# models/local/my_optimized_model.py -""" -My Optimized Model v1.0 -Better than standard with custom improvements. -""" - -import torch -import torch.nn as nn - -class MyOptimizedModel(nn.Module): - def __init__(self, **params): - super().__init__() - # Your custom architecture - pass - - def forward(self, x): - # Your custom forward pass - pass - -def create_my_optimized_model(**params): - """Factory function.""" - return MyOptimizedModel(**params) -``` - -**Step 3: Test** - -```python -from rdagent.components.model_loader import load_model - -# Auto-loads your optimized model! -model_factory = load_model("my_optimized_model") -model = model_factory() -``` - ---- - -### Backup Your Private Assets - -**Backup Prompts & Models to Private Repo:** - -```bash -# Create private repo on GitHub: predix-private-assets - -# Clone private repo -cd ~/Dev -git clone git@github.com:TPTBusiness/predix-private-assets.git - -# Copy local assets -cp -r ~/Predix/prompts/local/* ~/predix-private-assets/prompts/ -cp -r ~/Predix/models/local/* ~/predix-private-assets/models/ - -# Commit to private repo -cd ~/predix-private-assets -git add . -git commit -m "Backup: prompts v2, models (Transformer, TCN, PatchTST, CNN+LSTM)" -git push -``` - -**Auto-Sync Script:** - -```bash -# ~/Predix/sync_private.sh -#!/bin/bash -echo "Syncing private assets..." -rsync -av prompts/local/ ~/predix-private-assets/prompts/ -rsync -av models/local/ ~/predix-private-assets/models/ -cd ~/predix-private-assets && git add . && git commit -m "Auto-sync $(date)" && git push -echo "Done!" -``` - ---- - -### Security Best Practices - -**What to Keep Private:** - -✅ Your proprietary model architectures -✅ Optimized prompt templates -✅ Best-performing factors -✅ Evolution weights -✅ Trade secrets & alpha-generating logic - -**What NOT to Commit:** - -❌ Anything in `prompts/local/` -❌ Anything in `models/local/` -❌ `.env` (API keys) -❌ `results/` (backtest performance) -❌ `git_ignore_folder/` (trading data) - -**Verify Before Committing:** - -```bash -# Check what will be committed -git status -git diff --staged - -# Verify .gitignore is working -git status -# Should NOT show prompts/local/, models/local/, .env, results/ -``` - ---- - -## Development Guidelines for AI Assistant - -### 🌍 CRITICAL: Open Source Compatibility - -**BEFORE implementing ANY feature, ask yourself:** - -1. **Can a GitHub user run this without our local files?** - - ✅ YES → Good, proceed - - ❌ NO → Add fallback or graceful degradation - -2. **Does this break if optional dependencies are missing?** - - Example: `stable-baselines3`, `llama.cpp`, `Ollama` - - Solution: Try/except with clear warning messages - -3. **Is this feature documented for external users?** - - Update README.md with usage instructions - - Ensure installation guide covers all dependencies - -**PRINCIPLE:** The project on GitHub MUST be fully functional for users. Our closed-source assets (`models/local/`, `prompts/local/`, `.env`) are ENHANCEMENTS, not requirements. - -### ⚠️ MANDATORY Rules for ALL Development - -**When implementing NEW features or making SIGNIFICANT changes, you MUST:** - -#### 1. 📝 Update QWEN.md - -**When:** Every time you add a new feature, module, or change existing architecture. - -**What to update:** -- Architecture section (if structure changes) -- Important Files section -- Testing section -- Key Metrics (if targets change) -- Project Status -- Next Steps - -**Example:** -```markdown -### Architecture -├── rdagent/ -│ └── components/ -│ └── backtesting/ -│ └── protections/ # NEW: Trading protection system -│ ├── base.py -│ ├── max_drawdown.py -│ └── protection_manager.py -``` - -#### 2. 📖 Update README.md - -**When:** Every user-facing feature change or major update. - -**What to update:** -- Features list -- Installation instructions -- Usage examples -- Configuration examples - -**Keep it user-focused:** -```markdown -## Features -- ✅ Trading Protection System (NEW) - * Automatic drawdown protection - * Cooldown periods after losses - * Stoploss cluster detection -``` - -#### 3. 📦 Update requirements.txt - -**When:** Adding new dependencies or removing unused ones. - -**What to update:** -- `requirements.txt` (main dependencies) -- `requirements/lint.txt` (dev dependencies) -- `requirements/test.txt` (test dependencies) - -**Example:** -```bash -# If you add a new library -echo "new-library==1.0.0" >> requirements.txt - -# If you add a new test dependency -echo "pytest-mock" >> requirements/test.txt -``` - -#### 4. ✅ Extend Tests - -**When:** EVERY time you add new code. - -**Rule:** New features MUST have tests with >80% coverage. - -**What to create:** -- Unit tests in `test/` directory -- Integration tests in `test/integration/` -- Update existing tests if behavior changed - -**Test structure:** -```python -# test/feature_type/test_new_feature.py -"""Tests for New Feature""" - -class TestNewFeature: - """Test new feature thoroughly.""" - - def test_basic_functionality(self): ... - def test_edge_cases(self): ... - def test_error_handling(self): ... - def test_integration_with_existing(self): ... -``` - -**Update integration tests:** -```python -# Add to test/integration/test_all_features.py -class TestNewFeature: - """Test new feature integration.""" - - def test_imports(self): ... - def test_initialization(self): ... - def test_full_workflow(self): ... -``` - -#### 5. 🔄 Pre-Commit Checklist - -**BEFORE every commit with new features:** - -```bash -# 1. Run ALL tests -pytest test/ -v - -# 2. Run integration tests -pytest test/integration/test_all_features.py -v - -# 3. Check test coverage -pytest --cov=rdagent.components.new_module -v - -# 4. Run security scan -bandit -r rdagent/ -c .bandit.yml - -# 5. Verify tests updated -git status -# Should show test files modified -``` - -### Documentation Priority Order - -1. **QWEN.md** - Internal AI assistant context (UPDATE ALWAYS) -2. **Test files** - Code documentation through tests (MANDATORY) -3. **README.md** - User-facing documentation (UPDATE for user-visible changes) -4. **requirements.txt** - Dependencies (UPDATE when adding libraries) -5. **Inline code comments** - English only (ALWAYS) - -### Example Workflow: Adding New Feature - -``` -1. Plan feature - ↓ -2. Implement code - ↓ -3. Write unit tests (test/...) - ↓ -4. Write integration tests (test/integration/...) - ↓ -5. Run ALL tests → Must pass - ↓ -6. Update QWEN.md ← MANDATORY - ↓ -7. Update README.md (if user-visible) - ↓ -8. Update requirements.txt (if new deps) - ↓ -9. Commit with clear message - ↓ -10. Pre-commit hooks run automatically - ↓ -11. Push to remote -``` - -### Penalties for Not Following Rules - -**If you forget to update:** -- ❌ Missing tests → Code cannot be committed (pre-commit blocks) -- ❌ Missing QWEN.md update → Next AI assistant will work with outdated context -- ❌ Missing README update → Users won't understand new features -- ❌ Missing requirements.txt → Installation will fail - -**Remember:** These rules ensure: -1. Code quality through tests -2. AI assistant has current context -3. Users understand changes -4. Dependencies are tracked - ---- - ---- - -## 🚀 COMPLETE 5-PHASE ARCHITECTURE - -### Phase 1: Factor Generation (Open Source - ALWAYS ACTIVE) - -``` -1. Hypothesis Generation (LLM v3 Prompt) - → MultiIndex code examples (unstack/stack pattern) - → Working code templates - → Volume warning (FX volume = 0 often) - -2. CoSTEER Code Validation - → Execute factor code - → Validate result.h5 output - → Retry with feedback (max 3 retries) - -3. Qlib Docker Backtest - → LightGBM training on factor - → Portfolio backtest (TopkDropoutStrategy) - → IC, Sharpe, Max DD, Win Rate calculation - -4. Results Storage - → results/factors/{name}.json (Code + Description + Metrics) - → results/db/backtest_results.db (SQLite) - → results/logs/ (Running logs) - -⚡ CONTINUE UNTIL 5000+ VALID FACTORS REACHED -``` - -### Phase 2: ML Model Training (Closed Source - Local Only) - -``` -5. Load Top 50 Factors (by IC ≥ 0.01) - → From results/factors/ with valid IC - → Extract factor values from workspaces - -6. Build Feature Matrix - → X = factor values (samples × factors) - → y = forward returns (96-bar shift) - -7. Train LightGBM Model - → Split: 80% train, 20% validate - → Early stopping (50 rounds) - → Feature importance analysis - -8. Model Validation - → IC (train vs valid) - → Sharpe-like metric - → Overfitting detection - -9. Save Model - → results/models/{name}/model.txt - → results/models/{name}/metadata.json -``` - -### Phase 3: Portfolio Optimization (Closed Source - Local Only) - -``` -10. Load Top 30 Factors - → Compute correlation matrix - → Select uncorrelated factors (max corr = 0.3) - -11. Optimize Weights - → Weight by absolute IC - → Normalize to sum = 1.0 - -12. Backtest Portfolio - → Combined factor score = Σ(weight_i × factor_i) - → Calculate IC, Sharpe, Max DD, Win Rate - -13. Save Portfolio - → results/portfolios/{name}.json -``` - -### Phase 4: Strategy Generation (Closed Source - Local Only) - -``` -14. Generate Trading Rules - → Entry signals (factor thresholds) - → Exit signals (take profit, stop loss) - → Position sizing (Kelly criterion) - -15. Add Risk Management - → Max drawdown protection - → Cooldown periods after losses - → Stoploss cluster detection - -16. Save Strategy - → results/strategies/{name}.json -``` - -### Phase 5: Iterative Improvement (Closed Source - Local Only) - -``` -17. ML Feedback Loop - → Use model performance to guide factor generation - → Identify feature importance patterns - → Generate factors targeting weak areas - -18. Portfolio Feedback - → Use portfolio performance to refine weights - → Add new uncorrelated factors - → Remove degraded factors - -19. Loop Back to Phase 1 - → Generate NEW factors with ML insights - → Retrain model with expanded factor set - → Continuous improvement cycle -``` - -### Phase 6: ML Feedback Integrator (P6 - Closed Source) - -``` -20. Hook into QuantRDLoop.feedback() - → Check factor count every feedback call - → Trigger ML training every 500 factors - → Trigger strategy generation every 1000 factors - → Trigger portfolio optimization every 2000 factors - -21. Feature Importance → Prompt Feedback - → Read results/models/feature_importance.json - → Generate prompt suggestions for next factor generation - → Save to prompts/local/ml_feedback.yaml - -22. MLFeedbackMixin Implementation - → Mixin class for QuantRDLoop - → Patches via multiple inheritance - → Graceful degradation if modules missing - → All errors caught - never breaks main loop - -File: rdagent/scenarios/qlib/local/feedback_integrator.py -Tests: test/local/test_feedback_integrator.py (18 tests) -``` - -### Phase 7: Portfolio Optimizer (P7 - Closed Source) - -``` -23. Correlation Analysis - → Load top 30 strategies by Sharpe - → Calculate return correlation matrix - → Select uncorrelated subset (max corr 0.3) - -24. Mean-Variance Optimization (max Sharpe) - → Expected returns from strategy backtests - → Covariance matrix from strategy returns - → scipy.optimize.minimize (SLSQP) - → Long-only, weights sum to 1 - -25. Risk Parity (equal risk contribution) - → Each strategy contributes equally to portfolio risk - → Alternative to mean-variance - -26. IC-Weighted Portfolio - → Weight proportional to |IC| - → Simple, robust fallback - -27. Portfolio Backtest - → Weighted combination of strategy signals - → Calculate portfolio IC, Sharpe, Max DD, Win Rate - -28. Results Persistence - → Save to results/portfolios/{timestamp}_portfolio_{method}.json - → Full metrics for each optimization - -File: rdagent/scenarios/qlib/local/portfolio_optimizer.py -Tests: test/local/test_portfolio_optimizer.py (28 tests) -``` - -### Phase 8: Integration Tests (P8) - -``` -29. End-to-End Pipeline Tests - → Data Loading → Strategy Gen → Backtest → Accept - → Mock LLM calls for speed - → Verify all intermediate outputs - -30. Parallelization Tests - → 4 Workers, parallel factor evaluation - → No race conditions - → All results collected correctly - -31. FTMO Compliance Tests - → SL ≤ 2%, DD ≤ 10%, Daily Loss ≤ 5% - → All accepted strategies pass compliance - -32. Error Handling Tests - → Missing imports, empty data, edge cases - → Graceful degradation verified - -File: test/integration/test_full_pipeline.py (27 tests) -``` - -### Phase 9: Documentation (P9) - -``` -33. QWEN.md Updated - → Architecture section: new local modules listed - → Project Status: P6-P8 marked complete - → Next Steps: P6-P9 checkpoints added - → Module descriptions added - -34. Test Documentation - → 18 unit tests for feedback_integrator - → 28 unit tests for portfolio_optimizer - → 27 integration tests for full pipeline - → Total: 73 new tests, all passing -``` - ---- - -## 📊 CURRENT RESULTS (as of April 2026) - -### Factor Evaluation (1009 factors, FULL DATA 2020-2026) - -| Metric | Value | -|--------|-------| -| Total evaluated | 1,009 | -| Successful | 337 (33%) | -| Failed | 672 (67%) | -| Best IC | **0.255** (daily_close_open_mom) | -| Avg IC (valid) | 0.011 | -| Best Sharpe | 1.71 (DCP) | - -### Top 10 Factors by IC - -| # | Factor | IC | Sharpe | -|---|--------|-----|--------| -| 1 | daily_close_open_mom | **0.255** | 0.007 | -| 2 | daily_ret_log_1d | 0.255 | 0.003 | -| 3 | daily_ret_close_1d | 0.255 | 0.005 | -| 4 | daily_close_to_close_return | 0.255 | 0.005 | -| 5 | daily_ret_vol_adj_1d | 0.235 | -0.007 | -| 6 | daily_ols_slope_96 | 0.227 | 0.002 | -| 7 | DCP | 0.199 | **1.71** | -| 8 | DailyTrendStrength_Raw | 0.143 | -0.016 | -| 9 | daily_c2c_return | 0.129 | 0.001 | -| 10 | daily_momentum | 0.129 | -0.001 | - -### Failure Analysis (672 failed) - -| Error Type | Count | % | Cause | -|------------|-------|-----|-------| -| Code crashed | 540 | 80.4% | MultiIndex errors (FIXED in v3 prompt) | -| All NaN values | 97 | 14.4% | Volume=0, rolling window too large | -| Other errors | 28 | 4.2% | Various | -| Timeout (120s) | 5 | 0.7% | Computationally expensive | -| Too little overlap | 2 | 0.3% | Data mismatch | - ---- - -## 💡 OPTIMIZATION POTENTIAL (HIGH-END UPGRADES) - -### 1. Code Quality Improvements -- **Current**: 33% success rate -- **Target**: 70%+ with v3 prompt (MultiIndex examples) -- **Expected**: ~700 valid factors from 1009 generated - -### 2. ML Pipeline Enhancements -- **Feature Selection**: Use SHAP values for importance -- **Ensemble Models**: Combine LightGBM + XGBoost + Neural Net -- **Cross-Validation**: Time-series split to prevent overfitting -- **Hyperparameter Optimization**: Optuna for automatic tuning - -### 3. Portfolio Optimization -- **Risk Parity**: Equal risk contribution instead of IC-weighted -- **Black-Litterman**: Incorporate LLM views as priors -- **Regime Detection**: Switch portfolios based on market state -- **Dynamic Rebalancing**: Adjust weights based on rolling IC - -### 4. Strategy Generation -- **Regime-Specific Rules**: Different signals for trending vs mean-reverting -- **Multi-Timeframe**: Combine 1min, 5min, 15min signals -- **Adaptive Thresholds**: Dynamic entry/exit based on volatility -- **News Integration**: Avoid trading during high-impact news - -### 5. Execution Optimization -- **Parallel Factor Generation**: 8+ workers instead of 4 -- **Smart Retry Logic**: Learn from failures, adjust prompts -- **Early Stopping**: Skip factors that show promise in first 1000 bars -- **Incremental Evaluation**: Evaluate factors as they're generated - -### 6. Risk Management -- **VaR/ES**: Value at Risk and Expected Shortfall calculations -- **Correlation Monitoring**: Track factor correlation drift -- **Performance Attribution**: Understand which factors drive returns -- **Stress Testing**: Test strategies on historical crises - -### 7. Infrastructure -- **GPU Acceleration**: Use RTX 5060 Ti for LightGBM training -- **Database Optimization**: Index queries for faster factor selection -- **Caching Layer**: Cache expensive computations -- **Monitoring Dashboard**: Real-time performance tracking - ---- - ---- - -## ⚠️ CRITICAL: NEVER COMMIT TRADING STRATEGIES OR JSON FILES - -### 🚫 NEVER COMMIT THESE: - -``` -❌ *.json (strategy files in root directory) -❌ results/strategies_new/*.json -❌ strategies/*.json -❌ Any file containing trading strategy output -❌ Backtest results JSON files -❌ Generated strategy code files -``` - -### ✅ WHY? - -1. **Git Repository is for CODE ONLY** - Not for generated data -2. **Strategy files are auto-generated** - They belong in `results/` (gitignored) -3. **JSON files pollute the repository** - 204+ files were accidentally committed -4. **History cleanup was painful** - Required BFG Repo-Cleaner to fix - -### ✅ WHAT TO COMMIT INSTEAD: - -``` -✅ Source code (*.py, *.yaml, *.sh) -✅ Configuration files (.env.example, data_config.yaml) -✅ Documentation (QWEN.md, README.md) -✅ Tests (test/*.py) -✅ Prompts (prompts/*.yaml) -✅ Models (models/standard/*.py) -``` - -### ✅ WHERE DO STRATEGIES BELONG? - -``` -results/strategies_new/ ← Auto-generated strategies (gitignored) -results/strategies/ ← StrategyCoSTEER output (gitignored) -results/factors/ ← Factor data (gitignored) -results/backtests/ ← Backtest results (gitignored) -``` - -### ✅ HOW TO PREVENT ACCIDENTAL COMMITS: - -```bash -# 1. .gitignore is configured: -*.json # Excludes ALL JSON files -!package.json # Except package files -!package-lock.json - -# 2. Always check before commit: -git status # Verify no JSON files staged -git diff --staged # Review what will be committed - -# 3. If JSON files appear: -git restore --staged *.json # Unstage them -git checkout -- *.json # Discard changes -``` - -### 📜 LESSON LEARNED: - -On April 9, 2026: -- 204+ JSON strategy files were accidentally committed to root -- Required BFG Repo-Cleaner to remove from history -- 13,221 git objects had to be cleaned -- History rewrite required force push - -**NEVER let this happen again.** - ---- - ---- - -## 🔒 CRITICAL: NEVER COMMIT CLOSED-SOURCE / PRIVATE ASSETS - -### 🚫 NEVER COMMIT THESE (CLOSED SOURCE): - -``` -❌ git_ignore_folder/ ← Trading scripts, OHLCV data, credentials -❌ rdagent/scenarios/qlib/local/ ← Advanced closed-source components -❌ models/local/ ← Your improved models (Transformer, TCN, etc.) -❌ prompts/local/ ← Your improved prompts (v2.0 optimized) -❌ .env ← API keys, credentials -❌ results/ ← Backtest results, strategies, logs -❌ *.db ← SQLite databases -❇️ *.log ← Log files -``` - -### ✅ WHY? - -1. **Alpha Protection** - Your best models/prompts are your competitive edge -2. **Security** - API keys and credentials must stay private -3. **Repository Size** - OHLCV data and results are massive -4. **Open Source Framework** - Users get the framework, not your alpha - -### 📂 WHAT IS OPEN SOURCE vs CLOSED SOURCE: - -#### 🟢 OPEN SOURCE (Public on GitHub - SAFE TO COMMIT): -``` -rdagent/ ← Core framework -models/standard/ ← Base models (XGBoost, LightGBM) -prompts/standard_prompts.yaml ← Base prompts -web/ ← Dashboards -test/ ← ALL tests -requirements.txt ← Dependencies -data_config.yaml ← Configuration -scripts/ ← Utility scripts -``` - -#### 🔒 CLOSED SOURCE (Local Only - NEVER COMMIT): -``` -git_ignore_folder/ -├── predix_live_trader.py ← Live trading scripts -├── LIVE_TRADING_SETUP.md ← Setup guides -└── factor_implementation_source_data/ ← OHLCV data - -rdagent/scenarios/qlib/local/ -├── strategy_coster.py ← StrategyCoSTEER -├── strategy_evaluator.py ← Strategy metrics -├── strategy_runner.py ← Strategy execution -├── ml_trainer.py ← ML training -├── feedback_integrator.py ← ML feedback loop -├── portfolio_optimizer.py ← Portfolio optimization -└── strategy_discovery_v1.yaml ← LLM prompts - -models/local/ -├── transformer_factor.py -├── tcn_factor.py -├── patchtst_factor.py -└── cnn_lstm_hybrid.py - -prompts/local/ -└── factor_discovery_v2.yaml ← Your improved prompts - -.env ← API keys -results/ ← All generated data -``` - -### ✅ HOW .gitignore PROTECTS YOU: - -```bash -# Already in .gitignore: -.gitignore contains: - git_ignore_folder/ ← Trading data - .env ← Credentials - results/ ← Generated data - *.db ← Databases - *.log ← Log files - models/local/ ← Your improved models - prompts/local/ ← Your improved prompts - rdagent/scenarios/qlib/local/ ← Advanced components -``` - -### ✅ HOW TO VERIFY BEFORE COMMIT: - -```bash -# ALWAYS run these before commit: -git status # Should NOT show closed-source files -git diff --staged # Review EVERY file - -# If you see any of these, STOP: -git_ignore_folder/ -.env -results/ -models/local/ -prompts/local/ -rdagent/scenarios/qlib/local/ -``` - -### ✅ BACKUP YOUR PRIVATE ASSETS SAFELY: - -```bash -# Create a PRIVATE repo on GitHub: -git clone git@github.com:YOUR-USERNAME/predix-private-assets.git - -# Copy closed-source assets: -cp -r ~/Predix/models/local/* ~/predix-private-assets/models/ -cp -r ~/Predix/prompts/local/* ~/predix-private-assets/prompts/ -cp -r ~/Predix/git_ignore_folder/* ~/predix-private-assets/data/ - -# Commit to PRIVATE repo: -cd ~/predix-private-assets -git add . -git commit -m "Backup: models v2, prompts v2, trading scripts" -git push -``` - -### 📜 LESSON LEARNED: - -Your **competitive edge** (alpha) is in: -- `models/local/` - Your improved model architectures -- `prompts/local/` - Your optimized prompt templates -- `git_ignore_folder/` - Your trading scripts and data - -**These MUST stay private.** The open-source framework is the skeleton - your closed-source assets are the brain. - ---- diff --git a/TODO.md b/TODO.md deleted file mode 100644 index fe21a556..00000000 --- a/TODO.md +++ /dev/null @@ -1,150 +0,0 @@ -# Predix — Implementierungs-Tracker - -## Übersicht -- **Start:** April 2026 -- **Ziel:** Vollständig integriertes Quant Trading System in `fin_quant` Loop -- **Status:** ✅ ALLE PHASEN ABGESCHLOSSEN (P0-P9) - ---- - -## Phasen - -### P0: Data Loader (2h) ✅ ABGESCHLOSSEN -- [x] `rdagent/scenarios/qlib/local/data_loader.py` erstellt - - [x] OHLCV Loading + Caching (HDF5 → pd.Series) - - [x] Faktor Loading + Metadaten (JSON + Parquet) - - [x] Feature Matrix Builder (alignment mit close) - - [x] Thread-sicheres Caching -- [x] Tests: `test/local/test_data_loader.py` (11 passed) -- [x] Abhängigkeiten: Keine - -### P1: Strategy Worker (4h) ✅ ABGESCHLOSSEN -- [x] `rdagent/scenarios/qlib/local/strategy_worker.py` erstellt - - [x] LLM Call Wrapper (llama.cpp :8081) - - [x] Backtest Engine (subprocess mit OHLCV) - - [x] Acceptance Gate (FTMO-konform) - - [x] FTMO Compliance Check -- [x] Tests: `test/local/test_strategy_worker.py` (41 passed) -- [x] Abhängigkeiten: P0 (data_loader.py) - -### P2: Strategy Orchestrator (6h) ✅ ABGESCHLOSSEN -- [x] `rdagent/scenarios/qlib/local/strategy_orchestrator.py` erstellt - - [x] Multi-Process Pool (4-8 Workers) - - [x] Task Queue (random factor selection) - - [x] LLM Semaphore (max 2 parallel für llama.cpp) - - [x] Result Collection + Deduplizierung - - [x] Strategy Saver (JSON + Reports) -- [x] CLI Command: `rdagent generate_strategies` -- [x] Tests: `test/local/test_strategy_orchestrator.py` (30 passed) -- [x] Abhängigkeiten: P1 - -### P3: Optuna Optimizer (4h) ✅ ABGESCHLOSSEN -- [x] `rdagent/scenarios/qlib/local/optuna_optimizer.py` erstellt - - [x] Parameter Space Definition (FTMO-konform) - - [x] Objective Function (Sharpe × |IC| × √trades) - - [x] FTMO Penalty Logic - - [x] TPE Sampler + MedianPruner - - [x] 20-50 Trials pro Strategie -- [x] Integration in Strategy Orchestrator -- [x] Tests: `test/local/test_optuna_optimizer.py` (60 passed) -- [x] Abhängigkeiten: P1, `pip install optuna` - -### P4: CLI Commands (2h) ✅ ABGESCHLOSSEN -- [x] `rdagent/app/cli.py` erweitert - - [x] `generate_strategies` Command - - [x] CLI Parameter (count, workers, style, optuna) - - [x] Rich Console Output - - [ ] Integration in `fin_quant` Loop -- [x] Tests: `test/integration/test_cli_commands.py` (21 tests) -- [x] Abhängigkeiten: P2, P3 - -### P5: ML Training Pipeline (6h) ✅ ABGESCHLOSSEN -- [x] `rdagent/scenarios/qlib/local/ml_trainer.py` erstellt - - [x] Feature Matrix Builder (alle Top-N Faktoren) - - [x] Time-Series Train/Val Split - - [x] LightGBM Training (early stopping) - - [x] Feature Importance Analysis - - [x] Model Save/Load -- [x] CLI Command: `rdagent train_models` -- [x] Tests: `test/local/test_ml_trainer.py` (46 passed) -- [x] Abhängigkeiten: P0, `pip install lightgbm` - -### P6: Feedback an fin_quant Loop (3h) ✅ ABGESCHLOSSEN -- [ ] Hook in `QuantRDLoop.feedback()` einbauen - - [ ] `_trigger_ml_training()` alle 500 Faktoren - - [ ] `_trigger_strategy_generation()` alle 1000 Faktoren - - [ ] ML Feature Importance → Prompt Feedback -- [ ] Prompt-Loader erweitern (local/ml_feedback.yaml) -- [ ] Tests: `test/local/test_feedback_integration.py` -- [ ] Abhängigkeiten: P5 - -### P7: Portfolio Optimizer (6h) ✅ ABGESCHLOSSEN -- [ ] `rdagent/scenarios/qlib/local/portfolio_optimizer.py` erstellen - - [ ] Korrelationsmatrix (max 0.3) - - [ ] Mean-Variance Optimization - - [ ] Risk Parity (gleicher Risiko-Beitrag) - - [ ] Black-Litterman (LLM Views als Priors) - - [ ] Portfolio-Backtest -- [ ] CLI Command: `rdagent optimize_portfolio` -- [ ] Tests: `test/local/test_portfolio_optimizer.py` -- [ ] Abhängigkeiten: P5 - -### P8: Integrationstests (4h) ✅ ABGESCHLOSSEN -- [ ] End-to-End Pipeline Test - - [ ] Data Loading → Strategy Gen → Backtest → Accept -- [ ] Parallelisierung Test - - [ ] 4 Workers, 2 LLM parallel, keine Race Conditions -- [ ] Optuna Test - - [ ] 20 Trials, Konvergenz prüfen -- [ ] FTMO Compliance Test - - [ ] SL ≤ 2%, DD ≤ 10%, Daily Loss ≤ 5% -- [ ] Tests: `test/integration/test_full_pipeline.py` -- [ ] Abhängigkeiten: P0-P7 - -### P9: Dokumentation (3h) ✅ ABGESCHLOSSEN -- [ ] README.md aktualisieren - - [ ] Neue Commands dokumentieren - - [ ] Architektur-Diagramm - - [ ] Setup-Anleitung -- [ ] QWEN.md aktualisieren - - [ ] Neue Module in Architecture Section - - [ ] Data Flow Diagram - - [ ] Project Status -- [ ] Abhängigkeiten: P0-P8 - ---- - -## Dependencies Checklist - -- [ ] `pip install optuna` (P3) -- [ ] `pip install lightgbm` (P5) -- [ ] `pip install xgboost` (P5, optional) -- [ ] llama.cpp Server auf :8081 mit `--parallel 2` (P1-P3) - -## FTMO Compliance Rules - -| Regel | Limit | Prüfung | -|-------|-------|---------| -| Max Stop Loss | 2% | Hard-coded in Optuna Space | -| Max Drawdown | 10% | Acceptance Gate | -| Max Daily Loss | 5% | Risk Management Layer | -| Risk/Reward | ≥ 2:1 | TP ≥ 2× SL enforced | -| Max Positions | 1 | Strategy Code enforced | - -## Risiko-Metriken (Ziele) - -| Metrik | Target | Minimum | -|--------|--------|---------| -| Strategie IC | > 0.03 | > 0.02 | -| Strategie Sharpe | > 1.5 | > 0.5 | -| Max Drawdown | < 10% | < 15% | -| Win Rate | > 50% | > 45% | -| Monthly Return | 1-3% | > 0.5% | -| Trades/Monat | > 20 | > 10 | - -## Notizen - -- **Alles in fin_quant Loop**: Keine separaten Skripte mehr (`predix_gen_strategies_real_bt.py`, `predix_smart_strategy_gen.py` deprecated) -- **Graceful Degradation**: Wenn llama.cpp nicht läuft, skippt Strategy Generation mit klarer Warnung -- **LLM Parallelisierung**: Max 2 parallele Calls (llama.cpp `--parallel 2`) -- **Optuna Trials**: Standard 30, kann via CLI überschrieben werden diff --git a/CHANGELOG.md b/docs/CHANGELOG.md similarity index 100% rename from CHANGELOG.md rename to docs/CHANGELOG.md diff --git a/IMPLEMENTATION_SUMMARY.md b/docs/IMPLEMENTATION_SUMMARY.md similarity index 100% rename from IMPLEMENTATION_SUMMARY.md rename to docs/IMPLEMENTATION_SUMMARY.md diff --git a/STRATEGY_BUILDER_DESIGN.md b/docs/STRATEGY_BUILDER_DESIGN.md similarity index 100% rename from STRATEGY_BUILDER_DESIGN.md rename to docs/STRATEGY_BUILDER_DESIGN.md