mirror of https://github.com/NicolasBohn/NexQuant.git synced 2026-07-27 23:47:46 +00:00

Files

T

TPTBusiness 24a58e970e docs: Add live trading system documentation to QWEN.md

Complete live trading guide for cTrader + RiskMgmt integration:
- Architecture diagram and how it works (5 steps)
- Setup instructions and API configuration
- Usage examples (paper/live trading)
- Risk management and monitoring
- Troubleshooting guide
- Future enhancements roadmap

Documentation kept in QWEN.md only (internal, not public README).

2026-04-07 12:41:07 +02:00

44 KiB

Raw Blame History

Predix - QWEN.md Context File

Project Overview

Predix is an autonomous AI-powered quantitative trading agent for EUR/USD forex markets. Built on the RD-Agent framework, it automates the full research and development cycle for trading strategies.

Core Purpose

Generate trading factors (signals) autonomously using LLMs
Backtest and validate factors on 1-minute EUR/USD data
Generate AI strategies with LLM + REAL OHLCV backtest (96-bar forward returns)
Optimize portfolios using modern portfolio theory
Target: 1-3% monthly returns with Sharpe > 2.0

Key Technologies

Python 3.10/3.11 - Primary language
PyTorch - Deep learning models
Qlib - Backtesting engine
LLM (Qwen3.5-35B via OpenRouter) - Factor/strategy generation
Flask - Web dashboard API
SQLite - Results database
Rich/Typer - CLI interface
Matplotlib/Seaborn - Performance report charts

Architecture

Predix/
├── rdagent/                    # Core agent framework
│   ├── app/
│   │   └── cli.py              # Main CLI entry point (rdagent command)
│   ├── components/
│   │   ├── backtesting/        # Backtest engine, metrics, database
│   │   ├── coder/
│   │   │   ├── factor_coder/   # Factor generation & EURUSD-specific modules
│   │   │   └── rl/             # RL Trading Agent
│   │   ├── loader.py           # Prompt loader (auto-loads local prompts)
│   │   └── model_loader.py     # Model loader (auto-loads local models)
│   └── scenarios/
│       └── qlib/               # Qlib integration for FX trading
├── predix.py                   # Main CLI wrapper (predix.py commands)
├── predix_parallel.py          # Parallel factor evolution
├── predix_gen_strategies_real_bt.py  # AI Strategy Gen + REAL OHLCV Backtest
├── predix_strategy_report.py   # Performance report generator (charts + PDF)
├── debug_backtest.py           # Debug backtest alignment & IC
├── prompts/                    # LLM Prompts
│   ├── standard_prompts.yaml   # Standard prompts (in Git)
│   └── local/                  # Your improved prompts (NOT in Git!)
├── models/                     # ML Models
│   ├── standard/               # Standard models (in Git)
│   └── local/                  # Your improved models (NOT in Git!)
├── results/                    # Backtest results (NOT in git)
│   ├── factors/                # ~872 evaluated factors
│   │   └── values/             # Factor time-series parquet files (862)
│   ├── strategies_new/         # AI-generated strategies with real backtests
│   └── strategy_reports/       # Performance reports with charts
├── git_ignore_folder/          # OHLCV data (intraday_pv.h5)
└── .env                        # Environment config (API keys)

CLI Commands Reference

Trading Loop

rdagent fin_quant                        # Start factor evolution
rdagent fin_quant --loop-n 5             # 5 evolution loops
rdagent fin_quant --with-dashboard       # With web dashboard
rdagent fin_quant --cli-dashboard        # With CLI Rich dashboard

Parallel Execution

python predix_parallel.py --runs 5 --api-keys 1 -m openrouter   # 5 parallel runs
python predix_parallel.py --runs 20 --api-keys 2 -m openrouter  # 20 runs, 2 keys

AI Strategy Generation (REAL OHLCV Backtest)

python predix_gen_strategies_real_bt.py          # Generate 10 strategies
python predix_gen_strategies_real_bt.py 20       # Generate 20 strategies
python predix_gen_strategies_real_bt.py 5        # Generate 5 (faster test)

Each accepted strategy gets:

JSON file in results/strategies_new/
Performance report with charts in results/strategy_reports/
Dashboard PNG (equity curve, drawdown, signals, monthly returns)
Text report with full metrics

Strategy Reports

python predix_strategy_report.py                 # Reports for ALL strategies
python predix_strategy_report.py <path.json>     # Report for single strategy

Factor Evaluation

python predix.py evaluate --all                  # Evaluate all factors
python predix.py top -n 20                       # Top 20 factors by IC
python predix.py portfolio-simple                # Portfolio optimization

Debug

python debug_backtest.py                         # Debug alignment & IC

🚀 Live Trading System (cTrader + FTMO)

Overview

Predix includes a complete live trading system that executes strategies on cTrader via Open API with FTMO broker.

All live trading code is CLOSED SOURCE and stored in git_ignore_folder/ (never committed to Git).

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    PREDIX LIVE TRADING                       │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Strategy JSON  →  Factor Calculator  →  Signal Generator   │
│       ↓                    ↓                       ↓        │
│  results/strategies   Live OHLCV Data        LONG/SHORT     │
│  _new/*.json          (cTrader API)         /NEUTRAL        │
│                                                    ↓        │
│                                              Risk Manager   │
│                                                    ↓        │
│                                        cTrader Orders API   │
│                                                    ↓        │
│                                        FTMO Account (Live)  │
│                                                              │
│  Logging: results/live_trading/                              │
│    - trades_*.json  (trade log)                              │
│    - trading_*.log  (detailed log)                           │
└──────────────────────────────────────────────────────────────┘

Files (Closed Source)

git_ignore_folder/
├── predix_live_trader.py          ← Main live trading script
└── LIVE_TRADING_SETUP.md          ← Setup guide

results/live_trading/
├── trades_*.json                  ← Trade log
└── trading_*.log                  ← Detailed log

Prerequisites

cTrader Account with FTMO broker
cTrader Open API credentials: https://developers.ctrader.com/
- Client ID
- Client Secret
- Broker ID
- Access Token
Python 3.10+ with requests, pandas, numpy, python-dotenv

Setup cTrader API

Register Application:
- Go to https://developers.ctrader.com/
- Login with your cTrader credentials
- Create new application
- Note down: Client ID, Client Secret, Broker ID
Generate Access Token:
- OAuth2 flow or generate in dashboard
- Token expires - refresh as needed
Configure .env:

# Add to .env file:
CTRADE_API_BASE=https://api.ctrader.com
CTRADE_CLIENT_ID=your_client_id
CTRADE_CLIENT_SECRET=your_client_secret
CTRADE_ACCESS_TOKEN=your_access_token
CTRADE_BROKER_ID=your_broker_id

# Trading parameters
TRADING_SYMBOL=EURUSD
TRADING_TIMEFRAME=M1
DEFAULT_LOT_SIZE=0.01
MAX_DAILY_LOSS_PCT=2.0
MAX_POSITIONS=1

How It Works

1. Strategy Loading

# Loads strategy from JSON
strategy = json.load(open('results/strategies_new/123_MomentumDivergenceZScore.json'))
code = strategy['code']          # Strategy Python code
factors = strategy['factor_names']  # Factor names list

2. Factor Calculation

# Computes factors from live OHLCV
ohlcv = client.get_ohlcv('EURUSD', 'M1', count=1000)
factors_df = compute_factors(ohlcv)
# Calculates: daily_close_return_96, daily_session_momentum_divergence_1d, etc.

3. Signal Generation

# Executes strategy code
exec(strategy_code, {'factors': factors_df}, local_vars)
signal = local_vars['signal']  # 1=LONG, -1=SHORT, 0=NEUTRAL

4. Order Execution

if signal != last_signal and signal != 0:
    # Close opposite positions
    if signal == 1: close_all_shorts()
    if signal == -1: close_all_longs()
    
    # Place new order
    client.place_order(
        symbol='EURUSD',
        side='LONG' if signal == 1 else 'SHORT',
        lot_size=calculate_position_size(),
        stop_loss=0.0050,    # 50 pips
        take_profit=0.0100,  # 100 pips
        comment='Predix-{strategy_name}'
    )

5. Risk Management

Daily Loss Limit: Stops trading if daily loss > 2%
Max Positions: Only 1 position at a time
Position Sizing: Dynamic based on balance and ATR
Stop Loss: 50 pips automatic
Take Profit: 100 pips automatic

Usage

Paper Trading (TEST FIRST!)

python git_ignore_folder/predix_live_trader.py \
  --strategy results/strategies_new/1775543215_MomentumDivergenceZScore.json \
  --paper

Live Trading (REAL MONEY)

python git_ignore_folder/predix_live_trader.py \
  --strategy results/strategies_new/1775543215_MomentumDivergenceZScore.json \
  --lot-size 0.01

Custom Parameters

python git_ignore_folder/predix_live_trader.py \
  --strategy results/strategies_new/123_MyStrategy.json \
  --lot-size 0.02 \
  --symbol EURUSD \
  --timeframe M5

CLI Options

Option	Short	Description	Default
`--strategy`	`-s`	Path to strategy JSON	Required
`--paper`	`-p`	Paper trading mode	False
`--lot-size`	`-l`	Fixed lot size	0.01
`--symbol`		Trading symbol	EURUSD
`--timeframe`		Timeframe	M1

Monitoring

Log Files

# View trade log
cat results/live_trading/trades_*.json | jq .

# View detailed log
tail -f results/live_trading/trading_*.log

Trade Log Format

[
  {
    "timestamp": "2026-04-07T12:05:30",
    "signal": 1,
    "side": "LONG",
    "lot_size": 0.01,
    "result": { "orderId": "12345", "price": 1.08500 }
  }
]

⚠️ Critical Warnings

ALWAYS test in paper mode first - Never go live without testing
Start small - Use 0.01 lots initially
Monitor daily - Check logs every day
FTMO rules - Respect max drawdown limits (usually 10%)
Token expiry - Refresh API tokens before they expire
Internet required - System stops if connection drops
No guarantees - Past performance ≠ future results

Troubleshooting

Error	Cause	Solution
"Connection failed"	Wrong API credentials	Check .env values
"No OHLCV data"	cTrader not running	Start cTrader platform
"Signal error"	Missing factors	Strategy needs factors not in live data
"Order failed"	Insufficient margin	Check FTMO account balance
"Daily loss limit"	Hit 2% daily loss	System stopped - wait for next day

cTrader API Endpoints

The system uses these cTrader Open API endpoints:

GET  /api/accounts          # Get account info
GET  /api/positions         # Get open positions
GET  /api/cbars             # Get OHLCV data
POST /api/orders            # Place order
DELETE /api/positions/{id}  # Close position

Future Enhancements

Multi-strategy portfolio trading
Dynamic stop loss/take profit
Trailing stop loss
Webhook alerts for trades
Telegram notifications
Auto-restart on disconnect
Backtest with live data sync

│ └── local/ # Your improved models (NOT in Git!) │ ├── transformer_factor.py │ ├── tcn_factor.py │ ├── patchtst_factor.py │ └── cnn_lstm_hybrid.py ├── results/ # Backtest results (NOT in git) │ ├── backtests/ # Individual factor backtests (JSON/CSV) │ ├── db/ # SQLite database │ ├── factors/ # Factor analysis │ ├── runs/ # Run results & risk reports │ └── logs/ # Backtest logs ├── web/ # Dashboard frontend │ ├── dashboard_api.py # Flask API backend │ └── dashboard.html # Web UI ├── .env # Environment config (API keys, etc.) ├── data_config.yaml # EURUSD data configuration └── requirements.txt # Python dependencies


### Open Source vs. Closed Source

**🟢 OPEN SOURCE (Public on GitHub - FULLY WORKING):**
- `rdagent/` - Core framework (ALL components)
- `models/standard/` - Base models (XGBoost, LightGBM)
- `prompts/standard_prompts.yaml` - Base prompts
- `web/` - Dashboards
- `test/` - ALL tests (integration, unit, security)
- `rdagent/components/coder/rl/` - RL Trading System (with fallback)
- `rdagent/components/backtesting/protections/` - Trading Protection System
- `scripts/` - Utility scripts

**GitHub users get:**
✅ Full working trading system
✅ RL Trading with graceful fallback (no stable-baselines3 needed)
✅ Protection Manager (drawdown, cooldown, stoploss guard)
✅ Backtesting Engine with RL support
✅ CLI commands (`fin_quant`, `rl_trading`, etc.)
✅ Web and CLI dashboards
✅ All 200+ integration tests

**🔒 CLOSED SOURCE (Local Only - NOT on GitHub):**
- `models/local/` - Your improved models (Transformer, TCN, PatchTST, CNN+LSTM)
- `prompts/local/` - Your improved prompts (v2.0 optimized)
- `rdagent/scenarios/qlib/local/` - Advanced components:
  - `strategy_coster.py` - StrategyCoSTEER (LLM strategy generation)
  - `strategy_evaluator.py` - Comprehensive strategy metrics
  - `strategy_runner.py` - Strategy execution & backtesting
  - `strategy_discovery_v1.yaml` - LLM prompts for strategy generation
  - Plus: ml_trainer, portfolio_optimizer, quant_loop_advanced, etc.
- `.env` - API keys
- `results/` - Backtest results
- `git_ignore_folder/` - Trading data
- `QWEN.md`, `TODO.md` - Internal docs

**Protection:**
- `.gitignore` excludes all `local/` directories
- Your competitive edge (alpha) stays private
- Framework is open, but your best models/prompts are closed

### Open Source Fallback Strategy

**For users without stable-baselines3:**
The RL system provides graceful degradation:
- ❌ No stable-baselines3 → Uses simple momentum-based fallback
- ✅ Still fully functional: CLI, backtesting, protections work
- ✅ No errors or broken features
- ✅ Clear warning message with installation instructions

**For users without LLM (llama.cpp):**
- Factor evolution degrades gracefully
- System still works with standard models
- Clear error messages for missing LLM

**PRINCIPLE:** Every GitHub user MUST be able to run the full system. Missing optional components should never break the project.

## Building and Running

### Installation

```bash
# Clone repository
git clone https://github.com/PredixAI/predix
cd predix

# Create conda environment
conda create -n predix python=3.10
conda activate predix

# Install in editable mode
pip install -e .[test,lint]

Configuration

Create .env file:

# Local LLM (llama.cpp)
OPENAI_API_KEY=local
OPENAI_API_BASE=http://localhost:8081/v1
CHAT_MODEL=qwen3.5-35b

# Embedding (Ollama)
LITELLM_PROXY_API_KEY=local
LITELLM_PROXY_API_BASE=http://localhost:11434/v1
EMBEDDING_MODEL=nomic-embed-text

# Paths
QLIB_DATA_DIR=~/.qlib/qlib_data/eurusd_1min_data

Start LLM server (llama.cpp):

~/llama.cpp/build/bin/llama-server \
  --model ~/models/qwen3.5/Qwen3.5-35B-A3B-Q3_K_M.gguf \
  --n-gpu-layers 36 \
  --ctx-size 80000 \
  --port 8081

Running the Trading Loop

# Start trading loop (24/7)
./start_loop.sh

# Or single run
rdagent fin_quant

# With dashboard
rdagent fin_quant --with-dashboard

# With CLI dashboard
rdagent fin_quant --cli-dashboard

Running the Dashboard

# Web dashboard (runs with fin_quant --with-dashboard)
# Access at: http://localhost:5000/dashboard.html

# Or standalone
python web/dashboard_api.py

Testing

Integration Test Suite (ALL Features)

Comprehensive test system that validates ALL 13 implemented features:

# Run ALL integration tests (60 tests, ~7.5 seconds)
pytest test/integration/test_all_features.py -v

# Run with coverage report
pytest test/integration/test_all_features.py --cov=rdagent.components.backtesting -v

# Run via test runner script
./scripts/run_all_tests.sh

# Test specific features only
pytest test/integration/test_all_features.py -k "backtest or database" -v

# Skip slow tests
pytest test/integration/test_all_features.py -m "not slow" -v

Tested Features (60 Tests, ALL MUST PASS):

#	Feature	Tests	Status
1	Factor Evolution	5	✅ LLM generates trading factors autonomously
2	Model Evolution	5	✅ ML models auto-improved
3	Quant Loop (fin_quant)	4	✅ Main 24/7 trading loop
4	Backtesting Engine	5	✅ IC, Sharpe, Drawdown, Win Rate
5	Results Database	5	✅ SQLite with queries
6	Risk Management	6	✅ Correlation, Portfolio Optimization
7	CLI Dashboard	4	✅ Rich live-progress display
8	Web Dashboard	4	✅ Flask API + HTML
9	Health Check	4	✅ Environment validation
10	Streamlit UI	3	✅ Alternative dashboard
11	LLM Integration	5	✅ llama.cpp (Qwen3.5-35B)
12	Embedding	3	✅ Ollama (nomic-embed-text)
13	Security Scanning	5	✅ Bandit pre-commit hook

⚠️ MANDATORY: These tests run BEFORE every commit and MUST pass!

Unit Tests

# Run all unit tests
pytest test/

# Run with coverage
pytest --cov=rdagent --cov-report=html

# Test backtesting module
python rdagent/components/backtesting/backtest_engine.py
python rdagent/components/backtesting/results_db.py
python rdagent/components/backtesting/risk_management.py

Code Quality

# Linting
ruff check rdagent/

# Type checking
mypy rdagent/

# Format
black rdagent/

# Pre-commit (install first)
pre-commit install
pre-commit run --all-files

Development Conventions

Language Policy

ALL code comments and documentation MUST be in English.

❌ Wrong (German):

# Inspiriert von: TradingAgents
# Berechnet den Sharpe Ratio
# Achtung: Division durch Null möglich!
# Hinweis: Diese Funktion ist experimentell

✅ Correct (English):

# Inspired by: TradingAgents
# Calculates the Sharpe ratio
# Warning: Division by zero possible!
# Note: This function is experimental

Rationale:

International collaboration
Better searchability
Professional codebase
Consistent with commit messages (also English-only)

Enforcement:

All new code must have English comments
Existing German comments should be translated when modified
PRs with German comments will be rejected

Code Style

Line length: 120 characters (configured in pyproject.toml)
Type hints: Required for all public functions
Docstrings: Google style for public APIs
Imports: Sorted automatically with isort

Testing Practices

Unit tests in test/ directory
Test files named test_*.py
Use pytest fixtures for common setup
Mock external APIs (LLM, yfinance)
Minimum 80% coverage target

Commit Conventions

git commit --author="TPTBusiness <tpt.requests@pm.me>" -m "type: description"

# Types:
# - feat: New feature
# - fix: Bug fix
# - docs: Documentation
# - style: Formatting
# - refactor: Code restructuring
# - test: Tests
# - chore: Maintenance

Module Structure

"""
Module Name - Brief description

Longer description if needed.
"""

import numpy as np
import pandas as pd
from typing import Dict, List, Optional
from datetime import datetime

class ClassName:
    """Class docstring."""
    
    def __init__(self, param: type) -> None:
        """Initialize."""
        pass
    
    def method(self, param: type) -> ReturnType:
        """
        Method docstring.
        
        Parameters
        ----------
        param : type
            Description
        
        Returns
        -------
        ReturnType
            Description
        """
        pass

Backtesting Module Usage

from rdagent.components.backtesting import (
    FactorBacktester,
    ResultsDatabase,
    PortfolioOptimizer,
    AdvancedRiskManager
)

# Run backtest
backtester = FactorBacktester()
metrics = backtester.run_backtest(
    factor_values=factor_series,
    forward_returns=forward_returns,
    factor_name="MyFactor"
)

# Save to database
db = ResultsDatabase()
db.add_backtest("MyFactor", metrics)

# Query top factors
top = db.get_top_factors('sharpe_ratio', limit=20)

# Portfolio optimization
optimizer = PortfolioOptimizer()
weights = optimizer.mean_variance(expected_returns, cov_matrix)

# Risk management
risk_manager = AdvancedRiskManager()
report = risk_manager.generate_risk_report(returns, weights)

Key Metrics

Metric	Target	Minimum
IC (Information Coefficient)	> 0.05	> 0.02
Sharpe Ratio	> 2.0	> 1.0
Max Drawdown	< 15%	< 25%
Win Rate	> 55%	> 45%
Annualized Return	> 10%	> 5%

Important Files

rdagent/app/cli.py - Main CLI entry point
rdagent/components/backtesting/ - Backtest engine
rdagent/components/coder/factor_coder/ - Factor generation
results/README.md - Results documentation
data_config.yaml - EURUSD configuration
web/dashboard_api.py - Dashboard API
requirements.txt - Dependencies

External Dependencies

llama.cpp - Local LLM inference (Qwen3.5-35B)
Ollama - Embedding models
Qlib - Backtesting engine
yfinance - Live market data

Common Issues

LLM Connection Errors: Ensure llama.cpp server is running on port 8081
Embedding Errors: Check Ollama is running with nomic-embed-text loaded
Database Lock: Close all connections before running multiple processes
Memory Issues: Reduce batch size or context length for LLM

Project Status

✅ Factor Generation (110+ factors created)
✅ Backtesting Engine (IC, Sharpe, Drawdown, RL support)
✅ Results Database (SQLite with queries)
✅ Risk Management (Correlation, Portfolio Optimization)
✅ Trading Protection System (Drawdown, Cooldown, Stoploss Guard, Low Performance)
✅ RL Trading Agent (PPO/A2C/SAC with Gymnasium environment + fallback)
✅ Dashboards (Web + CLI)
✅ CLI Commands (fin_quant, rl_trading, health_check, etc.)
✅ Integration Tests (200+ tests, run before EVERY commit)
✅ Security Scanning (Bandit pre-commit hook)
⏳ Live Trading (Paper trading - in development)

Next Steps

✅ Connect RL with Protection Manager (DONE)
✅ Connect RL with Backtesting Engine (DONE)
✅ Add CLI command for RL Trading (DONE)
✅ Ensure GitHub users can run full system (DONE - fallback system)
Backtest all 110 factors
Select top 20 by IC/Sharpe
Portfolio optimization
4 weeks paper trading
Live trading with small capital

Git Commit Guidelines

Language Policy

ALL commit messages MUST be in English.

❌ Wrong (German):

git commit -m "feat: Neue Funktion hinzugefügt"
git commit -m "fix: Fehler behoben"
git commit -m "chore: QWEN.md zu .gitignore hinzugefügt"

✅ Correct (English):

git commit -m "feat: Add new feature"
git commit -m "fix: Fix bug"
git commit -m "chore: Add QWEN.md to .gitignore"

Pre-Commit Checklist

BEFORE every commit, you MUST:

Run git status and verify:
- Only intended files are staged
- No generated files (.qwen/, results/, *.db, etc.)
- No sensitive data (.env, API keys, etc.)

Check .gitignore is working:

git status
# Verify .qwen/, results/, *.db are NOT shown

Review staged changes:

git diff --staged
# Review what will be committed

Run tests (if applicable):

pytest test/backtesting/ -v
# Ensure all tests pass

Commit Message Format

Use Conventional Commits:

<type>: <description in English>

[optional body]

Types:

feat: - New feature
fix: - Bug fix
test: - Tests
docs: - Documentation
chore: - Maintenance
style: - Formatting
refactor: - Code restructuring

Examples:

feat: Add backtesting tests with 98% coverage
fix: Remove .qwen/ from Git tracking
test: Add unit tests for ResultsDatabase
docs: Update QWEN.md with commit guidelines
chore: Add pytest to requirements.txt

Protected Files (NEVER commit)

These files/directories MUST NEVER be committed:

.qwen/              # AI agent files (generated)
results/            # Backtest results (sensitive data)
*.db                # SQLite databases
.env                # Environment variables (API keys!)
git_ignore_folder/  # Generated data
*.log               # Log files

If you accidentally commit any of these:

# Remove from last commit (keeps files locally)
git reset HEAD~1

# Or remove from tracking
git rm -r --cached .qwen/
git commit -m "chore: Remove .qwen/ from tracking"

Fixing Past Commits

To fix the last 3-5 commits:

# For last 5 commits
git rebase -i HEAD~5

# In the editor, change 'pick' to 'reword' for commits to rename
# Save and close
# Write new English message for each commit

To fix older commits (advanced):

# Find the commit hash
git log --oneline

# Start rebase from that commit
git rebase -i <commit-hash>^

# Follow same process as above

Current German commits to fix (as of April 2026):

73140b68 test: Backtesting Tests mit 98.77% Coverage
     → test: Add backtesting tests with 98.77% coverage

5148d17d chore: QWEN.md zu .gitignore hinzugefügt
     → chore: Add QWEN.md to .gitignore

df93e162 feat: Intelligent Embedding Chunking statt Kürzung
     → feat: Intelligent embedding chunking instead of truncation

01aa183a fix: CLI Dashboard in separatem Terminal-Fenster
     → fix: CLI dashboard in separate terminal window

df356978 feat: predix.py Wrapper für Dashboard-Support
     → feat: predix.py wrapper for dashboard support

89d01f5d feat: Beautiful CLI Dashboard + korrigierter Start-Befehl
     → feat: Beautiful CLI dashboard + corrected start command

48e4f44e feat: Auto-Start Dashboard für fin_quant
     → feat: Auto-start dashboard for fin_quant

59122a19 feat: Dashboard + Live-Daten Integration (Phase 4)
     → feat: Dashboard + live data integration (Phase 4)

a0f414ed feat: EURUSD Trading-Verbesserungen (Phase 2 & 3)
     → feat: EURUSD trading improvements (Phase 2 & 3)

e8b962b5 feat: EURUSD Trading-Verbesserungen implementiert (Phase 1)
     → feat: Implement EURUSD trading improvements (Phase 1)

⚠️ Warning: Rewriting history changes commit hashes. If you've already pushed:

# After rebasing locally
git push --force-with-lease origin master

# Tell team members to re-clone:
git clone <repo-url>

Push Policy

BEFORE pushing:

Verify commit messages are in English
Verify no protected files are included
Run tests one final time

git status
git log -3 --oneline  # Verify last 3 commits
pytest test/backtesting/ -v  # Quick test
git push origin master

Enforcement

All PRs will be rejected if commit messages are not in English
Protected files in commits will be rejected
Tests must pass before merging

Remember: Consistent English commit messages ensure:

International collaboration
Better searchability
Professional project history

Implementation Guide: Prompts & Models

Using the Prompt Loader

Auto-Load Prompts (Local First):

from rdagent.components.loader import load_prompt

# Load factor discovery prompt
# Automatically loads from prompts/local/ if exists!
prompt = load_prompt("factor_discovery")

# Load specific section
system_prompt = load_prompt("factor_discovery", section="system")
user_prompt = load_prompt("factor_discovery", section="user")

# Force local only (raise error if not found)
prompt = load_prompt("factor_discovery", local_only=True)

# List available prompts
from rdagent.components.loader import list_available_prompts
available = list_available_prompts()
print(f"Standard: {available['standard']}")
print(f"Local: {available['local']}")

Priority:

prompts/local/factor_discovery_v2.yaml (loaded first if exists)
prompts/local/factor_discovery.yaml
prompts/standard_prompts.yaml (fallback)

Using the Model Loader

Auto-Load Models (Local First):

from rdagent.components.model_loader import load_model

# Load XGBoost model
# Automatically loads from models/local/ if exists!
model_factory = load_model("xgboost_factor")

# Create model instance
model = model_factory(max_depth=8, learning_rate=0.03)

# Train
model.fit(X_train, y_train, epochs=50, batch_size=64)

# Predict
predictions = model.predict(X_test)

# Save/Load
model.save("models/my_model.pth")
model.load("models/my_model.pth")

Available Models:

Model	Location	Use Case
`xgboost_factor`	`models/standard/`	Tabular data, fast training
`lightgbm_factor`	`models/standard/`	Large datasets, faster than XGBoost
`transformer_factor`	`models/local/`	Time-series, long-range dependencies
`tcn_factor`	`models/local/`	Multi-scale patterns
`patchtst_factor`	`models/local/`	SOTA for time-series forecasting
`cnn_lstm_hybrid`	`models/local/`	Complex pattern recognition

Priority:

models/local/{name}_v2.py (loaded first if exists)
models/local/{name}.py
models/standard/{name}.py (fallback)

Creating Your Improved Prompts

Step 1: Create Local Prompt

mkdir -p prompts/local
nano prompts/local/factor_discovery_v3.yaml

Step 2: Add Your Improvements

# prompts/local/factor_discovery_v3.yaml

factor_discovery:
  system: |-
    YOUR IMPROVED SYSTEM PROMPT HERE
    
    Add your proprietary insights:
    - Specific EURUSD patterns you've discovered
    - Your unique factor formulas
    - Custom session filters
    - Proprietary risk management rules
    
  user: |-
    YOUR IMPROVED USER PROMPT HERE

Step 3: Test

from rdagent.components.loader import load_prompt

# Auto-loads your v3!
prompt = load_prompt("factor_discovery")

Creating Your Improved Models

Step 1: Create Local Model

mkdir -p models/local
nano models/local/my_optimized_model.py

Step 2: Implement Model

# models/local/my_optimized_model.py
"""
My Optimized Model v1.0
Better than standard with custom improvements.
"""

import torch
import torch.nn as nn

class MyOptimizedModel(nn.Module):
    def __init__(self, **params):
        super().__init__()
        # Your custom architecture
        pass
    
    def forward(self, x):
        # Your custom forward pass
        pass

def create_my_optimized_model(**params):
    """Factory function."""
    return MyOptimizedModel(**params)

Step 3: Test

from rdagent.components.model_loader import load_model

# Auto-loads your optimized model!
model_factory = load_model("my_optimized_model")
model = model_factory()

Backup Your Private Assets

Backup Prompts & Models to Private Repo:

# Create private repo on GitHub: predix-private-assets

# Clone private repo
cd ~/Dev
git clone git@github.com:TPTBusiness/predix-private-assets.git

# Copy local assets
cp -r ~/Predix/prompts/local/* ~/predix-private-assets/prompts/
cp -r ~/Predix/models/local/* ~/predix-private-assets/models/

# Commit to private repo
cd ~/predix-private-assets
git add .
git commit -m "Backup: prompts v2, models (Transformer, TCN, PatchTST, CNN+LSTM)"
git push

Auto-Sync Script:

# ~/Predix/sync_private.sh
#!/bin/bash
echo "Syncing private assets..."
rsync -av prompts/local/ ~/predix-private-assets/prompts/
rsync -av models/local/ ~/predix-private-assets/models/
cd ~/predix-private-assets && git add . && git commit -m "Auto-sync $(date)" && git push
echo "Done!"

Security Best Practices

What to Keep Private:

✅ Your proprietary model architectures ✅ Optimized prompt templates ✅ Best-performing factors ✅ Evolution weights ✅ Trade secrets & alpha-generating logic

What NOT to Commit:

❌ Anything in prompts/local/ ❌ Anything in models/local/ ❌ .env (API keys) ❌ results/ (backtest performance) ❌ git_ignore_folder/ (trading data)

Verify Before Committing:

# Check what will be committed
git status
git diff --staged

# Verify .gitignore is working
git status
# Should NOT show prompts/local/, models/local/, .env, results/

Development Guidelines for AI Assistant

🌍 CRITICAL: Open Source Compatibility

BEFORE implementing ANY feature, ask yourself:

Can a GitHub user run this without our local files?
- ✅ YES → Good, proceed
- ❌ NO → Add fallback or graceful degradation
Does this break if optional dependencies are missing?
- Example: stable-baselines3, llama.cpp, Ollama
- Solution: Try/except with clear warning messages
Is this feature documented for external users?
- Update README.md with usage instructions
- Ensure installation guide covers all dependencies

PRINCIPLE: The project on GitHub MUST be fully functional for users. Our closed-source assets (models/local/, prompts/local/, .env) are ENHANCEMENTS, not requirements.

⚠️ MANDATORY Rules for ALL Development

When implementing NEW features or making SIGNIFICANT changes, you MUST:

1. 📝 Update QWEN.md

When: Every time you add a new feature, module, or change existing architecture.

What to update:

Architecture section (if structure changes)
Important Files section
Testing section
Key Metrics (if targets change)
Project Status
Next Steps

Example:

### Architecture
├── rdagent/
│   └── components/
│       └── backtesting/
│           └── protections/          # NEW: Trading protection system
│               ├── base.py
│               ├── max_drawdown.py
│               └── protection_manager.py

2. 📖 Update README.md

When: Every user-facing feature change or major update.

What to update:

Features list
Installation instructions
Usage examples
Configuration examples

Keep it user-focused:

## Features
- ✅ Trading Protection System (NEW)
  * Automatic drawdown protection
  * Cooldown periods after losses
  * Stoploss cluster detection

3. 📦 Update requirements.txt

When: Adding new dependencies or removing unused ones.

What to update:

requirements.txt (main dependencies)
requirements/lint.txt (dev dependencies)
requirements/test.txt (test dependencies)

Example:

# If you add a new library
echo "new-library==1.0.0" >> requirements.txt

# If you add a new test dependency
echo "pytest-mock" >> requirements/test.txt

4. ✅ Extend Tests

When: EVERY time you add new code.

Rule: New features MUST have tests with >80% coverage.

What to create:

Unit tests in test/ directory
Integration tests in test/integration/
Update existing tests if behavior changed

Test structure:

# test/feature_type/test_new_feature.py
"""Tests for New Feature"""

class TestNewFeature:
    """Test new feature thoroughly."""
    
    def test_basic_functionality(self): ...
    def test_edge_cases(self): ...
    def test_error_handling(self): ...
    def test_integration_with_existing(self): ...

Update integration tests:

# Add to test/integration/test_all_features.py
class TestNewFeature:
    """Test new feature integration."""
    
    def test_imports(self): ...
    def test_initialization(self): ...
    def test_full_workflow(self): ...

5. 🔄 Pre-Commit Checklist

BEFORE every commit with new features:

# 1. Run ALL tests
pytest test/ -v

# 2. Run integration tests
pytest test/integration/test_all_features.py -v

# 3. Check test coverage
pytest --cov=rdagent.components.new_module -v

# 4. Run security scan
bandit -r rdagent/ -c .bandit.yml

# 5. Verify tests updated
git status
# Should show test files modified

Documentation Priority Order

QWEN.md - Internal AI assistant context (UPDATE ALWAYS)
Test files - Code documentation through tests (MANDATORY)
README.md - User-facing documentation (UPDATE for user-visible changes)
requirements.txt - Dependencies (UPDATE when adding libraries)
Inline code comments - English only (ALWAYS)

Example Workflow: Adding New Feature

1. Plan feature
   ↓
2. Implement code
   ↓
3. Write unit tests (test/...)
   ↓
4. Write integration tests (test/integration/...)
   ↓
5. Run ALL tests → Must pass
   ↓
6. Update QWEN.md ← MANDATORY
   ↓
7. Update README.md (if user-visible)
   ↓
8. Update requirements.txt (if new deps)
   ↓
9. Commit with clear message
   ↓
10. Pre-commit hooks run automatically
    ↓
11. Push to remote

Penalties for Not Following Rules

If you forget to update:

❌ Missing tests → Code cannot be committed (pre-commit blocks)
❌ Missing QWEN.md update → Next AI assistant will work with outdated context
❌ Missing README update → Users won't understand new features
❌ Missing requirements.txt → Installation will fail

Remember: These rules ensure:

Code quality through tests
AI assistant has current context
Users understand changes
Dependencies are tracked

🚀 COMPLETE 5-PHASE ARCHITECTURE

Phase 1: Factor Generation (Open Source - ALWAYS ACTIVE)

1. Hypothesis Generation (LLM v3 Prompt)
   → MultiIndex code examples (unstack/stack pattern)
   → Working code templates
   → Volume warning (FX volume = 0 often)

2. CoSTEER Code Validation
   → Execute factor code
   → Validate result.h5 output
   → Retry with feedback (max 3 retries)

3. Qlib Docker Backtest
   → LightGBM training on factor
   → Portfolio backtest (TopkDropoutStrategy)
   → IC, Sharpe, Max DD, Win Rate calculation

4. Results Storage
   → results/factors/{name}.json (Code + Description + Metrics)
   → results/db/backtest_results.db (SQLite)
   → results/logs/ (Running logs)

⚡ CONTINUE UNTIL 5000+ VALID FACTORS REACHED

Phase 2: ML Model Training (Closed Source - Local Only)

5. Load Top 50 Factors (by IC ≥ 0.01)
   → From results/factors/ with valid IC
   → Extract factor values from workspaces

6. Build Feature Matrix
   → X = factor values (samples × factors)
   → y = forward returns (96-bar shift)

7. Train LightGBM Model
   → Split: 80% train, 20% validate
   → Early stopping (50 rounds)
   → Feature importance analysis

8. Model Validation
   → IC (train vs valid)
   → Sharpe-like metric
   → Overfitting detection

9. Save Model
   → results/models/{name}/model.txt
   → results/models/{name}/metadata.json

Phase 3: Portfolio Optimization (Closed Source - Local Only)

10. Load Top 30 Factors
    → Compute correlation matrix
    → Select uncorrelated factors (max corr = 0.3)

11. Optimize Weights
    → Weight by absolute IC
    → Normalize to sum = 1.0

12. Backtest Portfolio
    → Combined factor score = Σ(weight_i × factor_i)
    → Calculate IC, Sharpe, Max DD, Win Rate

13. Save Portfolio
    → results/portfolios/{name}.json

Phase 4: Strategy Generation (Closed Source - Local Only)

14. Generate Trading Rules
    → Entry signals (factor thresholds)
    → Exit signals (take profit, stop loss)
    → Position sizing (Kelly criterion)

15. Add Risk Management
    → Max drawdown protection
    → Cooldown periods after losses
    → Stoploss cluster detection

16. Save Strategy
    → results/strategies/{name}.json

Phase 5: Iterative Improvement (Closed Source - Local Only)

17. ML Feedback Loop
    → Use model performance to guide factor generation
    → Identify feature importance patterns
    → Generate factors targeting weak areas

18. Portfolio Feedback
    → Use portfolio performance to refine weights
    → Add new uncorrelated factors
    → Remove degraded factors

19. Loop Back to Phase 1
    → Generate NEW factors with ML insights
    → Retrain model with expanded factor set
    → Continuous improvement cycle

📊 CURRENT RESULTS (as of April 2026)

Factor Evaluation (1009 factors, FULL DATA 2020-2026)

Metric	Value
Total evaluated	1,009
Successful	337 (33%)
Failed	672 (67%)
Best IC	0.255 (daily_close_open_mom)
Avg IC (valid)	0.011
Best Sharpe	1.71 (DCP)

Top 10 Factors by IC

#	Factor	IC	Sharpe
1	daily_close_open_mom	0.255	0.007
2	daily_ret_log_1d	0.255	0.003
3	daily_ret_close_1d	0.255	0.005
4	daily_close_to_close_return	0.255	0.005
5	daily_ret_vol_adj_1d	0.235	-0.007
6	daily_ols_slope_96	0.227	0.002
7	DCP	0.199	1.71
8	DailyTrendStrength_Raw	0.143	-0.016
9	daily_c2c_return	0.129	0.001
10	daily_momentum	0.129	-0.001

Failure Analysis (672 failed)

Error Type	Count	%	Cause
Code crashed	540	80.4%	MultiIndex errors (FIXED in v3 prompt)
All NaN values	97	14.4%	Volume=0, rolling window too large
Other errors	28	4.2%	Various
Timeout (120s)	5	0.7%	Computationally expensive
Too little overlap	2	0.3%	Data mismatch

💡 OPTIMIZATION POTENTIAL (HIGH-END UPGRADES)

1. Code Quality Improvements

Current: 33% success rate
Target: 70%+ with v3 prompt (MultiIndex examples)
Expected: ~700 valid factors from 1009 generated

2. ML Pipeline Enhancements

Feature Selection: Use SHAP values for importance
Ensemble Models: Combine LightGBM + XGBoost + Neural Net
Cross-Validation: Time-series split to prevent overfitting
Hyperparameter Optimization: Optuna for automatic tuning

3. Portfolio Optimization

Risk Parity: Equal risk contribution instead of IC-weighted
Black-Litterman: Incorporate LLM views as priors
Regime Detection: Switch portfolios based on market state
Dynamic Rebalancing: Adjust weights based on rolling IC

4. Strategy Generation

Regime-Specific Rules: Different signals for trending vs mean-reverting
Multi-Timeframe: Combine 1min, 5min, 15min signals
Adaptive Thresholds: Dynamic entry/exit based on volatility
News Integration: Avoid trading during high-impact news

5. Execution Optimization

Parallel Factor Generation: 8+ workers instead of 4
Smart Retry Logic: Learn from failures, adjust prompts
Early Stopping: Skip factors that show promise in first 1000 bars
Incremental Evaluation: Evaluate factors as they're generated

6. Risk Management

VaR/ES: Value at Risk and Expected Shortfall calculations
Correlation Monitoring: Track factor correlation drift
Performance Attribution: Understand which factors drive returns
Stress Testing: Test strategies on historical crises

7. Infrastructure

GPU Acceleration: Use RTX 5060 Ti for LightGBM training
Database Optimization: Index queries for faster factor selection
Caching Layer: Cache expensive computations
Monitoring Dashboard: Real-time performance tracking

44 KiB Raw Blame History Unescape Escape

Predix - QWEN.md Context File

Project Overview

Core Purpose

Key Technologies

Architecture

CLI Commands Reference

Trading Loop

Parallel Execution

AI Strategy Generation (REAL OHLCV Backtest)

Strategy Reports

Factor Evaluation

Debug

🚀 Live Trading System (cTrader + FTMO)

Overview

Architecture

Files (Closed Source)

Prerequisites

Setup cTrader API

How It Works

1. Strategy Loading

2. Factor Calculation

3. Signal Generation

4. Order Execution

5. Risk Management

Usage

Paper Trading (TEST FIRST!)

Live Trading (REAL MONEY)

Custom Parameters

CLI Options

Monitoring

Log Files

Trade Log Format

⚠️ Critical Warnings

Troubleshooting

cTrader API Endpoints

Future Enhancements

Configuration

Running the Trading Loop

Running the Dashboard

Testing

Integration Test Suite (ALL Features)

Unit Tests

Code Quality

Development Conventions

Language Policy

Code Style

Testing Practices

Commit Conventions

Module Structure

Backtesting Module Usage

Key Metrics

Important Files

External Dependencies

Common Issues

Project Status

Next Steps

Git Commit Guidelines

Language Policy

Pre-Commit Checklist

Commit Message Format

Protected Files (NEVER commit)

Fixing Past Commits

Push Policy

Enforcement

Implementation Guide: Prompts & Models

Using the Prompt Loader

Using the Model Loader

Creating Your Improved Prompts

Creating Your Improved Models

Backup Your Private Assets

Security Best Practices

Development Guidelines for AI Assistant

🌍 CRITICAL: Open Source Compatibility

⚠️ MANDATORY Rules for ALL Development

1. 📝 Update QWEN.md

2. 📖 Update README.md

3. 📦 Update requirements.txt

4. ✅ Extend Tests

5. 🔄 Pre-Commit Checklist

44 KiB

Raw Blame History