diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..8e28c52 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,22 @@ +# Only the API code is needed β€” keep context minimal +.git/ +.venv/ +venv/ +__pycache__/ +*.pyc +node_modules/ +.next/ +*.log +*.pkl +*.joblib +docs/ +archive/ +backtests/ +tests/ +scripts/ +models/ +logs/ +data/ +.env +.env.local +*.md diff --git a/.env.docker.example b/.env.docker.example new file mode 100644 index 0000000..d97cf64 --- /dev/null +++ b/.env.docker.example @@ -0,0 +1,39 @@ +# =========================================== +# XAUBot AI - Docker Environment Configuration +# =========================================== + +# ============== MT5 CONNECTION ============== +# Your MetaTrader 5 account credentials +MT5_LOGIN=your_mt5_login +MT5_PASSWORD=your_mt5_password +MT5_SERVER=your_mt5_server +MT5_PATH=/path/to/mt5/terminal + +# ============== TRADING CONFIG ============== +SYMBOL=XAUUSD +CAPITAL=10000 + +# ============== DATABASE ============== +DB_HOST=postgres +DB_PORT=5432 +DB_USER=trading_bot +DB_PASSWORD=trading_bot_2026 +DB_NAME=trading_db + +# ============== TELEGRAM (Optional) ============== +TELEGRAM_BOT_TOKEN= +TELEGRAM_CHAT_ID= + +# ============== PORTS ============== +# Ports accessible from host machine +API_PORT=8000 # Trading API (FastAPI) +DASHBOARD_PORT=3000 # Web Dashboard (Next.js) +DB_PORT=5432 # PostgreSQL +PGADMIN_PORT=5050 # pgAdmin (optional) + +# ============== PGADMIN (Optional) ============== +PGADMIN_EMAIL=admin@trading.local +PGADMIN_PASSWORD=admin123 + +# ============== TIMEZONE ============== +TZ=Asia/Jakarta diff --git a/DOCKER-INTEGRATION.md b/DOCKER-INTEGRATION.md new file mode 100644 index 0000000..3d3e609 --- /dev/null +++ b/DOCKER-INTEGRATION.md @@ -0,0 +1,255 @@ +# Dashboard Integration with Existing Docker Setup + +## 🎯 Overview + +Dashboard dan API telah diintegrasikan ke dalam Docker setup yang **sudah ada**. Database PostgreSQL yang sudah running **TIDAK AKAN DIGANGGU**. + +## βœ… Existing Setup (Tidak Berubah) + +Yang sudah jalan dan **tetap aman**: +- βœ… `trading_bot_db` - PostgreSQL database +- βœ… `trading_bot_network` - Docker network +- βœ… Database schema dengan 7 tables (trades, signals, dll) +- βœ… Volume `postgres_data` untuk persistence + +## πŸ†• New Services Added + +Layanan baru yang ditambahkan: +1. **trading-api** - FastAPI backend untuk dashboard +2. **dashboard** - Next.js web interface +3. **pgadmin** - Database management (optional) + +## πŸš€ Quick Start + +### Option 1: Gunakan Helper Script (Recommended) + +```cmd +# Tambahkan dashboard ke setup yang sudah ada +docker-add-dashboard.bat +``` + +Script ini akan: +1. Check database yang sudah running +2. Build API & Dashboard services +3. Start kedua services baru +4. Connect ke database & network yang sudah ada + +### Option 2: Manual Docker Compose + +```cmd +# Build hanya services baru +docker-compose build trading-api dashboard + +# Start hanya services baru +docker-compose up -d trading-api dashboard +``` + +## πŸ“Š Access Points + +Setelah services running: +- **Dashboard:** http://localhost:3000 +- **API:** http://localhost:8000 +- **API Docs:** http://localhost:8000/docs +- **Database:** localhost:5432 (sudah running) + +## πŸ”§ Service Management + +### Check Status +```cmd +# Lihat status semua services +docker-status.bat + +# Atau manual +docker-compose ps +``` + +### View Logs +```cmd +# Logs dashboard +docker-compose logs -f dashboard + +# Logs API +docker-compose logs -f trading-api + +# Logs database +docker-compose logs -f postgres +``` + +### Restart Services +```cmd +# Restart hanya dashboard +docker-compose restart dashboard + +# Restart hanya API +docker-compose restart trading-api + +# Restart semua (termasuk database) +docker-compose restart +``` + +### Remove Dashboard (Keep Database) +```cmd +# Hapus dashboard tapi tetap keep database +docker-remove-dashboard.bat + +# Atau manual +docker-compose stop trading-api dashboard +docker-compose rm -f trading-api dashboard +``` + +## πŸ”— Service Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ trading_bot_network β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β”‚ +β”‚ πŸ“Š Dashboard (NEW) β”‚ +β”‚ Port: 3000 β”‚ +β”‚ └─> http://trading-api:8000 β”‚ +β”‚ β”‚ +β”‚ πŸ”Œ Trading API (NEW) β”‚ +β”‚ Port: 8000 β”‚ +β”‚ └─> postgres:5432 β”‚ +β”‚ β”‚ +β”‚ πŸ—„οΈ PostgreSQL (EXISTING - NO CHANGE) β”‚ +β”‚ Port: 5432 β”‚ +β”‚ Status: Already Running β”‚ +β”‚ Volume: postgres_data β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## πŸ“ Environment Variables + +Edit `.env` untuk konfigurasi: + +```env +# MT5 (Required for API) +MT5_LOGIN=your_login +MT5_PASSWORD=your_password +MT5_SERVER=your_server +MT5_PATH=C:/Program Files/MetaTrader 5/terminal64.exe + +# Trading +SYMBOL=XAUUSD +CAPITAL=10000 + +# Database (Already configured) +DB_USER=trading_bot +DB_PASSWORD=trading_bot_2026 +DB_NAME=trading_db + +# Ports +API_PORT=8000 +DASHBOARD_PORT=3000 +DB_PORT=5432 +``` + +## πŸ› Troubleshooting + +### Dashboard tidak bisa connect ke API + +**Check API health:** +```cmd +curl http://localhost:8000/api/health +``` + +**View API logs:** +```cmd +docker-compose logs -f trading-api +``` + +### API tidak bisa connect ke database + +**Check database:** +```cmd +docker exec trading_bot_db pg_isready -U trading_bot +``` + +**Check network:** +```cmd +docker network inspect trading_bot_network +``` + +### Port conflict + +Edit `.env` untuk ganti port: +```env +API_PORT=8001 +DASHBOARD_PORT=3001 +``` + +Then restart: +```cmd +docker-compose down trading-api dashboard +docker-compose up -d trading-api dashboard +``` + +## πŸ’Ύ Data Persistence + +**Database data tetap aman:** +- Volume `postgres_data` tetap ada +- Hapus container tidak hapus data +- Data tersimpan di Docker volume + +**Check volume:** +```cmd +docker volume ls | findstr postgres +docker volume inspect trading_bot_postgres_data +``` + +## πŸ”„ Updates + +**Update code dan rebuild:** +```cmd +# Pull latest code +git pull + +# Rebuild services baru +docker-compose build trading-api dashboard + +# Restart +docker-compose up -d trading-api dashboard +``` + +**Database tidak perlu rebuild** karena schema sudah ada. + +## ⚠️ Important Notes + +1. **Database tidak boleh dihapus** - Data trades ada di sini +2. **Jangan run `docker-compose down -v`** - Ini akan hapus volumes +3. **Untuk stop semua:** `docker-compose stop` (data aman) +4. **Untuk restart:** `docker-compose restart` atau `docker-compose up -d` + +## πŸ“š Files Structure + +``` +xaubot-ai/ +β”œβ”€β”€ docker-compose.yml # Main orchestration (UPDATED) +β”œβ”€β”€ Dockerfile # API image (NEW) +β”œβ”€β”€ .env # Environment config +β”œβ”€β”€ .dockerignore # Build exclusions +β”œβ”€β”€ docker-add-dashboard.bat # Add dashboard script (NEW) +β”œβ”€β”€ docker-remove-dashboard.bat # Remove dashboard script (NEW) +β”œβ”€β”€ docker-status.bat # Status check script (NEW) +β”œβ”€β”€ docker/ +β”‚ └── init-db/ +β”‚ └── 01-schema.sql # Database schema (EXISTING) +└── web-dashboard/ + β”œβ”€β”€ Dockerfile # Dashboard image (NEW) + └── .dockerignore # Build exclusions +``` + +## 🎯 Summary + +βœ… **Database tetap jalan** - Tidak ada perubahan +βœ… **Services baru ditambahkan** - API & Dashboard +βœ… **Data aman** - Volume persistence +βœ… **Easy management** - Helper scripts +βœ… **Independent** - Bisa start/stop tanpa ganggu database + +--- + +**Integration completed:** Feb 6, 2026 +**Status:** Dashboard integrated with existing Docker setup ✨ diff --git a/DOCKER-SETUP-SUMMARY.md b/DOCKER-SETUP-SUMMARY.md new file mode 100644 index 0000000..9c446e1 --- /dev/null +++ b/DOCKER-SETUP-SUMMARY.md @@ -0,0 +1,401 @@ +# XAUBot AI - Docker Integration Summary + +## βœ… Completed Tasks + +### 1. **Created Dockerfile for Next.js Dashboard** +- Multi-stage build for optimization +- Standalone output for minimal image size +- Production-ready configuration +- Non-root user for security + +**Location:** `web-dashboard/Dockerfile` + +### 2. **Created Dockerfile for Python Trading API** +- Python 3.11-slim base image +- FastAPI server with health checks +- Proper dependency management +- Volume mounts for data/logs/models + +**Location:** `Dockerfile` (root directory) + +### 3. **Updated Docker Compose Configuration** +- 4 services: postgres, trading-api, dashboard, pgadmin +- Proper service dependencies and health checks +- Custom bridge network for inter-service communication +- Environment variable support via .env file +- Volume persistence for database and pgadmin + +**Location:** `docker-compose.yml` + +### 4. **Created Environment Configuration** +- Template with all required variables +- Clear documentation for each setting +- Default values for non-sensitive configs + +**Location:** `.env.docker.example` + +### 5. **Created Docker Ignore Files** +- Excludes unnecessary files from images +- Reduces build context size +- Improves build performance + +**Locations:** +- `web-dashboard/.dockerignore` +- `.dockerignore` (root) + +### 6. **Created Helper Scripts** + +#### Windows Batch Scripts: +- `docker-start.bat` - Start all services +- `docker-stop.bat` - Stop services with options +- `docker-logs.bat` - View service logs + +#### Linux/Mac Shell Scripts: +- `docker-start.sh` - Start all services +- `docker-stop.sh` - Stop services with options +- `docker-logs.sh` - View service logs + +### 7. **Updated Next.js Configuration** +- Enabled standalone output for Docker +- Optimized for production builds + +**Location:** `web-dashboard/next.config.ts` + +### 8. **Created Comprehensive Documentation** +- Complete Docker setup guide +- Architecture diagram +- Service management commands +- Troubleshooting section +- Security best practices +- Performance tuning tips + +**Location:** `DOCKER.md` + +### 9. **Updated Main README** +- Added Docker deployment section as recommended method +- Clear quick start instructions +- Links to full documentation + +**Location:** `README.md` + +## πŸ—οΈ Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Docker Network β”‚ +β”‚ (trading_bot_network) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Dashboard │────────▢│ Trading API β”‚ β”‚ +β”‚ β”‚ (Next.js) β”‚ HTTP β”‚ (FastAPI) β”‚ β”‚ +β”‚ β”‚ Port: 3000 β”‚ β”‚ Port: 8000 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”‚ PostgreSQL β”‚ +β”‚ β”‚ Protocol β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ PostgreSQL β”‚ β”‚ +β”‚ β”‚ Database β”‚ β”‚ +β”‚ β”‚ Port: 5432 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” (Optional - Admin Profile) β”‚ +β”‚ β”‚ pgAdmin β”‚ β”‚ +β”‚ β”‚ Port: 5050 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↕ Exposed Ports + localhost:3000 (Dashboard) + localhost:8000 (API) + localhost:5432 (Database) + localhost:5050 (pgAdmin) +``` + +## πŸš€ Quick Start Guide + +### 1. Initial Setup (One-time) + +```bash +# Navigate to project +cd "Smart Automatic Trading BOT + AI" + +# Create environment file +copy .env.docker.example .env + +# Edit .env with your MT5 credentials +notepad .env +``` + +**Required credentials in .env:** +```env +MT5_LOGIN=your_login +MT5_PASSWORD=your_password +MT5_SERVER=your_server +MT5_PATH=/path/to/mt5/terminal +``` + +### 2. Start Services (Windows) + +**Option A: Using helper script (Recommended)** +```cmd +REM Start core services +docker-start.bat + +REM Or start with pgAdmin +docker-start.bat --admin +``` + +**Option B: Manual docker-compose** +```cmd +REM Build and start +docker-compose up -d + +REM With pgAdmin +docker-compose --profile admin up -d +``` + +### 3. Access the Dashboard + +Open your browser and go to: +- **Dashboard:** http://localhost:3000 + +You'll see: +- Real-time price updates +- Account balance and equity +- Trading signals (SMC + ML) +- Market regime +- Open positions +- Risk status +- Activity logs + +### 4. Check Other Services + +- **API Docs:** http://localhost:8000/docs +- **API Health:** http://localhost:8000/api/health +- **API Status:** http://localhost:8000/api/status +- **pgAdmin:** http://localhost:5050 (if started with --admin) + +## πŸ“‹ Common Commands + +### View Logs +```cmd +REM All services +docker-logs.bat + +REM Specific service +docker-logs.bat trading-api +docker-logs.bat dashboard +docker-logs.bat postgres +``` + +### Check Status +```cmd +docker-compose ps +``` + +### Restart Services +```cmd +REM Restart all +docker-compose restart + +REM Restart specific +docker-compose restart trading-api +docker-compose restart dashboard +``` + +### Stop Services +```cmd +REM Stop (keeps data) +docker-stop.bat + +REM Stop and remove containers (keeps data) +docker-stop.bat --remove + +REM Stop and remove everything including data (⚠️ DANGER!) +docker-stop.bat --clean +``` + +### Update Code and Rebuild +```cmd +REM Pull latest code +git pull + +REM Rebuild and restart +docker-compose build +docker-compose up -d +``` + +## πŸ”§ Configuration + +### Port Configuration + +Default ports can be changed in `.env`: + +```env +API_PORT=8000 # Trading API +DASHBOARD_PORT=3000 # Web Dashboard +DB_PORT=5432 # PostgreSQL +PGADMIN_PORT=5050 # pgAdmin +``` + +### Environment Variables + +All configuration is in `.env`: + +| Category | Variables | +|----------|-----------| +| **MT5** | MT5_LOGIN, MT5_PASSWORD, MT5_SERVER, MT5_PATH | +| **Trading** | SYMBOL, CAPITAL | +| **Database** | DB_USER, DB_PASSWORD, DB_NAME | +| **Telegram** | TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID | +| **Ports** | API_PORT, DASHBOARD_PORT, DB_PORT, PGADMIN_PORT | + +## πŸ› Troubleshooting + +### Dashboard Shows "Connection Error" + +**Check if API is running:** +```cmd +curl http://localhost:8000/api/health +``` + +**View API logs:** +```cmd +docker-logs.bat trading-api +``` + +### Port Already in Use + +**Find what's using the port:** +```cmd +netstat -ano | findstr :3000 +netstat -ano | findstr :8000 +``` + +**Change port in .env:** +```env +DASHBOARD_PORT=3001 +API_PORT=8001 +``` + +**Restart services:** +```cmd +docker-compose down +docker-compose up -d +``` + +### Can't Connect to MT5 + +1. Check credentials in `.env` +2. Ensure MT5 terminal is accessible +3. View API logs for connection errors: + ```cmd + docker-logs.bat trading-api + ``` + +### Database Connection Issues + +**Check database health:** +```cmd +docker-compose ps postgres +``` + +**Test connection:** +```cmd +docker exec -it trading_bot_db pg_isready -U trading_bot +``` + +**View database logs:** +```cmd +docker-logs.bat postgres +``` + +## πŸ“Š Monitoring + +### View Real-time Logs +```cmd +REM Follow all logs +docker-compose logs -f + +REM Follow specific service +docker-compose logs -f trading-api +``` + +### Check Resource Usage +```cmd +docker stats +``` + +### Service Health +```cmd +REM All services +docker-compose ps + +REM Detailed info +docker inspect trading_bot_api +docker inspect trading_bot_dashboard +``` + +## πŸ” Security Notes + +1. **Never commit .env file** - It contains sensitive credentials +2. **Change default passwords** - Especially for database and pgAdmin +3. **Use strong passwords** - For all services +4. **Limit port exposure** - Only expose ports you need +5. **Keep Docker updated** - Regular security updates + +## πŸ“ File Structure + +``` +xaubot-ai/ +β”œβ”€β”€ Dockerfile # Python API Docker image +β”œβ”€β”€ docker-compose.yml # Service orchestration +β”œβ”€β”€ .env # Environment variables (DO NOT COMMIT) +β”œβ”€β”€ .env.docker.example # Environment template +β”œβ”€β”€ .dockerignore # Files to exclude from build +β”œβ”€β”€ docker-start.bat # Windows start script +β”œβ”€β”€ docker-stop.bat # Windows stop script +β”œβ”€β”€ docker-logs.bat # Windows logs script +β”œβ”€β”€ docker-start.sh # Linux/Mac start script +β”œβ”€β”€ docker-stop.sh # Linux/Mac stop script +β”œβ”€β”€ docker-logs.sh # Linux/Mac logs script +β”œβ”€β”€ DOCKER.md # Full Docker documentation +└── web-dashboard/ + β”œβ”€β”€ Dockerfile # Next.js dashboard image + β”œβ”€β”€ .dockerignore # Dashboard build exclusions + └── next.config.ts # Next.js config (standalone output) +``` + +## 🎯 Benefits of Docker Setup + +βœ… **Easy Setup** - One command to start everything +βœ… **Consistent Environment** - Same setup on any machine +βœ… **Isolated Services** - No conflicts with other software +βœ… **Easy Updates** - Rebuild and restart to update +βœ… **Production Ready** - Same setup for dev and production +βœ… **Automatic Restarts** - Services auto-restart on crash +βœ… **Health Monitoring** - Built-in health checks +βœ… **Volume Persistence** - Data survives container restarts + +## πŸ“š Additional Resources + +- **Full Documentation:** [DOCKER.md](DOCKER.md) +- **Styling Guide:** [web-dashboard/STYLING-GUIDE.md](web-dashboard/STYLING-GUIDE.md) +- **Docker Docs:** https://docs.docker.com +- **Docker Compose:** https://docs.docker.com/compose + +## πŸ†˜ Support + +If you encounter issues: + +1. Check the logs: `docker-logs.bat` +2. Verify services: `docker-compose ps` +3. Review troubleshooting section in [DOCKER.md](DOCKER.md) +4. Check service health: `curl http://localhost:8000/api/health` + +--- + +**Setup completed:** Feb 6, 2026 +**Ready to deploy!** πŸš€ diff --git a/DOCKER.md b/DOCKER.md new file mode 100644 index 0000000..9b29394 --- /dev/null +++ b/DOCKER.md @@ -0,0 +1,437 @@ +# XAUBot AI - Docker Setup Guide + +Complete guide to running the XAUBot AI trading system with Docker. + +## πŸ“‹ Prerequisites + +- Docker Engine 20.10+ +- Docker Compose 2.0+ +- 4GB+ RAM available +- MetaTrader 5 account credentials + +## πŸ—οΈ Architecture + +The Docker setup includes 4 services: + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Host Machine β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Dashboard │─────▢│ Trading API β”‚ β”‚ +β”‚ β”‚ Next.js β”‚ β”‚ FastAPI β”‚ β”‚ +β”‚ β”‚ Port: 3000 β”‚ β”‚ Port: 8000 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ PostgreSQL β”‚ β”‚ +β”‚ β”‚ Port: 5432 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” (Optional - Profile: admin) β”‚ +β”‚ β”‚ pgAdmin β”‚ β”‚ +β”‚ β”‚ Port: 5050 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Services + +1. **postgres** - PostgreSQL 16 database for trade logging +2. **trading-api** - Python FastAPI backend serving trading data +3. **dashboard** - Next.js web interface for monitoring +4. **pgadmin** - Database management UI (optional, admin profile) + +## πŸš€ Quick Start + +### 1. Clone & Setup + +```bash +cd "Smart Automatic Trading BOT + AI" + +# Copy environment template +cp .env.docker.example .env +``` + +### 2. Configure Environment + +Edit `.env` file with your credentials: + +```bash +# Required +MT5_LOGIN=your_login +MT5_PASSWORD=your_password +MT5_SERVER=your_server +MT5_PATH=/path/to/mt5 + +# Optional - adjust ports if needed +API_PORT=8000 +DASHBOARD_PORT=3000 +DB_PORT=5432 +``` + +### 3. Start Services + +**Option A: All services (without pgAdmin)** +```bash +docker-compose up -d +``` + +**Option B: All services including pgAdmin** +```bash +docker-compose --profile admin up -d +``` + +**Option C: Specific services only** +```bash +# Just database and API +docker-compose up -d postgres trading-api + +# Add dashboard +docker-compose up -d dashboard +``` + +### 4. Access Services + +- **Dashboard**: http://localhost:3000 +- **Trading API**: http://localhost:8000 +- **API Docs**: http://localhost:8000/docs +- **pgAdmin**: http://localhost:5050 (if using admin profile) +- **PostgreSQL**: localhost:5432 + +## πŸ“Š Service Management + +### View Logs + +```bash +# All services +docker-compose logs -f + +# Specific service +docker-compose logs -f dashboard +docker-compose logs -f trading-api +docker-compose logs -f postgres + +# Last 50 lines +docker-compose logs --tail=50 trading-api +``` + +### Check Status + +```bash +# List running containers +docker-compose ps + +# Check health +docker-compose ps --format json | jq '.[].Health' + +# Detailed status +docker inspect trading_bot_api +``` + +### Restart Services + +```bash +# Restart all +docker-compose restart + +# Restart specific service +docker-compose restart trading-api +docker-compose restart dashboard +``` + +### Stop Services + +```bash +# Stop all (keeps data) +docker-compose stop + +# Stop and remove containers (keeps data) +docker-compose down + +# Stop and remove everything including volumes (⚠️ deletes data!) +docker-compose down -v +``` + +## πŸ”§ Development & Debugging + +### Access Container Shell + +```bash +# Trading API container +docker exec -it trading_bot_api bash + +# Dashboard container +docker exec -it trading_bot_dashboard sh + +# Database +docker exec -it trading_bot_db psql -U trading_bot -d trading_db +``` + +### Rebuild After Code Changes + +```bash +# Rebuild all +docker-compose build + +# Rebuild specific service +docker-compose build trading-api +docker-compose build dashboard + +# Rebuild and restart +docker-compose up -d --build +``` + +### View Resource Usage + +```bash +# CPU, Memory, Network +docker stats + +# Specific container +docker stats trading_bot_api +``` + +## πŸ—„οΈ Database Management + +### Connect to PostgreSQL + +```bash +# Via Docker +docker exec -it trading_bot_db psql -U trading_bot -d trading_db + +# Via host (if port exposed) +psql -h localhost -p 5432 -U trading_bot -d trading_db +``` + +### Backup Database + +```bash +# Create backup +docker exec trading_bot_db pg_dump -U trading_bot trading_db > backup_$(date +%Y%m%d).sql + +# Restore backup +docker exec -i trading_bot_db psql -U trading_bot -d trading_db < backup_20260206.sql +``` + +### Using pgAdmin + +1. Start with admin profile: + ```bash + docker-compose --profile admin up -d + ``` + +2. Open http://localhost:5050 + +3. Login: + - Email: admin@trading.local + - Password: admin123 + +4. Add Server: + - Host: postgres + - Port: 5432 + - Database: trading_db + - Username: trading_bot + - Password: trading_bot_2026 + +## πŸ” Troubleshooting + +### Container Won't Start + +```bash +# Check logs +docker-compose logs trading-api + +# Check events +docker events --filter container=trading_bot_api + +# Inspect container +docker inspect trading_bot_api +``` + +### Port Already in Use + +```bash +# Find what's using the port +netstat -ano | findstr :3000 +netstat -ano | findstr :8000 + +# Change port in .env +DASHBOARD_PORT=3001 +API_PORT=8001 + +# Restart +docker-compose down +docker-compose up -d +``` + +### API Can't Connect to MT5 + +1. Check MT5 credentials in `.env` +2. Ensure MT5 terminal is running (if running on host) +3. Check container logs: + ```bash + docker-compose logs trading-api | grep MT5 + ``` + +### Dashboard Shows Connection Error + +1. Check if API is healthy: + ```bash + curl http://localhost:8000/api/health + ``` + +2. Check API logs: + ```bash + docker-compose logs trading-api + ``` + +3. Verify API_URL in dashboard: + ```bash + docker exec -it trading_bot_dashboard env | grep API + ``` + +### Database Connection Issues + +```bash +# Check if postgres is healthy +docker-compose ps postgres + +# Test connection +docker exec -it trading_bot_db pg_isready -U trading_bot + +# Check logs +docker-compose logs postgres +``` + +## πŸ” Security Best Practices + +1. **Change Default Passwords** + ```bash + # In .env + DB_PASSWORD=strong_password_here + PGADMIN_PASSWORD=another_strong_password + ``` + +2. **Don't Expose Unnecessary Ports** + ```yaml + # In docker-compose.yml, comment out if not needed: + # ports: + # - "5432:5432" # Only if you need external DB access + ``` + +3. **Use Secrets for Production** + ```bash + # Use Docker secrets instead of .env + docker secret create mt5_password password.txt + ``` + +4. **Restrict Network Access** + ```bash + # Only expose dashboard port + docker-compose up -d postgres trading-api + # Then separately: docker-compose up -d dashboard + ``` + +## πŸ“ˆ Performance Tuning + +### Allocate More Resources + +```yaml +# In docker-compose.yml +services: + trading-api: + deploy: + resources: + limits: + cpus: '2.0' + memory: 2G + reservations: + cpus: '1.0' + memory: 1G +``` + +### Optimize Database + +```bash +# Connect to DB +docker exec -it trading_bot_db psql -U trading_bot -d trading_db + +# Run vacuum +VACUUM ANALYZE; + +# Check table sizes +SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size +FROM pg_tables +WHERE schemaname = 'public' +ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC; +``` + +## πŸ”„ Updates & Maintenance + +### Update Images + +```bash +# Pull latest base images +docker-compose pull + +# Rebuild +docker-compose build --no-cache + +# Restart +docker-compose up -d +``` + +### Clean Up + +```bash +# Remove unused images +docker image prune -a + +# Remove unused volumes (⚠️ careful!) +docker volume prune + +# Remove everything unused +docker system prune -a --volumes +``` + +## πŸ“ Environment Variables Reference + +| Variable | Default | Description | +|----------|---------|-------------| +| `MT5_LOGIN` | - | MT5 account login | +| `MT5_PASSWORD` | - | MT5 account password | +| `MT5_SERVER` | - | MT5 server name | +| `MT5_PATH` | - | Path to MT5 terminal | +| `SYMBOL` | XAUUSD | Trading symbol | +| `CAPITAL` | 10000 | Trading capital | +| `API_PORT` | 8000 | API port on host | +| `DASHBOARD_PORT` | 3000 | Dashboard port on host | +| `DB_PORT` | 5432 | Database port on host | +| `DB_USER` | trading_bot | Database username | +| `DB_PASSWORD` | trading_bot_2026 | Database password | +| `DB_NAME` | trading_db | Database name | +| `TELEGRAM_BOT_TOKEN` | - | Telegram bot token (optional) | +| `TELEGRAM_CHAT_ID` | - | Telegram chat ID (optional) | + +## πŸ“š Additional Resources + +- **Docker Docs**: https://docs.docker.com +- **Docker Compose**: https://docs.docker.com/compose +- **FastAPI**: https://fastapi.tiangolo.com +- **Next.js**: https://nextjs.org + +## πŸ†˜ Getting Help + +If you encounter issues: + +1. Check logs: `docker-compose logs -f` +2. Verify services: `docker-compose ps` +3. Check health: `curl http://localhost:8000/api/health` +4. Review this guide's troubleshooting section +5. Open an issue on GitHub + +--- + +**Last Updated:** Feb 6, 2026 diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..bc9c833 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,24 @@ +# Lightweight API server β€” reads bot_status.json from mounted volume +FROM python:3.11-slim + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 + +WORKDIR /app + +# Install only API dependencies +COPY web-dashboard/api/requirements.txt requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +# Copy only the API code +COPY web-dashboard/api/main.py main.py + +# Create data directory (will be overridden by volume mount) +RUN mkdir -p data + +EXPOSE 8000 + +HEALTHCHECK --interval=30s --timeout=10s --start-period=10s --retries=3 \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" || exit 1 + +CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/QUICK-START.md b/QUICK-START.md new file mode 100644 index 0000000..aeea0c5 --- /dev/null +++ b/QUICK-START.md @@ -0,0 +1,161 @@ +# Quick Start - Tambah Dashboard ke Docker Existing + +## Status Saat Ini + +βœ… **Docker Compose sudah ada** +βœ… **Service `postgres` sudah running** (container: `trading_bot_db`) +βœ… **Service `trading-api` dan `dashboard` sudah didefinisikan** tapi belum di-build + +## πŸš€ Cara Menjalankan + +### 1. Setup Environment (Kalau Belum) + +```cmd +cd "C:\Users\Administrator\Videos\Smart Automatic Trading BOT + AI" + +REM Copy environment template kalau belum ada +copy .env.docker.example .env + +REM Edit dengan MT5 credentials Anda +notepad .env +``` + +Pastikan isi `.env`: +```env +MT5_LOGIN=your_login +MT5_PASSWORD=your_password +MT5_SERVER=your_server +MT5_PATH=C:/Program Files/MetaTrader 5/terminal64.exe + +SYMBOL=XAUUSD +CAPITAL=10000 +``` + +### 2. Build Services Baru + +```cmd +REM Build trading-api dan dashboard +docker-compose build trading-api dashboard +``` + +Ini akan: +- Build Dockerfile untuk Python API +- Build Dockerfile untuk Next.js Dashboard +- Tidak ganggu database yang sudah running + +### 3. Start Services Baru + +```cmd +REM Start trading-api dan dashboard +docker-compose up -d trading-api dashboard +``` + +### 4. Check Status + +```cmd +docker-compose ps +``` + +Output akan menunjukkan: +``` +NAME STATUS PORTS +trading_bot_db Up (healthy) 0.0.0.0:5432->5432/tcp +trading_bot_api Up (healthy) 0.0.0.0:8000->8000/tcp +trading_bot_dashboard Up (healthy) 0.0.0.0:3000->3000/tcp +``` + +### 5. Akses Dashboard + +Buka browser: +- **Dashboard:** http://localhost:3000 +- **API:** http://localhost:8000 +- **API Docs:** http://localhost:8000/docs + +## πŸ“‹ Commands Penting + +```cmd +# Lihat logs +docker-compose logs -f dashboard +docker-compose logs -f trading-api + +# Restart service +docker-compose restart trading-api +docker-compose restart dashboard + +# Stop service +docker-compose stop trading-api dashboard + +# Start lagi +docker-compose up -d trading-api dashboard + +# Rebuild setelah update code +docker-compose build trading-api dashboard +docker-compose up -d trading-api dashboard +``` + +## πŸ” Troubleshooting + +### Build Error + +```cmd +# Clean build +docker-compose build --no-cache trading-api dashboard +``` + +### Service Tidak Start + +```cmd +# Check logs +docker-compose logs trading-api +docker-compose logs dashboard + +# Check health +curl http://localhost:8000/api/health +curl http://localhost:3000 +``` + +### Port Conflict + +Edit `.env`: +```env +API_PORT=8001 +DASHBOARD_PORT=3001 +``` + +Lalu restart: +```cmd +docker-compose down trading-api dashboard +docker-compose up -d trading-api dashboard +``` + +## ⚑ One-Liner (All in One) + +```cmd +cd "C:\Users\Administrator\Videos\Smart Automatic Trading BOT + AI" && docker-compose build trading-api dashboard && docker-compose up -d trading-api dashboard && docker-compose ps +``` + +## πŸ“Š Arsitektur + +``` +Docker Compose Project: "smart-automatic-trading-bot-ai" +β”œβ”€β”€ postgres (RUNNING) βœ… +β”‚ └── trading_bot_db +β”œβ”€β”€ trading-api (BUILD & START) ⚑ +β”‚ └── trading_bot_api +└── dashboard (BUILD & START) ⚑ + └── trading_bot_dashboard +``` + +## βœ… Checklist + +- [ ] Copy `.env.docker.example` ke `.env` +- [ ] Edit `.env` dengan MT5 credentials +- [ ] Run: `docker-compose build trading-api dashboard` +- [ ] Run: `docker-compose up -d trading-api dashboard` +- [ ] Check: `docker-compose ps` +- [ ] Open: http://localhost:3000 +- [ ] Test API: http://localhost:8000/api/health + +--- + +**That's it!** Simple kan? πŸŽ‰ diff --git a/README.md b/README.md index 2b95f5b..970cdff 100644 --- a/README.md +++ b/README.md @@ -119,13 +119,44 @@ xaubot-ai/ ## Installation -### Prerequisites +### 🐳 Docker Deployment (Recommended) +**Quick Start:** + +```bash +# 1. Clone the repository +git clone https://github.com/GifariKemal/xaubot-ai.git +cd xaubot-ai + +# 2. Configure environment +cp .env.docker.example .env +# Edit .env with your MT5 credentials + +# 3. Start all services (Windows) +docker-start.bat + +# 3. Start all services (Linux/Mac) +./docker-start.sh +``` + +**Services will be available at:** +- πŸ“Š Dashboard: http://localhost:3000 +- πŸ”Œ API: http://localhost:8000 +- πŸ“š API Docs: http://localhost:8000/docs +- πŸ—„οΈ Database: localhost:5432 + +**Full Docker documentation:** See [DOCKER.md](DOCKER.md) + +--- + +### 🐍 Manual Installation + +**Prerequisites:** - Python 3.11+ - MetaTrader 5 terminal (Windows) - PostgreSQL (optional, for trade logging) -### Setup +**Setup:** ```bash # Clone the repository diff --git a/SIMPLE-START.md b/SIMPLE-START.md new file mode 100644 index 0000000..05617ad --- /dev/null +++ b/SIMPLE-START.md @@ -0,0 +1,170 @@ +# Simple Start Guide - XAUBot AI Dashboard + +## 🎯 Cara Tercepat (1 Command) + +```cmd +start-all.bat +``` + +Script ini akan: +1. βœ… Check database Docker container +2. πŸš€ Start Trading API di http://localhost:8000 +3. πŸš€ Start Dashboard di http://localhost:3000 + +Dua window akan terbuka otomatis! + +## πŸ“‹ Manual Start (Jika Perlu) + +### Option 1: Start Semua Sekaligus +```cmd +start-all.bat +``` + +### Option 2: Start Satu-satu + +**Terminal 1: API** +```cmd +start-api.bat +``` + +**Terminal 2: Dashboard** +```cmd +start-dashboard.bat +``` + +## βœ… Pre-requisites + +### 1. Database (Docker) +Database harus sudah running: +```cmd +# Check status +docker ps | findstr trading_bot_db + +# Start jika belum running +docker-compose up -d postgres +``` + +### 2. Python Environment +- Python 3.11+ installed +- Virtual environment akan dibuat otomatis + +### 3. Node.js +- Node.js 18+ installed +- npm dependencies akan diinstall otomatis + +## 🌐 Access Points + +Setelah start: +- **Dashboard:** http://localhost:3000 +- **API:** http://localhost:8000 +- **API Docs:** http://localhost:8000/docs +- **Health Check:** http://localhost:8000/api/health +- **Status:** http://localhost:8000/api/status + +## πŸ›‘ Stop Services + +Close kedua command windows atau tekan `Ctrl+C` di masing-masing window. + +## πŸ” Troubleshooting + +### API Error: "Module not found" + +Install dependencies: +```cmd +pip install -r requirements.txt +``` + +### Dashboard Error: "Module not found" + +Install dependencies: +```cmd +cd web-dashboard +npm install +``` + +### Port Already in Use + +**Change API Port:** +Edit `web-dashboard/api/main.py` line terakhir: +```python +uvicorn.run(app, host="0.0.0.0", port=8001) # Change 8000 to 8001 +``` + +**Change Dashboard Port:** +Edit `web-dashboard/.env.local`: +``` +NEXT_PUBLIC_API_URL=http://localhost:8001 +``` + +Then start dashboard on different port: +```cmd +cd web-dashboard +set PORT=3001 && npm run dev +``` + +### Database Not Running + +Start database: +```cmd +docker-compose up -d postgres + +# Check status +docker ps +``` + +## πŸ“Š Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Windows Host Machine β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β”‚ +β”‚ πŸ“Š Dashboard (Port 3000) β”‚ +β”‚ npm run dev β”‚ +β”‚ ↓ HTTP β”‚ +β”‚ πŸ”Œ API (Port 8000) β”‚ +β”‚ uvicorn main:app β”‚ +β”‚ ↓ PostgreSQL β”‚ +β”‚ πŸ—„οΈ Database (Docker) β”‚ +β”‚ trading_bot_db β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## 🎨 Features + +Dashboard akan menampilkan: +- ⏰ Real-time XAUUSD price +- πŸ’° Account balance & equity +- πŸ“ˆ Price history chart +- 🎯 Trading signals (SMC + ML) +- 🌊 Market regime +- ⚠️ Risk status +- πŸ“‹ Open positions +- πŸ“ Activity logs + +## πŸ’‘ Tips + +1. **Auto-start Database:** + Tambahkan Docker Desktop ke Windows startup + +2. **Keep API Running:** + Minimize command windows, jangan close + +3. **Monitor Logs:** + Lihat output di command windows untuk debug + +4. **Quick Restart:** + Close windows dan run `start-all.bat` lagi + +## πŸ“ Files + +``` +start-all.bat # Start API + Dashboard +start-api.bat # Start API only +start-dashboard.bat # Start Dashboard only +``` + +--- + +**Super Simple!** Tinggal double-click `start-all.bat` πŸŽ‰ diff --git a/backtests/backtest_26_sell_improvement.py b/backtests/backtest_26_sell_improvement.py new file mode 100644 index 0000000..4045acb --- /dev/null +++ b/backtests/backtest_26_sell_improvement.py @@ -0,0 +1,1010 @@ +""" +Backtest #26 β€” SELL Improvement +================================ +Base: #24B (19B+20B+22D) β€” 739 trades, 80.4% WR, $2,235, Sharpe 2.87 + +Problem: SELL WR 76.5% vs BUY 83.1% (6.6pp gap) +Goal: Improve SELL quality without reducing BUY performance + +Configs: + A: Higher SELL confidence threshold (>=0.55 vs no filter) + B: SELL requires bearish BOS (not just CHoCH) + C: Asymmetric RR (SELL 1:1.2, BUY 1:1.5) + D: A+B combined (higher confidence + bearish BOS) + E: A+B+C combined (all three improvements) + +Usage: + python backtests/backtest_26_sell_improvement.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + sell_filtered: int = 0 # NEW: track SELL-specific filters + + +# ─── SELL Improvement Backtest ────────────────────────────────── + +class SellImprovementBacktest: + """#24B base + SELL-specific improvements.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base settings + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # ═══ #26 SELL IMPROVEMENT PARAMS ═══ + sell_min_confidence: float = 0.0, # Min confidence for SELL (0 = no filter) + sell_require_bos: bool = False, # Require bearish BOS for SELL + sell_rr_multiplier: float = 1.0, # Multiply SELL RR (0.8 = 1:1.2 instead of 1:1.5) + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + + # #26 SELL params + self.sell_min_confidence = sell_min_confidence + self.sell_require_bos = sell_require_bos + self.sell_rr_multiplier = sell_rr_multiplier + + config = get_config() + self.smc = SMCAnalyzer( + swing_length=config.smc.swing_length, + ob_lookback=config.smc.ob_lookback, + ) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2260000 + + # ── Session filter (#19B: skip Tokyo-London) ── + + def _get_session_from_time(self, dt: datetime) -> Tuple[str, bool, float]: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt: datetime) -> float: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt: datetime) -> bool: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30: + return True + return False + + # ── Lot sizing (synced) ── + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + # ── Full exit simulation (#24B base) ── + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ) -> Tuple[float, float, ExitReason, int, float]: + pip_value = 10 + + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + # ═══ #22D: ATR-ADAPTIVE exit levels ═══ + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + profit_history = [] + price_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = (close - entry_price) / 0.1 + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = (entry_price - close) / 0.1 + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + price_history.append(close) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + + profit_growing = momentum > 0 + + # ════════════════════════════════════════════════ + # A) SmartPositionManager (ATR-ADAPTIVE) + # ════════════════════════════════════════════════ + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven (ATR-ADAPTIVE) + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + 2 + else: + current_sl = entry_price - 2 + breakeven_moved = True + + # A.2 Trailing SL (ATR-ADAPTIVE) + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5: + if i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "NEUTRAL" + if ma_fast > ma_slow * 1.001: + trend = "BULLISH" + elif ma_fast < ma_slow * 0.999: + trend = "BEARISH" + + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + + if cached_ml_confidence > 0.75: + if direction == "BUY" and cached_ml_signal == "SELL": + should_exit = True + urgency += 2 + elif direction == "SELL" and cached_ml_signal == "BUY": + should_exit = True + urgency += 2 + + if rsi_val: + if rsi_val > 75 and direction == "BUY": + should_exit = True + urgency += 2 + elif rsi_val < 25 and direction == "SELL": + should_exit = True + urgency += 2 + + if direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH": + should_exit = True + urgency += 3 + elif direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH": + should_exit = True + urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + elif current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # ════════════════════════════════════════════════ + # B) SmartRiskManager (#20B early cut tuned) + # ════════════════════════════════════════════════ + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = ( + (direction == "BUY" and cached_ml_signal == "SELL") or + (direction == "SELL" and cached_ml_signal == "BUY") + ) + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut (#20B: momentum < -50 instead of -30) + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal + is_ml_reversal = False + if direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold: + is_ml_reversal = True + reversal_warnings += 1 + elif direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold: + is_ml_reversal = True + reversal_warnings += 1 + + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall detection + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # ════════════════════════════════════════════════ + # C) Time-based exit + # ════════════════════════════════════════════════ + + if bars_since_entry >= 16: + if current_profit < 5 and not profit_growing: + if current_profit >= 0: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + elif current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + if bars_since_entry >= 24: + if current_profit < 10 or not profit_growing: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if direction == "BUY" and mom < -reversal_momentum_threshold: + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + elif direction == "SELL" and mom > reversal_momentum_threshold: + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + if direction == "BUY": + pips = (final_price - entry_price) / 0.1 + else: + pips = (entry_price - final_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + # ── Main run ── + + def run(self, df, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df.columns] + + times = df["time"].to_list() + start_idx = next((i for i, t in enumerate(times) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times) if t > end_date), len(df) - 100) if end_date else len(df) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #26 SELL filters: min_conf={self.sell_min_confidence}, require_bos={self.sell_require_bos}, rr_mult={self.sell_rr_multiplier}") + print(f" Date range: {times[start_idx]} to {times[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times[i] + + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + # ═══ Detect SMC components (direction-aware for #26) ═══ + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + # Direction-specific BOS check for SELL filter + has_bearish_bos = -1 in recent_bos + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = ( + (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + ) + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + # ═══ #26 SELL FILTERS ═══ + if smc_signal.signal_type == "SELL": + # Filter A: Min confidence for SELL + if self.sell_min_confidence > 0 and confidence < self.sell_min_confidence: + stats.sell_filtered += 1 + continue + + # Filter B: Require bearish BOS for SELL + if self.sell_require_bos and not has_bearish_bos: + stats.sell_filtered += 1 + continue + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + # ═══ #26 Filter C: Asymmetric RR for SELL ═══ + if smc_signal.signal_type == "SELL" and self.sell_rr_multiplier != 1.0: + # Tighter TP for SELL (e.g., 0.8 = 1:1.2 instead of 1:1.5) + new_rr = rr * self.sell_rr_multiplier + take_profit_price = entry_price - (risk * new_rr) + rr = new_rr + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times[exit_idx] if exit_idx < len(times) else times[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + ) + stats.trades.append(trade) + + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #26 SELL Improvement") + print("Base: #24B | Modified: SELL-specific filters") + print("=" * 70) + + config = get_config() + mt5 = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df = mt5.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df) == 0: + print("ERROR: No data") + mt5.disconnect() + return + + print(f" Received {len(df)} bars") + times = df["time"].to_list() + print(f" Data range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df = features.calculate_all(df, include_ml_features=True) + df = smc.calculate_all(df) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df = regime_detector.predict(df) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + print(" Indicators calculated") + + baseline_24b_pnl = 2235.0 # #24B result + + # ═══ CONFIGS ═══ + configs = [ + # (name, sell_min_conf, sell_require_bos, sell_rr_mult) + ("A: SELL conf>=0.55", 0.55, False, 1.0), + ("B: SELL require BOS", 0.0, True, 1.0), + ("C: SELL RR 1:1.2", 0.0, False, 0.8), + ("D: A+B (conf+BOS)", 0.55, True, 1.0), + ("E: A+B+C (all)", 0.55, True, 0.8), + ] + + all_results = [] + + for cfg_name, sell_conf, sell_bos, sell_rr in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = SellImprovementBacktest( + sell_min_confidence=sell_conf, + sell_require_bos=sell_bos, + sell_rr_multiplier=sell_rr, + ) + stats = bt.run(df=df, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_24b_pnl + + # Direction breakdown + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" SELL filtered: {stats.sell_filtered}") + print(f" BUY: {len(buy_trades)} trades, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)} trades, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" vs #24B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, len(buy_trades), buy_wr, buy_pnl, len(sell_trades), sell_wr, sell_pnl)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#26 SELL IMPROVEMENT β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'Filt':>5} {'vs #24B':>10}") + print(f" {'-' * 85}") + print(f" {'#1 BASELINE':<25} {'686':>6} {'72.2%':>6} {'$1,449.86':>10} {'5.4%':>6} {'1.98':>7} {'1.52':>5} {'β€”':>5} {'β€”':>10}") + print(f" {'#24B (base)':<25} {'739':>6} {'80.4%':>6} {'$2,235.00':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'β€”':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, _, _, _, _, _, _ in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {stats.sell_filtered:>5} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + best_data = None + for entry in all_results: + cfg_name, stats, net_pnl = entry[0], entry[1], entry[2] + if net_pnl > best_pnl: + best_pnl = net_pnl + best_name = cfg_name + best_stats = stats + best_data = entry + + print(f"\n Best config: {best_name}") + + # Direction breakdown for best + print(f"\n Direction (best config):") + _, _, _, _, buy_n, buy_wr, buy_pnl, sell_n, sell_wr, sell_pnl = best_data + print(f" BUY: {buy_n} trades, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {sell_n} trades, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + + # Direction comparison table + print(f"\n SELL Performance Comparison:") + print(f" {'Config':<25} {'SELL Trades':>11} {'SELL WR':>8} {'SELL PnL':>10} {'BUY WR':>7}") + print(f" {'-' * 65}") + print(f" {'#24B (base)':<25} {'~125':>11} {'76.5%':>8} {'$496':>10} {'83.1%':>7}") + for cfg_name, stats, net_pnl, diff, buy_n, buy_wr, buy_pnl, sell_n, sell_wr, sell_pnl in all_results: + print(f" {cfg_name:<25} {sell_n:>11} {sell_wr:>7.1f}% ${sell_pnl:>9,.2f} {buy_wr:>6.1f}%") + + # Exit reasons for best + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "26_sell_improvement_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"sell_improve_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#26 SELL Improvement Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #24B (739 trades, 80.4% WR, $2,235)\n\n") + for cfg_name, stats, net_pnl, diff, buy_n, buy_wr, buy_pnl, sell_n, sell_wr, sell_pnl in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"SELL filtered: {stats.sell_filtered}, " + f"BUY: {buy_n}@{buy_wr:.1f}%/${buy_pnl:,.2f}, " + f"SELL: {sell_n}@{sell_wr:.1f}%/${sell_pnl:,.2f}, " + f"vs #24B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"sell_improve_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_27_regime_aware_entry.py b/backtests/backtest_27_regime_aware_entry.py new file mode 100644 index 0000000..b305055 --- /dev/null +++ b/backtests/backtest_27_regime_aware_entry.py @@ -0,0 +1,1004 @@ +""" +Backtest #27 β€” Regime-Aware Entry +================================== +Base: #24B (19B+20B+22D) β€” 739 trades, 80.4% WR, $2,235, Sharpe 2.87 + +HMM regime (low/medium/high vol) is only used for exits currently. +Goal: Use regime to FILTER entries and improve quality. + +Configs: + A: Regime-conditional confidence thresholds + low_vol=0.50, medium=0.55, high_vol=0.70 + B: Skip SELL in low volatility (gold grinds up in calm markets, SELL fails) + C: Skip ALL trades in high volatility (only trade low+medium vol) + D: A+B combined (regime thresholds + skip SELL in low vol) + E: Regime-direction matrix (skip worst combos: SELL in low vol, BUY in high vol crisis) + +Usage: + python backtests/backtest_27_regime_aware_entry.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + regime_filtered: int = 0 + + +# ─── Regime-Aware Entry Backtest ────────────────────────────── + +class RegimeAwareBacktest: + """#24B base + regime-aware entry filtering.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base settings + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # ═══ #27 REGIME-AWARE ENTRY PARAMS ═══ + regime_conf_thresholds: Dict[str, float] = None, # {regime: min_confidence} + skip_sell_in_low_vol: bool = False, + skip_all_in_high_vol: bool = False, + regime_direction_filter: Dict[str, List[str]] = None, # {regime: [blocked_directions]} + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + + # #27 Regime params + self.regime_conf_thresholds = regime_conf_thresholds or {} + self.skip_sell_in_low_vol = skip_sell_in_low_vol + self.skip_all_in_high_vol = skip_all_in_high_vol + self.regime_direction_filter = regime_direction_filter or {} + + config = get_config() + self.smc = SMCAnalyzer( + swing_length=config.smc.swing_length, + ob_lookback=config.smc.ob_lookback, + ) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2270000 + + # ── Session filter (#19B) ── + + def _get_session_from_time(self, dt: datetime) -> Tuple[str, bool, float]: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt: datetime) -> float: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt: datetime) -> bool: + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30: + return True + return False + + # ── Lot sizing ── + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + # ── Exit simulation (identical to #24B) ── + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ) -> Tuple[float, float, ExitReason, int, float]: + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + current_sl = entry_price + (2 if direction == "BUY" else -2) + breakeven_moved = True + + # A.2 Trailing SL + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # C) Time-based + if bars_since_entry >= 16 and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + # ── Main run ── + + def run(self, df, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df.columns] + + times = df["time"].to_list() + start_idx = next((i for i, t in enumerate(times) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times) if t > end_date), len(df) - 100) if end_date else len(df) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #27 Regime filters: thresholds={self.regime_conf_thresholds}, skip_sell_low={self.skip_sell_in_low_vol}, skip_high={self.skip_all_in_high_vol}, dir_filter={self.regime_direction_filter}") + print(f" Date range: {times[start_idx]} to {times[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + # ═══ #27 REGIME FILTER: Skip all in high vol ═══ + if self.skip_all_in_high_vol and regime == "high_volatility": + stats.regime_filtered += 1 + continue + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = ( + (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + ) + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + # ═══ #27 REGIME FILTERS ═══ + + # Filter A: Regime-conditional confidence thresholds + if self.regime_conf_thresholds: + min_conf = self.regime_conf_thresholds.get(regime, 0.0) + if min_conf > 0 and confidence < min_conf: + stats.regime_filtered += 1 + continue + + # Filter B: Skip SELL in low volatility + if self.skip_sell_in_low_vol and regime == "low_volatility" and smc_signal.signal_type == "SELL": + stats.regime_filtered += 1 + continue + + # Filter E: Regime-direction matrix + if self.regime_direction_filter: + blocked_dirs = self.regime_direction_filter.get(regime, []) + if smc_signal.signal_type in blocked_dirs: + stats.regime_filtered += 1 + continue + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times[exit_idx] if exit_idx < len(times) else times[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + ) + stats.trades.append(trade) + + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #27 Regime-Aware Entry") + print("Base: #24B | Modified: Regime-based entry filtering") + print("=" * 70) + + config = get_config() + mt5 = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df = mt5.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df) == 0: + print("ERROR: No data") + mt5.disconnect() + return + + print(f" Received {len(df)} bars") + times = df["time"].to_list() + print(f" Data range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df = features.calculate_all(df, include_ml_features=True) + df = smc.calculate_all(df) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df = regime_detector.predict(df) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + print(" Indicators calculated") + + # ── First: analyze regime distribution in #24B baseline ── + print("\n Regime distribution analysis (pre-backtest)...") + regime_col = df["regime_name"].to_list() if "regime_name" in df.columns else [] + if regime_col: + from collections import Counter + regime_counts = Counter(r for r in regime_col if r is not None) + total_bars = sum(regime_counts.values()) + for r, c in sorted(regime_counts.items(), key=lambda x: -x[1]): + print(f" {r}: {c} bars ({c/total_bars*100:.1f}%)") + + baseline_24b_pnl = 2235.0 + + # ═══ CONFIGS ═══ + configs = [ + { + "name": "A: Regime thresholds", + "regime_conf_thresholds": {"low_volatility": 0.50, "medium_volatility": 0.55, "high_volatility": 0.70}, + "skip_sell_in_low_vol": False, + "skip_all_in_high_vol": False, + "regime_direction_filter": {}, + }, + { + "name": "B: Skip SELL in low vol", + "regime_conf_thresholds": {}, + "skip_sell_in_low_vol": True, + "skip_all_in_high_vol": False, + "regime_direction_filter": {}, + }, + { + "name": "C: Skip all high vol", + "regime_conf_thresholds": {}, + "skip_sell_in_low_vol": False, + "skip_all_in_high_vol": True, + "regime_direction_filter": {}, + }, + { + "name": "D: A+B combined", + "regime_conf_thresholds": {"low_volatility": 0.50, "medium_volatility": 0.55, "high_volatility": 0.70}, + "skip_sell_in_low_vol": True, + "skip_all_in_high_vol": False, + "regime_direction_filter": {}, + }, + { + "name": "E: Direction matrix", + "regime_conf_thresholds": {}, + "skip_sell_in_low_vol": False, + "skip_all_in_high_vol": False, + "regime_direction_filter": { + "low_volatility": ["SELL"], # No SELL in calm markets + "high_volatility": ["SELL"], # No SELL in volatile markets (whipsaw) + }, + }, + ] + + all_results = [] + + for cfg in configs: + cfg_name = cfg["name"] + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = RegimeAwareBacktest( + regime_conf_thresholds=cfg["regime_conf_thresholds"], + skip_sell_in_low_vol=cfg["skip_sell_in_low_vol"], + skip_all_in_high_vol=cfg["skip_all_in_high_vol"], + regime_direction_filter=cfg["regime_direction_filter"], + ) + stats = bt.run(df=df, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_24b_pnl + + # Direction breakdown + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + # Regime breakdown + regime_stats = {} + for t in stats.trades: + r = t.regime + if r not in regime_stats: + regime_stats[r] = {"w": 0, "l": 0, "p": 0.0} + if t.result == TradeResult.WIN: + regime_stats[r]["w"] += 1 + else: + regime_stats[r]["l"] += 1 + regime_stats[r]["p"] += t.profit_usd + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" Regime filtered: {stats.regime_filtered}") + print(f" BUY: {len(buy_trades)} trades, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)} trades, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" Per regime:") + for r, d in sorted(regime_stats.items(), key=lambda x: -x[1]["p"]): + total = d["w"] + d["l"] + wr = d["w"] / total * 100 if total > 0 else 0 + print(f" {r:<20}: {total:>3} trades, {wr:>5.1f}% WR, ${d['p']:>8,.2f}") + print(f" vs #24B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, len(buy_trades), buy_wr, buy_pnl, len(sell_trades), sell_wr, sell_pnl, regime_stats)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#27 REGIME-AWARE ENTRY β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'Filt':>5} {'vs #24B':>10}") + print(f" {'-' * 85}") + print(f" {'#1 BASELINE':<25} {'686':>6} {'72.2%':>6} {'$1,449.86':>10} {'5.4%':>6} {'1.98':>7} {'1.52':>5} {'β€”':>5} {'β€”':>10}") + print(f" {'#24B (base)':<25} {'739':>6} {'80.4%':>6} {'$2,235.00':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'β€”':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, _, _, _, _, _, _, _ in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {stats.regime_filtered:>5} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + best_data = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + best_data = entry + + print(f"\n Best config: {best_name}") + + # Exit reasons for best + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "27_regime_aware_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"regime_aware_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#27 Regime-Aware Entry Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #24B (739 trades, 80.4% WR, $2,235)\n\n") + for cfg_name, stats, net_pnl, diff, buy_n, buy_wr, buy_pnl, sell_n, sell_wr, sell_pnl, regime_stats in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"Filtered: {stats.regime_filtered}, " + f"BUY: {buy_n}@{buy_wr:.1f}%, SELL: {sell_n}@{sell_wr:.1f}%, " + f"vs #24B: ${diff:+,.2f}\n") + for r, d in sorted(regime_stats.items(), key=lambda x: -x[1]["p"]): + total = d["w"] + d["l"] + wr = d["w"] / total * 100 if total > 0 else 0 + f.write(f" {r}: {total} trades, {wr:.1f}% WR, ${d['p']:,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"regime_aware_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_28_smart_breakeven.py b/backtests/backtest_28_smart_breakeven.py new file mode 100644 index 0000000..1422c42 --- /dev/null +++ b/backtests/backtest_28_smart_breakeven.py @@ -0,0 +1,919 @@ +""" +Backtest #28 β€” Smart Breakeven + Loss Reduction +================================================= +Base: #24B (19B+20B+22D) β€” 739 trades, 80.4% WR, $2,235, Sharpe 2.87 + +Problem: 31.9% of exits are breakeven β€” trades that went profitable then +reversed back to entry. These are dead weight (~$0 profit each). +Also: 10.4% early_cut + 2.7% max_loss = preventable losses. + +Configs: + A: Smart BE β€” lock small profit at entry + 0.3x ATR (not exact entry + $2) + B: Smart BE β€” lock at entry + 0.5x ATR (more aggressive profit lock) + C: First-candle adverse exit β€” if bar 1 goes against >0.5x ATR, cut immediately + D: A+C combined (smart BE + first-candle exit) + E: B+C combined (aggressive BE + first-candle exit) + +Usage: + python backtests/backtest_28_smart_breakeven.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + FIRST_CANDLE_CUT = "first_candle_cut" # NEW + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + first_candle_cuts: int = 0 + + +# ─── Smart Breakeven Backtest ───────────────────────────────── + +class SmartBreakevenBacktest: + """#24B base + smart breakeven and first-candle adverse exit.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # ═══ #28 SMART BREAKEVEN PARAMS ═══ + be_profit_lock_atr_mult: float = 0.0, # Lock profit at entry + X * ATR (0 = use $2 like #24B) + first_candle_cut_atr_mult: float = 0.0, # Cut if bar 1 adverse > X * ATR (0 = disabled) + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + + # #28 params + self.be_profit_lock_atr_mult = be_profit_lock_atr_mult + self.first_candle_cut_atr_mult = first_candle_cut_atr_mult + + config = get_config() + self.smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2280000 + + def _get_session_from_time(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + return wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30 + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ): + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + # ═══ #28: Smart breakeven profit lock ═══ + # Instead of entry + $2, lock at entry + (ATR * be_profit_lock_atr_mult) + if self.be_profit_lock_atr_mult > 0: + be_lock_distance = atr * self.be_profit_lock_atr_mult # in price terms + else: + be_lock_distance = 2.0 # Original $2 buffer + + # ═══ #28: First-candle adverse threshold ═══ + first_candle_adverse_threshold = 0.0 + first_candle_cut_triggered = False + if self.first_candle_cut_atr_mult > 0: + first_candle_adverse_threshold = atr * self.first_candle_cut_atr_mult + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + # ═══ #28: FIRST-CANDLE ADVERSE EXIT ═══ + if bars_since_entry == 1 and first_candle_adverse_threshold > 0: + if direction == "BUY": + adverse_move = entry_price - low # How far price went against us + else: + adverse_move = high - entry_price + if adverse_move > first_candle_adverse_threshold: + first_candle_cut_triggered = True + pips = current_pips + return current_profit, pips, ExitReason.FIRST_CANDLE_CUT, i, close + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven (#28: SMART β€” lock profit at entry + ATR*mult) + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + be_lock_distance + else: + current_sl = entry_price - be_lock_distance + breakeven_moved = True + + # A.2 Trailing SL + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # C) Time-based + if bars_since_entry >= 16 and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + # ── Main run (identical to #24B except passes first_candle_cuts) ── + + def run(self, df, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df.columns] + + times = df["time"].to_list() + start_idx = next((i for i, t in enumerate(times) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times) if t > end_date), len(df) - 100) if end_date else len(df) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #28 BE lock ATR mult: {self.be_profit_lock_atr_mult}, First-candle cut ATR mult: {self.first_candle_cut_atr_mult}") + print(f" Date range: {times[start_idx]} to {times[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or \ + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + if exit_reason == ExitReason.FIRST_CANDLE_CUT: + stats.first_candle_cuts += 1 + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times[exit_idx] if exit_idx < len(times) else times[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + ) + stats.trades.append(trade) + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #28 Smart Breakeven + Loss Reduction") + print("Base: #24B | Modified: Smart BE profit lock + first-candle exit") + print("=" * 70) + + config = get_config() + mt5 = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df = mt5.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df) == 0: + print("ERROR: No data") + mt5.disconnect() + return + + print(f" Received {len(df)} bars") + times = df["time"].to_list() + print(f" Data range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df = features.calculate_all(df, include_ml_features=True) + df = smc.calculate_all(df) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df = regime_detector.predict(df) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + print(" Indicators calculated") + + baseline_24b_pnl = 2235.0 + + # ═══ CONFIGS ═══ + configs = [ + # (name, be_profit_lock_atr_mult, first_candle_cut_atr_mult) + ("A: Smart BE 0.3x ATR", 0.3, 0.0), # Lock small profit + ("B: Smart BE 0.5x ATR", 0.5, 0.0), # Lock more profit + ("C: First-candle 0.5x ATR", 0.0, 0.5), # Cut if bar 1 adverse + ("D: A+C (0.3x BE + FC)", 0.3, 0.5), # Combined + ("E: B+C (0.5x BE + FC)", 0.5, 0.5), # Combined aggressive + ] + + all_results = [] + + for cfg_name, be_lock, fc_cut in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = SmartBreakevenBacktest( + be_profit_lock_atr_mult=be_lock, + first_candle_cut_atr_mult=fc_cut, + ) + stats = bt.run(df=df, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_24b_pnl + + # Count BE exits and their avg profit + be_exits = [t for t in stats.trades if t.exit_reason == ExitReason.BREAKEVEN_EXIT] + be_avg_profit = np.mean([t.profit_usd for t in be_exits]) if be_exits else 0 + be_wins = sum(1 for t in be_exits if t.profit_usd > 0) + + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" BE exits: {len(be_exits)} (avg ${be_avg_profit:.2f}, {be_wins} profitable)") + print(f" First-candle cuts: {stats.first_candle_cuts}") + print(f" BUY: {len(buy_trades)}, {buy_wr:.1f}% WR | SELL: {len(sell_trades)}, {sell_wr:.1f}% WR") + print(f" vs #24B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, len(be_exits), be_avg_profit, be_wins, stats.first_candle_cuts)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#28 SMART BREAKEVEN β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'BE#':>4} {'BE$':>6} {'FC#':>4} {'vs #24B':>10}") + print(f" {'-' * 100}") + print(f" {'#24B (base)':<25} {'739':>6} {'80.4%':>6} {'$2,235':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'236':>4} {'$0.0':>6} {'β€”':>4} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, be_n, be_avg, be_w, fc_n in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {be_n:>4} ${be_avg:>5.1f} {fc_n:>4} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + + print(f"\n Best config: {best_name}") + + # Exit reasons for best + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "28_smart_breakeven_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"smart_be_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#28 Smart Breakeven Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #24B (739 trades, 80.4% WR, $2,235)\n\n") + for cfg_name, stats, net_pnl, diff, be_n, be_avg, be_w, fc_n in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"BE: {be_n} (avg ${be_avg:.2f}, {be_w} wins), FC cuts: {fc_n}, " + f"vs #24B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"smart_be_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_29_confluence_scoring.py b/backtests/backtest_29_confluence_scoring.py new file mode 100644 index 0000000..9fe7b53 --- /dev/null +++ b/backtests/backtest_29_confluence_scoring.py @@ -0,0 +1,953 @@ +""" +Backtest #29 β€” Confluence Scoring +================================== +Base: #28B (Smart BE 0.5x ATR) β€” 741 trades, 79.8% WR, $2,464, Sharpe 3.23 + +Idea: Require minimum number of SMC confirmations (BOS, CHoCH, FVG, OB) +before entering. Currently any single SMC signal triggers entry. By requiring +more confirmations, we filter weak signals and keep only high-quality setups. + +Configs: + A: Min 2 SMC elements (any 2 of BOS/CHoCH/FVG/OB) + B: Min confidence >= 0.55 (threshold filter) + C: Min confidence >= 0.60 + D: A+B combined (2 elements + conf >= 0.55) + E: Min 3 SMC elements (BOS/CHoCH + FVG or OB) + +Usage: + python backtests/backtest_29_confluence_scoring.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + smc_element_count: int = 0 # NEW: count of SMC elements + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + confluence_filtered: int = 0 # NEW + + +# ─── Confluence Scoring Backtest ───────────────────────────── + +class ConfluenceScoringBacktest: + """#28B base + confluence scoring entry filter.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # #28B: Smart breakeven + be_profit_lock_atr_mult: float = 0.5, + # ═══ #29 CONFLUENCE SCORING PARAMS ═══ + min_smc_elements: int = 0, # Minimum number of SMC elements (BOS, CHoCH, FVG, OB) + min_confidence: float = 0.0, # Minimum confidence threshold + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + self.be_profit_lock_atr_mult = be_profit_lock_atr_mult + + # #29 params + self.min_smc_elements = min_smc_elements + self.min_confidence = min_confidence + + config = get_config() + self.smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2290000 + + def _get_session_from_time(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + return wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30 + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ): + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + # #28B: Smart breakeven profit lock + if self.be_profit_lock_atr_mult > 0: + be_lock_distance = atr * self.be_profit_lock_atr_mult + else: + be_lock_distance = 2.0 + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven (#28B: Smart β€” lock profit at entry + ATR*0.5) + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + be_lock_distance + else: + current_sl = entry_price - be_lock_distance + breakeven_moved = True + + # A.2 Trailing SL + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # C) Time-based + if bars_since_entry >= 16 and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + # ── Main run ── + + def run(self, df, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df.columns] + + times = df["time"].to_list() + start_idx = next((i for i, t in enumerate(times) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times) if t > end_date), len(df) - 100) if end_date else len(df) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #29 Min SMC elements: {self.min_smc_elements}, Min confidence: {self.min_confidence}") + print(f" Date range: {times[start_idx]} to {times[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + # ═══ SMC ELEMENT DETECTION ═══ + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + # ═══ #29: COUNT SMC ELEMENTS ═══ + smc_element_count = sum([has_bos, has_choch, has_fvg, has_ob]) + + # ═══ #29: CONFLUENCE FILTER ═══ + if self.min_smc_elements > 0 and smc_element_count < self.min_smc_elements: + stats.confluence_filtered += 1 + continue + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or \ + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + # ═══ #29: CONFIDENCE FILTER ═══ + if self.min_confidence > 0 and confidence < self.min_confidence: + stats.confluence_filtered += 1 + continue + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times[exit_idx] if exit_idx < len(times) else times[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + smc_element_count=smc_element_count, + ) + stats.trades.append(trade) + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #29 Confluence Scoring") + print("Base: #28B (Smart BE 0.5x ATR) | Modified: Confluence entry filters") + print("=" * 70) + + config = get_config() + mt5 = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df = mt5.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df) == 0: + print("ERROR: No data") + mt5.disconnect() + return + + print(f" Received {len(df)} bars") + times = df["time"].to_list() + print(f" Data range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df = features.calculate_all(df, include_ml_features=True) + df = smc.calculate_all(df) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df = regime_detector.predict(df) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + print(" Indicators calculated") + + # ═══ ANALYZE ELEMENT DISTRIBUTION BEFORE BACKTEST ═══ + print("\n SMC Element Distribution (pre-analysis)...") + if "bos" in df.columns: + bos_count = df.filter(pl.col("bos") != 0).height + choch_count = df.filter(pl.col("choch") != 0).height if "choch" in df.columns else 0 + fvg_bull = df.filter(pl.col("is_fvg_bull") == True).height if "is_fvg_bull" in df.columns else 0 + fvg_bear = df.filter(pl.col("is_fvg_bear") == True).height if "is_fvg_bear" in df.columns else 0 + ob_count = df.filter(pl.col("ob") != 0).height if "ob" in df.columns else 0 + print(f" BOS: {bos_count} bars | CHoCH: {choch_count} bars | FVG: {fvg_bull + fvg_bear} bars | OB: {ob_count} bars") + + baseline_28b_pnl = 2463.80 + + # ═══ CONFIGS ═══ + configs = [ + # (name, min_smc_elements, min_confidence) + ("A: Min 2 SMC elements", 2, 0.0), # Require 2 of BOS/CHoCH/FVG/OB + ("B: Min conf >= 0.55", 0, 0.55), # Confidence threshold + ("C: Min conf >= 0.60", 0, 0.60), # Higher confidence + ("D: 2 elem + conf>=0.55", 2, 0.55), # Combined + ("E: Min 3 SMC elements", 3, 0.0), # Strict: 3 out of 4 elements + ] + + all_results = [] + + for cfg_name, min_elem, min_conf in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = ConfluenceScoringBacktest( + min_smc_elements=min_elem, + min_confidence=min_conf, + ) + stats = bt.run(df=df, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_28b_pnl + + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + # Element distribution for trades taken + elem_dist = {} + for t in stats.trades: + c = t.smc_element_count + elem_dist[c] = elem_dist.get(c, 0) + 1 + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" Confluence filtered: {stats.confluence_filtered}") + print(f" BUY: {len(buy_trades)}, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)}, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" Element dist: {dict(sorted(elem_dist.items()))}") + print(f" vs #28B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, stats.confluence_filtered, elem_dist)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#29 CONFLUENCE SCORING β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'Filt':>5} {'vs #28B':>10}") + print(f" {'-' * 90}") + print(f" {'#24B (base) ':<25} {'739':>6} {'80.4%':>6} {'$2,235':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'β€”':>5} {'β€”':>10}") + print(f" {'#28B (smart BE)':<25} {'741':>6} {'79.8%':>6} {'$2,464':>10} {'3.5%':>6} {'3.23':>7} {'1.83':>5} {'β€”':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, filt, elem_dist in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {filt:>5} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + + print(f"\n Best config: {best_name}") + + # Element analysis for best config + print(f"\n Element Analysis (best config):") + for elem_count in sorted(set(t.smc_element_count for t in best_stats.trades)): + elem_trades = [t for t in best_stats.trades if t.smc_element_count == elem_count] + elem_wins = sum(1 for t in elem_trades if t.result == TradeResult.WIN) + elem_wr = elem_wins / len(elem_trades) * 100 if elem_trades else 0 + elem_pnl = sum(t.profit_usd for t in elem_trades) + print(f" {elem_count} elements: {len(elem_trades)} trades, {elem_wr:.1f}% WR, ${elem_pnl:,.2f}") + + # Direction analysis + print(f"\n Direction (best config):") + buy_trades = [t for t in best_stats.trades if t.direction == "BUY"] + sell_trades = [t for t in best_stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + print(f" BUY: {len(buy_trades)} trades, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)} trades, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + + # Exit reasons + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "29_confluence_scoring_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"confluence_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#29 Confluence Scoring Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #28B (741 trades, 79.8% WR, $2,464)\n\n") + for cfg_name, stats, net_pnl, diff, filt, elem_dist in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"Filtered: {filt}, Elem dist: {dict(sorted(elem_dist.items()))}, " + f"vs #28B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"confluence_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_30_dynamic_rr.py b/backtests/backtest_30_dynamic_rr.py new file mode 100644 index 0000000..acd4703 --- /dev/null +++ b/backtests/backtest_30_dynamic_rr.py @@ -0,0 +1,999 @@ +""" +Backtest #30 β€” Dynamic Risk-Reward +==================================== +Base: #28B (Smart BE 0.5x ATR) β€” 741 trades, 79.8% WR, $2,464, Sharpe 3.23 + +Idea: Adjust TP distance based on session, ATR, and conditions. +Currently, SMC uses a fixed RR of 1.5-2.0. What if we: +- Use tighter TP in Asian session (smaller moves) +- Use wider TP in Golden session (bigger moves) +- Scale TP with ATR (high vol = wider TP) + +Configs: + A: Session-based RR (Golden=2.0x, London=1.5x, Asian=1.0x of SMC TP) + B: ATR-scaled TP (TP = entry + direction * ATR * 3.0) + C: ATR-scaled TP wider (TP = entry + direction * ATR * 4.0) + D: A + ATR floor (session RR but min TP = ATR * 2.5) + E: Tighter TP across board (RR mult 0.8 = closer TP for higher hit rate) + +Usage: + python backtests/backtest_30_dynamic_rr.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + original_tp: float = 0.0 # NEW: track original TP for comparison + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + tp_modified: int = 0 # NEW + + +# ─── Dynamic RR Backtest ───────────────────────────────────── + +class DynamicRRBacktest: + """#28B base + dynamic risk-reward TP adjustment.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # #28B: Smart breakeven + be_profit_lock_atr_mult: float = 0.5, + # ═══ #30 DYNAMIC RR PARAMS ═══ + session_rr_multipliers: Optional[Dict[str, float]] = None, # session -> TP multiplier + atr_tp_mult: float = 0.0, # If > 0, override TP with entry +/- ATR * mult + tp_rr_multiplier: float = 1.0, # Global TP distance multiplier + atr_tp_floor_mult: float = 0.0, # Minimum TP distance = ATR * this + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + self.be_profit_lock_atr_mult = be_profit_lock_atr_mult + + # #30 params + self.session_rr_multipliers = session_rr_multipliers or {} + self.atr_tp_mult = atr_tp_mult + self.tp_rr_multiplier = tp_rr_multiplier + self.atr_tp_floor_mult = atr_tp_floor_mult + + config = get_config() + self.smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2300000 + + def _get_session_from_time(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + return wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30 + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + def _adjust_tp(self, direction, entry_price, original_tp, stop_loss, session_name, atr): + """#30: Adjust take profit based on session/ATR/multiplier.""" + tp = original_tp + modified = False + + # Method 1: ATR-based TP override + if self.atr_tp_mult > 0: + if direction == "BUY": + tp = entry_price + atr * self.atr_tp_mult + else: + tp = entry_price - atr * self.atr_tp_mult + modified = True + + # Method 2: Session-based multiplier + elif self.session_rr_multipliers: + mult = self.session_rr_multipliers.get(session_name, 1.0) + tp_distance = abs(original_tp - entry_price) + new_tp_distance = tp_distance * mult + if direction == "BUY": + tp = entry_price + new_tp_distance + else: + tp = entry_price - new_tp_distance + if mult != 1.0: + modified = True + + # Method 3: Global TP multiplier + if self.tp_rr_multiplier != 1.0 and self.atr_tp_mult == 0 and not self.session_rr_multipliers: + tp_distance = abs(original_tp - entry_price) + new_tp_distance = tp_distance * self.tp_rr_multiplier + if direction == "BUY": + tp = entry_price + new_tp_distance + else: + tp = entry_price - new_tp_distance + modified = True + + # Floor: ensure minimum TP distance + if self.atr_tp_floor_mult > 0: + min_tp_distance = atr * self.atr_tp_floor_mult + current_tp_distance = abs(tp - entry_price) + if current_tp_distance < min_tp_distance: + if direction == "BUY": + tp = entry_price + min_tp_distance + else: + tp = entry_price - min_tp_distance + modified = True + + return tp, modified + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ): + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + if self.be_profit_lock_atr_mult > 0: + be_lock_distance = atr * self.be_profit_lock_atr_mult + else: + be_lock_distance = 2.0 + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven (#28B: Smart) + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + be_lock_distance + else: + current_sl = entry_price - be_lock_distance + breakeven_moved = True + + # A.2 Trailing SL + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # C) Time-based + if bars_since_entry >= 16 and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + def run(self, df, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df.columns] + + times = df["time"].to_list() + start_idx = next((i for i, t in enumerate(times) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times) if t > end_date), len(df) - 100) if end_date else len(df) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + sess_rr_str = str(self.session_rr_multipliers) if self.session_rr_multipliers else "none" + print(f" #30 Session RR: {sess_rr_str}, ATR TP: {self.atr_tp_mult}, TP mult: {self.tp_rr_multiplier}, ATR floor: {self.atr_tp_floor_mult}") + print(f" Date range: {times[start_idx]} to {times[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or \ + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + original_tp = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + + # ═══ #30: ADJUST TP ═══ + take_profit_price, tp_was_modified = self._adjust_tp( + direction=smc_signal.signal_type, + entry_price=entry_price, + original_tp=original_tp, + stop_loss=stop_loss_price, + session_name=session_name, + atr=atr_at_entry, + ) + if tp_was_modified: + stats.tp_modified += 1 + + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times[exit_idx] if exit_idx < len(times) else times[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + original_tp=original_tp, + ) + stats.trades.append(trade) + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #30 Dynamic Risk-Reward") + print("Base: #28B (Smart BE 0.5x ATR) | Modified: Dynamic TP adjustment") + print("=" * 70) + + config = get_config() + mt5 = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df = mt5.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df) == 0: + print("ERROR: No data") + mt5.disconnect() + return + + print(f" Received {len(df)} bars") + times = df["time"].to_list() + print(f" Data range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df = features.calculate_all(df, include_ml_features=True) + df = smc.calculate_all(df) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df = regime_detector.predict(df) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + print(" Indicators calculated") + + baseline_28b_pnl = 2463.80 + + # ═══ CONFIGS ═══ + configs = [ + ("A: Session RR", { + "session_rr_multipliers": { + "Sydney-Tokyo": 0.8, + "London Early": 1.0, + "London-NY Overlap (Golden)": 1.3, + "NY Session": 1.0, + }, + }), + ("B: ATR TP 3.0x", { + "atr_tp_mult": 3.0, + }), + ("C: ATR TP 4.0x", { + "atr_tp_mult": 4.0, + }), + ("D: Session + ATR floor", { + "session_rr_multipliers": { + "Sydney-Tokyo": 0.8, + "London Early": 1.0, + "London-NY Overlap (Golden)": 1.3, + "NY Session": 1.0, + }, + "atr_tp_floor_mult": 2.5, + }), + ("E: Tighter TP 0.8x", { + "tp_rr_multiplier": 0.8, + }), + ] + + all_results = [] + + for cfg_name, cfg_params in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = DynamicRRBacktest(**cfg_params) + stats = bt.run(df=df, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_28b_pnl + + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + # TP hit rate + tp_hits = sum(1 for t in stats.trades if t.exit_reason == ExitReason.TAKE_PROFIT) + tp_rate = tp_hits / stats.total_trades * 100 if stats.total_trades > 0 else 0 + + # Avg RR + avg_rr = np.mean([t.rr_ratio for t in stats.trades]) if stats.trades else 0 + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" TP modified: {stats.tp_modified} | TP hit rate: {tp_rate:.1f}% | Avg RR: {avg_rr:.2f}") + print(f" BUY: {len(buy_trades)}, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)}, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" vs #28B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, stats.tp_modified, tp_rate, avg_rr)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#30 DYNAMIC RISK-REWARD β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'TP%':>5} {'RR':>5} {'vs #28B':>10}") + print(f" {'-' * 95}") + print(f" {'#24B (base) ':<25} {'739':>6} {'80.4%':>6} {'$2,235':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'3.5%':>5} {'1.6':>5} {'β€”':>10}") + print(f" {'#28B (smart BE)':<25} {'741':>6} {'79.8%':>6} {'$2,464':>10} {'3.5%':>6} {'3.23':>7} {'1.83':>5} {'3.0%':>5} {'1.6':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, tp_mod, tp_rate, avg_rr in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {tp_rate:>4.1f}% {avg_rr:>5.2f} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + + print(f"\n Best config: {best_name}") + + # Per-session analysis for best + print(f"\n Per-Session (best config):") + sessions = set(t.session for t in best_stats.trades) + for sess in sorted(sessions): + sess_trades = [t for t in best_stats.trades if t.session == sess] + sess_wins = sum(1 for t in sess_trades if t.result == TradeResult.WIN) + sess_wr = sess_wins / len(sess_trades) * 100 if sess_trades else 0 + sess_pnl = sum(t.profit_usd for t in sess_trades) + print(f" {sess:30s}: {len(sess_trades):>4} trades, {sess_wr:>5.1f}% WR, ${sess_pnl:>8,.2f}") + + # Exit reasons + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "30_dynamic_rr_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"dynamic_rr_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#30 Dynamic Risk-Reward Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #28B (741 trades, 79.8% WR, $2,464)\n\n") + for cfg_name, stats, net_pnl, diff, tp_mod, tp_rate, avg_rr in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"TP modified: {tp_mod}, TP rate: {tp_rate:.1f}%, Avg RR: {avg_rr:.2f}, " + f"vs #28B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"dynamic_rr_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_31_multi_tf_h1.py b/backtests/backtest_31_multi_tf_h1.py new file mode 100644 index 0000000..f929f83 --- /dev/null +++ b/backtests/backtest_31_multi_tf_h1.py @@ -0,0 +1,1016 @@ +""" +Backtest #31 β€” Multi-Timeframe H1 Confirmation +================================================ +Base: #28B (Smart BE 0.5x ATR) β€” 741 trades, 79.8% WR, $2,464, Sharpe 3.23 + +Idea: Use H1 (1-hour) timeframe for trend confirmation before entering on M15. +If H1 trend disagrees with M15 signal, skip the trade. + +H1 trend detection methods: + - EMA alignment (20 EMA vs 50 EMA) + - Market structure (last BOS direction) + - Price above/below EMA + +Configs: + A: H1 EMA trend filter (signal must align with H1 20/50 EMA trend) + B: H1 price vs EMA20 (BUY only if price > H1 EMA20, SELL only if < H1 EMA20) + C: H1 BOS direction (signal must match last H1 BOS direction) + D: H1 filter on SELL only (only filter SELL trades with H1 trend) + E: H1 filter relaxed (allow if H1 is NEUTRAL, only block opposing trend) + +Usage: + python backtests/backtest_31_multi_tf_h1.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + h1_trend: str = "NEUTRAL" # NEW + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + h1_filtered: int = 0 # NEW + + +# ─── Multi-TF Backtest ────────────────────────────────────── + +class MultiTFBacktest: + """#28B base + H1 multi-timeframe confirmation.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + trend_reversal_threshold: float = 0.75, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + trend_reversal_mult: float = 0.6, + # #24B base + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # #28B: Smart breakeven + be_profit_lock_atr_mult: float = 0.5, + # ═══ #31 MULTI-TF PARAMS ═══ + h1_filter_mode: str = "ema", # "ema", "price_vs_ema", "bos", "sell_only", "relaxed" + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.trend_reversal_threshold = trend_reversal_threshold + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + self.be_profit_lock_atr_mult = be_profit_lock_atr_mult + + # #31 params + self.h1_filter_mode = h1_filter_mode + + config = get_config() + self.smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2310000 + + def _get_session_from_time(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + return wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30 + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + def _get_h1_trend(self, df_h1_slice, mode): + """Get H1 trend direction based on filter mode.""" + if df_h1_slice is None or len(df_h1_slice) < 50: + return "NEUTRAL" + + closes = df_h1_slice["close"].to_list() + + if mode in ["ema", "relaxed", "sell_only"]: + # EMA 20 vs EMA 50 alignment + if len(closes) < 50: + return "NEUTRAL" + ema20 = self._calc_ema(closes, 20) + ema50 = self._calc_ema(closes, 50) + if ema20 > ema50 * 1.0005: + return "BULLISH" + elif ema20 < ema50 * 0.9995: + return "BEARISH" + return "NEUTRAL" + + elif mode == "price_vs_ema": + # Price above/below EMA20 + if len(closes) < 20: + return "NEUTRAL" + ema20 = self._calc_ema(closes, 20) + current_price = closes[-1] + if current_price > ema20 * 1.001: + return "BULLISH" + elif current_price < ema20 * 0.999: + return "BEARISH" + return "NEUTRAL" + + elif mode == "bos": + # Last H1 BOS direction + if "bos" in df_h1_slice.columns: + recent_bos = df_h1_slice.tail(10)["bos"].to_list() + # Find last non-zero BOS + for b in reversed(recent_bos): + if b == 1: + return "BULLISH" + elif b == -1: + return "BEARISH" + return "NEUTRAL" + + return "NEUTRAL" + + def _calc_ema(self, data, period): + """Calculate EMA for given data and period.""" + if len(data) < period: + return data[-1] if data else 0 + multiplier = 2 / (period + 1) + ema = np.mean(data[:period]) # SMA for initial + for val in data[period:]: + ema = (val - ema) * multiplier + ema + return ema + + def _h1_allows_trade(self, h1_trend, signal_direction, mode): + """Check if H1 trend allows the trade.""" + if mode == "relaxed": + # Only block if H1 actively opposes + if signal_direction == "BUY" and h1_trend == "BEARISH": + return False + if signal_direction == "SELL" and h1_trend == "BULLISH": + return False + return True # Allow NEUTRAL + + elif mode == "sell_only": + # Only filter SELL trades + if signal_direction == "SELL" and h1_trend == "BULLISH": + return False + return True + + else: + # Strict: signal must match H1 trend (NEUTRAL blocks too) + if signal_direction == "BUY" and h1_trend != "BULLISH": + return False + if signal_direction == "SELL" and h1_trend != "BEARISH": + return False + return True + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ): + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + if self.be_profit_lock_atr_mult > 0: + be_lock_distance = atr * self.be_profit_lock_atr_mult + else: + be_lock_distance = 2.0 + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + be_lock_distance + else: + current_sl = entry_price - be_lock_distance + breakeven_moved = True + + if pip_profit_from_entry >= adaptive_trail_start_pips: + trail_distance = adaptive_trail_step_pips * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.trend_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.trend_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + if bars_since_entry >= 16 and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + def run(self, df_m15, df_h1, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df_m15.columns] + + times_m15 = df_m15["time"].to_list() + times_h1 = df_h1["time"].to_list() if df_h1 is not None else [] + + start_idx = next((i for i, t in enumerate(times_m15) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times_m15) if t > end_date), len(df_m15) - 100) if end_date else len(df_m15) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #31 H1 filter mode: {self.h1_filter_mode}") + print(f" H1 bars available: {len(df_h1) if df_h1 is not None else 0}") + print(f" Date range: {times_m15[start_idx]} to {times_m15[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times_m15[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df_m15.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + # ═══ #31: H1 TREND FILTER ═══ + h1_trend = "NEUTRAL" + if df_h1 is not None and len(times_h1) > 0: + # Find H1 bars up to current M15 time + h1_idx = 0 + for j, t in enumerate(times_h1): + if t <= current_time: + h1_idx = j + else: + break + if h1_idx > 50: + df_h1_slice = df_h1.head(h1_idx + 1) + h1_trend = self._get_h1_trend(df_h1_slice, self.h1_filter_mode) + + if not self._h1_allows_trade(h1_trend, smc_signal.signal_type, self.h1_filter_mode): + stats.h1_filtered += 1 + continue + + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or \ + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df_m15, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times_m15[exit_idx] if exit_idx < len(times_m15) else times_m15[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + h1_trend=h1_trend, + ) + stats.trades.append(trade) + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #31 Multi-Timeframe H1 Confirmation") + print("Base: #28B (Smart BE 0.5x ATR) | Modified: H1 trend filter") + print("=" * 70) + + config = get_config() + mt5_conn = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5_conn.connect() + print(f"\nConnected to MT5") + + # Fetch M15 data (primary) + print("Fetching XAUUSD M15 historical data...") + df_m15 = mt5_conn.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + if len(df_m15) == 0: + print("ERROR: No M15 data") + mt5_conn.disconnect() + return + print(f" M15: {len(df_m15)} bars") + + # Fetch H1 data (for multi-TF) + print("Fetching XAUUSD H1 historical data...") + df_h1 = mt5_conn.get_market_data(symbol="XAUUSD", timeframe="H1", count=15000) + if len(df_h1) == 0: + print("WARNING: No H1 data β€” running without H1 filter") + df_h1 = None + else: + print(f" H1: {len(df_h1)} bars") + + times = df_m15["time"].to_list() + print(f" M15 range: {times[0]} to {times[-1]}") + if df_h1 is not None: + h1_times = df_h1["time"].to_list() + print(f" H1 range: {h1_times[0]} to {h1_times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + # Calculate indicators for M15 + print("\nCalculating M15 indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df_m15 = features.calculate_all(df_m15, include_ml_features=True) + df_m15 = smc.calculate_all(df_m15) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df_m15 = regime_detector.predict(df_m15) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + + # Calculate H1 indicators (for BOS mode) + if df_h1 is not None: + print("Calculating H1 indicators...") + df_h1 = features.calculate_all(df_h1, include_ml_features=False) + df_h1 = smc.calculate_all(df_h1) + print(" H1 indicators calculated") + + print(" All indicators calculated") + + baseline_28b_pnl = 2463.80 + + # ═══ CONFIGS ═══ + configs = [ + ("A: H1 EMA strict", "ema"), + ("B: H1 price vs EMA20", "price_vs_ema"), + ("C: H1 BOS direction", "bos"), + ("D: H1 SELL only", "sell_only"), + ("E: H1 relaxed", "relaxed"), + ] + + all_results = [] + + for cfg_name, h1_mode in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = MultiTFBacktest(h1_filter_mode=h1_mode) + stats = bt.run(df_m15=df_m15, df_h1=df_h1, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_28b_pnl + + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + # H1 trend distribution + h1_dist = {} + for t in stats.trades: + h1_dist[t.h1_trend] = h1_dist.get(t.h1_trend, 0) + 1 + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" H1 filtered: {stats.h1_filtered}") + print(f" BUY: {len(buy_trades)}, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)}, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" H1 trend dist: {dict(sorted(h1_dist.items()))}") + print(f" vs #28B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff, stats.h1_filtered, h1_dist)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#31 MULTI-TIMEFRAME H1 β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'Filt':>5} {'vs #28B':>10}") + print(f" {'-' * 90}") + print(f" {'#24B (base) ':<25} {'739':>6} {'80.4%':>6} {'$2,235':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'β€”':>5} {'β€”':>10}") + print(f" {'#28B (smart BE)':<25} {'741':>6} {'79.8%':>6} {'$2,464':>10} {'3.5%':>6} {'3.23':>7} {'1.83':>5} {'β€”':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff, filt, h1_dist in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} {filt:>5} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + + print(f"\n Best config: {best_name}") + + # H1 trend analysis for best + print(f"\n H1 Trend WR Analysis (best config):") + for trend_val in sorted(set(t.h1_trend for t in best_stats.trades)): + trend_trades = [t for t in best_stats.trades if t.h1_trend == trend_val] + trend_wins = sum(1 for t in trend_trades if t.result == TradeResult.WIN) + trend_wr = trend_wins / len(trend_trades) * 100 if trend_trades else 0 + trend_pnl = sum(t.profit_usd for t in trend_trades) + print(f" {trend_val:10s}: {len(trend_trades):>4} trades, {trend_wr:>5.1f}% WR, ${trend_pnl:>8,.2f}") + + # Exit reasons + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "31_multi_tf_h1_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"multi_tf_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#31 Multi-Timeframe H1 Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #28B (741 trades, 79.8% WR, $2,464)\n\n") + for cfg_name, stats, net_pnl, diff, filt, h1_dist in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"H1 filtered: {filt}, H1 dist: {dict(sorted(h1_dist.items()))}, " + f"vs #28B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"multi_tf_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5_conn.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_32_ml_exit_optimizer.py b/backtests/backtest_32_ml_exit_optimizer.py new file mode 100644 index 0000000..b806252 --- /dev/null +++ b/backtests/backtest_32_ml_exit_optimizer.py @@ -0,0 +1,983 @@ +""" +Backtest #32 β€” ML Exit Optimizer +================================== +Base: #31B (H1 Price vs EMA20) β€” 625 trades, 81.8% WR, $2,807, Sharpe 3.97 + +Idea: Use XGBoost ML predictions more aggressively for exit decisions. +Currently ML is only used for trend reversal detection in exits. +What if we: +- Lower the ML reversal confidence threshold (catch reversals earlier) +- Use ML to tighten trailing SL when ML opposes +- Use ML agreement to hold winners longer + +Configs: + A: Lower ML reversal threshold (0.75 β†’ 0.65) β€” catch reversals earlier + B: ML-tightened trail (if ML opposes, use 2x ATR trail instead of 3x) + C: ML hold boost (if ML agrees, extend timeout from 16β†’24 bars) + D: A+B combined (earlier reversal + tighter trail when opposed) + E: A+B+C combined (full ML exit optimization) + +Usage: + python backtests/backtest_32_ml_exit_optimizer.py +""" + +import polars as pl +import pandas as pd +import numpy as np +from datetime import datetime, timedelta, date +from typing import Dict, List, Tuple, Optional +from dataclasses import dataclass, field +from enum import Enum +import sys +import os +from zoneinfo import ZoneInfo + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from src.mt5_connector import MT5Connector +from src.smc_polars import SMCAnalyzer, SMCSignal +from src.feature_eng import FeatureEngineer +from src.regime_detector import MarketRegimeDetector, MarketRegime +from src.ml_model import TradingModel +from src.config import get_config +from src.dynamic_confidence import DynamicConfidenceManager, create_dynamic_confidence, MarketQuality +from loguru import logger + +logger.remove() +logger.add(sys.stderr, level="WARNING") + +WIB = ZoneInfo("Asia/Jakarta") + + +# ─── Enums & Dataclasses ────────────────────────────────────── + +class TradeResult(Enum): + WIN = "WIN" + LOSS = "LOSS" + BREAKEVEN = "BREAKEVEN" + +class ExitReason(Enum): + TAKE_PROFIT = "take_profit" + SMART_TP = "smart_tp" + PEAK_PROTECT = "peak_protect" + EARLY_EXIT = "early_exit" + EARLY_CUT = "early_cut" + MAX_LOSS = "max_loss" + STALL = "stall" + TREND_REVERSAL = "trend_reversal" + TIMEOUT = "timeout" + WEEKEND_CLOSE = "weekend_close" + TRAILING_SL = "trailing_sl" + BREAKEVEN_EXIT = "breakeven_exit" + DAILY_LIMIT = "daily_limit" + REGIME_DANGER = "regime_danger" + MARKET_SIGNAL = "market_signal" + +class TradingMode(Enum): + NORMAL = "normal" + RECOVERY = "recovery" + PROTECTED = "protected" + STOPPED = "stopped" + +@dataclass +class SimulatedTrade: + ticket: int + entry_time: datetime + exit_time: datetime + direction: str + entry_price: float + exit_price: float + stop_loss: float + take_profit: float + lot_size: float + profit_usd: float + profit_pips: float + result: TradeResult + exit_reason: ExitReason + smc_confidence: float + regime: str + session: str + signal_reason: str + has_bos: bool = False + has_choch: bool = False + has_fvg: bool = False + has_ob: bool = False + atr_at_entry: float = 0.0 + rr_ratio: float = 0.0 + trading_mode: str = "normal" + h1_trend: str = "NEUTRAL" + +@dataclass +class BacktestStats: + total_trades: int = 0 + wins: int = 0 + losses: int = 0 + total_profit: float = 0.0 + total_loss: float = 0.0 + max_drawdown: float = 0.0 + max_drawdown_usd: float = 0.0 + win_rate: float = 0.0 + profit_factor: float = 0.0 + avg_win: float = 0.0 + avg_loss: float = 0.0 + avg_trade: float = 0.0 + expectancy: float = 0.0 + sharpe_ratio: float = 0.0 + trades: List[SimulatedTrade] = field(default_factory=list) + equity_curve: List[float] = field(default_factory=list) + avoided_signals: int = 0 + daily_limit_stops: int = 0 + recovery_mode_trades: int = 0 + session_blocked: int = 0 + h1_filtered: int = 0 + + +# ─── ML Exit Optimizer Backtest ────────────────────────────── + +class MLExitBacktest: + """#31B base + ML exit optimization.""" + + def __init__( + self, + capital: float = 5000.0, + max_daily_loss_percent: float = 5.0, + max_loss_per_trade_percent: float = 1.0, + base_lot_size: float = 0.01, + max_lot_size: float = 0.02, + recovery_lot_size: float = 0.01, + max_concurrent_positions: int = 2, + min_profit_to_protect: float = 5.0, + max_drawdown_from_peak: float = 50.0, + trade_cooldown_bars: int = 10, + # #24B base + skip_tokyo_london: bool = True, + early_cut_momentum: float = -50.0, + early_cut_loss_pct: float = 30.0, + be_mult: float = 2.0, + trail_start_mult: float = 4.0, + trail_step_mult: float = 3.0, + # #28B: Smart breakeven + be_profit_lock_atr_mult: float = 0.5, + # ═══ #32 ML EXIT OPTIMIZER PARAMS ═══ + ml_reversal_threshold: float = 0.75, # ML confidence to trigger reversal exit + ml_tighten_trail: bool = False, # If ML opposes, tighten trail step + ml_tighten_trail_mult: float = 2.0, # Tightened trail step multiplier (vs 3.0 default) + ml_hold_boost: bool = False, # If ML agrees, extend timeout + ml_hold_timeout_bars: int = 24, # Extended timeout when ML agrees + trend_reversal_mult: float = 0.6, + ): + self.capital = capital + self.max_daily_loss_usd = capital * (max_daily_loss_percent / 100) + self.max_loss_per_trade = capital * (max_loss_per_trade_percent / 100) + self.base_lot_size = base_lot_size + self.max_lot_size = max_lot_size + self.recovery_lot_size = recovery_lot_size + self.max_concurrent_positions = max_concurrent_positions + self.min_profit_to_protect = min_profit_to_protect + self.max_drawdown_from_peak = max_drawdown_from_peak + self.trade_cooldown_bars = trade_cooldown_bars + self.trend_reversal_mult = trend_reversal_mult + + self.skip_tokyo_london = skip_tokyo_london + self.early_cut_momentum = early_cut_momentum + self.early_cut_loss_pct = early_cut_loss_pct + self.be_mult = be_mult + self.trail_start_mult = trail_start_mult + self.trail_step_mult = trail_step_mult + self.be_profit_lock_atr_mult = be_profit_lock_atr_mult + + # #32 params + self.ml_reversal_threshold = ml_reversal_threshold + self.ml_tighten_trail = ml_tighten_trail + self.ml_tighten_trail_mult = ml_tighten_trail_mult + self.ml_hold_boost = ml_hold_boost + self.ml_hold_timeout_bars = ml_hold_timeout_bars + + config = get_config() + self.smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + self.features = FeatureEngineer() + self.dynamic_confidence = create_dynamic_confidence() + + self.ml_model = TradingModel(model_path="models/xgboost_model.pkl") + try: + self.ml_model.load() + print(" ML model loaded (for exit evaluation)") + except Exception: + print(" [WARN] ML model not loaded") + + self.regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + self.regime_detector.load() + except Exception: + print(" [WARN] HMM model not loaded") + + self._ticket_counter = 2320000 + + def _get_session_from_time(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib_time = dt.astimezone(WIB) + hour = wib_time.hour + if 6 <= hour < 15: + return "Sydney-Tokyo", True, 0.5 + elif 15 <= hour < 16: + if self.skip_tokyo_london: + return "Tokyo-London Overlap", False, 0.0 + return "Tokyo-London Overlap", True, 0.75 + elif 16 <= hour < 19: + return "London Early", True, 0.8 + elif 19 <= hour < 24: + return "London-NY Overlap (Golden)", True, 1.0 + elif 0 <= hour < 4: + return "NY Session", True, 0.9 + else: + return "Off Hours", False, 0.0 + + def _hours_to_golden(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + if 19 <= wib.hour < 24: + return 0 + target = wib.replace(hour=19, minute=0, second=0, microsecond=0) + if wib.hour >= 19: + target += timedelta(days=1) + return max(0, (target - wib).total_seconds() / 3600) + + def _is_near_weekend_close(self, dt): + if dt.tzinfo is None: + dt = dt.replace(tzinfo=ZoneInfo("UTC")) + wib = dt.astimezone(WIB) + return wib.weekday() == 5 and wib.hour >= 4 and wib.minute >= 30 + + def _calculate_lot_size(self, confidence, regime, trading_mode, session_mult): + if trading_mode == TradingMode.STOPPED: + return 0 + lot = self.base_lot_size + if trading_mode in (TradingMode.RECOVERY, TradingMode.PROTECTED): + lot = self.recovery_lot_size + else: + if confidence >= 0.65: + lot = self.max_lot_size + elif confidence >= 0.55: + lot = self.base_lot_size + else: + lot = self.recovery_lot_size + if regime.lower() in ["high_volatility", "crisis"]: + lot = self.recovery_lot_size + lot = max(0.01, lot * session_mult) + return round(lot, 2) + + def _calc_ema(self, data, period): + if len(data) < period: + return data[-1] if data else 0 + multiplier = 2 / (period + 1) + ema = np.mean(data[:period]) + for val in data[period:]: + ema = (val - ema) * multiplier + ema + return ema + + def _get_h1_trend(self, df_h1_slice): + """#31B: H1 Price vs EMA20.""" + if df_h1_slice is None or len(df_h1_slice) < 20: + return "NEUTRAL" + closes = df_h1_slice["close"].to_list() + ema20 = self._calc_ema(closes, 20) + current_price = closes[-1] + if current_price > ema20 * 1.001: + return "BULLISH" + elif current_price < ema20 * 0.999: + return "BEARISH" + return "NEUTRAL" + + def _simulate_trade_exit( + self, df, entry_idx, direction, entry_price, take_profit, stop_loss, + lot_size, daily_loss_so_far, feature_cols, max_bars=100, + ): + pip_value = 10 + highs = df["high"].to_list() + lows = df["low"].to_list() + closes = df["close"].to_list() + times = df["time"].to_list() + + atr = 12.0 + if "atr" in df.columns: + atr_list = df["atr"].to_list() + if entry_idx < len(atr_list) and atr_list[entry_idx] is not None: + atr = atr_list[entry_idx] + + adaptive_breakeven_pips = atr * self.be_mult + adaptive_trail_start_pips = atr * self.trail_start_mult + adaptive_trail_step_pips = atr * self.trail_step_mult + # #32: ML-tightened trail step + tightened_trail_step_pips = atr * self.ml_tighten_trail_mult + reversal_momentum_threshold = atr * self.trend_reversal_mult + min_loss_for_reversal_exit = atr * 0.8 + + if self.be_profit_lock_atr_mult > 0: + be_lock_distance = atr * self.be_profit_lock_atr_mult + else: + be_lock_distance = 2.0 + + profit_history = [] + peak_profit = 0.0 + stall_count = 0 + reversal_warnings = 0 + current_sl = stop_loss + breakeven_moved = False + + if direction == "BUY": + target_tp_profit = (take_profit - entry_price) / 0.1 * pip_value * lot_size + else: + target_tp_profit = (entry_price - take_profit) / 0.1 * pip_value * lot_size + + cached_ml_signal = "" + cached_ml_confidence = 0.5 + ml_agrees_with_trade = False + + for i in range(entry_idx + 1, min(entry_idx + max_bars, len(df))): + high = highs[i] + low = lows[i] + close = closes[i] + current_time = times[i] + + if direction == "BUY": + current_pips = (close - entry_price) / 0.1 + pip_profit_from_entry = current_pips + else: + current_pips = (entry_price - close) / 0.1 + pip_profit_from_entry = current_pips + current_profit = current_pips * pip_value * lot_size + + profit_history.append(current_profit) + if current_profit > peak_profit: + peak_profit = current_profit + + bars_since_entry = i - entry_idx + + if bars_since_entry % 4 == 0 and self.ml_model.fitted: + try: + df_slice = df.head(i + 1) + ml_pred = self.ml_model.predict(df_slice, feature_cols) + cached_ml_signal = ml_pred.signal + cached_ml_confidence = ml_pred.confidence + # #32: Track ML agreement + ml_agrees_with_trade = ( + (direction == "BUY" and cached_ml_signal == "BUY") or + (direction == "SELL" and cached_ml_signal == "SELL") + ) + except Exception: + pass + + momentum = 0.0 + if len(profit_history) >= 3: + recent = profit_history[-5:] if len(profit_history) >= 5 else profit_history + profit_change = recent[-1] - recent[0] + momentum = max(-100, min(100, (profit_change / 10) * 50)) + profit_growing = momentum > 0 + + # A.0 TP hit + if direction == "BUY" and high >= take_profit: + pips = (take_profit - entry_price) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + elif direction == "SELL" and low <= take_profit: + pips = (entry_price - take_profit) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TAKE_PROFIT, i, take_profit + + # A.0b Trailing SL hit + if breakeven_moved and current_sl > 0: + if direction == "BUY" and low <= current_sl: + pips = (current_sl - entry_price) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + elif direction == "SELL" and high >= current_sl: + pips = (entry_price - current_sl) / 0.1 + reason = ExitReason.TRAILING_SL if pip_profit_from_entry >= adaptive_trail_start_pips else ExitReason.BREAKEVEN_EXIT + return pips * pip_value * lot_size, pips, reason, i, current_sl + + # A.1 Breakeven (#28B: Smart) + if pip_profit_from_entry >= adaptive_breakeven_pips and not breakeven_moved: + if direction == "BUY": + current_sl = entry_price + be_lock_distance + else: + current_sl = entry_price - be_lock_distance + breakeven_moved = True + + # A.2 Trailing SL (#32: tighter when ML opposes) + if pip_profit_from_entry >= adaptive_trail_start_pips: + # #32B: Use tighter trail if ML opposes the trade direction + if self.ml_tighten_trail and not ml_agrees_with_trade and cached_ml_confidence >= 0.6: + active_trail_step = tightened_trail_step_pips + else: + active_trail_step = adaptive_trail_step_pips + + trail_distance = active_trail_step * 0.1 + if direction == "BUY": + new_trail_sl = close - trail_distance + if new_trail_sl > current_sl: + current_sl = new_trail_sl + else: + new_trail_sl = close + trail_distance + if current_sl == 0 or new_trail_sl < current_sl: + current_sl = new_trail_sl + + # A.3 Peak protect + if peak_profit > self.min_profit_to_protect: + drawdown_pct = ((peak_profit - current_profit) / peak_profit) * 100 if peak_profit > 0 else 0 + if drawdown_pct > self.max_drawdown_from_peak: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + + # A.4 Market analysis + if bars_since_entry % 5 == 0 and bars_since_entry >= 5 and i >= 20: + ma_fast = np.mean(closes[i-4:i+1]) + ma_slow = np.mean(closes[i-19:i+1]) + trend = "BULLISH" if ma_fast > ma_slow * 1.001 else ("BEARISH" if ma_fast < ma_slow * 0.999 else "NEUTRAL") + roc = (closes[i] / closes[max(0,i-4)] - 1) * 100 + mom_dir = "BULLISH" if roc > 0.3 else ("BEARISH" if roc < -0.3 else "NEUTRAL") + + rsi_val = None + if "rsi" in df.columns: + rsi_list = df["rsi"].to_list() + if i < len(rsi_list): + rsi_val = rsi_list[i] + + urgency = 0 + should_exit = False + if cached_ml_confidence > 0.75: + if (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY"): + should_exit = True; urgency += 2 + if rsi_val: + if (rsi_val > 75 and direction == "BUY") or (rsi_val < 25 and direction == "SELL"): + should_exit = True; urgency += 2 + if (direction == "BUY" and trend == "BEARISH" and mom_dir == "BEARISH") or \ + (direction == "SELL" and trend == "BULLISH" and mom_dir == "BULLISH"): + should_exit = True; urgency += 3 + + if should_exit and current_profit > self.min_profit_to_protect / 2: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + if urgency >= 7 and current_profit > 0: + return current_profit, current_pips, ExitReason.MARKET_SIGNAL, i, close + + # A.5 Weekend close + if self._is_near_weekend_close(current_time): + if current_profit > 0 or current_profit > -10: + return current_profit, current_pips, ExitReason.WEEKEND_CLOSE, i, close + + # B.1 Smart TP + if current_profit >= 15: + if current_profit >= 40: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if current_profit >= 25 and momentum < -30: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + if peak_profit > 30 and current_profit < peak_profit * 0.6: + return current_profit, current_pips, ExitReason.PEAK_PROTECT, i, close + if current_profit >= 20: + progress = (current_profit / target_tp_profit) * 100 if target_tp_profit > 0 else 0 + progress_score = min(40, max(0, progress * 0.4)) + momentum_score = ((momentum + 100) / 200) * 30 + time_penalty = min(10, bars_since_entry / 4 * 2) + tp_probability = progress_score + momentum_score + 10 - time_penalty + if tp_probability < 25: + return current_profit, current_pips, ExitReason.SMART_TP, i, close + + # B.2 Smart Early Exit + if 5 <= current_profit < 15: + if momentum < -50 and cached_ml_confidence >= 0.65: + is_reversal = (direction == "BUY" and cached_ml_signal == "SELL") or (direction == "SELL" and cached_ml_signal == "BUY") + if is_reversal: + return current_profit, current_pips, ExitReason.EARLY_EXIT, i, close + + # B.3 Early cut + if current_profit < 0: + loss_percent_of_max = abs(current_profit) / self.max_loss_per_trade * 100 + if momentum < self.early_cut_momentum and loss_percent_of_max >= self.early_cut_loss_pct: + return current_profit, current_pips, ExitReason.EARLY_CUT, i, close + + # B.4 Trend Reversal (#32A: configurable ML threshold) + is_ml_reversal = False + if (direction == "BUY" and cached_ml_signal == "SELL" and cached_ml_confidence >= self.ml_reversal_threshold) or \ + (direction == "SELL" and cached_ml_signal == "BUY" and cached_ml_confidence >= self.ml_reversal_threshold): + is_ml_reversal = True + reversal_warnings += 1 + loss_moderate = abs(current_profit) > (self.max_loss_per_trade * 0.4) + if is_ml_reversal and current_profit < -8 and loss_moderate: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + if reversal_warnings >= 3 and current_profit < -10: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + # B.5 Max loss + if current_profit <= -(self.max_loss_per_trade * 0.50): + htg = self._hours_to_golden(current_time) + if htg <= 1 and htg > 0 and momentum > -40: + pass + else: + return current_profit, current_pips, ExitReason.MAX_LOSS, i, close + + # B.6 Stall + if len(profit_history) >= 10: + recent_range = max(profit_history[-10:]) - min(profit_history[-10:]) + if recent_range < 3 and current_profit < -15: + stall_count += 1 + if stall_count >= 5: + return current_profit, current_pips, ExitReason.STALL, i, close + + # B.7 Daily loss limit + potential_daily_loss = daily_loss_so_far + abs(min(0, current_profit)) + if potential_daily_loss >= self.max_daily_loss_usd: + return current_profit, current_pips, ExitReason.DAILY_LIMIT, i, close + + # C) Time-based (#32C: ML hold boost extends timeout) + base_timeout = 16 + if self.ml_hold_boost and ml_agrees_with_trade and cached_ml_confidence >= 0.6: + base_timeout = self.ml_hold_timeout_bars + + if bars_since_entry >= base_timeout and current_profit < 5 and not profit_growing: + if current_profit >= 0 or current_profit > -15: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 24 and (current_profit < 10 or not profit_growing): + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + if bars_since_entry >= 32: + return current_profit, current_pips, ExitReason.TIMEOUT, i, close + + # C.2 ATR trend reversal + if bars_since_entry > 10: + recent_closes = closes[i-5:i+1] + mom = recent_closes[-1] - recent_closes[0] + if (direction == "BUY" and mom < -reversal_momentum_threshold) or \ + (direction == "SELL" and mom > reversal_momentum_threshold): + if current_profit < -min_loss_for_reversal_exit: + return current_profit, current_pips, ExitReason.TREND_REVERSAL, i, close + + final_idx = min(entry_idx + max_bars - 1, len(df) - 1) + final_price = closes[final_idx] + pips = ((final_price - entry_price) if direction == "BUY" else (entry_price - final_price)) / 0.1 + return pips * pip_value * lot_size, pips, ExitReason.TIMEOUT, final_idx, final_price + + def run(self, df_m15, df_h1, start_date=None, end_date=None, initial_capital=5000.0): + stats = BacktestStats() + capital = initial_capital + peak_capital = initial_capital + stats.equity_curve.append(capital) + + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + consecutive_losses = 0 + trading_mode = TradingMode.NORMAL + current_date = None + + feature_cols = [] + if self.ml_model.fitted and self.ml_model.feature_names: + feature_cols = [f for f in self.ml_model.feature_names if f in df_m15.columns] + + times_m15 = df_m15["time"].to_list() + times_h1 = df_h1["time"].to_list() if df_h1 is not None else [] + + start_idx = next((i for i, t in enumerate(times_m15) if t >= start_date), 100) if start_date else 100 + end_idx = next((i for i, t in enumerate(times_m15) if t > end_date), len(df_m15) - 100) if end_date else len(df_m15) - 100 + + last_trade_idx = -self.trade_cooldown_bars * 2 + + print(f" #32 ML reversal threshold: {self.ml_reversal_threshold}, tighten trail: {self.ml_tighten_trail}, hold boost: {self.ml_hold_boost}") + print(f" Date range: {times_m15[start_idx]} to {times_m15[end_idx - 1]}") + print(f" Total bars: {end_idx - start_idx}") + + for i in range(start_idx, end_idx): + if i - last_trade_idx < self.trade_cooldown_bars: + continue + + current_time = times_m15[i] + trade_date = current_time.date() if hasattr(current_time, 'date') else current_time + if current_date is None or trade_date != current_date: + daily_loss = 0.0 + daily_profit = 0.0 + daily_trades = 0 + current_date = trade_date + if consecutive_losses < 2: + trading_mode = TradingMode.NORMAL + + if trading_mode == TradingMode.STOPPED: + continue + + session_name, can_trade, lot_mult = self._get_session_from_time(current_time) + if not can_trade: + if session_name == "Tokyo-London Overlap": + stats.session_blocked += 1 + continue + + if hasattr(current_time, 'weekday') and current_time.weekday() >= 5: + continue + + df_slice = df_m15.head(i + 1) + + regime = "normal" + try: + if self.regime_detector.fitted: + regime_state = self.regime_detector.get_current_state(df_slice) + if regime_state: + regime = regime_state.regime.value + if regime_state.regime == MarketRegime.CRISIS: + continue + if regime_state.recommendation == "SLEEP": + continue + except Exception: + pass + + try: + ml_signal = "" + ml_confidence = 0.5 + if self.ml_model.fitted and feature_cols: + ml_pred = self.ml_model.predict(df_slice, feature_cols) + ml_signal = ml_pred.signal + ml_confidence = ml_pred.confidence + + market_analysis = self.dynamic_confidence.analyze_market( + session=session_name, regime=regime, volatility="medium", + trend_direction=regime, has_smc_signal=True, + ml_signal=ml_signal, ml_confidence=ml_confidence, + ) + if market_analysis.quality == MarketQuality.AVOID: + stats.avoided_signals += 1 + continue + except Exception: + pass + + try: + smc_signal = self.smc.generate_signal(df_slice) + except Exception: + continue + + if smc_signal is None: + continue + + # #31B: H1 Price vs EMA20 filter + h1_trend = "NEUTRAL" + if df_h1 is not None and len(times_h1) > 0: + h1_idx = 0 + for j, t in enumerate(times_h1): + if t <= current_time: + h1_idx = j + else: + break + if h1_idx > 20: + df_h1_slice = df_h1.head(h1_idx + 1) + h1_trend = self._get_h1_trend(df_h1_slice) + + # Strict H1 filter: signal must match H1 trend direction + if smc_signal.signal_type == "BUY" and h1_trend != "BULLISH": + stats.h1_filtered += 1 + continue + if smc_signal.signal_type == "SELL" and h1_trend != "BEARISH": + stats.h1_filtered += 1 + continue + + recent_df = df_slice.tail(10) + recent_bos = recent_df["bos"].to_list() if "bos" in df_slice.columns else [] + recent_choch = recent_df["choch"].to_list() if "choch" in df_slice.columns else [] + recent_fvg_bull = recent_df["is_fvg_bull"].to_list() if "is_fvg_bull" in df_slice.columns else [] + recent_fvg_bear = recent_df["is_fvg_bear"].to_list() if "is_fvg_bear" in df_slice.columns else [] + recent_obs = recent_df["ob"].to_list() if "ob" in df_slice.columns else [] + + has_bos = 1 in recent_bos or -1 in recent_bos + has_choch = 1 in recent_choch or -1 in recent_choch + has_fvg = any(recent_fvg_bull) or any(recent_fvg_bear) + has_ob = 1 in recent_obs or -1 in recent_obs + + atr_at_entry = 12.0 + if "atr" in df_slice.columns: + atr_val = df_slice.tail(1)["atr"].item() + if atr_val is not None and atr_val > 0: + atr_at_entry = atr_val + + confidence = smc_signal.confidence + ml_agrees = (smc_signal.signal_type == "BUY" and ml_signal == "BUY") or \ + (smc_signal.signal_type == "SELL" and ml_signal == "SELL") + if ml_agrees: + confidence = (smc_signal.confidence + ml_confidence) / 2 + if regime == "high_volatility": + confidence *= 0.9 + + lot_size = self._calculate_lot_size(confidence, regime, trading_mode, lot_mult) + if lot_size <= 0: + continue + + if trading_mode == TradingMode.RECOVERY: + stats.recovery_mode_trades += 1 + + entry_price = smc_signal.entry_price + take_profit_price = smc_signal.take_profit + stop_loss_price = smc_signal.stop_loss + risk = abs(entry_price - stop_loss_price) + rr = abs(take_profit_price - entry_price) / risk if risk > 0 else 0 + + profit, pips, exit_reason, exit_idx, exit_price = self._simulate_trade_exit( + df=df_m15, entry_idx=i, direction=smc_signal.signal_type, + entry_price=entry_price, take_profit=take_profit_price, + stop_loss=stop_loss_price, lot_size=lot_size, + daily_loss_so_far=daily_loss, feature_cols=feature_cols, + ) + + self._ticket_counter += 1 + result = TradeResult.WIN if profit > 0 else (TradeResult.LOSS if profit < 0 else TradeResult.BREAKEVEN) + + trade = SimulatedTrade( + ticket=self._ticket_counter, + entry_time=current_time, + exit_time=times_m15[exit_idx] if exit_idx < len(times_m15) else times_m15[-1], + direction=smc_signal.signal_type, + entry_price=entry_price, exit_price=exit_price, + stop_loss=stop_loss_price, take_profit=take_profit_price, + lot_size=lot_size, profit_usd=profit, profit_pips=pips, + result=result, exit_reason=exit_reason, + smc_confidence=confidence, regime=regime, + session=session_name, signal_reason=smc_signal.reason, + has_bos=has_bos, has_choch=has_choch, + has_fvg=has_fvg, has_ob=has_ob, + atr_at_entry=atr_at_entry, rr_ratio=rr, + trading_mode=trading_mode.value, + h1_trend=h1_trend, + ) + stats.trades.append(trade) + stats.total_trades += 1 + daily_trades += 1 + capital += profit + + if profit > 0: + stats.wins += 1 + stats.total_profit += profit + daily_profit += profit + consecutive_losses = 0 + if trading_mode == TradingMode.RECOVERY: + trading_mode = TradingMode.NORMAL + else: + stats.losses += 1 + stats.total_loss += abs(profit) + daily_loss += abs(profit) + consecutive_losses += 1 + + if daily_loss >= self.max_daily_loss_usd: + trading_mode = TradingMode.STOPPED + stats.daily_limit_stops += 1 + elif consecutive_losses >= 3 or daily_loss >= self.max_daily_loss_usd * 0.6: + trading_mode = TradingMode.PROTECTED + elif consecutive_losses >= 2: + trading_mode = TradingMode.RECOVERY + + if capital > peak_capital: + peak_capital = capital + drawdown_pct = (peak_capital - capital) / peak_capital * 100 + drawdown_usd = peak_capital - capital + if drawdown_pct > stats.max_drawdown: + stats.max_drawdown = drawdown_pct + stats.max_drawdown_usd = drawdown_usd + + stats.equity_curve.append(capital) + last_trade_idx = exit_idx + + if stats.total_trades % 100 == 0: + print(f" {stats.total_trades} trades processed...") + + if stats.total_trades > 0: + stats.win_rate = stats.wins / stats.total_trades * 100 + stats.avg_win = stats.total_profit / stats.wins if stats.wins > 0 else 0 + stats.avg_loss = stats.total_loss / stats.losses if stats.losses > 0 else 0 + stats.avg_trade = (stats.total_profit - stats.total_loss) / stats.total_trades + stats.profit_factor = stats.total_profit / stats.total_loss if stats.total_loss > 0 else float("inf") + win_prob = stats.wins / stats.total_trades + loss_prob = stats.losses / stats.total_trades + stats.expectancy = (win_prob * stats.avg_win) - (loss_prob * stats.avg_loss) + returns = [t.profit_usd for t in stats.trades] + if len(returns) > 1: + avg_return = np.mean(returns) + std_return = np.std(returns) + stats.sharpe_ratio = (avg_return / std_return) * np.sqrt(252) if std_return > 0 else 0 + + return stats + + +# ─── Main ────────────────────────────────────────────────────── + +def main(): + print("=" * 70) + print("XAUBOT AI β€” #32 ML Exit Optimizer") + print("Base: #31B (H1 Price vs EMA20) | Modified: ML-enhanced exits") + print("=" * 70) + + config = get_config() + mt5_conn = MT5Connector( + login=config.mt5_login, password=config.mt5_password, + server=config.mt5_server, path=config.mt5_path, + ) + mt5_conn.connect() + print(f"\nConnected to MT5") + + print("Fetching XAUUSD M15 historical data...") + df_m15 = mt5_conn.get_market_data(symbol="XAUUSD", timeframe="M15", count=50000) + print(f" M15: {len(df_m15)} bars") + + print("Fetching XAUUSD H1 historical data...") + df_h1 = mt5_conn.get_market_data(symbol="XAUUSD", timeframe="H1", count=15000) + print(f" H1: {len(df_h1)} bars") + + times = df_m15["time"].to_list() + print(f" M15 range: {times[0]} to {times[-1]}") + + end_date = datetime.now() + start_date = datetime(2025, 8, 1) + data_start = times[0] + if hasattr(data_start, 'replace') and data_start.tzinfo: + start_date = start_date.replace(tzinfo=data_start.tzinfo) + end_date = end_date.replace(tzinfo=data_start.tzinfo) + if data_start > start_date: + start_date = data_start + timedelta(days=5) + + print(f"\n Backtest period: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}") + + print("\nCalculating M15 indicators...") + features = FeatureEngineer() + smc = SMCAnalyzer(swing_length=config.smc.swing_length, ob_lookback=config.smc.ob_lookback) + df_m15 = features.calculate_all(df_m15, include_ml_features=True) + df_m15 = smc.calculate_all(df_m15) + + regime_detector = MarketRegimeDetector(model_path="models/hmm_regime.pkl") + try: + regime_detector.load() + df_m15 = regime_detector.predict(df_m15) + print(" HMM regime loaded") + except Exception: + print(" [WARN] HMM not available") + + print("Calculating H1 indicators...") + df_h1 = features.calculate_all(df_h1, include_ml_features=False) + print(" All indicators calculated") + + baseline_31b_pnl = 2806.56 + + # ═══ CONFIGS ═══ + configs = [ + ("A: ML reversal 0.65", { + "ml_reversal_threshold": 0.65, + }), + ("B: ML tighten trail", { + "ml_tighten_trail": True, + "ml_tighten_trail_mult": 2.0, + }), + ("C: ML hold boost", { + "ml_hold_boost": True, + "ml_hold_timeout_bars": 24, + }), + ("D: A+B combined", { + "ml_reversal_threshold": 0.65, + "ml_tighten_trail": True, + "ml_tighten_trail_mult": 2.0, + }), + ("E: A+B+C all", { + "ml_reversal_threshold": 0.65, + "ml_tighten_trail": True, + "ml_tighten_trail_mult": 2.0, + "ml_hold_boost": True, + "ml_hold_timeout_bars": 24, + }), + ] + + all_results = [] + + for cfg_name, cfg_params in configs: + print(f"\n{'=' * 60}") + print(f" Config: {cfg_name}") + + bt = MLExitBacktest(**cfg_params) + stats = bt.run(df_m15=df_m15, df_h1=df_h1, start_date=start_date, end_date=end_date, initial_capital=5000.0) + net_pnl = stats.total_profit - stats.total_loss + diff = net_pnl - baseline_31b_pnl + + buy_trades = [t for t in stats.trades if t.direction == "BUY"] + sell_trades = [t for t in stats.trades if t.direction == "SELL"] + buy_wins = sum(1 for t in buy_trades if t.result == TradeResult.WIN) + sell_wins = sum(1 for t in sell_trades if t.result == TradeResult.WIN) + buy_wr = buy_wins / len(buy_trades) * 100 if buy_trades else 0 + sell_wr = sell_wins / len(sell_trades) * 100 if sell_trades else 0 + buy_pnl = sum(t.profit_usd for t in buy_trades) + sell_pnl = sum(t.profit_usd for t in sell_trades) + + print(f"\n [{cfg_name}] Results:") + print(f" Trades: {stats.total_trades} | WR: {stats.win_rate:.1f}%") + print(f" Net PnL: ${net_pnl:,.2f} | PF: {stats.profit_factor:.2f}") + print(f" Max DD: {stats.max_drawdown:.1f}% | Sharpe: {stats.sharpe_ratio:.2f}") + print(f" BUY: {len(buy_trades)}, {buy_wr:.1f}% WR, ${buy_pnl:,.2f}") + print(f" SELL: {len(sell_trades)}, {sell_wr:.1f}% WR, ${sell_pnl:,.2f}") + print(f" vs #31B: ${diff:+,.2f}") + + all_results.append((cfg_name, stats, net_pnl, diff)) + + # ═══ FINAL SUMMARY ═══ + print(f"\n{'=' * 70}") + print("#32 ML EXIT OPTIMIZER β€” ALL CONFIGURATIONS") + print("=" * 70) + + print(f"\n {'Config':<25} {'Trades':>6} {'WR':>6} {'Net PnL':>10} {'DD':>6} {'Sharpe':>7} {'PF':>5} {'vs #31B':>10}") + print(f" {'-' * 85}") + print(f" {'#24B (base) ':<25} {'739':>6} {'80.4%':>6} {'$2,235':>10} {'3.4%':>6} {'2.87':>7} {'1.77':>5} {'β€”':>10}") + print(f" {'#28B (smart BE)':<25} {'741':>6} {'79.8%':>6} {'$2,464':>10} {'3.5%':>6} {'3.23':>7} {'1.83':>5} {'β€”':>10}") + print(f" {'#31B (H1 filter)':<25} {'625':>6} {'81.8%':>6} {'$2,807':>10} {'2.5%':>6} {'3.97':>7} {'2.19':>5} {'β€”':>10}") + for cfg_name, stats, net_pnl, diff in all_results: + print(f" {cfg_name:<25} {stats.total_trades:>6} {stats.win_rate:>5.1f}% ${net_pnl:>9,.2f} {stats.max_drawdown:>5.1f}% {stats.sharpe_ratio:>7.2f} {stats.profit_factor:>5.2f} ${diff:>+9,.2f}") + + best_pnl = -999999 + best_name = "" + best_stats = None + for entry in all_results: + if entry[2] > best_pnl: + best_pnl = entry[2] + best_name = entry[0] + best_stats = entry[1] + + print(f"\n Best config: {best_name}") + + # Exit reasons + print(f"\n Exit Reasons (best config):") + exit_counts = {} + for t in best_stats.trades: + r = t.exit_reason.value + exit_counts[r] = exit_counts.get(r, 0) + 1 + for reason, count in sorted(exit_counts.items(), key=lambda x: -x[1]): + pct = count / best_stats.total_trades * 100 if best_stats.total_trades > 0 else 0 + print(f" {reason:20s}: {count} ({pct:.1f}%)") + + # Save + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "32_ml_exit_results") + os.makedirs(output_dir, exist_ok=True) + + log_path = os.path.join(output_dir, f"ml_exit_{timestamp}.log") + with open(log_path, "w") as f: + f.write(f"#32 ML Exit Optimizer Results\n") + f.write(f"Generated: {datetime.now()}\n") + f.write(f"Base: #31B (625 trades, 81.8% WR, $2,807)\n\n") + for cfg_name, stats, net_pnl, diff in all_results: + f.write(f" {cfg_name}: {stats.total_trades} trades, {stats.win_rate:.1f}% WR, " + f"${net_pnl:,.2f}, DD: {stats.max_drawdown:.1f}%, " + f"Sharpe: {stats.sharpe_ratio:.2f}, PF: {stats.profit_factor:.2f}, " + f"vs #31B: ${diff:+,.2f}\n") + f.write(f"\nBest: {best_name}\n") + print(f" Log saved: {log_path}") + + try: + from backtests.backtest_01_smc_only import generate_xlsx_report as gen_xlsx + xlsx_path = os.path.join(output_dir, f"ml_exit_{timestamp}.xlsx") + gen_xlsx(best_stats, xlsx_path, start_date, end_date) + print(f"\n Report saved: {xlsx_path}") + except Exception as e: + print(f" [WARN] XLSX: {e}") + + mt5_conn.disconnect() + + print(f"\n{'=' * 70}") + print(f"Output: {output_dir}") + print(f" Log: {os.path.basename(log_path)}") + print("=" * 70) + print("Backtest complete!") + + +if __name__ == "__main__": + main() diff --git a/backtests/backtest_live_sync.py b/backtests/backtest_live_sync.py index c0247fc..60c3026 100644 --- a/backtests/backtest_live_sync.py +++ b/backtests/backtest_live_sync.py @@ -4,12 +4,14 @@ Backtest Live Sync - 100% Identical to main_live.py This backtest MUST be identical to live trading logic. SYNCED with Critical & Major Fixes (Feb 2025): -1. SMC Signal: No lookahead bias, current_close entry, min RR 2.0 +1. SMC Signal: No lookahead bias, current_close entry, Fixed RR 1:1.5 2. Pullback Filter: ATR-based thresholds (not hardcoded $2, $1.5) 3. Time-Based Exit: Checks profit_growing + ML agreement before exit -4. Trend Reversal: ATR-based momentum thresholds +4. Trend Reversal: ATR-based momentum thresholds (0.6x multiplier) 5. Signal Persistence: Index-based cleanup (prevents memory leak) 6. Calibrated Confidence: Uses SMC's weighted confidence calculation +7. Dynamic RR: 1.5 (ranging) to 2.0 (strong trend) based on market conditions +8. SELL Filter: Requires ML agreement + 55% confidence Synchronized elements: 1. ML Model: XGBoost with same features, 50-bar train/test gap @@ -25,9 +27,9 @@ Synchronized elements: 6. Position Sizing: Based on ML confidence tiers (0.01-0.02 lot) 7. Trade Cooldown: 20 bars (~5 hours on M15) 8. Exit Logic: - - TP hit (RR 1:2 enforced) + - TP hit (Dynamic RR 1.5-2.0) - ML reversal (>65% opposite signal) - - Trend reversal (ATR-based momentum shift) + - Trend reversal (ATR * 0.6 momentum shift) - Smart timeout (checks profit_growing before exit) - Max loss per trade ($50 default) diff --git a/docker-add-dashboard.bat b/docker-add-dashboard.bat new file mode 100644 index 0000000..a37eb13 --- /dev/null +++ b/docker-add-dashboard.bat @@ -0,0 +1,64 @@ +@echo off +REM Add Dashboard & API to existing Docker setup + +echo. +echo ======================================== +echo Adding Dashboard to Existing Setup +echo ======================================== +echo. + +REM Check if .env exists +if not exist .env ( + echo WARNING: .env file not found! + echo Creating .env from template... + copy .env.docker.example .env + echo. + echo Please edit .env with your MT5 credentials: + echo - MT5_LOGIN + echo - MT5_PASSWORD + echo - MT5_SERVER + echo - MT5_PATH + echo. + pause +) + +REM Check if Docker is running +docker info >nul 2>&1 +if errorlevel 1 ( + echo ERROR: Docker is not running! + echo Please start Docker Desktop and try again. + pause + exit /b 1 +) + +echo [1/4] Checking existing services... +docker ps --filter "name=trading_bot_db" --format "table {{.Names}}\t{{.Status}}" + +echo. +echo [2/4] Building new services API and Dashboard... +docker-compose build trading-api dashboard + +echo. +echo [3/4] Starting new services... +docker-compose up -d trading-api dashboard + +echo. +echo [4/4] Checking all services... +docker-compose ps + +echo. +echo ======================================== +echo Dashboard Added Successfully! +echo ======================================== +echo. +echo Access Points: +echo Dashboard: http://localhost:3000 +echo API: http://localhost:8000 +echo API Docs: http://localhost:8000/docs +echo Database: localhost:5432 (already running) +echo. +echo View logs: +echo docker-compose logs -f dashboard +echo docker-compose logs -f trading-api +echo. +pause diff --git a/docker-compose.yml b/docker-compose.yml index 5d2d20f..fc2f6ec 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,5 +1,3 @@ -version: '3.8' - services: # PostgreSQL Database postgres: @@ -21,8 +19,57 @@ services: interval: 10s timeout: 5s retries: 5 + networks: + - trading_bot_network - # pgAdmin (Optional - for database management) + # Trading Bot API (reads bot_status.json from shared volume) + trading-api: + build: + context: . + dockerfile: Dockerfile + container_name: trading_bot_api + restart: unless-stopped + environment: + TZ: Asia/Jakarta + ports: + - "${API_PORT:-8000}:8000" + volumes: + # Mount data/ so API can read bot_status.json written by the bot on host + - ./data:/app/data:ro + depends_on: + postgres: + condition: service_healthy + networks: + - trading_bot_network + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 10s + + # Web Dashboard (Next.js Frontend) + dashboard: + build: + context: ./web-dashboard + dockerfile: Dockerfile + container_name: trading_bot_dashboard + restart: unless-stopped + ports: + - "${DASHBOARD_PORT:-3000}:3000" + depends_on: + trading-api: + condition: service_healthy + networks: + - trading_bot_network + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + + # pgAdmin (OPTIONAL - for database management) pgadmin: image: dpage/pgadmin4:latest container_name: trading_bot_pgadmin @@ -38,6 +85,8 @@ services: depends_on: postgres: condition: service_healthy + networks: + - trading_bot_network profiles: - admin # Only start with: docker-compose --profile admin up @@ -48,5 +97,6 @@ volumes: name: trading_bot_pgadmin_data networks: - default: + trading_bot_network: name: trading_bot_network + driver: bridge diff --git a/docker-logs.bat b/docker-logs.bat new file mode 100644 index 0000000..4ad724e --- /dev/null +++ b/docker-logs.bat @@ -0,0 +1,18 @@ +@echo off +REM XAUBot AI - Docker Logs Viewer (Windows) + +set SERVICE=%1 +set LINES=%2 +if "%LINES%"=="" set LINES=100 + +if "%SERVICE%"=="" ( + echo Viewing logs for all services... + echo Tip: Use 'docker-logs.bat SERVICE [LINES]' to view specific service + echo Available: trading-api, dashboard, postgres, pgadmin + echo. + docker-compose logs -f --tail=%LINES% +) else ( + echo Viewing logs for: %SERVICE% last %LINES% lines + echo. + docker-compose logs -f --tail=%LINES% %SERVICE% +) diff --git a/docker-logs.sh b/docker-logs.sh new file mode 100644 index 0000000..0093f44 --- /dev/null +++ b/docker-logs.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# XAUBot AI - Docker Logs Viewer + +set -e + +SERVICE="$1" +LINES="${2:-100}" + +if [ -z "$SERVICE" ]; then + echo "πŸ“‹ Viewing logs for all services..." + echo "πŸ’‘ Tip: Use './docker-logs.sh SERVICE [LINES]' to view specific service" + echo " Available: trading-api, dashboard, postgres, pgadmin" + echo "" + docker-compose logs -f --tail=$LINES +else + echo "πŸ“‹ Viewing logs for: $SERVICE (last $LINES lines)" + echo "" + docker-compose logs -f --tail=$LINES $SERVICE +fi diff --git a/docker-remove-dashboard.bat b/docker-remove-dashboard.bat new file mode 100644 index 0000000..6178e00 --- /dev/null +++ b/docker-remove-dashboard.bat @@ -0,0 +1,34 @@ +@echo off +REM Remove Dashboard & API while keeping database + +echo. +echo ======================================== +echo Removing Dashboard & API Services +echo ======================================== +echo. +echo This will stop and remove: +echo - trading_bot_dashboard +echo - trading_bot_api +echo. +echo Database (trading_bot_db) will remain running. +echo. +pause + +echo Stopping services... +docker-compose stop trading-api dashboard + +echo. +echo Removing containers... +docker-compose rm -f trading-api dashboard + +echo. +echo ======================================== +echo Dashboard Removed! +echo ======================================== +echo. +echo Database is still running: +docker ps --filter "name=trading_bot_db" --format "table {{.Names}}\t{{.Status}}" +echo. +echo To add dashboard back: docker-add-dashboard.bat +echo. +pause diff --git a/docker-start.bat b/docker-start.bat new file mode 100644 index 0000000..68f56a8 --- /dev/null +++ b/docker-start.bat @@ -0,0 +1,91 @@ +@echo off +REM XAUBot AI - Docker Start Script (Windows) + +echo. +echo Starting XAUBot AI Docker Services... +echo. + +REM Check if .env exists +if not exist .env ( + echo WARNING: .env file not found! + echo Creating .env from template... + copy .env.docker.example .env + echo. + echo .env created. Please edit it with your MT5 credentials: + echo - MT5_LOGIN + echo - MT5_PASSWORD + echo - MT5_SERVER + echo - MT5_PATH + echo. + pause +) + +REM Check if Docker is running +docker info >nul 2>&1 +if errorlevel 1 ( + echo ERROR: Docker is not running! + echo Please start Docker Desktop and try again. + pause + exit /b 1 +) + +echo Docker is running +echo. + +REM Parse arguments +set PROFILE_FLAG= +if "%1"=="--admin" set PROFILE_FLAG=--profile admin +if "%1"=="-a" set PROFILE_FLAG=--profile admin + +if defined PROFILE_FLAG ( + echo Starting with pgAdmin admin profile... +) else ( + echo Starting core services postgres, api, dashboard... + echo Tip: Use 'docker-start.bat --admin' to include pgAdmin +) + +echo. +echo Pulling latest base images... +docker-compose pull + +echo. +echo Building services... +docker-compose build + +echo. +echo Starting services... +docker-compose %PROFILE_FLAG% up -d + +echo. +echo Waiting for services to be healthy... +timeout /t 10 /nobreak >nul + +echo. +echo Checking service health... +docker-compose ps + +echo. +echo ======================================== +echo Services started successfully! +echo ======================================== +echo. +echo Access Points: +echo Dashboard: http://localhost:3000 +echo API: http://localhost:8000 +echo API Docs: http://localhost:8000/docs +echo Database: localhost:5432 + +if defined PROFILE_FLAG ( + echo pgAdmin: http://localhost:5050 +) + +echo. +echo Useful Commands: +echo View logs: docker-compose logs -f +echo View API logs: docker-compose logs -f trading-api +echo Stop services: docker-compose down +echo Restart: docker-compose restart +echo. +echo Full documentation: DOCKER.md +echo. +pause diff --git a/docker-start.sh b/docker-start.sh new file mode 100644 index 0000000..8468a23 --- /dev/null +++ b/docker-start.sh @@ -0,0 +1,87 @@ +#!/bin/bash +# XAUBot AI - Docker Start Script + +set -e + +echo "πŸš€ Starting XAUBot AI Docker Services..." +echo "" + +# Check if .env exists +if [ ! -f .env ]; then + echo "⚠️ .env file not found!" + echo "πŸ“ Creating .env from template..." + cp .env.docker.example .env + echo "βœ… .env created. Please edit it with your MT5 credentials:" + echo " - MT5_LOGIN" + echo " - MT5_PASSWORD" + echo " - MT5_SERVER" + echo " - MT5_PATH" + echo "" + read -p "Press Enter after editing .env to continue..." +fi + +# Check if Docker is running +if ! docker info > /dev/null 2>&1; then + echo "❌ Docker is not running!" + echo "Please start Docker Desktop and try again." + exit 1 +fi + +echo "βœ… Docker is running" +echo "" + +# Parse command line arguments +PROFILE_FLAG="" +if [ "$1" == "--admin" ] || [ "$1" == "-a" ]; then + PROFILE_FLAG="--profile admin" + echo "πŸ“Š Starting with pgAdmin (admin profile)..." +else + echo "πŸ“Š Starting core services (postgres, api, dashboard)..." + echo "πŸ’‘ Use './docker-start.sh --admin' to include pgAdmin" +fi + +echo "" + +# Pull latest images +echo "πŸ“₯ Pulling latest base images..." +docker-compose pull + +echo "" +echo "πŸ”¨ Building services..." +docker-compose build + +echo "" +echo "🎯 Starting services..." +docker-compose $PROFILE_FLAG up -d + +echo "" +echo "⏳ Waiting for services to be healthy..." +sleep 10 + +# Check health +echo "" +echo "πŸ₯ Checking service health..." +docker-compose ps + +echo "" +echo "βœ… Services started successfully!" +echo "" +echo "πŸ“ Access Points:" +echo " Dashboard: http://localhost:3000" +echo " API: http://localhost:8000" +echo " API Docs: http://localhost:8000/docs" +echo " Database: localhost:5432" + +if [ "$1" == "--admin" ] || [ "$1" == "-a" ]; then + echo " pgAdmin: http://localhost:5050" +fi + +echo "" +echo "πŸ“‹ Useful Commands:" +echo " View logs: docker-compose logs -f" +echo " View API logs: docker-compose logs -f trading-api" +echo " Stop services: docker-compose down" +echo " Restart: docker-compose restart" +echo "" +echo "πŸ“– Full documentation: DOCKER.md" +echo "" diff --git a/docker-status.bat b/docker-status.bat new file mode 100644 index 0000000..5c00f60 --- /dev/null +++ b/docker-status.bat @@ -0,0 +1,55 @@ +@echo off +REM Check status of all trading bot services + +echo. +echo ======================================== +echo Trading Bot Docker Services Status +echo ======================================== +echo. + +docker-compose ps + +echo. +echo ======================================== +echo Service Health Checks +echo ======================================== +echo. + +echo [Database] +docker exec trading_bot_db pg_isready -U trading_bot 2>nul +if %errorlevel%==0 ( + echo Status: HEALTHY +) else ( + echo Status: NOT RUNNING +) + +echo. +echo [API] +curl -s http://localhost:8000/api/health >nul 2>&1 +if %errorlevel%==0 ( + echo Status: HEALTHY + echo URL: http://localhost:8000 +) else ( + echo Status: NOT RUNNING or UNHEALTHY +) + +echo. +echo [Dashboard] +curl -s http://localhost:3000 >nul 2>&1 +if %errorlevel%==0 ( + echo Status: HEALTHY + echo URL: http://localhost:3000 +) else ( + echo Status: NOT RUNNING or UNHEALTHY +) + +echo. +echo ======================================== +echo Quick Commands +echo ======================================== +echo View logs: docker-compose logs -f +echo Restart: docker-compose restart +echo Stop all: docker-compose stop +echo Start all: docker-compose up -d +echo. +pause diff --git a/docker-stop.bat b/docker-stop.bat new file mode 100644 index 0000000..1ffa8fd --- /dev/null +++ b/docker-stop.bat @@ -0,0 +1,44 @@ +@echo off +REM XAUBot AI - Docker Stop Script (Windows) + +echo. +echo Stopping XAUBot AI Docker Services... +echo. + +if "%1"=="--remove" goto remove +if "%1"=="-r" goto remove +if "%1"=="--clean" goto clean +if "%1"=="-c" goto clean +goto stop + +:remove +echo Stopping and removing containers... +docker-compose down +echo. +echo Containers stopped and removed +goto end + +:clean +echo WARNING: This will remove all data including database! +set /p confirm="Are you sure? (yes/no): " +if /i "%confirm%"=="yes" ( + docker-compose down -v + echo. + echo Containers, networks, and volumes removed +) else ( + echo. + echo Cancelled +) +goto end + +:stop +echo Stopping containers data will be preserved... +docker-compose stop +echo. +echo Containers stopped + +:end +echo. +echo To restart: docker-start.bat +echo. +pause diff --git a/docker-stop.sh b/docker-stop.sh new file mode 100644 index 0000000..7c3cd6f --- /dev/null +++ b/docker-stop.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# XAUBot AI - Docker Stop Script + +set -e + +echo "πŸ›‘ Stopping XAUBot AI Docker Services..." +echo "" + +# Parse arguments +if [ "$1" == "--remove" ] || [ "$1" == "-r" ]; then + echo "⚠️ Stopping and removing containers..." + docker-compose down + echo "βœ… Containers stopped and removed" +elif [ "$1" == "--clean" ] || [ "$1" == "-c" ]; then + echo "⚠️ WARNING: This will remove all data including database!" + read -p "Are you sure? (yes/no): " confirm + if [ "$confirm" == "yes" ]; then + docker-compose down -v + echo "βœ… Containers, networks, and volumes removed" + else + echo "❌ Cancelled" + fi +else + echo "πŸ”„ Stopping containers (data will be preserved)..." + docker-compose stop + echo "βœ… Containers stopped" +fi + +echo "" +echo "πŸ“‹ To restart: ./docker-start.sh" +echo "" diff --git a/main_live.py b/main_live.py index eceaf60..cbcabec 100644 --- a/main_live.py +++ b/main_live.py @@ -525,15 +525,14 @@ class TradingBot: # --- H1 Multi-Timeframe Bias (Fix 5) --- def _get_h1_bias(self) -> str: """ - Determine H1 higher-timeframe bias using SMC structure. + Determine H1 higher-timeframe bias using Price vs EMA20 (#31B). Returns: "BULLISH", "BEARISH", or "NEUTRAL" - Logic: + Logic (#31B: backtest +$343, WR 81.8%, Sharpe 3.97, DD 2.5%): - Fetch H1 data (100 bars) - - Run SMC analysis (BOS, CHoCH, OB, FVG) - - Last BOS/CHoCH direction = H1 bias - - If H1 has bullish OB near price β†’ BULLISH zone - - If H1 has bearish OB near price β†’ BEARISH zone + - Calculate EMA20 on H1 closes + - If price > EMA20 * 1.001 β†’ BULLISH (allow BUY only) + - If price < EMA20 * 0.999 β†’ BEARISH (allow SELL only) """ try: # Cache H1 bias β€” only update every 4 candles (1 hour) since H1 changes slowly @@ -550,73 +549,31 @@ class TradingBot: if len(df_h1) < 20: return "NEUTRAL" - # Run SMC on H1 data - from src.smc_polars import SMCAnalyzer - h1_smc = SMCAnalyzer(swing_length=5, fvg_min_gap_pips=5.0, ob_lookback=10) - df_h1 = h1_smc.calculate_all(df_h1) + # #31B: Price vs EMA20 method (backtested winner) + import numpy as np + closes = df_h1["close"].to_list() + current_price = closes[-1] - current_price = df_h1["close"].tail(1).item() + # Calculate EMA20 + period = 20 + multiplier = 2 / (period + 1) + ema = np.mean(closes[:period]) + for val in closes[period:]: + ema = (val - ema) * multiplier + ema + + # Determine bias with small buffer (0.1% threshold) bias = "NEUTRAL" - - # 1. Check last BOS direction on H1 - bos_col = df_h1["bos"].to_list() - last_bos = 0 - for v in reversed(bos_col[-20:]): - if v != 0: - last_bos = v - break - - # 2. Check last CHoCH direction on H1 - choch_col = df_h1["choch"].to_list() - last_choch = 0 - for v in reversed(choch_col[-20:]): - if v != 0: - last_choch = v - break - - # 3. Check if price is near H1 Order Block - ob_col = df_h1["ob"].to_list() - highs = df_h1["high"].to_list() - lows = df_h1["low"].to_list() - near_bullish_ob = False - near_bearish_ob = False - - for i in range(-10, 0): # Last 10 H1 candles - idx = len(ob_col) + i - if idx < 0: - continue - ob_val = ob_col[idx] - if ob_val == 1: # Bullish OB - # Price within OB zone (low to high of that candle) - if lows[idx] <= current_price <= highs[idx] * 1.002: - near_bullish_ob = True - elif ob_val == -1: # Bearish OB - if lows[idx] * 0.998 <= current_price <= highs[idx]: - near_bearish_ob = True - - # Determine bias: BOS > CHoCH > OB proximity - if last_bos == 1: + if current_price > ema * 1.001: bias = "BULLISH" - elif last_bos == -1: + elif current_price < ema * 0.999: bias = "BEARISH" - elif last_choch == 1: - bias = "BULLISH" - elif last_choch == -1: - bias = "BEARISH" - - # OB proximity can override if no clear structure - if bias == "NEUTRAL": - if near_bullish_ob: - bias = "BULLISH" - elif near_bearish_ob: - bias = "BEARISH" # Cache result self._h1_bias_cache = bias self._h1_bias_loop = self._loop_count if self._loop_count % 4 == 0: - logger.info(f"H1 Bias: {bias} (BOS={last_bos}, CHoCH={last_choch}, near_bull_OB={near_bullish_ob}, near_bear_OB={near_bearish_ob})") + logger.info(f"H1 Bias: {bias} (price={current_price:.2f}, EMA20={ema:.2f})") return bias @@ -962,10 +919,18 @@ class TradingBot: if final_signal is None: return - # 10.1 H1 Multi-Timeframe Filter - DISABLED (SMC-only mode) - # H1 bias still logged for dashboard but does NOT block trades + # 10.1 H1 Multi-Timeframe Filter (#31B: Price vs EMA20 β€” backtest +$343) + # BUY only when H1 is BULLISH, SELL only when H1 is BEARISH if h1_bias != "NEUTRAL": - logger.info(f"H1 Bias: {h1_bias} (monitoring only, not blocking)") + if (final_signal.signal_type == "BUY" and h1_bias != "BULLISH") or \ + (final_signal.signal_type == "SELL" and h1_bias != "BEARISH"): + logger.info(f"H1 Filter: {final_signal.signal_type} blocked (H1={h1_bias})") + return + logger.info(f"H1 Filter: {final_signal.signal_type} aligned with H1={h1_bias}") + else: + # H1 NEUTRAL = block both directions (strict mode from backtest) + logger.info(f"H1 Filter: {final_signal.signal_type} blocked (H1=NEUTRAL)") + return # 10.5 Check trade cooldown if self._last_trade_time: diff --git a/requirements-docker.txt b/requirements-docker.txt new file mode 100644 index 0000000..75893ef --- /dev/null +++ b/requirements-docker.txt @@ -0,0 +1,38 @@ +# Docker-compatible requirements (Linux) +# Note: MetaTrader5 is Windows-only, excluded from Docker build + +# Core Data Engine (Rust-based, NOT Pandas) +polars>=1.37.0 +pyarrow>=15.0.0 + +# Machine Learning +xgboost>=2.1.0 +scikit-learn>=1.4.0 +hmmlearn>=0.3.3 +joblib>=1.4.0 + +# Asynchronous Processing +asyncio-throttle>=1.0.2 + +# Logging and Monitoring +loguru>=0.7.2 + +# Environment Variables +python-dotenv>=1.0.1 + +# Numerical Computing +numpy>=1.26.0 + +# HTTP Client +aiohttp>=3.9.0 + +# PostgreSQL Database +psycopg2-binary>=2.9.9 + +# FastAPI for Web Dashboard API +fastapi>=0.109.0 +uvicorn[standard]>=0.27.0 +pydantic>=2.6.0 + +# CORS middleware +python-multipart>=0.0.6 diff --git a/requirements.txt b/requirements.txt index 53ff2db..d565a94 100644 --- a/requirements.txt +++ b/requirements.txt @@ -33,6 +33,11 @@ aiohttp>=3.9.0 # PostgreSQL Database psycopg2-binary>=2.9.9 +# FastAPI for Web Dashboard API +fastapi>=0.109.0 +uvicorn[standard]>=0.27.0 +pydantic>=2.6.0 + # Optional: For backtesting # vectorbt>=0.26.2 diff --git a/src/position_manager.py b/src/position_manager.py index 248c23e..bf20777 100644 --- a/src/position_manager.py +++ b/src/position_manager.py @@ -554,9 +554,10 @@ class SmartPositionManager: trail_start = self.trail_start_pips trail_step = self.trail_step_pips - # 5. Breakeven protection + # 5. Breakeven protection (#28B: smart BE locks profit at 0.5*ATR instead of fixed $2) if pip_profit >= be_pips and current_sl != 0: - breakeven_sl = entry_price + (1 if is_buy else -1) * 2 # 2 points buffer + be_lock_distance = current_atr * 0.5 if (current_atr is not None and current_atr > 0) else 2.0 + breakeven_sl = entry_price + (1 if is_buy else -1) * be_lock_distance if is_buy and current_sl < breakeven_sl: return PositionAction( diff --git a/src/smc_polars.py b/src/smc_polars.py index 7e4419f..2d9b63f 100644 --- a/src/smc_polars.py +++ b/src/smc_polars.py @@ -135,6 +135,80 @@ class SMCAnalyzer: # Cap confidence at 0.85 (never 100% certain) return min(conf, 0.85) + def _calculate_dynamic_rr( + self, + market_structure: int, + has_bullish_break: bool, + has_bearish_break: bool, + has_fvg: bool, + has_ob: bool, + df: Optional[pl.DataFrame] = None, + ) -> float: + """ + Calculate dynamic Risk:Reward ratio based on market conditions. + + Returns RR between 1.5 and 2.0: + - 2.0: Strong trend, high confidence -> let profits run + - 1.5: Ranging/uncertain -> take profit earlier (higher hit rate) + + Factors considered: + 1. Market structure strength (trending vs ranging) + 2. Number of confirmations (BOS, FVG, OB) + 3. Trend strength (multiple BOS in same direction) + 4. Volatility (high vol = lower RR for faster exit) + """ + # Start with base RR + rr = 1.5 # Conservative base + + # === Factor 1: Market Structure === + # Strong trend = higher RR + if market_structure != 0: # Trending (bullish or bearish) + rr += 0.15 + + # === Factor 2: Structure Break Confirmation === + if has_bullish_break or has_bearish_break: + rr += 0.10 # BOS/CHoCH adds confidence + + # === Factor 3: Entry Zone Confirmation === + if has_fvg: + rr += 0.05 # FVG present + if has_ob: + rr += 0.05 # Order Block present + + # === Factor 4: Trend Strength (multiple BOS) === + if df is not None and "bos" in df.columns: + recent_bos = df.tail(20)["bos"].to_list() + bos_count = sum(1 for b in recent_bos if b != 0) + if bos_count >= 3: # Strong trend with multiple breaks + rr += 0.10 + elif bos_count >= 2: + rr += 0.05 + + # === Factor 5: Volatility Adjustment === + # High volatility = reduce RR (take profit faster) + if df is not None and "atr" in df.columns: + atr = df.tail(1)["atr"].item() + if atr is not None: + # Typical XAUUSD ATR is ~$10-15 + if atr > 18: # High volatility + rr -= 0.15 # Take profit faster + elif atr > 15: # Above average volatility + rr -= 0.05 + + # === Factor 6: Check for ranging market (low BOS count) === + if df is not None and "bos" in df.columns: + recent_bos = df.tail(30)["bos"].to_list() + bos_count = sum(1 for b in recent_bos if b != 0) + if bos_count == 0: # No structure breaks = ranging + rr = 1.5 # Use minimum RR in ranging market + + # Clamp RR between 1.5 and 2.0 + rr = max(1.5, min(2.0, rr)) + + logger.debug(f"Dynamic RR: {rr:.2f} (struct={market_structure}, break={has_bullish_break or has_bearish_break}, fvg={has_fvg}, ob={has_ob})") + + return rr + def calculate_all(self, df: pl.DataFrame) -> pl.DataFrame: """ Calculate all SMC indicators. @@ -710,9 +784,11 @@ class SMCAnalyzer: # SL: 1.5-2 ATR distance (protects against noise) min_sl_distance = 1.5 * atr - # TP: Must be at least 2x risk (RR 1:2 minimum) - # With 1.5 ATR SL, TP should be at least 3 ATR - min_rr_ratio = 2.0 # ENFORCED: Minimum Risk:Reward 1:2 + + # === FIXED RR RATIO 1:1.5 === + # Based on backtest analysis: RR 1:2 only hits TP 14% of the time + # RR 1:1.5 is more realistic for higher hit rate + min_rr_ratio = 1.5 # BULLISH SIGNAL CONDITIONS # Need: bullish structure OR recent bullish break, AND (FVG OR OB) @@ -738,7 +814,7 @@ class SMCAnalyzer: if entry - sl < min_sl_distance: sl = entry - min_sl_distance - # FIX: TP at EXACTLY min_rr_ratio (1:2) - ENFORCED + # FIXED TP at RR 1:1.5 risk = entry - sl tp = entry + (risk * min_rr_ratio) @@ -797,7 +873,7 @@ class SMCAnalyzer: if sl - entry < min_sl_distance: sl = entry + min_sl_distance - # FIX: TP at EXACTLY min_rr_ratio (1:2) - ENFORCED + # FIXED TP at RR 1:1.5 risk = sl - entry tp = entry - (risk * min_rr_ratio) diff --git a/start-all.bat b/start-all.bat new file mode 100644 index 0000000..d2a9868 --- /dev/null +++ b/start-all.bat @@ -0,0 +1,52 @@ +@echo off +REM Start API and Dashboard in separate windows + +echo. +echo ======================================== +echo XAUBot AI - Starting All Services +echo ======================================== +echo. + +cd "%~dp0" + +REM Check database +echo [1/3] Checking database... +docker ps --filter "name=trading_bot_db" --format "{{.Names}}: {{.Status}}" 2>nul +if errorlevel 1 ( + echo. + echo WARNING: Database not running! + echo Please start with: docker-compose up -d postgres + echo. + pause + exit /b 1 +) + +echo. +echo [2/3] Starting API... +start "Trading API" cmd /k start-api.bat + +echo Waiting for API to start... +timeout /t 5 /nobreak >nul + +echo. +echo [3/3] Starting Dashboard... +start "Web Dashboard" cmd /k start-dashboard.bat + +echo. +echo ======================================== +echo All Services Started! +echo ======================================== +echo. +echo Access Points: +echo - Dashboard: http://localhost:3000 +echo - API: http://localhost:8000 +echo - API Docs: http://localhost:8000/docs +echo - Database: localhost:5432 +echo. +echo Two windows will open: +echo 1. Trading API (FastAPI) +echo 2. Web Dashboard (Next.js) +echo. +echo Close this window when done. +echo. +pause diff --git a/start-api.bat b/start-api.bat new file mode 100644 index 0000000..7beb0db --- /dev/null +++ b/start-api.bat @@ -0,0 +1,35 @@ +@echo off +REM Start Trading API (FastAPI) + +echo. +echo ======================================== +echo Starting Trading API +echo ======================================== +echo. + +cd "%~dp0" + +REM Check if virtual environment exists +if not exist "venv" ( + echo Creating virtual environment... + python -m venv venv + echo. +) + +REM Activate virtual environment +call venv\Scripts\activate.bat + +REM Install/update dependencies +echo Installing dependencies... +pip install -q fastapi uvicorn pydantic python-dotenv aiohttp + +echo. +echo ======================================== +echo API Starting on http://localhost:8000 +echo ======================================== +echo. +echo Press Ctrl+C to stop +echo. + +REM Start the API +python web-dashboard\api\main.py diff --git a/start-dashboard.bat b/start-dashboard.bat new file mode 100644 index 0000000..2a59da1 --- /dev/null +++ b/start-dashboard.bat @@ -0,0 +1,28 @@ +@echo off +REM Start Next.js Dashboard + +echo. +echo ======================================== +echo Starting Web Dashboard +echo ======================================== +echo. + +cd "%~dp0web-dashboard" + +REM Check if node_modules exists +if not exist "node_modules" ( + echo Installing dependencies... + npm install + echo. +) + +echo. +echo ======================================== +echo Dashboard Starting on http://localhost:3000 +echo ======================================== +echo. +echo Press Ctrl+C to stop +echo. + +REM Start dashboard +npm run dev diff --git a/web-dashboard/.dockerignore b/web-dashboard/.dockerignore new file mode 100644 index 0000000..891557b --- /dev/null +++ b/web-dashboard/.dockerignore @@ -0,0 +1,48 @@ +# Dependencies +node_modules +npm-debug.log* +yarn-debug.log* +yarn-error.log* +.pnpm-debug.log* + +# Next.js +.next/ +out/ +build +dist + +# Testing +coverage + +# Misc +.DS_Store +*.pem + +# Debug +*.log + +# Local env files +.env*.local +.env + +# Vercel +.vercel + +# TypeScript +*.tsbuildinfo +next-env.d.ts + +# IDE +.vscode +.idea + +# Git +.git +.gitignore +README.md + +# Docs +*.md + +# Exclude v3 tailwind config (use @theme in CSS for v4) +tailwind.config.ts diff --git a/web-dashboard/Dockerfile b/web-dashboard/Dockerfile new file mode 100644 index 0000000..02da77a --- /dev/null +++ b/web-dashboard/Dockerfile @@ -0,0 +1,48 @@ +# Multi-stage build for Next.js Dashboard +FROM node:20-alpine AS base + +# Install dependencies only when needed +FROM base AS deps +RUN apk add --no-cache libc6-compat +WORKDIR /app + +# Copy package files +COPY package.json package-lock.json* ./ +RUN npm ci + +# Build the source code +FROM base AS builder +WORKDIR /app +COPY --from=deps /app/node_modules ./node_modules +COPY . . + +# NEXT_PUBLIC_API_URL defaults to http://localhost:8000 in use-trading-data.ts +# The browser fetches from the host machine, not Docker internal network +ENV NEXT_TELEMETRY_DISABLED=1 +RUN npm run build + +# Production image +FROM base AS runner +WORKDIR /app + +ENV NODE_ENV=production +ENV NEXT_TELEMETRY_DISABLED=1 + +RUN addgroup --system --gid 1001 nodejs +RUN adduser --system --uid 1001 nextjs + +# Copy built files +COPY --from=builder /app/public ./public + +# standalone output includes server.js + required node_modules +COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ +COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static + +USER nextjs + +EXPOSE 3000 + +ENV PORT=3000 +ENV HOSTNAME="0.0.0.0" + +CMD ["node", "server.js"] diff --git a/web-dashboard/MIGRATION-SUMMARY.md b/web-dashboard/MIGRATION-SUMMARY.md new file mode 100644 index 0000000..c167b9d --- /dev/null +++ b/web-dashboard/MIGRATION-SUMMARY.md @@ -0,0 +1,217 @@ +# Web Dashboard Styling Migration - Summary + +## βœ… Completed Changes + +### 1. **Created Tailwind Configuration** (`tailwind.config.ts`) +- Custom dark theme colors based on SURGE-AI-Trading design +- Extended color palette with semantic colors (success, warning, danger, info) +- Custom animations (fade-in, slide-up, shimmer) +- Custom font families (Inter for sans, JetBrains Mono for mono) +- Responsive design utilities + +### 2. **Updated Global Styles** (`src/app/globals.css`) +- Dark theme color variables using HSL +- Custom scrollbar styling +- Utility classes for: + - Text gradient effects + - Card variations (glass, hover) + - Badge variants (success, warning, danger, info) + - Button utilities + - Number formatting (font-number) + - Price colors (price-up, price-down, price-neutral) + - Live pulse indicator + - Loading skeleton with shimmer + - Input styling + +### 3. **Enhanced Utility Functions** (`src/lib/utils.ts`) +Added comprehensive utility functions: +- **Formatting:** formatUSD, formatGoldPrice, formatPercent, formatCompact +- **Date/Time:** formatTime, formatDate, formatDateTime, formatDateTimeWIB, getRelativeTime +- **Colors:** getValueColor, getValueBgColor, getSignalColor, getSignalBadgeColor +- **Confidence:** getConfidenceLevel, getConfidenceColor +- **Helpers:** calcProgress, debounce, generateId, sleep + +### 4. **Updated shadcn/ui Components** + +#### Badge Component (`src/components/ui/badge.tsx`) +- Added semantic variants: success, warning, danger, info +- Improved styling consistency +- Better hover effects + +#### Card Component (`src/components/ui/card.tsx`) +- Simplified implementation +- Better border and shadow styling +- Consistent with shadcn/ui patterns + +### 5. **Updated Dashboard Components** + +#### PriceCard (`src/components/dashboard/price-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses `formatGoldPrice` and `getValueColor` +- βœ… Uses `font-number` for numeric displays +- βœ… Uppercase + tracking-wider for title +- βœ… Proper semantic colors + +#### AccountCard (`src/components/dashboard/account-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses `formatUSD` for currency display +- βœ… Uses `getValueColor` for profit/loss +- βœ… Uses `font-number` for numeric displays +- βœ… Proper border styling with `border-border` + +#### SignalCard (`src/components/dashboard/signal-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses `getSignalColor` for signal colors +- βœ… Uses `getConfidenceColor` for confidence display +- βœ… Improved progress bar colors +- βœ… Better probability display formatting +- βœ… Uses `font-number` for numeric displays + +#### SessionCard (`src/components/dashboard/session-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses semantic badge variants (success/danger) +- βœ… Improved golden time indicator with proper colors +- βœ… Better visual hierarchy +- βœ… Uppercase + tracking-wider for title + +#### RiskCard (`src/components/dashboard/risk-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses `formatUSD` for currency display +- βœ… Dynamic risk level colors (success/warning/danger) +- βœ… Better progress bar with semantic colors +- βœ… Improved risk status indicator +- βœ… Uses `font-number` for numeric displays + +#### RegimeCard (`src/components/dashboard/regime-card.tsx`) +- βœ… Uses `glass` effect +- βœ… Uses Badge component for regime display +- βœ… Uses `getConfidenceColor` for confidence display +- βœ… Better regime color mapping (danger/success/info/warning) +- βœ… Uses `font-number` for numeric displays + +#### Header (`src/components/dashboard/header.tsx`) +- βœ… Improved branding with gradient text effect +- βœ… Better badge styling with semantic variants +- βœ… Added primary color accent box for logo +- βœ… Improved time display with proper formatting +- βœ… Responsive design (hide time on small screens) +- βœ… Uses `font-number` for time display + +### 6. **Updated Configuration** (`components.json`) +- Changed style from "new-york" to "default" +- Added `tailwind.config.ts` reference +- Changed baseColor from "neutral" to "slate" +- Added shadcn registry configuration + +### 7. **Created Documentation** + +#### STYLING-GUIDE.md +Comprehensive guide covering: +- Color system with hex and HSL values +- Component styling examples +- Utility classes documentation +- Utility functions API reference +- Typography guidelines +- Responsive design patterns +- Best practices +- Example implementations +- Migration checklist + +## πŸ“ Migration Notes + +### Color Changes +- `text-green-500` β†’ `text-success` +- `text-red-500` β†’ `text-danger` +- `text-amber-500` β†’ `text-warning` +- `text-blue-500` β†’ `text-info` +- `bg-card/50 backdrop-blur` β†’ `glass` + +### Formatting Changes +- Manual `.toLocaleString()` β†’ `formatUSD()`, `formatGoldPrice()` +- Manual percentage formatting β†’ `formatPercent()` +- Manual color logic β†’ `getValueColor()`, `getSignalColor()` + +### Component Improvements +- All cards now use consistent `glass` effect +- All numeric displays use `font-number` class +- All titles use `uppercase tracking-wider` +- Consistent spacing with `space-y-*` utilities +- Better badge variants with semantic colors + +## 🎨 Design System + +### Primary Colors +- **Primary:** #6366f1 (Indigo) - Main brand color +- **Accent:** #8b5cf6 (Purple) - Highlights and accents + +### Semantic Colors +- **Success:** #22c55e (Green) - Positive values, buy signals +- **Warning:** #f59e0b (Orange) - Caution, hold signals +- **Danger:** #ef4444 (Red) - Negative values, sell signals +- **Info:** #3b82f6 (Blue) - Informational content + +### Background Hierarchy +1. `background` (#0a0a0f) - Page background +2. `surface` (#121218) - Card background +3. `surface-light` (#1a1a24) - Nested elements +4. `surface-hover` (#22222e) - Hover states + +## πŸ”„ Remaining Components to Migrate + +The following components still need to be updated: +- [ ] `positions-card.tsx` +- [ ] `log-card.tsx` +- [ ] `price-chart.tsx` +- [ ] `equity-chart.tsx` + +These should follow the same pattern: +1. Add `glass` effect to cards +2. Use utility formatting functions +3. Apply `font-number` to numbers +4. Use semantic colors +5. Apply uppercase + tracking-wider to titles + +## πŸš€ Next Steps + +1. **Test the dashboard:** + ```bash + cd web-dashboard + npm run dev + ``` + +2. **Add more shadcn/ui components as needed:** + ```bash + npx shadcn@latest add tooltip + npx shadcn@latest add dialog + npx shadcn@latest add dropdown-menu + ``` + +3. **Migrate remaining components** using the patterns in STYLING-GUIDE.md + +4. **Consider adding:** + - Toast notifications (sonner) + - Loading states (spinner) + - Error boundaries + - Tooltips for detailed info + +## πŸ“š Resources + +- **STYLING-GUIDE.md** - Complete styling reference +- **tailwind.config.ts** - Theme configuration +- **src/lib/utils.ts** - Utility functions +- **shadcn/ui docs:** https://ui.shadcn.com + +## 🎯 Benefits + +1. **Consistent Design** - All components follow the same design system +2. **Better Maintainability** - Centralized theme and utilities +3. **Improved Readability** - Semantic colors and proper formatting +4. **Type Safety** - TypeScript utility functions +5. **Performance** - Optimized Tailwind CSS with PurgeCSS +6. **Accessibility** - Better color contrast and semantic HTML +7. **Developer Experience** - Clear utility functions and documentation + +--- + +**Migration completed:** Feb 6, 2026 +**By:** Claude Sonnet 4.5 diff --git a/web-dashboard/STYLING-GUIDE.md b/web-dashboard/STYLING-GUIDE.md new file mode 100644 index 0000000..3a84ddd --- /dev/null +++ b/web-dashboard/STYLING-GUIDE.md @@ -0,0 +1,314 @@ +# XAUBot AI Dashboard - Styling Guide + +## Overview + +The dashboard uses **shadcn/ui** components with **Tailwind CSS** and a custom dark theme inspired by nof1.ai and SURGE-AI-Trading. + +## Color System + +### Theme Colors +```typescript +// Background & Surface +background: #0a0a0f (HSL: 222 47% 6%) +surface: #121218 (HSL: 222 25% 7%) +surface-light: #1a1a24 (HSL: 222 20% 10%) +surface-hover: #22222e (HSL: 222 18% 14%) + +// Primary & Accent +primary: #6366f1 (Indigo) +primary-dark: #4f46e5 +accent: #8b5cf6 (Purple) + +// Semantic Colors +success: #22c55e (Green) +warning: #f59e0b (Orange) +danger: #ef4444 (Red) +info: #3b82f6 (Blue) + +// Each semantic color has a background variant with 12.5% opacity +success-bg: #22c55e20 +warning-bg: #f59e0b20 +danger-bg: #ef444420 +info-bg: #3b82f620 +``` + +### Border & Text +```typescript +border: #2a2a3a +border-light: #3a3a4a +foreground: #ffffff +muted-foreground: #a1a1aa +``` + +## Component Styling + +### Cards +```tsx +// Glass effect card (recommended for dashboard) + + ... + ... + + +// Custom card utilities +.glass β†’ bg-surface/80 + backdrop-blur +.card-custom β†’ bg-surface + rounded-xl + border +.card-hover β†’ card-custom + hover effect +``` + +### Badges +```tsx +// Available badge variants +Primary +Success +Warning +Danger +Info +Outline +``` + +### Buttons +```tsx +// Utility classes for buttons +className="btn-primary" β†’ Primary button +className="btn-success" β†’ Success button +className="btn-danger" β†’ Danger button +className="btn-outline" β†’ Outline button +``` + +## Utility Classes + +### Text & Numbers +```css +.font-number β†’ font-mono + tabular-nums (for prices, numbers) +.text-gradient β†’ gradient from primary to accent + +.price-up β†’ text-success +.price-down β†’ text-danger +.price-neutral β†’ text-muted-foreground +``` + +### Animations +```css +.animate-pulse-slow β†’ 3s pulse +.animate-fade-in β†’ fade in effect +.animate-slide-up β†’ slide up effect +.animate-shimmer β†’ shimmer loading effect +.skeleton β†’ loading skeleton with shimmer +``` + +### Live Indicators +```tsx +// Adds a pulsing dot indicator +
LIVE
+``` + +## Utility Functions + +### Formatting +```typescript +import { + formatUSD, // β†’ $1,234.56 + formatGoldPrice, // β†’ 2345.67 + formatPercent, // β†’ +2.45% + formatCompact, // β†’ 1.2M, 3.4K + formatTime, // β†’ 14:23:45 + formatDate, // β†’ Jan 17, 2026 + formatDateTime, // β†’ Jan 17, 2026 14:23:45 + formatDateTimeWIB // β†’ 17 Jan 2026 14:23:45 WIB +} from '@/lib/utils'; +``` + +### Color Helpers +```typescript +import { + getValueColor, // β†’ Returns color class based on +/- + getValueBgColor, // β†’ Returns bg color class based on +/- + getSignalColor, // β†’ Returns color for BUY/SELL/HOLD + getSignalBadgeColor, // β†’ Returns badge variant for signals + getConfidenceColor, // β†’ Returns color based on confidence % + getConfidenceLevel // β†’ Returns "Very High", "High", etc. +} from '@/lib/utils'; +``` + +### Other Utilities +```typescript +import { + cn, // Merge Tailwind classes + calcProgress, // Calculate progress % (capped at 100) + debounce, // Debounce function + generateId, // Generate unique ID + sleep // Async sleep +} from '@/lib/utils'; +``` + +## Usage Examples + +### Price Display +```tsx +import { formatGoldPrice, getValueColor } from '@/lib/utils'; + + + ${formatGoldPrice(price)} + +``` + +### Signal Badge +```tsx +import { getSignalBadgeColor } from '@/lib/utils'; + + + {signal} + +``` + +### Confidence Display +```tsx +import { getConfidenceColor, getConfidenceLevel } from '@/lib/utils'; + +const confidencePercent = confidence * 100; + + + {confidencePercent.toFixed(0)}% - {getConfidenceLevel(confidencePercent)} + +``` + +### Profit/Loss Display +```tsx +import { formatUSD, getValueColor } from '@/lib/utils'; + + + {profit >= 0 ? '+' : ''}{formatUSD(profit)} + +``` + +## Typography + +### Fonts +- **Sans:** Inter, system-ui, sans-serif +- **Mono:** JetBrains Mono, Fira Code, monospace + +### Font Classes +```tsx +Regular text +Code or numbers +Numbers (tabular-nums) +``` + +## Responsive Design + +The dashboard is optimized for desktop but responsive: +```tsx +
Desktop only
+
Mobile only
+
+ Responsive grid +
+``` + +## Adding New shadcn/ui Components + +1. Check available components: +```bash +npx shadcn@latest view @shadcn +``` + +2. Add a component: +```bash +npx shadcn@latest add button +npx shadcn@latest add tooltip +npx shadcn@latest add dialog +``` + +3. Components will be added to `src/components/ui/` + +## Best Practices + +1. **Always use utility functions** for formatting numbers, dates, and colors +2. **Use the `cn()` helper** to merge Tailwind classes +3. **Apply `font-number`** to all numeric displays for consistent monospace formatting +4. **Use semantic colors** (success, warning, danger, info) instead of raw colors +5. **Apply `glass` effect** to cards for depth and consistency +6. **Use uppercase + tracking-wider** for card titles: `className="uppercase tracking-wider"` +7. **Add proper spacing** with `space-y-*` or `gap-*` utilities +8. **Keep contrast in mind** - use `text-muted-foreground` for secondary text + +## Example Card Component + +```tsx +"use client"; + +import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; +import { Badge } from "@/components/ui/badge"; +import { TrendingUp } from "lucide-react"; +import { cn, formatUSD, getValueColor } from "@/lib/utils"; + +interface ExampleCardProps { + title: string; + value: number; + change: number; + status: "active" | "inactive"; +} + +export function ExampleCard({ title, value, change, status }: ExampleCardProps) { + return ( + + + + + {title} + + + +
+ {formatUSD(value)} +
+
+ + {change >= 0 ? '+' : ''}{change.toFixed(2)}% + + + {status} + +
+ + + ); +} +``` + +## Migration Checklist + +When updating existing components to the new styling: + +- [ ] Replace hardcoded colors with theme colors (text-green-500 β†’ text-success) +- [ ] Add `glass` class to cards +- [ ] Use utility formatting functions instead of manual formatting +- [ ] Apply `font-number` to numeric displays +- [ ] Use uppercase + tracking-wider for titles +- [ ] Replace manual color logic with utility functions (getValueColor, etc.) +- [ ] Update Badge variants to semantic ones (success, warning, danger, info) +- [ ] Add proper spacing with space-y or gap utilities +- [ ] Ensure proper use of `cn()` for class merging + +## Resources + +- shadcn/ui docs: https://ui.shadcn.com +- Tailwind CSS docs: https://tailwindcss.com +- Lucide Icons: https://lucide.dev + +--- + +Last updated: Feb 6, 2026 diff --git a/web-dashboard/api/main.py b/web-dashboard/api/main.py index 83bd15a..c731422 100644 --- a/web-dashboard/api/main.py +++ b/web-dashboard/api/main.py @@ -1,42 +1,21 @@ """ -FastAPI Backend for Web Dashboard -================================= +FastAPI Backend for Web Dashboard (Docker-compatible) +===================================================== Serves trading bot status data to the web frontend. + +Reads from data/bot_status.json which is written by main_live.py. +This allows the API to run in Docker without needing MT5 (Windows-only). """ -import sys +import json from pathlib import Path from datetime import datetime from zoneinfo import ZoneInfo -from collections import deque -import asyncio -from typing import Optional -import json -# Add parent directory to path for imports -sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - -from fastapi import FastAPI, WebSocket, WebSocketDisconnect +from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware -from pydantic import BaseModel -from dotenv import load_dotenv -load_dotenv() - -# Import bot components -try: - from src.mt5_connector import MT5Connector - from src.smc_polars import SMCAnalyzer - from src.ml_model import TradingModel - from src.regime_detector import MarketRegimeDetector - from src.session_filter import SessionFilter - from src.feature_eng import FeatureEngineer - from src.config import TradingConfig -except ImportError as e: - print(f"Import error: {e}") - print("Make sure you're running from the correct directory") - -app = FastAPI(title="Trading Bot API", version="1.0.0") +app = FastAPI(title="Trading Bot API", version="2.0.0") # CORS for frontend app.add_middleware( @@ -47,246 +26,68 @@ app.add_middleware( allow_headers=["*"], ) -# Global state -class BotState: - def __init__(self): - self.mt5: Optional[MT5Connector] = None - self.smc: Optional[SMCAnalyzer] = None - self.ml: Optional[TradingModel] = None - self.hmm: Optional[MarketRegimeDetector] = None - self.session: Optional[SessionFilter] = None - self.feature_eng: Optional[FeatureEngineer] = None - self.config: Optional[TradingConfig] = None - self.connected = False +# Status file path (mounted as volume in Docker) +STATUS_FILE = Path("/app/data/bot_status.json") - # History buffers - self.price_history = deque(maxlen=120) - self.equity_history = deque(maxlen=120) - self.balance_history = deque(maxlen=120) - self.logs = deque(maxlen=50) - - # Last known values - self.last_price = 0.0 - self.last_update = None - -state = BotState() - - -def add_log(level: str, message: str): - """Add log entry to buffer""" - now = datetime.now(ZoneInfo("Asia/Jakarta")) - state.logs.append({ - "time": now.strftime("%H:%M:%S"), - "level": level, - "message": message - }) - - -@app.on_event("startup") -async def startup(): - """Initialize bot components on startup""" - add_log("info", "Starting API server...") - - try: - state.config = TradingConfig() - state.mt5 = MT5Connector( - login=state.config.mt5_login, - password=state.config.mt5_password, - server=state.config.mt5_server, - path=state.config.mt5_path, - ) - - if state.mt5.connect(): - state.connected = True - add_log("info", "MT5 connected successfully") - - # Initialize components - state.smc = SMCAnalyzer() - state.ml = TradingModel(model_path="models/xgboost_model") - state.ml.load() - state.hmm = MarketRegimeDetector(model_path="models/hmm_regime") - state.hmm.load() - state.session = SessionFilter() - state.feature_eng = FeatureEngineer() - - add_log("info", f"ML Model loaded ({len(state.ml.feature_names)} features)") - else: - add_log("error", "Failed to connect to MT5") - - except Exception as e: - add_log("error", f"Startup error: {e}") - - -@app.on_event("shutdown") -async def shutdown(): - """Cleanup on shutdown""" - if state.mt5: - state.mt5.disconnect() - add_log("info", "API server stopped") +# Default empty response +DEFAULT_STATUS = { + "timestamp": "00:00:00", + "connected": False, + "price": 0.0, + "spread": 0.0, + "priceChange": 0.0, + "priceHistory": [], + "balance": 0.0, + "equity": 0.0, + "profit": 0.0, + "equityHistory": [], + "balanceHistory": [], + "session": "Unknown", + "isGoldenTime": False, + "canTrade": False, + "dailyLoss": 0.0, + "dailyProfit": 0.0, + "consecutiveLosses": 0, + "riskPercent": 0.0, + "smc": {"signal": "", "confidence": 0.0, "reason": ""}, + "ml": {"signal": "", "confidence": 0.0, "buyProb": 0.0, "sellProb": 0.0}, + "regime": {"name": "", "volatility": 0.0, "confidence": 0.0}, + "positions": [], + "logs": [], +} @app.get("/api/status") async def get_status(): - """Get current trading status""" - wib = ZoneInfo("Asia/Jakarta") - now = datetime.now(wib) - - result = { - "timestamp": now.strftime("%H:%M:%S"), - "connected": state.connected, - "price": 0.0, - "spread": 0.0, - "priceChange": 0.0, - "priceHistory": list(state.price_history), - "balance": 0.0, - "equity": 0.0, - "profit": 0.0, - "equityHistory": list(state.equity_history), - "balanceHistory": list(state.balance_history), - "session": "Unknown", - "isGoldenTime": 19 <= now.hour < 23, - "canTrade": False, - "dailyLoss": 0.0, - "dailyProfit": 0.0, - "consecutiveLosses": 0, - "riskPercent": 0.0, - "smc": {"signal": "", "confidence": 0.0, "reason": ""}, - "ml": {"signal": "", "confidence": 0.0, "buyProb": 0.0, "sellProb": 0.0}, - "regime": {"name": "", "volatility": 0.0, "confidence": 0.0}, - "positions": [], - "logs": list(state.logs), - } - - if not state.connected or not state.mt5: - return result - - try: - # Price - tick = state.mt5.get_tick(state.config.symbol) - if tick: - price = (tick.bid + tick.ask) / 2 - spread = (tick.ask - tick.bid) * 100 - - # Calculate change - price_change = price - state.last_price if state.last_price > 0 else 0 - state.last_price = price - - # Update history - state.price_history.append(price) - - result["price"] = price - result["spread"] = spread - result["priceChange"] = price_change - result["priceHistory"] = list(state.price_history) - - # Account - balance = state.mt5.account_balance or 0 - equity = state.mt5.account_equity or 0 - profit = equity - balance - - state.equity_history.append(equity) - state.balance_history.append(balance) - - result["balance"] = balance - result["equity"] = equity - result["profit"] = profit - result["equityHistory"] = list(state.equity_history) - result["balanceHistory"] = list(state.balance_history) - - # Session - if state.session: - session_info = state.session.get_status_report() - if session_info: - result["session"] = session_info.get('current_session', 'Unknown') - can_trade, _, _ = state.session.can_trade() - result["canTrade"] = can_trade - - # Risk state from file - risk_file = Path("data/risk_state.txt") - if risk_file.exists(): - content = risk_file.read_text() - for line in content.strip().split('\n'): - if ':' in line: - key, value = line.split(':', 1) - key = key.strip() - value = value.strip() - if key == 'daily_loss': - result["dailyLoss"] = float(value) - elif key == 'daily_profit': - result["dailyProfit"] = float(value) - elif key == 'consecutive_losses': - result["consecutiveLosses"] = int(value) - - # Calculate risk percent - max_loss = state.config.capital * (state.config.risk.max_daily_loss / 100) - if max_loss > 0: - result["riskPercent"] = (result["dailyLoss"] / max_loss) * 100 - - # Signals - df = state.mt5.get_market_data(state.config.symbol, state.config.execution_timeframe, 200) - if df is not None and len(df) > 50: - # Feature engineering - df = state.feature_eng.calculate_all(df, include_ml_features=True) - df = state.smc.calculate_all(df) - - # Regime - if state.hmm: - df = state.hmm.predict(df) - regime = state.hmm.get_current_state(df) - if regime: - result["regime"] = { - "name": regime.regime.value.replace('_', ' ').title(), - "volatility": regime.volatility, - "confidence": regime.confidence, - } - - # SMC Signal - smc_signal = state.smc.generate_signal(df) - if smc_signal: - result["smc"] = { - "signal": smc_signal.signal_type, - "confidence": smc_signal.confidence, - "reason": smc_signal.reason or "", - } - - # ML Prediction - if state.ml and state.ml.fitted: - available_features = [f for f in state.ml.feature_names if f in df.columns] - ml_pred = state.ml.predict(df, available_features) - if ml_pred: - result["ml"] = { - "signal": ml_pred.signal, - "confidence": ml_pred.confidence, - "buyProb": ml_pred.probability, - "sellProb": 1.0 - ml_pred.probability, - } - - # Positions - positions = state.mt5.get_open_positions(state.config.symbol) - if positions is not None and not positions.is_empty(): - pos_list = [] - for row in positions.iter_rows(named=True): - pos_list.append({ - "ticket": row.get('ticket', 0), - "type": "BUY" if row.get('type', 0) == 0 else "SELL", - "volume": row.get('volume', 0), - "priceOpen": row.get('price_open', 0), - "profit": row.get('profit', 0), - }) - result["positions"] = pos_list - - state.last_update = now - - except Exception as e: - add_log("error", f"Status error: {str(e)[:50]}") + """Get current trading status from bot's status file.""" + # Try local path first (non-Docker), then Docker path + for path in [STATUS_FILE, Path("data/bot_status.json")]: + if path.exists(): + try: + data = json.loads(path.read_text()) + return data + except (json.JSONDecodeError, OSError): + continue + # No status file β€” bot not running + now = datetime.now(ZoneInfo("Asia/Jakarta")) + result = DEFAULT_STATUS.copy() + result["timestamp"] = now.strftime("%H:%M:%S") + result["logs"] = [ + { + "time": now.strftime("%H:%M:%S"), + "level": "warning", + "message": "Bot is not running β€” waiting for bot_status.json", + } + ] return result @app.get("/api/health") async def health(): - """Health check endpoint""" - return {"status": "ok", "connected": state.connected} + """Health check endpoint.""" + bot_running = STATUS_FILE.exists() or Path("data/bot_status.json").exists() + return {"status": "ok", "bot_running": bot_running} if __name__ == "__main__": diff --git a/web-dashboard/api/requirements.txt b/web-dashboard/api/requirements.txt index 9cbed03..4660ff4 100644 --- a/web-dashboard/api/requirements.txt +++ b/web-dashboard/api/requirements.txt @@ -1,4 +1,2 @@ fastapi>=0.109.0 -uvicorn>=0.27.0 -python-dotenv>=1.0.0 -pydantic>=2.5.0 +uvicorn[standard]>=0.27.0 diff --git a/web-dashboard/components.json b/web-dashboard/components.json index 03909d9..9b52978 100644 --- a/web-dashboard/components.json +++ b/web-dashboard/components.json @@ -1,12 +1,12 @@ { "$schema": "https://ui.shadcn.com/schema.json", - "style": "new-york", + "style": "default", "rsc": true, "tsx": true, "tailwind": { - "config": "", + "config": "tailwind.config.ts", "css": "src/app/globals.css", - "baseColor": "neutral", + "baseColor": "slate", "cssVariables": true, "prefix": "" }, @@ -19,5 +19,7 @@ "lib": "@/lib", "hooks": "@/hooks" }, - "registries": {} + "registries": { + "@shadcn": "https://ui.shadcn.com/r" + } } diff --git a/web-dashboard/next.config.ts b/web-dashboard/next.config.ts index e9ffa30..f9a8d82 100644 --- a/web-dashboard/next.config.ts +++ b/web-dashboard/next.config.ts @@ -1,7 +1,13 @@ import type { NextConfig } from "next"; const nextConfig: NextConfig = { - /* config options here */ + // Enable standalone output for Docker + output: 'standalone', + + // Disable static optimization for dynamic data + experimental: { + // Enable if needed for better performance + } }; export default nextConfig; diff --git a/web-dashboard/src/app/globals.css b/web-dashboard/src/app/globals.css index 6cf72ed..268eef7 100644 --- a/web-dashboard/src/app/globals.css +++ b/web-dashboard/src/app/globals.css @@ -1,125 +1,246 @@ @import "tailwindcss"; -@import "tw-animate-css"; -@custom-variant dark (&:is(.dark *)); +@theme { + /* Background layers β€” soft dark, GitHub Dark Dimmed inspired */ + --color-background: oklch(0.21 0.01 250); + --color-foreground: oklch(0.85 0.01 250); -@theme inline { - --color-background: var(--background); - --color-foreground: var(--foreground); - --font-sans: var(--font-geist-sans); - --font-mono: var(--font-geist-mono); - --color-sidebar-ring: var(--sidebar-ring); - --color-sidebar-border: var(--sidebar-border); - --color-sidebar-accent-foreground: var(--sidebar-accent-foreground); - --color-sidebar-accent: var(--sidebar-accent); - --color-sidebar-primary-foreground: var(--sidebar-primary-foreground); - --color-sidebar-primary: var(--sidebar-primary); - --color-sidebar-foreground: var(--sidebar-foreground); - --color-sidebar: var(--sidebar); - --color-chart-5: var(--chart-5); - --color-chart-4: var(--chart-4); - --color-chart-3: var(--chart-3); - --color-chart-2: var(--chart-2); - --color-chart-1: var(--chart-1); - --color-ring: var(--ring); - --color-input: var(--input); - --color-border: var(--border); - --color-destructive: var(--destructive); - --color-accent-foreground: var(--accent-foreground); - --color-accent: var(--accent); - --color-muted-foreground: var(--muted-foreground); - --color-muted: var(--muted); - --color-secondary-foreground: var(--secondary-foreground); - --color-secondary: var(--secondary); - --color-primary-foreground: var(--primary-foreground); - --color-primary: var(--primary); - --color-popover-foreground: var(--popover-foreground); - --color-popover: var(--popover); - --color-card-foreground: var(--card-foreground); - --color-card: var(--card); - --radius-sm: calc(var(--radius) - 4px); - --radius-md: calc(var(--radius) - 2px); - --radius-lg: var(--radius); - --radius-xl: calc(var(--radius) + 4px); - --radius-2xl: calc(var(--radius) + 8px); - --radius-3xl: calc(var(--radius) + 12px); - --radius-4xl: calc(var(--radius) + 16px); -} - -:root { - --radius: 0.625rem; - --background: oklch(1 0 0); - --foreground: oklch(0.145 0 0); - --card: oklch(1 0 0); - --card-foreground: oklch(0.145 0 0); - --popover: oklch(1 0 0); - --popover-foreground: oklch(0.145 0 0); - --primary: oklch(0.205 0 0); - --primary-foreground: oklch(0.985 0 0); - --secondary: oklch(0.97 0 0); - --secondary-foreground: oklch(0.205 0 0); - --muted: oklch(0.97 0 0); - --muted-foreground: oklch(0.556 0 0); - --accent: oklch(0.97 0 0); - --accent-foreground: oklch(0.205 0 0); - --destructive: oklch(0.577 0.245 27.325); - --border: oklch(0.922 0 0); - --input: oklch(0.922 0 0); - --ring: oklch(0.708 0 0); - --chart-1: oklch(0.646 0.222 41.116); - --chart-2: oklch(0.6 0.118 184.704); - --chart-3: oklch(0.398 0.07 227.392); - --chart-4: oklch(0.828 0.189 84.429); - --chart-5: oklch(0.769 0.188 70.08); - --sidebar: oklch(0.985 0 0); - --sidebar-foreground: oklch(0.145 0 0); - --sidebar-primary: oklch(0.205 0 0); - --sidebar-primary-foreground: oklch(0.985 0 0); - --sidebar-accent: oklch(0.97 0 0); - --sidebar-accent-foreground: oklch(0.205 0 0); - --sidebar-border: oklch(0.922 0 0); - --sidebar-ring: oklch(0.708 0 0); -} - -.dark { - --background: oklch(0.145 0 0); - --foreground: oklch(0.985 0 0); - --card: oklch(0.205 0 0); - --card-foreground: oklch(0.985 0 0); - --popover: oklch(0.205 0 0); - --popover-foreground: oklch(0.985 0 0); - --primary: oklch(0.922 0 0); - --primary-foreground: oklch(0.205 0 0); - --secondary: oklch(0.269 0 0); - --secondary-foreground: oklch(0.985 0 0); - --muted: oklch(0.269 0 0); - --muted-foreground: oklch(0.708 0 0); - --accent: oklch(0.269 0 0); - --accent-foreground: oklch(0.985 0 0); - --destructive: oklch(0.704 0.191 22.216); - --border: oklch(1 0 0 / 10%); - --input: oklch(1 0 0 / 15%); - --ring: oklch(0.556 0 0); - --chart-1: oklch(0.488 0.243 264.376); - --chart-2: oklch(0.696 0.17 162.48); - --chart-3: oklch(0.769 0.188 70.08); - --chart-4: oklch(0.627 0.265 303.9); - --chart-5: oklch(0.645 0.246 16.439); - --sidebar: oklch(0.205 0 0); - --sidebar-foreground: oklch(0.985 0 0); - --sidebar-primary: oklch(0.488 0.243 264.376); - --sidebar-primary-foreground: oklch(0.985 0 0); - --sidebar-accent: oklch(0.269 0 0); - --sidebar-accent-foreground: oklch(0.985 0 0); - --sidebar-border: oklch(1 0 0 / 10%); - --sidebar-ring: oklch(0.556 0 0); + --color-surface: oklch(0.25 0.01 250); + --color-surface-light: oklch(0.30 0.008 250); + --color-surface-hover: oklch(0.34 0.008 250); + + --color-card: oklch(0.25 0.01 250); + --color-card-foreground: oklch(0.85 0.01 250); + + --color-popover: oklch(0.25 0.01 250); + --color-popover-foreground: oklch(0.85 0.01 250); + + /* Primary β€” calm blue */ + --color-primary: oklch(0.62 0.18 255); + --color-primary-foreground: oklch(0.98 0 0); + --color-primary-dark: oklch(0.56 0.18 255); + + --color-secondary: oklch(0.30 0.008 250); + --color-secondary-foreground: oklch(0.85 0.01 250); + + --color-muted: oklch(0.30 0.008 250); + --color-muted-foreground: oklch(0.58 0.01 250); + + --color-accent: oklch(0.62 0.17 290); + --color-accent-foreground: oklch(0.98 0 0); + + --color-destructive: oklch(0.62 0.19 25); + --color-destructive-foreground: oklch(0.98 0 0); + + /* Borders β€” gentle, not harsh */ + --color-border: oklch(0.34 0.008 250); + --color-border-light: oklch(0.40 0.006 250); + + --color-input: oklch(0.34 0.008 250); + --color-ring: oklch(0.62 0.18 255); + + /* Semantic colors β€” softer, less saturated */ + --color-success: oklch(0.68 0.15 155); + --color-success-bg: oklch(0.68 0.15 155 / 0.12); + + --color-warning: oklch(0.76 0.14 75); + --color-warning-bg: oklch(0.76 0.14 75 / 0.12); + + --color-danger: oklch(0.62 0.19 25); + --color-danger-bg: oklch(0.62 0.19 25 / 0.12); + + --color-info: oklch(0.65 0.15 250); + --color-info-bg: oklch(0.65 0.15 250 / 0.12); + + /* Charts */ + --color-chart-1: oklch(0.62 0.18 255); + --color-chart-2: oklch(0.68 0.15 155); + --color-chart-3: oklch(0.76 0.14 75); + --color-chart-4: oklch(0.62 0.17 290); + --color-chart-5: oklch(0.62 0.19 25); + + /* Radius */ + --radius-sm: calc(0.625rem - 4px); + --radius-md: calc(0.625rem - 2px); + --radius-lg: 0.625rem; + --radius-xl: 0.875rem; + + /* Fonts */ + --font-sans: var(--font-inter), 'Inter', system-ui, sans-serif; + --font-mono: var(--font-jetbrains), 'JetBrains Mono', 'Fira Code', monospace; + + /* Animations */ + --animate-pulse-slow: pulse 3s cubic-bezier(0.4, 0, 0.6, 1) infinite; + --animate-fade-in: fadeIn 0.4s ease-out; + --animate-slide-up: slideUp 0.4s ease-out; + --animate-shimmer: shimmer 2s ease-in-out infinite; } +/* ─── Base ─── */ @layer base { * { - @apply border-border outline-ring/50; + border-color: var(--color-border); + outline-color: color-mix(in oklch, var(--color-ring) 50%, transparent); } - body { - @apply bg-background text-foreground; + + html { + color-scheme: dark; + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; + } + + html, body { + @apply bg-background text-foreground font-sans; + height: 100%; + overflow: hidden; + font-feature-settings: "cv02", "cv03", "cv04", "cv11"; } } + +/* ─── Scrollbar ─── */ +::-webkit-scrollbar { + width: 6px; + height: 6px; +} + +::-webkit-scrollbar-track { + background: transparent; +} + +::-webkit-scrollbar-thumb { + background: var(--color-border); + border-radius: 3px; +} + +::-webkit-scrollbar-thumb:hover { + background: var(--color-border-light); +} + +/* ─── Utilities ─── */ +@layer utilities { + /* Glass β€” soft frosted effect */ + .glass { + background: color-mix(in oklch, var(--color-surface) 90%, transparent); + backdrop-filter: blur(10px) saturate(120%); + -webkit-backdrop-filter: blur(10px) saturate(120%); + border: 1px solid color-mix(in oklch, var(--color-border) 50%, transparent); + box-shadow: + 0 1px 2px rgba(0, 0, 0, 0.12), + 0 0 1px rgba(0, 0, 0, 0.08); + transition: border-color 0.2s ease; + } + + .glass:hover { + border-color: var(--color-border-light); + } + + /* Monospace numbers with tabular figures */ + .font-number { + font-family: var(--font-mono); + font-variant-numeric: tabular-nums; + letter-spacing: -0.01em; + } + + /* Section label */ + .section-label { + @apply text-[11px] font-medium text-muted-foreground uppercase; + letter-spacing: 0.1em; + } + + /* Signal border accents */ + .signal-buy { + border-left: 3px solid var(--color-success); + } + + .signal-sell { + border-left: 3px solid var(--color-danger); + } + + .signal-hold { + border-left: 3px solid var(--color-warning); + } + + .signal-none { + border-left: 3px solid var(--color-muted); + } + + /* Badge variants */ + .badge-success { + @apply inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-success-bg text-success; + } + + .badge-warning { + @apply inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-warning-bg text-warning; + } + + .badge-danger { + @apply inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-danger-bg text-danger; + } + + .badge-info { + @apply inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-info-bg text-info; + } + + /* Text gradient */ + .text-gradient { + @apply bg-gradient-to-r from-primary to-accent bg-clip-text text-transparent; + } + + /* Skeleton */ + .skeleton { + @apply bg-surface-light rounded; + animation: shimmer 2s ease-in-out infinite; + background: linear-gradient( + 90deg, + var(--color-surface) 0%, + var(--color-surface-light) 50%, + var(--color-surface) 100% + ); + background-size: 200% 100%; + } + + /* Live pulse dot */ + .pulse-live::before { + content: ''; + @apply absolute -left-2 top-1/2 -translate-y-1/2 w-1.5 h-1.5 bg-success rounded-full; + animation: pulse-dot 2s infinite; + } + + .pulse-stale::before { + content: ''; + @apply absolute -left-2 top-1/2 -translate-y-1/2 w-1.5 h-1.5 bg-warning rounded-full; + animation: pulse-dot 1.5s infinite; + } + + .pulse-dead::before { + content: ''; + @apply absolute -left-2 top-1/2 -translate-y-1/2 w-1.5 h-1.5 bg-danger rounded-full; + } +} + +/* ─── Keyframes ─── */ +@keyframes pulse-dot { + 0%, 100% { + opacity: 1; + transform: translateY(-50%) scale(1); + } + 50% { + opacity: 0.4; + transform: translateY(-50%) scale(1.8); + } +} + +@keyframes fadeIn { + 0% { opacity: 0; transform: translateY(8px); } + 100% { opacity: 1; transform: translateY(0); } +} + +@keyframes slideUp { + 0% { transform: translateY(12px); opacity: 0; } + 100% { transform: translateY(0); opacity: 1; } +} + +@keyframes shimmer { + 0% { background-position: -200% 0; } + 100% { background-position: 200% 0; } +} diff --git a/web-dashboard/src/app/layout.tsx b/web-dashboard/src/app/layout.tsx index 60e1009..b0dacd8 100644 --- a/web-dashboard/src/app/layout.tsx +++ b/web-dashboard/src/app/layout.tsx @@ -1,20 +1,22 @@ import type { Metadata } from "next"; -import { Geist, Geist_Mono } from "next/font/google"; +import { Inter, JetBrains_Mono } from "next/font/google"; import "./globals.css"; -const geistSans = Geist({ - variable: "--font-geist-sans", +const inter = Inter({ + variable: "--font-inter", subsets: ["latin"], + display: "swap", }); -const geistMono = Geist_Mono({ - variable: "--font-geist-mono", +const jetbrainsMono = JetBrains_Mono({ + variable: "--font-jetbrains", subsets: ["latin"], + display: "swap", }); export const metadata: Metadata = { - title: "AI Trading Bot - Monitor", - description: "Real-time monitoring dashboard for AI Trading Bot", + title: "XAUBOT AI β€” Trading Monitor", + description: "Real-time monitoring dashboard for XAUBOT AI Trading Bot", }; export default function RootLayout({ @@ -25,7 +27,7 @@ export default function RootLayout({ return ( {children} diff --git a/web-dashboard/src/app/page.tsx b/web-dashboard/src/app/page.tsx index 5ab2ce2..5a7336e 100644 --- a/web-dashboard/src/app/page.tsx +++ b/web-dashboard/src/app/page.tsx @@ -12,27 +12,41 @@ import { PositionsCard, LogCard, PriceChart, - EquityChart, + SettingsCard, } from "@/components/dashboard"; import { Skeleton } from "@/components/ui/skeleton"; function LoadingSkeleton() { return ( -
- {[...Array(8)].map((_, i) => ( - - ))} +
+
+ {[...Array(4)].map((_, i) => ( + + ))} +
+
+ {[...Array(4)].map((_, i) => ( + + ))} +
+
+ + +
); } function ErrorDisplay({ message }: { message: string }) { return ( -
-
-

Connection Error

-

{message}

-

+

+
+
+ ! +
+

Connection Error

+

{message}

+

Make sure the API server is running on port 8000

@@ -43,19 +57,18 @@ function ErrorDisplay({ message }: { message: string }) { export default function Dashboard() { const { data, loading, error, dataAge } = useTradingData(); - // Format current time for header const now = new Date(); - const wibTime = now.toLocaleTimeString('en-US', { - timeZone: 'Asia/Jakarta', + const wibTime = now.toLocaleTimeString("en-US", { + timeZone: "Asia/Jakarta", hour12: false, - hour: '2-digit', - minute: '2-digit', - second: '2-digit', + hour: "2-digit", + minute: "2-digit", + second: "2-digit", }); if (loading && !data) { return ( -
+
@@ -64,7 +77,7 @@ export default function Dashboard() { if (error && !data) { return ( -
+
@@ -74,86 +87,116 @@ export default function Dashboard() { if (!data) return null; return ( -
+
-
-
- {/* Row 1: Price Chart (full width) */} - +
+ {/* ── Row 1: Status ── */} +
+
+ +
+
+ +
+
+ +
+
+ +
+
- {/* Row 2: Price & Account */} - - + {/* ── Row 2: Signals ── */} +
+
+ +
+
+ +
+
+ +
+
+ {data.settings ? ( + + ) : ( +
+ )} +
+
- {/* Row 3: Session & Risk */} - - - - {/* Row 4: SMC & ML */} - - - - {/* Row 5: Regime & Positions */} - - - - {/* Row 6: Equity Chart (full width) */} - - - {/* Row 7: Log (full width) */} - + {/* ── Row 3: Chart + Sidebar (fills remaining) ── */} +
+
+ +
+
+
+ +
+
+ +
+
- - {/* Footer Status */} -
-
- Last update: {data.timestamp} - AI Trading Bot Monitor v1.0 -
-
); } diff --git a/web-dashboard/src/components/dashboard/account-card.tsx b/web-dashboard/src/components/dashboard/account-card.tsx index 57937d7..8e5d5f3 100644 --- a/web-dashboard/src/components/dashboard/account-card.tsx +++ b/web-dashboard/src/components/dashboard/account-card.tsx @@ -2,39 +2,52 @@ import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { Wallet } from "lucide-react"; +import { Sparkline } from "./sparkline"; +import { cn, formatUSD, getValueColor } from "@/lib/utils"; interface AccountCardProps { balance: number; equity: number; profit: number; + equityHistory?: number[]; } -export function AccountCard({ balance, equity, profit }: AccountCardProps) { +export function AccountCard({ balance, equity, profit, equityHistory = [] }: AccountCardProps) { const isProfit = profit >= 0; return ( - - - - - ACCOUNT + + + + + Account - +
- Balance - ${balance.toLocaleString(undefined, { minimumFractionDigits: 2 })} + Balance + {formatUSD(balance)}
- Equity - ${equity.toLocaleString(undefined, { minimumFractionDigits: 2 })} + Equity + {formatUSD(equity)}
-
- P/L - - {isProfit ? '+' : ''}${profit.toFixed(2)} +
+ P/L + + {isProfit ? "+" : ""}{formatUSD(profit)}
+ + {equityHistory.length > 2 && ( +
+ +
+ )} ); diff --git a/web-dashboard/src/components/dashboard/equity-chart.tsx b/web-dashboard/src/components/dashboard/equity-chart.tsx index 1275f02..2310a6e 100644 --- a/web-dashboard/src/components/dashboard/equity-chart.tsx +++ b/web-dashboard/src/components/dashboard/equity-chart.tsx @@ -17,58 +17,68 @@ export function EquityChart({ equityData, balanceData }: EquityChartProps) { })); return ( - - - - - EQUITY vs BALANCE (2H) + + + + + Equity vs Balance (2H) + {equityData.length > 0 && ( + + ${equityData[equityData.length - 1]?.toFixed(2)} + + )} -
+
{equityData.length > 1 ? ( - + [ `$${value.toFixed(2)}`, - name === 'equity' ? 'Equity' : 'Balance' + name === "equity" ? "Equity" : "Balance", ]} /> - + ) : ( -
- Waiting for data... +
+
+ +

Collecting data...

+
)}
diff --git a/web-dashboard/src/components/dashboard/header.tsx b/web-dashboard/src/components/dashboard/header.tsx index df1f536..328d3d3 100644 --- a/web-dashboard/src/components/dashboard/header.tsx +++ b/web-dashboard/src/components/dashboard/header.tsx @@ -2,6 +2,7 @@ import { Badge } from "@/components/ui/badge"; import { Bot, Wifi, WifiOff, Clock } from "lucide-react"; +import { cn } from "@/lib/utils"; interface HeaderProps { connected: boolean; @@ -10,36 +11,47 @@ interface HeaderProps { } export function Header({ connected, lastUpdate, dataAge }: HeaderProps) { - const isStale = dataAge > 5; + const getDataStatus = () => { + if (dataAge > 45) return { label: "OFFLINE", variant: "danger" as const, dot: "bg-danger" }; + if (dataAge > 15) return { label: `STALE ${dataAge.toFixed(0)}s`, variant: "warning" as const, dot: "bg-warning animate-pulse" }; + return { label: `LIVE ${dataAge.toFixed(1)}s`, variant: "success" as const, dot: "bg-success" }; + }; + + const status = getDataStatus(); return ( -
-
-
- -
-

AI TRADING BOT

- MONITOR +
+
+ {/* Brand */} +
+
+
+

XAUBOT AI

+ + Monitor +
-
- {/* Data Freshness */} - - - {isStale ? `STALE (${dataAge.toFixed(0)}s)` : `LIVE (${dataAge.toFixed(1)}s)`} + {/* Status */} +
+ + + {status.label} - {/* Connection Status */} - + {connected ? : } - {connected ? 'Connected' : 'Disconnected'} + {connected ? "Connected" : "Disconnected"} - {/* Time */} - - {lastUpdate || '--:--:--'} WIB - +
+ + + {lastUpdate || "--:--:--"} + + WIB +
diff --git a/web-dashboard/src/components/dashboard/index.ts b/web-dashboard/src/components/dashboard/index.ts index ce307e3..f8c9499 100644 --- a/web-dashboard/src/components/dashboard/index.ts +++ b/web-dashboard/src/components/dashboard/index.ts @@ -1,11 +1,13 @@ -export { PriceCard } from './price-card'; -export { AccountCard } from './account-card'; -export { SessionCard } from './session-card'; -export { RiskCard } from './risk-card'; -export { SignalCard } from './signal-card'; -export { RegimeCard } from './regime-card'; -export { PositionsCard } from './positions-card'; -export { LogCard } from './log-card'; -export { PriceChart } from './price-chart'; -export { EquityChart } from './equity-chart'; -export { Header } from './header'; +export { PriceCard } from "./price-card"; +export { AccountCard } from "./account-card"; +export { SessionCard } from "./session-card"; +export { RiskCard } from "./risk-card"; +export { SignalCard } from "./signal-card"; +export { RegimeCard } from "./regime-card"; +export { PositionsCard } from "./positions-card"; +export { LogCard } from "./log-card"; +export { PriceChart } from "./price-chart"; +export { EquityChart } from "./equity-chart"; +export { Header } from "./header"; +export { Sparkline } from "./sparkline"; +export { SettingsCard } from "./settings-card"; diff --git a/web-dashboard/src/components/dashboard/log-card.tsx b/web-dashboard/src/components/dashboard/log-card.tsx index 01b7ff3..75c3015 100644 --- a/web-dashboard/src/components/dashboard/log-card.tsx +++ b/web-dashboard/src/components/dashboard/log-card.tsx @@ -1,7 +1,6 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { ScrollArea } from "@/components/ui/scroll-area"; import { Terminal } from "lucide-react"; import type { LogEntry } from "@/types/trading"; @@ -12,48 +11,48 @@ interface LogCardProps { export function LogCard({ logs }: LogCardProps) { const getLevelColor = (level: string) => { switch (level) { - case 'error': return 'text-red-500'; - case 'warn': return 'text-amber-500'; - case 'trade': return 'text-cyan-400'; - default: return 'text-green-400'; + case "error": return "text-danger"; + case "warn": return "text-warning"; + case "trade": return "text-info"; + default: return "text-success"; } }; const getLevelBadge = (level: string) => { switch (level) { - case 'error': return 'ERR'; - case 'warn': return 'WRN'; - case 'trade': return 'TRD'; - default: return 'INF'; + case "error": return "ERR"; + case "warn": return "WRN"; + case "trade": return "TRD"; + default: return "INF"; } }; return ( - - - - - AI ACTIVITY LOG + + + + + Activity - - + +
{logs.length === 0 ? ( -

Waiting for activity...

+

Waiting for activity...

) : ( -
+
{logs.map((log, i) => ( -
- [{log.time}] - - [{getLevelBadge(log.level)}] +
+ {log.time} + + {getLevelBadge(log.level)} - {log.message} + {log.message}
))}
)} - +
); diff --git a/web-dashboard/src/components/dashboard/positions-card.tsx b/web-dashboard/src/components/dashboard/positions-card.tsx index 3ac7a7e..9d403cd 100644 --- a/web-dashboard/src/components/dashboard/positions-card.tsx +++ b/web-dashboard/src/components/dashboard/positions-card.tsx @@ -1,9 +1,9 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { ScrollArea } from "@/components/ui/scroll-area"; import { Badge } from "@/components/ui/badge"; -import { Layers } from "lucide-react"; +import { Layers, Inbox } from "lucide-react"; +import { cn } from "@/lib/utils"; import type { Position } from "@/types/trading"; interface PositionsCardProps { @@ -12,43 +12,55 @@ interface PositionsCardProps { export function PositionsCard({ positions }: PositionsCardProps) { return ( - - - - - OPEN POSITIONS + + + + + Positions {positions.length > 0 && ( - {positions.length} + + {positions.length} + )} - - - {positions.length === 0 ? ( -

- No open positions -

- ) : ( -
- {positions.map((pos) => ( -
-
- - {pos.type} - - {pos.volume} @ {pos.priceOpen.toFixed(2)} -
- = 0 ? 'text-green-500' : 'text-red-500'}`}> - {pos.profit >= 0 ? '+' : ''}${pos.profit.toFixed(2)} + + {positions.length === 0 ? ( +
+ +

No open positions

+
+ ) : ( +
+ {positions.map((pos) => ( +
+
+ + {pos.type} + + + {pos.volume} @ {pos.priceOpen.toFixed(2)}
- ))} -
- )} - + = 0 ? "text-success" : "text-danger" + )}> + {pos.profit >= 0 ? "+" : ""}${pos.profit.toFixed(2)} + +
+ ))} +
+ )} ); diff --git a/web-dashboard/src/components/dashboard/price-card.tsx b/web-dashboard/src/components/dashboard/price-card.tsx index f359899..3cdcc61 100644 --- a/web-dashboard/src/components/dashboard/price-card.tsx +++ b/web-dashboard/src/components/dashboard/price-card.tsx @@ -2,43 +2,58 @@ import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { TrendingUp, TrendingDown } from "lucide-react"; +import { Sparkline } from "./sparkline"; +import { cn, formatGoldPrice, getValueColor } from "@/lib/utils"; interface PriceCardProps { price: number; spread: number; priceChange: number; + priceHistory?: number[]; } -export function PriceCard({ price, spread, priceChange }: PriceCardProps) { +export function PriceCard({ price, spread, priceChange, priceHistory = [] }: PriceCardProps) { const isUp = priceChange >= 0; return ( - - - - PRICE + + + + XAUUSD -
- - {price.toFixed(2)} - - XAUUSD -
-
- {isUp ? ( - - ) : ( - - )} - - {isUp ? '+' : ''}{priceChange.toFixed(2)} +
+ + ${formatGoldPrice(price)}
-

- Spread: {spread.toFixed(1)} pips -

+ +
+
+ {isUp ? ( + + ) : ( + + )} + + {isUp ? "+" : ""}{priceChange.toFixed(2)} + +
+ + {spread.toFixed(1)}p + +
+ + {priceHistory.length > 2 && ( +
+ +
+ )} ); diff --git a/web-dashboard/src/components/dashboard/price-chart.tsx b/web-dashboard/src/components/dashboard/price-chart.tsx index 334b08e..bf0671a 100644 --- a/web-dashboard/src/components/dashboard/price-chart.tsx +++ b/web-dashboard/src/components/dashboard/price-chart.tsx @@ -1,7 +1,7 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { LineChart, Line, XAxis, YAxis, ResponsiveContainer, Tooltip } from "recharts"; +import { AreaChart, Area, XAxis, YAxis, ResponsiveContainer, Tooltip } from "recharts"; import { TrendingUp } from "lucide-react"; interface PriceChartProps { @@ -12,48 +12,58 @@ export function PriceChart({ data }: PriceChartProps) { const chartData = data.map((price, i) => ({ index: i, price })); return ( - - - - - PRICE CHART (2H) + + + + + Price Chart (2H) + {data.length > 0 && ( + + ${data[data.length - 1]?.toFixed(2)} + + )} - -
+ +
{data.length > 1 ? ( - + - + [`$${value.toFixed(2)}`, 'Price']} + labelStyle={{ display: "none" }} + formatter={(value: number) => [`$${value.toFixed(2)}`, "Price"]} /> - - + + - - + ) : ( -
- Waiting for data... +
+
+ +

Collecting data...

+
)}
diff --git a/web-dashboard/src/components/dashboard/regime-card.tsx b/web-dashboard/src/components/dashboard/regime-card.tsx index aaa2599..d75449b 100644 --- a/web-dashboard/src/components/dashboard/regime-card.tsx +++ b/web-dashboard/src/components/dashboard/regime-card.tsx @@ -1,44 +1,74 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { Activity } from "lucide-react"; +import { Badge } from "@/components/ui/badge"; +import { Activity, Clock } from "lucide-react"; +import { cn, getConfidenceColor } from "@/lib/utils"; interface RegimeCardProps { name: string; volatility: number; confidence: number; + updatedAt?: string; + h1Bias?: string; } -export function RegimeCard({ name, volatility, confidence }: RegimeCardProps) { - const getRegimeColor = (regime: string) => { - if (regime.toLowerCase().includes('high')) return 'text-red-500'; - if (regime.toLowerCase().includes('low')) return 'text-green-500'; - return 'text-amber-500'; +export function RegimeCard({ name, volatility, confidence, updatedAt, h1Bias }: RegimeCardProps) { + const getRegimeBadgeVariant = (regime: string) => { + const lower = regime.toLowerCase(); + if (lower.includes("high") || lower.includes("volatile") || lower.includes("crisis")) return "danger"; + if (lower.includes("low") || lower.includes("ranging")) return "success"; + if (lower.includes("trend")) return "info"; + return "warning"; }; + const confidencePercent = confidence * 100; + return ( - - - - - MARKET REGIME + + + + + Market Regime + {updatedAt && ( + + + {updatedAt} + + )} - -
- - {name || '---'} - -
+ + + {name || "Unknown"} +
- Volatility - {volatility.toFixed(2)} + Volatility + {volatility.toFixed(2)}
- Confidence - {(confidence * 100).toFixed(0)}% + Confidence + + {confidencePercent.toFixed(0)}% +
+ {h1Bias && ( +
+ H1 Bias + + {h1Bias === "BULLISH" ? "↑ " : h1Bias === "BEARISH" ? "↓ " : ""}{h1Bias} + +
+ )} ); diff --git a/web-dashboard/src/components/dashboard/risk-card.tsx b/web-dashboard/src/components/dashboard/risk-card.tsx index 4ac1d87..b1237b1 100644 --- a/web-dashboard/src/components/dashboard/risk-card.tsx +++ b/web-dashboard/src/components/dashboard/risk-card.tsx @@ -1,8 +1,8 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { Progress } from "@/components/ui/progress"; -import { ShieldAlert } from "lucide-react"; +import { ShieldAlert, AlertTriangle } from "lucide-react"; +import { cn, formatUSD } from "@/lib/utils"; interface RiskCardProps { dailyLoss: number; @@ -12,42 +12,75 @@ interface RiskCardProps { } export function RiskCard({ dailyLoss, dailyProfit, consecutiveLosses, riskPercent }: RiskCardProps) { - const isHighRisk = riskPercent >= 80; - const isMediumRisk = riskPercent >= 50; + const isCritical = riskPercent >= 100; + const isHigh = riskPercent >= 80; + const isMedium = riskPercent >= 50; + + const getRiskColor = () => { + if (isHigh) return "text-danger"; + if (isMedium) return "text-warning"; + return "text-success"; + }; + + const getSegmentFill = () => { + if (isHigh) return "bg-danger"; + if (isMedium) return "bg-warning"; + return "bg-success"; + }; return ( - - - - - RISK STATUS + + + + + Risk + {isCritical && ( + + + BREACHED + + )} - +
- Daily Loss - ${dailyLoss.toFixed(2)} + Daily Loss + {formatUSD(dailyLoss)}
- Daily Profit - ${dailyProfit.toFixed(2)} + Daily Profit + {formatUSD(dailyProfit)}
- Consec. Losses - {consecutiveLosses} + Consec. Losses + = 3 ? "text-warning" : "text-foreground" + )}> + {consecutiveLosses} +
-
+
- Risk Used - + Risk Used + {riskPercent.toFixed(0)}%
- div]:bg-red-500' : isMediumRisk ? '[&>div]:bg-amber-500' : '[&>div]:bg-green-500'}`} - /> +
+
+
diff --git a/web-dashboard/src/components/dashboard/session-card.tsx b/web-dashboard/src/components/dashboard/session-card.tsx index 41ff17d..f7d3b0f 100644 --- a/web-dashboard/src/components/dashboard/session-card.tsx +++ b/web-dashboard/src/components/dashboard/session-card.tsx @@ -2,7 +2,8 @@ import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { Badge } from "@/components/ui/badge"; -import { Clock, Sparkles } from "lucide-react"; +import { Clock, Sparkles, CheckCircle2, XCircle } from "lucide-react"; +import { cn } from "@/lib/utils"; interface SessionCardProps { session: string; @@ -11,31 +12,48 @@ interface SessionCardProps { } export function SessionCard({ session, isGoldenTime, canTrade }: SessionCardProps) { + const getSessionColor = (s: string) => { + const lower = s.toLowerCase(); + if (lower.includes("london")) return "text-info"; + if (lower.includes("new york") || lower.includes("ny")) return "text-success"; + if (lower.includes("sydney") || lower.includes("asian")) return "text-accent"; + return "text-warning"; + }; + return ( - - - - - SESSION + + + + + Session - -
- {session} + + + {session || "Closed"} + + +
+ + + {isGoldenTime ? "Golden Hour" : "Standard Hours"} +
-
-
- - - GOLDEN: {isGoldenTime ? 'YES' : 'NO'} - -
-
- -
- - {canTrade ? 'CAN TRADE' : 'NO TRADE'} +
+ {canTrade ? ( + + ) : ( + + )} + + {canTrade ? "CAN TRADE" : "NO TRADE"}
diff --git a/web-dashboard/src/components/dashboard/settings-card.tsx b/web-dashboard/src/components/dashboard/settings-card.tsx new file mode 100644 index 0000000..19adc7a --- /dev/null +++ b/web-dashboard/src/components/dashboard/settings-card.tsx @@ -0,0 +1,47 @@ +"use client"; + +import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; +import { Settings2 } from "lucide-react"; +import type { BotSettings } from "@/types/trading"; + +interface SettingsCardProps { + settings: BotSettings; +} + +export function SettingsCard({ settings }: SettingsCardProps) { + const rows: { label: string; value: string }[] = [ + { label: "Mode", value: settings.capitalMode.toUpperCase() }, + { label: "Capital", value: `$${settings.capital.toLocaleString()}` }, + { label: "TF", value: `${settings.executionTF}/${settings.trendTF}` }, + { label: "Risk", value: `${settings.riskPerTrade}%` }, + { label: "Max Loss", value: `${settings.maxDailyLoss}%` }, + { label: "Leverage", value: `1:${settings.leverage}` }, + { label: "Max Lot", value: `${settings.maxLotSize}` }, + { label: "Max Pos", value: `${settings.maxPositions}` }, + { label: "R:R", value: `1:${settings.minRR}` }, + { label: "ML Conf", value: `${(settings.mlConfidence * 100).toFixed(0)}%` }, + { label: "Cooldown", value: `${settings.cooldownSeconds}s` }, + { label: "Symbol", value: settings.symbol }, + ]; + + return ( + + + + + Bot Settings + + + +
+ {rows.map((row) => ( +
+ {row.label} + {row.value} +
+ ))} +
+ + + ); +} diff --git a/web-dashboard/src/components/dashboard/signal-card.tsx b/web-dashboard/src/components/dashboard/signal-card.tsx index 43038e4..055ac9c 100644 --- a/web-dashboard/src/components/dashboard/signal-card.tsx +++ b/web-dashboard/src/components/dashboard/signal-card.tsx @@ -1,8 +1,8 @@ "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { Progress } from "@/components/ui/progress"; -import { Brain, BarChart3 } from "lucide-react"; +import { Brain, BarChart3, Clock } from "lucide-react"; +import { cn, getSignalColor, getConfidenceColor } from "@/lib/utils"; interface SignalCardProps { title: string; @@ -12,54 +12,119 @@ interface SignalCardProps { detail?: string; buyProb?: number; sellProb?: number; + updatedAt?: string; + threshold?: number; + marketQuality?: string; } -export function SignalCard({ title, icon, signal, confidence, detail, buyProb, sellProb }: SignalCardProps) { - const getSignalColor = (sig: string) => { - if (sig === 'BUY') return 'text-green-500'; - if (sig === 'SELL') return 'text-red-500'; - if (sig === 'HOLD') return 'text-amber-500'; - return 'text-muted-foreground'; +export function SignalCard({ + title, + icon, + signal, + confidence, + detail, + buyProb, + sellProb, + updatedAt, + threshold, + marketQuality, +}: SignalCardProps) { + const confidencePercent = confidence * 100; + const hasSignal = signal && signal.toUpperCase() !== "NO SIGNAL" && signal !== ""; + const normalized = (signal || "").toUpperCase(); + + const getBorderClass = () => { + if (normalized === "BUY") return "signal-buy"; + if (normalized === "SELL") return "signal-sell"; + if (normalized === "HOLD") return "signal-hold"; + return "signal-none"; }; - const getProgressColor = (sig: string) => { - if (sig === 'BUY') return '[&>div]:bg-green-500'; - if (sig === 'SELL') return '[&>div]:bg-red-500'; - if (sig === 'HOLD') return '[&>div]:bg-amber-500'; - return ''; + const getBarColor = () => { + if (normalized === "BUY") return "bg-success"; + if (normalized === "SELL") return "bg-danger"; + if (normalized === "HOLD") return "bg-warning"; + return "bg-muted"; }; return ( - - - - {icon === 'smc' ? : } + + + + {icon === "smc" ? : } {title} + {updatedAt && ( + + + {updatedAt} + + )} - -
- - {signal || 'NO SIGNAL'} - -
+ + + {signal || "NO SIGNAL"} + + {/* Confidence bar */}
- Confidence - {(confidence * 100).toFixed(0)}% + Confidence + + {confidencePercent.toFixed(0)}% + {threshold !== undefined && ( + + /{(threshold * 100).toFixed(0)}% + + )} +
- +
+
+ {threshold !== undefined && ( +
+ )} +
+ {threshold !== undefined && ( +
+ + {confidencePercent >= threshold * 100 ? "βœ“ Above" : "βœ— Below"} threshold + + {marketQuality && ( + + Mkt: {marketQuality} + + )} +
+ )}
{detail && ( -

{detail}

+

{detail}

)} {buyProb !== undefined && sellProb !== undefined && ( -
- Buy: {(buyProb * 100).toFixed(0)}% - Sell: {(sellProb * 100).toFixed(0)}% +
+ + Buy + {(buyProb * 100).toFixed(0)}% + + + Sell + {(sellProb * 100).toFixed(0)}% +
)} diff --git a/web-dashboard/src/components/dashboard/sparkline.tsx b/web-dashboard/src/components/dashboard/sparkline.tsx new file mode 100644 index 0000000..8fd0b56 --- /dev/null +++ b/web-dashboard/src/components/dashboard/sparkline.tsx @@ -0,0 +1,33 @@ +"use client"; + +import { LineChart, Line, ResponsiveContainer, YAxis } from "recharts"; + +interface SparklineProps { + data: number[]; + color?: string; + height?: number; +} + +export function Sparkline({ data, color = "#22c55e", height = 28 }: SparklineProps) { + if (data.length < 2) return null; + + const chartData = data.map((v, i) => ({ i, v })); + + return ( +
+ + + + + + +
+ ); +} diff --git a/web-dashboard/src/components/ui/badge.tsx b/web-dashboard/src/components/ui/badge.tsx index beb56ed..acb2b9c 100644 --- a/web-dashboard/src/components/ui/badge.tsx +++ b/web-dashboard/src/components/ui/badge.tsx @@ -1,23 +1,27 @@ import * as React from "react" import { cva, type VariantProps } from "class-variance-authority" -import { Slot } from "radix-ui" - import { cn } from "@/lib/utils" const badgeVariants = cva( - "inline-flex items-center justify-center rounded-full border border-transparent px-2 py-0.5 text-xs font-medium w-fit whitespace-nowrap shrink-0 [&>svg]:size-3 gap-1 [&>svg]:pointer-events-none focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-[3px] aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive transition-[color,box-shadow] overflow-hidden", + "inline-flex items-center justify-center rounded-full border border-transparent px-2.5 py-0.5 text-xs font-medium transition-colors focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2", { variants: { variant: { - default: "bg-primary text-primary-foreground [a&]:hover:bg-primary/90", + default: + "border-transparent bg-primary text-primary-foreground hover:bg-primary/80", secondary: - "bg-secondary text-secondary-foreground [a&]:hover:bg-secondary/90", + "border-transparent bg-secondary text-secondary-foreground hover:bg-secondary/80", destructive: - "bg-destructive text-white [a&]:hover:bg-destructive/90 focus-visible:ring-destructive/20 dark:focus-visible:ring-destructive/40 dark:bg-destructive/60", - outline: - "border-border text-foreground [a&]:hover:bg-accent [a&]:hover:text-accent-foreground", - ghost: "[a&]:hover:bg-accent [a&]:hover:text-accent-foreground", - link: "text-primary underline-offset-4 [a&]:hover:underline", + "border-transparent bg-destructive text-destructive-foreground hover:bg-destructive/80", + outline: "text-foreground border-border", + success: + "border-transparent bg-success-bg text-success hover:bg-success-bg/80", + warning: + "border-transparent bg-warning-bg text-warning hover:bg-warning-bg/80", + danger: + "border-transparent bg-danger-bg text-danger hover:bg-danger-bg/80", + info: + "border-transparent bg-info-bg text-info hover:bg-info-bg/80", }, }, defaultVariants: { @@ -26,22 +30,13 @@ const badgeVariants = cva( } ) -function Badge({ - className, - variant = "default", - asChild = false, - ...props -}: React.ComponentProps<"span"> & - VariantProps & { asChild?: boolean }) { - const Comp = asChild ? Slot.Root : "span" +export interface BadgeProps + extends React.HTMLAttributes, + VariantProps {} +function Badge({ className, variant, ...props }: BadgeProps) { return ( - +
) } diff --git a/web-dashboard/src/components/ui/card.tsx b/web-dashboard/src/components/ui/card.tsx index 681ad98..accd904 100644 --- a/web-dashboard/src/components/ui/card.tsx +++ b/web-dashboard/src/components/ui/card.tsx @@ -1,92 +1,78 @@ import * as React from "react" - import { cn } from "@/lib/utils" -function Card({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} +const Card = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)) +Card.displayName = "Card" -function CardHeader({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} +const CardHeader = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)) +CardHeader.displayName = "CardHeader" -function CardTitle({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} +const CardTitle = React.forwardRef< + HTMLParagraphElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +

+)) +CardTitle.displayName = "CardTitle" -function CardDescription({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} +const CardDescription = React.forwardRef< + HTMLParagraphElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +

+)) +CardDescription.displayName = "CardDescription" -function CardAction({ className, ...props }: React.ComponentProps<"div">) { - return ( -

- ) -} +const CardContent = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)) +CardContent.displayName = "CardContent" -function CardContent({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} +const CardFooter = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)) +CardFooter.displayName = "CardFooter" -function CardFooter({ className, ...props }: React.ComponentProps<"div">) { - return ( -
- ) -} - -export { - Card, - CardHeader, - CardFooter, - CardTitle, - CardAction, - CardDescription, - CardContent, -} +export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent } diff --git a/web-dashboard/src/types/trading.ts b/web-dashboard/src/types/trading.ts index 7b3edc2..6d57042 100644 --- a/web-dashboard/src/types/trading.ts +++ b/web-dashboard/src/types/trading.ts @@ -33,17 +33,20 @@ export interface TradingStatus { signal: string; confidence: number; reason: string; + updatedAt?: string; }; ml: { signal: string; confidence: number; buyProb: number; sellProb: number; + updatedAt?: string; }; regime: { name: string; volatility: number; confidence: number; + updatedAt?: string; }; // Positions @@ -51,6 +54,31 @@ export interface TradingStatus { // Log logs: LogEntry[]; + + // Bot Settings + settings?: BotSettings; + + // Entry Conditions + h1Bias?: string; + dynamicThreshold?: number; + marketQuality?: string; + marketScore?: number; +} + +export interface BotSettings { + capitalMode: string; + capital: number; + riskPerTrade: number; + maxDailyLoss: number; + maxPositions: number; + maxLotSize: number; + leverage: number; + executionTF: string; + trendTF: string; + minRR: number; + mlConfidence: number; + cooldownSeconds: number; + symbol: string; } export interface Position { diff --git a/web-dashboard/tailwind.config.ts b/web-dashboard/tailwind.config.ts new file mode 100644 index 0000000..62d954f --- /dev/null +++ b/web-dashboard/tailwind.config.ts @@ -0,0 +1,117 @@ +import type { Config } from 'tailwindcss' + +const config: Config = { + darkMode: ['class', '.dark'], + content: [ + './src/pages/**/*.{js,ts,jsx,tsx,mdx}', + './src/components/**/*.{js,ts,jsx,tsx,mdx}', + './src/app/**/*.{js,ts,jsx,tsx,mdx}', + ], + theme: { + extend: { + colors: { + // Dark theme colors (nof1.ai / SURGE-AI inspired) + background: 'hsl(var(--background))', + foreground: 'hsl(var(--foreground))', + surface: 'hsl(var(--surface))', + 'surface-light': 'hsl(var(--surface-light))', + 'surface-hover': 'hsl(var(--surface-hover))', + + card: { + DEFAULT: 'hsl(var(--card))', + foreground: 'hsl(var(--card-foreground))', + }, + popover: { + DEFAULT: 'hsl(var(--popover))', + foreground: 'hsl(var(--popover-foreground))', + }, + primary: { + DEFAULT: 'hsl(var(--primary))', + foreground: 'hsl(var(--primary-foreground))', + dark: 'hsl(var(--primary-dark))', + }, + secondary: { + DEFAULT: 'hsl(var(--secondary))', + foreground: 'hsl(var(--secondary-foreground))', + }, + muted: { + DEFAULT: 'hsl(var(--muted))', + foreground: 'hsl(var(--muted-foreground))', + }, + accent: { + DEFAULT: 'hsl(var(--accent))', + foreground: 'hsl(var(--accent-foreground))', + }, + destructive: { + DEFAULT: 'hsl(var(--destructive))', + foreground: 'hsl(var(--destructive-foreground))', + }, + border: { + DEFAULT: 'hsl(var(--border))', + light: 'hsl(var(--border-light))', + }, + input: 'hsl(var(--input))', + ring: 'hsl(var(--ring))', + + // Semantic colors + success: { + DEFAULT: 'hsl(var(--success))', + bg: 'hsl(var(--success-bg))', + }, + warning: { + DEFAULT: 'hsl(var(--warning))', + bg: 'hsl(var(--warning-bg))', + }, + danger: { + DEFAULT: 'hsl(var(--danger))', + bg: 'hsl(var(--danger-bg))', + }, + info: { + DEFAULT: 'hsl(var(--info))', + bg: 'hsl(var(--info-bg))', + }, + + // Chart colors + chart: { + '1': 'hsl(var(--chart-1))', + '2': 'hsl(var(--chart-2))', + '3': 'hsl(var(--chart-3))', + '4': 'hsl(var(--chart-4))', + '5': 'hsl(var(--chart-5))', + }, + }, + borderRadius: { + lg: 'var(--radius)', + md: 'calc(var(--radius) - 2px)', + sm: 'calc(var(--radius) - 4px)', + }, + fontFamily: { + sans: ['Inter', 'system-ui', 'sans-serif'], + mono: ['JetBrains Mono', 'Fira Code', 'monospace'], + }, + animation: { + 'pulse-slow': 'pulse 3s cubic-bezier(0.4, 0, 0.6, 1) infinite', + 'fade-in': 'fadeIn 0.3s ease-in-out', + 'slide-up': 'slideUp 0.3s ease-out', + 'shimmer': 'shimmer 1.5s infinite', + }, + keyframes: { + fadeIn: { + '0%': { opacity: '0' }, + '100%': { opacity: '1' }, + }, + slideUp: { + '0%': { transform: 'translateY(10px)', opacity: '0' }, + '100%': { transform: 'translateY(0)', opacity: '1' }, + }, + shimmer: { + '0%': { backgroundPosition: '-200% 0' }, + '100%': { backgroundPosition: '200% 0' }, + }, + }, + }, + }, + plugins: [], +} + +export default config