refactor: restructure repository and add README, CLAUDE.md, LICENSE

- Move utility scripts to scripts/ (check_market, check_positions, etc.)
- Move test files to tests/ (test_modules, test_mt5_connection, etc.)
- Move deprecated dashboards to archive/
- Move research files to docs/research/
- Add sys.path fix to all moved Python files
- Rewrite README.md with architecture diagram and badges
- Add CLAUDE.md project guide
- Add MIT LICENSE
- Update .gitignore with archive/ pattern

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
GifariKemal
2026-02-06 13:22:46 +07:00
parent 20dc1385c3
commit 0d25548ed5
25 changed files with 364 additions and 82 deletions
+6
View File
@@ -78,3 +78,9 @@ desktop.ini
# Docker
docker/data/
# Archive
archive/
# Junk
nul
+124
View File
@@ -0,0 +1,124 @@
# CLAUDE.md — XAUBot AI
## Project Overview
XAUBot AI is an automated XAUUSD (Gold) trading bot that combines Machine Learning (XGBoost), Smart Money Concepts (SMC), and Hidden Markov Model (HMM) regime detection. It runs on MetaTrader 5 via an async Python loop, executing trades on M15 candles.
## Directory Structure
```
.
├── main_live.py # Main async trading orchestrator
├── train_models.py # Model training script
├── src/ # Core modules
│ ├── config.py # Trading configuration & capital modes
│ ├── mt5_connector.py # MetaTrader 5 connection layer
│ ├── smc_polars.py # Smart Money Concepts (Polars-based)
│ ├── ml_model.py # XGBoost trading model
│ ├── feature_eng.py # Feature engineering (37 features)
│ ├── regime_detector.py # HMM market regime detection
│ ├── risk_engine.py # Risk calculations & validation
│ ├── smart_risk_manager.py # Dynamic risk management
│ ├── session_filter.py # Trading session filter (Sydney/London/NY)
│ ├── position_manager.py # Open position management
│ ├── dynamic_confidence.py # Adaptive confidence thresholds
│ ├── auto_trainer.py # Auto-retraining pipeline
│ ├── news_agent.py # Economic news filtering
│ ├── telegram_notifier.py # Telegram alerts
│ ├── trade_logger.py # Trade logging to DB
│ ├── utils.py # Utility functions
│ └── db/ # Database schemas
├── backtests/ # Backtesting scripts
│ ├── backtest_live_sync.py # Main backtest (synced with live logic)
│ └── archive/ # Old backtest versions
├── scripts/ # Utility scripts
│ ├── check_market.py # Quick SMC market analysis
│ ├── check_positions.py # View open positions
│ ├── check_status.py # Account status check
│ ├── close_positions.py # Close all positions
│ ├── modify_tp.py # Modify take-profit levels
│ └── get_trade_history.py # Pull trade history from MT5
├── tests/ # Test scripts
│ ├── test_modules.py # Module integration tests
│ ├── test_mt5_connection.py# MT5 connection test
│ └── test_risk_settings.py # Risk settings test
├── models/ # Trained models (.pkl)
├── data/ # Market data & trade logs
├── docs/ # Documentation
│ ├── arsitektur-ai/ # Architecture docs (23 components)
│ └── research/ # Research & analysis files
├── web-dashboard/ # Next.js monitoring dashboard
├── docker/ # Docker configuration
└── logs/ # Runtime logs
```
## Key Commands
```bash
# Run the live trading bot
python main_live.py
# Train/retrain ML models
python train_models.py
# Run backtest with threshold tuning
python backtests/backtest_live_sync.py --tune
# Run backtest with specific threshold
python backtests/backtest_live_sync.py --threshold 0.50 --save
# Run module tests
python tests/test_modules.py
# Check market status
python scripts/check_market.py
```
## Architecture
The bot runs an **async candle-based loop** on M15 timeframe:
1. **Data Fetch** — Pull OHLCV from MT5, convert to Polars DataFrame
2. **Feature Engineering** — Calculate 37 technical features (RSI, ATR, MACD, Bollinger, etc.)
3. **SMC Analysis** — Detect Order Blocks, Fair Value Gaps, BOS, CHoCH
4. **Regime Detection** — HMM classifies market as trending/ranging/volatile
5. **ML Prediction** — XGBoost outputs BUY/SELL/HOLD with confidence score
6. **Entry Filtering** — 11 entry filters must pass (session, regime, spread, cooldown, etc.)
7. **Risk Sizing** — ATR-based SL, dynamic position sizing, Kelly criterion
8. **Trade Execution** — Send order to MT5 with broker-level SL/TP
9. **Position Management** — 10 exit conditions (trailing SL, time exit, regime change, etc.)
10. **Logging** — Trade logged to PostgreSQL + Telegram notification
## Tech Stack
- **Python 3.11+** — Main runtime
- **Polars** — Data engine (not Pandas)
- **XGBoost** — ML model for signal prediction
- **hmmlearn** — Hidden Markov Model for regime detection
- **MetaTrader5** — Broker connection
- **asyncio + aiohttp** — Async execution & HTTP
- **loguru** — Structured logging
- **PostgreSQL** — Trade database
- **Next.js** — Web dashboard (optional)
## Configuration
All secrets in `.env`:
- `MT5_LOGIN`, `MT5_PASSWORD`, `MT5_SERVER`, `MT5_PATH` — Broker credentials
- `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID` — Notifications
- `CAPITAL` — Trading capital
- `SYMBOL` — Default: XAUUSD
Capital modes auto-configure risk parameters:
- **MICRO** (<$500): 2% risk/trade
- **SMALL** ($500-$10k): 1.5% risk/trade
- **MEDIUM** ($10k-$100k): 0.5% risk/trade
- **LARGE** (>$100k): 0.25% risk/trade
## Important Notes
- All data processing uses **Polars**, not Pandas
- The bot targets **< 50ms per loop** iteration
- Models are stored as `.pkl` files in `models/`
- Backtest logic is **synced with live** (`backtest_live_sync.py` mirrors `main_live.py`)
- Scripts in `scripts/` and `tests/` include `sys.path` fix so they work from any directory
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025-2026 Gifari Kemal
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+179 -74
View File
@@ -1,19 +1,112 @@
# Smart Automatic Trading BOT + AI
# XAUBot AI
An intelligent automated trading system for XAUUSD (Gold) using Machine Learning and Smart Money Concepts (SMC).
**AI-powered XAUUSD (Gold) trading bot** with XGBoost ML, Smart Money Concepts (SMC), and HMM regime detection for MetaTrader 5.
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![MetaTrader 5](https://img.shields.io/badge/broker-MetaTrader%205-orange.svg)](https://www.metatrader5.com/)
---
## Features
- **ML-Powered Predictions**: XGBoost model with 37 features for market direction prediction
- **Smart Money Concepts (SMC)**: Order Blocks, Fair Value Gaps, Break of Structure, Change of Character
- **HMM Regime Detection**: Hidden Markov Model for market regime classification
- **Dynamic Risk Management**: ATR-based stop loss, position sizing, and smart exits
- **Session-Aware Trading**: Optimized for different market sessions (Sydney, London, NY)
- **Auto-Retraining**: Models automatically retrain based on market conditions
- **Telegram Notifications**: Real-time trade alerts and market updates
- **Web Dashboard**: Real-time monitoring interface
| Feature | Description |
|---------|-------------|
| **XGBoost ML Model** | 37-feature model predicting BUY/SELL/HOLD with calibrated confidence |
| **Smart Money Concepts** | Order Blocks, Fair Value Gaps, Break of Structure, Change of Character |
| **HMM Regime Detection** | 3-state Hidden Markov Model classifying trending/ranging/volatile markets |
| **Dynamic Risk Management** | ATR-based stop loss, Kelly criterion sizing, daily loss limits |
| **Session-Aware Trading** | Optimized for Sydney, London, and New York sessions |
| **Auto-Retraining** | Models automatically retrain when market conditions shift |
| **Telegram Alerts** | Real-time trade notifications and daily summaries |
| **Web Dashboard** | Next.js monitoring interface for live tracking |
## Performance (Backtest Jan 2025 - Feb 2026)
## Architecture
```
┌─────────────────┐
│ MetaTrader 5 │
│ (XAUUSD M15) │
└────────┬─────────┘
│ OHLCV
┌────────▼─────────┐
│ Data Pipeline │
│ (Polars Engine) │
└────────┬─────────┘
┌─────────────────┼─────────────────┐
│ │ │
┌────────▼───────┐ ┌──────▼───────┐ ┌───────▼──────┐
│ SMC Analyzer │ │ Feature Eng │ │ HMM Regime │
│ (OB/FVG/BOS) │ │ (37 features) │ │ Detector │
└────────┬───────┘ └──────┬───────┘ └───────┬──────┘
│ │ │
└─────────────────┼─────────────────┘
┌────────▼─────────┐
│ XGBoost Model │
│ (Signal + Conf) │
└────────┬─────────┘
┌─────────────────┼─────────────────┐
│ │ │
┌────────▼───────┐ ┌──────▼───────┐ ┌───────▼──────┐
│ 11 Entry │ │ Risk Engine │ │ Position │
│ Filters │ │ (ATR + Kelly)│ │ Manager │
└────────┬───────┘ └──────┬───────┘ └───────┬──────┘
│ │ │
└────────────────┼──────────────────┘
┌────────▼─────────┐
│ Trade Execution │
│ (MT5 + Logging) │
└───────────────────┘
```
## Project Structure
```
xaubot-ai/
├── main_live.py # Main async trading orchestrator
├── train_models.py # Model training script
├── src/ # Core modules
│ ├── config.py # Trading configuration & capital modes
│ ├── mt5_connector.py # MetaTrader 5 connection layer
│ ├── smc_polars.py # Smart Money Concepts analyzer
│ ├── ml_model.py # XGBoost trading model
│ ├── feature_eng.py # Feature engineering (37 features)
│ ├── regime_detector.py # HMM market regime detection
│ ├── risk_engine.py # Risk calculations & validation
│ ├── smart_risk_manager.py # Dynamic risk management
│ ├── session_filter.py # Session filter (Sydney/London/NY)
│ ├── position_manager.py # Open position management
│ ├── dynamic_confidence.py # Adaptive confidence thresholds
│ ├── auto_trainer.py # Auto-retraining pipeline
│ ├── news_agent.py # Economic news filtering
│ ├── telegram_notifier.py # Telegram alerts
│ ├── trade_logger.py # Trade logging to DB
│ └── utils.py # Utility functions
├── backtests/ # Backtesting
│ ├── backtest_live_sync.py # Main backtest (synced with live)
│ └── archive/ # Historical versions
├── scripts/ # Utility scripts
│ ├── check_market.py # Quick SMC market analysis
│ ├── check_positions.py # View open positions
│ ├── check_status.py # Account status check
│ ├── close_positions.py # Emergency close all
│ ├── modify_tp.py # Modify take-profit levels
│ └── get_trade_history.py # Pull trade history
├── tests/ # Tests
├── models/ # Trained models (.pkl)
├── data/ # Market data & trade logs
├── docs/ # Documentation
│ ├── arsitektur-ai/ # Architecture docs (23 components)
│ └── research/ # Research & analysis
├── web-dashboard/ # Next.js monitoring dashboard
└── docker/ # Docker configuration
```
## Backtest Results (Jan 2025 - Feb 2026)
| Metric | Value |
|--------|-------|
@@ -24,80 +117,92 @@ An intelligent automated trading system for XAUUSD (Gold) using Machine Learning
| Max Drawdown | 2.2% |
| Sharpe Ratio | 4.83 |
## Architecture
## Installation
### Prerequisites
- Python 3.11+
- MetaTrader 5 terminal (Windows)
- PostgreSQL (optional, for trade logging)
### Setup
```bash
# Clone the repository
git clone https://github.com/GifariKemal/xaubot-ai.git
cd xaubot-ai
# Install dependencies
pip install -r requirements.txt
# Configure environment
cp .env.example .env
# Edit .env with your MT5 credentials and Telegram token
```
├── main_live.py # Main trading orchestrator
├── src/
│ ├── ml_model.py # XGBoost ML model
│ ├── smc_polars.py # Smart Money Concepts analyzer
│ ├── regime_detector.py # HMM market regime detection
│ ├── smart_risk_manager.py # Risk management system
│ ├── feature_eng.py # Feature engineering
│ ├── mt5_connector.py # MetaTrader 5 connection
│ ├── session_filter.py # Trading session management
│ └── ...
├── backtests/
│ ├── backtest_live_sync.py # Main backtest (synced with live)
│ └── archive/ # Historical backtest scripts
├── models/ # Trained ML models (.pkl)
├── data/ # Market data and trade logs
├── docs/ # Documentation
└── web-dashboard/ # Next.js monitoring dashboard
### Configuration
Key settings in `.env`:
```env
# MetaTrader 5
MT5_LOGIN=your_login
MT5_PASSWORD=your_password
MT5_SERVER=your_server
MT5_PATH=C:/Program Files/MetaTrader 5/terminal64.exe
# Telegram Notifications
TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_CHAT_ID=your_chat_id
# Trading
CAPITAL=5000
SYMBOL=XAUUSD
```
### Run
```bash
# Train models first
python train_models.py
# Start the bot
python main_live.py
# Run backtest
python backtests/backtest_live_sync.py --tune
```
## Risk Management
- **ATR-Based Stop Loss**: Minimum 1.5 ATR distance
- **Broker-Level Protection**: Emergency SL at broker level
- **Time-Based Exit**: Max 6 hours per trade
- **Daily Loss Limit**: 5% of capital
- **Position Limit**: Max 2 concurrent positions
| Protection | Details |
|-----------|---------|
| **ATR-Based Stop Loss** | Minimum 1.5x ATR distance |
| **Broker-Level SL** | Emergency SL set at broker level |
| **Position Sizing** | Kelly criterion with capital mode scaling |
| **Daily Loss Limit** | 5% of capital per day |
| **Total Loss Limit** | 10% of capital |
| **Position Limit** | Max 2 concurrent positions |
| **Time-Based Exit** | Max 6 hours per trade |
| **Session Filter** | Only trades during active sessions |
| **Spread Filter** | Rejects trades during high spread |
| **Cooldown** | Minimum time between trades |
## Installation
## Tech Stack
1. Clone the repository
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. Copy `.env.example` to `.env` and configure:
- MT5 credentials
- Telegram bot token
- Database connection
4. Train models:
```bash
python train_models.py
```
5. Run the bot:
```bash
python main_live.py
```
## Configuration
Key settings in `.env`:
- `MT5_LOGIN`, `MT5_PASSWORD`, `MT5_SERVER` - MetaTrader 5 credentials
- `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID` - Telegram notifications
- `CAPITAL` - Trading capital amount
- `SYMBOL` - Trading symbol (default: XAUUSD)
## Backtest
Run backtest with threshold tuning:
```bash
python backtests/backtest_live_sync.py --tune
```
Run backtest with specific threshold:
```bash
python backtests/backtest_live_sync.py --threshold 0.50 --save
```
- **Polars** — High-performance data engine (not Pandas)
- **XGBoost** — Gradient boosted ML model
- **hmmlearn** — Hidden Markov Model for regime detection
- **MetaTrader5** — Broker connection API
- **asyncio** — Async event loop for low-latency execution
- **loguru** — Structured logging
- **PostgreSQL** — Trade database
- **Next.js** — Web dashboard
## Disclaimer
This software is for educational purposes only. Trading involves substantial risk of loss. Past performance is not indicative of future results. Use at your own risk.
> This software is for **educational and research purposes only**. Trading foreign exchange (Forex) and commodities on margin carries a high level of risk and may not be suitable for all investors. Past performance is not indicative of future results. You could lose some or all of your investment. **Use at your own risk.**
## License
MIT License
[MIT License](LICENSE) - Copyright (c) 2025-2026 Gifari Kemal
@@ -1,4 +1,8 @@
"""Quick market analysis script"""
# Run from project root: python scripts/check_market.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
load_dotenv()
@@ -1,5 +1,8 @@
"""Check open positions and account status."""
import os
# Run from project root: python scripts/check_positions.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
load_dotenv()
+4 -1
View File
@@ -1,7 +1,10 @@
"""Quick status check script."""
# Run from project root: python scripts/check_status.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import MetaTrader5 as mt5
from dotenv import load_dotenv
import os
from datetime import datetime, timedelta
load_dotenv()
@@ -1,5 +1,8 @@
"""Close all open positions."""
import os
# Run from project root: python scripts/close_positions.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
load_dotenv()
@@ -1,5 +1,8 @@
"""Get real trading history from MT5."""
import os
# Run from project root: python scripts/get_trade_history.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from datetime import datetime, timedelta
from dotenv import load_dotenv
load_dotenv()
+4 -1
View File
@@ -1,5 +1,8 @@
"""Modify TP of open positions to closer targets."""
import os
# Run from project root: python scripts/modify_tp.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
load_dotenv()
+3 -1
View File
@@ -3,8 +3,10 @@ Module Test Script
==================
Tests all modules to ensure they work correctly.
"""
# Run from project root: python tests/test_modules.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import sys
import polars as pl
import numpy as np
from datetime import datetime, timedelta
@@ -3,9 +3,10 @@ Test MT5 Connection
===================
Quick test to verify MT5 connection and data retrieval.
"""
# Run from project root: python tests/test_mt5_connection.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import os
import sys
from loguru import logger
# Configure logging
@@ -1,4 +1,8 @@
"""Test new risk settings."""
# Run from project root: python tests/test_risk_settings.py
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src.smart_risk_manager import create_smart_risk_manager
# Test dengan modal $50