Update
This commit is contained in:
zhutoutoutousan
@@ -0,0 +1,240 @@
|
||||
# RSI Divergence ONNX Trading System for BTCUSD
|
||||
|
||||
A complete AI-powered trading system that uses machine learning to identify genuine RSI (Relative Strength Index) divergences and execute trades on MetaTrader 5.
|
||||
|
||||
## Overview
|
||||
|
||||
This system trains a neural network to classify RSI divergences into 5 categories:
|
||||
- **NONE** (0): No divergence detected
|
||||
- **REGULAR_BULLISH** (1): Price makes lower low, RSI makes higher low (reversal signal)
|
||||
- **REGULAR_BEARISH** (2): Price makes higher high, RSI makes lower high (reversal signal)
|
||||
- **HIDDEN_BULLISH** (3): Price makes higher low, RSI makes lower low (continuation signal)
|
||||
- **HIDDEN_BEARISH** (4): Price makes lower high, RSI makes higher high (continuation signal)
|
||||
|
||||
The trained model is exported to ONNX format and used in a MetaTrader 5 Expert Advisor for live trading.
|
||||
|
||||
## Features
|
||||
|
||||
- **Advanced Divergence Detection**: Identifies both regular and hidden RSI divergences
|
||||
- **Machine Learning Classification**: Uses LSTM neural network to learn genuine divergence patterns
|
||||
- **ONNX Integration**: Model runs efficiently in MetaTrader 5 using ONNX Runtime
|
||||
- **Comprehensive Backtesting**: Test model performance on historical data
|
||||
- **Risk Management**: Built-in stop loss, take profit, trailing stop, and position time limits
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
ai/rsi-divergence/
|
||||
├── rsi_divergence_detector.py # Core divergence detection module
|
||||
├── collect_btcusd_data.py # Data collection and labeling script
|
||||
├── train_onnx_model.py # Model training script
|
||||
├── backtest_model.py # Backtesting script
|
||||
├── RSIDivergence_EA.mq5 # MetaTrader 5 Expert Advisor
|
||||
├── requirements.txt # Python dependencies
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Install Python Dependencies
|
||||
|
||||
```bash
|
||||
cd ai/rsi-divergence
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Setup MetaTrader 5
|
||||
|
||||
1. Install MetaTrader 5
|
||||
2. Enable automated trading in MT5 settings
|
||||
3. Copy `RSIDivergence_EA.mq5` to `MT5_Data_Folder/MQL5/Experts/`
|
||||
4. Compile the EA in MetaEditor
|
||||
|
||||
## Usage
|
||||
|
||||
### Step 1: Collect and Label Data
|
||||
|
||||
Collect BTCUSD historical data and label it with RSI divergence signals:
|
||||
|
||||
```bash
|
||||
python collect_btcusd_data.py \
|
||||
--symbol BTCUSD \
|
||||
--timeframe H1 \
|
||||
--days 365 \
|
||||
--rsi-period 14 \
|
||||
--output data \
|
||||
--min-strength 0.15
|
||||
```
|
||||
|
||||
This will:
|
||||
- Fetch BTCUSD data from MetaTrader 5
|
||||
- Calculate RSI and other technical indicators
|
||||
- Detect and label RSI divergences
|
||||
- Save labeled data to `data/BTCUSD_H1_labeled.csv`
|
||||
|
||||
### Step 2: Train the Model
|
||||
|
||||
Train the neural network to classify divergences:
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py \
|
||||
--data data/BTCUSD_H1_labeled.csv \
|
||||
--lookback 60 \
|
||||
--epochs 50 \
|
||||
--batch-size 32 \
|
||||
--output models
|
||||
```
|
||||
|
||||
This will:
|
||||
- Load labeled data
|
||||
- Train an LSTM-based classification model
|
||||
- Export model to ONNX format
|
||||
- Save scaler and feature list for inference
|
||||
|
||||
Output files:
|
||||
- `models/BTCUSD_H1_rsi_divergence_model.onnx` - ONNX model
|
||||
- `models/BTCUSD_H1_rsi_divergence_scaler.pkl` - Feature scaler
|
||||
- `models/BTCUSD_H1_rsi_divergence_features.pkl` - Feature list
|
||||
|
||||
### Step 3: Backtest the Model
|
||||
|
||||
Test the trained model on historical data:
|
||||
|
||||
```bash
|
||||
python backtest_model.py \
|
||||
--model models/BTCUSD_H1_rsi_divergence_model.onnx \
|
||||
--scaler models/BTCUSD_H1_rsi_divergence_scaler.pkl \
|
||||
--features models/BTCUSD_H1_rsi_divergence_features.pkl \
|
||||
--symbol BTCUSD \
|
||||
--timeframe H1 \
|
||||
--days 90 \
|
||||
--balance 10000 \
|
||||
--lot-size 0.01 \
|
||||
--min-confidence 0.7
|
||||
```
|
||||
|
||||
This will:
|
||||
- Load the trained model
|
||||
- Run backtest on historical data
|
||||
- Generate performance metrics
|
||||
- Save trade history to CSV
|
||||
|
||||
### Step 4: Deploy to MetaTrader 5
|
||||
|
||||
1. **Copy Model Files**:
|
||||
- Copy `BTCUSD_H1_rsi_divergence_model.onnx` to `MT5_Data_Folder/MQL5/Files/models/`
|
||||
- Create the `models` folder if it doesn't exist
|
||||
|
||||
2. **Attach EA to Chart**:
|
||||
- Open BTCUSD chart in MT5
|
||||
- Drag `RSIDivergence_EA` from Navigator to chart
|
||||
- Configure parameters:
|
||||
- `InpModelPath`: Path to ONNX model (e.g., `models\\BTCUSD_H1_rsi_divergence_model.onnx`)
|
||||
- `InpMinConfidence`: Minimum confidence threshold (0.7 recommended)
|
||||
- `InpLotSize`: Position size
|
||||
- `InpStopLoss`: Stop loss in pips
|
||||
- `InpTakeProfit`: Take profit in pips
|
||||
|
||||
3. **Enable AutoTrading**:
|
||||
- Click "AutoTrading" button in MT5 toolbar
|
||||
- EA will start analyzing and trading automatically
|
||||
|
||||
## Parameters
|
||||
|
||||
### Data Collection Parameters
|
||||
|
||||
- `--symbol`: Trading symbol (default: BTCUSD)
|
||||
- `--timeframe`: Timeframe (M1, M5, M15, M30, H1, H4, D1)
|
||||
- `--days`: Number of days of historical data
|
||||
- `--rsi-period`: RSI calculation period (default: 14)
|
||||
- `--min-strength`: Minimum divergence strength (0-1)
|
||||
|
||||
### Training Parameters
|
||||
|
||||
- `--data`: Path to labeled CSV file
|
||||
- `--lookback`: Number of bars to look back (default: 60)
|
||||
- `--epochs`: Training epochs (default: 50)
|
||||
- `--batch-size`: Batch size (default: 32)
|
||||
|
||||
### EA Parameters
|
||||
|
||||
**ONNX Model Settings**:
|
||||
- `InpModelPath`: Path to ONNX model file
|
||||
- `InpLookback`: Lookback period (must match training)
|
||||
- `InpMinConfidence`: Minimum confidence to trade (0-1)
|
||||
|
||||
**Trading Settings**:
|
||||
- `InpLotSize`: Position size
|
||||
- `InpMagicNumber`: Unique identifier for EA trades
|
||||
- `InpStopLoss`: Stop loss in pips (0 = disabled)
|
||||
- `InpTakeProfit`: Take profit in pips (0 = disabled)
|
||||
- `InpMaxBarsInTrade`: Maximum bars to hold position (0 = disabled)
|
||||
|
||||
**Divergence Filter**:
|
||||
- `InpUseRegularBullish`: Enable regular bullish divergence trades
|
||||
- `InpUseRegularBearish`: Enable regular bearish divergence trades
|
||||
- `InpUseHiddenBullish`: Enable hidden bullish divergence trades
|
||||
- `InpUseHiddenBearish`: Enable hidden bearish divergence trades
|
||||
|
||||
**Risk Management**:
|
||||
- `InpUseTrailingStop`: Enable trailing stop
|
||||
- `InpTrailingStopPips`: Trailing stop distance in pips
|
||||
- `InpTrailingStepPips`: Trailing stop step in pips
|
||||
|
||||
## Understanding RSI Divergences
|
||||
|
||||
### Regular Divergences (Reversal Signals)
|
||||
|
||||
- **Bullish**: Price makes lower low, RSI makes higher low → Potential upward reversal
|
||||
- **Bearish**: Price makes higher high, RSI makes lower high → Potential downward reversal
|
||||
|
||||
### Hidden Divergences (Continuation Signals)
|
||||
|
||||
- **Bullish**: Price makes higher low, RSI makes lower low → Trend continuation upward
|
||||
- **Bearish**: Price makes lower high, RSI makes higher high → Trend continuation downward
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
1. **Data Quality**: Use more historical data (1-2 years) for better training
|
||||
2. **Feature Engineering**: Experiment with additional technical indicators
|
||||
3. **Model Tuning**: Adjust LSTM architecture, dropout rates, learning rate
|
||||
4. **Confidence Threshold**: Higher threshold = fewer but higher quality trades
|
||||
5. **Risk Management**: Always use stop loss and position sizing
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Model Not Loading in MT5
|
||||
|
||||
- Check model file path is correct
|
||||
- Ensure model file is in `MQL5/Files/models/` folder
|
||||
- Verify ONNX model version compatibility (opset 13)
|
||||
|
||||
### No Trades Executed
|
||||
|
||||
- Check confidence threshold (try lowering `InpMinConfidence`)
|
||||
- Verify divergence types are enabled
|
||||
- Check that sufficient historical data is available
|
||||
|
||||
### Poor Backtest Results
|
||||
|
||||
- Collect more training data
|
||||
- Adjust divergence detection parameters
|
||||
- Retrain with different model architecture
|
||||
- Test on different timeframes
|
||||
|
||||
## Notes
|
||||
|
||||
- **Model Compatibility**: ONNX model uses opset 13 for MT5 compatibility
|
||||
- **Feature Normalization**: Features are normalized using MinMaxScaler - ensure same normalization in EA
|
||||
- **Timeframe**: Model trained on H1 timeframe - retrain for other timeframes
|
||||
- **Symbol**: Model trained on BTCUSD - retrain for other symbols
|
||||
|
||||
## License
|
||||
|
||||
This project is provided as-is for educational and research purposes.
|
||||
|
||||
## References
|
||||
|
||||
- [MetaTrader 5 ONNX Documentation](https://www.mql5.com/en/docs/onnx/onnx_prepare)
|
||||
- [RSI Divergence Trading Strategies](https://www.investopedia.com/trading/using-relative-strength-index-rsi/)
|
||||
- [ONNX Runtime](https://onnxruntime.ai/)
|
||||
@@ -0,0 +1,531 @@
|
||||
//+------------------------------------------------------------------+
|
||||
//| RSIDivergence_EA.mq5 |
|
||||
//| RSI Divergence ONNX EA for MT5 |
|
||||
//| |
|
||||
//+------------------------------------------------------------------+
|
||||
#property copyright "RSI Divergence ONNX EA"
|
||||
#property link ""
|
||||
#property version "1.00"
|
||||
#property description "Expert Advisor using ONNX model to identify genuine RSI divergences"
|
||||
#property description "Based on: https://www.mql5.com/en/docs/onnx/onnx_prepare"
|
||||
|
||||
#include <Trade\Trade.mqh>
|
||||
|
||||
//--- Input parameters
|
||||
input group "=== ONNX Model Settings ==="
|
||||
input string InpModelPath = "models\\BTCUSD_H1_rsi_divergence_model.onnx"; // ONNX Model Path
|
||||
input string InpScalerPath = "models\\BTCUSD_H1_rsi_divergence_scaler.pkl"; // Scaler Path (not used in MQL5, for reference)
|
||||
input string InpFeaturesPath = "models\\BTCUSD_H1_rsi_divergence_features.pkl"; // Features Path (not used in MQL5, for reference)
|
||||
input int InpLookback = 60; // Lookback Period (bars)
|
||||
input double InpMinConfidence = 0.7; // Minimum Confidence (0-1)
|
||||
|
||||
input group "=== Trading Settings ==="
|
||||
input double InpLotSize = 0.01; // Lot Size
|
||||
input int InpMagicNumber = 88001; // Magic Number
|
||||
input int InpSlippage = 3; // Slippage (points)
|
||||
input int InpStopLoss = 100; // Stop Loss (pips, 0 = disabled)
|
||||
input int InpTakeProfit = 200; // Take Profit (pips, 0 = disabled)
|
||||
input int InpMaxBarsInTrade = 20; // Max Bars in Trade (0 = disabled)
|
||||
|
||||
input group "=== Divergence Filter ==="
|
||||
input bool InpUseRegularBullish = true; // Trade Regular Bullish Divergence
|
||||
input bool InpUseRegularBearish = true; // Trade Regular Bearish Divergence
|
||||
input bool InpUseHiddenBullish = true; // Trade Hidden Bullish Divergence
|
||||
input bool InpUseHiddenBearish = true; // Trade Hidden Bearish Divergence
|
||||
|
||||
input group "=== Risk Management ==="
|
||||
input bool InpUseTrailingStop = false; // Use Trailing Stop
|
||||
input int InpTrailingStopPips = 50; // Trailing Stop (pips)
|
||||
input int InpTrailingStepPips = 10; // Trailing Step (pips)
|
||||
|
||||
//--- Global variables
|
||||
CTrade trade;
|
||||
long onnx_handle = INVALID_HANDLE;
|
||||
datetime last_bar_time = 0;
|
||||
|
||||
// Divergence type constants (must match Python model)
|
||||
#define DIV_NONE 0
|
||||
#define DIV_REGULAR_BULLISH 1
|
||||
#define DIV_REGULAR_BEARISH 2
|
||||
#define DIV_HIDDEN_BULLISH 3
|
||||
#define DIV_HIDDEN_BEARISH 4
|
||||
|
||||
// Feature calculation buffers
|
||||
double rsi_buffer[];
|
||||
double ema20_buffer[];
|
||||
double ema50_buffer[];
|
||||
double atr_buffer[];
|
||||
double sma20_buffer[];
|
||||
double sma50_buffer[];
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert initialization function |
|
||||
//+------------------------------------------------------------------+
|
||||
int OnInit()
|
||||
{
|
||||
// Set trade parameters
|
||||
trade.SetExpertMagicNumber(InpMagicNumber);
|
||||
trade.SetDeviationInPoints(InpSlippage);
|
||||
trade.SetTypeFilling(ORDER_FILLING_FOK);
|
||||
|
||||
// Load ONNX model
|
||||
string model_path = InpModelPath;
|
||||
|
||||
// Convert relative path to full path
|
||||
if(StringFind(model_path, "\\") == 0 || StringFind(model_path, "/") == 0)
|
||||
{
|
||||
// Already absolute path
|
||||
}
|
||||
else
|
||||
{
|
||||
// Relative path - prepend terminal data folder
|
||||
model_path = TerminalInfoString(TERMINAL_DATA_PATH) + "\\MQL5\\Files\\" + model_path;
|
||||
}
|
||||
|
||||
// Replace forward slashes with backslashes for Windows
|
||||
StringReplace(model_path, "/", "\\");
|
||||
|
||||
Print("Loading ONNX model from: ", model_path);
|
||||
|
||||
onnx_handle = OnnxCreate(model_path, ONNX_DEFAULT);
|
||||
|
||||
if(onnx_handle == INVALID_HANDLE)
|
||||
{
|
||||
Print("ERROR: Failed to load ONNX model. Error: ", GetLastError());
|
||||
Print("Make sure the model file exists at: ", model_path);
|
||||
Print("Model file should be in: ", TerminalInfoString(TERMINAL_DATA_PATH) + "\\MQL5\\Files\\models\\");
|
||||
return(INIT_FAILED);
|
||||
}
|
||||
|
||||
// Get model info
|
||||
long input_count = OnnxGetInputCount(onnx_handle);
|
||||
long output_count = OnnxGetOutputCount(onnx_handle);
|
||||
|
||||
Print("ONNX Model loaded successfully");
|
||||
Print(" Inputs: ", input_count);
|
||||
Print(" Outputs: ", output_count);
|
||||
|
||||
if(input_count > 0)
|
||||
{
|
||||
string input_name = OnnxGetInputName(onnx_handle, 0);
|
||||
Print(" Input name: ", input_name);
|
||||
}
|
||||
|
||||
if(output_count > 0)
|
||||
{
|
||||
string output_name = OnnxGetOutputName(onnx_handle, 0);
|
||||
Print(" Output name: ", output_name);
|
||||
}
|
||||
|
||||
// Initialize indicator buffers
|
||||
ArraySetAsSeries(rsi_buffer, true);
|
||||
ArraySetAsSeries(ema20_buffer, true);
|
||||
ArraySetAsSeries(ema50_buffer, true);
|
||||
ArraySetAsSeries(atr_buffer, true);
|
||||
ArraySetAsSeries(sma20_buffer, true);
|
||||
ArraySetAsSeries(sma50_buffer, true);
|
||||
|
||||
Print("RSI Divergence EA initialized successfully");
|
||||
Print(" Symbol: ", _Symbol);
|
||||
Print(" Timeframe: ", EnumToString(PERIOD_CURRENT));
|
||||
Print(" Lookback: ", InpLookback);
|
||||
Print(" Min Confidence: ", InpMinConfidence);
|
||||
|
||||
return(INIT_SUCCEEDED);
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert deinitialization function |
|
||||
//+------------------------------------------------------------------+
|
||||
void OnDeinit(const int reason)
|
||||
{
|
||||
// Release ONNX model
|
||||
if(onnx_handle != INVALID_HANDLE)
|
||||
{
|
||||
OnnxRelease(onnx_handle);
|
||||
Print("ONNX model released");
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert tick function |
|
||||
//+------------------------------------------------------------------+
|
||||
void OnTick()
|
||||
{
|
||||
// Check if new bar
|
||||
datetime current_bar_time = iTime(_Symbol, PERIOD_CURRENT, 0);
|
||||
if(current_bar_time == last_bar_time)
|
||||
{
|
||||
// Check trailing stop on current bar
|
||||
if(InpUseTrailingStop)
|
||||
{
|
||||
ApplyTrailingStop();
|
||||
}
|
||||
return; // Still the same bar
|
||||
}
|
||||
last_bar_time = current_bar_time;
|
||||
|
||||
// Close positions that have been open too long
|
||||
if(InpMaxBarsInTrade > 0)
|
||||
{
|
||||
CloseOldPositions();
|
||||
}
|
||||
|
||||
// Prepare input data
|
||||
float input_data[];
|
||||
if(!PrepareInputData(input_data))
|
||||
{
|
||||
Print("ERROR: Failed to prepare input data");
|
||||
return;
|
||||
}
|
||||
|
||||
// Run ONNX model
|
||||
float output_data[];
|
||||
if(!RunONNXModel(input_data, output_data))
|
||||
{
|
||||
Print("ERROR: Failed to run ONNX model");
|
||||
return;
|
||||
}
|
||||
|
||||
// Get prediction
|
||||
if(ArraySize(output_data) < 5)
|
||||
{
|
||||
Print("ERROR: Invalid output from ONNX model");
|
||||
return;
|
||||
}
|
||||
|
||||
// Get predicted class and confidence
|
||||
int predicted_class = 0;
|
||||
double max_prob = 0.0;
|
||||
|
||||
for(int i = 0; i < 5; i++)
|
||||
{
|
||||
if(output_data[i] > max_prob)
|
||||
{
|
||||
max_prob = output_data[i];
|
||||
predicted_class = i;
|
||||
}
|
||||
}
|
||||
|
||||
double confidence = max_prob;
|
||||
|
||||
// Check if confidence meets threshold
|
||||
if(confidence < InpMinConfidence)
|
||||
{
|
||||
return; // Not confident enough
|
||||
}
|
||||
|
||||
// Check if we should trade this divergence type
|
||||
bool should_trade = false;
|
||||
int signal_type = 0; // 1 = BUY, -1 = SELL
|
||||
|
||||
if(predicted_class == DIV_REGULAR_BULLISH && InpUseRegularBullish)
|
||||
{
|
||||
should_trade = true;
|
||||
signal_type = 1; // BUY
|
||||
}
|
||||
else if(predicted_class == DIV_REGULAR_BEARISH && InpUseRegularBearish)
|
||||
{
|
||||
should_trade = true;
|
||||
signal_type = -1; // SELL
|
||||
}
|
||||
else if(predicted_class == DIV_HIDDEN_BULLISH && InpUseHiddenBullish)
|
||||
{
|
||||
should_trade = true;
|
||||
signal_type = 1; // BUY
|
||||
}
|
||||
else if(predicted_class == DIV_HIDDEN_BEARISH && InpUseHiddenBearish)
|
||||
{
|
||||
should_trade = true;
|
||||
signal_type = -1; // SELL
|
||||
}
|
||||
|
||||
if(!should_trade)
|
||||
{
|
||||
return; // Divergence type not enabled
|
||||
}
|
||||
|
||||
// Check if we already have a position
|
||||
if(PositionSelect(_Symbol))
|
||||
{
|
||||
return; // Already in a position
|
||||
}
|
||||
|
||||
// Execute trade
|
||||
double price = (signal_type == 1) ? SymbolInfoDouble(_Symbol, SYMBOL_ASK) : SymbolInfoDouble(_Symbol, SYMBOL_BID);
|
||||
double sl = 0, tp = 0;
|
||||
|
||||
// Calculate stop loss and take profit
|
||||
if(InpStopLoss > 0)
|
||||
{
|
||||
sl = (signal_type == 1) ? price - InpStopLoss * _Point * 10 : price + InpStopLoss * _Point * 10;
|
||||
}
|
||||
|
||||
if(InpTakeProfit > 0)
|
||||
{
|
||||
tp = (signal_type == 1) ? price + InpTakeProfit * _Point * 10 : price - InpTakeProfit * _Point * 10;
|
||||
}
|
||||
|
||||
// Open position
|
||||
string divergence_name = "";
|
||||
if(predicted_class == DIV_REGULAR_BULLISH) divergence_name = "Regular Bullish";
|
||||
else if(predicted_class == DIV_REGULAR_BEARISH) divergence_name = "Regular Bearish";
|
||||
else if(predicted_class == DIV_HIDDEN_BULLISH) divergence_name = "Hidden Bullish";
|
||||
else if(predicted_class == DIV_HIDDEN_BEARISH) divergence_name = "Hidden Bearish";
|
||||
|
||||
if(signal_type == 1)
|
||||
{
|
||||
if(trade.Buy(InpLotSize, _Symbol, price, sl, tp, divergence_name + " Divergence (Conf: " + DoubleToString(confidence, 2) + ")"))
|
||||
{
|
||||
Print("BUY order opened: ", divergence_name, " Divergence, Confidence: ", confidence);
|
||||
}
|
||||
else
|
||||
{
|
||||
Print("ERROR: Failed to open BUY order: ", trade.ResultRetcodeDescription());
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
if(trade.Sell(InpLotSize, _Symbol, price, sl, tp, divergence_name + " Divergence (Conf: " + DoubleToString(confidence, 2) + ")"))
|
||||
{
|
||||
Print("SELL order opened: ", divergence_name, " Divergence, Confidence: ", confidence);
|
||||
}
|
||||
else
|
||||
{
|
||||
Print("ERROR: Failed to open SELL order: ", trade.ResultRetcodeDescription());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Prepare input data for ONNX model |
|
||||
//+------------------------------------------------------------------+
|
||||
bool PrepareInputData(float &input_data[])
|
||||
{
|
||||
// We need to prepare features in the same order as training
|
||||
// This should match the feature_cols from the Python training script
|
||||
|
||||
int lookback = InpLookback;
|
||||
int num_features = 20; // Adjust based on your actual feature count
|
||||
|
||||
// Resize input array: (1, lookback, num_features)
|
||||
ArrayResize(input_data, lookback * num_features);
|
||||
ArrayInitialize(input_data, 0.0);
|
||||
|
||||
// Get price data
|
||||
double close[], open[], high[], low[], volume[];
|
||||
ArraySetAsSeries(close, true);
|
||||
ArraySetAsSeries(open, true);
|
||||
ArraySetAsSeries(high, true);
|
||||
ArraySetAsSeries(low, true);
|
||||
ArraySetAsSeries(volume, true);
|
||||
|
||||
CopyClose(_Symbol, PERIOD_CURRENT, 0, lookback, close);
|
||||
CopyOpen(_Symbol, PERIOD_CURRENT, 0, lookback, open);
|
||||
CopyHigh(_Symbol, PERIOD_CURRENT, 0, lookback, high);
|
||||
CopyLow(_Symbol, PERIOD_CURRENT, 0, lookback, low);
|
||||
CopyTickVolume(_Symbol, PERIOD_CURRENT, 0, lookback, volume);
|
||||
|
||||
// Calculate technical indicators
|
||||
int rsi_handle = iRSI(_Symbol, PERIOD_CURRENT, 14, PRICE_CLOSE);
|
||||
int ema20_handle = iMA(_Symbol, PERIOD_CURRENT, 20, 0, MODE_EMA, PRICE_CLOSE);
|
||||
int ema50_handle = iMA(_Symbol, PERIOD_CURRENT, 50, 0, MODE_EMA, PRICE_CLOSE);
|
||||
int sma20_handle = iMA(_Symbol, PERIOD_CURRENT, 20, 0, MODE_SMA, PRICE_CLOSE);
|
||||
int sma50_handle = iMA(_Symbol, PERIOD_CURRENT, 50, 0, MODE_SMA, PRICE_CLOSE);
|
||||
int atr_handle = iATR(_Symbol, PERIOD_CURRENT, 14);
|
||||
|
||||
ArraySetAsSeries(rsi_buffer, true);
|
||||
ArraySetAsSeries(ema20_buffer, true);
|
||||
ArraySetAsSeries(ema50_buffer, true);
|
||||
ArraySetAsSeries(sma20_buffer, true);
|
||||
ArraySetAsSeries(sma50_buffer, true);
|
||||
ArraySetAsSeries(atr_buffer, true);
|
||||
|
||||
if(CopyBuffer(rsi_handle, 0, 0, lookback, rsi_buffer) <= 0) return false;
|
||||
if(CopyBuffer(ema20_handle, 0, 0, lookback, ema20_buffer) <= 0) return false;
|
||||
if(CopyBuffer(ema50_handle, 0, 0, lookback, ema50_buffer) <= 0) return false;
|
||||
if(CopyBuffer(sma20_handle, 0, 0, lookback, sma20_buffer) <= 0) return false;
|
||||
if(CopyBuffer(sma50_handle, 0, 0, lookback, sma50_buffer) <= 0) return false;
|
||||
if(CopyBuffer(atr_handle, 0, 0, lookback, atr_buffer) <= 0) return false;
|
||||
|
||||
// Release indicator handles
|
||||
IndicatorRelease(rsi_handle);
|
||||
IndicatorRelease(ema20_handle);
|
||||
IndicatorRelease(ema50_handle);
|
||||
IndicatorRelease(sma20_handle);
|
||||
IndicatorRelease(sma50_handle);
|
||||
IndicatorRelease(atr_handle);
|
||||
|
||||
// Prepare features (must match Python feature order)
|
||||
// Note: Features need to be normalized - this is a simplified version
|
||||
// In production, you should use the same scaler from Python
|
||||
|
||||
for(int i = 0; i < lookback; i++)
|
||||
{
|
||||
int idx = i * num_features;
|
||||
int bar_idx = lookback - 1 - i; // Reverse for time series
|
||||
|
||||
// Basic OHLCV (normalized)
|
||||
input_data[idx + 0] = (float)(close[bar_idx] / close[0] - 1.0); // Normalized close
|
||||
input_data[idx + 1] = (float)(open[bar_idx] / close[0] - 1.0); // Normalized open
|
||||
input_data[idx + 2] = (float)(high[bar_idx] / close[0] - 1.0); // Normalized high
|
||||
input_data[idx + 3] = (float)(low[bar_idx] / close[0] - 1.0); // Normalized low
|
||||
input_data[idx + 4] = (float)(volume[bar_idx] / 1000000.0); // Normalized volume
|
||||
|
||||
// Returns
|
||||
if(bar_idx < lookback - 1)
|
||||
{
|
||||
input_data[idx + 5] = (float)((close[bar_idx] - close[bar_idx + 1]) / close[bar_idx + 1]);
|
||||
}
|
||||
|
||||
// Ratios
|
||||
input_data[idx + 6] = (float)(high[bar_idx] / (low[bar_idx] + 1e-10));
|
||||
input_data[idx + 7] = (float)(close[bar_idx] / (open[bar_idx] + 1e-10));
|
||||
|
||||
// Moving averages (normalized)
|
||||
input_data[idx + 8] = (float)(sma20_buffer[bar_idx] / close[0] - 1.0);
|
||||
input_data[idx + 9] = (float)(sma50_buffer[bar_idx] / close[0] - 1.0);
|
||||
input_data[idx + 10] = (float)(ema20_buffer[bar_idx] / close[0] - 1.0);
|
||||
input_data[idx + 11] = (float)(ema50_buffer[bar_idx] / close[0] - 1.0);
|
||||
|
||||
// ATR
|
||||
input_data[idx + 12] = (float)(atr_buffer[bar_idx] / close[0]);
|
||||
|
||||
// RSI (normalized to 0-1)
|
||||
input_data[idx + 13] = (float)(rsi_buffer[bar_idx] / 100.0);
|
||||
|
||||
// Volume features
|
||||
double volume_ma = 0;
|
||||
for(int j = 0; j < 20 && (bar_idx + j) < lookback; j++)
|
||||
{
|
||||
volume_ma += volume[bar_idx + j];
|
||||
}
|
||||
volume_ma /= 20.0;
|
||||
input_data[idx + 14] = (float)(volume[bar_idx] / (volume_ma + 1e-10));
|
||||
|
||||
// Price position (simplified)
|
||||
double min_low = low[bar_idx];
|
||||
double max_high = high[bar_idx];
|
||||
for(int j = 0; j < 20 && (bar_idx + j) < lookback; j++)
|
||||
{
|
||||
if(low[bar_idx + j] < min_low) min_low = low[bar_idx + j];
|
||||
if(high[bar_idx + j] > max_high) max_high = high[bar_idx + j];
|
||||
}
|
||||
input_data[idx + 15] = (float)((close[bar_idx] - min_low) / (max_high - min_low + 1e-10));
|
||||
|
||||
// Additional features (pad with zeros if needed)
|
||||
for(int j = 16; j < num_features; j++)
|
||||
{
|
||||
input_data[idx + j] = 0.0;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Run ONNX model |
|
||||
//+------------------------------------------------------------------+
|
||||
bool RunONNXModel(float &input_data[], float &output_data[])
|
||||
{
|
||||
if(onnx_handle == INVALID_HANDLE)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// Get input/output names
|
||||
string input_name = OnnxGetInputName(onnx_handle, 0);
|
||||
string output_name = OnnxGetOutputName(onnx_handle, 0);
|
||||
|
||||
// Prepare input shape: (1, lookback, num_features)
|
||||
long input_shape[] = {1, InpLookback, 20}; // Adjust num_features as needed
|
||||
long output_shape[] = {1, 5}; // 5 classes
|
||||
|
||||
// Run model
|
||||
if(!OnnxRun(onnx_handle, ONNX_NO_CONVERSION, input_data, input_shape, 3,
|
||||
output_data, output_shape))
|
||||
{
|
||||
Print("ERROR: OnnxRun failed. Error: ", GetLastError());
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Apply trailing stop |
|
||||
//+------------------------------------------------------------------+
|
||||
void ApplyTrailingStop()
|
||||
{
|
||||
if(!PositionSelect(_Symbol))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if(PositionGetInteger(POSITION_MAGIC) != InpMagicNumber)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
double position_open_price = PositionGetDouble(POSITION_PRICE_OPEN);
|
||||
double position_sl = PositionGetDouble(POSITION_SL);
|
||||
double position_tp = PositionGetDouble(POSITION_TP);
|
||||
long position_type = PositionGetInteger(POSITION_TYPE);
|
||||
|
||||
double current_price = (position_type == POSITION_TYPE_BUY) ?
|
||||
SymbolInfoDouble(_Symbol, SYMBOL_BID) :
|
||||
SymbolInfoDouble(_Symbol, SYMBOL_ASK);
|
||||
|
||||
double trailing_distance = InpTrailingStopPips * _Point * 10;
|
||||
double trailing_step = InpTrailingStepPips * _Point * 10;
|
||||
|
||||
if(position_type == POSITION_TYPE_BUY)
|
||||
{
|
||||
double new_sl = current_price - trailing_distance;
|
||||
|
||||
if(new_sl > position_open_price &&
|
||||
(position_sl == 0 || new_sl > position_sl + trailing_step))
|
||||
{
|
||||
trade.PositionModify(_Symbol, new_sl, position_tp);
|
||||
}
|
||||
}
|
||||
else if(position_type == POSITION_TYPE_SELL)
|
||||
{
|
||||
double new_sl = current_price + trailing_distance;
|
||||
|
||||
if(new_sl < position_open_price &&
|
||||
(position_sl == 0 || new_sl < position_sl - trailing_step))
|
||||
{
|
||||
trade.PositionModify(_Symbol, new_sl, position_tp);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Close positions that have been open too long |
|
||||
//+------------------------------------------------------------------+
|
||||
void CloseOldPositions()
|
||||
{
|
||||
if(!PositionSelect(_Symbol))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if(PositionGetInteger(POSITION_MAGIC) != InpMagicNumber)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
datetime position_open_time = (datetime)PositionGetInteger(POSITION_TIME);
|
||||
datetime current_time = TimeCurrent();
|
||||
|
||||
int bars_open = Bars(_Symbol, PERIOD_CURRENT, position_open_time, current_time);
|
||||
|
||||
if(bars_open >= InpMaxBarsInTrade)
|
||||
{
|
||||
trade.PositionClose(_Symbol);
|
||||
Print("Position closed: Max bars in trade reached (", bars_open, " bars)");
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
Binary file not shown.
@@ -0,0 +1,401 @@
|
||||
"""
|
||||
Backtesting Script for RSI Divergence ONNX Model
|
||||
Tests the trained model on historical data and evaluates trading performance.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import MetaTrader5 as mt5
|
||||
from datetime import datetime, timedelta
|
||||
import onnxruntime as ort
|
||||
import pickle
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
class RSIDivergenceBacktester:
|
||||
"""
|
||||
Backtests the RSI divergence ONNX model.
|
||||
"""
|
||||
|
||||
def __init__(self, model_path: str, scaler_path: str, features_path: str, lookback: int = 60):
|
||||
"""
|
||||
Initialize the backtester.
|
||||
|
||||
Args:
|
||||
model_path: Path to ONNX model file
|
||||
scaler_path: Path to scaler pickle file
|
||||
features_path: Path to features list pickle file
|
||||
lookback: Number of bars to look back
|
||||
"""
|
||||
self.lookback = lookback
|
||||
|
||||
# Load ONNX model
|
||||
print(f"Loading ONNX model from {model_path}...")
|
||||
self.session = ort.InferenceSession(model_path)
|
||||
print("ONNX model loaded successfully")
|
||||
|
||||
# Load scaler
|
||||
print(f"Loading scaler from {scaler_path}...")
|
||||
with open(scaler_path, 'rb') as f:
|
||||
self.scaler = pickle.load(f)
|
||||
print("Scaler loaded successfully")
|
||||
|
||||
# Load feature list
|
||||
print(f"Loading features from {features_path}...")
|
||||
with open(features_path, 'rb') as f:
|
||||
self.feature_cols = pickle.load(f)
|
||||
print(f"Using {len(self.feature_cols)} features")
|
||||
|
||||
# Divergence type mapping
|
||||
self.divergence_types = {
|
||||
0: 'NONE',
|
||||
1: 'REGULAR_BULLISH',
|
||||
2: 'REGULAR_BEARISH',
|
||||
3: 'HIDDEN_BULLISH',
|
||||
4: 'HIDDEN_BEARISH'
|
||||
}
|
||||
|
||||
def prepare_features(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Prepare features from raw data (same as in collect_btcusd_data.py)."""
|
||||
feature_df = df.copy()
|
||||
|
||||
# Price-based features
|
||||
feature_df['returns'] = feature_df['close'].pct_change()
|
||||
feature_df['high_low_ratio'] = feature_df['high'] / (feature_df['low'] + 1e-10)
|
||||
feature_df['close_open_ratio'] = feature_df['close'] / (feature_df['open'] + 1e-10)
|
||||
|
||||
# Moving averages
|
||||
feature_df['sma_20'] = feature_df['close'].rolling(window=20).mean()
|
||||
feature_df['sma_50'] = feature_df['close'].rolling(window=50).mean()
|
||||
feature_df['ema_20'] = feature_df['close'].ewm(span=20).mean()
|
||||
feature_df['ema_50'] = feature_df['close'].ewm(span=50).mean()
|
||||
|
||||
# ATR
|
||||
high_low = feature_df['high'] - feature_df['low']
|
||||
high_close = np.abs(feature_df['high'] - feature_df['close'].shift())
|
||||
low_close = np.abs(feature_df['low'] - feature_df['close'].shift())
|
||||
tr = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1)
|
||||
feature_df['atr'] = tr.rolling(window=14).mean()
|
||||
feature_df['atr_pct'] = feature_df['atr'] / (feature_df['close'] + 1e-10)
|
||||
|
||||
# Volume features
|
||||
if 'tick_volume' in feature_df.columns:
|
||||
feature_df['volume_ma'] = feature_df['tick_volume'].rolling(window=20).mean()
|
||||
feature_df['volume_ratio'] = feature_df['tick_volume'] / (feature_df['volume_ma'] + 1e-10)
|
||||
|
||||
# Price position relative to range
|
||||
feature_df['price_position'] = (feature_df['close'] - feature_df['low'].rolling(20).min()) / (
|
||||
feature_df['high'].rolling(20).max() - feature_df['low'].rolling(20).min() + 1e-10
|
||||
)
|
||||
|
||||
# Calculate RSI
|
||||
from rsi_divergence_detector import RSIDivergenceDetector
|
||||
detector = RSIDivergenceDetector()
|
||||
feature_df['rsi'] = detector.calculate_rsi(feature_df['close'])
|
||||
|
||||
return feature_df
|
||||
|
||||
def predict(self, df: pd.DataFrame, index: int) -> tuple:
|
||||
"""
|
||||
Make prediction at given index.
|
||||
|
||||
Args:
|
||||
df: DataFrame with features
|
||||
index: Current bar index
|
||||
|
||||
Returns:
|
||||
Tuple of (predicted_class, confidence)
|
||||
"""
|
||||
if index < self.lookback:
|
||||
return 0, 0.0
|
||||
|
||||
# Get feature sequence
|
||||
feature_data = df[self.feature_cols].iloc[index - self.lookback:index].values
|
||||
|
||||
# Scale features
|
||||
feature_data_scaled = self.scaler.transform(feature_data)
|
||||
|
||||
# Reshape for model input (1, lookback, features)
|
||||
feature_data_scaled = feature_data_scaled.reshape(1, self.lookback, -1)
|
||||
|
||||
# Run ONNX model
|
||||
input_name = self.session.get_inputs()[0].name
|
||||
output_name = self.session.get_outputs()[0].name
|
||||
|
||||
result = self.session.run([output_name], {input_name: feature_data_scaled.astype(np.float32)})
|
||||
|
||||
# Get prediction
|
||||
probabilities = result[0][0]
|
||||
predicted_class = int(np.argmax(probabilities))
|
||||
confidence = float(np.max(probabilities))
|
||||
|
||||
return predicted_class, confidence
|
||||
|
||||
def backtest(self, symbol: str, timeframe: int, start_date: datetime,
|
||||
end_date: datetime, initial_balance: float = 10000.0,
|
||||
lot_size: float = 0.01, min_confidence: float = 0.7) -> dict:
|
||||
"""
|
||||
Run backtest on historical data.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol
|
||||
timeframe: MT5 timeframe constant
|
||||
start_date: Start date
|
||||
end_date: End date
|
||||
initial_balance: Starting balance
|
||||
lot_size: Lot size per trade
|
||||
min_confidence: Minimum confidence to take a trade
|
||||
|
||||
Returns:
|
||||
Dictionary with backtest results
|
||||
"""
|
||||
print(f"\n{'='*60}")
|
||||
print("RSI Divergence Model Backtest")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
# Fetch data
|
||||
if not mt5.initialize():
|
||||
raise RuntimeError(f"MT5 initialization failed: {mt5.last_error()}")
|
||||
|
||||
try:
|
||||
print(f"Fetching {symbol} data from {start_date} to {end_date}...")
|
||||
rates = mt5.copy_rates_range(symbol, timeframe, start_date, end_date)
|
||||
|
||||
if rates is None or len(rates) == 0:
|
||||
raise ValueError(f"No data available for {symbol}")
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
df.set_index('time', inplace=True)
|
||||
df.columns = [col.lower() for col in df.columns]
|
||||
|
||||
print(f"Fetched {len(df)} bars")
|
||||
|
||||
# Prepare features
|
||||
print("Preparing features...")
|
||||
df = self.prepare_features(df)
|
||||
df = df.dropna()
|
||||
|
||||
print(f"Data ready: {len(df)} bars after feature preparation")
|
||||
|
||||
# Backtest simulation
|
||||
balance = initial_balance
|
||||
equity = initial_balance
|
||||
position = None # (type: 'BUY' or 'SELL', entry_price, entry_index, size)
|
||||
trades = []
|
||||
equity_curve = [initial_balance]
|
||||
|
||||
print("\nRunning backtest...")
|
||||
for i in tqdm(range(self.lookback, len(df))):
|
||||
current_price = df['close'].iloc[i]
|
||||
current_time = df.index[i]
|
||||
|
||||
# Make prediction
|
||||
predicted_class, confidence = self.predict(df, i)
|
||||
divergence_type = self.divergence_types[predicted_class]
|
||||
|
||||
# Close position if needed
|
||||
if position is not None:
|
||||
# Simple exit: close after 10 bars or on opposite signal
|
||||
bars_in_trade = i - position[2]
|
||||
|
||||
if bars_in_trade >= 10:
|
||||
# Close position
|
||||
if position[0] == 'BUY':
|
||||
pnl = (current_price - position[1]) * position[3]
|
||||
else:
|
||||
pnl = (position[1] - current_price) * position[3]
|
||||
|
||||
balance += pnl
|
||||
equity = balance
|
||||
|
||||
trades.append({
|
||||
'entry_time': df.index[position[2]],
|
||||
'exit_time': current_time,
|
||||
'type': position[0],
|
||||
'entry_price': position[1],
|
||||
'exit_price': current_price,
|
||||
'size': position[3],
|
||||
'pnl': pnl,
|
||||
'bars_held': bars_in_trade
|
||||
})
|
||||
|
||||
position = None
|
||||
|
||||
# Open new position based on prediction
|
||||
if position is None and confidence >= min_confidence:
|
||||
if divergence_type == 'REGULAR_BULLISH' or divergence_type == 'HIDDEN_BULLISH':
|
||||
# Buy signal
|
||||
position = ('BUY', current_price, i, lot_size)
|
||||
elif divergence_type == 'REGULAR_BEARISH' or divergence_type == 'HIDDEN_BEARISH':
|
||||
# Sell signal
|
||||
position = ('SELL', current_price, i, lot_size)
|
||||
|
||||
# Update equity (with unrealized PnL)
|
||||
if position is not None:
|
||||
if position[0] == 'BUY':
|
||||
unrealized_pnl = (current_price - position[1]) * position[3]
|
||||
else:
|
||||
unrealized_pnl = (position[1] - current_price) * position[3]
|
||||
equity = balance + unrealized_pnl
|
||||
else:
|
||||
equity = balance
|
||||
|
||||
equity_curve.append(equity)
|
||||
|
||||
# Close any remaining position
|
||||
if position is not None:
|
||||
final_price = df['close'].iloc[-1]
|
||||
if position[0] == 'BUY':
|
||||
pnl = (final_price - position[1]) * position[3]
|
||||
else:
|
||||
pnl = (position[1] - final_price) * position[3]
|
||||
|
||||
balance += pnl
|
||||
trades.append({
|
||||
'entry_time': df.index[position[2]],
|
||||
'exit_time': df.index[-1],
|
||||
'type': position[0],
|
||||
'entry_price': position[1],
|
||||
'exit_price': final_price,
|
||||
'size': position[3],
|
||||
'pnl': pnl,
|
||||
'bars_held': len(df) - position[2]
|
||||
})
|
||||
|
||||
# Calculate metrics
|
||||
trades_df = pd.DataFrame(trades)
|
||||
|
||||
if len(trades) > 0:
|
||||
total_trades = len(trades)
|
||||
winning_trades = len(trades_df[trades_df['pnl'] > 0])
|
||||
losing_trades = len(trades_df[trades_df['pnl'] <= 0])
|
||||
win_rate = winning_trades / total_trades * 100
|
||||
|
||||
total_pnl = trades_df['pnl'].sum()
|
||||
avg_win = trades_df[trades_df['pnl'] > 0]['pnl'].mean() if winning_trades > 0 else 0
|
||||
avg_loss = trades_df[trades_df['pnl'] <= 0]['pnl'].mean() if losing_trades > 0 else 0
|
||||
|
||||
profit_factor = abs(avg_win * winning_trades / (avg_loss * losing_trades)) if losing_trades > 0 and avg_loss != 0 else float('inf')
|
||||
|
||||
final_balance = balance
|
||||
total_return = (final_balance - initial_balance) / initial_balance * 100
|
||||
|
||||
# Drawdown
|
||||
equity_series = pd.Series(equity_curve)
|
||||
running_max = equity_series.expanding().max()
|
||||
drawdown = (equity_series - running_max) / running_max * 100
|
||||
max_drawdown = drawdown.min()
|
||||
else:
|
||||
total_trades = 0
|
||||
winning_trades = 0
|
||||
losing_trades = 0
|
||||
win_rate = 0
|
||||
total_pnl = 0
|
||||
avg_win = 0
|
||||
avg_loss = 0
|
||||
profit_factor = 0
|
||||
final_balance = initial_balance
|
||||
total_return = 0
|
||||
max_drawdown = 0
|
||||
|
||||
results = {
|
||||
'initial_balance': initial_balance,
|
||||
'final_balance': final_balance,
|
||||
'total_return_pct': total_return,
|
||||
'total_trades': total_trades,
|
||||
'winning_trades': winning_trades,
|
||||
'losing_trades': losing_trades,
|
||||
'win_rate': win_rate,
|
||||
'total_pnl': total_pnl,
|
||||
'avg_win': avg_win,
|
||||
'avg_loss': avg_loss,
|
||||
'profit_factor': profit_factor,
|
||||
'max_drawdown_pct': max_drawdown,
|
||||
'trades': trades_df
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(description='Backtest RSI divergence ONNX model')
|
||||
parser.add_argument('--model', type=str, required=True, help='Path to ONNX model file')
|
||||
parser.add_argument('--scaler', type=str, required=True, help='Path to scaler pickle file')
|
||||
parser.add_argument('--features', type=str, required=True, help='Path to features list pickle file')
|
||||
parser.add_argument('--symbol', type=str, default='BTCUSD', help='Trading symbol')
|
||||
parser.add_argument('--timeframe', type=str, default='H1',
|
||||
choices=['M1', 'M5', 'M15', 'M30', 'H1', 'H4', 'D1'],
|
||||
help='Timeframe')
|
||||
parser.add_argument('--days', type=int, default=90, help='Number of days to backtest')
|
||||
parser.add_argument('--balance', type=float, default=10000.0, help='Initial balance')
|
||||
parser.add_argument('--lot-size', type=float, default=0.01, help='Lot size per trade')
|
||||
parser.add_argument('--min-confidence', type=float, default=0.7,
|
||||
help='Minimum confidence to take a trade')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Convert timeframe
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[args.timeframe]
|
||||
|
||||
# Create backtester
|
||||
backtester = RSIDivergenceBacktester(
|
||||
args.model, args.scaler, args.features, lookback=60
|
||||
)
|
||||
|
||||
# Run backtest
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=args.days)
|
||||
|
||||
results = backtester.backtest(
|
||||
args.symbol, timeframe, start_date, end_date,
|
||||
initial_balance=args.balance,
|
||||
lot_size=args.lot_size,
|
||||
min_confidence=args.min_confidence
|
||||
)
|
||||
|
||||
# Print results
|
||||
print(f"\n{'='*60}")
|
||||
print("Backtest Results")
|
||||
print(f"{'='*60}")
|
||||
print(f"Initial Balance: ${results['initial_balance']:,.2f}")
|
||||
print(f"Final Balance: ${results['final_balance']:,.2f}")
|
||||
print(f"Total Return: {results['total_return_pct']:.2f}%")
|
||||
print(f"Max Drawdown: {results['max_drawdown_pct']:.2f}%")
|
||||
print(f"\nTrades:")
|
||||
print(f" Total: {results['total_trades']}")
|
||||
print(f" Winning: {results['winning_trades']}")
|
||||
print(f" Losing: {results['losing_trades']}")
|
||||
print(f" Win Rate: {results['win_rate']:.2f}%")
|
||||
print(f"\nPerformance:")
|
||||
print(f" Total P&L: ${results['total_pnl']:,.2f}")
|
||||
print(f" Avg Win: ${results['avg_win']:,.2f}")
|
||||
print(f" Avg Loss: ${results['avg_loss']:,.2f}")
|
||||
print(f" Profit Factor: {results['profit_factor']:.2f}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
# Save trades to CSV
|
||||
if len(results['trades']) > 0:
|
||||
output_file = f"backtest_trades_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv"
|
||||
results['trades'].to_csv(output_file, index=False)
|
||||
print(f"Trades saved to: {output_file}")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,221 @@
|
||||
"""
|
||||
Data Collection Script for BTCUSD RSI Divergence Training
|
||||
Fetches BTCUSD data from MetaTrader 5 and labels it with RSI divergence signals.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import MetaTrader5 as mt5
|
||||
from rsi_divergence_detector import RSIDivergenceDetector, DivergenceType
|
||||
import pickle
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
def fetch_mt5_data(symbol: str, timeframe: int, start_date: datetime, end_date: datetime) -> pd.DataFrame:
|
||||
"""
|
||||
Fetch historical data from MetaTrader 5.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol (e.g., 'BTCUSD')
|
||||
timeframe: MT5 timeframe constant
|
||||
start_date: Start date for data
|
||||
end_date: End date for data
|
||||
|
||||
Returns:
|
||||
DataFrame with OHLCV data
|
||||
"""
|
||||
print(f"Fetching {symbol} data from {start_date} to {end_date}...")
|
||||
|
||||
if not mt5.initialize():
|
||||
raise RuntimeError(f"MT5 initialization failed: {mt5.last_error()}")
|
||||
|
||||
try:
|
||||
rates = mt5.copy_rates_range(symbol, timeframe, start_date, end_date)
|
||||
|
||||
if rates is None or len(rates) == 0:
|
||||
raise ValueError(f"No data available for {symbol} in the specified date range")
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
df.set_index('time', inplace=True)
|
||||
|
||||
# Rename columns to lowercase
|
||||
df.columns = [col.lower() for col in df.columns]
|
||||
|
||||
print(f"Fetched {len(df)} bars")
|
||||
return df
|
||||
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
def prepare_features(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Prepare additional features for training.
|
||||
|
||||
Args:
|
||||
df: DataFrame with OHLCV data
|
||||
|
||||
Returns:
|
||||
DataFrame with additional features
|
||||
"""
|
||||
feature_df = df.copy()
|
||||
|
||||
# Price-based features
|
||||
feature_df['returns'] = feature_df['close'].pct_change()
|
||||
feature_df['high_low_ratio'] = feature_df['high'] / (feature_df['low'] + 1e-10)
|
||||
feature_df['close_open_ratio'] = feature_df['close'] / (feature_df['open'] + 1e-10)
|
||||
|
||||
# Moving averages
|
||||
feature_df['sma_20'] = feature_df['close'].rolling(window=20).mean()
|
||||
feature_df['sma_50'] = feature_df['close'].rolling(window=50).mean()
|
||||
feature_df['ema_20'] = feature_df['close'].ewm(span=20).mean()
|
||||
feature_df['ema_50'] = feature_df['close'].ewm(span=50).mean()
|
||||
|
||||
# ATR
|
||||
high_low = feature_df['high'] - feature_df['low']
|
||||
high_close = np.abs(feature_df['high'] - feature_df['close'].shift())
|
||||
low_close = np.abs(feature_df['low'] - feature_df['close'].shift())
|
||||
tr = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1)
|
||||
feature_df['atr'] = tr.rolling(window=14).mean()
|
||||
feature_df['atr_pct'] = feature_df['atr'] / (feature_df['close'] + 1e-10)
|
||||
|
||||
# Volume features
|
||||
if 'tick_volume' in feature_df.columns:
|
||||
feature_df['volume_ma'] = feature_df['tick_volume'].rolling(window=20).mean()
|
||||
feature_df['volume_ratio'] = feature_df['tick_volume'] / (feature_df['volume_ma'] + 1e-10)
|
||||
|
||||
# Price position relative to range
|
||||
feature_df['price_position'] = (feature_df['close'] - feature_df['low'].rolling(20).min()) / (
|
||||
feature_df['high'].rolling(20).max() - feature_df['low'].rolling(20).min() + 1e-10
|
||||
)
|
||||
|
||||
return feature_df
|
||||
|
||||
|
||||
def create_sequences(df: pd.DataFrame, lookback: int = 60, prediction_horizon: int = 5) -> tuple:
|
||||
"""
|
||||
Create sequences for training.
|
||||
|
||||
Args:
|
||||
df: Labeled DataFrame
|
||||
lookback: Number of bars to look back
|
||||
prediction_horizon: Number of bars ahead to predict
|
||||
|
||||
Returns:
|
||||
Tuple of (X, y) where X is features and y is labels
|
||||
"""
|
||||
# Feature columns (exclude labels and time-based columns)
|
||||
exclude_cols = ['divergence_type', 'divergence_confidence', 'divergence_strength', 'time']
|
||||
feature_cols = [col for col in df.columns if col not in exclude_cols]
|
||||
|
||||
X, y = [], []
|
||||
|
||||
for i in range(lookback, len(df) - prediction_horizon):
|
||||
# Get feature sequence
|
||||
X.append(df[feature_cols].iloc[i - lookback:i].values)
|
||||
|
||||
# Get label (divergence type at current bar)
|
||||
y.append(df['divergence_type'].iloc[i])
|
||||
|
||||
return np.array(X), np.array(y)
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(description='Collect and label BTCUSD data for RSI divergence training')
|
||||
parser.add_argument('--symbol', type=str, default='BTCUSD', help='Trading symbol')
|
||||
parser.add_argument('--timeframe', type=str, default='H1',
|
||||
choices=['M1', 'M5', 'M15', 'M30', 'H1', 'H4', 'D1'],
|
||||
help='Timeframe')
|
||||
parser.add_argument('--days', type=int, default=365, help='Number of days of historical data')
|
||||
parser.add_argument('--rsi-period', type=int, default=14, help='RSI period')
|
||||
parser.add_argument('--output', type=str, default='data', help='Output directory')
|
||||
parser.add_argument('--min-strength', type=float, default=0.15,
|
||||
help='Minimum divergence strength (0-1)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Convert timeframe string to MT5 constant
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[args.timeframe]
|
||||
|
||||
# Create output directory
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
# Fetch data
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=args.days)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("BTCUSD RSI Divergence Data Collection")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
df = fetch_mt5_data(args.symbol, timeframe, start_date, end_date)
|
||||
|
||||
# Prepare features
|
||||
print("\nPreparing features...")
|
||||
df = prepare_features(df)
|
||||
|
||||
# Detect and label divergences
|
||||
print("\nDetecting RSI divergences...")
|
||||
detector = RSIDivergenceDetector(
|
||||
rsi_period=args.rsi_period,
|
||||
min_divergence_strength=args.min_strength
|
||||
)
|
||||
|
||||
df = detector.label_data(df)
|
||||
|
||||
# Statistics
|
||||
total_bars = len(df)
|
||||
labeled_bars = len(df[df['divergence_type'] != DivergenceType.NONE.value])
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("Labeling Statistics:")
|
||||
print(f"{'='*60}")
|
||||
print(f"Total bars: {total_bars}")
|
||||
print(f"Bars with divergence: {labeled_bars} ({labeled_bars/total_bars*100:.2f}%)")
|
||||
|
||||
for div_type in DivergenceType:
|
||||
if div_type == DivergenceType.NONE:
|
||||
continue
|
||||
count = len(df[df['divergence_type'] == div_type.value])
|
||||
print(f" {div_type.name}: {count} ({count/total_bars*100:.2f}%)")
|
||||
|
||||
# Save labeled data
|
||||
output_file = os.path.join(args.output, f"{args.symbol}_{args.timeframe}_labeled.csv")
|
||||
df.to_csv(output_file)
|
||||
print(f"\nLabeled data saved to: {output_file}")
|
||||
|
||||
# Save detector parameters
|
||||
detector_params = {
|
||||
'rsi_period': args.rsi_period,
|
||||
'min_swing_bars': detector.min_swing_bars,
|
||||
'max_swing_bars': detector.max_swing_bars,
|
||||
'min_divergence_strength': args.min_strength
|
||||
}
|
||||
|
||||
params_file = os.path.join(args.output, f"{args.symbol}_{args.timeframe}_detector_params.pkl")
|
||||
with open(params_file, 'wb') as f:
|
||||
pickle.dump(detector_params, f)
|
||||
print(f"Detector parameters saved to: {params_file}")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("Data collection completed!")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,17 @@
|
||||
# ONNX and Machine Learning
|
||||
onnx>=1.12.0
|
||||
onnxruntime>=1.12.0
|
||||
tensorflow>=2.10.0
|
||||
tf2onnx>=1.13.0
|
||||
|
||||
# Data Processing
|
||||
pandas>=1.3.0
|
||||
numpy>=1.21.0
|
||||
scikit-learn>=1.0.0
|
||||
|
||||
# MetaTrader 5 Integration
|
||||
MetaTrader5>=5.0.45
|
||||
|
||||
# Visualization and Utilities
|
||||
matplotlib>=3.4.0
|
||||
tqdm>=4.64.0
|
||||
@@ -0,0 +1,396 @@
|
||||
"""
|
||||
RSI Divergence Detection Module
|
||||
Detects regular and hidden RSI divergences in price action.
|
||||
|
||||
Regular Divergence:
|
||||
- Bullish: Price makes lower low, RSI makes higher low (reversal signal)
|
||||
- Bearish: Price makes higher high, RSI makes lower high (reversal signal)
|
||||
|
||||
Hidden Divergence:
|
||||
- Bullish: Price makes higher low, RSI makes lower low (continuation signal)
|
||||
- Bearish: Price makes lower high, RSI makes higher high (continuation signal)
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from typing import Tuple, Optional, List, Dict
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class DivergenceType(Enum):
|
||||
"""Types of RSI divergences"""
|
||||
NONE = 0
|
||||
REGULAR_BULLISH = 1 # Price lower low, RSI higher low
|
||||
REGULAR_BEARISH = 2 # Price higher high, RSI lower high
|
||||
HIDDEN_BULLISH = 3 # Price higher low, RSI lower low
|
||||
HIDDEN_BEARISH = 4 # Price lower high, RSI higher high
|
||||
|
||||
|
||||
@dataclass
|
||||
class DivergenceSignal:
|
||||
"""Represents a detected divergence signal"""
|
||||
type: DivergenceType
|
||||
price_swing_start: int # Index of price swing start
|
||||
price_swing_end: int # Index of price swing end
|
||||
rsi_swing_start: int # Index of RSI swing start
|
||||
rsi_swing_end: int # Index of RSI swing end
|
||||
price_start: float # Price at swing start
|
||||
price_end: float # Price at swing end
|
||||
rsi_start: float # RSI at swing start
|
||||
rsi_end: float # RSI at swing end
|
||||
strength: float # Divergence strength (0-1)
|
||||
confidence: float # Confidence score (0-1)
|
||||
timestamp: pd.Timestamp
|
||||
|
||||
|
||||
class RSIDivergenceDetector:
|
||||
"""
|
||||
Detects RSI divergences in price data.
|
||||
"""
|
||||
|
||||
def __init__(self, rsi_period: int = 14, min_swing_bars: int = 5,
|
||||
max_swing_bars: int = 50, min_divergence_strength: float = 0.1):
|
||||
"""
|
||||
Initialize the RSI divergence detector.
|
||||
|
||||
Args:
|
||||
rsi_period: Period for RSI calculation
|
||||
min_swing_bars: Minimum bars for a valid swing
|
||||
max_swing_bars: Maximum bars to look back for swings
|
||||
min_divergence_strength: Minimum strength for valid divergence
|
||||
"""
|
||||
self.rsi_period = rsi_period
|
||||
self.min_swing_bars = min_swing_bars
|
||||
self.max_swing_bars = max_swing_bars
|
||||
self.min_divergence_strength = min_divergence_strength
|
||||
|
||||
def calculate_rsi(self, prices: pd.Series, period: int = None) -> pd.Series:
|
||||
"""
|
||||
Calculate RSI indicator.
|
||||
|
||||
Args:
|
||||
prices: Price series (typically close prices)
|
||||
period: RSI period (defaults to self.rsi_period)
|
||||
|
||||
Returns:
|
||||
RSI values
|
||||
"""
|
||||
if period is None:
|
||||
period = self.rsi_period
|
||||
|
||||
delta = prices.diff()
|
||||
gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
|
||||
loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
|
||||
|
||||
# Avoid division by zero
|
||||
rs = gain / (loss + 1e-10)
|
||||
rsi = 100 - (100 / (1 + rs))
|
||||
|
||||
return rsi
|
||||
|
||||
def find_swings(self, data: pd.Series, lookback: int = None) -> Tuple[List[int], List[int]]:
|
||||
"""
|
||||
Find swing highs and lows in the data.
|
||||
|
||||
Args:
|
||||
data: Series to find swings in (price or RSI)
|
||||
lookback: Number of bars to look back (defaults to max_swing_bars)
|
||||
|
||||
Returns:
|
||||
Tuple of (swing_highs, swing_lows) - lists of indices
|
||||
"""
|
||||
if lookback is None:
|
||||
lookback = self.max_swing_bars
|
||||
|
||||
swing_highs = []
|
||||
swing_lows = []
|
||||
|
||||
for i in range(lookback, len(data) - lookback):
|
||||
# Check for swing high
|
||||
is_swing_high = True
|
||||
for j in range(i - lookback, i + lookback + 1):
|
||||
if j != i and data.iloc[j] >= data.iloc[i]:
|
||||
is_swing_high = False
|
||||
break
|
||||
|
||||
if is_swing_high:
|
||||
swing_highs.append(i)
|
||||
|
||||
# Check for swing low
|
||||
is_swing_low = True
|
||||
for j in range(i - lookback, i + lookback + 1):
|
||||
if j != i and data.iloc[j] <= data.iloc[i]:
|
||||
is_swing_low = False
|
||||
break
|
||||
|
||||
if is_swing_low:
|
||||
swing_lows.append(i)
|
||||
|
||||
return swing_highs, swing_lows
|
||||
|
||||
def detect_divergence(self, df: pd.DataFrame, current_index: int) -> Optional[DivergenceSignal]:
|
||||
"""
|
||||
Detect divergence at the current index.
|
||||
|
||||
Args:
|
||||
df: DataFrame with 'close' and 'rsi' columns
|
||||
current_index: Current bar index to check for divergence
|
||||
|
||||
Returns:
|
||||
DivergenceSignal if found, None otherwise
|
||||
"""
|
||||
if current_index < self.max_swing_bars * 2:
|
||||
return None
|
||||
|
||||
# Get price and RSI data up to current index
|
||||
price_data = df['close'].iloc[:current_index + 1]
|
||||
rsi_data = df['rsi'].iloc[:current_index + 1]
|
||||
|
||||
# Find recent swings
|
||||
price_highs, price_lows = self.find_swings(price_data, self.max_swing_bars)
|
||||
rsi_highs, rsi_lows = self.find_swings(rsi_data, self.max_swing_bars)
|
||||
|
||||
if len(price_highs) < 2 or len(price_lows) < 2:
|
||||
return None
|
||||
if len(rsi_highs) < 2 or len(rsi_lows) < 2:
|
||||
return None
|
||||
|
||||
# Get the two most recent swings
|
||||
current_price = price_data.iloc[current_index]
|
||||
current_rsi = rsi_data.iloc[current_index]
|
||||
|
||||
# Check for regular bearish divergence (price higher high, RSI lower high)
|
||||
if len(price_highs) >= 2 and len(rsi_highs) >= 2:
|
||||
price_high_1_idx = price_highs[-1]
|
||||
price_high_2_idx = price_highs[-2] if len(price_highs) >= 2 else price_highs[-1]
|
||||
|
||||
rsi_high_1_idx = rsi_highs[-1]
|
||||
rsi_high_2_idx = rsi_highs[-2] if len(rsi_highs) >= 2 else rsi_highs[-1]
|
||||
|
||||
# Regular bearish: price higher high, RSI lower high
|
||||
if (price_high_1_idx == current_index or abs(price_high_1_idx - current_index) <= 3):
|
||||
if price_data.iloc[price_high_1_idx] > price_data.iloc[price_high_2_idx]:
|
||||
if rsi_data.iloc[rsi_high_1_idx] < rsi_data.iloc[rsi_high_2_idx]:
|
||||
strength = self._calculate_strength(
|
||||
price_data.iloc[price_high_2_idx], price_data.iloc[price_high_1_idx],
|
||||
rsi_data.iloc[rsi_high_2_idx], rsi_data.iloc[rsi_high_1_idx]
|
||||
)
|
||||
if strength >= self.min_divergence_strength:
|
||||
return DivergenceSignal(
|
||||
type=DivergenceType.REGULAR_BEARISH,
|
||||
price_swing_start=price_high_2_idx,
|
||||
price_swing_end=price_high_1_idx,
|
||||
rsi_swing_start=rsi_high_2_idx,
|
||||
rsi_swing_end=rsi_high_1_idx,
|
||||
price_start=price_data.iloc[price_high_2_idx],
|
||||
price_end=price_data.iloc[price_high_1_idx],
|
||||
rsi_start=rsi_data.iloc[rsi_high_2_idx],
|
||||
rsi_end=rsi_data.iloc[rsi_high_1_idx],
|
||||
strength=strength,
|
||||
confidence=self._calculate_confidence(df, price_high_1_idx, DivergenceType.REGULAR_BEARISH),
|
||||
timestamp=df.index[current_index]
|
||||
)
|
||||
|
||||
# Check for regular bullish divergence (price lower low, RSI higher low)
|
||||
if len(price_lows) >= 2 and len(rsi_lows) >= 2:
|
||||
price_low_1_idx = price_lows[-1]
|
||||
price_low_2_idx = price_lows[-2] if len(price_lows) >= 2 else price_lows[-1]
|
||||
|
||||
rsi_low_1_idx = rsi_lows[-1]
|
||||
rsi_low_2_idx = rsi_lows[-2] if len(rsi_lows) >= 2 else rsi_lows[-1]
|
||||
|
||||
# Regular bullish: price lower low, RSI higher low
|
||||
if (price_low_1_idx == current_index or abs(price_low_1_idx - current_index) <= 3):
|
||||
if price_data.iloc[price_low_1_idx] < price_data.iloc[price_low_2_idx]:
|
||||
if rsi_data.iloc[rsi_low_1_idx] > rsi_data.iloc[rsi_low_2_idx]:
|
||||
strength = self._calculate_strength(
|
||||
price_data.iloc[price_low_2_idx], price_data.iloc[price_low_1_idx],
|
||||
rsi_data.iloc[rsi_low_2_idx], rsi_data.iloc[rsi_low_1_idx],
|
||||
reverse=True
|
||||
)
|
||||
if strength >= self.min_divergence_strength:
|
||||
return DivergenceSignal(
|
||||
type=DivergenceType.REGULAR_BULLISH,
|
||||
price_swing_start=price_low_2_idx,
|
||||
price_swing_end=price_low_1_idx,
|
||||
rsi_swing_start=rsi_low_2_idx,
|
||||
rsi_swing_end=rsi_low_1_idx,
|
||||
price_start=price_data.iloc[price_low_2_idx],
|
||||
price_end=price_data.iloc[price_low_1_idx],
|
||||
rsi_start=rsi_data.iloc[rsi_low_2_idx],
|
||||
rsi_end=rsi_data.iloc[rsi_low_1_idx],
|
||||
strength=strength,
|
||||
confidence=self._calculate_confidence(df, price_low_1_idx, DivergenceType.REGULAR_BULLISH),
|
||||
timestamp=df.index[current_index]
|
||||
)
|
||||
|
||||
# Check for hidden bearish divergence (price lower high, RSI higher high)
|
||||
if len(price_highs) >= 2 and len(rsi_highs) >= 2:
|
||||
price_high_1_idx = price_highs[-1]
|
||||
price_high_2_idx = price_highs[-2] if len(price_highs) >= 2 else price_highs[-1]
|
||||
|
||||
rsi_high_1_idx = rsi_highs[-1]
|
||||
rsi_high_2_idx = rsi_highs[-2] if len(rsi_highs) >= 2 else rsi_highs[-1]
|
||||
|
||||
# Hidden bearish: price lower high, RSI higher high
|
||||
if (price_high_1_idx == current_index or abs(price_high_1_idx - current_index) <= 3):
|
||||
if price_data.iloc[price_high_1_idx] < price_data.iloc[price_high_2_idx]:
|
||||
if rsi_data.iloc[rsi_high_1_idx] > rsi_data.iloc[rsi_high_2_idx]:
|
||||
strength = self._calculate_strength(
|
||||
price_data.iloc[price_high_2_idx], price_data.iloc[price_high_1_idx],
|
||||
rsi_data.iloc[rsi_high_2_idx], rsi_data.iloc[rsi_high_1_idx]
|
||||
)
|
||||
if strength >= self.min_divergence_strength:
|
||||
return DivergenceSignal(
|
||||
type=DivergenceType.HIDDEN_BEARISH,
|
||||
price_swing_start=price_high_2_idx,
|
||||
price_swing_end=price_high_1_idx,
|
||||
rsi_swing_start=rsi_high_2_idx,
|
||||
rsi_swing_end=rsi_high_1_idx,
|
||||
price_start=price_data.iloc[price_high_2_idx],
|
||||
price_end=price_data.iloc[price_high_1_idx],
|
||||
rsi_start=rsi_data.iloc[rsi_high_2_idx],
|
||||
rsi_end=rsi_data.iloc[rsi_high_1_idx],
|
||||
strength=strength,
|
||||
confidence=self._calculate_confidence(df, price_high_1_idx, DivergenceType.HIDDEN_BEARISH),
|
||||
timestamp=df.index[current_index]
|
||||
)
|
||||
|
||||
# Check for hidden bullish divergence (price higher low, RSI lower low)
|
||||
if len(price_lows) >= 2 and len(rsi_lows) >= 2:
|
||||
price_low_1_idx = price_lows[-1]
|
||||
price_low_2_idx = price_lows[-2] if len(price_lows) >= 2 else price_lows[-1]
|
||||
|
||||
rsi_low_1_idx = rsi_lows[-1]
|
||||
rsi_low_2_idx = rsi_lows[-2] if len(rsi_lows) >= 2 else rsi_lows[-1]
|
||||
|
||||
# Hidden bullish: price higher low, RSI lower low
|
||||
if (price_low_1_idx == current_index or abs(price_low_1_idx - current_index) <= 3):
|
||||
if price_data.iloc[price_low_1_idx] > price_data.iloc[price_low_2_idx]:
|
||||
if rsi_data.iloc[rsi_low_1_idx] < rsi_data.iloc[rsi_low_2_idx]:
|
||||
strength = self._calculate_strength(
|
||||
price_data.iloc[price_low_2_idx], price_data.iloc[price_low_1_idx],
|
||||
rsi_data.iloc[rsi_low_2_idx], rsi_data.iloc[rsi_low_1_idx],
|
||||
reverse=True
|
||||
)
|
||||
if strength >= self.min_divergence_strength:
|
||||
return DivergenceSignal(
|
||||
type=DivergenceType.HIDDEN_BULLISH,
|
||||
price_swing_start=price_low_2_idx,
|
||||
price_swing_end=price_low_1_idx,
|
||||
rsi_swing_start=rsi_low_2_idx,
|
||||
rsi_swing_end=rsi_low_1_idx,
|
||||
price_start=price_data.iloc[price_low_2_idx],
|
||||
price_end=price_data.iloc[price_low_1_idx],
|
||||
rsi_start=rsi_data.iloc[rsi_low_2_idx],
|
||||
rsi_end=rsi_data.iloc[rsi_low_1_idx],
|
||||
strength=strength,
|
||||
confidence=self._calculate_confidence(df, price_low_1_idx, DivergenceType.HIDDEN_BULLISH),
|
||||
timestamp=df.index[current_index]
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
def _calculate_strength(self, price1: float, price2: float,
|
||||
rsi1: float, rsi2: float, reverse: bool = False) -> float:
|
||||
"""
|
||||
Calculate divergence strength (0-1).
|
||||
|
||||
Args:
|
||||
price1: First price value
|
||||
price2: Second price value
|
||||
rsi1: First RSI value
|
||||
rsi2: Second RSI value
|
||||
reverse: If True, reverse the calculation for bullish divergences
|
||||
|
||||
Returns:
|
||||
Strength score (0-1)
|
||||
"""
|
||||
if price1 == 0 or price2 == 0:
|
||||
return 0.0
|
||||
|
||||
price_change_pct = abs((price2 - price1) / price1)
|
||||
rsi_change = abs(rsi2 - rsi1)
|
||||
|
||||
# Normalize to 0-1 range
|
||||
price_strength = min(price_change_pct * 10, 1.0) # Scale price change
|
||||
rsi_strength = min(rsi_change / 20.0, 1.0) # Scale RSI change (max ~20 points)
|
||||
|
||||
# Combined strength
|
||||
strength = (price_strength + rsi_strength) / 2.0
|
||||
|
||||
return min(max(strength, 0.0), 1.0)
|
||||
|
||||
def _calculate_confidence(self, df: pd.DataFrame, signal_index: int,
|
||||
divergence_type: DivergenceType) -> float:
|
||||
"""
|
||||
Calculate confidence score for a divergence signal.
|
||||
|
||||
Args:
|
||||
df: DataFrame with market data
|
||||
signal_index: Index where divergence was detected
|
||||
divergence_type: Type of divergence
|
||||
|
||||
Returns:
|
||||
Confidence score (0-1)
|
||||
"""
|
||||
confidence = 0.5 # Base confidence
|
||||
|
||||
# Check RSI extremes
|
||||
if signal_index < len(df):
|
||||
rsi = df['rsi'].iloc[signal_index]
|
||||
|
||||
# Higher confidence if RSI is in extreme zones
|
||||
if divergence_type in [DivergenceType.REGULAR_BULLISH, DivergenceType.HIDDEN_BULLISH]:
|
||||
if rsi < 30:
|
||||
confidence += 0.2
|
||||
elif rsi < 40:
|
||||
confidence += 0.1
|
||||
elif divergence_type in [DivergenceType.REGULAR_BEARISH, DivergenceType.HIDDEN_BEARISH]:
|
||||
if rsi > 70:
|
||||
confidence += 0.2
|
||||
elif rsi > 60:
|
||||
confidence += 0.1
|
||||
|
||||
# Check volume (if available)
|
||||
if 'tick_volume' in df.columns and signal_index < len(df):
|
||||
volume = df['tick_volume'].iloc[signal_index]
|
||||
avg_volume = df['tick_volume'].rolling(20).mean().iloc[signal_index] if signal_index >= 20 else volume
|
||||
|
||||
if avg_volume > 0:
|
||||
volume_ratio = volume / avg_volume
|
||||
if volume_ratio > 1.2: # Higher volume increases confidence
|
||||
confidence += 0.1
|
||||
|
||||
return min(max(confidence, 0.0), 1.0)
|
||||
|
||||
def label_data(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Label entire dataset with divergence signals.
|
||||
|
||||
Args:
|
||||
df: DataFrame with 'close' column and datetime index
|
||||
|
||||
Returns:
|
||||
DataFrame with 'divergence_type' and 'divergence_confidence' columns
|
||||
"""
|
||||
# Calculate RSI
|
||||
if 'rsi' not in df.columns:
|
||||
df['rsi'] = self.calculate_rsi(df['close'], self.rsi_period)
|
||||
|
||||
# Initialize labels
|
||||
df['divergence_type'] = DivergenceType.NONE.value
|
||||
df['divergence_confidence'] = 0.0
|
||||
df['divergence_strength'] = 0.0
|
||||
|
||||
# Detect divergences at each point
|
||||
for i in range(self.max_swing_bars * 2, len(df)):
|
||||
signal = self.detect_divergence(df, i)
|
||||
|
||||
if signal:
|
||||
df.loc[df.index[i], 'divergence_type'] = signal.type.value
|
||||
df.loc[df.index[i], 'divergence_confidence'] = signal.confidence
|
||||
df.loc[df.index[i], 'divergence_strength'] = signal.strength
|
||||
|
||||
return df
|
||||
@@ -0,0 +1,337 @@
|
||||
"""
|
||||
ONNX Model Training Script for RSI Divergence Classification
|
||||
Trains a neural network to identify genuine RSI divergences and exports to ONNX format.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import tensorflow as tf
|
||||
from tensorflow import keras
|
||||
from tensorflow.keras import layers
|
||||
from sklearn.preprocessing import MinMaxScaler, LabelEncoder
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn.metrics import classification_report, confusion_matrix
|
||||
import tf2onnx
|
||||
import onnx
|
||||
import pickle
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
class RSIDivergenceTrainer:
|
||||
"""
|
||||
Trainer class for creating ONNX models to classify RSI divergences.
|
||||
"""
|
||||
|
||||
def __init__(self, lookback: int = 60, num_classes: int = 5):
|
||||
"""
|
||||
Initialize the trainer.
|
||||
|
||||
Args:
|
||||
lookback: Number of bars to look back for prediction
|
||||
num_classes: Number of divergence classes (5: NONE + 4 divergence types)
|
||||
"""
|
||||
self.lookback = lookback
|
||||
self.num_classes = num_classes
|
||||
self.scaler = MinMaxScaler()
|
||||
self.label_encoder = LabelEncoder()
|
||||
self.model = None
|
||||
|
||||
def load_data(self, data_path: str) -> tuple:
|
||||
"""
|
||||
Load labeled data from CSV file.
|
||||
|
||||
Args:
|
||||
data_path: Path to labeled CSV file
|
||||
|
||||
Returns:
|
||||
Tuple of (X, y) where X is features and y is labels
|
||||
"""
|
||||
print(f"Loading data from {data_path}...")
|
||||
df = pd.read_csv(data_path, index_col=0, parse_dates=True)
|
||||
|
||||
# Exclude label columns from features
|
||||
exclude_cols = ['divergence_type', 'divergence_confidence', 'divergence_strength']
|
||||
feature_cols = [col for col in df.columns if col not in exclude_cols]
|
||||
|
||||
# Remove any remaining non-numeric columns
|
||||
feature_cols = [col for col in feature_cols if df[col].dtype in [np.float64, np.int64, np.float32, np.int32]]
|
||||
|
||||
print(f"Using {len(feature_cols)} features: {feature_cols[:10]}...")
|
||||
|
||||
# Prepare sequences
|
||||
X, y = [], []
|
||||
|
||||
for i in range(self.lookback, len(df)):
|
||||
# Get feature sequence
|
||||
X.append(df[feature_cols].iloc[i - self.lookback:i].values)
|
||||
|
||||
# Get label (divergence type at current bar)
|
||||
y.append(int(df['divergence_type'].iloc[i]))
|
||||
|
||||
X = np.array(X)
|
||||
y = np.array(y)
|
||||
|
||||
print(f"Created {len(X)} sequences")
|
||||
print(f"Label distribution: {np.bincount(y)}")
|
||||
|
||||
return X, y, feature_cols
|
||||
|
||||
def prepare_data(self, X: np.ndarray, y: np.ndarray) -> tuple:
|
||||
"""
|
||||
Prepare and scale data for training.
|
||||
|
||||
Args:
|
||||
X: Feature sequences
|
||||
y: Labels
|
||||
|
||||
Returns:
|
||||
Tuple of (X_scaled, y_encoded, X_train, X_test, y_train, y_test)
|
||||
"""
|
||||
# Scale features
|
||||
print("Scaling features...")
|
||||
original_shape = X.shape
|
||||
X_reshaped = X.reshape(-1, X.shape[-1])
|
||||
X_scaled = self.scaler.fit_transform(X_reshaped)
|
||||
X_scaled = X_scaled.reshape(original_shape)
|
||||
|
||||
# Encode labels (already integers, but ensure they're 0-4)
|
||||
y_encoded = y.astype(int)
|
||||
|
||||
# Split data (no shuffle to preserve temporal order)
|
||||
X_train, X_test, y_train, y_test = train_test_split(
|
||||
X_scaled, y_encoded, test_size=0.2, shuffle=False
|
||||
)
|
||||
|
||||
print(f"Training samples: {len(X_train)}")
|
||||
print(f"Test samples: {len(X_test)}")
|
||||
|
||||
return X_scaled, y_encoded, X_train, X_test, y_train, y_test
|
||||
|
||||
def build_model(self, input_shape: tuple) -> keras.Model:
|
||||
"""
|
||||
Build the neural network model for classification.
|
||||
|
||||
Args:
|
||||
input_shape: Shape of input data (lookback, features)
|
||||
|
||||
Returns:
|
||||
Compiled Keras model
|
||||
"""
|
||||
model = keras.Sequential([
|
||||
# LSTM layers for sequence learning
|
||||
layers.LSTM(128, return_sequences=True, input_shape=input_shape),
|
||||
layers.Dropout(0.3),
|
||||
layers.LSTM(64, return_sequences=True),
|
||||
layers.Dropout(0.3),
|
||||
layers.LSTM(32),
|
||||
layers.Dropout(0.3),
|
||||
|
||||
# Dense layers for classification
|
||||
layers.Dense(64, activation='relu'),
|
||||
layers.Dropout(0.2),
|
||||
layers.Dense(32, activation='relu'),
|
||||
layers.Dropout(0.2),
|
||||
layers.Dense(self.num_classes, activation='softmax') # Multi-class classification
|
||||
])
|
||||
|
||||
model.compile(
|
||||
optimizer=keras.optimizers.Adam(learning_rate=0.001),
|
||||
loss='sparse_categorical_crossentropy',
|
||||
metrics=['accuracy']
|
||||
)
|
||||
|
||||
return model
|
||||
|
||||
def train(self, X_train: np.ndarray, y_train: np.ndarray,
|
||||
X_test: np.ndarray, y_test: np.ndarray,
|
||||
epochs: int = 50, batch_size: int = 32, verbose: int = 1):
|
||||
"""
|
||||
Train the model.
|
||||
|
||||
Args:
|
||||
X_train: Training features
|
||||
y_train: Training labels
|
||||
X_test: Test features
|
||||
y_test: Test labels
|
||||
epochs: Number of training epochs
|
||||
batch_size: Batch size for training
|
||||
verbose: Verbosity level
|
||||
"""
|
||||
# Build model
|
||||
self.model = self.build_model((X_train.shape[1], X_train.shape[2]))
|
||||
|
||||
print("\nModel architecture:")
|
||||
self.model.summary()
|
||||
|
||||
# Handle class imbalance with class weights
|
||||
from sklearn.utils.class_weight import compute_class_weight
|
||||
class_weights = compute_class_weight(
|
||||
'balanced',
|
||||
classes=np.unique(y_train),
|
||||
y=y_train
|
||||
)
|
||||
class_weight_dict = {i: weight for i, weight in enumerate(class_weights)}
|
||||
|
||||
print(f"\nClass weights: {class_weight_dict}")
|
||||
|
||||
# Train model
|
||||
print("\nTraining model...")
|
||||
history = self.model.fit(
|
||||
X_train, y_train,
|
||||
batch_size=batch_size,
|
||||
epochs=epochs,
|
||||
validation_data=(X_test, y_test),
|
||||
verbose=verbose,
|
||||
class_weight=class_weight_dict,
|
||||
callbacks=[
|
||||
keras.callbacks.EarlyStopping(
|
||||
monitor='val_loss',
|
||||
patience=15,
|
||||
restore_best_weights=True,
|
||||
verbose=1
|
||||
),
|
||||
keras.callbacks.ReduceLROnPlateau(
|
||||
monitor='val_loss',
|
||||
factor=0.5,
|
||||
patience=5,
|
||||
min_lr=0.0001,
|
||||
verbose=1
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
# Evaluate
|
||||
train_loss, train_acc = self.model.evaluate(X_train, y_train, verbose=0)
|
||||
test_loss, test_acc = self.model.evaluate(X_test, y_test, verbose=0)
|
||||
|
||||
print(f"\nTraining - Loss: {train_loss:.4f}, Accuracy: {train_acc:.4f}")
|
||||
print(f"Test - Loss: {test_loss:.4f}, Accuracy: {test_acc:.4f}")
|
||||
|
||||
# Classification report
|
||||
y_pred = self.model.predict(X_test, verbose=0)
|
||||
y_pred_classes = np.argmax(y_pred, axis=1)
|
||||
|
||||
print("\nClassification Report:")
|
||||
print(classification_report(y_test, y_pred_classes,
|
||||
target_names=['NONE', 'REGULAR_BULLISH', 'REGULAR_BEARISH',
|
||||
'HIDDEN_BULLISH', 'HIDDEN_BEARISH']))
|
||||
|
||||
return history
|
||||
|
||||
def export_to_onnx(self, output_path: str, num_features: int):
|
||||
"""
|
||||
Export the trained model to ONNX format.
|
||||
|
||||
Args:
|
||||
output_path: Path to save ONNX model
|
||||
num_features: Number of input features
|
||||
"""
|
||||
if self.model is None:
|
||||
raise ValueError("Model must be trained before exporting")
|
||||
|
||||
print(f"\nExporting model to ONNX format: {output_path}")
|
||||
|
||||
# Create functional model from Sequential
|
||||
input_layer = keras.Input(shape=(self.lookback, num_features), name="input")
|
||||
x = input_layer
|
||||
|
||||
# Rebuild model as functional
|
||||
for layer in self.model.layers:
|
||||
x = layer(x)
|
||||
|
||||
functional_model = keras.Model(inputs=input_layer, outputs=x)
|
||||
|
||||
# Convert to ONNX
|
||||
spec = (tf.TensorSpec((None, self.lookback, num_features), tf.float32, name="input"),)
|
||||
|
||||
try:
|
||||
onnx_model_proto, _ = tf2onnx.convert.from_keras(
|
||||
functional_model,
|
||||
input_signature=spec,
|
||||
opset=13
|
||||
)
|
||||
|
||||
onnx.save_model(onnx_model_proto, output_path)
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
|
||||
# Verify ONNX model
|
||||
onnx_model = onnx.load(output_path)
|
||||
onnx.checker.check_model(onnx_model)
|
||||
print("ONNX model validation passed")
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to export ONNX model: {str(e)}")
|
||||
|
||||
def save_scaler(self, output_path: str):
|
||||
"""Save the scaler for consistent normalization."""
|
||||
with open(output_path, 'wb') as f:
|
||||
pickle.dump(self.scaler, f)
|
||||
print(f"Scaler saved to: {output_path}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(description='Train ONNX model for RSI divergence classification')
|
||||
parser.add_argument('--data', type=str, required=True,
|
||||
help='Path to labeled CSV data file')
|
||||
parser.add_argument('--lookback', type=int, default=60,
|
||||
help='Number of bars to look back')
|
||||
parser.add_argument('--epochs', type=int, default=50, help='Training epochs')
|
||||
parser.add_argument('--batch-size', type=int, default=32, help='Batch size')
|
||||
parser.add_argument('--output', type=str, default='models',
|
||||
help='Output directory for ONNX model')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create output directory
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
# Create trainer
|
||||
trainer = RSIDivergenceTrainer(lookback=args.lookback)
|
||||
|
||||
try:
|
||||
# Load data
|
||||
X, y, feature_cols = trainer.load_data(args.data)
|
||||
|
||||
# Prepare data
|
||||
X_scaled, y_encoded, X_train, X_test, y_train, y_test = trainer.prepare_data(X, y)
|
||||
|
||||
# Train model
|
||||
trainer.train(X_train, y_train, X_test, y_test,
|
||||
epochs=args.epochs, batch_size=args.batch_size)
|
||||
|
||||
# Export to ONNX
|
||||
num_features = len(feature_cols)
|
||||
model_name = "BTCUSD_H1_rsi_divergence_model.onnx"
|
||||
output_path = os.path.join(args.output, model_name)
|
||||
trainer.export_to_onnx(output_path, num_features)
|
||||
|
||||
# Save scaler
|
||||
scaler_name = "BTCUSD_H1_rsi_divergence_scaler.pkl"
|
||||
scaler_path = os.path.join(args.output, scaler_name)
|
||||
trainer.save_scaler(scaler_path)
|
||||
|
||||
# Save feature list
|
||||
features_name = "BTCUSD_H1_rsi_divergence_features.pkl"
|
||||
features_path = os.path.join(args.output, features_name)
|
||||
with open(features_path, 'wb') as f:
|
||||
pickle.dump(feature_cols, f)
|
||||
print(f"Feature list saved to: {features_path}")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("Training completed successfully!")
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nError: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user