Update
This commit is contained in:
zhutoutoutousan
@@ -0,0 +1,53 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Models and Data
|
||||
models/
|
||||
*.onnx
|
||||
*.h5
|
||||
*.pb
|
||||
*.pkl
|
||||
*.csv
|
||||
*.npy
|
||||
|
||||
# TensorFlow
|
||||
*.ckpt
|
||||
checkpoints/
|
||||
logs/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
*.ipynb
|
||||
@@ -0,0 +1,121 @@
|
||||
# MetaTrader 5 ONNX EA Setup Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **Copy Model Files to MT5**
|
||||
- Copy `models/XAUUSD_H1_model.onnx` to: `MT5_Data_Folder/MQL5/Files/models/`
|
||||
- The EA will look for the model at: `models\XAUUSD_H1_model.onnx`
|
||||
|
||||
2. **Compile the EA**
|
||||
- Open `ai/ONNX_EA.mq5` in MetaEditor
|
||||
- Press F7 to compile
|
||||
- Check for any errors
|
||||
|
||||
3. **Attach to Chart**
|
||||
- Open XAUUSD H1 chart in MT5
|
||||
- Drag `ONNX_EA.ex5` from Navigator to chart
|
||||
- Configure parameters (see below)
|
||||
|
||||
## Model Information
|
||||
|
||||
- **Model Type**: LSTM Neural Network
|
||||
- **Input**: 60 bars × 13 features
|
||||
- **Output**: Price change percentage (e.g., -0.003 = -0.3% decrease)
|
||||
- **Features**: OHLC, volume, RSI, EMA20, EMA50, ATR, price_change, high_low_ratio, volume_ma, volume_ratio
|
||||
|
||||
## EA Parameters
|
||||
|
||||
### ONNX Model Settings
|
||||
- **InpModelPath**: `models\\XAUUSD_H1_model.onnx` (path relative to MQL5/Files/)
|
||||
- **InpLookback**: `60` (must match training)
|
||||
- **InpUsePrediction**: `true` (enable/disable predictions)
|
||||
|
||||
### Trading Settings
|
||||
- **InpLotSize**: `0.01` (start small for testing)
|
||||
- **InpMagicNumber**: `123456` (unique identifier)
|
||||
- **InpSlippage**: `3` (points)
|
||||
- **InpStopLoss**: `50` (pips)
|
||||
- **InpTakeProfit**: `100` (pips)
|
||||
|
||||
### Prediction Settings
|
||||
- **InpPredictionThreshold**: `0.00005` (0.005% as decimal, minimum change to trade)
|
||||
- **InpUseConfidence**: `true` (enable confidence filter)
|
||||
- **InpMinConfidence**: `0.1` (10% minimum confidence)
|
||||
|
||||
## Important Notes
|
||||
|
||||
### Feature Normalization
|
||||
⚠️ **The EA uses simplified normalization that may not exactly match training.**
|
||||
|
||||
For best results:
|
||||
1. The training script saves a scaler (`XAUUSD_H1_scaler.pkl`)
|
||||
2. You should implement the same MinMaxScaler logic in MQL5
|
||||
3. Or export scaler parameters (min/max) from Python and use in MQL5
|
||||
|
||||
Current implementation uses:
|
||||
- OHLC: Raw values (should be normalized by scaler)
|
||||
- Volume: Divided by 1,000,000
|
||||
- RSI: Divided by 100
|
||||
- EMAs/ATR: Normalized differences
|
||||
- Price change: Percentage
|
||||
- Volume MA: Divided by 1,000,000
|
||||
|
||||
### Prediction Format
|
||||
The new model predicts **price change percentage** directly:
|
||||
- Example: `-0.003` = price will decrease by 0.3%
|
||||
- Old format (absolute price) is also supported for backward compatibility
|
||||
|
||||
### Testing Recommendations
|
||||
|
||||
1. **Start with Strategy Tester**
|
||||
- Use Visual Mode to see predictions
|
||||
- Check Expert tab for prediction logs
|
||||
- Verify predictions make sense
|
||||
|
||||
2. **Monitor Logs**
|
||||
- Check "Experts" tab for prediction values
|
||||
- Verify confidence calculations
|
||||
- Watch for any errors
|
||||
|
||||
3. **Adjust Parameters**
|
||||
- If too many trades: Increase `InpPredictionThreshold` or `InpMinConfidence`
|
||||
- If no trades: Decrease thresholds
|
||||
- Adjust stop loss/take profit based on volatility
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Failed to load ONNX model"
|
||||
- Check model path is correct
|
||||
- Ensure model file exists in `MQL5/Files/models/`
|
||||
- Check file permissions
|
||||
|
||||
### "Failed to prepare input data"
|
||||
- Ensure enough historical data (need 60+ bars)
|
||||
- Check indicator calculations
|
||||
- Verify symbol is XAUUSD
|
||||
|
||||
### "Empty output from ONNX model"
|
||||
- Check model input shape matches (1, 60, 13)
|
||||
- Verify feature preparation matches training
|
||||
- Check ONNX runtime version compatibility
|
||||
|
||||
### Predictions seem wrong
|
||||
- Feature normalization may not match training
|
||||
- Implement proper MinMaxScaler from training
|
||||
- Check feature order matches training (13 features in correct order)
|
||||
|
||||
## Model Training Info
|
||||
|
||||
- **Training Date**: 2026-01-06
|
||||
- **Training MAE**: 0.0013 (0.13%)
|
||||
- **Validation MAE**: 0.0018 (0.18%)
|
||||
- **Data Period**: Last 2 years
|
||||
- **Timeframe**: H1
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Test in Strategy Tester first
|
||||
2. Compare predictions with Python backtest
|
||||
3. Adjust parameters based on results
|
||||
4. Consider implementing proper scaler normalization
|
||||
5. Test on demo account before live trading
|
||||
@@ -0,0 +1,472 @@
|
||||
//+------------------------------------------------------------------+
|
||||
//| ONNX_EA.mq5 |
|
||||
//| Copyright 2025, MetaQuotes Ltd. |
|
||||
//| https://www.mql5.com |
|
||||
//+------------------------------------------------------------------+
|
||||
#property copyright "Copyright 2025, MetaQuotes Ltd."
|
||||
#property link "https://www.mql5.com"
|
||||
#property version "1.00"
|
||||
#property description "Expert Advisor using ONNX model for price prediction"
|
||||
#property description "Based on: https://www.mql5.com/en/docs/onnx/onnx_prepare"
|
||||
|
||||
#include <Trade\Trade.mqh>
|
||||
|
||||
//--- Input parameters
|
||||
input group "ONNX Model Settings"
|
||||
input string InpModelPath = "models\\XAUUSD_H1_model.onnx"; // ONNX Model Path
|
||||
input int InpLookback = 60; // Lookback Period (bars)
|
||||
input bool InpUsePrediction = true; // Use Model Prediction
|
||||
|
||||
input group "Trading Settings"
|
||||
input double InpLotSize = 0.01; // Lot Size
|
||||
input int InpMagicNumber = 123456; // Magic Number
|
||||
input int InpSlippage = 3; // Slippage (points)
|
||||
input int InpStopLoss = 50; // Stop Loss (pips)
|
||||
input int InpTakeProfit = 100; // Take Profit (pips)
|
||||
|
||||
input group "Prediction Settings"
|
||||
input double InpPredictionThreshold = 0.00005; // Min Prediction Change (0.005% as decimal, e.g., 0.00005 = 0.005%)
|
||||
input bool InpUseConfidence = true; // Use Confidence Filter
|
||||
input double InpMinConfidence = 0.1; // Minimum Confidence (0.1 = 10%)
|
||||
|
||||
//--- Global variables
|
||||
CTrade trade;
|
||||
long onnx_handle = INVALID_HANDLE;
|
||||
datetime last_bar_time = 0;
|
||||
double last_prediction = 0.0;
|
||||
double last_confidence = 0.0;
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert initialization function |
|
||||
//+------------------------------------------------------------------+
|
||||
int OnInit()
|
||||
{
|
||||
// Set trade parameters
|
||||
trade.SetExpertMagicNumber(InpMagicNumber);
|
||||
trade.SetDeviationInPoints(InpSlippage);
|
||||
trade.SetTypeFilling(ORDER_FILLING_FOK);
|
||||
|
||||
// Load ONNX model
|
||||
string model_path = InpModelPath;
|
||||
|
||||
// Convert relative path to full path
|
||||
if(StringFind(model_path, "\\") == 0 || StringFind(model_path, "/") == 0)
|
||||
{
|
||||
// Already absolute path
|
||||
}
|
||||
else
|
||||
{
|
||||
// Relative path - prepend terminal data folder
|
||||
model_path = TerminalInfoString(TERMINAL_DATA_PATH) + "\\MQL5\\Files\\" + model_path;
|
||||
}
|
||||
|
||||
// Replace forward slashes with backslashes for Windows
|
||||
StringReplace(model_path, "/", "\\");
|
||||
|
||||
Print("Loading ONNX model from: ", model_path);
|
||||
|
||||
onnx_handle = OnnxCreate(model_path, ONNX_DEFAULT);
|
||||
|
||||
if(onnx_handle == INVALID_HANDLE)
|
||||
{
|
||||
Print("ERROR: Failed to load ONNX model. Error: ", GetLastError());
|
||||
Print("Make sure the model file exists at: ", model_path);
|
||||
return(INIT_FAILED);
|
||||
}
|
||||
|
||||
// Get model info
|
||||
long input_count = OnnxGetInputCount(onnx_handle);
|
||||
long output_count = OnnxGetOutputCount(onnx_handle);
|
||||
|
||||
Print("ONNX Model loaded successfully");
|
||||
Print(" Inputs: ", input_count);
|
||||
Print(" Outputs: ", output_count);
|
||||
|
||||
if(input_count > 0)
|
||||
{
|
||||
string input_name = OnnxGetInputName(onnx_handle, 0);
|
||||
Print(" Input name: ", input_name);
|
||||
}
|
||||
|
||||
if(output_count > 0)
|
||||
{
|
||||
string output_name = OnnxGetOutputName(onnx_handle, 0);
|
||||
Print(" Output name: ", output_name);
|
||||
}
|
||||
|
||||
return(INIT_SUCCEEDED);
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert deinitialization function |
|
||||
//+------------------------------------------------------------------+
|
||||
void OnDeinit(const int reason)
|
||||
{
|
||||
// Release ONNX model
|
||||
if(onnx_handle != INVALID_HANDLE)
|
||||
{
|
||||
OnnxRelease(onnx_handle);
|
||||
Print("ONNX model released");
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Expert tick function |
|
||||
//+------------------------------------------------------------------+
|
||||
void OnTick()
|
||||
{
|
||||
// Check if new bar
|
||||
datetime current_bar_time = iTime(_Symbol, PERIOD_CURRENT, 0);
|
||||
if(current_bar_time == last_bar_time)
|
||||
{
|
||||
return; // Still the same bar
|
||||
}
|
||||
last_bar_time = current_bar_time;
|
||||
|
||||
// Check if we should use prediction
|
||||
if(!InpUsePrediction)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Prepare input data
|
||||
float input_data[];
|
||||
if(!PrepareInputData(input_data))
|
||||
{
|
||||
Print("ERROR: Failed to prepare input data");
|
||||
return;
|
||||
}
|
||||
|
||||
// Run ONNX model
|
||||
float output_data[];
|
||||
if(!RunONNXModel(input_data, output_data))
|
||||
{
|
||||
Print("ERROR: Failed to run ONNX model");
|
||||
return;
|
||||
}
|
||||
|
||||
// Get prediction
|
||||
if(ArraySize(output_data) == 0)
|
||||
{
|
||||
Print("ERROR: Empty output from ONNX model");
|
||||
return;
|
||||
}
|
||||
|
||||
// Model now predicts price change percentage directly (e.g., -0.003 = -0.3%)
|
||||
double predicted_change_pct = output_data[0];
|
||||
double current_price = SymbolInfoDouble(_Symbol, SYMBOL_BID);
|
||||
|
||||
// Check if prediction is percentage (between -1 and 1) or absolute price (old format)
|
||||
double price_change_pct;
|
||||
double predicted_price;
|
||||
|
||||
if(MathAbs(predicted_change_pct) < 1.0)
|
||||
{
|
||||
// New format: percentage (e.g., -0.003 = -0.3%)
|
||||
price_change_pct = predicted_change_pct * 100.0; // Convert to percentage
|
||||
predicted_price = current_price * (1.0 + predicted_change_pct); // Calculate predicted price
|
||||
}
|
||||
else
|
||||
{
|
||||
// Old format: absolute price
|
||||
predicted_price = predicted_change_pct;
|
||||
double price_change = predicted_price - current_price;
|
||||
price_change_pct = (price_change / current_price) * 100.0;
|
||||
}
|
||||
|
||||
// Calculate confidence (for percentage predictions: 0.001 = 0.1% = 10% confidence)
|
||||
double confidence;
|
||||
if(MathAbs(price_change_pct) < 1.0)
|
||||
{
|
||||
// It's a decimal percentage (e.g., 0.001 = 0.1%)
|
||||
confidence = MathMin(MathAbs(predicted_change_pct) / 0.01, 1.0); // 0.01 = 1% = 100% confidence
|
||||
}
|
||||
else
|
||||
{
|
||||
// It's already in percentage form
|
||||
confidence = MathMin(MathAbs(price_change_pct) / 1.0, 1.0);
|
||||
}
|
||||
|
||||
last_prediction = predicted_price;
|
||||
last_confidence = confidence;
|
||||
|
||||
// Log prediction
|
||||
Print("Prediction: Current=", current_price,
|
||||
" Predicted Change=", price_change_pct, "%",
|
||||
" Predicted Price=", predicted_price,
|
||||
" Confidence=", confidence);
|
||||
|
||||
// Check if we should trade
|
||||
if(!InpUseConfidence || confidence >= InpMinConfidence)
|
||||
{
|
||||
// Check if prediction is significant
|
||||
// price_change_pct is now in percentage (e.g., 0.1 = 0.1%), so compare with threshold * 100
|
||||
// OR: if model outputs decimal (0.001), compare directly
|
||||
double abs_change_decimal = MathAbs(predicted_change_pct); // Use raw prediction for threshold check
|
||||
if(abs_change_decimal >= InpPredictionThreshold)
|
||||
{
|
||||
// Check existing position
|
||||
if(PositionSelect(_Symbol))
|
||||
{
|
||||
// Manage existing position
|
||||
ManagePosition(predicted_price, price_change_pct);
|
||||
}
|
||||
else
|
||||
{
|
||||
// Open new position based on prediction
|
||||
// Use raw prediction (decimal format) for threshold comparison
|
||||
if(predicted_change_pct > InpPredictionThreshold)
|
||||
{
|
||||
OpenBuyPosition();
|
||||
}
|
||||
else if(predicted_change_pct < -InpPredictionThreshold)
|
||||
{
|
||||
OpenSellPosition();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Prepare input data for ONNX model |
|
||||
//+------------------------------------------------------------------+
|
||||
bool PrepareInputData(float &input_array[])
|
||||
{
|
||||
// We need to prepare data similar to training
|
||||
// This is a simplified version - you may need to adjust based on your model
|
||||
|
||||
int lookback = InpLookback;
|
||||
int features = 13; // OHLC(4) + volume(1) + RSI(1) + EMA20(1) + EMA50(1) + ATR(1) + price_change(1) + high_low_ratio(1) + volume_ma(1) + volume_ratio(1) = 13
|
||||
|
||||
ArrayResize(input_array, lookback * features);
|
||||
ArrayInitialize(input_array, 0.0);
|
||||
|
||||
// Get historical data
|
||||
double open[], high[], low[], close[];
|
||||
long volume[]; // CopyTickVolume requires long[] not double[]
|
||||
ArraySetAsSeries(open, true);
|
||||
ArraySetAsSeries(high, true);
|
||||
ArraySetAsSeries(low, true);
|
||||
ArraySetAsSeries(close, true);
|
||||
ArraySetAsSeries(volume, true);
|
||||
|
||||
if(CopyOpen(_Symbol, PERIOD_CURRENT, 0, lookback + 50, open) < lookback)
|
||||
return false;
|
||||
if(CopyHigh(_Symbol, PERIOD_CURRENT, 0, lookback + 50, high) < lookback)
|
||||
return false;
|
||||
if(CopyLow(_Symbol, PERIOD_CURRENT, 0, lookback + 50, low) < lookback)
|
||||
return false;
|
||||
if(CopyClose(_Symbol, PERIOD_CURRENT, 0, lookback + 50, close) < lookback)
|
||||
return false;
|
||||
if(CopyTickVolume(_Symbol, PERIOD_CURRENT, 0, lookback + 50, volume) < lookback)
|
||||
return false;
|
||||
|
||||
// Calculate indicators (simplified - you may need to match training exactly)
|
||||
double rsi[], ema20[], ema50[], atr[];
|
||||
ArraySetAsSeries(rsi, true);
|
||||
ArraySetAsSeries(ema20, true);
|
||||
ArraySetAsSeries(ema50, true);
|
||||
ArraySetAsSeries(atr, true);
|
||||
|
||||
// Calculate RSI
|
||||
int rsi_handle = iRSI(_Symbol, PERIOD_CURRENT, 14, PRICE_CLOSE);
|
||||
if(rsi_handle == INVALID_HANDLE) return false;
|
||||
if(CopyBuffer(rsi_handle, 0, 0, lookback + 50, rsi) < lookback)
|
||||
{
|
||||
IndicatorRelease(rsi_handle);
|
||||
return false;
|
||||
}
|
||||
IndicatorRelease(rsi_handle);
|
||||
|
||||
// Calculate EMAs
|
||||
int ema20_handle = iMA(_Symbol, PERIOD_CURRENT, 20, 0, MODE_EMA, PRICE_CLOSE);
|
||||
int ema50_handle = iMA(_Symbol, PERIOD_CURRENT, 50, 0, MODE_EMA, PRICE_CLOSE);
|
||||
if(ema20_handle == INVALID_HANDLE || ema50_handle == INVALID_HANDLE) return false;
|
||||
|
||||
if(CopyBuffer(ema20_handle, 0, 0, lookback + 50, ema20) < lookback ||
|
||||
CopyBuffer(ema50_handle, 0, 0, lookback + 50, ema50) < lookback)
|
||||
{
|
||||
IndicatorRelease(ema20_handle);
|
||||
IndicatorRelease(ema50_handle);
|
||||
return false;
|
||||
}
|
||||
IndicatorRelease(ema20_handle);
|
||||
IndicatorRelease(ema50_handle);
|
||||
|
||||
// Calculate ATR
|
||||
int atr_handle = iATR(_Symbol, PERIOD_CURRENT, 14);
|
||||
if(atr_handle == INVALID_HANDLE) return false;
|
||||
if(CopyBuffer(atr_handle, 0, 0, lookback + 50, atr) < lookback)
|
||||
{
|
||||
IndicatorRelease(atr_handle);
|
||||
return false;
|
||||
}
|
||||
IndicatorRelease(atr_handle);
|
||||
|
||||
// Calculate volume MA for normalization
|
||||
double volume_ma[];
|
||||
ArraySetAsSeries(volume_ma, true);
|
||||
ArrayResize(volume_ma, lookback);
|
||||
ArrayInitialize(volume_ma, 0.0);
|
||||
|
||||
// Calculate volume MA (20-period rolling average)
|
||||
for(int j = 0; j < lookback; j++)
|
||||
{
|
||||
double sum = 0.0;
|
||||
int count = 0;
|
||||
for(int k = j; k < j + 20 && k < ArraySize(volume); k++)
|
||||
{
|
||||
sum += (double)volume[k]; // Convert long to double
|
||||
count++;
|
||||
}
|
||||
volume_ma[j] = count > 0 ? sum / count : (double)volume[j]; // Convert long to double
|
||||
}
|
||||
|
||||
// Prepare features - MUST match Python training exactly (13 features)
|
||||
// IMPORTANT: This uses simplified normalization. For best results, implement MinMaxScaler from training.
|
||||
// The scaler is saved as models/XAUUSD_H1_scaler.pkl - you may need to export scaler parameters to MQL5
|
||||
int idx = 0;
|
||||
for(int i = 0; i < lookback; i++)
|
||||
{
|
||||
// Feature 1-4: OHLC (raw values, will be normalized by scaler)
|
||||
input_array[idx++] = (float)open[i];
|
||||
input_array[idx++] = (float)high[i];
|
||||
input_array[idx++] = (float)low[i];
|
||||
input_array[idx++] = (float)close[i];
|
||||
|
||||
// Feature 5: Volume (normalized by 1,000,000)
|
||||
input_array[idx++] = (float)((double)volume[i] / 1000000.0); // Convert long to double first
|
||||
|
||||
// Feature 6: RSI (normalized by 100)
|
||||
input_array[idx++] = (float)(rsi[i] / 100.0);
|
||||
|
||||
// Feature 7: EMA20 normalized difference
|
||||
input_array[idx++] = (float)((ema20[i] - close[i]) / close[i]);
|
||||
|
||||
// Feature 8: EMA50 normalized difference
|
||||
input_array[idx++] = (float)((ema50[i] - close[i]) / close[i]);
|
||||
|
||||
// Feature 9: ATR normalized
|
||||
input_array[idx++] = (float)(atr[i] / close[i]);
|
||||
|
||||
// Feature 10: Price change (percentage)
|
||||
double price_change = i > 0 ? (close[i] - close[i+1]) / close[i+1] : 0.0;
|
||||
input_array[idx++] = (float)price_change;
|
||||
|
||||
// Feature 11: High/Low ratio
|
||||
input_array[idx++] = (float)(high[i] / low[i]);
|
||||
|
||||
// Feature 12: Volume MA (normalized by 1,000,000)
|
||||
input_array[idx++] = (float)(volume_ma[i] / 1000000.0);
|
||||
|
||||
// Feature 13: Volume ratio
|
||||
double vol_ratio = volume_ma[i] > 0 ? (double)volume[i] / volume_ma[i] : 1.0; // Convert long to double
|
||||
input_array[idx++] = (float)vol_ratio;
|
||||
}
|
||||
|
||||
// Reshape for model: (1, lookback, features)
|
||||
// ONNX expects shape [1, lookback, features]
|
||||
float reshaped[];
|
||||
ArrayResize(reshaped, 1 * lookback * features);
|
||||
ArrayCopy(reshaped, input_array);
|
||||
|
||||
ArrayCopy(input_array, reshaped);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Run ONNX model |
|
||||
//+------------------------------------------------------------------+
|
||||
bool RunONNXModel(float &input_data[], float &output_data[])
|
||||
{
|
||||
if(onnx_handle == INVALID_HANDLE)
|
||||
return false;
|
||||
|
||||
// Set input shape
|
||||
long input_shape[] = {1, InpLookback, 13}; // 13 features
|
||||
if(!OnnxSetInputShape(onnx_handle, 0, input_shape))
|
||||
{
|
||||
Print("ERROR: Failed to set input shape. Error: ", GetLastError());
|
||||
return false;
|
||||
}
|
||||
|
||||
// Run model
|
||||
if(!OnnxRun(onnx_handle, ONNX_NO_CONVERSION, input_data, output_data))
|
||||
{
|
||||
Print("ERROR: Failed to run ONNX model. Error: ", GetLastError());
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Open buy position |
|
||||
//+------------------------------------------------------------------+
|
||||
void OpenBuyPosition()
|
||||
{
|
||||
double price = SymbolInfoDouble(_Symbol, SYMBOL_ASK);
|
||||
double sl = InpStopLoss > 0 ? price - InpStopLoss * _Point * 10 : 0;
|
||||
double tp = InpTakeProfit > 0 ? price + InpTakeProfit * _Point * 10 : 0;
|
||||
|
||||
if(trade.Buy(InpLotSize, _Symbol, price, sl, tp, "ONNX Buy Signal"))
|
||||
{
|
||||
Print("Buy order opened. Ticket: ", trade.ResultOrder());
|
||||
}
|
||||
else
|
||||
{
|
||||
Print("Failed to open buy order. Error: ", trade.ResultRetcodeDescription());
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Open sell position |
|
||||
//+------------------------------------------------------------------+
|
||||
void OpenSellPosition()
|
||||
{
|
||||
double price = SymbolInfoDouble(_Symbol, SYMBOL_BID);
|
||||
double sl = InpStopLoss > 0 ? price + InpStopLoss * _Point * 10 : 0;
|
||||
double tp = InpTakeProfit > 0 ? price - InpTakeProfit * _Point * 10 : 0;
|
||||
|
||||
if(trade.Sell(InpLotSize, _Symbol, price, sl, tp, "ONNX Sell Signal"))
|
||||
{
|
||||
Print("Sell order opened. Ticket: ", trade.ResultOrder());
|
||||
}
|
||||
else
|
||||
{
|
||||
Print("Failed to open sell order. Error: ", trade.ResultRetcodeDescription());
|
||||
}
|
||||
}
|
||||
|
||||
//+------------------------------------------------------------------+
|
||||
//| Manage existing position |
|
||||
//+------------------------------------------------------------------+
|
||||
void ManagePosition(double predicted_price, double price_change_pct)
|
||||
{
|
||||
if(!PositionSelect(_Symbol))
|
||||
return;
|
||||
|
||||
long position_type = PositionGetInteger(POSITION_TYPE);
|
||||
double position_open_price = PositionGetDouble(POSITION_PRICE_OPEN);
|
||||
double current_profit = PositionGetDouble(POSITION_PROFIT);
|
||||
|
||||
// Simple management: close if prediction reverses
|
||||
if(position_type == POSITION_TYPE_BUY && price_change_pct < -InpPredictionThreshold)
|
||||
{
|
||||
// Prediction turned bearish, close long
|
||||
if(trade.PositionClose(_Symbol))
|
||||
{
|
||||
Print("Closed long position due to bearish prediction");
|
||||
}
|
||||
}
|
||||
else if(position_type == POSITION_TYPE_SELL && price_change_pct > InpPredictionThreshold)
|
||||
{
|
||||
// Prediction turned bullish, close short
|
||||
if(trade.PositionClose(_Symbol))
|
||||
{
|
||||
Print("Closed short position due to bullish prediction");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
# Quick Start Guide - ONNX with MetaTrader 5
|
||||
|
||||
Get started with ONNX machine learning models in MT5 in 5 minutes!
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **MetaTrader 5** installed and running
|
||||
2. **Python 3.8+** installed
|
||||
3. **MT5 account** (demo or live) with access to historical data
|
||||
|
||||
## Step 1: Install Dependencies
|
||||
|
||||
```bash
|
||||
cd ai
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
This installs:
|
||||
- TensorFlow/Keras for model training
|
||||
- ONNX runtime for model inference
|
||||
- MetaTrader5 Python module
|
||||
- Other required libraries
|
||||
|
||||
## Step 2: Verify MT5 Connection
|
||||
|
||||
Make sure MT5 is running and you're logged in. The scripts will automatically connect to MT5.
|
||||
|
||||
## Step 3: Train Your First Model
|
||||
|
||||
Train a price prediction model for Gold (XAUUSD):
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1 --epochs 30
|
||||
```
|
||||
|
||||
This will:
|
||||
- Download 2 years of historical data
|
||||
- Train an LSTM neural network
|
||||
- Save the model as `models/XAUUSD_H1_model.onnx`
|
||||
|
||||
**Expected time**: 5-15 minutes depending on your hardware.
|
||||
|
||||
## Step 4: Test the Model
|
||||
|
||||
Make a prediction with your trained model:
|
||||
|
||||
```bash
|
||||
python predict_with_onnx.py --model models/XAUUSD_H1_model.onnx --symbol XAUUSD
|
||||
```
|
||||
|
||||
You should see output like:
|
||||
```
|
||||
Current XAUUSD price: 2650.12345
|
||||
Making 1 prediction(s)...
|
||||
|
||||
Predicted next price: 2652.54321
|
||||
Expected change: 2.41976 (0.09%)
|
||||
```
|
||||
|
||||
## Step 5: Use in MetaTrader 5
|
||||
|
||||
### Option A: Copy Model to MT5
|
||||
|
||||
1. Copy your ONNX model to MT5's Files folder:
|
||||
```
|
||||
<MT5 Data Folder>\MQL5\Files\models\XAUUSD_H1_model.onnx
|
||||
```
|
||||
|
||||
Default locations:
|
||||
- Windows: `C:\Users\<YourName>\AppData\Roaming\MetaQuotes\Terminal\<TerminalID>\MQL5\Files\`
|
||||
- Or find it: MT5 → File → Open Data Folder → MQL5 → Files
|
||||
|
||||
2. Open `ONNX_EA.mq5` in MetaEditor
|
||||
|
||||
3. Compile (F7)
|
||||
|
||||
4. Attach to chart:
|
||||
- Model path: `models\XAUUSD_H1_model.onnx`
|
||||
- Lookback: `60` (must match training)
|
||||
- Set your trading parameters
|
||||
|
||||
### Option B: Run from MetaEditor
|
||||
|
||||
If you have Python integration enabled in MetaEditor:
|
||||
|
||||
1. Open `train_onnx_model.py` in MetaEditor
|
||||
2. Press F7 (Compile) to run
|
||||
3. The model will be saved to the project folder
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Train for Different Symbols
|
||||
|
||||
```bash
|
||||
# EUR/USD on 15-minute charts
|
||||
python train_onnx_model.py --symbol EURUSD --timeframe M15
|
||||
|
||||
# Bitcoin on 4-hour charts
|
||||
python train_onnx_model.py --symbol BTCUSD --timeframe H4
|
||||
```
|
||||
|
||||
### Custom Training Parameters
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py \
|
||||
--symbol XAUUSD \
|
||||
--timeframe H1 \
|
||||
--lookback 100 \
|
||||
--epochs 100 \
|
||||
--batch-size 64
|
||||
```
|
||||
|
||||
### Multiple Predictions
|
||||
|
||||
```bash
|
||||
python predict_with_onnx.py \
|
||||
--model models/XAUUSD_H1_model.onnx \
|
||||
--symbol XAUUSD \
|
||||
--predictions 5
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "MT5 initialization failed"
|
||||
- ✅ Make sure MT5 is running
|
||||
- ✅ Log into your account in MT5
|
||||
- ✅ Check that the symbol exists (e.g., XAUUSD, not GOLD)
|
||||
|
||||
### "No data available"
|
||||
- ✅ Ensure you have historical data downloaded in MT5
|
||||
- ✅ Check the date range (script uses last 2 years)
|
||||
- ✅ Verify symbol name is correct
|
||||
|
||||
### "Failed to load ONNX model" in EA
|
||||
- ✅ Check the file path is correct
|
||||
- ✅ Ensure model is in MT5's Files folder
|
||||
- ✅ Verify the model file exists and is not corrupted
|
||||
|
||||
### Model predictions seem wrong
|
||||
- ✅ Ensure `InpLookback` in EA matches training `--lookback`
|
||||
- ✅ Check that you're using the same symbol/timeframe
|
||||
- ✅ Verify feature normalization matches training
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Experiment with different models**:
|
||||
- Try different lookback periods
|
||||
- Adjust network architecture
|
||||
- Add more features
|
||||
|
||||
2. **Optimize trading parameters**:
|
||||
- Test different prediction thresholds
|
||||
- Tune stop loss/take profit
|
||||
- Adjust confidence levels
|
||||
|
||||
3. **Backtest thoroughly**:
|
||||
- Use MT5 Strategy Tester
|
||||
- Test on different time periods
|
||||
- Analyze performance metrics
|
||||
|
||||
4. **Monitor and improve**:
|
||||
- Track prediction accuracy
|
||||
- Retrain models periodically
|
||||
- Adjust based on market conditions
|
||||
|
||||
## Example Workflow
|
||||
|
||||
```bash
|
||||
# 1. Train model
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1 --epochs 50
|
||||
|
||||
# 2. Test predictions
|
||||
python predict_with_onnx.py --model models/XAUUSD_H1_model.onnx --symbol XAUUSD
|
||||
|
||||
# 3. Copy model to MT5 Files folder
|
||||
# (Manual step)
|
||||
|
||||
# 4. Compile and attach ONNX_EA.mq5 to chart
|
||||
|
||||
# 5. Monitor and adjust parameters
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- 🎯 Start with longer timeframes (H1, H4) for more stable predictions
|
||||
- 📊 Use multiple models for different market conditions
|
||||
- 🔄 Retrain models periodically (weekly/monthly)
|
||||
- ⚠️ Always test on demo account first
|
||||
- 📈 Monitor model performance and adjust parameters
|
||||
|
||||
## Need Help?
|
||||
|
||||
- Check the main [README.md](README.md) for detailed documentation
|
||||
- Review the MQL5 ONNX documentation: https://www.mql5.com/en/docs/onnx/onnx_prepare
|
||||
- Examine the code comments for implementation details
|
||||
|
||||
Happy trading! 🚀
|
||||
@@ -0,0 +1,310 @@
|
||||
# ONNX Models with MetaTrader 5
|
||||
|
||||
A complete framework for training and using ONNX machine learning models in MetaTrader 5 for algorithmic trading.
|
||||
|
||||
Based on the [MQL5 ONNX documentation](https://www.mql5.com/en/docs/onnx/onnx_prepare).
|
||||
|
||||
## Overview
|
||||
|
||||
This framework allows you to:
|
||||
1. **Train neural network models** in Python using MetaTrader 5 historical data
|
||||
2. **Export models to ONNX format** for use in MQL5
|
||||
3. **Use ONNX models in Expert Advisors** for real-time trading predictions
|
||||
4. **Test predictions** using Python scripts
|
||||
|
||||
## Features
|
||||
|
||||
- 🧠 **LSTM Neural Networks** for price prediction
|
||||
- 📊 **Technical Indicators** as features (RSI, EMA, ATR, etc.)
|
||||
- 🔄 **ONNX Export** for MQL5 integration
|
||||
- 📈 **Real-time Prediction** in Expert Advisors
|
||||
- 🎯 **Flexible Configuration** for different symbols and timeframes
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Install Python Dependencies
|
||||
|
||||
```bash
|
||||
cd ai
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Install MetaTrader 5
|
||||
|
||||
- Download and install [MetaTrader 5](https://www.metatrader5.com/en/download)
|
||||
- Create a demo or live account
|
||||
- Enable Python integration in MT5 settings:
|
||||
- Tools → Options → Expert Advisors
|
||||
- Check "Allow DLL imports"
|
||||
- Check "Integration with Python" (if available)
|
||||
|
||||
### 3. Configure MetaEditor (Optional)
|
||||
|
||||
If you want to run Python scripts from MetaEditor:
|
||||
- MetaEditor → Tools → Options → Compiler
|
||||
- Set Python executable path
|
||||
- Or click "Install" to download Python
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Train an ONNX Model
|
||||
|
||||
Train a model for price prediction:
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1 --lookback 60 --epochs 50
|
||||
```
|
||||
|
||||
This will:
|
||||
- Fetch 2 years of historical data from MT5
|
||||
- Prepare features (OHLCV + technical indicators)
|
||||
- Train an LSTM neural network
|
||||
- Export the model to `models/XAUUSD_H1_model.onnx`
|
||||
|
||||
### Step 2: Test the Model
|
||||
|
||||
Make predictions using the trained model:
|
||||
|
||||
```bash
|
||||
python predict_with_onnx.py --model models/XAUUSD_H1_model.onnx --symbol XAUUSD --timeframe H1
|
||||
```
|
||||
|
||||
### Step 3: Use in Expert Advisor
|
||||
|
||||
1. Copy the ONNX model to MT5's Files folder:
|
||||
```
|
||||
<MT5 Data Folder>\MQL5\Files\models\XAUUSD_H1_model.onnx
|
||||
```
|
||||
|
||||
2. Compile `ONNX_EA.mq5` in MetaEditor
|
||||
|
||||
3. Attach the EA to a chart with:
|
||||
- Model path: `models\XAUUSD_H1_model.onnx`
|
||||
- Your trading parameters
|
||||
|
||||
## Detailed Usage
|
||||
|
||||
### Training Models
|
||||
|
||||
#### Basic Training
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py \
|
||||
--symbol XAUUSD \
|
||||
--timeframe H1 \
|
||||
--lookback 60 \
|
||||
--epochs 50 \
|
||||
--batch-size 32
|
||||
```
|
||||
|
||||
#### Advanced Options
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py \
|
||||
--symbol EURUSD \
|
||||
--timeframe M15 \
|
||||
--lookback 100 \
|
||||
--epochs 100 \
|
||||
--batch-size 64 \
|
||||
--output custom_models
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--symbol`: Trading symbol (XAUUSD, EURUSD, BTCUSD, etc.)
|
||||
- `--timeframe`: M1, M5, M15, M30, H1, H4, D1
|
||||
- `--lookback`: Number of bars to use for prediction (default: 60)
|
||||
- `--epochs`: Training epochs (default: 50)
|
||||
- `--batch-size`: Batch size (default: 32)
|
||||
- `--output`: Output directory (default: models)
|
||||
|
||||
### Making Predictions
|
||||
|
||||
#### Single Prediction
|
||||
|
||||
```bash
|
||||
python predict_with_onnx.py \
|
||||
--model models/XAUUSD_H1_model.onnx \
|
||||
--symbol XAUUSD \
|
||||
--timeframe H1
|
||||
```
|
||||
|
||||
#### Multiple Predictions
|
||||
|
||||
```bash
|
||||
python predict_with_onnx.py \
|
||||
--model models/XAUUSD_H1_model.onnx \
|
||||
--symbol XAUUSD \
|
||||
--timeframe H1 \
|
||||
--predictions 5
|
||||
```
|
||||
|
||||
### Expert Advisor Configuration
|
||||
|
||||
The `ONNX_EA.mq5` Expert Advisor includes:
|
||||
|
||||
**ONNX Model Settings:**
|
||||
- `InpModelPath`: Path to ONNX model file
|
||||
- `InpLookback`: Lookback period (must match training)
|
||||
- `InpUsePrediction`: Enable/disable model predictions
|
||||
|
||||
**Trading Settings:**
|
||||
- `InpLotSize`: Position size
|
||||
- `InpMagicNumber`: Magic number for trades
|
||||
- `InpStopLoss`: Stop loss in pips
|
||||
- `InpTakeProfit`: Take profit in pips
|
||||
|
||||
**Prediction Settings:**
|
||||
- `InpPredictionThreshold`: Minimum prediction change to trade (0.01% = 0.0001)
|
||||
- `InpUseConfidence`: Enable confidence filtering
|
||||
- `InpMinConfidence`: Minimum confidence level (0.0-1.0)
|
||||
|
||||
## Model Architecture
|
||||
|
||||
The default model uses:
|
||||
- **Input**: 60 bars × 12 features
|
||||
- **Architecture**:
|
||||
- LSTM(128) → Dropout(0.2)
|
||||
- LSTM(64) → Dropout(0.2)
|
||||
- LSTM(32) → Dropout(0.2)
|
||||
- Dense(16, ReLU)
|
||||
- Dense(1) - Price prediction
|
||||
- **Features**:
|
||||
- OHLC prices
|
||||
- Tick volume
|
||||
- RSI (14)
|
||||
- EMA(20), EMA(50)
|
||||
- ATR(14)
|
||||
- Price changes
|
||||
- High/Low ratio
|
||||
- Volume ratios
|
||||
|
||||
## Customization
|
||||
|
||||
### Modify Features
|
||||
|
||||
Edit `train_onnx_model.py` to add/remove features:
|
||||
|
||||
```python
|
||||
def prepare_features(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
feature_df = df[['open', 'high', 'low', 'close', 'tick_volume']].copy()
|
||||
|
||||
# Add your custom indicators
|
||||
feature_df['custom_indicator'] = your_calculation(df)
|
||||
|
||||
return feature_df
|
||||
```
|
||||
|
||||
### Change Model Architecture
|
||||
|
||||
Modify `build_model()` in `train_onnx_model.py`:
|
||||
|
||||
```python
|
||||
def build_model(self, input_shape: tuple) -> keras.Model:
|
||||
model = keras.Sequential([
|
||||
layers.LSTM(256, return_sequences=True, input_shape=input_shape),
|
||||
# Add your layers here
|
||||
layers.Dense(1)
|
||||
])
|
||||
return model
|
||||
```
|
||||
|
||||
### Adjust Expert Advisor Logic
|
||||
|
||||
Edit `ONNX_EA.mq5` to customize trading logic:
|
||||
- Entry conditions
|
||||
- Exit conditions
|
||||
- Position management
|
||||
- Risk management
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
ai/
|
||||
├── requirements.txt # Python dependencies
|
||||
├── train_onnx_model.py # Model training script
|
||||
├── predict_with_onnx.py # Prediction testing script
|
||||
├── ONNX_EA.mq5 # MQL5 Expert Advisor
|
||||
├── README.md # This file
|
||||
└── models/ # Trained ONNX models (created after training)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### MT5 Connection Issues
|
||||
|
||||
**Error**: "MT5 initialization failed"
|
||||
- Ensure MetaTrader 5 is installed and running
|
||||
- Log into a demo or live account
|
||||
- Check that the symbol exists in MT5
|
||||
|
||||
### Model Loading Issues
|
||||
|
||||
**Error**: "Failed to load ONNX model"
|
||||
- Verify the model file path is correct
|
||||
- Ensure the model file is in MT5's Files folder
|
||||
- Check that the model was exported correctly
|
||||
|
||||
### Prediction Issues
|
||||
|
||||
**Error**: "Failed to prepare input data"
|
||||
- Ensure enough historical data is available
|
||||
- Check that lookback period matches training
|
||||
- Verify indicators can be calculated
|
||||
|
||||
### Shape Mismatch Errors
|
||||
|
||||
If you get shape mismatch errors:
|
||||
1. Check that `InpLookback` in EA matches training `--lookback`
|
||||
2. Verify feature count matches (default: 12 features)
|
||||
3. Ensure input normalization matches training
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Data Quality**: Use high-quality historical data
|
||||
2. **Feature Engineering**: Experiment with different indicators
|
||||
3. **Model Validation**: Always validate on out-of-sample data
|
||||
4. **Risk Management**: Use stop loss and position sizing
|
||||
5. **Backtesting**: Test thoroughly before live trading
|
||||
6. **Monitoring**: Monitor model performance regularly
|
||||
|
||||
## Example Workflow
|
||||
|
||||
1. **Train Model**:
|
||||
```bash
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1
|
||||
```
|
||||
|
||||
2. **Test Predictions**:
|
||||
```bash
|
||||
python predict_with_onnx.py --model models/XAUUSD_H1_model.onnx --symbol XAUUSD
|
||||
```
|
||||
|
||||
3. **Backtest in MT5**:
|
||||
- Use Strategy Tester with `ONNX_EA.mq5`
|
||||
- Test on historical data
|
||||
- Analyze results
|
||||
|
||||
4. **Optimize Parameters**:
|
||||
- Adjust prediction threshold
|
||||
- Tune confidence levels
|
||||
- Optimize stop loss/take profit
|
||||
|
||||
5. **Deploy**:
|
||||
- Start with small position sizes
|
||||
- Monitor performance
|
||||
- Adjust as needed
|
||||
|
||||
## References
|
||||
|
||||
- [MQL5 ONNX Documentation](https://www.mql5.com/en/docs/onnx/onnx_prepare)
|
||||
- [ONNX Model Zoo](https://github.com/onnx/models)
|
||||
- [MetaTrader 5 Python Module](https://pypi.org/project/MetaTrader5/)
|
||||
- [TensorFlow to ONNX](https://github.com/onnx/tensorflow-onnx)
|
||||
|
||||
## Disclaimer
|
||||
|
||||
Trading involves substantial risk of loss. This framework is provided for educational purposes only. Always test thoroughly on a demo account before using with real money. Past performance does not guarantee future results.
|
||||
|
||||
## License
|
||||
|
||||
This framework is provided for educational and research purposes.
|
||||
@@ -0,0 +1,94 @@
|
||||
# XAUUSD ONNX Model Training and Backtesting Summary
|
||||
|
||||
## Status: ✅ Model Trained, ⚠️ Predictions Need Investigation
|
||||
|
||||
### Completed
|
||||
1. ✅ **Model Training**: Successfully trained XAUUSD H1 ONNX model
|
||||
- Model: `models/XAUUSD_H1_model.onnx`
|
||||
- Scaler: `models/XAUUSD_H1_scaler.pkl`
|
||||
- Training Loss: 177939.59, MAE: 349.43
|
||||
- Validation Loss: 1749893.25, MAE: 1280.39
|
||||
|
||||
2. ✅ **Backtest Framework**: Working correctly
|
||||
- Processes 2888 bars successfully
|
||||
- No errors in execution
|
||||
|
||||
3. ✅ **Parameter Optimization Tools**: Created
|
||||
- Grid search and random search support
|
||||
- Can test multiple parameter combinations
|
||||
|
||||
### Issue Identified
|
||||
⚠️ **Model Predictions Are Unrealistic**
|
||||
- Model predicts prices around **2669** when current price is **3800+**
|
||||
- This suggests a **-31% price change**, which is unrealistic
|
||||
- All parameter combinations result in **0 trades**
|
||||
|
||||
### Possible Causes
|
||||
1. **Model Training Issue**:
|
||||
- High validation MAE (1280) suggests model may not be learning well
|
||||
- Model might be predicting from wrong data range
|
||||
|
||||
2. **Feature Mismatch**:
|
||||
- Features used in backtesting might not match training features exactly
|
||||
- Normalization might be inconsistent
|
||||
|
||||
3. **Model Architecture**:
|
||||
- LSTM might need more training or different architecture
|
||||
- Current model might be underfitting
|
||||
|
||||
### Recommendations
|
||||
|
||||
#### Immediate Actions
|
||||
1. **Check Model Predictions**:
|
||||
```bash
|
||||
python inspect_predictions.py
|
||||
```
|
||||
This shows actual prediction values and statistics
|
||||
|
||||
2. **Retrain with Better Settings**:
|
||||
- Increase training epochs (try 50-100)
|
||||
- Use more recent data
|
||||
- Consider predicting price changes instead of absolute prices
|
||||
- Add more regularization to prevent overfitting
|
||||
|
||||
3. **Alternative Approach**:
|
||||
- Train model to predict **price change percentage** instead of absolute price
|
||||
- This would be more stable and easier to interpret
|
||||
|
||||
#### Next Steps
|
||||
1. Investigate why predictions are so far off
|
||||
2. Consider retraining with:
|
||||
- Price change prediction instead of absolute price
|
||||
- Better feature engineering
|
||||
- More training data
|
||||
- Different model architecture
|
||||
|
||||
### Files Created
|
||||
- `ai/train_onnx_model.py` - Model training script
|
||||
- `ai/quick_backtest.py` - Quick backtest script
|
||||
- `ai/optimize_onnx_params.py` - Parameter optimization
|
||||
- `ai/debug_onnx_predictions.py` - Debug predictions
|
||||
- `ai/inspect_predictions.py` - Detailed prediction inspection
|
||||
- `ai/test_very_low_threshold.py` - Test with very low thresholds
|
||||
- `backtesting/MT5/onnx_backtest_strategy.py` - ONNX strategy class
|
||||
- `backtesting/MT5/indicator_utils.py` - Indicator calculation utilities
|
||||
|
||||
### Usage
|
||||
```bash
|
||||
# Quick backtest
|
||||
cd ai
|
||||
python quick_backtest.py
|
||||
|
||||
# Optimize parameters
|
||||
python optimize_onnx_params.py 2 30
|
||||
|
||||
# Inspect predictions
|
||||
python inspect_predictions.py
|
||||
```
|
||||
|
||||
### Model Details
|
||||
- **Symbol**: XAUUSD
|
||||
- **Timeframe**: H1
|
||||
- **Lookback**: 60 bars
|
||||
- **Features**: 13 (OHLC + volume + RSI + EMA20 + EMA50 + ATR + price_change + high_low_ratio + volume_ma + volume_ratio)
|
||||
- **Architecture**: LSTM(128) → LSTM(64) → LSTM(32) → Dense(16) → Dense(1)
|
||||
@@ -0,0 +1,112 @@
|
||||
# XAUUSD ONNX Model Training and Backtesting Guide
|
||||
|
||||
This guide will help you train an ONNX model for XAUUSD and backtest it.
|
||||
|
||||
## Step 1: Install Dependencies
|
||||
|
||||
Make sure you have all required packages installed:
|
||||
|
||||
```bash
|
||||
cd ai
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
If you encounter issues, install individually:
|
||||
```bash
|
||||
pip install tensorflow onnx onnxruntime tf2onnx scikit-learn pandas numpy MetaTrader5
|
||||
```
|
||||
|
||||
## Step 2: Train the Model
|
||||
|
||||
Train an ONNX model for XAUUSD:
|
||||
|
||||
```bash
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1 --lookback 60 --epochs 30
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--symbol XAUUSD`: Trading symbol (Gold)
|
||||
- `--timeframe H1`: 1-hour timeframe
|
||||
- `--lookback 60`: Use 60 bars for prediction
|
||||
- `--epochs 30`: Training epochs (adjust based on your needs)
|
||||
|
||||
**Expected Output:**
|
||||
- Model: `models/XAUUSD_H1_model.onnx`
|
||||
- Scaler: `models/XAUUSD_H1_scaler.pkl`
|
||||
|
||||
**Training Time:** 5-15 minutes depending on your hardware and data availability.
|
||||
|
||||
## Step 3: Run Backtest
|
||||
|
||||
After training, backtest the model:
|
||||
|
||||
```bash
|
||||
python run_xauusd_backtest.py
|
||||
```
|
||||
|
||||
Or use the combined script:
|
||||
|
||||
```bash
|
||||
python train_and_backtest_xauusd.py
|
||||
```
|
||||
|
||||
## Step 4: Review Results
|
||||
|
||||
The backtest will generate:
|
||||
- Performance summary in console
|
||||
- Equity curve chart
|
||||
- Drawdown chart
|
||||
- Monthly returns chart
|
||||
- Trades CSV file
|
||||
|
||||
All files are saved in `onnx_xauusd_backtest/` directory.
|
||||
|
||||
## Model Configuration
|
||||
|
||||
The trained model uses:
|
||||
- **Input**: 60 bars × 12 features
|
||||
- **Features**: OHLC, volume, RSI, EMA, ATR, price changes, ratios
|
||||
- **Output**: Predicted next close price
|
||||
- **Architecture**: LSTM(128) → LSTM(64) → LSTM(32) → Dense layers
|
||||
|
||||
## Backtest Strategy Parameters
|
||||
|
||||
Default backtest parameters:
|
||||
- **Prediction Threshold**: 0.01% (minimum price change to trade)
|
||||
- **Min Confidence**: 30%
|
||||
- **Lot Size**: 0.1
|
||||
- **Stop Loss**: 50 pips
|
||||
- **Take Profit**: 100 pips
|
||||
|
||||
You can adjust these in `run_xauusd_backtest.py`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Model not found"
|
||||
- Make sure you've trained the model first
|
||||
- Check that the model file exists in `models/` directory
|
||||
|
||||
### "MT5 initialization failed"
|
||||
- Ensure MetaTrader 5 is running
|
||||
- Log into your account
|
||||
- Check that XAUUSD symbol is available
|
||||
|
||||
### "Insufficient data"
|
||||
- Make sure you have historical data downloaded in MT5
|
||||
- Check the date range in the backtest script
|
||||
- Verify symbol name is correct
|
||||
|
||||
## Next Steps
|
||||
|
||||
After successful backtesting:
|
||||
1. Review performance metrics
|
||||
2. Optimize prediction threshold and confidence levels
|
||||
3. Adjust stop loss/take profit if needed
|
||||
4. Test on demo account before live trading
|
||||
5. Consider using the model in `ONNX_EA.mq5` for live trading
|
||||
|
||||
## Files Created
|
||||
|
||||
- `models/XAUUSD_H1_model.onnx`: Trained ONNX model
|
||||
- `models/XAUUSD_H1_scaler.pkl`: Feature scaler for normalization
|
||||
- `onnx_xauusd_backtest/`: Backtest results directory
|
||||
@@ -0,0 +1,176 @@
|
||||
"""
|
||||
Debug ONNX Model Predictions
|
||||
|
||||
This script helps debug why the model isn't generating trades.
|
||||
It shows actual predictions and checks if they meet trading criteria.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import onnxruntime as ort
|
||||
import pickle
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
|
||||
|
||||
def main():
|
||||
"""Debug ONNX predictions."""
|
||||
print("="*60)
|
||||
print("ONNX Model Prediction Debug")
|
||||
print("="*60)
|
||||
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
|
||||
if not os.path.exists(model_path):
|
||||
print(f"ERROR: Model not found: {model_path}")
|
||||
return
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
return
|
||||
|
||||
try:
|
||||
# Load model and scaler
|
||||
session = ort.InferenceSession(model_path)
|
||||
input_name = session.get_inputs()[0].name
|
||||
output_name = session.get_outputs()[0].name
|
||||
|
||||
with open(scaler_path, 'rb') as f:
|
||||
scaler = pickle.load(f)
|
||||
|
||||
# Get recent data
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=10)
|
||||
|
||||
rates = mt5.copy_rates_range(symbol, timeframe, start_date, end_date)
|
||||
if rates is None or len(rates) == 0:
|
||||
print("ERROR: No data available")
|
||||
return
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
|
||||
print(f"\nLoaded {len(df)} bars")
|
||||
print(f"Date range: {df['time'].min()} to {df['time'].max()}")
|
||||
|
||||
# Create strategy to get features
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=10000.0,
|
||||
prediction_threshold=0.0001,
|
||||
min_confidence=0.3,
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
|
||||
# Simulate a few bars
|
||||
print("\n" + "="*60)
|
||||
print("Predictions Analysis")
|
||||
print("="*60)
|
||||
|
||||
predictions_data = []
|
||||
|
||||
for i in range(60, min(100, len(df))): # Start from bar 60 (need lookback)
|
||||
bar = df.iloc[i]
|
||||
bar_time = bar['time'] if isinstance(bar['time'], datetime) else datetime.fromtimestamp(bar['time'])
|
||||
current_price = bar['close']
|
||||
|
||||
# Build historical buffer
|
||||
bar_data = {
|
||||
'time': bar_time,
|
||||
'open': float(bar['open']),
|
||||
'high': float(bar['high']),
|
||||
'low': float(bar['low']),
|
||||
'close': float(bar['close']),
|
||||
'tick_volume': int(bar['tick_volume']),
|
||||
'rsi': 50.0, # Simplified
|
||||
'ema': current_price, # Simplified
|
||||
'atr': 0.0 # Simplified
|
||||
}
|
||||
|
||||
strategy.historical_bars.append(bar_data)
|
||||
|
||||
if len(strategy.historical_bars) >= strategy.lookback:
|
||||
# Get prediction
|
||||
features = strategy.prepare_features()
|
||||
if features is not None:
|
||||
input_data = features.astype(np.float32)
|
||||
outputs = session.run([output_name], {input_name: input_data})
|
||||
predicted_price = float(outputs[0][0][0])
|
||||
|
||||
# Calculate metrics
|
||||
price_change = predicted_price - current_price
|
||||
price_change_pct = (price_change / current_price) if current_price > 0 else 0.0
|
||||
confidence = min(abs(price_change_pct) / 0.01, 1.0)
|
||||
|
||||
predictions_data.append({
|
||||
'time': bar_time,
|
||||
'current_price': current_price,
|
||||
'predicted_price': predicted_price,
|
||||
'price_change': price_change,
|
||||
'price_change_pct': price_change_pct * 100,
|
||||
'confidence': confidence,
|
||||
'meets_threshold': abs(price_change_pct) >= 0.0001,
|
||||
'meets_confidence': confidence >= 0.3,
|
||||
'would_trade': abs(price_change_pct) >= 0.0001 and confidence >= 0.3
|
||||
})
|
||||
|
||||
# Display results
|
||||
if predictions_data:
|
||||
pred_df = pd.DataFrame(predictions_data)
|
||||
print(f"\nAnalyzed {len(pred_df)} predictions")
|
||||
print(f"\nPrediction Statistics:")
|
||||
print(f" Mean price change: {pred_df['price_change_pct'].mean():.4f}%")
|
||||
print(f" Std price change: {pred_df['price_change_pct'].std():.4f}%")
|
||||
print(f" Min price change: {pred_df['price_change_pct'].min():.4f}%")
|
||||
print(f" Max price change: {pred_df['price_change_pct'].max():.4f}%")
|
||||
print(f"\n Mean confidence: {pred_df['confidence'].mean():.4f}")
|
||||
print(f" Predictions meeting threshold: {pred_df['meets_threshold'].sum()}/{len(pred_df)}")
|
||||
print(f" Predictions meeting confidence: {pred_df['meets_confidence'].sum()}/{len(pred_df)}")
|
||||
print(f" Predictions that would trade: {pred_df['would_trade'].sum()}/{len(pred_df)}")
|
||||
|
||||
print(f"\nSample predictions (first 10):")
|
||||
print(pred_df[['time', 'current_price', 'predicted_price', 'price_change_pct', 'confidence', 'would_trade']].head(10).to_string(index=False))
|
||||
|
||||
if pred_df['would_trade'].sum() == 0:
|
||||
print("\n" + "="*60)
|
||||
print("RECOMMENDATIONS:")
|
||||
print("="*60)
|
||||
print("No trades would be generated. Try:")
|
||||
print(f" 1. Lower prediction_threshold (current: 0.0001)")
|
||||
print(f" Suggested: {pred_df['price_change_pct'].abs().quantile(0.1):.6f}")
|
||||
print(f" 2. Lower min_confidence (current: 0.3)")
|
||||
print(f" Suggested: {pred_df['confidence'].quantile(0.1):.2f}")
|
||||
print(f" 3. Check if model predictions are reasonable")
|
||||
else:
|
||||
print("No predictions generated (need more historical data)")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,219 @@
|
||||
"""
|
||||
Debug ONNX Strategy - Find out why no trades are generated
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
|
||||
|
||||
def main():
|
||||
"""Debug strategy to find why no trades."""
|
||||
print("="*60)
|
||||
print("Debugging ONNX Strategy - Why No Trades?")
|
||||
print("="*60)
|
||||
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
initial_balance = 10000.0
|
||||
|
||||
if not os.path.exists(model_path):
|
||||
print(f"ERROR: Model not found: {model_path}")
|
||||
return
|
||||
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=30) # Shorter period for debugging
|
||||
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Date Range: {start_date.date()} to {end_date.date()}")
|
||||
print(f"Parameters:")
|
||||
print(f" Prediction Threshold: 0.00005 (0.005%)")
|
||||
print(f" Min Confidence: 0.1 (10%)")
|
||||
print("\n")
|
||||
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
return
|
||||
|
||||
try:
|
||||
# Create strategy with debug enabled
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=initial_balance,
|
||||
prediction_threshold=0.00005,
|
||||
min_confidence=0.1,
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
|
||||
# Override on_bar to add detailed debugging
|
||||
original_on_bar = strategy.on_bar
|
||||
|
||||
def debug_on_bar(bar_data):
|
||||
"""Debug version of on_bar."""
|
||||
# Add current bar to historical buffer
|
||||
strategy.historical_bars.append(bar_data.copy())
|
||||
|
||||
# Keep only necessary history
|
||||
if len(strategy.historical_bars) > strategy.lookback + 50:
|
||||
strategy.historical_bars = strategy.historical_bars[-(strategy.lookback + 50):]
|
||||
|
||||
# Check if we have enough data
|
||||
if len(strategy.historical_bars) < strategy.lookback:
|
||||
if len(strategy.historical_bars) % 20 == 0:
|
||||
print(f" [Bar {len(strategy.historical_bars)}] Not enough data yet (need {strategy.lookback})")
|
||||
return
|
||||
|
||||
current_price = bar_data['close']
|
||||
|
||||
# Check existing position
|
||||
if strategy.position is not None:
|
||||
strategy.check_stop_loss_take_profit(current_price)
|
||||
return
|
||||
|
||||
# Make prediction
|
||||
try:
|
||||
predicted_change_pct = strategy.predict_price()
|
||||
if predicted_change_pct is None:
|
||||
if len(strategy.historical_bars) % 10 == 0:
|
||||
print(f" [Bar {len(strategy.historical_bars)}] Prediction returned None - checking why...")
|
||||
# Try to debug why prediction is None
|
||||
features = strategy.prepare_features()
|
||||
if features is None:
|
||||
print(f" -> Features preparation returned None")
|
||||
else:
|
||||
print(f" -> Features shape: {features.shape}")
|
||||
return
|
||||
except Exception as e:
|
||||
print(f" [Bar {len(strategy.historical_bars)}] Prediction exception: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return
|
||||
|
||||
# Process prediction
|
||||
if abs(predicted_change_pct) < 1.0:
|
||||
price_change_pct = predicted_change_pct
|
||||
else:
|
||||
predicted_price = predicted_change_pct
|
||||
if predicted_price <= 0 or predicted_price > 10000:
|
||||
if len(strategy.historical_bars) % 50 == 0:
|
||||
print(f" [Bar {len(strategy.historical_bars)}] Invalid prediction: {predicted_price}")
|
||||
return
|
||||
price_change = predicted_price - current_price
|
||||
price_change_pct = (price_change / current_price) if current_price > 0 else 0.0
|
||||
|
||||
# Calculate confidence
|
||||
if abs(price_change_pct) < 1.0:
|
||||
confidence = min(abs(price_change_pct) / 0.01, 1.0)
|
||||
else:
|
||||
confidence = min(abs(price_change_pct) / 1.0, 1.0)
|
||||
|
||||
# Debug output for every 10th bar
|
||||
if len(strategy.historical_bars) % 10 == 0:
|
||||
print(f"\n [Bar {len(strategy.historical_bars)}]")
|
||||
print(f" Current Price: {current_price:.2f}")
|
||||
print(f" Raw Prediction: {predicted_change_pct:.6f}")
|
||||
print(f" Price Change %: {price_change_pct*100:.4f}%")
|
||||
print(f" Abs Change: {abs(price_change_pct):.6f}")
|
||||
print(f" Threshold: {strategy.prediction_threshold:.6f}")
|
||||
print(f" Confidence: {confidence:.3f}")
|
||||
print(f" Min Confidence: {strategy.min_confidence:.2f}")
|
||||
print(f" Threshold Check: {abs(price_change_pct) >= strategy.prediction_threshold} (need True)")
|
||||
print(f" Confidence Check: {confidence >= strategy.min_confidence} (need True)")
|
||||
|
||||
if abs(price_change_pct) >= strategy.prediction_threshold and confidence >= strategy.min_confidence:
|
||||
print(f" -> WOULD TRADE! Direction: {'BUY' if price_change_pct > 0 else 'SELL'}")
|
||||
else:
|
||||
if abs(price_change_pct) < strategy.prediction_threshold:
|
||||
print(f" -> BLOCKED: Abs change {abs(price_change_pct):.6f} < threshold {strategy.prediction_threshold:.6f}")
|
||||
if confidence < strategy.min_confidence:
|
||||
print(f" -> BLOCKED: Confidence {confidence:.3f} < min {strategy.min_confidence:.2f}")
|
||||
|
||||
# Check if we should trade
|
||||
if confidence < strategy.min_confidence:
|
||||
return
|
||||
|
||||
if abs(price_change_pct) < strategy.prediction_threshold:
|
||||
return
|
||||
|
||||
# Open position based on prediction
|
||||
if price_change_pct > strategy.prediction_threshold:
|
||||
# Bullish prediction
|
||||
sl = current_price - (strategy.stop_loss_pips / 10000) if strategy.stop_loss_pips > 0 else None
|
||||
tp = current_price + (strategy.take_profit_pips / 10000) if strategy.take_profit_pips > 0 else None
|
||||
print(f"\n *** ATTEMPTING BUY POSITION at bar {len(strategy.historical_bars)} ***")
|
||||
print(f" Price: {current_price:.2f}, Predicted Change: {price_change_pct*100:.4f}%")
|
||||
print(f" SL: {sl:.2f}, TP: {tp:.2f}, Volume: {strategy.lot_size}")
|
||||
print(f" Equity: {strategy.equity:.2f}, Current Position: {strategy.position}")
|
||||
# Check margin requirement manually
|
||||
contract_size = 100000
|
||||
margin_required = strategy.lot_size * contract_size * current_price * 0.01
|
||||
print(f" Margin Required: {margin_required:.2f}, Available: {strategy.equity * 0.9:.2f}")
|
||||
result = strategy.open_position('BUY', strategy.lot_size, current_price, sl, tp, 'ONNX Buy')
|
||||
print(f" Open Position Result: {result}")
|
||||
if result:
|
||||
print(f" -> Position opened! New position: {strategy.position}")
|
||||
else:
|
||||
if strategy.position is not None:
|
||||
print(f" -> Position NOT opened! Reason: Already have position")
|
||||
else:
|
||||
print(f" -> Position NOT opened! Reason: Margin insufficient or other validation failed")
|
||||
elif price_change_pct < -strategy.prediction_threshold:
|
||||
# Bearish prediction
|
||||
sl = current_price + (strategy.stop_loss_pips / 10000) if strategy.stop_loss_pips > 0 else None
|
||||
tp = current_price - (strategy.take_profit_pips / 10000) if strategy.take_profit_pips > 0 else None
|
||||
print(f"\n *** ATTEMPTING SELL POSITION at bar {len(strategy.historical_bars)} ***")
|
||||
print(f" Price: {current_price:.2f}, Predicted Change: {price_change_pct*100:.4f}%")
|
||||
print(f" SL: {sl:.2f}, TP: {tp:.2f}, Volume: {strategy.lot_size}")
|
||||
print(f" Equity: {strategy.equity:.2f}, Current Position: {strategy.position}")
|
||||
# Check margin requirement manually
|
||||
contract_size = 100000
|
||||
margin_required = strategy.lot_size * contract_size * current_price * 0.01
|
||||
print(f" Margin Required: {margin_required:.2f}, Available: {strategy.equity * 0.9:.2f}")
|
||||
result = strategy.open_position('SELL', strategy.lot_size, current_price, sl, tp, 'ONNX Sell')
|
||||
print(f" Open Position Result: {result}")
|
||||
if result:
|
||||
print(f" -> Position opened! New position: {strategy.position}")
|
||||
else:
|
||||
if strategy.position is not None:
|
||||
print(f" -> Position NOT opened! Reason: Already have position")
|
||||
else:
|
||||
print(f" -> Position NOT opened! Reason: Margin insufficient or other validation failed")
|
||||
|
||||
strategy.on_bar = debug_on_bar
|
||||
|
||||
print("Running backtest with detailed debugging...\n")
|
||||
engine = BacktestEngine(strategy, start_date, end_date)
|
||||
results = engine.run()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Backtest Complete")
|
||||
print("="*60)
|
||||
print(f"Total Trades: {len(strategy.closed_trades)}")
|
||||
print(f"Open Positions: {1 if strategy.position else 0}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,159 @@
|
||||
"""
|
||||
Complete Example Workflow for ONNX + MT5
|
||||
|
||||
This script demonstrates the complete workflow:
|
||||
1. Train an ONNX model
|
||||
2. Test predictions
|
||||
3. Show how to use in MT5
|
||||
|
||||
Run this to see the full process in action.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Import our modules
|
||||
from train_onnx_model import ONNXModelTrainer
|
||||
from predict_with_onnx import ONNXPredictor
|
||||
|
||||
|
||||
def main():
|
||||
"""Complete workflow example."""
|
||||
print("="*60)
|
||||
print("ONNX + MetaTrader 5 - Complete Workflow Example")
|
||||
print("="*60)
|
||||
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe_str = 'H1'
|
||||
lookback = 60
|
||||
epochs = 20 # Reduced for quick demo
|
||||
|
||||
# Convert timeframe
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[timeframe_str]
|
||||
|
||||
# Create output directory
|
||||
models_dir = 'models'
|
||||
os.makedirs(models_dir, exist_ok=True)
|
||||
|
||||
# Step 1: Train Model
|
||||
print("\n" + "="*60)
|
||||
print("STEP 1: Training ONNX Model")
|
||||
print("="*60)
|
||||
|
||||
trainer = ONNXModelTrainer(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
lookback=lookback
|
||||
)
|
||||
|
||||
try:
|
||||
print(f"\nTraining model for {symbol} on {timeframe_str} timeframe...")
|
||||
print(f"Lookback: {lookback} bars")
|
||||
print(f"Epochs: {epochs}")
|
||||
print("\nThis may take several minutes...\n")
|
||||
|
||||
trainer.train(epochs=epochs, batch_size=32, verbose=1)
|
||||
|
||||
# Export model
|
||||
model_name = f"{symbol}_{timeframe_str}_model.onnx"
|
||||
model_path = os.path.join(models_dir, model_name)
|
||||
|
||||
print(f"\nExporting model to ONNX format...")
|
||||
trainer.export_to_onnx(model_path)
|
||||
|
||||
print(f"\n✓ Model saved to: {model_path}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Training failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
trainer.cleanup()
|
||||
return
|
||||
|
||||
finally:
|
||||
trainer.cleanup()
|
||||
|
||||
# Step 2: Test Predictions
|
||||
print("\n" + "="*60)
|
||||
print("STEP 2: Testing Predictions")
|
||||
print("="*60)
|
||||
|
||||
predictor = ONNXPredictor(model_path)
|
||||
|
||||
try:
|
||||
# Get current price
|
||||
symbol_info = mt5.symbol_info(symbol)
|
||||
if symbol_info:
|
||||
current_price = symbol_info.bid
|
||||
print(f"\nCurrent {symbol} price: {current_price:.5f}")
|
||||
else:
|
||||
print(f"\nWarning: Could not get current price for {symbol}")
|
||||
current_price = 0
|
||||
|
||||
# Make predictions
|
||||
print(f"\nMaking predictions...")
|
||||
predictions = predictor.predict_batch(symbol, timeframe, n_predictions=3)
|
||||
|
||||
print("\nPredictions:")
|
||||
for i, pred in enumerate(predictions, 1):
|
||||
if current_price > 0:
|
||||
change = pred - current_price
|
||||
change_pct = (change / current_price) * 100
|
||||
print(f" {i}. {pred:.5f} (change: {change:+.5f}, {change_pct:+.2f}%)")
|
||||
else:
|
||||
print(f" {i}. {pred:.5f}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Prediction failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
predictor.cleanup()
|
||||
|
||||
# Step 3: Instructions for MT5
|
||||
print("\n" + "="*60)
|
||||
print("STEP 3: Using in MetaTrader 5")
|
||||
print("="*60)
|
||||
|
||||
print(f"\nTo use this model in MetaTrader 5:")
|
||||
print(f"\n1. Copy the model file to MT5's Files folder:")
|
||||
print(f" {model_path}")
|
||||
print(f" → <MT5 Data Folder>\\MQL5\\Files\\models\\{os.path.basename(model_path)}")
|
||||
print(f"\n2. Open ONNX_EA.mq5 in MetaEditor")
|
||||
print(f"\n3. Set EA parameters:")
|
||||
print(f" - Model Path: models\\{os.path.basename(model_path)}")
|
||||
print(f" - Lookback: {lookback}")
|
||||
print(f" - Your trading parameters")
|
||||
print(f"\n4. Compile and attach to chart")
|
||||
print(f"\n5. Monitor performance")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Workflow completed!")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Check MT5 connection first
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
print("Make sure MetaTrader 5 is running and you're logged in.")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nInterrupted by user")
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
@@ -0,0 +1,212 @@
|
||||
"""
|
||||
Inspect ONNX Model Predictions in Detail
|
||||
|
||||
This script directly tests the model and shows what it's predicting.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import onnxruntime as ort
|
||||
import pickle
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from indicator_utils import calculate_rsi, calculate_ema, calculate_atr
|
||||
|
||||
|
||||
def main():
|
||||
"""Inspect model predictions."""
|
||||
print("="*60)
|
||||
print("ONNX Model Prediction Inspection")
|
||||
print("="*60)
|
||||
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
|
||||
if not os.path.exists(model_path):
|
||||
print(f"ERROR: Model not found: {model_path}")
|
||||
return
|
||||
|
||||
# Load model
|
||||
session = ort.InferenceSession(model_path)
|
||||
input_name = session.get_inputs()[0].name
|
||||
output_name = session.get_outputs()[0].name
|
||||
input_shape = session.get_inputs()[0].shape
|
||||
lookback = int(input_shape[1]) if input_shape[1] else 60
|
||||
|
||||
print(f"Model Input Shape: {input_shape}")
|
||||
print(f"Lookback: {lookback}")
|
||||
|
||||
# Load scaler
|
||||
with open(scaler_path, 'rb') as f:
|
||||
scaler = pickle.load(f)
|
||||
|
||||
print(f"Scaler Feature Count: {scaler.n_features_in_}")
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
return
|
||||
|
||||
try:
|
||||
# Get data
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=100) # Get more data
|
||||
|
||||
rates = mt5.copy_rates_range(symbol, timeframe, start_date, end_date)
|
||||
if rates is None or len(rates) == 0:
|
||||
print("ERROR: No data")
|
||||
return
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
df.set_index('time', inplace=True)
|
||||
|
||||
print(f"\nLoaded {len(df)} bars")
|
||||
|
||||
# Calculate indicators (same as training)
|
||||
df['rsi'] = calculate_rsi(df['close'], period=14)
|
||||
df['ema_20'] = calculate_ema(df['close'], period=20)
|
||||
df['ema_50'] = calculate_ema(df['close'], period=50)
|
||||
df['atr'] = calculate_atr(df, period=14)
|
||||
df['price_change'] = df['close'].pct_change()
|
||||
df['high_low_ratio'] = df['high'] / df['low']
|
||||
df['volume_ma'] = df['tick_volume'].rolling(window=20).mean()
|
||||
df['volume_ratio'] = df['tick_volume'] / df['volume_ma']
|
||||
|
||||
# Drop NaN
|
||||
df = df.dropna()
|
||||
|
||||
print(f"After indicator calculation: {len(df)} bars")
|
||||
print(f"Features: {len(['open', 'high', 'low', 'close', 'tick_volume', 'rsi', 'ema_20', 'ema_50', 'atr', 'price_change', 'high_low_ratio', 'volume_ma', 'volume_ratio'])}")
|
||||
|
||||
# Test predictions
|
||||
print("\n" + "="*60)
|
||||
print("Testing Predictions")
|
||||
print("="*60)
|
||||
|
||||
predictions = []
|
||||
|
||||
for i in range(lookback, min(lookback + 50, len(df))):
|
||||
# Prepare features (same as training)
|
||||
feature_rows = []
|
||||
|
||||
for j in range(i - lookback, i):
|
||||
bar = df.iloc[j]
|
||||
feature_row = [
|
||||
bar['open'],
|
||||
bar['high'],
|
||||
bar['low'],
|
||||
bar['close'],
|
||||
bar['tick_volume'] / 1000000.0,
|
||||
bar['rsi'] / 100.0,
|
||||
(bar['ema_20'] - bar['close']) / bar['close'] if bar['close'] > 0 else 0.0,
|
||||
(bar['ema_50'] - bar['close']) / bar['close'] if bar['close'] > 0 else 0.0,
|
||||
bar['atr'] / bar['close'] if bar['close'] > 0 else 0.0,
|
||||
bar['price_change'],
|
||||
bar['high_low_ratio'],
|
||||
bar['volume_ma'] / 1000000.0,
|
||||
bar['volume_ratio']
|
||||
]
|
||||
feature_rows.append(feature_row)
|
||||
|
||||
features = np.array(feature_rows, dtype=np.float32)
|
||||
|
||||
# Scale
|
||||
original_shape = features.shape
|
||||
features_flat = features.reshape(-1, features.shape[-1])
|
||||
features_scaled = scaler.transform(features_flat)
|
||||
features = features_scaled.reshape(original_shape)
|
||||
|
||||
# Reshape for model
|
||||
input_data = features.reshape(1, lookback, -1)
|
||||
|
||||
# Predict
|
||||
outputs = session.run([output_name], {input_name: input_data})
|
||||
predicted_change_pct = float(outputs[0][0][0])
|
||||
|
||||
current_price = df.iloc[i]['close']
|
||||
|
||||
# Model predicts price change percentage directly
|
||||
# If it's between -1 and 1, it's already a percentage
|
||||
if abs(predicted_change_pct) < 1.0:
|
||||
price_change_pct = predicted_change_pct * 100 # Convert to percentage (e.g., 0.001 -> 0.1%)
|
||||
predicted_price = current_price * (1 + predicted_change_pct / 100) # Calculate predicted price
|
||||
price_change = predicted_price - current_price
|
||||
else:
|
||||
# Old format: absolute price
|
||||
predicted_price = predicted_change_pct
|
||||
price_change = predicted_price - current_price
|
||||
price_change_pct = (price_change / current_price) * 100 if current_price > 0 else 0.0
|
||||
|
||||
predictions.append({
|
||||
'time': df.index[i],
|
||||
'current_price': current_price,
|
||||
'predicted_price': predicted_price,
|
||||
'price_change': price_change,
|
||||
'price_change_pct': price_change_pct,
|
||||
'abs_change_pct': abs(price_change_pct)
|
||||
})
|
||||
|
||||
if predictions:
|
||||
pred_df = pd.DataFrame(predictions)
|
||||
|
||||
print(f"\nAnalyzed {len(pred_df)} predictions:")
|
||||
print(f"\nPrice Change Statistics:")
|
||||
print(f" Mean: {pred_df['price_change_pct'].mean():.6f}%")
|
||||
print(f" Std: {pred_df['price_change_pct'].std():.6f}%")
|
||||
print(f" Min: {pred_df['price_change_pct'].min():.6f}%")
|
||||
print(f" Max: {pred_df['price_change_pct'].max():.6f}%")
|
||||
print(f" Median: {pred_df['price_change_pct'].median():.6f}%")
|
||||
|
||||
print(f"\nAbsolute Price Change Statistics:")
|
||||
print(f" Mean: {pred_df['abs_change_pct'].mean():.6f}%")
|
||||
print(f" Min: {pred_df['abs_change_pct'].min():.6f}%")
|
||||
print(f" Max: {pred_df['abs_change_pct'].max():.6f}%")
|
||||
print(f" Median: {pred_df['abs_change_pct'].median():.6f}%")
|
||||
|
||||
print(f"\nSample Predictions (first 10):")
|
||||
print(pred_df[['time', 'current_price', 'predicted_price', 'price_change_pct']].head(10).to_string(index=False))
|
||||
|
||||
# Check thresholds
|
||||
threshold_0001 = (pred_df['abs_change_pct'] >= 0.01).sum()
|
||||
threshold_00005 = (pred_df['abs_change_pct'] >= 0.005).sum()
|
||||
threshold_00001 = (pred_df['abs_change_pct'] >= 0.001).sum()
|
||||
|
||||
print(f"\nPredictions meeting thresholds:")
|
||||
print(f" >= 0.01% (0.0001): {threshold_0001}/{len(pred_df)}")
|
||||
print(f" >= 0.005% (0.00005): {threshold_00005}/{len(pred_df)}")
|
||||
print(f" >= 0.001% (0.00001): {threshold_00001}/{len(pred_df)}")
|
||||
|
||||
if threshold_00001 == 0:
|
||||
print("\n" + "="*60)
|
||||
print("ISSUE DETECTED!")
|
||||
print("="*60)
|
||||
print("Even with 0.001% threshold, no predictions qualify.")
|
||||
print("The model may be predicting prices that are too close to current prices.")
|
||||
print("\nPossible solutions:")
|
||||
print(" 1. Retrain model to predict price changes instead of absolute prices")
|
||||
print(" 2. Use a different prediction target (e.g., next bar high/low)")
|
||||
print(" 3. Adjust the model architecture")
|
||||
else:
|
||||
print("No predictions generated")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,142 @@
|
||||
[
|
||||
{
|
||||
"prediction_threshold": 0.0007803532317523569,
|
||||
"min_confidence": 0.4778214378844623,
|
||||
"stop_loss_pips": 126,
|
||||
"take_profit_pips": 111,
|
||||
"lot_size": 0.19966462104925914,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.00035423634886275126,
|
||||
"min_confidence": 0.1201975341512912,
|
||||
"stop_loss_pips": 94,
|
||||
"take_profit_pips": 127,
|
||||
"lot_size": 0.13342715278475548,
|
||||
"total_return": -6.90000000000009,
|
||||
"max_drawdown": 6.90000000000009,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 1,
|
||||
"final_balance": -59000.0000000009
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.001265431347313738,
|
||||
"min_confidence": 0.19890411118369217,
|
||||
"stop_loss_pips": 67,
|
||||
"take_profit_pips": 101,
|
||||
"lot_size": 0.13129583050668675,
|
||||
"total_return": -6.90000000000009,
|
||||
"max_drawdown": 6.90000000000009,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 1,
|
||||
"final_balance": -59000.0000000009
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.0012316219508229722,
|
||||
"min_confidence": 0.46683539533100704,
|
||||
"stop_loss_pips": 60,
|
||||
"take_profit_pips": 67,
|
||||
"lot_size": 0.2657758564688984,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 6.0768128391024685e-05,
|
||||
"min_confidence": 0.41695764280467534,
|
||||
"stop_loss_pips": 100,
|
||||
"take_profit_pips": 202,
|
||||
"lot_size": 0.2475438851328014,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.0001953737551755531,
|
||||
"min_confidence": 0.49409912147023277,
|
||||
"stop_loss_pips": 107,
|
||||
"take_profit_pips": 168,
|
||||
"lot_size": 0.0996789203835431,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.0019322478491650692,
|
||||
"min_confidence": 0.3231654114590081,
|
||||
"stop_loss_pips": 60,
|
||||
"take_profit_pips": 196,
|
||||
"lot_size": 0.2505492451885099,
|
||||
"total_return": -6.90000000000009,
|
||||
"max_drawdown": 6.90000000000009,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 1,
|
||||
"final_balance": -59000.0000000009
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.0019242854474812309,
|
||||
"min_confidence": 0.4300402319051681,
|
||||
"stop_loss_pips": 101,
|
||||
"take_profit_pips": 92,
|
||||
"lot_size": 0.19668779141596204,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.0018569847882978986,
|
||||
"min_confidence": 0.3772723981353894,
|
||||
"stop_loss_pips": 34,
|
||||
"take_profit_pips": 254,
|
||||
"lot_size": 0.18020856500645593,
|
||||
"total_return": -2.349999999999909,
|
||||
"max_drawdown": 2.349999999999909,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 1,
|
||||
"final_balance": -13499.99999999909
|
||||
},
|
||||
{
|
||||
"prediction_threshold": 0.00011106092028833925,
|
||||
"min_confidence": 0.42902814856774935,
|
||||
"stop_loss_pips": 63,
|
||||
"take_profit_pips": 201,
|
||||
"lot_size": 0.1487875590004536,
|
||||
"total_return": 0.0,
|
||||
"max_drawdown": 0.0,
|
||||
"sharpe_ratio": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"win_rate": 0.0,
|
||||
"total_trades": 0,
|
||||
"final_balance": 10000.0
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,359 @@
|
||||
"""
|
||||
Parameter Optimization for ONNX Strategy
|
||||
|
||||
This script optimizes strategy parameters (prediction_threshold, min_confidence,
|
||||
stop_loss_pips, take_profit_pips, lot_size) using grid search or random search.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from itertools import product
|
||||
import json
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
from performance_analyzer import PerformanceAnalyzer
|
||||
|
||||
|
||||
class ONNXParameterOptimizer:
|
||||
"""Optimize ONNX strategy parameters."""
|
||||
|
||||
def __init__(self, symbol: str, timeframe: int, model_path: str, scaler_path: str,
|
||||
start_date: datetime, end_date: datetime, initial_balance: float = 10000.0):
|
||||
"""
|
||||
Initialize optimizer.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol
|
||||
timeframe: MT5 timeframe
|
||||
model_path: Path to ONNX model
|
||||
scaler_path: Path to scaler
|
||||
start_date: Backtest start date
|
||||
end_date: Backtest end date
|
||||
initial_balance: Starting balance
|
||||
"""
|
||||
self.symbol = symbol
|
||||
self.timeframe = timeframe
|
||||
self.model_path = model_path
|
||||
self.scaler_path = scaler_path
|
||||
self.start_date = start_date
|
||||
self.end_date = end_date
|
||||
self.initial_balance = initial_balance
|
||||
|
||||
def grid_search(self, param_grid: dict, metric: str = 'sharpe_ratio') -> pd.DataFrame:
|
||||
"""
|
||||
Perform grid search optimization.
|
||||
|
||||
Args:
|
||||
param_grid: Dictionary of parameter ranges
|
||||
Example: {
|
||||
'prediction_threshold': [0.0001, 0.0002, 0.0005],
|
||||
'min_confidence': [0.2, 0.3, 0.4],
|
||||
'stop_loss_pips': [30, 50, 70],
|
||||
'take_profit_pips': [60, 100, 150],
|
||||
'lot_size': [0.1, 0.2]
|
||||
}
|
||||
metric: Metric to optimize ('sharpe_ratio', 'total_return', 'max_drawdown', 'profit_factor')
|
||||
|
||||
Returns:
|
||||
DataFrame with results sorted by metric
|
||||
"""
|
||||
print("="*60)
|
||||
print("Grid Search Parameter Optimization")
|
||||
print("="*60)
|
||||
|
||||
# Generate all parameter combinations
|
||||
param_names = list(param_grid.keys())
|
||||
param_values = list(param_grid.values())
|
||||
combinations = list(product(*param_values))
|
||||
|
||||
total_combinations = len(combinations)
|
||||
print(f"\nTotal parameter combinations: {total_combinations}")
|
||||
print(f"Optimizing for: {metric}\n")
|
||||
|
||||
results = []
|
||||
|
||||
for i, combo in enumerate(combinations, 1):
|
||||
params = dict(zip(param_names, combo))
|
||||
|
||||
print(f"[{i}/{total_combinations}] Testing: {params}")
|
||||
|
||||
try:
|
||||
# Create strategy with these parameters
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=self.symbol,
|
||||
timeframe=self.timeframe,
|
||||
model_path=self.model_path,
|
||||
scaler_path=self.scaler_path,
|
||||
initial_balance=self.initial_balance,
|
||||
**params
|
||||
)
|
||||
|
||||
# Run backtest
|
||||
engine = BacktestEngine(strategy, self.start_date, self.end_date)
|
||||
backtest_results = engine.run()
|
||||
|
||||
# Calculate metrics
|
||||
analyzer = PerformanceAnalyzer(backtest_results)
|
||||
metrics = analyzer.metrics
|
||||
|
||||
# Store results
|
||||
result = params.copy()
|
||||
# Map metric names to match what we're looking for
|
||||
result['total_return'] = metrics.get('total_return_pct', 0) / 100.0
|
||||
result['max_drawdown'] = metrics.get('max_drawdown_pct', 0) / 100.0
|
||||
result['sharpe_ratio'] = metrics.get('sharpe_ratio', 0.0) if 'sharpe_ratio' in metrics else 0.0
|
||||
result['profit_factor'] = metrics.get('profit_factor', 0.0)
|
||||
result['win_rate'] = metrics.get('win_rate_pct', 0) / 100.0
|
||||
result['total_trades'] = metrics.get('total_trades', 0)
|
||||
result['final_balance'] = metrics.get('final_balance', self.initial_balance)
|
||||
results.append(result)
|
||||
|
||||
metric_value = result.get(metric, 0)
|
||||
print(f" -> {metric}: {metric_value:.4f} | Trades: {result['total_trades']}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" X Error: {e}")
|
||||
continue
|
||||
|
||||
# Convert to DataFrame
|
||||
df_results = pd.DataFrame(results)
|
||||
|
||||
if len(df_results) == 0:
|
||||
raise ValueError("No successful backtests!")
|
||||
|
||||
# Sort by metric (descending for most metrics, ascending for max_drawdown)
|
||||
if metric == 'max_drawdown':
|
||||
df_results = df_results.sort_values(metric, ascending=True)
|
||||
else:
|
||||
df_results = df_results.sort_values(metric, ascending=False)
|
||||
|
||||
return df_results
|
||||
|
||||
def random_search(self, param_ranges: dict, n_iter: int = 50,
|
||||
metric: str = 'sharpe_ratio') -> pd.DataFrame:
|
||||
"""
|
||||
Perform random search optimization.
|
||||
|
||||
Args:
|
||||
param_ranges: Dictionary of parameter ranges
|
||||
Example: {
|
||||
'prediction_threshold': (0.0001, 0.001),
|
||||
'min_confidence': (0.1, 0.5),
|
||||
'stop_loss_pips': (20, 100),
|
||||
'take_profit_pips': (40, 200),
|
||||
'lot_size': (0.1, 0.5)
|
||||
}
|
||||
n_iter: Number of random combinations to test
|
||||
metric: Metric to optimize
|
||||
|
||||
Returns:
|
||||
DataFrame with results sorted by metric
|
||||
"""
|
||||
print("="*60)
|
||||
print("Random Search Parameter Optimization")
|
||||
print("="*60)
|
||||
print(f"\nTesting {n_iter} random parameter combinations")
|
||||
print(f"Optimizing for: {metric}\n")
|
||||
|
||||
results = []
|
||||
np.random.seed(42) # For reproducibility
|
||||
|
||||
for i in range(1, n_iter + 1):
|
||||
# Generate random parameters
|
||||
params = {}
|
||||
for param_name, (min_val, max_val) in param_ranges.items():
|
||||
if isinstance(min_val, int) and isinstance(max_val, int):
|
||||
params[param_name] = np.random.randint(min_val, max_val + 1)
|
||||
else:
|
||||
params[param_name] = np.random.uniform(min_val, max_val)
|
||||
|
||||
print(f"[{i}/{n_iter}] Testing: {params}")
|
||||
|
||||
try:
|
||||
# Create strategy
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=self.symbol,
|
||||
timeframe=self.timeframe,
|
||||
model_path=self.model_path,
|
||||
scaler_path=self.scaler_path,
|
||||
initial_balance=self.initial_balance,
|
||||
**params
|
||||
)
|
||||
|
||||
# Run backtest
|
||||
engine = BacktestEngine(strategy, self.start_date, self.end_date)
|
||||
backtest_results = engine.run()
|
||||
|
||||
# Calculate metrics
|
||||
analyzer = PerformanceAnalyzer(backtest_results)
|
||||
metrics = analyzer.metrics
|
||||
|
||||
# Store results
|
||||
result = params.copy()
|
||||
# Map metric names to match what we're looking for
|
||||
result['total_return'] = metrics.get('total_return_pct', 0) / 100.0
|
||||
result['max_drawdown'] = metrics.get('max_drawdown_pct', 0) / 100.0
|
||||
result['sharpe_ratio'] = metrics.get('sharpe_ratio', 0.0) if 'sharpe_ratio' in metrics else 0.0
|
||||
result['profit_factor'] = metrics.get('profit_factor', 0.0)
|
||||
result['win_rate'] = metrics.get('win_rate_pct', 0) / 100.0
|
||||
result['total_trades'] = metrics.get('total_trades', 0)
|
||||
result['final_balance'] = metrics.get('final_balance', self.initial_balance)
|
||||
results.append(result)
|
||||
|
||||
metric_value = result.get(metric, 0)
|
||||
print(f" -> {metric}: {metric_value:.4f} | Trades: {result['total_trades']}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" X Error: {e}")
|
||||
continue
|
||||
|
||||
# Convert to DataFrame
|
||||
df_results = pd.DataFrame(results)
|
||||
|
||||
if len(df_results) == 0:
|
||||
raise ValueError("No successful backtests!")
|
||||
|
||||
# Sort by metric
|
||||
if metric == 'max_drawdown':
|
||||
df_results = df_results.sort_values(metric, ascending=True)
|
||||
else:
|
||||
df_results = df_results.sort_values(metric, ascending=False)
|
||||
|
||||
return df_results
|
||||
|
||||
def save_results(self, df_results: pd.DataFrame, output_file: str = 'optimization_results.csv'):
|
||||
"""Save optimization results to CSV."""
|
||||
df_results.to_csv(output_file, index=False)
|
||||
print(f"\nResults saved to: {output_file}")
|
||||
|
||||
# Also save top 10 as JSON
|
||||
top_10 = df_results.head(10).to_dict('records')
|
||||
json_file = output_file.replace('.csv', '_top10.json')
|
||||
with open(json_file, 'w') as f:
|
||||
json.dump(top_10, f, indent=2, default=str)
|
||||
print(f"Top 10 results saved to: {json_file}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main optimization function."""
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
initial_balance = 10000.0
|
||||
|
||||
# Backtest date range (use last 6 months for optimization)
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=180)
|
||||
|
||||
# Check if model exists
|
||||
if not os.path.exists(model_path):
|
||||
print(f"ERROR: Model not found: {model_path}")
|
||||
print("Please train the model first using train_onnx_model.py")
|
||||
return
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
return
|
||||
|
||||
try:
|
||||
# Create optimizer
|
||||
optimizer = ONNXParameterOptimizer(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
initial_balance=initial_balance
|
||||
)
|
||||
|
||||
# Choose optimization method (use command line args or defaults)
|
||||
import sys
|
||||
choice = "2" # Default to random search
|
||||
n_iter = 30 # Default iterations
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
choice = sys.argv[1]
|
||||
if len(sys.argv) > 2:
|
||||
n_iter = int(sys.argv[2])
|
||||
|
||||
print("\nOptimization Configuration:")
|
||||
print(f"Method: {'Grid Search' if choice == '1' else 'Random Search'}")
|
||||
if choice != "1":
|
||||
print(f"Iterations: {n_iter}")
|
||||
print()
|
||||
|
||||
if choice == "1":
|
||||
# Grid search parameters
|
||||
param_grid = {
|
||||
'prediction_threshold': [0.0001, 0.0002, 0.0005, 0.001],
|
||||
'min_confidence': [0.1, 0.2, 0.3, 0.4],
|
||||
'stop_loss_pips': [30, 50, 70, 100],
|
||||
'take_profit_pips': [60, 100, 150, 200],
|
||||
'lot_size': [0.1, 0.2]
|
||||
}
|
||||
|
||||
results = optimizer.grid_search(param_grid, metric='sharpe_ratio')
|
||||
|
||||
else:
|
||||
# Random search parameters
|
||||
param_ranges = {
|
||||
'prediction_threshold': (0.00005, 0.002), # Lower threshold to get more trades
|
||||
'min_confidence': (0.05, 0.5), # Lower confidence requirement
|
||||
'stop_loss_pips': (20, 150),
|
||||
'take_profit_pips': (40, 300),
|
||||
'lot_size': (0.05, 0.3)
|
||||
}
|
||||
|
||||
results = optimizer.random_search(param_ranges, n_iter=n_iter, metric='sharpe_ratio')
|
||||
|
||||
# Display top results
|
||||
print("\n" + "="*60)
|
||||
print("Top 10 Results")
|
||||
print("="*60)
|
||||
print(results.head(10).to_string(index=False))
|
||||
|
||||
# Save results
|
||||
optimizer.save_results(results, 'onnx_optimization_results.csv')
|
||||
|
||||
# Display best parameters
|
||||
best = results.iloc[0]
|
||||
print("\n" + "="*60)
|
||||
print("Best Parameters")
|
||||
print("="*60)
|
||||
print(f"Prediction Threshold: {best['prediction_threshold']:.6f}")
|
||||
print(f"Min Confidence: {best['min_confidence']:.2f}")
|
||||
print(f"Stop Loss (pips): {best['stop_loss_pips']}")
|
||||
print(f"Take Profit (pips): {best['take_profit_pips']}")
|
||||
print(f"Lot Size: {best['lot_size']:.2f}")
|
||||
print(f"\nPerformance Metrics:")
|
||||
print(f" Sharpe Ratio: {best.get('sharpe_ratio', 'N/A'):.4f}")
|
||||
print(f" Total Return: {best.get('total_return', 'N/A'):.2%}")
|
||||
print(f" Max Drawdown: {best.get('max_drawdown', 'N/A'):.2%}")
|
||||
print(f" Profit Factor: {best.get('profit_factor', 'N/A'):.2f}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nOptimization interrupted by user")
|
||||
except Exception as e:
|
||||
print(f"\nERROR: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,261 @@
|
||||
"""
|
||||
ONNX Model Prediction Script
|
||||
|
||||
This script loads a trained ONNX model and makes predictions using MT5 data.
|
||||
Can be run from MetaEditor or directly in Python.
|
||||
|
||||
Usage:
|
||||
python predict_with_onnx.py --model models/XAUUSD_H1_model.onnx --symbol XAUUSD
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import MetaTrader5 as mt5
|
||||
import onnxruntime as ort
|
||||
from datetime import datetime
|
||||
from sklearn.preprocessing import MinMaxScaler
|
||||
import pickle
|
||||
import os
|
||||
|
||||
|
||||
class ONNXPredictor:
|
||||
"""
|
||||
Predictor class for using ONNX models with MT5 data.
|
||||
"""
|
||||
|
||||
def __init__(self, model_path: str, scaler_path: str = None):
|
||||
"""
|
||||
Initialize the predictor.
|
||||
|
||||
Args:
|
||||
model_path: Path to ONNX model file
|
||||
scaler_path: Path to saved scaler (optional, will create if not provided)
|
||||
"""
|
||||
self.model_path = model_path
|
||||
self.scaler_path = scaler_path
|
||||
|
||||
# Load ONNX model
|
||||
if not os.path.exists(model_path):
|
||||
raise FileNotFoundError(f"ONNX model not found: {model_path}")
|
||||
|
||||
self.session = ort.InferenceSession(model_path)
|
||||
|
||||
# Get input/output info
|
||||
self.input_name = self.session.get_inputs()[0].name
|
||||
self.output_name = self.session.get_outputs()[0].name
|
||||
self.input_shape = self.session.get_inputs()[0].shape
|
||||
|
||||
print(f"Loaded ONNX model: {model_path}")
|
||||
print(f"Input shape: {self.input_shape}")
|
||||
print(f"Input name: {self.input_name}")
|
||||
print(f"Output name: {self.output_name}")
|
||||
|
||||
# Load or create scaler
|
||||
if scaler_path and os.path.exists(scaler_path):
|
||||
with open(scaler_path, 'rb') as f:
|
||||
self.scaler = pickle.load(f)
|
||||
print(f"Loaded scaler from: {scaler_path}")
|
||||
else:
|
||||
self.scaler = MinMaxScaler()
|
||||
print("Using default scaler (will need to fit)")
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
raise RuntimeError(f"MT5 initialization failed: {mt5.last_error()}")
|
||||
|
||||
def prepare_features(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Prepare features from raw OHLCV data (same as training).
|
||||
|
||||
Args:
|
||||
df: Raw OHLCV data
|
||||
|
||||
Returns:
|
||||
DataFrame with features
|
||||
"""
|
||||
features = ['open', 'high', 'low', 'close', 'tick_volume']
|
||||
feature_df = df[features].copy()
|
||||
|
||||
# Add technical indicators
|
||||
feature_df['rsi'] = self._calculate_rsi(df['close'], period=14)
|
||||
feature_df['ema_20'] = df['close'].ewm(span=20).mean()
|
||||
feature_df['ema_50'] = df['close'].ewm(span=50).mean()
|
||||
feature_df['atr'] = self._calculate_atr(df, period=14)
|
||||
feature_df['price_change'] = df['close'].pct_change()
|
||||
feature_df['high_low_ratio'] = df['high'] / df['low']
|
||||
feature_df['volume_ma'] = df['tick_volume'].rolling(window=20).mean()
|
||||
feature_df['volume_ratio'] = df['tick_volume'] / feature_df['volume_ma']
|
||||
|
||||
feature_df = feature_df.dropna()
|
||||
return feature_df
|
||||
|
||||
def _calculate_rsi(self, prices: pd.Series, period: int = 14) -> pd.Series:
|
||||
"""Calculate RSI indicator."""
|
||||
delta = prices.diff()
|
||||
gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
|
||||
loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
|
||||
rs = gain / loss
|
||||
rsi = 100 - (100 / (1 + rs))
|
||||
return rsi
|
||||
|
||||
def _calculate_atr(self, df: pd.DataFrame, period: int = 14) -> pd.Series:
|
||||
"""Calculate ATR indicator."""
|
||||
high_low = df['high'] - df['low']
|
||||
high_close = np.abs(df['high'] - df['close'].shift())
|
||||
low_close = np.abs(df['low'] - df['close'].shift())
|
||||
tr = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1)
|
||||
atr = tr.rolling(window=period).mean()
|
||||
return atr
|
||||
|
||||
def get_latest_data(self, symbol: str, timeframe: int, lookback: int) -> np.ndarray:
|
||||
"""
|
||||
Get latest data from MT5 and prepare for prediction.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol
|
||||
timeframe: MT5 timeframe
|
||||
lookback: Number of bars needed
|
||||
|
||||
Returns:
|
||||
Prepared feature array ready for model input
|
||||
"""
|
||||
# Fetch data
|
||||
rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, lookback + 50)
|
||||
|
||||
if rates is None or len(rates) < lookback:
|
||||
raise ValueError(f"Insufficient data for {symbol}")
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
|
||||
# Prepare features
|
||||
feature_df = self.prepare_features(df)
|
||||
|
||||
# Get last lookback bars
|
||||
feature_data = feature_df.values[-lookback:]
|
||||
|
||||
# Scale features
|
||||
if hasattr(self.scaler, 'scale_'):
|
||||
feature_data_scaled = self.scaler.transform(feature_data)
|
||||
else:
|
||||
# Fit scaler if not already fitted
|
||||
print("Warning: Scaler not fitted, fitting on current data...")
|
||||
feature_data_scaled = self.scaler.fit_transform(feature_data)
|
||||
|
||||
# Reshape for model input: (1, lookback, features)
|
||||
feature_data_scaled = feature_data_scaled.reshape(1, lookback, -1)
|
||||
|
||||
return feature_data_scaled.astype(np.float32)
|
||||
|
||||
def predict(self, symbol: str, timeframe: int, lookback: int = None) -> float:
|
||||
"""
|
||||
Make a prediction for the next price.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol
|
||||
timeframe: MT5 timeframe
|
||||
lookback: Number of bars to use (default: from model input shape)
|
||||
|
||||
Returns:
|
||||
Predicted price
|
||||
"""
|
||||
if lookback is None:
|
||||
lookback = self.input_shape[1] if self.input_shape[1] else 60
|
||||
|
||||
# Get and prepare data
|
||||
input_data = self.get_latest_data(symbol, timeframe, lookback)
|
||||
|
||||
# Make prediction
|
||||
outputs = self.session.run([self.output_name], {self.input_name: input_data})
|
||||
prediction = outputs[0][0][0]
|
||||
|
||||
return float(prediction)
|
||||
|
||||
def predict_batch(self, symbol: str, timeframe: int, n_predictions: int = 5) -> list:
|
||||
"""
|
||||
Make multiple predictions.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol
|
||||
timeframe: MT5 timeframe
|
||||
n_predictions: Number of predictions to make
|
||||
|
||||
Returns:
|
||||
List of predictions
|
||||
"""
|
||||
predictions = []
|
||||
for _ in range(n_predictions):
|
||||
pred = self.predict(symbol, timeframe)
|
||||
predictions.append(pred)
|
||||
return predictions
|
||||
|
||||
def cleanup(self):
|
||||
"""Clean up MT5 connection."""
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(description='Make predictions using ONNX model')
|
||||
parser.add_argument('--model', type=str, required=True,
|
||||
help='Path to ONNX model file')
|
||||
parser.add_argument('--symbol', type=str, default='XAUUSD',
|
||||
help='Trading symbol')
|
||||
parser.add_argument('--timeframe', type=str, default='H1',
|
||||
choices=['M1', 'M5', 'M15', 'M30', 'H1', 'H4', 'D1'],
|
||||
help='Timeframe')
|
||||
parser.add_argument('--scaler', type=str, default=None,
|
||||
help='Path to saved scaler (optional)')
|
||||
parser.add_argument('--predictions', type=int, default=1,
|
||||
help='Number of predictions to make')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Convert timeframe
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[args.timeframe]
|
||||
|
||||
# Create predictor
|
||||
predictor = ONNXPredictor(args.model, args.scaler)
|
||||
|
||||
try:
|
||||
# Get current price
|
||||
symbol_info = mt5.symbol_info(args.symbol)
|
||||
current_price = symbol_info.bid if symbol_info else 0
|
||||
|
||||
print(f"\nCurrent {args.symbol} price: {current_price:.5f}")
|
||||
print(f"Making {args.predictions} prediction(s)...\n")
|
||||
|
||||
# Make predictions
|
||||
if args.predictions == 1:
|
||||
prediction = predictor.predict(args.symbol, timeframe)
|
||||
print(f"Predicted next price: {prediction:.5f}")
|
||||
print(f"Expected change: {(prediction - current_price):.5f} "
|
||||
f"({((prediction - current_price) / current_price * 100):.2f}%)")
|
||||
else:
|
||||
predictions = predictor.predict_batch(args.symbol, timeframe, args.predictions)
|
||||
print("Predictions:")
|
||||
for i, pred in enumerate(predictions, 1):
|
||||
change = pred - current_price
|
||||
change_pct = (change / current_price * 100) if current_price > 0 else 0
|
||||
print(f" {i}. {pred:.5f} (change: {change:+.5f}, {change_pct:+.2f}%)")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nError: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
predictor.cleanup()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""
|
||||
Quick Backtest Script for ONNX Model
|
||||
|
||||
Simple script to quickly backtest the trained ONNX model with default parameters.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
from performance_analyzer import PerformanceAnalyzer
|
||||
|
||||
|
||||
def main():
|
||||
"""Run quick backtest."""
|
||||
print("="*60)
|
||||
print("XAUUSD ONNX Model Quick Backtest")
|
||||
print("="*60)
|
||||
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
initial_balance = 10000.0
|
||||
|
||||
# Check if model exists
|
||||
if not os.path.exists(model_path):
|
||||
print(f"\nERROR: Model not found: {model_path}")
|
||||
print("Please train the model first using:")
|
||||
print(" python train_onnx_model.py --symbol XAUUSD --timeframe H1")
|
||||
return
|
||||
|
||||
# Backtest date range
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=180) # Last 6 months
|
||||
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Symbol: {symbol}")
|
||||
print(f"Timeframe: H1")
|
||||
print(f"Date Range: {start_date.date()} to {end_date.date()}")
|
||||
print(f"Initial Balance: ${initial_balance:,.2f}\n")
|
||||
|
||||
# Adjusted parameters (more relaxed to generate trades)
|
||||
print("Strategy Parameters (Adjusted for Testing):")
|
||||
print(" Prediction Threshold: 0.00005 (0.005%) - LOWERED")
|
||||
print(" Min Confidence: 0.1 (10%) - LOWERED")
|
||||
print(" Stop Loss: 50 pips")
|
||||
print(" Take Profit: 100 pips")
|
||||
print(" Lot Size: 0.1\n")
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
print("Make sure MetaTrader 5 is running and you're logged in.")
|
||||
return
|
||||
|
||||
try:
|
||||
# Create strategy with relaxed parameters
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=initial_balance,
|
||||
prediction_threshold=0.00005, # Lowered from 0.0001
|
||||
min_confidence=0.1, # Lowered from 0.3
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
|
||||
# Run backtest
|
||||
print("Running backtest...\n")
|
||||
engine = BacktestEngine(strategy, start_date, end_date)
|
||||
results = engine.run()
|
||||
|
||||
# Analyze results
|
||||
print("\n" + "="*60)
|
||||
print("Performance Summary")
|
||||
print("="*60)
|
||||
|
||||
analyzer = PerformanceAnalyzer(results)
|
||||
metrics = analyzer.metrics
|
||||
|
||||
print(f"\nTotal Return: {metrics.get('total_return_pct', 0):.2f}%")
|
||||
print(f"Max Drawdown: {metrics.get('max_drawdown_pct', 0):.2f}%")
|
||||
print(f"Profit Factor: {metrics.get('profit_factor', 0):.2f}")
|
||||
print(f"Win Rate: {metrics.get('win_rate_pct', 0):.2f}%")
|
||||
print(f"Total Trades: {metrics.get('total_trades', 0)}")
|
||||
print(f"Final Balance: ${metrics.get('final_balance', initial_balance):,.2f}")
|
||||
|
||||
# Generate report
|
||||
analyzer.generate_report('onnx_xauusd_quick_backtest')
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Backtest Completed!")
|
||||
print("="*60)
|
||||
print(f"\nResults saved to: onnx_xauusd_quick_backtest/")
|
||||
print("\nTo optimize parameters, run:")
|
||||
print(" python optimize_onnx_params.py")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: Backtest failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,17 @@
|
||||
# ONNX and Machine Learning
|
||||
onnx>=1.12.0
|
||||
onnxruntime>=1.12.0
|
||||
tensorflow>=2.10.0
|
||||
tf2onnx>=1.13.0
|
||||
|
||||
# Data Processing
|
||||
pandas>=1.3.0
|
||||
numpy>=1.21.0
|
||||
scikit-learn>=1.0.0
|
||||
|
||||
# MetaTrader 5 Integration
|
||||
MetaTrader5>=5.0.45
|
||||
|
||||
# Visualization and Utilities
|
||||
matplotlib>=3.4.0
|
||||
tqdm>=4.64.0
|
||||
@@ -0,0 +1,115 @@
|
||||
"""
|
||||
Retrain XAUUSD ONNX Model with Improved Settings
|
||||
|
||||
This script retrains the model with:
|
||||
- Price change percentage prediction (instead of absolute price)
|
||||
- More training epochs
|
||||
- Better model architecture
|
||||
- Improved data preprocessing
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, current_dir)
|
||||
|
||||
from train_onnx_model import ONNXModelTrainer
|
||||
|
||||
|
||||
def main():
|
||||
"""Retrain XAUUSD model with improved settings."""
|
||||
print("="*60)
|
||||
print("Retraining XAUUSD ONNX Model (Improved)")
|
||||
print("="*60)
|
||||
|
||||
symbol = 'XAUUSD'
|
||||
timeframe_str = 'H1'
|
||||
lookback = 60
|
||||
epochs = 50 # More epochs for better training
|
||||
|
||||
# Convert timeframe
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[timeframe_str]
|
||||
|
||||
# Create models directory
|
||||
models_dir = 'models'
|
||||
os.makedirs(models_dir, exist_ok=True)
|
||||
|
||||
print(f"\nConfiguration:")
|
||||
print(f" Symbol: {symbol}")
|
||||
print(f" Timeframe: {timeframe_str}")
|
||||
print(f" Lookback: {lookback} bars")
|
||||
print(f" Epochs: {epochs}")
|
||||
print(f" Prediction: Price change percentage (improved)")
|
||||
print("\nThis will take 10-20 minutes...\n")
|
||||
|
||||
trainer = ONNXModelTrainer(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
lookback=lookback
|
||||
)
|
||||
|
||||
try:
|
||||
# Train model
|
||||
trainer.train(epochs=epochs, batch_size=32, verbose=1)
|
||||
|
||||
# Export model
|
||||
model_name = f"{symbol}_{timeframe_str}_model.onnx"
|
||||
model_path = os.path.join(models_dir, model_name)
|
||||
|
||||
print(f"\nExporting model to ONNX format...")
|
||||
trainer.export_to_onnx(model_path)
|
||||
|
||||
# Save scaler
|
||||
scaler_name = f"{symbol}_{timeframe_str}_scaler.pkl"
|
||||
scaler_path = os.path.join(models_dir, scaler_name)
|
||||
import pickle
|
||||
with open(scaler_path, 'wb') as f:
|
||||
pickle.dump(trainer.scaler, f)
|
||||
print(f"Scaler saved to: {scaler_path}")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("Retraining Completed Successfully!")
|
||||
print(f"{'='*60}")
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Scaler: {scaler_path}")
|
||||
print("\nNext steps:")
|
||||
print(" 1. Run: python quick_backtest.py")
|
||||
print(" 2. Or: python optimize_onnx_params.py 2 30")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: Training failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
trainer.cleanup()
|
||||
return
|
||||
|
||||
finally:
|
||||
trainer.cleanup()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Check MT5 connection
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
print("Make sure MetaTrader 5 is running and you're logged in.")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nTraining interrupted by user")
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
@@ -0,0 +1,115 @@
|
||||
"""
|
||||
Backtest XAUUSD ONNX Model
|
||||
|
||||
This script backtests a trained ONNX model for XAUUSD.
|
||||
Make sure you have trained the model first using train_onnx_model.py
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
from performance_analyzer import PerformanceAnalyzer
|
||||
|
||||
|
||||
def main():
|
||||
"""Run backtest for XAUUSD ONNX model."""
|
||||
print("="*60)
|
||||
print("XAUUSD ONNX Model Backtest")
|
||||
print("="*60)
|
||||
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
initial_balance = 10000.0
|
||||
|
||||
# Check if model exists
|
||||
if not os.path.exists(model_path):
|
||||
print(f"\nERROR: Model not found: {model_path}")
|
||||
print("Please train the model first using:")
|
||||
print(" python train_onnx_model.py --symbol XAUUSD --timeframe H1")
|
||||
return
|
||||
|
||||
if not os.path.exists(scaler_path):
|
||||
print(f"\nWARNING: Scaler not found: {scaler_path}")
|
||||
print("Will use default normalization (may affect accuracy)")
|
||||
scaler_path = None
|
||||
|
||||
# Backtest date range
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=180) # Last 6 months
|
||||
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Symbol: {symbol}")
|
||||
print(f"Timeframe: H1")
|
||||
print(f"Date Range: {start_date.date()} to {end_date.date()}")
|
||||
print(f"Initial Balance: ${initial_balance:,.2f}\n")
|
||||
|
||||
# Create strategy
|
||||
try:
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=initial_balance,
|
||||
prediction_threshold=0.0001, # 0.01% minimum change
|
||||
min_confidence=0.3, # 30% minimum confidence
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"ERROR: Failed to create strategy: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
print("Make sure MetaTrader 5 is running and you're logged in.")
|
||||
return
|
||||
|
||||
try:
|
||||
# Run backtest
|
||||
print("Running backtest...\n")
|
||||
engine = BacktestEngine(strategy, start_date, end_date)
|
||||
results = engine.run()
|
||||
|
||||
# Analyze results
|
||||
print("\n" + "="*60)
|
||||
print("Performance Analysis")
|
||||
print("="*60)
|
||||
|
||||
analyzer = PerformanceAnalyzer(results)
|
||||
analyzer.generate_report('onnx_xauusd_backtest')
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Backtest Completed!")
|
||||
print("="*60)
|
||||
print(f"\nResults saved to: onnx_xauusd_backtest/")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: Backtest failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nInterrupted by user")
|
||||
@@ -0,0 +1,21 @@
|
||||
"""
|
||||
Quick script to train XAUUSD ONNX model
|
||||
|
||||
Run this to train a model for XAUUSD.
|
||||
After training, you can use the model for backtesting or live trading.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Ensure we're in the right directory
|
||||
os.chdir(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Train the model
|
||||
print("Training XAUUSD ONNX model...")
|
||||
print("This will take several minutes...\n")
|
||||
|
||||
os.system('python train_onnx_model.py --symbol XAUUSD --timeframe H1 --lookback 60 --epochs 30 --batch-size 32')
|
||||
|
||||
print("\nTraining completed! Model saved to models/XAUUSD_H1_model.onnx")
|
||||
print("Scaler saved to models/XAUUSD_H1_scaler.pkl")
|
||||
@@ -0,0 +1,103 @@
|
||||
"""
|
||||
Test with very low thresholds to see if we can get any trades
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
from performance_analyzer import PerformanceAnalyzer
|
||||
|
||||
|
||||
def main():
|
||||
"""Test with very low thresholds."""
|
||||
print("="*60)
|
||||
print("Testing with VERY LOW Thresholds")
|
||||
print("="*60)
|
||||
|
||||
symbol = 'XAUUSD'
|
||||
timeframe = mt5.TIMEFRAME_H1
|
||||
model_path = 'models/XAUUSD_H1_model.onnx'
|
||||
scaler_path = 'models/XAUUSD_H1_scaler.pkl'
|
||||
initial_balance = 10000.0
|
||||
|
||||
if not os.path.exists(model_path):
|
||||
print(f"ERROR: Model not found: {model_path}")
|
||||
return
|
||||
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=180)
|
||||
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Date Range: {start_date.date()} to {end_date.date()}")
|
||||
print(f"\nVERY RELAXED Parameters:")
|
||||
print(" Prediction Threshold: 0.00001 (0.001%)")
|
||||
print(" Min Confidence: 0.05 (5%)")
|
||||
print(" Stop Loss: 50 pips")
|
||||
print(" Take Profit: 100 pips")
|
||||
print(" Lot Size: 0.1\n")
|
||||
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
return
|
||||
|
||||
try:
|
||||
# Create strategy with VERY low thresholds
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=initial_balance,
|
||||
prediction_threshold=0.00001, # Very low: 0.001%
|
||||
min_confidence=0.05, # Very low: 5%
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
|
||||
print("Running backtest...\n")
|
||||
engine = BacktestEngine(strategy, start_date, end_date)
|
||||
results = engine.run()
|
||||
|
||||
analyzer = PerformanceAnalyzer(results)
|
||||
metrics = analyzer.metrics
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Results")
|
||||
print("="*60)
|
||||
print(f"Total Trades: {metrics.get('total_trades', 0)}")
|
||||
print(f"Final Balance: ${metrics.get('final_balance', initial_balance):,.2f}")
|
||||
print(f"Total Return: {metrics.get('total_return_pct', 0):.2f}%")
|
||||
|
||||
if metrics.get('total_trades', 0) == 0:
|
||||
print("\n" + "="*60)
|
||||
print("STILL NO TRADES!")
|
||||
print("="*60)
|
||||
print("This suggests the model predictions may be:")
|
||||
print(" 1. Too small in magnitude")
|
||||
print(" 2. Not meeting even very low thresholds")
|
||||
print(" 3. Or there's an issue with the prediction logic")
|
||||
print("\nNext steps:")
|
||||
print(" - Check model predictions directly")
|
||||
print(" - Verify feature preparation matches training")
|
||||
print(" - Consider retraining with different architecture")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nERROR: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Train ONNX Model for XAUUSD and Backtest
|
||||
|
||||
This script:
|
||||
1. Trains an ONNX model for XAUUSD
|
||||
2. Runs backtest using the trained model
|
||||
3. Generates performance report
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import MetaTrader5 as mt5
|
||||
|
||||
# Add paths
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
backtest_dir = os.path.join(os.path.dirname(current_dir), 'backtesting', 'MT5')
|
||||
sys.path.insert(0, current_dir)
|
||||
sys.path.insert(0, backtest_dir)
|
||||
|
||||
from train_onnx_model import ONNXModelTrainer
|
||||
from backtest_engine import BacktestEngine
|
||||
from onnx_backtest_strategy import ONNXBacktestStrategy
|
||||
from performance_analyzer import PerformanceAnalyzer
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function to train model and run backtest."""
|
||||
print("="*60)
|
||||
print("XAUUSD ONNX Model Training and Backtesting")
|
||||
print("="*60)
|
||||
|
||||
# Configuration
|
||||
symbol = 'XAUUSD'
|
||||
timeframe_str = 'H1'
|
||||
lookback = 60
|
||||
epochs = 30 # Reduced for faster training
|
||||
initial_balance = 10000.0
|
||||
|
||||
# Convert timeframe
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[timeframe_str]
|
||||
|
||||
# Create models directory
|
||||
models_dir = 'models'
|
||||
os.makedirs(models_dir, exist_ok=True)
|
||||
|
||||
# Step 1: Train Model
|
||||
print("\n" + "="*60)
|
||||
print("STEP 1: Training ONNX Model")
|
||||
print("="*60)
|
||||
|
||||
trainer = ONNXModelTrainer(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
lookback=lookback
|
||||
)
|
||||
|
||||
try:
|
||||
print(f"\nTraining model for {symbol} on {timeframe_str} timeframe...")
|
||||
print(f"Lookback: {lookback} bars")
|
||||
print(f"Epochs: {epochs}")
|
||||
print("\nThis may take several minutes...\n")
|
||||
|
||||
trainer.train(epochs=epochs, batch_size=32, verbose=1)
|
||||
|
||||
# Export model
|
||||
model_name = f"{symbol}_{timeframe_str}_model.onnx"
|
||||
model_path = os.path.join(models_dir, model_name)
|
||||
|
||||
print(f"\nExporting model to ONNX format...")
|
||||
trainer.export_to_onnx(model_path)
|
||||
|
||||
# Save scaler
|
||||
scaler_name = f"{symbol}_{timeframe_str}_scaler.pkl"
|
||||
scaler_path = os.path.join(models_dir, scaler_name)
|
||||
import pickle
|
||||
with open(scaler_path, 'wb') as f:
|
||||
pickle.dump(trainer.scaler, f)
|
||||
print(f"✓ Scaler saved to: {scaler_path}")
|
||||
|
||||
print(f"\n✓ Model saved to: {model_path}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Training failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
trainer.cleanup()
|
||||
return
|
||||
|
||||
finally:
|
||||
trainer.cleanup()
|
||||
|
||||
# Step 2: Run Backtest
|
||||
print("\n" + "="*60)
|
||||
print("STEP 2: Running Backtest")
|
||||
print("="*60)
|
||||
|
||||
# Backtest date range (last 6 months for testing)
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=180)
|
||||
|
||||
# Create strategy
|
||||
strategy = ONNXBacktestStrategy(
|
||||
symbol=symbol,
|
||||
timeframe=timeframe,
|
||||
model_path=model_path,
|
||||
scaler_path=scaler_path,
|
||||
initial_balance=initial_balance,
|
||||
prediction_threshold=0.0001, # 0.01% minimum change
|
||||
min_confidence=0.3, # 30% minimum confidence
|
||||
lot_size=0.1,
|
||||
stop_loss_pips=50,
|
||||
take_profit_pips=100
|
||||
)
|
||||
|
||||
# Run backtest
|
||||
try:
|
||||
print(f"\nRunning backtest from {start_date.date()} to {end_date.date()}...")
|
||||
engine = BacktestEngine(strategy, start_date, end_date)
|
||||
results = engine.run()
|
||||
|
||||
# Analyze results
|
||||
print("\n" + "="*60)
|
||||
print("STEP 3: Performance Analysis")
|
||||
print("="*60)
|
||||
|
||||
analyzer = PerformanceAnalyzer(results)
|
||||
analyzer.generate_report('onnx_backtest_results')
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Training and Backtesting Completed!")
|
||||
print("="*60)
|
||||
print(f"\nModel: {model_path}")
|
||||
print(f"Scaler: {scaler_path}")
|
||||
print(f"Results: onnx_backtest_results/")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Backtest failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Check MT5 connection
|
||||
if not mt5.initialize():
|
||||
print("ERROR: Failed to initialize MT5")
|
||||
print("Make sure MetaTrader 5 is running and you're logged in.")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nInterrupted by user")
|
||||
finally:
|
||||
mt5.shutdown()
|
||||
@@ -0,0 +1,493 @@
|
||||
"""
|
||||
ONNX Model Training Script for MetaTrader 5
|
||||
|
||||
This script trains a neural network model for price prediction and exports it to ONNX format.
|
||||
Based on MQL5 ONNX documentation: https://www.mql5.com/en/docs/onnx/onnx_prepare
|
||||
|
||||
Usage:
|
||||
python train_onnx_model.py --symbol XAUUSD --timeframe H1 --lookback 60 --epochs 50
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import MetaTrader5 as mt5
|
||||
import tensorflow as tf
|
||||
from tensorflow import keras
|
||||
from tensorflow.keras import layers
|
||||
from sklearn.preprocessing import MinMaxScaler
|
||||
from sklearn.model_selection import train_test_split
|
||||
import tf2onnx
|
||||
import onnx
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
class ONNXModelTrainer:
|
||||
"""
|
||||
Trainer class for creating ONNX models from MT5 data.
|
||||
"""
|
||||
|
||||
def __init__(self, symbol: str, timeframe: int, lookback: int = 60,
|
||||
prediction_horizon: int = 1, features: list = None):
|
||||
"""
|
||||
Initialize the trainer.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol (e.g., 'XAUUSD', 'EURUSD')
|
||||
timeframe: MT5 timeframe constant
|
||||
lookback: Number of bars to look back for prediction
|
||||
prediction_horizon: Number of bars ahead to predict
|
||||
features: List of features to use (default: OHLC + volume)
|
||||
"""
|
||||
self.symbol = symbol
|
||||
self.timeframe = timeframe
|
||||
self.lookback = lookback
|
||||
self.prediction_horizon = prediction_horizon
|
||||
self.features = features or ['open', 'high', 'low', 'close', 'tick_volume']
|
||||
|
||||
self.scaler = MinMaxScaler()
|
||||
self.model = None
|
||||
|
||||
# Initialize MT5
|
||||
if not mt5.initialize():
|
||||
raise RuntimeError(f"MT5 initialization failed: {mt5.last_error()}")
|
||||
|
||||
def fetch_data(self, start_date: datetime, end_date: datetime) -> pd.DataFrame:
|
||||
"""
|
||||
Fetch historical data from MT5.
|
||||
|
||||
Args:
|
||||
start_date: Start date for data
|
||||
end_date: End date for data
|
||||
|
||||
Returns:
|
||||
DataFrame with OHLCV data
|
||||
"""
|
||||
print(f"Fetching data for {self.symbol} from {start_date} to {end_date}...")
|
||||
|
||||
rates = mt5.copy_rates_range(self.symbol, self.timeframe, start_date, end_date)
|
||||
|
||||
if rates is None or len(rates) == 0:
|
||||
raise ValueError(f"No data available for {self.symbol} in the specified date range")
|
||||
|
||||
df = pd.DataFrame(rates)
|
||||
df['time'] = pd.to_datetime(df['time'], unit='s')
|
||||
df.set_index('time', inplace=True)
|
||||
|
||||
print(f"Fetched {len(df)} bars")
|
||||
return df
|
||||
|
||||
def prepare_features(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Prepare features for training.
|
||||
|
||||
Args:
|
||||
df: Raw OHLCV data
|
||||
|
||||
Returns:
|
||||
DataFrame with features
|
||||
"""
|
||||
feature_df = df[self.features].copy()
|
||||
|
||||
# Add technical indicators as features
|
||||
feature_df['rsi'] = self._calculate_rsi(df['close'], period=14)
|
||||
feature_df['ema_20'] = df['close'].ewm(span=20).mean()
|
||||
feature_df['ema_50'] = df['close'].ewm(span=50).mean()
|
||||
feature_df['atr'] = self._calculate_atr(df, period=14)
|
||||
|
||||
# Price changes
|
||||
feature_df['price_change'] = df['close'].pct_change()
|
||||
feature_df['high_low_ratio'] = df['high'] / df['low']
|
||||
|
||||
# Volume features
|
||||
feature_df['volume_ma'] = df['tick_volume'].rolling(window=20).mean()
|
||||
feature_df['volume_ratio'] = df['tick_volume'] / feature_df['volume_ma']
|
||||
|
||||
# Drop NaN values
|
||||
feature_df = feature_df.dropna()
|
||||
|
||||
return feature_df
|
||||
|
||||
def _calculate_rsi(self, prices: pd.Series, period: int = 14) -> pd.Series:
|
||||
"""Calculate RSI indicator."""
|
||||
delta = prices.diff()
|
||||
gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
|
||||
loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
|
||||
rs = gain / loss
|
||||
rsi = 100 - (100 / (1 + rs))
|
||||
return rsi
|
||||
|
||||
def _calculate_atr(self, df: pd.DataFrame, period: int = 14) -> pd.Series:
|
||||
"""Calculate ATR indicator."""
|
||||
high_low = df['high'] - df['low']
|
||||
high_close = np.abs(df['high'] - df['close'].shift())
|
||||
low_close = np.abs(df['low'] - df['close'].shift())
|
||||
tr = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1)
|
||||
atr = tr.rolling(window=period).mean()
|
||||
return atr
|
||||
|
||||
def create_sequences(self, data: np.ndarray, target: np.ndarray) -> tuple:
|
||||
"""
|
||||
Create sequences for LSTM/RNN training.
|
||||
|
||||
Args:
|
||||
data: Feature data
|
||||
target: Target values (price change percentages)
|
||||
|
||||
Returns:
|
||||
Tuple of (X, y) sequences
|
||||
"""
|
||||
X, y = [], []
|
||||
|
||||
for i in range(self.lookback, len(data) - self.prediction_horizon + 1):
|
||||
X.append(data[i - self.lookback:i])
|
||||
# Target is already the price change percentage at position i
|
||||
y.append(target[i])
|
||||
|
||||
return np.array(X), np.array(y)
|
||||
|
||||
def build_model(self, input_shape: tuple) -> keras.Model:
|
||||
"""
|
||||
Build the neural network model.
|
||||
|
||||
Args:
|
||||
input_shape: Shape of input data (lookback, features)
|
||||
|
||||
Returns:
|
||||
Compiled Keras model
|
||||
"""
|
||||
model = keras.Sequential([
|
||||
layers.LSTM(128, return_sequences=True, input_shape=input_shape),
|
||||
layers.Dropout(0.3),
|
||||
layers.LSTM(64, return_sequences=True),
|
||||
layers.Dropout(0.3),
|
||||
layers.LSTM(32),
|
||||
layers.Dropout(0.3),
|
||||
layers.Dense(32, activation='relu'),
|
||||
layers.Dense(16, activation='relu'),
|
||||
layers.Dense(1) # Predict price change percentage
|
||||
])
|
||||
|
||||
model.compile(
|
||||
optimizer=keras.optimizers.Adam(learning_rate=0.0005), # Lower learning rate for stability
|
||||
loss='mse',
|
||||
metrics=['mae']
|
||||
)
|
||||
|
||||
return model
|
||||
|
||||
def train(self, epochs: int = 50, batch_size: int = 32,
|
||||
validation_split: float = 0.2, verbose: int = 1):
|
||||
"""
|
||||
Train the model.
|
||||
|
||||
Args:
|
||||
epochs: Number of training epochs
|
||||
batch_size: Batch size for training
|
||||
validation_split: Fraction of data to use for validation
|
||||
verbose: Verbosity level
|
||||
"""
|
||||
# Fetch data (last 2 years)
|
||||
end_date = datetime.now()
|
||||
start_date = end_date - timedelta(days=730)
|
||||
|
||||
df = self.fetch_data(start_date, end_date)
|
||||
feature_df = self.prepare_features(df)
|
||||
|
||||
# Prepare data - align target with features after dropna
|
||||
feature_data = feature_df.values
|
||||
|
||||
# Get target data aligned with feature_df (after dropna)
|
||||
# Use .loc to align by index, then convert to values
|
||||
close_prices = df.loc[feature_df.index, 'close'].values
|
||||
|
||||
# Predict price change percentage instead of absolute price (more stable)
|
||||
# Calculate future price change: (future_price - current_price) / current_price
|
||||
target_data = []
|
||||
for i in range(len(close_prices)):
|
||||
if i + self.prediction_horizon < len(close_prices):
|
||||
current_price = close_prices[i]
|
||||
future_price = close_prices[i + self.prediction_horizon]
|
||||
price_change_pct = (future_price - current_price) / current_price if current_price > 0 else 0.0
|
||||
target_data.append(price_change_pct)
|
||||
else:
|
||||
target_data.append(0.0)
|
||||
target_data = np.array(target_data)
|
||||
|
||||
# Scale features
|
||||
feature_data_scaled = self.scaler.fit_transform(feature_data)
|
||||
|
||||
# Create sequences
|
||||
X, y = self.create_sequences(feature_data_scaled, target_data)
|
||||
|
||||
# Split data
|
||||
X_train, X_test, y_train, y_test = train_test_split(
|
||||
X, y, test_size=validation_split, shuffle=False
|
||||
)
|
||||
|
||||
print(f"\nTraining data shape: {X_train.shape}")
|
||||
print(f"Validation data shape: {X_test.shape}")
|
||||
|
||||
# Build model - use actual feature count from data
|
||||
actual_num_features = X_train.shape[2]
|
||||
print(f"Actual number of features: {actual_num_features}")
|
||||
self.model = self.build_model((X_train.shape[1], X_train.shape[2]))
|
||||
|
||||
print("\nModel architecture:")
|
||||
self.model.summary()
|
||||
|
||||
# Train model
|
||||
print("\nTraining model...")
|
||||
history = self.model.fit(
|
||||
X_train, y_train,
|
||||
batch_size=batch_size,
|
||||
epochs=epochs,
|
||||
validation_data=(X_test, y_test),
|
||||
verbose=verbose,
|
||||
callbacks=[
|
||||
keras.callbacks.EarlyStopping(
|
||||
monitor='val_loss',
|
||||
patience=10,
|
||||
restore_best_weights=True
|
||||
),
|
||||
keras.callbacks.ReduceLROnPlateau(
|
||||
monitor='val_loss',
|
||||
factor=0.5,
|
||||
patience=5,
|
||||
min_lr=0.0001
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
# Evaluate
|
||||
train_loss = self.model.evaluate(X_train, y_train, verbose=0)
|
||||
test_loss = self.model.evaluate(X_test, y_test, verbose=0)
|
||||
|
||||
print(f"\nTraining Loss: {train_loss[0]:.4f}, MAE: {train_loss[1]:.4f}")
|
||||
print(f"Validation Loss: {test_loss[0]:.4f}, MAE: {test_loss[1]:.4f}")
|
||||
|
||||
return history
|
||||
|
||||
def export_to_onnx(self, output_path: str):
|
||||
"""
|
||||
Export the trained model to ONNX format.
|
||||
|
||||
Args:
|
||||
output_path: Path to save ONNX model
|
||||
"""
|
||||
if self.model is None:
|
||||
raise ValueError("Model must be trained before exporting")
|
||||
|
||||
print(f"\nExporting model to ONNX format: {output_path}")
|
||||
|
||||
# Get actual number of features from model input shape
|
||||
# The model was built with the actual feature count during training
|
||||
if self.model is not None and hasattr(self.model, 'input_shape'):
|
||||
num_features = self.model.input_shape[2] if len(self.model.input_shape) > 2 else self.model.input_shape[1]
|
||||
else:
|
||||
# Fallback calculation
|
||||
# Base features: open, high, low, close, tick_volume (5)
|
||||
# Added features: rsi, ema_20, ema_50, atr, price_change, high_low_ratio, volume_ma, volume_ratio (8)
|
||||
num_features = len(self.features) + 8
|
||||
|
||||
print(f"Using {num_features} features for ONNX export")
|
||||
|
||||
# Create input signature
|
||||
input_shape = (None, self.lookback, num_features)
|
||||
spec = (tf.TensorSpec(input_shape, tf.float32, name="input"),)
|
||||
|
||||
# Convert to ONNX using tf2onnx
|
||||
# Workaround for tf2onnx 1.16.1 issue with Sequential models (GitHub issue #2319)
|
||||
# Fix: Add output_names attribute to Sequential model if missing
|
||||
if hasattr(self.model, 'output_names') is False:
|
||||
# Workaround: Create a wrapper or use functional API
|
||||
try:
|
||||
# Try to get output names from model outputs
|
||||
if hasattr(self.model, 'outputs') and self.model.outputs:
|
||||
self.model.output_names = [f'output_{i}' for i in range(len(self.model.outputs))]
|
||||
else:
|
||||
self.model.output_names = ['output']
|
||||
except:
|
||||
pass
|
||||
|
||||
# Create input signature tuple
|
||||
spec = (tf.TensorSpec((None, self.lookback, num_features), tf.float32, name="input"),)
|
||||
|
||||
# Skip direct Sequential conversion - use Functional API directly
|
||||
# This avoids the 'output_names' attribute error
|
||||
try:
|
||||
# Method 1: Convert Sequential to Functional API model (more reliable)
|
||||
print("Converting Sequential model to Functional API...")
|
||||
|
||||
# Create functional model from Sequential
|
||||
input_layer = keras.Input(shape=(self.lookback, num_features), name="input")
|
||||
x = input_layer
|
||||
|
||||
# Rebuild model as functional
|
||||
for layer in self.model.layers:
|
||||
x = layer(x)
|
||||
|
||||
functional_model = keras.Model(inputs=input_layer, outputs=x)
|
||||
|
||||
# Convert functional model
|
||||
onnx_model_proto, _ = tf2onnx.convert.from_keras(
|
||||
functional_model,
|
||||
input_signature=spec,
|
||||
opset=13
|
||||
)
|
||||
|
||||
onnx.save_model(onnx_model_proto, output_path)
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
|
||||
except Exception as e1:
|
||||
# Method 2: Use concrete function approach
|
||||
try:
|
||||
print("Trying concrete function method...")
|
||||
|
||||
# Create concrete function
|
||||
input_spec = tf.TensorSpec(shape=(None, self.lookback, num_features), dtype=tf.float32)
|
||||
|
||||
@tf.function
|
||||
def model_func(x):
|
||||
return self.model(x)
|
||||
|
||||
# Get concrete function
|
||||
concrete_func = model_func.get_concrete_function(input_spec)
|
||||
|
||||
# Convert with input_signature as list
|
||||
input_signature_list = [input_spec]
|
||||
onnx_model_proto, _ = tf2onnx.convert.from_function(
|
||||
concrete_func,
|
||||
input_signature=input_signature_list,
|
||||
opset=13
|
||||
)
|
||||
|
||||
onnx.save_model(onnx_model_proto, output_path)
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
|
||||
except Exception as e2:
|
||||
# Method 3: Try alternative conversion method
|
||||
try:
|
||||
print("Trying alternative conversion method...")
|
||||
|
||||
# Save model first, then convert
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Save as .keras format
|
||||
keras_path = os.path.join(tmpdir, "model.keras")
|
||||
self.model.save(keras_path)
|
||||
|
||||
# Load and convert
|
||||
loaded_model = keras.models.load_model(keras_path)
|
||||
|
||||
# Convert Sequential to Functional
|
||||
input_layer = keras.Input(shape=(self.lookback, num_features), name="input")
|
||||
x = input_layer
|
||||
for layer in loaded_model.layers:
|
||||
x = layer(x)
|
||||
functional_model = keras.Model(inputs=input_layer, outputs=x)
|
||||
|
||||
# Try conversion again
|
||||
onnx_model_proto, _ = tf2onnx.convert.from_keras(
|
||||
functional_model,
|
||||
input_signature=spec,
|
||||
opset=13
|
||||
)
|
||||
|
||||
onnx.save_model(onnx_model_proto, output_path)
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
|
||||
except Exception as e3:
|
||||
raise RuntimeError(
|
||||
f"Failed to export ONNX model.\n"
|
||||
f"Error 1 (functional): {str(e1)[:200]}\n"
|
||||
f"Error 2 (concrete): {str(e2)[:200]}\n"
|
||||
f"Error 3 (alternative): {str(e3)[:200]}\n\n"
|
||||
f"Please try upgrading tf2onnx: pip install --upgrade tf2onnx"
|
||||
)
|
||||
|
||||
# Verify ONNX model
|
||||
try:
|
||||
onnx_model = onnx.load(output_path)
|
||||
onnx.checker.check_model(onnx_model)
|
||||
print("ONNX model validation passed")
|
||||
except Exception as e:
|
||||
print(f"⚠ ONNX model validation warning: {e}")
|
||||
|
||||
def cleanup(self):
|
||||
"""Clean up MT5 connection."""
|
||||
mt5.shutdown()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(description='Train ONNX model for MT5 price prediction')
|
||||
parser.add_argument('--symbol', type=str, default='XAUUSD', help='Trading symbol')
|
||||
parser.add_argument('--timeframe', type=str, default='H1',
|
||||
choices=['M1', 'M5', 'M15', 'M30', 'H1', 'H4', 'D1'],
|
||||
help='Timeframe')
|
||||
parser.add_argument('--lookback', type=int, default=60,
|
||||
help='Number of bars to look back')
|
||||
parser.add_argument('--epochs', type=int, default=50, help='Training epochs')
|
||||
parser.add_argument('--batch-size', type=int, default=32, help='Batch size')
|
||||
parser.add_argument('--output', type=str, default='models',
|
||||
help='Output directory for ONNX model')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Convert timeframe string to MT5 constant
|
||||
timeframe_map = {
|
||||
'M1': mt5.TIMEFRAME_M1,
|
||||
'M5': mt5.TIMEFRAME_M5,
|
||||
'M15': mt5.TIMEFRAME_M15,
|
||||
'M30': mt5.TIMEFRAME_M30,
|
||||
'H1': mt5.TIMEFRAME_H1,
|
||||
'H4': mt5.TIMEFRAME_H4,
|
||||
'D1': mt5.TIMEFRAME_D1
|
||||
}
|
||||
timeframe = timeframe_map[args.timeframe]
|
||||
|
||||
# Create output directory
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
# Create trainer
|
||||
trainer = ONNXModelTrainer(
|
||||
symbol=args.symbol,
|
||||
timeframe=timeframe,
|
||||
lookback=args.lookback
|
||||
)
|
||||
|
||||
try:
|
||||
# Train model
|
||||
trainer.train(epochs=args.epochs, batch_size=args.batch_size)
|
||||
|
||||
# Export to ONNX
|
||||
model_name = f"{args.symbol}_{args.timeframe}_model.onnx"
|
||||
output_path = os.path.join(args.output, model_name)
|
||||
trainer.export_to_onnx(output_path)
|
||||
|
||||
# Save scaler for consistent normalization
|
||||
scaler_path = os.path.join(args.output, f"{args.symbol}_{args.timeframe}_scaler.pkl")
|
||||
import pickle
|
||||
with open(scaler_path, 'wb') as f:
|
||||
pickle.dump(trainer.scaler, f)
|
||||
print(f"Scaler saved to: {scaler_path}")
|
||||
print(f" Use this with predict_with_onnx.py for consistent normalization")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("Training completed successfully!")
|
||||
print(f"ONNX model saved to: {output_path}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nError: {e}")
|
||||
sys.exit(1)
|
||||
finally:
|
||||
trainer.cleanup()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user