Initial commit

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Mohammad Aghdam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,226 @@
+# AlphaFlow ML & DL Trading Bot Project
+
+A comprehensive **machine learning and deep learning trading framework** that covers the entire workflow:
+
+1. **Data loading** from MetaTrader 5
+2. **Feature engineering** (technical indicators, custom features, labeling)
+3. **Model training** (RandomForest, XGBoost, LightGBM, deep learning models, etc.)
+4. **Hyperparameter tuning** (RandomizedSearchCV, GridSearchCV or Optuna)
+5. **Time-based / walk-forward cross-validation**
+6. **Backtesting** (VectorBT or simple custom code)
+7. **Live trading** integration with MetaTrader 5
+
+### Supported Labeling Strategies:
+- **Regression** on next-bar returns
+- **Multi-bar classification**
+- **Double-barrier labeling** (López de Prado style)
+- **Regime detection** (simple up/down/sideways approach)
+
+This project provides a flexible **template** for you to **create and add your own** custom labeling functions or feature engineering steps, allowing you to experiment with new ideas and strategies.
+
+## Table of Contents
+1. [Features](#features)
+2. [Repository Structure](#repository-structure)
+3. [Setup & Installation](#setup--installation)
+4. [Usage](#usage)
+    - Backtesting Notebooks
+    - Live Trading Scripts
+5. [Key Modules](#key-modules)
+6. [Extending the Project](#extending-the-project)
+7. [Disclaimer](#disclaimer)
+8. [License](#license)
+
+## Features
+- **MetaTrader 5** data retrieval (`data_loader.py`)
+- **TA** library for feature engineering (`ta.add_all_ta_features`)
+- Multiple **labeling methods**: next-bar, multi-bar, double-barrier, regime detection, etc.
+- **Time-based** or **walk-forward** cross-validation to avoid data leakage
+- **RandomizedSearchCV** or **GridSearchCV** for hyperparameter tuning
+- **VectorBT** or custom backtesting scripts for performance evaluation
+- **Live trading** scripts with real-time MetaTrader 5 order sending
+
+## Repository Structure
+```bash
+# ML Bot Trading Repository Structure
+
+ml_bot_trading/
+├── data/
+│   ├── data_loader.py  # MetaTrader 5 data retrieval
+│
+├── features/
+│   ├── feature_engineering.py  # Technical indicators, custom features
+│   ├── labeling.py  # Labeling methods: next-bar, multi-bar, double-barrier, regime detection
+│
+├── models/
+│   ├── model_training.py  # Model selection, hyperparam tuning
+│   ├── saved_models/  # Folder for .pkl pipelines (best_rf_pipeline.pkl, etc.)
+│
+├── backtests/
+│   ├── simple_backtest.py  # Simple Pythonic backtest logic
+│   ├── vectorbt_backtest.py  # VectorBT-based backtesting template
+│
+├── live_trading/
+│   ├── regression_returns.py  # Live trading script for regression returns
+│   ├── multi_bar.py  # Live trading script for multi-bar classification
+│   ├── double_barrier.py  # Live trading script for double-barrier labeling
+│   ├── regime_detection.py  # Live trading script for regime detection
+│
+├── notebooks/
+│   ├── dl_notebooks/
+│   │   ├── 00_eda_visualization.ipynb
+│   │   ├── 01_backtests_regression_returns_dl.ipynb
+│   │   ├── 01_live_trading_regression_returns_dl.ipynb
+│   │   ├── 02_time_series_arima_sarima_var_lstmprice.ipynb
+│   │
+│   ├── eda_notebooks/
+│   │   ├── 00_eda_visualization.ipynb
+│   │
+│   ├── ml_notebooks/
+│   │   ├── 01_backtests_regression_returns.ipynb
+│   │   ├── 01_live_trading_regression_returns.ipynb
+│   │   ├── 02_backtests_multi_bar_classification.ipynb
+│   │   ├── 02_live_trading_multi_bar_classification.ipynb
+│   │   ├── 03_backtests_double_barrier_labeling.ipynb
+│   │   ├── 03_live_trading_double_barrier_labeling.ipynb
+│   │   ├── 04_backtests_regime_detection.ipynb
+│   │   ├── 04_live_trading_regime_detection.ipynb
+│
+├── requirements.txt
+├── README.md
+```
+
+## Setup & Installation
+
+### 1. Clone this repository:
+```bash
+git clone https://github.com/maghdam/AlphaFlow-Trading-Bot.git
+cd ml_bot_trading
+```
+
+### 2. Create and activate a Python environment (conda or venv):
+```bash
+conda create -n ml_trading python>=3.9
+conda activate ml_trading
+```
+
+### 3. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+- Make sure you have **MetaTrader5** installed [IC Markets MT5](https://www.icmarkets.com/global/en/forex-trading-platform-metatrader/metatrader-5).
+
+### 4. (Optional) Install Jupyter Notebook:
+```bash
+pip install jupyter
+```
+
+## Usage
+### Backtesting Notebooks
+1. Navigate to `ml_notebooks/` or `dl_notebooks/`, pick a relevant file (e.g., `02_backtests_multi_bar_classification.ipynb`), and run it:
+   ```bash
+   jupyter notebook
+   ```
+2. Inside the notebook, you can see how we do:
+   - Feature engineering
+   - Labeling
+   - Walk-forward splits
+   - Train & tune
+   - VectorBT or custom backtesting
+
+### Live Trading Scripts
+1. Navigate to `ml_notebooks/` or `dl_notebooks/`, pick a relevant live trading file (e.g., 2_live_trading_multi_bar_classification.ipynb), or go to `live_trading/` folder and pick the script for your labeling approach:
+   - `regression_returns.py`
+   - `multi_bar.py`
+   - `double_barrier.py`
+   - `regime_detection.py`
+2. Adjust **MetaTrader 5 credentials** (login, server, password) in the script.
+3. Run from terminal:
+   ```bash
+   python live_trading/multi_bar.py.py
+   ```
+4. The script will:
+   - Load the pipeline (e.g., `best_rf_mb_pipeline.pkl`)
+   - Fetch new bars from MetaTrader 5
+   - Predict SHIFTED classes `[0, 1, 2]` => SHIFT back to `[-1, 0, +1]`
+   - Place orders if signals = ±1
+
+## Key Modules
+- **`data/data_loader.py`**: Connects to MetaTrader 5, fetches bars with `copy_rates_from_pos`.
+- **`features/feature_engineering.py`**: Uses the **TA** library and additional custom features (spreads, autocorrelation, etc.).
+- **`features/labeling.py`**:
+  - `calculate_future_return(...)`
+  - `create_labels_multi_bar(...)`
+  - `create_labels_double_barrier(...)`
+  - `create_labels_regime_detection(...)`
+- **`models/model_training.py`**:
+  - `select_features_rf_reg(...)`
+  - Time-based splits, random/grid search for hyperparams.
+- **`backtests/`**:
+  - `simple_backtest.py` or `vectorbt_backtest.py`
+- **`live_trading/`**:
+  - Each script loads a pipeline (`.pkl`), connects to MT5, and places trades based on predictions.
+
+## Extending the Project
+- **Add your own label**: Create a new function in `features/labeling.py` (e.g. `create_labels_custom(...)` that returns a new column with `[-1, 0, +1]` (or your custom classes)).
+- **Add your own features**: Implement them in `features/feature_engineering.py` or create a new file.
+- **Train a new model**: Adapt `models/model_training.py` or your notebooks to handle new classifiers/regressors.
+- **Explore new backtest approaches**: Either integrate with `vectorbt` in a notebook or write a custom `.py` in `backtests/`.
+
+## Disclaimer
+We share this code for **learning and development/research purposes only**. Nothing herein constitutes financial advice or a recommendation to trade real money. **Trading involves substantial risk.** Always do your own due diligence, consult professionals, and only risk capital you can afford to lose.
+
+## License
+This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.
+
+
+## Backtest Results - US30 - H4
+
+
+```
+Loaded best classification model from 'best_rf_mb_pipeline.pkl'
+
+Out-of-Sample Accuracy: 0.5439
+
+Running Full Backtest on the Last 5000 Bars...
+```
+
+```
+Full Backtest Results:
+Accuracy=0.54, Return=0.30%, Sharpe=1.21
+Start                               2021-12-01 16:00:00
+End                                 2025-02-28 00:00:00
+Period                                832 days 12:00:00
+Start Value                                     10000.0
+End Value                                  12971.402323
+Total Return [%]                              29.714023
+Benchmark Return [%]                          24.515943
+Max Gross Exposure [%]                            100.0
+Total Fees Paid                              236.291366
+Max Drawdown [%]                              13.645737
+Max Drawdown Duration                 295 days 04:00:00
+Total Trades                                         56
+Total Closed Trades                                  56
+Total Open Trades                                     0
+Open Trade PnL                                      0.0
+Win Rate [%]                                  60.714286
+Best Trade [%]                                13.156291
+Worst Trade [%]                               -3.463323
+Avg Winning Trade [%]                          1.517335
+Avg Losing Trade [%]                          -1.094661
+Avg Winning Trade Duration    7 days 03:03:31.764705882
+Avg Losing Trade Duration               1 days 00:00:00
+Profit Factor                                  2.150809
+Expectancy                                    53.060756
+Sharpe Ratio                                    1.20547
+Calmar Ratio                                   0.885441
+Omega Ratio                                    1.156419
+Sortino Ratio                                  1.793852
+dtype: object
+
+```
+
+![Fold 1 Performance](images/backtest.png)
+
+
+
+

backtests/simple_backtest.py

@@ -0,0 +1,86 @@
+# simple_backtest.py
+
+import numpy as np
+import pandas as pd
+
+# backtests/simple_backtest.py
+
+
+def simulate_trading(signals, df, cost=0.0002):
+    """
+    A simple backtest function that simulates trading based on signals (+1/-1/0).
+    
+    Parameters
+    ----------
+    signals : array-like of int
+        Sequence of +1, -1, or 0 indicating long, short, or flat.
+    df : pd.DataFrame
+        Must contain at least a 'close' column with the same length as 'signals'.
+    cost : float
+        Transaction cost fraction per position change (e.g. 0.0002 = 0.02%).
+
+    Returns
+    -------
+    daily_returns : np.array
+        The sequence of returns from the strategy for each bar.
+    total_return : float
+        The total percentage return (e.g., 10.0 = +10%).
+    """
+    if len(signals) != len(df):
+        raise ValueError("Length of signals must match length of df.")
+    if 'close' not in df.columns:
+        raise ValueError("df must contain a 'close' column.")
+    
+    # 1) Calculate price returns bar to bar
+    df['price_return'] = df['close'].pct_change().fillna(0)
+    
+    # 2) Strategy returns = signals * price_return
+    #    But we must subtract cost each time we change position.
+    #    If signals[i] != signals[i-1], we pay cost.
+    daily_returns = np.zeros(len(signals))
+    
+    prev_signal = 0
+    for i in range(len(signals)):
+        # Base return from price movement
+        daily_returns[i] = signals[i] * df['price_return'].iloc[i]
+        
+        # Check if position changed from previous bar
+        if i > 0 and signals[i] != prev_signal:
+            # Subtract cost
+            daily_returns[i] -= cost
+        prev_signal = signals[i]
+    
+    # 3) Compute total return in percent
+    cumulative_return = (1 + daily_returns).prod() - 1
+    total_return = cumulative_return * 100.0
+    
+    return daily_returns, total_return
+
+
+def calculate_sharpe_ratio(returns, risk_free=0.0):
+    """
+    Calculates a simple Sharpe ratio for a series of returns.
+    
+    Parameters
+    ----------
+    returns : list or np.array
+        A sequence of returns per bar/day.
+    risk_free : float, optional
+        Risk-free rate per bar/day, default is 0.0 (no risk-free rate).
+    
+    Returns
+    -------
+    float
+        The Sharpe ratio = (mean(returns - risk_free)) / std(returns).
+        If std is zero, returns np.nan.
+    """
+    returns = np.array(returns)
+    excess_returns = returns - risk_free
+    avg_excess = np.mean(excess_returns)
+    std_excess = np.std(excess_returns)
+
+    if std_excess == 0:
+        return np.nan
+    
+    sharpe = avg_excess / std_excess
+    return sharpe

backtests/vectorbt_backtest.py

@@ -0,0 +1,80 @@
+# vectorbt_backtest.py
+
+import numpy as np
+import pandas as pd
+import vectorbt as vbt
+
+def run_vectorbt_backtest(
+    model,
+    X,
+    selected_features,
+    data,
+    scaler,
+    init_cash=10000,
+    freq='4H',
+    threshold=0.0
+):
+    """
+    Runs a vectorbt backtest for a given pre-trained model.
+
+    Parameters
+    ----------
+    model : fitted scikit-learn model
+        Already fitted model (e.g. RandomForestRegressor).
+    X : pd.DataFrame
+        The full feature DataFrame (or the portion you want to backtest).
+    selected_features : list
+        List of feature names used by the model.
+    data : pd.DataFrame
+        Original DataFrame containing at least a 'close' column.
+    scaler : fitted scaler
+        The StandardScaler (or other) used to scale features.
+    init_cash : float
+        Starting capital for the backtest.
+    freq : str
+        Frequency for vectorbt (e.g. '4H', '1D').
+    threshold : float
+        Minimum absolute predicted return to place a trade (optional).
+
+    Returns
+    -------
+    pf : vbt.Portfolio
+        The resulting vectorbt portfolio object.
+    """
+    # 1) Subset X to the selected features
+    X_sel = X[selected_features]
+
+    # 2) Scale
+    X_scaled = scaler.transform(X_sel)
+
+    # 3) Generate predictions
+    preds = model.predict(X_scaled)
+
+    # 4) Convert predictions to signals
+    #    Optionally use threshold to reduce whipsaws
+    if threshold > 0.0:
+        signals = np.where(preds > threshold, 1, np.where(preds < -threshold, -1, 0))
+    else:
+        signals = np.sign(preds)
+
+    # 5) Align signals with close prices
+    close_prices = data.loc[X_sel.index, "close"]
+    # If signals is shorter or the same length
+    if len(signals) < len(close_prices):
+        # Pad signals with 0 if needed
+        signals = np.append(signals, [0]*(len(close_prices)-len(signals)))
+
+    signals_s = pd.Series(signals, index=close_prices.index)
+    # Align if any missing indexes
+    close_prices, signals_s = close_prices.align(signals_s, join="inner", axis=0)
+
+    # 6) Run vectorbt Portfolio
+    pf = vbt.Portfolio.from_signals(
+        close_prices,
+        entries=signals_s > 0,
+        exits=signals_s < 0,
+        init_cash=init_cash,
+        freq=freq
+    )
+
+    return pf

data/data_loader.py

@@ -0,0 +1,31 @@
+# data_loader.py
+
+import MetaTrader5 as mt5
+import pandas as pd
+import numpy as np
+from datetime import datetime
+
+def get_data_mt5(symbol: str, n_bars: int, timeframe, start_pos=None) -> pd.DataFrame:
+    """
+    Fetch historical data from MetaTrader 5.
+    
+    - `symbol`: Trading instrument (e.g., "BTCUSD").
+    - `n_bars`: Number of bars to retrieve.
+    - `timeframe`: MT5 timeframe (e.g., mt5.TIMEFRAME_H1).
+    - `start_pos`: Offset from the most recent bar (default `None` for live trading).
+    
+    If `start_pos` is `None`, fetches the latest `n_bars` (useful for live trading).
+    If `start_pos` is given, fetches `n_bars` from that historical position (useful for backtesting).
+    """
+    if start_pos is None:
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, n_bars)  # Latest n_bars for live trading
+    else:
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, start_pos, n_bars)  # Historical data for backtesting
+
+    if rates is None:
+        raise ValueError(f"Could not retrieve data for {symbol}")
+
+    df = pd.DataFrame(rates)
+    df['time'] = pd.to_datetime(df['time'], unit='s')
+    df.set_index('time', inplace=True)
+    return df

features/feature_engineering.py

@@ -0,0 +1,381 @@
+# feature_engineering.py
+
+import numpy as np
+import pandas as pd
+import math
+import ta
+from statsmodels.tsa.stattools import adfuller
+from scipy.fftpack import fft
+from sklearn.preprocessing import StandardScaler
+
+
+
+
+
+# --------------------------------------------------------------------
+# 1) TA-LIB FEATURES (add_all_ta_features)
+# --------------------------------------------------------------------
+def add_all_ta_features(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Adds a wide range of technical analysis indicators to the DataFrame
+    using the 'ta' library. Modifies the DataFrame in place.
+    """
+    df = ta.add_all_ta_features(
+        df, open="open", high="high", low="low", close="close", volume="tick_volume", fillna=True
+    )
+    return df
+
+
+
+def create_custom_feature(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Example custom feature. For instance, a rolling mean of the close price.
+    """
+    df["rolling_mean_10"] = df["close"].rolling(window=10).mean()
+    return df
+
+# --------------------------------------------------------------------
+# 2) MISCELLANEOUS FEATURES
+# --------------------------------------------------------------------
+def spread(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Calculates the spread between 'high' and 'low' columns.
+    """
+    df_copy = df.copy()
+    df_copy["spread"] = df_copy["high"] - df_copy["low"]
+    return df_copy
+
+def auto_corr_multi(df: pd.DataFrame, col: str, n: int = 50, lags: list = [1, 3, 5, 10]) -> pd.DataFrame:
+    """
+    Computes rolling autocorrelation for multiple lags.
+    """
+    df_copy = df.copy()
+    for lag in lags:
+        df_copy[f"autocorr_{lag}"] = (
+            df_copy[col]
+            .rolling(window=n, min_periods=n)
+            .apply(lambda x: x.autocorr(lag=lag), raw=False)
+        )
+    return df_copy
+
+
+def candle_information(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Adds candle-specific features:
+      - candle_way
+      - fill
+      - amplitude
+    """
+    df_copy = df.copy()
+    df_copy["candle_way"] = 0
+    df_copy.loc[df_copy["close"] > df_copy["open"], "candle_way"] = 1
+    
+    df_copy["fill"] = (
+        np.abs(df_copy["close"] - df_copy["open"]) 
+        / (df_copy["high"] - df_copy["low"] + 1e-5)
+    )
+    df_copy["amplitude"] = (
+        np.abs(df_copy["close"] - df_copy["open"]) 
+        / (df_copy["open"] + 1e-5)
+    )
+    return df_copy
+
+def log_transform(df: pd.DataFrame, col: str, n: int) -> pd.DataFrame:
+    """
+    Log-transform a column + compute % change over 'n' bars.
+    """
+    df_copy = df.copy()
+    df_copy[f"log_{col}"] = np.log(df_copy[col])
+    df_copy[f"ret_log_{n}"] = df_copy[f"log_{col}"].pct_change(periods=n)
+    return df_copy
+
+def mathematical_derivatives(df: pd.DataFrame, col: str) -> pd.DataFrame:
+    """
+    Adds 'velocity' and 'acceleration' for a given column.
+    """
+    df_copy = df.copy()
+    df_copy["velocity"] = df_copy[col].diff()
+    df_copy["acceleration"] = df_copy["velocity"].diff()
+    return df_copy
+
+# --------------------------------------------------------------------
+# 3) VOLATILITY ESTIMATORS
+# --------------------------------------------------------------------
+def parkinson_estimator(window: pd.DataFrame) -> float:
+    n = len(window)
+    if n < 1:
+        return np.nan
+    sum_sq = np.sum(np.log(window['high'] / window['low']) ** 2)
+    return math.sqrt(sum_sq / (4 * math.log(2) * n))
+
+def moving_parkinson_estimator(df: pd.DataFrame, window_size: int = 30) -> pd.DataFrame:
+    df_copy = df.copy()
+    rolling_vol = pd.Series(dtype="float64", index=df_copy.index)
+    for i in range(window_size, len(df_copy)):
+        w = df_copy.iloc[i - window_size : i]
+        rolling_vol.iloc[i] = parkinson_estimator(w)
+    df_copy["rolling_volatility_parkinson"] = rolling_vol
+    return df_copy
+
+def yang_zhang_estimator(window: pd.DataFrame) -> float:
+    n = len(window)
+    if n < 1:
+        return np.nan
+    term1 = np.log(window['high'] / window['low']) ** 2
+    term2 = np.log(window['close'] / window['open']) ** 2
+    return math.sqrt(np.mean(term1 + term2))
+
+def moving_yang_zhang_estimator(df: pd.DataFrame, window_size: int = 30) -> pd.DataFrame:
+    df_copy = df.copy()
+    rolling_vol = pd.Series(dtype="float64", index=df_copy.index)
+    for i in range(window_size, len(df_copy)):
+        w = df_copy.iloc[i - window_size : i]
+        rolling_vol.iloc[i] = yang_zhang_estimator(w)
+    df_copy["rolling_volatility_yang_zhang"] = rolling_vol
+    return df_copy
+
+# --------------------------------------------------------------------
+# 4) MARKET REGIME / DC EVENTS
+# --------------------------------------------------------------------
+def dc_event(P: float, Pext: float, threshold: float) -> int:
+    dc = 0
+    var = (P - Pext) / Pext
+    if var >= threshold:
+        dc = 1
+    elif var <= -threshold:
+        dc = -1
+    return dc
+
+def calculate_dc(df: pd.DataFrame, threshold: float = 0.01) -> tuple:
+    df_copy = df.copy()
+    prices = df_copy['close'].values
+    dc_events_up, dc_events_down = [], []
+    Pext = prices[0]
+    direction = 0
+    for i in range(1, len(prices)):
+        P = prices[i]
+        dc_flag = dc_event(P, Pext, threshold)
+        if dc_flag == 1:
+            dc_events_up.append(i)
+            direction = 1
+            Pext = P
+        elif dc_flag == -1:
+            dc_events_down.append(i)
+            direction = -1
+            Pext = P
+        else:
+            if direction == 1 and P > Pext:
+                Pext = P
+            elif direction == -1 and P < Pext:
+                Pext = P
+    return dc_events_up, dc_events_down
+
+def calculate_trend(dc_events_up: list, dc_events_down: list, df: pd.DataFrame):
+    trend_events_down = []
+    trend_events_up = []
+    trend_events_down.extend(sorted(dc_events_down))
+    trend_events_up.extend(sorted(dc_events_up))
+    return trend_events_down, trend_events_up
+
+def market_regime_dc(df: pd.DataFrame, threshold: float = 0.01) -> pd.DataFrame:
+    df_copy = df.copy()
+    dc_up, dc_down = calculate_dc(df_copy, threshold=threshold)
+    t_down, t_up = calculate_trend(dc_up, dc_down, df_copy)
+    df_copy['market_regime'] = np.nan
+    df_copy.loc[t_up, 'market_regime'] = 1
+    df_copy.loc[t_down, 'market_regime'] = 0
+    df_copy['market_regime'] = df_copy['market_regime'].ffill().bfill()
+    return df_copy
+
+def kama_market_regime(df: pd.DataFrame, col: str = 'close', n1: int = 10, n2: int = 30) -> pd.DataFrame:
+    df_copy = df.copy()
+    short_kama = df_copy[col].ewm(span=n1, adjust=False).mean()
+    long_kama  = df_copy[col].ewm(span=n2, adjust=False).mean()
+    df_copy['kama_diff'] = short_kama - long_kama
+    df_copy['kama_trend'] = (df_copy['kama_diff'] >= 0).astype(int)
+    return df_copy
+
+# --------------------------------------------------------------------
+# 5) GAP & DISPLACEMENT
+# --------------------------------------------------------------------
+def gap_detection(df: pd.DataFrame, lookback: int = 1) -> pd.DataFrame:
+    df_copy = df.copy()
+    df_copy['Bullish_gap_inf'] = np.nan
+    df_copy['Bullish_gap_sup'] = np.nan
+    df_copy['Bullish_gap_size'] = np.nan
+    df_copy['Bearish_gap_inf'] = np.nan
+    df_copy['Bearish_gap_sup'] = np.nan
+    df_copy['Bearish_gap_size'] = np.nan
+    for i in range(lookback, len(df_copy)):
+        prev_high = df_copy['high'].iloc[i - lookback]
+        prev_low = df_copy['low'].iloc[i - lookback]
+        curr_high = df_copy['high'].iloc[i]
+        curr_low = df_copy['low'].iloc[i]
+        if curr_low > prev_high:
+            df_copy.at[df_copy.index[i], 'Bullish_gap_inf'] = prev_high
+            df_copy.at[df_copy.index[i], 'Bullish_gap_sup'] = curr_low
+            df_copy.at[df_copy.index[i], 'Bullish_gap_size'] = curr_low - prev_high
+        if curr_high < prev_low:
+            df_copy.at[df_copy.index[i], 'Bearish_gap_inf'] = curr_high
+            df_copy.at[df_copy.index[i], 'Bearish_gap_sup'] = prev_low
+            df_copy.at[df_copy.index[i], 'Bearish_gap_size'] = prev_low - curr_high
+    return df_copy
+
+def displacement_detection(
+    df: pd.DataFrame, 
+    type_range: str = 'standard', 
+    strenght: float = 3.0, 
+    period: int = 20
+) -> pd.DataFrame:
+    df_copy = df.copy()
+    if type_range == 'standard':
+        df_copy['candle_range'] = np.abs(df_copy['close'] - df_copy['open'])
+    elif type_range == 'extrem':
+        df_copy['candle_range'] = np.abs(df_copy['high'] - df_copy['low'])
+    else:
+        raise ValueError("Invalid 'type_range'. Use 'standard' or 'extrem'.")
+
+    df_copy['Variation'] = np.abs(df_copy['close'] / df_copy['open'] - 1)
+    df_copy['STD'] = df_copy['candle_range'].rolling(period).std()
+    df_copy['displacement'] = 0
+    mask = df_copy['candle_range'] > strenght * df_copy['STD']
+    df_copy.loc[mask, 'displacement'] = 1
+    df_copy['red_displacement'] = (
+        df_copy['displacement'] & df_copy['displacement'].shift(1).fillna(0)
+    ).astype(int)
+    return df_copy
+
+# --------------------------------------------------------------------
+# 6) ROLLING ADF (Stationarity)
+# --------------------------------------------------------------------
+def rolling_adf_with_flag(df: pd.DataFrame, col: str = 'close', window_size: int = 50, p_value_threshold=0.05) -> pd.DataFrame:
+    """
+    Computes rolling ADF test and adds a stationarity flag (1=stationary, 0=non-stationary).
+    """
+    df_copy = df.copy()
+    adf_stat = pd.Series(dtype="float64", index=df_copy.index)
+    adf_pval = pd.Series(dtype="float64", index=df_copy.index)
+    stationarity_flag = pd.Series(dtype="int", index=df_copy.index)
+
+    for i in range(window_size, len(df_copy)):
+        slice_data = df_copy[col].iloc[i - window_size : i].values
+        try:
+            result = adfuller(slice_data, autolag='AIC')
+            adf_stat.iloc[i] = result[0]
+            adf_pval.iloc[i] = result[1]
+            stationarity_flag.iloc[i] = 1 if result[1] < p_value_threshold else 0
+        except:
+            adf_stat.iloc[i] = np.nan
+            adf_pval.iloc[i] = np.nan
+            stationarity_flag.iloc[i] = np.nan
+
+    df_copy['rolling_adf_stat'] = adf_stat
+    df_copy['rolling_adf_pval'] = adf_pval
+    df_copy['stationary_flag'] = stationarity_flag  # 1 = stationary, 0 = non-stationary
+
+    return df_copy
+
+
+# --------------------------------------------------------------------
+# 7) DOUBLE-BARRIER LABEL
+# --------------------------------------------------------------------
+def set_double_barrier_label(
+    df: pd.DataFrame, 
+    up: float = 0.005, 
+    down: float = 0.005, 
+    horizon: int = 50
+) -> pd.DataFrame:
+    df_copy = df.copy()
+    closes = df_copy["close"].values
+    labels = np.full(len(closes), np.nan)
+
+    for i in range(len(closes)):
+        current_price = closes[i]
+        upper_barrier = current_price * (1 + up)
+        lower_barrier = current_price * (1 - down)
+        end = min(i + horizon, len(closes))
+        for forward_i in range(i + 1, end):
+            if closes[forward_i] >= upper_barrier:
+                labels[i] = 1
+                break
+            elif closes[forward_i] <= lower_barrier:
+                labels[i] = 0
+                break
+    df_copy["barrier_label"] = labels
+    df_copy.dropna(subset=["barrier_label"], inplace=True)
+    return df_copy
+
+# --------------------------------------------------------------------
+# 8) FUTURE MARKET REGIME (Directional-Change Example)
+# --------------------------------------------------------------------
+def future_DC_market_regime(df: pd.DataFrame, threshold: float = 0.03, horizon: int = 10) -> pd.DataFrame:
+    df_copy = df.copy()
+    df_copy['future_return'] = df_copy['close'].shift(-horizon) / df_copy['close'] - 1.0
+    df_copy['future_market_regime'] = np.nan
+    df_copy.loc[df_copy['future_return'] >= threshold, 'future_market_regime'] = 1
+    df_copy.loc[df_copy['future_return'] <= -threshold, 'future_market_regime'] = 0
+    df_copy.dropna(subset=['future_market_regime'], inplace=True)
+    return df_copy
+
+
+
+# --------------------------------------------------------------------
+# 9) Introduce Fourier & Wavelet Features for Cyclical Pattern Recognition
+# --------------------------------------------------------------------
+
+def add_fourier_features(df: pd.DataFrame, col: str = "close", n_components: int = 5) -> pd.DataFrame:
+    """
+    Extracts the top 'n_components' Fourier coefficients from price data.
+    """
+    fft_vals = np.abs(fft(df[col].values))
+    for i in range(1, n_components + 1):
+        df[f'fft_comp_{i}'] = fft_vals[i]
+    return df
+
+
+# --------------------------------------------------------------------
+# 10) Optimize ADF Test for Model Selection
+# --------------------------------------------------------------------
+
+def apply_differencing_if_needed(df: pd.DataFrame, col: str = "close", threshold: float = 0.05) -> pd.DataFrame:
+    """
+    If ADF p-value > threshold (non-stationary), apply first differencing.
+    """
+    if df['rolling_adf_pval'].iloc[-1] > threshold:  # Check last rolling p-value
+        df[f"{col}_diff"] = df[col] - df[col].shift(1)  # First differencing
+    return df.dropna()
+
+
+
+# --------------------------------------------------------------------
+# 11) Normalize Feature Distributions (Scaling)
+# --------------------------------------------------------------------
+def scale_features(df: pd.DataFrame, cols_to_scale: list) -> pd.DataFrame:
+    scaler = StandardScaler()
+    df[cols_to_scale] = scaler.fit_transform(df[cols_to_scale])
+    return df
+
+
+
+# 12) SINGLE PIPELINE EXAMPLE
+# --------------------------------------------------------------------
+def create_features(df: pd.DataFrame, col: str = "close", window_size: int = 30) -> pd.DataFrame:
+    """
+    Optimized pipeline integrating TA, autocorrelation, stationarity, Fourier transform, and normalization.
+    """
+    df = add_all_ta_features(df)          # Adds TA indicators
+    df = spread(df)                        # Adds 'spread'
+    df = auto_corr_multi(df, col='close')  # Multi-lag autocorrelation
+    df = rolling_adf_with_flag(df)         # ADF with stationarity flag
+    
+    df = log_transform(df, col, 5)         # Log transform
+    df = moving_yang_zhang_estimator(df, window_size)
+    df = moving_parkinson_estimator(df, window_size)
+    
+    df = add_fourier_features(df, col="close")  # Fourier Transform for cyclic detection
+    df = apply_differencing_if_needed(df, col="close")  # Ensure stationarity
+
+    # Normalize all numeric features
+    df = scale_features(df, df.select_dtypes(include=[np.number]).columns.tolist())
+
+    return df

features/labeling_schemes.py

@@ -0,0 +1,145 @@
+import pandas as pd
+import numpy as np   # <-- Make sure this is present
+
+
+def calculate_future_returns(df: pd.DataFrame, horizon: int = 1) -> pd.DataFrame:
+    """
+    Calculates future returns for a given horizon. By default, horizon=1
+    means next-bar returns. The function appends a new column 'future_returns'.
+    """
+    df["future_returns"] = df["close"].pct_change(periods=horizon).shift(-horizon)
+    return df.dropna(subset=["future_returns"])
+
+def create_labels_multi_bar(df, horizon=5, threshold=0.005):
+    """
+    Creates classification labels for a multi-bar horizon.
+    +1 if future return >= +threshold
+    -1 if future return <= -threshold
+     0 otherwise (could keep as neutral or drop).
+    
+    df must have a 'close' column.
+    Returns a new DataFrame with:
+      - 'future_return_h' (the h-bar future return)
+      - 'multi_bar_label' (the classification label)
+    """
+    df_copy = df.copy()
+    
+    # 1) Compute the horizon-based future returns
+    df_copy["future_return_h"] = df_copy["close"].pct_change(periods=horizon).shift(-horizon)
+    
+    # 2) Create classification labels
+    df_copy["multi_bar_label"] = 0
+    df_copy.loc[df_copy["future_return_h"] >= threshold, "multi_bar_label"] = 1
+    df_copy.loc[df_copy["future_return_h"] <= -threshold, "multi_bar_label"] = -1
+    
+    # 3) Drop rows where future_return_h is NaN (the last 'horizon' bars)
+    df_copy.dropna(subset=["future_return_h"], inplace=True)
+    
+    # If you prefer a pure up/down classification, do:
+    # df_copy = df_copy[df_copy["multi_bar_label"] != 0]
+    
+    return df_copy
+
+
+def create_labels_double_barrier(df, up=0.005, down=0.005, horizon=20):
+    """
+    Double-barrier labeling:
+      - For each index i, define:
+          upper_barrier = close_i * (1 + up)
+          lower_barrier = close_i * (1 - down)
+      - Look ahead up to 'horizon' bars to see which barrier is touched first.
+      - Label = +1 if upper barrier touched first,
+                -1 if lower barrier touched first,
+                 0 if neither is touched within horizon.
+    df must have a 'close' column.
+    Returns a new DataFrame with a 'barrier_label' in {-1, 0, +1}.
+    """
+    df_copy = df.copy()
+    closes = df_copy["close"].values
+    
+    labels = np.full(len(closes), np.nan)
+    
+    for i in range(len(closes)):
+        current_price = closes[i]
+        upper_barrier = current_price * (1 + up)
+        lower_barrier = current_price * (1 - down)
+        
+        # Look ahead up to horizon bars (or until dataset ends)
+        end = min(i + horizon, len(closes))
+        for fwd_i in range(i+1, end):
+            if closes[fwd_i] >= upper_barrier:
+                labels[i] = 1
+                break
+            elif closes[fwd_i] <= lower_barrier:
+                labels[i] = -1
+                break
+        # if we exit loop without setting label => neither barrier hit => 0
+        if np.isnan(labels[i]):
+            labels[i] = 0
+    
+    df_copy["barrier_label"] = labels
+    return df_copy
+
+
+
+def create_labels_double_barrier(df, up=0.005, down=0.005, horizon=20):
+    """
+    Double-barrier labeling:
+      +1 if upper barrier is touched first,
+      -1 if lower barrier is touched first,
+       0 if neither is touched within horizon.
+    df must have a 'close' column.
+    Returns a new DataFrame with a 'barrier_label' column in {-1, 0, +1}.
+    """
+    df_copy = df.copy()
+    closes = df_copy["close"].values
+    labels = np.full(len(closes), np.nan)
+    
+    for i in range(len(closes)):
+        current_price = closes[i]
+        upper_barrier = current_price * (1 + up)
+        lower_barrier = current_price * (1 - down)
+        
+        end = min(i + horizon, len(closes))
+        for fwd_i in range(i+1, end):
+            if closes[fwd_i] >= upper_barrier:
+                labels[i] = 1
+                break
+            elif closes[fwd_i] <= lower_barrier:
+                labels[i] = -1
+                break
+        if np.isnan(labels[i]):
+            labels[i] = 0
+    
+    df_copy["barrier_label"] = labels
+    return df_copy
+
+
+
+def create_labels_regime_detection(df, short_window=20, long_window=50):
+    """
+    Simple regime detection:
+      +1 if short MA > long MA (up)
+      -1 if short MA < long MA (down)
+       0 otherwise (sideways)
+    df must have 'close' column.
+    Returns a new DataFrame with 'regime_label' in {-1, 0, +1}.
+    """
+    df_copy = df.copy()
+    
+    # 1) Compute short and long MAs
+    df_copy["ma_short"] = df_copy["close"].rolling(short_window).mean()
+    df_copy["ma_long"] = df_copy["close"].rolling(long_window).mean()
+    
+    # 2) Label each bar
+    df_copy["regime_label"] = 0
+    up_mask = df_copy["ma_short"] > df_copy["ma_long"]
+    down_mask = df_copy["ma_short"] < df_copy["ma_long"]
+    
+    df_copy.loc[up_mask, "regime_label"] = 1
+    df_copy.loc[down_mask, "regime_label"] = -1
+    
+    # 3) Drop rows where MAs are NaN (the first 'long_window' bars)
+    df_copy.dropna(subset=["ma_short", "ma_long"], inplace=True)
+    
+    return df_copy

live_trading/double_barrier.py

@@ -0,0 +1,410 @@
+# LIVE TRADING CODE FOR DOUBLE-BARRIER CLASSIFICATION
+
+import sys
+import os
+import warnings
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# 1) SET PROJECT ROOT AND UPDATE PATH/WORKING DIRECTORY
+# ---------------------------------------------------------------------------
+project_root = Path.cwd().parent.parent  # Adjust if your notebook is in notebooks/time_series
+sys.path.append(str(project_root))
+os.chdir(str(project_root))
+warnings.filterwarnings("ignore")
+
+import warnings
+warnings.filterwarnings("ignore")
+
+import MetaTrader5 as mt5
+import pandas as pd
+import numpy as np
+import ta
+from datetime import datetime, timedelta
+import time
+import logging
+import joblib
+
+# Setup logging
+logging.basicConfig(
+    filename='models/saved_models/trading_app_db.log',
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s:%(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S'
+)
+
+def log_and_print(message, is_error=False):
+    """
+    Logs and prints a message.
+    If is_error=True, logs at the ERROR level; otherwise logs at INFO level.
+    """
+    if is_error:
+        logging.error(message)
+    else:
+        logging.info(message)
+    print(message)
+
+# Update the login credentials and server information accordingly
+name = 66677507
+key = 'ST746$nG38'
+serv = 'ICMarketsSC-Demo'
+
+# Global variables
+SYMBOL = "EURUSD"
+LOT_SIZE = 0.01
+TIMEFRAME = mt5.TIMEFRAME_D1
+N_BARS = 50000
+MAGIC_NUMBER = 234003
+SLEEP_TIME = 86400  # e.g. 24 hours
+COMMENT_ML = "DoubleBarrier-ML"
+
+class TradingApp:
+    def __init__(self, symbol, lot_size, magic_number):
+        self.symbol = symbol
+        self.lot_size = lot_size
+        self.magic_number = magic_number
+        self.pipeline = None  # We'll store the loaded classification pipeline here
+        self.last_retrain_time = None
+
+    def get_data(self, symbol, n, timeframe):
+        """
+        Fetch the last 'n' bars from MetaTrader 5 for the given timeframe.
+        """
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, n)
+        rates_frame = pd.DataFrame(rates)
+        rates_frame['time'] = pd.to_datetime(rates_frame['time'], unit='s')
+        rates_frame.set_index('time', inplace=True)
+        return rates_frame
+
+    def add_all_ta_features(self, df):
+        """
+        Add technical analysis features to the DataFrame using 'ta' library.
+        """
+        df = ta.add_all_ta_features(
+            df, open="open", high="high", low="low", close="close", volume="tick_volume", fillna=True
+        )
+        return df
+
+    def load_pipeline(self, pipeline_path):
+        """
+        Loads a pre-trained classification pipeline (e.g., final_production_pipeline.pkl)
+        that was trained on SHIFTED double-barrier labels in {0,1,2}.
+        """
+        self.pipeline = joblib.load(pipeline_path)
+        logging.info(f"Loaded pipeline from {pipeline_path}")
+        log_and_print(f"Loaded pipeline from {pipeline_path}")
+
+    def ml_signal_generation(self, symbol, n_bars, timeframe):
+        """
+        Generate buy/sell signals using the loaded classification pipeline.
+        The pipeline outputs SHIFTED labels in {0,1,2} => SHIFT them back to {-1,0,+1}.
+        We'll interpret +1 => buy, -1 => sell, 0 => no trade.
+
+        Double-Barrier labeling was used offline to train this pipeline,
+        so we just replicate the same feature engineering steps and let the model predict.
+        """
+        if self.pipeline is None:
+            logging.error("No pipeline loaded. Call load_pipeline(...) first.")
+            return False, False, True, True
+
+        # 1) Fetch new data
+        df = self.get_data(symbol, n_bars, timeframe)
+        # 2) Add TA features
+        df = self.add_all_ta_features(df)
+        df.fillna(method='ffill', inplace=True)
+
+        # 3) Prepare the features
+        X_new = df
+
+        # 4) Predict SHIFTED classes
+        preds_shifted = self.pipeline.predict(X_new)
+        # SHIFT them back: 0->-1, 1->0, 2->+1
+        preds = preds_shifted - 1
+
+        # Get the latest predicted class
+        latest_pred = preds[-1]
+        # If latest_pred == +1 => buy signal
+        # If latest_pred == -1 => sell signal
+        # If 0 => no trade
+        buy_signal = (latest_pred == 1)
+        sell_signal = (latest_pred == -1)
+
+        return buy_signal, sell_signal, not buy_signal, not sell_signal
+
+    def orders(self, symbol, lot, is_buy=True, id_position=None, sl=None, tp=None):
+        """
+        Send an order (buy/sell) to MetaTrader 5.
+        """
+        symbol_info = mt5.symbol_info(symbol)
+        if symbol_info is None:
+            log_and_print(f"Symbol {symbol} not found, can't place order.", is_error=True)
+            return "Symbol not found"
+
+        # Make sure symbol is visible
+        if not symbol_info.visible:
+            if not mt5.symbol_select(symbol, True):
+                log_and_print(f"Failed to select symbol {symbol}", is_error=True)
+                return "Symbol not visible or could not be selected."
+
+        tick_info = mt5.symbol_info_tick(symbol)
+        if tick_info is None:
+            log_and_print(f"Could not get tick info for {symbol}.", is_error=True)
+            return "Tick info unavailable"
+
+        # Check for valid bid/ask
+        if tick_info.bid <= 0 or tick_info.ask <= 0:
+            log_and_print(
+                f"Zero or invalid bid/ask for {symbol}: bid={tick_info.bid}, ask={tick_info.ask}",
+                is_error=True
+            )
+            return "Invalid prices"
+
+        # LOT SIZE VALIDATION
+        lot = max(lot, symbol_info.volume_min)
+        step = symbol_info.volume_step
+        if step > 0:
+            remainder = lot % step
+            if remainder != 0:
+                lot = lot - remainder + step
+        if lot > symbol_info.volume_max:
+            lot = symbol_info.volume_max
+
+        log_and_print(
+            f"Adjusted lot size to {lot} (min={symbol_info.volume_min}, "
+            f"step={symbol_info.volume_step}, max={symbol_info.volume_max})"
+        )
+
+        # Force ORDER_FILLING_IOC
+        filling_mode = mt5.ORDER_FILLING_IOC
+
+        order_type = mt5.ORDER_TYPE_BUY if is_buy else mt5.ORDER_TYPE_SELL
+        deviation = 20
+
+        request = {
+            "action": mt5.TRADE_ACTION_DEAL,
+            "symbol": symbol,
+            "volume": lot,
+            "type": order_type,
+            "deviation": deviation,
+            "magic": self.magic_number,
+            "comment": COMMENT_ML,
+            "type_time": mt5.ORDER_TIME_GTC,
+            "type_filling": filling_mode,
+        }
+
+        if sl is not None:
+            request["sl"] = sl
+        if tp is not None:
+            request["tp"] = tp
+        if id_position is not None:
+            request["position"] = id_position
+
+        log_and_print(f"Sending order request: {request}")
+        result = mt5.order_send(request)
+
+        order_type_str = "BUY" if is_buy else "SELL"
+        if result is None or result.retcode != mt5.TRADE_RETCODE_DONE:
+            error_message = f"Order failed for {symbol}"
+            if result:
+                error_message += f", retcode={result.retcode}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}\n"
+                f"Request: {request}\n"
+                f"Result: {result}"
+            )
+            # If you need notifications, you could log or handle them differently here.
+            log_and_print(f"Order failed details: {additional_info}", is_error=True)
+        else:
+            success_message = f"Order successful for {symbol}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}"
+            )
+            # If you need notifications, you could log or handle them differently here.
+            log_and_print(success_message)
+
+    def get_positions_by_magic(self, symbol, magic_number):
+        """
+        Retrieve positions for a specific symbol and magic number.
+        """
+        all_positions = mt5.positions_get(symbol=symbol)
+        if not all_positions:
+            log_and_print("No positions found.", is_error=False)
+            return []
+        return [pos for pos in all_positions if pos.magic == magic_number]
+
+    def run_strategy(self, symbol, lot, buy_signal, sell_signal):
+        """
+        Decide whether to open a buy or sell order based on signals,
+        close opposite positions if needed, etc.
+        """
+        log_and_print("------------------------------------------------------------------")
+        log_and_print(
+            f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}, "
+            f"SYMBOL: {symbol}, BUY SIGNAL: {buy_signal}, SELL SIGNAL: {sell_signal}"
+        )
+
+        positions = self.get_positions_by_magic(symbol, self.magic_number)
+        has_buy = any(pos.type == mt5.POSITION_TYPE_BUY for pos in positions)
+        has_sell = any(pos.type == mt5.POSITION_TYPE_SELL for pos in positions)
+
+        if buy_signal and not has_buy:
+            if has_sell:
+                log_and_print("Existing sell positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=True):
+                    log_and_print("Sell positions closed. Placing new buy order.")
+                    self.orders(symbol, lot, is_buy=True)
+                else:
+                    log_and_print("Failed to close sell positions.")
+            else:
+                self.orders(symbol, lot, is_buy=True)
+        elif sell_signal and not has_sell:
+            if has_buy:
+                log_and_print("Existing buy positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=False):
+                    log_and_print("Buy positions closed. Placing new sell order.")
+                    self.orders(symbol, lot, is_buy=False)
+                else:
+                    log_and_print("Failed to close buy positions.")
+            else:
+                self.orders(symbol, lot, is_buy=False)
+        else:
+            log_and_print("Appropriate position already exists or no signal to act on.")
+
+    def close_position(self, symbol, is_buy):
+        """
+        Close all positions of the opposite type for the given symbol & magic.
+        """
+        positions = mt5.positions_get(symbol=symbol)
+        if not positions:
+            log_and_print(f"No positions to close for symbol: {symbol}")
+            return False
+
+        initial_balance = mt5.account_info().balance
+        closed_any = False
+
+        for position in positions:
+            if position.magic == self.magic_number:
+                # if is_buy==True => we want to close SELL positions
+                # if is_buy==False => we want to close BUY positions
+                if ((is_buy and position.type == mt5.POSITION_TYPE_SELL) or
+                    (not is_buy and position.type == mt5.POSITION_TYPE_BUY)):
+                    close_request = {
+                        "action": mt5.TRADE_ACTION_DEAL,
+                        "symbol": symbol,
+                        "volume": position.volume,
+                        "type": mt5.ORDER_TYPE_BUY if position.type == mt5.POSITION_TYPE_SELL else mt5.ORDER_TYPE_SELL,
+                        "position": position.ticket,
+                        "deviation": 20,
+                        "magic": self.magic_number,
+                        "comment": COMMENT_ML,
+                        "type_time": mt5.ORDER_TIME_GTC,
+                        "type_filling": mt5.ORDER_FILLING_RETURN,
+                    }
+                    result = mt5.order_send(close_request)
+                    if result.retcode != mt5.TRADE_RETCODE_DONE:
+                        error_message = (
+                            f"Failed to close position {position.ticket} for {symbol}: {result.retcode}"
+                        )
+                        log_and_print(error_message, is_error=True)
+                        # If you need notifications, you could log or handle them differently here.
+                    else:
+                        log_and_print(f"Successfully closed position {position.ticket} for {symbol}")
+                        closed_any = True
+
+        if closed_any:
+            final_balance = mt5.account_info().balance
+            profit = final_balance - initial_balance
+            success_message = f"Closed positions successfully, Profit: {profit}"
+            log_and_print(success_message)
+            return True
+
+        return False
+
+    def check_and_execute_trades(self):
+        """
+        Called in the main loop: generate signals, run strategy, etc.
+        """
+        mt5.symbol_select(self.symbol, True)
+        buy, sell, _, _ = self.ml_signal_generation(self.symbol, N_BARS, TIMEFRAME)
+        self.run_strategy(self.symbol, self.lot_size, buy, sell)
+        mt5.symbol_select(self.symbol, False)
+        log_and_print("Waiting for new signals...")
+
+def is_market_open():
+    """
+    Check if the current time is within the typical Forex trading session, adjusted for CET/CEST.
+    Market closes at Friday 10:00 PM CET and opens at Sunday 11:00 PM CET.
+    """
+    current_time_utc = datetime.utcnow()
+    current_time_cet = (
+        current_time_utc + timedelta(hours=2) 
+        if time.localtime().tm_isdst 
+        else current_time_utc + timedelta(hours=1)
+    )
+
+    # Market closes Friday after 10 PM CET
+    if current_time_cet.weekday() == 4 and current_time_cet.hour >= 22:
+        return False
+    # Market opens Sunday after 11 PM CET
+    elif current_time_cet.weekday() == 6 and current_time_cet.hour < 23:
+        return False
+    # Closed all day Saturday
+    elif current_time_cet.weekday() == 5:
+        return False
+    return True
+
+if __name__ == "__main__":
+    try:
+        if not mt5.initialize(login=name, server=serv, password=key):
+            log_and_print("Failed to initialize MetaTrader 5", is_error=True)
+            exit()
+
+        app = TradingApp(symbol=SYMBOL, lot_size=LOT_SIZE, magic_number=MAGIC_NUMBER)
+
+        # 1) Load the classification pipeline
+        # Make sure this pipeline is a classification model expecting SHIFTED double-barrier labels in {0,1,2}
+        pipeline_path = "models/saved_models/best_rf_db_pipeline.pkl"
+        app.load_pipeline(pipeline_path)
+
+        while True:
+            log_and_print("Checking market status...")
+            if is_market_open():
+                log_and_print("Market is open. Executing trades...")
+
+                # 2) Generate signals using the loaded pipeline
+                #    This pipeline is classification-based => SHIFTED labels {0,1,2}
+                #    ml_signal_generation() SHIFTs them back to [-1,0,+1]
+                buy_signal, sell_signal, _, _ = app.ml_signal_generation(
+                    symbol=app.symbol,
+                    n_bars=N_BARS,
+                    timeframe=TIMEFRAME
+                )
+
+                # 3) Run strategy
+                app.run_strategy(app.symbol, app.lot_size, buy_signal, sell_signal)
+            else:
+                log_and_print("Market is closed. No actions performed.")
+
+            time.sleep(SLEEP_TIME)
+
+    except KeyboardInterrupt:
+        log_and_print("Shutdown signal received.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    except Exception as e:
+        error_message = f"An error occurred: {e}"
+        log_and_print(error_message, is_error=True)
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    finally:
+        mt5.shutdown()
+        log_and_print("MetaTrader 5 shutdown completed.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).

live_trading/multi_bar.py

@@ -0,0 +1,419 @@
+# LIVE TRADING CODE FOR MULTI-BAR CLASSIFICATION
+
+import sys
+import os
+import warnings
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# 1) SET PROJECT ROOT AND UPDATE PATH/WORKING DIRECTORY
+# ---------------------------------------------------------------------------
+project_root = Path.cwd().parent.parent  # Adjust if your notebook is in notebooks/time_series
+sys.path.append(str(project_root))
+os.chdir(str(project_root))
+warnings.filterwarnings("ignore")
+
+
+import warnings
+warnings.filterwarnings("ignore")
+import MetaTrader5 as mt5
+import pandas as pd
+import numpy as np
+import ta
+from datetime import datetime, timedelta
+import time
+import logging
+import joblib
+
+# Setup logging
+logging.basicConfig(
+    filename='models/saved_models/trading_app1.log',
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s:%(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S'
+)
+
+def log_and_print(message, is_error=False):
+    """
+    Logs and prints a message.
+    If is_error=True, logs at the ERROR level; otherwise logs at INFO level.
+    """
+    if is_error:
+        logging.error(message)
+    else:
+        logging.info(message)
+    print(message)
+
+# Update the login credentials and server information accordingly
+name = 66677507
+key = 'ST746$nG38'
+serv = 'ICMarketsSC-Demo'
+
+# Global variables
+SYMBOL = "EURUSD"
+LOT_SIZE = 0.01
+TIMEFRAME = mt5.TIMEFRAME_D1
+N_BARS = 50000
+MAGIC_NUMBER = 234003
+SLEEP_TIME = 86400  # 24 hours in seconds
+COMMENT_ML = "RFFV-D"
+
+# If you still need feature selection, you can keep this helper function:
+def select_features_rf_reg(X, y, estimator, max_features=20):
+    """
+    Example helper function for feature selection using RandomForest.
+    """
+    from sklearn.feature_selection import SelectFromModel
+    selector = SelectFromModel(estimator=estimator, threshold=-np.inf, max_features=max_features).fit(X, y)
+    X_transformed = selector.transform(X)
+    selected_features_mask = selector.get_support()
+    return X_transformed, selected_features_mask
+
+class TradingApp:
+    def __init__(self, symbol, lot_size, magic_number):
+        self.symbol = symbol
+        self.lot_size = lot_size
+        self.magic_number = magic_number
+        self.pipeline = None  # We'll store the loaded classification pipeline here
+        self.last_retrain_time = None
+
+    def get_data(self, symbol, n, timeframe):
+        """
+        Fetch 'n' bars of historical data for the given symbol and timeframe.
+        """
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, n)
+        rates_frame = pd.DataFrame(rates)
+        rates_frame['time'] = pd.to_datetime(rates_frame['time'], unit='s')
+        rates_frame.set_index('time', inplace=True)
+        return rates_frame
+
+    def add_all_ta_features(self, df):
+        """
+        Add technical analysis features to the DataFrame using the 'ta' library.
+        """
+        df = ta.add_all_ta_features(
+            df, open="open", high="high", low="low", close="close", volume="tick_volume", fillna=True
+        )
+        return df
+
+    def load_pipeline(self, pipeline_path):
+        """
+        Loads a pre-trained classification pipeline (e.g., 'best_rf_pipeline.pkl').
+        This pipeline is expected to produce SHIFTED labels [0,1,2].
+        """
+        self.pipeline = joblib.load(pipeline_path)
+        logging.info(f"Loaded pipeline from {pipeline_path}")
+        log_and_print(f"Loaded pipeline from {pipeline_path}")
+
+    def ml_signal_generation(self, symbol, n_bars, timeframe):
+        """
+        Generate buy/sell signals using the loaded classification pipeline.
+        The pipeline outputs SHIFTED labels in {0,1,2} => we SHIFT them back to {-1,0,+1}.
+        We'll interpret +1 => buy, -1 => sell, 0 => no trade.
+        """
+        if self.pipeline is None:
+            logging.error("No pipeline loaded. Call load_pipeline(...) first.")
+            return False, False, True, True
+
+        # 1) Fetch new data
+        df = self.get_data(symbol, n_bars, timeframe)
+
+        # 2) Add TA features
+        df = self.add_all_ta_features(df)
+        df.fillna(method='ffill', inplace=True)
+
+        # 3) Prepare the features
+        X_new = df  # The pipeline must handle columns in the correct order.
+
+        # 4) Predict SHIFTED classes
+        preds_shifted = self.pipeline.predict(X_new)
+        # SHIFT them back: 0->-1, 1->0, 2->+1
+        preds = preds_shifted - 1
+
+        # Get the latest predicted class
+        latest_pred = preds[-1]
+        # If latest_pred == +1 => buy signal
+        # If latest_pred == -1 => sell signal
+        # If 0 => do nothing
+        buy_signal = (latest_pred == 1)
+        sell_signal = (latest_pred == -1)
+
+        return buy_signal, sell_signal, not buy_signal, not sell_signal
+
+    def orders(self, symbol, lot, is_buy=True, id_position=None, sl=None, tp=None):
+        """
+        Place an order (BUY or SELL) for the specified symbol and lot size.
+        """
+        symbol_info = mt5.symbol_info(symbol)
+        if symbol_info is None:
+            log_and_print(f"Symbol {symbol} not found, can't place order.", is_error=True)
+            return "Symbol not found"
+
+        # Make sure symbol is visible
+        if not symbol_info.visible:
+            if not mt5.symbol_select(symbol, True):
+                log_and_print(f"Failed to select symbol {symbol}", is_error=True)
+                return "Symbol not visible or could not be selected."
+
+        tick_info = mt5.symbol_info_tick(symbol)
+        if tick_info is None:
+            log_and_print(f"Could not get tick info for {symbol}.", is_error=True)
+            return "Tick info unavailable"
+
+        # Check for valid bid/ask
+        if tick_info.bid <= 0 or tick_info.ask <= 0:
+            log_and_print(
+                f"Zero or invalid bid/ask for {symbol}: bid={tick_info.bid}, ask={tick_info.ask}",
+                is_error=True
+            )
+            return "Invalid prices"
+
+        # LOT SIZE VALIDATION
+        lot = max(lot, symbol_info.volume_min)
+        step = symbol_info.volume_step
+        if step > 0:
+            remainder = lot % step
+            if remainder != 0:
+                lot = lot - remainder + step
+        if lot > symbol_info.volume_max:
+            lot = symbol_info.volume_max
+
+        log_and_print(
+            f"Adjusted lot size to {lot} (min={symbol_info.volume_min}, "
+            f"step={symbol_info.volume_step}, max={symbol_info.volume_max})"
+        )
+
+        # Force ORDER_FILLING_IOC
+        filling_mode = 1  # ORDER_FILLING_IOC
+
+        order_type = mt5.ORDER_TYPE_BUY if is_buy else mt5.ORDER_TYPE_SELL
+        order_price = tick_info.ask if is_buy else tick_info.bid
+        deviation = 20
+
+        request = {
+            "action": mt5.TRADE_ACTION_DEAL,
+            "symbol": symbol,
+            "volume": lot,
+            "type": order_type,
+            "deviation": deviation,
+            "magic": self.magic_number,
+            "comment": COMMENT_ML,
+            "type_time": mt5.ORDER_TIME_GTC,
+            "type_filling": filling_mode,
+        }
+
+        if sl is not None:
+            request["sl"] = sl
+        if tp is not None:
+            request["tp"] = tp
+        if id_position is not None:
+            request["position"] = id_position
+
+        log_and_print(f"Sending order request: {request}")
+        result = mt5.order_send(request)
+
+        order_type_str = "BUY" if is_buy else "SELL"
+        if result is None or result.retcode != mt5.TRADE_RETCODE_DONE:
+            error_message = f"Order failed for {symbol}"
+            if result:
+                error_message += f", retcode={result.retcode}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}\n"
+                f"Request: {request}\n"
+                f"Result: {result}"
+            )
+            # If you want notifications, you could log or handle them differently here.
+            log_and_print(f"Order failed details: {additional_info}", is_error=True)
+        else:
+            success_message = f"Order successful for {symbol}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}"
+            )
+            # If you want notifications, you could log or handle them differently here.
+            log_and_print(success_message)
+
+    def get_positions_by_magic(self, symbol, magic_number):
+        """
+        Retrieve positions for a specific symbol and magic number.
+        """
+        all_positions = mt5.positions_get(symbol=symbol)
+        if not all_positions:
+            log_and_print("No positions found.", is_error=False)
+            return []
+        return [pos for pos in all_positions if pos.magic == magic_number]
+
+    def run_strategy(self, symbol, lot, buy_signal, sell_signal):
+        """
+        Run the trading strategy logic based on buy/sell signals.
+        """
+        log_and_print("------------------------------------------------------------------")
+        log_and_print(
+            f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}, "
+            f"SYMBOL: {symbol}, BUY SIGNAL: {buy_signal}, SELL SIGNAL: {sell_signal}"
+        )
+
+        positions = self.get_positions_by_magic(symbol, self.magic_number)
+        has_buy = any(pos.type == mt5.POSITION_TYPE_BUY for pos in positions)
+        has_sell = any(pos.type == mt5.POSITION_TYPE_SELL for pos in positions)
+
+        if buy_signal and not has_buy:
+            if has_sell:
+                log_and_print("Existing sell positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=True):
+                    log_and_print("Sell positions closed. Placing new buy order.")
+                    self.orders(symbol, lot, is_buy=True)
+                else:
+                    log_and_print("Failed to close sell positions.")
+            else:
+                self.orders(symbol, lot, is_buy=True)
+        elif sell_signal and not has_sell:
+            if has_buy:
+                log_and_print("Existing buy positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=False):
+                    log_and_print("Buy positions closed. Placing new sell order.")
+                    self.orders(symbol, lot, is_buy=False)
+                else:
+                    log_and_print("Failed to close buy positions.")
+            else:
+                self.orders(symbol, lot, is_buy=False)
+        else:
+            log_and_print("Appropriate position already exists or no signal to act on.")
+
+    def close_position(self, symbol, is_buy):
+        """
+        Closes positions of the opposite type (BUY/SELL) for this app's magic number.
+        """
+        positions = mt5.positions_get(symbol=symbol)
+        if not positions:
+            log_and_print(f"No positions to close for symbol: {symbol}")
+            return False
+
+        initial_balance = mt5.account_info().balance
+        closed_any = False
+
+        for position in positions:
+            # Close positions of the opposite type with the same magic number
+            if position.magic == self.magic_number and (
+                (is_buy and position.type == mt5.POSITION_TYPE_SELL) or
+                (not is_buy and position.type == mt5.POSITION_TYPE_BUY)
+            ):
+                close_request = {
+                    "action": mt5.TRADE_ACTION_DEAL,
+                    "symbol": symbol,
+                    "volume": position.volume,
+                    "type": mt5.ORDER_TYPE_BUY if position.type == mt5.POSITION_TYPE_SELL else mt5.ORDER_TYPE_SELL,
+                    "position": position.ticket,
+                    "deviation": 20,
+                    "magic": self.magic_number,
+                    "comment": COMMENT_ML,
+                    "type_time": mt5.ORDER_TIME_GTC,
+                    "type_filling": mt5.ORDER_FILLING_RETURN,
+                }
+                result = mt5.order_send(close_request)
+                if result.retcode != mt5.TRADE_RETCODE_DONE:
+                    error_message = f"Failed to close position {position.ticket} for {symbol}: {result.retcode}"
+                    log_and_print(error_message, is_error=True)
+                    # If you want notifications, you could log or handle them differently here.
+                else:
+                    log_and_print(f"Successfully closed position {position.ticket} for {symbol}")
+                    closed_any = True
+
+        if closed_any:
+            final_balance = mt5.account_info().balance
+            profit = final_balance - initial_balance
+            success_message = f"Closed positions successfully, Profit: {profit}"
+            log_and_print(success_message)
+            return True
+
+        return False
+
+    def check_and_execute_trades(self):
+        """
+        Convenience method to perform the entire flow:
+        generate signals, run strategy, and deselect symbol.
+        """
+        mt5.symbol_select(self.symbol, True)
+        buy, sell, _, _ = self.ml_signal_generation(self.symbol, N_BARS, TIMEFRAME)
+        self.run_strategy(self.symbol, self.lot_size, buy, sell)
+        mt5.symbol_select(self.symbol, False)
+        log_and_print("Waiting for new signals...")
+
+def is_market_open():
+    """
+    Check if the current time is within the typical Forex trading session, adjusted for CET/CEST.
+    Market closes at Friday 10:00 PM CET and opens at Sunday 11:00 PM CET. 
+    It is closed all day Saturday.
+    """
+    current_time_utc = datetime.utcnow()
+    # Adjust for Central European Time (UTC+1) or Central European Summer Time (UTC+2)
+    current_time_cet = (
+        current_time_utc + timedelta(hours=2) 
+        if time.localtime().tm_isdst 
+        else current_time_utc + timedelta(hours=1)
+    )
+
+    # Friday after 10 PM CET
+    if current_time_cet.weekday() == 4 and current_time_cet.hour >= 22:
+        return False
+    # Sunday before 11 PM CET
+    elif current_time_cet.weekday() == 6 and current_time_cet.hour < 23:
+        return False
+    # All day Saturday
+    elif current_time_cet.weekday() == 5:
+        return False
+    return True
+
+if __name__ == "__main__":
+    try:
+        if not mt5.initialize(login=name, server=serv, password=key):
+            log_and_print("Failed to initialize MetaTrader 5", is_error=True)
+            exit()
+
+        app = TradingApp(symbol=SYMBOL, lot_size=LOT_SIZE, magic_number=MAGIC_NUMBER)
+
+        # 1) Load the classification pipeline
+        pipeline_path = "models/saved_models/best_rf_mb_pipeline.pkl"
+        app.load_pipeline(pipeline_path)
+
+        while True:
+            log_and_print("Checking market status...")
+            if is_market_open():
+                log_and_print("Market is open. Executing trades...")
+
+                # 2) Generate signals using the loaded pipeline
+                #    This pipeline is classification-based => SHIFTED labels [0,1,2]
+                #    ml_signal_generation() SHIFTs them back to [-1,0,+1] for signals
+                buy_signal, sell_signal, _, _ = app.ml_signal_generation(
+                    symbol=app.symbol,
+                    n_bars=N_BARS,
+                    timeframe=TIMEFRAME
+                )
+
+                # 3) Run strategy
+                app.run_strategy(app.symbol, app.lot_size, buy_signal, sell_signal)
+            else:
+                log_and_print("Market is closed. No actions performed.")
+
+            time.sleep(SLEEP_TIME)
+
+    except KeyboardInterrupt:
+        log_and_print("Shutdown signal received.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    except Exception as e:
+        error_message = f"An error occurred: {e}"
+        log_and_print(error_message, is_error=True)
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    finally:
+        mt5.shutdown()
+        log_and_print("MetaTrader 5 shutdown completed.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).

live_trading/regime_detection.py

@@ -0,0 +1,407 @@
+# LIVE TRADING CODE FOR REGIME DETECTION CLASSIFICATION
+import sys
+import os
+import warnings
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# 1) SET PROJECT ROOT AND UPDATE PATH/WORKING DIRECTORY
+# ---------------------------------------------------------------------------
+project_root = Path.cwd().parent.parent  # Adjust if your notebook is in notebooks/time_series
+sys.path.append(str(project_root))
+os.chdir(str(project_root))
+warnings.filterwarnings("ignore")
+
+import warnings
+warnings.filterwarnings("ignore")
+
+import MetaTrader5 as mt5
+import pandas as pd
+import numpy as np
+import ta
+from datetime import datetime, timedelta
+import time
+import logging
+import joblib
+
+# Setup logging
+logging.basicConfig(
+    filename='models/saved_models/trading_app.log',
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s:%(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S'
+)
+
+def log_and_print(message, is_error=False):
+    """
+    Logs and prints a message.
+    If is_error=True, logs at the ERROR level; otherwise logs at INFO level.
+    """
+    if is_error:
+        logging.error(message)
+    else:
+        logging.info(message)
+    print(message)
+
+# Update the login credentials and server information accordingly
+name = 66677507
+key = 'ST746$nG38'
+serv = 'ICMarketsSC-Demo'
+
+# Global variables
+SYMBOL = "EURUSD"
+LOT_SIZE = 0.01
+TIMEFRAME = mt5.TIMEFRAME_D1
+N_BARS = 50000
+MAGIC_NUMBER = 234003
+SLEEP_TIME = 86400  # 24 hours in seconds
+COMMENT_ML = "Regime-Detection"
+
+class TradingApp:
+    def __init__(self, symbol, lot_size, magic_number):
+        self.symbol = symbol
+        self.lot_size = lot_size
+        self.magic_number = magic_number
+        self.pipeline = None  # We'll store the loaded classification pipeline here
+        self.last_retrain_time = None
+
+    def get_data(self, symbol, n, timeframe):
+        """
+        Fetch the last 'n' bars from MetaTrader 5 for the given timeframe.
+        """
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, n)
+        rates_frame = pd.DataFrame(rates)
+        rates_frame['time'] = pd.to_datetime(rates_frame['time'], unit='s')
+        rates_frame.set_index('time', inplace=True)
+        return rates_frame
+
+    def add_all_ta_features(self, df):
+        """
+        Add technical analysis features to the DataFrame using the 'ta' library.
+        """
+        df = ta.add_all_ta_features(
+            df, open="open", high="high", low="low", close="close", volume="tick_volume", fillna=True
+        )
+        return df
+
+    def load_pipeline(self, pipeline_path):
+        """
+        Loads a pre-trained classification pipeline (e.g., final_production_pipeline.pkl).
+        This pipeline is expected to produce SHIFTED labels [0,1,2].
+        """
+        self.pipeline = joblib.load(pipeline_path)
+        logging.info(f"Loaded pipeline from {pipeline_path}")
+        log_and_print(f"Loaded pipeline from {pipeline_path}")
+
+    def ml_signal_generation(self, symbol, n_bars, timeframe):
+        """
+        Generate buy/sell signals using the loaded classification pipeline.
+        The pipeline outputs SHIFTED labels in {0,1,2} => SHIFT them back to {-1,0,+1}.
+        We'll interpret +1 => buy, -1 => sell, 0 => no trade.
+        """
+        if self.pipeline is None:
+            logging.error("No pipeline loaded. Call load_pipeline(...) first.")
+            return False, False, True, True
+
+        # 1) Fetch new data
+        df = self.get_data(symbol, n_bars, timeframe)
+        # 2) Add TA features
+        df = self.add_all_ta_features(df)
+        df.fillna(method='ffill', inplace=True)
+
+        # 3) Prepare the features (the pipeline must handle columns in correct order)
+        X_new = df
+
+        # 4) Predict SHIFTED classes
+        preds_shifted = self.pipeline.predict(X_new)
+        # SHIFT them back: 0->-1, 1->0, 2->+1
+        preds = preds_shifted - 1
+
+        # Get the latest predicted class
+        latest_pred = preds[-1]
+        # If latest_pred == +1 => buy signal
+        # If latest_pred == -1 => sell signal
+        # If 0 => no trade
+        buy_signal = (latest_pred == 1)
+        sell_signal = (latest_pred == -1)
+
+        return buy_signal, sell_signal, not buy_signal, not sell_signal
+
+    def orders(self, symbol, lot, is_buy=True, id_position=None, sl=None, tp=None):
+        """
+        Send an order (buy/sell) to MetaTrader 5.
+        """
+        symbol_info = mt5.symbol_info(symbol)
+        if symbol_info is None:
+            log_and_print(f"Symbol {symbol} not found, can't place order.", is_error=True)
+            return "Symbol not found"
+
+        # Make sure symbol is visible
+        if not symbol_info.visible:
+            if not mt5.symbol_select(symbol, True):
+                log_and_print(f"Failed to select symbol {symbol}", is_error=True)
+                return "Symbol not visible or could not be selected."
+
+        tick_info = mt5.symbol_info_tick(symbol)
+        if tick_info is None:
+            log_and_print(f"Could not get tick info for {symbol}.", is_error=True)
+            return "Tick info unavailable"
+
+        # Check for valid bid/ask
+        if tick_info.bid <= 0 or tick_info.ask <= 0:
+            log_and_print(
+                f"Zero or invalid bid/ask for {symbol}: bid={tick_info.bid}, ask={tick_info.ask}",
+                is_error=True
+            )
+            return "Invalid prices"
+
+        # LOT SIZE VALIDATION
+        lot = max(lot, symbol_info.volume_min)
+        step = symbol_info.volume_step
+        if step > 0:
+            remainder = lot % step
+            if remainder != 0:
+                lot = lot - remainder + step
+        if lot > symbol_info.volume_max:
+            lot = symbol_info.volume_max
+
+        log_and_print(
+            f"Adjusted lot size to {lot} (min={symbol_info.volume_min}, "
+            f"step={symbol_info.volume_step}, max={symbol_info.volume_max})"
+        )
+
+        # Force ORDER_FILLING_IOC
+        filling_mode = mt5.ORDER_FILLING_IOC
+
+        order_type = mt5.ORDER_TYPE_BUY if is_buy else mt5.ORDER_TYPE_SELL
+        deviation = 20
+        order_price = tick_info.ask if is_buy else tick_info.bid
+
+        request = {
+            "action": mt5.TRADE_ACTION_DEAL,
+            "symbol": symbol,
+            "volume": lot,
+            "type": order_type,
+            "deviation": deviation,
+            "magic": self.magic_number,
+            "comment": COMMENT_ML,
+            "type_time": mt5.ORDER_TIME_GTC,
+            "type_filling": filling_mode,
+        }
+
+        if sl is not None:
+            request["sl"] = sl
+        if tp is not None:
+            request["tp"] = tp
+        if id_position is not None:
+            request["position"] = id_position
+
+        log_and_print(f"Sending order request: {request}")
+        result = mt5.order_send(request)
+
+        order_type_str = "BUY" if is_buy else "SELL"
+        if result is None or result.retcode != mt5.TRADE_RETCODE_DONE:
+            error_message = f"Order failed for {symbol}"
+            if result:
+                error_message += f", retcode={result.retcode}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}\n"
+                f"Request: {request}\n"
+                f"Result: {result}"
+            )
+            # If you need notifications, you could log or handle them differently here.
+            log_and_print(f"Order failed details: {additional_info}", is_error=True)
+        else:
+            success_message = f"Order successful for {symbol}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}"
+            )
+            # If you need notifications, you could log or handle them differently here.
+            log_and_print(success_message)
+
+    def get_positions_by_magic(self, symbol, magic_number):
+        """
+        Retrieve positions for a specific symbol and magic number.
+        """
+        all_positions = mt5.positions_get(symbol=symbol)
+        if not all_positions:
+            log_and_print("No positions found.", is_error=False)
+            return []
+        return [pos for pos in all_positions if pos.magic == magic_number]
+
+    def run_strategy(self, symbol, lot, buy_signal, sell_signal):
+        """
+        Decide whether to open a buy or sell order based on signals,
+        close opposite positions if needed, etc.
+        """
+        log_and_print("------------------------------------------------------------------")
+        log_and_print(
+            f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}, "
+            f"SYMBOL: {symbol}, BUY SIGNAL: {buy_signal}, SELL SIGNAL: {sell_signal}"
+        )
+
+        positions = self.get_positions_by_magic(symbol, self.magic_number)
+        has_buy = any(pos.type == mt5.POSITION_TYPE_BUY for pos in positions)
+        has_sell = any(pos.type == mt5.POSITION_TYPE_SELL for pos in positions)
+
+        if buy_signal and not has_buy:
+            if has_sell:
+                log_and_print("Existing sell positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=True):
+                    log_and_print("Sell positions closed. Placing new buy order.")
+                    self.orders(symbol, lot, is_buy=True)
+                else:
+                    log_and_print("Failed to close sell positions.")
+            else:
+                self.orders(symbol, lot, is_buy=True)
+        elif sell_signal and not has_sell:
+            if has_buy:
+                log_and_print("Existing buy positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=False):
+                    log_and_print("Buy positions closed. Placing new sell order.")
+                    self.orders(symbol, lot, is_buy=False)
+                else:
+                    log_and_print("Failed to close buy positions.")
+            else:
+                self.orders(symbol, lot, is_buy=False)
+        else:
+            log_and_print("Appropriate position already exists or no signal to act on.")
+
+    def close_position(self, symbol, is_buy):
+        """
+        Close all positions of the opposite type for the given symbol & magic.
+        """
+        positions = mt5.positions_get(symbol=symbol)
+        if not positions:
+            log_and_print(f"No positions to close for symbol: {symbol}")
+            return False
+
+        initial_balance = mt5.account_info().balance
+        closed_any = False
+
+        for position in positions:
+            if position.magic == self.magic_number:
+                # if is_buy==True => we want to close SELL positions
+                # if is_buy==False => we want to close BUY positions
+                if ((is_buy and position.type == mt5.POSITION_TYPE_SELL) or
+                    (not is_buy and position.type == mt5.POSITION_TYPE_BUY)):
+                    close_request = {
+                        "action": mt5.TRADE_ACTION_DEAL,
+                        "symbol": symbol,
+                        "volume": position.volume,
+                        "type": mt5.ORDER_TYPE_BUY if position.type == mt5.POSITION_TYPE_SELL else mt5.ORDER_TYPE_SELL,
+                        "position": position.ticket,
+                        "deviation": 20,
+                        "magic": self.magic_number,
+                        "comment": COMMENT_ML,
+                        "type_time": mt5.ORDER_TIME_GTC,
+                        "type_filling": mt5.ORDER_FILLING_RETURN,
+                    }
+                    result = mt5.order_send(close_request)
+                    if result.retcode != mt5.TRADE_RETCODE_DONE:
+                        error_message = (
+                            f"Failed to close position {position.ticket} for {symbol}: {result.retcode}"
+                        )
+                        log_and_print(error_message, is_error=True)
+                        # If you need notifications, you could log or handle them differently here.
+                    else:
+                        log_and_print(f"Successfully closed position {position.ticket} for {symbol}")
+                        closed_any = True
+
+        if closed_any:
+            final_balance = mt5.account_info().balance
+            profit = final_balance - initial_balance
+            success_message = f"Closed positions successfully, Profit: {profit}"
+            log_and_print(success_message)
+            return True
+
+        return False
+
+    def check_and_execute_trades(self):
+        """
+        Called in the main loop: generate signals, run strategy, etc.
+        """
+        mt5.symbol_select(self.symbol, True)
+        buy, sell, _, _ = self.ml_signal_generation(self.symbol, N_BARS, TIMEFRAME)
+        self.run_strategy(self.symbol, self.lot_size, buy, sell)
+        mt5.symbol_select(self.symbol, False)
+        log_and_print("Waiting for new signals...")
+
+def is_market_open():
+    """
+    Check if the current time is within the typical Forex trading session, adjusted for CET/CEST.
+    Market closes at Friday 10:00 PM CET and opens at Sunday 11:00 PM CET.
+    """
+    current_time_utc = datetime.utcnow()
+    current_time_cet = (
+        current_time_utc + timedelta(hours=2) 
+        if time.localtime().tm_isdst 
+        else current_time_utc + timedelta(hours=1)
+    )
+
+    # Market closes Friday after 10 PM CET
+    if current_time_cet.weekday() == 4 and current_time_cet.hour >= 22:
+        return False
+    # Market opens Sunday after 11 PM CET
+    elif current_time_cet.weekday() == 6 and current_time_cet.hour < 23:
+        return False
+    # Closed all day Saturday
+    elif current_time_cet.weekday() == 5:
+        return False
+    return True
+
+if __name__ == "__main__":
+    try:
+        if not mt5.initialize(login=name, server=serv, password=key):
+            log_and_print("Failed to initialize MetaTrader 5", is_error=True)
+            exit()
+
+        app = TradingApp(symbol=SYMBOL, lot_size=LOT_SIZE, magic_number=MAGIC_NUMBER)
+
+        # 1) Load the classification pipeline
+        # Make sure this pipeline is a classification model expecting SHIFTED labels [0,1,2]
+        pipeline_path = "models/saved_models/best_rf_rd_pipeline.pkl"
+        app.load_pipeline(pipeline_path)
+
+        while True:
+            log_and_print("Checking market status...")
+            if is_market_open():
+                log_and_print("Market is open. Executing trades...")
+
+                # 2) Generate signals using the loaded pipeline
+                #    This pipeline is classification-based => SHIFTED labels [0,1,2]
+                #    ml_signal_generation() SHIFTs them back to [-1,0,+1] for signals
+                buy_signal, sell_signal, _, _ = app.ml_signal_generation(
+                    symbol=app.symbol,
+                    n_bars=N_BARS,
+                    timeframe=TIMEFRAME
+                )
+
+                # 3) Run strategy
+                app.run_strategy(app.symbol, app.lot_size, buy_signal, sell_signal)
+            else:
+                log_and_print("Market is closed. No actions performed.")
+
+            time.sleep(SLEEP_TIME)
+
+    except KeyboardInterrupt:
+        log_and_print("Shutdown signal received.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    except Exception as e:
+        error_message = f"An error occurred: {e}"
+        log_and_print(error_message, is_error=True)
+        # If you need a notification here, handle it (e.g., log, email, etc.).
+    finally:
+        mt5.shutdown()
+        log_and_print("MetaTrader 5 shutdown completed.")
+        # If you need a notification here, handle it (e.g., log, email, etc.).

live_trading/regression_returns.py

@@ -0,0 +1,416 @@
+import sys
+import os
+import warnings
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# 1) SET PROJECT ROOT AND UPDATE PATH/WORKING DIRECTORY
+# ---------------------------------------------------------------------------
+project_root = Path.cwd().parent.parent  # Adjust if your notebook is in notebooks/time_series
+sys.path.append(str(project_root))
+os.chdir(str(project_root))
+warnings.filterwarnings("ignore")
+import warnings
+warnings.filterwarnings("ignore")
+
+import MetaTrader5 as mt5
+import pandas as pd
+import numpy as np
+import ta
+from datetime import datetime, timedelta
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import StandardScaler
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.feature_selection import SelectFromModel
+import time
+import logging
+import joblib
+
+# Setup logging
+logging.basicConfig(
+    filename='models/saved_models/trading_app.log',
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s:%(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S'
+)
+
+def log_and_print(message, is_error=False):
+    """
+    Logs and prints a message. 
+    If is_error=True, logs at the ERROR level; otherwise logs at INFO level.
+    """
+    if is_error:
+        logging.error(message)
+    else:
+        logging.info(message)
+    print(message)
+
+# Update the login credentials and server information accordingly
+name = 52868686
+key = 'kkk7s$zzz6'
+serv = 'ICMarketsSC-Demo'
+
+# Global variables
+SYMBOL = "EURUSD"
+LOT_SIZE = 0.01
+TIMEFRAME = mt5.TIMEFRAME_D1
+N_BARS = 50000
+MAGIC_NUMBER = 234003
+SLEEP_TIME = 86400  # 4 hours in seconds
+COMMENT_ML = "regression return"
+
+def select_features_rf_reg(X, y, estimator, max_features=20):
+    """
+    Use a RandomForest (or similar) to select top 'max_features' features.
+    """
+    selector = SelectFromModel(estimator=estimator, threshold=-np.inf, max_features=max_features).fit(X, y)
+    X_transformed = selector.transform(X)
+    selected_features_mask = selector.get_support()
+    return X_transformed, selected_features_mask
+
+class TradingApp:
+    def __init__(self, symbol, lot_size, magic_number):
+        self.symbol = symbol
+        self.lot_size = lot_size
+        self.magic_number = magic_number
+        self.pipeline = None  # We'll store the loaded pipeline here
+        self.last_retrain_time = None
+
+    def get_data(self, symbol, n, timeframe):
+        """
+        Fetch 'n' bars of historical data for the given symbol and timeframe.
+        """
+        rates = mt5.copy_rates_from_pos(symbol, timeframe, 0, n)
+        rates_frame = pd.DataFrame(rates)
+        rates_frame['time'] = pd.to_datetime(rates_frame['time'], unit='s')
+        rates_frame.set_index('time', inplace=True)
+        return rates_frame
+
+    def add_all_ta_features(self, df):
+        """
+        Add technical analysis features to the DataFrame using the 'ta' library.
+        """
+        df = ta.add_all_ta_features(
+            df,
+            open="open",
+            high="high",
+            low="low",
+            close="close",
+            volume="tick_volume",
+            fillna=True
+        )
+        return df
+
+    def load_pipeline(self, pipeline_path):
+        """
+        Load a pre-trained pipeline (scaler + model + possibly feature selection)
+        from disk, e.g. 'best_rf_pipeline.pkl'.
+        """
+        self.pipeline = joblib.load(pipeline_path)
+        logging.info(f"Loaded pipeline from {pipeline_path}")
+
+    def ml_signal_generation(self, symbol, n_bars, timeframe):
+        """
+        Generate buy/sell signals using the loaded pipeline.
+        Make sure the pipeline expects the same features as we create below.
+        """
+        if self.pipeline is None:
+            logging.error("No pipeline loaded. Call load_pipeline(...) first.")
+            return False, False, True, True
+
+        # 1) Fetch new data
+        df = self.get_data(symbol, n_bars, timeframe)
+        # 2) Add TA features (if your pipeline doesn't handle feature eng, do it here)
+        df = self.add_all_ta_features(df)
+        df.fillna(method='ffill', inplace=True)
+
+        # 3) Prepare the features (the pipeline will do scaling/selection if included)
+        X_new = df  # If your pipeline expects specific columns, subset accordingly.
+
+        # 4) Predict with the pipeline
+        predictions = self.pipeline.predict(X_new)
+        latest_pred = predictions[-1]  # Get the most recent bar's prediction
+
+        buy_signal = latest_pred > 0
+        sell_signal = latest_pred < 0
+
+        return buy_signal, sell_signal, not buy_signal, not sell_signal
+
+    def calculate_future_returns(self, df):
+        """
+        (Optional) Example function to calculate future returns for labeling.
+        """
+        df["future_returns"] = df["close"].pct_change().shift(-1)
+        return df.dropna()
+
+    def orders(self, symbol, lot, is_buy=True, id_position=None, sl=None, tp=None):
+        """
+        Place an order (BUY or SELL) for the specified symbol and lot size.
+        """
+        symbol_info = mt5.symbol_info(symbol)
+        if symbol_info is None:
+            log_and_print(f"Symbol {symbol} not found, can't place order.", is_error=True)
+            return "Symbol not found"
+
+        # Make sure symbol is selected/visible
+        if not symbol_info.visible:
+            if not mt5.symbol_select(symbol, True):
+                log_and_print(f"Failed to select symbol {symbol}", is_error=True)
+                return "Symbol not visible or could not be selected."
+
+        tick_info = mt5.symbol_info_tick(symbol)
+        if tick_info is None:
+            log_and_print(f"Could not get tick info for {symbol}.", is_error=True)
+            return "Tick info unavailable"
+
+        # Check for valid bid/ask
+        if tick_info.bid <= 0 or tick_info.ask <= 0:
+            log_and_print(
+                f"Zero or invalid bid/ask for {symbol}: bid={tick_info.bid}, ask={tick_info.ask}",
+                is_error=True
+            )
+            return "Invalid prices"
+
+        # ----------- LOT SIZE VALIDATION -----------
+        lot = max(lot, symbol_info.volume_min)
+        step = symbol_info.volume_step
+        if step > 0:
+            remainder = lot % step
+            if remainder != 0:
+                lot = lot - remainder + step
+        if lot > symbol_info.volume_max:
+            lot = symbol_info.volume_max
+
+        log_and_print(
+            f"Adjusted lot size to {lot} (min={symbol_info.volume_min}, "
+            f"step={symbol_info.volume_step}, max={symbol_info.volume_max})"
+        )
+
+        # ----------- FORCE ORDER_FILLING_IOC -----------
+        filling_mode = 1  # ORDER_FILLING_IOC
+
+        order_type = mt5.ORDER_TYPE_BUY if is_buy else mt5.ORDER_TYPE_SELL
+        deviation = 20
+
+        request = {
+            "action": mt5.TRADE_ACTION_DEAL,
+            "symbol": symbol,
+            "volume": lot,
+            "type": order_type,
+            "deviation": deviation,
+            "magic": self.magic_number,
+            "comment": COMMENT_ML,
+            "type_time": mt5.ORDER_TIME_GTC,
+            "type_filling": filling_mode,
+        }
+
+        if sl is not None:
+            request["sl"] = sl
+        if tp is not None:
+            request["tp"] = tp
+        if id_position is not None:
+            request["position"] = id_position
+
+        log_and_print(f"Sending order request: {request}")
+        result = mt5.order_send(request)
+
+        order_type_str = "BUY" if is_buy else "SELL"
+        if result is None or result.retcode != mt5.TRADE_RETCODE_DONE:
+            error_message = f"Order failed for {symbol}"
+            if result:
+                error_message += f", retcode={result.retcode}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}\n"
+                f"Request: {request}\n"
+                f"Result: {result}"
+            )
+            log_and_print(f"Order failed details: {additional_info}", is_error=True)
+        else:
+            success_message = f"Order successful for {symbol}, comment={result.comment}"
+            additional_info = (
+                f"Date/Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
+                f"Order Type: {order_type_str}\n"
+                f"Lot Size: {lot}\n"
+                f"SL: {sl if sl else 'None'}\n"
+                f"TP: {tp if tp else 'None'}\n"
+                f"Comment: {COMMENT_ML}"
+            )
+            log_and_print(success_message)
+
+    def get_positions_by_magic(self, symbol, magic_number):
+        """
+        Retrieve open positions for the specified symbol and magic number.
+        """
+        all_positions = mt5.positions_get(symbol=symbol)
+        if not all_positions:
+            log_and_print("No positions found.", is_error=False)
+            return []
+        return [pos for pos in all_positions if pos.magic == magic_number]
+
+    def run_strategy(self, symbol, lot, buy_signal, sell_signal):
+        """
+        Based on buy/sell signals, decide whether to open or close positions.
+        """
+        log_and_print("------------------------------------------------------------------")
+        log_and_print(
+            f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}, "
+            f"SYMBOL: {symbol}, BUY SIGNAL: {buy_signal}, SELL SIGNAL: {sell_signal}"
+        )
+
+        # Retrieve positions based on the magic number to manage trades specific to this instance
+        positions = self.get_positions_by_magic(symbol, self.magic_number)
+        has_buy = any(pos.type == mt5.POSITION_TYPE_BUY for pos in positions)
+        has_sell = any(pos.type == mt5.POSITION_TYPE_SELL for pos in positions)
+
+        # Decision making based on current signals and existing positions
+        if buy_signal and not has_buy:
+            if has_sell:
+                log_and_print("Existing sell positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=True):
+                    log_and_print("Sell positions closed. Placing new buy order.")
+                    self.orders(symbol, lot, is_buy=True)
+                else:
+                    log_and_print("Failed to close sell positions.")
+            else:
+                self.orders(symbol, lot, is_buy=True)
+        elif sell_signal and not has_sell:
+            if has_buy:
+                log_and_print("Existing buy positions found. Attempting to close...")
+                if self.close_position(symbol, is_buy=False):
+                    log_and_print("Buy positions closed. Placing new sell order.")
+                    self.orders(symbol, lot, is_buy=False)
+                else:
+                    log_and_print("Failed to close buy positions.")
+            else:
+                self.orders(symbol, lot, is_buy=False)
+        else:
+            log_and_print("Appropriate position already exists or no signal to act on.")
+
+    def close_position(self, symbol, is_buy):
+        """
+        Closes positions of the opposite type (BUY/SELL) for this app's magic number.
+        """
+        positions = mt5.positions_get(symbol=symbol)
+        if not positions:
+            log_and_print(f"No positions to close for symbol: {symbol}")
+            return False
+
+        initial_balance = mt5.account_info().balance
+        closed_any = False
+
+        for position in positions:
+            # Close positions of the opposite type with the same magic number
+            if position.magic == self.magic_number and (
+                (is_buy and position.type == mt5.POSITION_TYPE_SELL) or
+                (not is_buy and position.type == mt5.POSITION_TYPE_BUY)
+            ):
+                close_request = {
+                    "action": mt5.TRADE_ACTION_DEAL,
+                    "symbol": symbol,
+                    "volume": position.volume,
+                    "type": mt5.ORDER_TYPE_BUY if position.type == mt5.POSITION_TYPE_SELL else mt5.ORDER_TYPE_SELL,
+                    "position": position.ticket,
+                    "deviation": 20,
+                    "magic": self.magic_number,
+                    "comment": COMMENT_ML,
+                    "type_time": mt5.ORDER_TIME_GTC,
+                    "type_filling": mt5.ORDER_FILLING_RETURN,
+                }
+                result = mt5.order_send(close_request)
+                if result.retcode != mt5.TRADE_RETCODE_DONE:
+                    error_message = (
+                        f"Failed to close position {position.ticket} for {symbol}: {result.retcode}"
+                    )
+                    log_and_print(error_message, is_error=True)
+                else:
+                    log_and_print(f"Successfully closed position {position.ticket} for {symbol}")
+                    closed_any = True
+
+        if closed_any:
+            final_balance = mt5.account_info().balance
+            profit = final_balance - initial_balance
+            success_message = f"Closed positions successfully, Profit: {profit}"
+            log_and_print(success_message)
+            return True
+
+        return False
+
+    def check_and_execute_trades(self):
+        """
+        Convenience method to perform the entire flow: 
+        generate signals, run strategy, and deselect symbol.
+        """
+        mt5.symbol_select(self.symbol, True)
+        buy, sell, _, _ = self.ml_signal_generation(self.symbol, N_BARS, TIMEFRAME)
+        self.run_strategy(self.symbol, self.lot_size, buy, sell)
+        mt5.symbol_select(self.symbol, False)
+        log_and_print("Waiting for new signals...")
+
+def is_market_open():
+    """
+    Check if the current time is within typical Forex trading session hours (CET/CEST).
+    - Closes: Friday 10:00 PM CET
+    - Opens: Sunday 11:00 PM CET
+    - Closed all day Saturday
+    """
+    current_time_utc = datetime.utcnow()
+    # Adjust for CET (UTC+1) or CEST (UTC+2)
+    current_time_cet = current_time_utc + timedelta(hours=2) if time.localtime().tm_isdst else current_time_utc + timedelta(hours=1)
+
+    # Friday after 10 PM CET
+    if current_time_cet.weekday() == 4 and current_time_cet.hour >= 22:
+        return False
+    # Sunday before 11 PM CET
+    elif current_time_cet.weekday() == 6 and current_time_cet.hour < 23:
+        return False
+    # All day Saturday
+    elif current_time_cet.weekday() == 5:
+        return False
+    return True
+
+if __name__ == "__main__":
+    try:
+        if not mt5.initialize(login=name, server=serv, password=key):
+            log_and_print("Failed to initialize MetaTrader 5", is_error=True)
+            exit()
+
+        app = TradingApp(symbol=SYMBOL, lot_size=LOT_SIZE, magic_number=MAGIC_NUMBER)
+
+        # Load a previously trained pipeline (scaler + model, etc.)
+        pipeline_path = "models/saved_models/best_rf_pipeline.pkl"
+        app.load_pipeline(pipeline_path)
+        log_and_print(f"Loaded final pipeline for {app.symbol}")
+
+        while True:
+            log_and_print("Checking market status...")
+            if is_market_open():
+                log_and_print("Market is open. Executing trades...")
+
+                # Generate signals using the loaded pipeline
+                buy_signal, sell_signal, _, _ = app.ml_signal_generation(
+                    symbol=app.symbol,
+                    n_bars=N_BARS,
+                    timeframe=TIMEFRAME
+                )
+
+                # Run strategy
+                app.run_strategy(app.symbol, app.lot_size, buy_signal, sell_signal)
+            else:
+                log_and_print("Market is closed. No actions performed.")
+
+            # Sleep for the configured interval (e.g., 4 hours)
+            time.sleep(SLEEP_TIME)
+
+    except KeyboardInterrupt:
+        log_and_print("Shutdown signal received.")
+    except Exception as e:
+        error_message = f"An error occurred: {e}"
+        log_and_print(error_message, is_error=True)
+    finally:
+        mt5.shutdown()
+        log_and_print("MetaTrader 5 shutdown completed.")