fx-quant/README.md

# fx-quant

Algorithmic FX trading system with backtesting, paper/live execution via OANDA, and a Flask web dashboard.

## Quick Start (New Machine Setup)

### 1. Clone & install dependencies

```bash
git clone <your-repo-url>
cd fx-quant
python -m pip install -r requirements.txt
```

### 2. Configure environment variables

Copy the template and fill in your credentials:

```bash
cp config/.env.example config/.env
```

Required variables in `config/.env`:

| Variable | Description |
|----------|-------------|
| `SUPABASE_URL` | Your Supabase project URL |
| `SUPABASE_KEY` | Supabase anon/service key |
| `OANDA_API_KEY` | OANDA v20 API token |
| `OANDA_ACCOUNT_ID` | OANDA account ID |
| `OANDA_ENV` | `practice` or `live` (default: practice) |
| `DASHBOARD_PASSWORD` | Password for web dashboard login |
| `TRADING_ECONOMICS_API_KEY` | Trading Economics API key (for economic calendar) |

### 3. Verify connectivity

```bash
python src/test_connection.py
```

### 4. Load historical data (if Supabase table is empty)

```bash
python src/historical_loader.py
```

### 5. Run the backtester

```bash
python src/backtester.py
```

Output goes to `logs/`:
- `backtest_trades_<INSTRUMENT>_<GRANULARITY>.csv` -- trade log
- `backtest_summary_<INSTRUMENT>_<GRANULARITY>.json` -- metrics + monthly P&L

### 6. Visualize trades on a chart

```bash
python src/chart_trades.py --start 2025-02-24 --end 2025-02-27
python src/chart_trades.py --granularity M15 --start 2025-04-01 --end 2025-04-15
```

Saves PNG charts to `logs/chart_trades_*.png`.

---

## Project Structure

```
fx-quant/
  config/
    system.yaml          # Main configuration (strategy, features, execution)
    .env                 # Secrets (not committed)
  src/
    config_loader.py     # Loads system.yaml + .env
    data_engine.py       # Feature engineering (SMA, EMA, RSI, ATR, VWAP, pivots, engulfing)
    backtester.py        # Backtesting engine (SMA cross + pivot retest strategies)
    order_executor.py    # Paper/live order execution via OANDA
    ai_wrapper.py        # ML ensemble (logistic reg, RF, gradient boosting) signal validation
    dashboard.py         # Flask web dashboard
    chart_trades.py      # Matplotlib/mplfinance trade visualization
    get_candles.py       # Fetch candles from OANDA API
    economic_calendar.py # Economic calendar fetch/upload pipeline
    historical_loader.py # Bulk historical data loader
    supabase_upload.py   # Upload candle data to Supabase
    param_sweep.py       # Strategy parameter optimization
    test_connection.py   # OANDA + Supabase connectivity check
  templates/             # Flask HTML templates (chart, backtest, config, logs)
  logs/                  # Output: trade CSVs, JSON summaries, chart PNGs
  models/                # Saved ML models
  sql/                   # Database schemas
  Dockerfile             # Container (live engine default CMD)
  docker-compose.yml     # live-engine + dashboard services
  railway.toml           # Railway build/deploy config
  requirements.txt       # Python dependencies (portable)
  requirements.docker.txt# Docker-specific deps
```

## Phase 1: Strategy Backtesting Status

Backtested 6 strategy families across 10 currency pairs on 2021-2024 data. Full results in `results/phase1/`, detailed learnings in `results/STRATEGY_LEARNINGS.md`.

### Working Strategies

**S3 — Key Level Momentum Breakout** (H1 timeframe) — Best performer
| Pair | Trades | Win Rate | PF | PnL (pips) | Max DD |
|------|--------|----------|-----|------------|--------|
| GBP_JPY | 155 | 52.3% | 0.99 | +408 | -13.5% |
| GBP_USD | 179 | 53.1% | 1.02 | +97 | -10.2% |
| USD_JPY | 138 | 52.9% | 1.00 | +66 | -9.8% |

Breakout of horizontal S/R levels (3+ touch clusters) with volume confirmation, strong candle close, MACD alignment, and ADX > 20. Simple, 4-filter approach on H1. Needs SL/TP tuning to push PF above 1.0 consistently.

### Strategies With Potential (Need Tweaks)

**S4-F — EMA Ribbon Trend Context** (EUR_AUD only): 95 trades, 45.3% WR, PF 1.06, +173 pips. Only profitable on EUR_AUD. Needs pair-specific tuning and SL/TP restructuring.

**S6 — EMA Bounce** (EUR_AUD/GBP_USD): 59-60% win rate but PF 0.83-0.84. Win rate is strong — needs tighter SL or trailing stop to fix risk/reward.

### Strategies Retired

| Strategy | Issue | Status |
|----------|-------|--------|
| S1 — MA Breakout | 35-47% WR, PF 0.40-0.88 | No edge |
| S2 — VWAP Reversal | 20-25% WR, 26 consecutive losses | Disabled |
| S4 — EMA Ribbon (6 variants) | Extensively tested D/E/F/F-v2/G/G-Minimal. Only EUR_AUD S4-F marginal. | Exhausted |
| S5 — Momentum Exhaustion | High trade count but PF 0.43-0.77 | Too much noise |

### Key Learnings
1. **Momentum + mean-reversion filters are contradictory** — don't combine in one strategy
2. **3-4 hard filters max** — more gates compound multiplicatively and kill trade count
3. **Always validate thresholds against data distributions** before running backtests
4. **Simple strategies beat complex ones** — S3 (4 filters) outperforms S4 (7+ filters)
5. **H1 timeframe has natural edge** — lower timeframes (M5/M15) struggle with noise

Full filter analysis and design principles in [`results/STRATEGY_LEARNINGS.md`](results/STRATEGY_LEARNINGS.md).

### Next Phase
Moving to Smart Money / institutional flow strategies. Will also revisit S3 (SL/TP tuning, expanded pairs) and S6 (risk/reward restructuring).

---

## Phase 2: Live Paper Trading + Extended Analytics

Phase 2 advances 5 strategies to live paper trading and extended backtesting analytics.

### Resuming on a New Machine

```bash
git clone https://github.com/BrentNeale1/fx-quant.git
cd fx-quant
pip install pandas numpy requests pyyaml
```

**Set your OANDA credentials** in `config/.env`:
```
OANDA_API_KEY=your-key-here
OANDA_ACCOUNT_ID=your-account-id    # <-- REQUIRED, currently empty
OANDA_ENV=practice
```

The API key is already populated. You need to add your **OANDA_ACCOUNT_ID** (find it in the OANDA fxTrade Practice platform under Account Settings).

### Running the Live Engine

```bash
# Single cycle (test connectivity + signal checks)
python src/live/run.py --once

# Continuous mode (polls every 60s, runs overnight)
python src/live/run.py
```

The engine runs 5 strategy slots in paper mode:

| Slot | Strategy | Pair | Timeframe | Signal Type |
|------|----------|------|-----------|-------------|
| 1 | S7 Tight | GBP_JPY | H1 | Liquidity sweep reversal |
| 2 | S9 | GBP_USD | H1 | London session breakout |
| 3 | S9 Filtered | GBP_AUD | H1 | London session + per-pair filters |
| 4 | S4-F | EUR_AUD | M15 + H1 | EMA ribbon trend context |
| 5 | S3 | GBP_JPY | H1 | Key level momentum breakout |

**Signals fire during London/NY sessions (07:00-17:00 UTC)**. Running outside those hours will show "No signal" which is expected.

### Monitoring

- **Live state**: `logs/live_state.json` (equity, open positions per slot)
- **Trade log**: `logs/live_trades.csv` (closed trades with PnL)
- **Paper orders**: `logs/paper_trades.csv` (all order attempts)
- **Kill switch**: Create `STOP_ALL_TRADING` file in project root to halt

### Running Analytics

```bash
# Per-year performance breakdown (2021/2022/2023)
python src/run_regime_analysis.py

# Signal overlap + portfolio metrics
python src/run_correlation_analysis.py

# Kelly criterion + Monte Carlo drawdown simulation
python src/run_kelly_sizing.py
```

Results output to `results/phase2/`.

### Phase 2 Analytics Summary

**Regime Analysis** — 4 of 5 strategies show improving PF over time:
| Strategy | 2021 PF | 2022 PF | 2023 PF | Trend |
|----------|---------|---------|---------|-------|
| S7 Tight / GBP_JPY | 0.47 | 0.89 | 1.80 | UP |
| S9 / GBP_USD | 0.71 | 0.83 | 1.38 | UP |
| S9 Filtered / GBP_AUD | 0.58 | 2.57 | 2.98 | UP |
| S4F / EUR_AUD | 1.11 | 1.79 | 0.48 | DOWN |
| S3 / GBP_JPY | 0.90 | 1.21 | 1.07 | UP |

**Correlation** — S7+S3 on GBP_JPY: 16.9% overlap (moderate, all same-direction). S9 vs S9_Filtered: 12% temporal overlap (good diversification).

**Kelly Sizing** — S9_Filtered: half-Kelly 7.3% (p95 DD 4.8%). S4F: 2.4%. S3: 1.6%. S7/S9 base: Kelly<=0 on full dataset.

### Railway Cloud Deployment (24/7)

The live engine and dashboard are deployed to [Railway](https://railway.com) for 24/7 operation without a local machine.

| Service | Description | URL |
|---------|-------------|-----|
| **live-engine** | Runs `src/live/run.py`, polls every 60s | Internal (no public URL) |
| **dashboard** | Flask web dashboard | https://dashboard-production-73e7.up.railway.app |

**Dashboard login:** `admin` / (password in Railway env vars)

**Infrastructure:**
- Both services auto-deploy on push to `main`
- Health checks on `/health` for both services
- Persistent volume mounted at `/app/logs` on live-engine (survives redeployments)
- Restart policy: on-failure with max 5 retries

**Railway environment variables:**

| Service | Variables |
|---------|-----------|
| live-engine | `OANDA_API_KEY`, `OANDA_ACCOUNT_ID`, `OANDA_ENV=practice` |
| dashboard | `DASHBOARD_PASSWORD`, `OANDA_API_KEY`, `OANDA_ACCOUNT_ID`, `OANDA_ENV` |

**Known limitations:**
- Kill switch toggle from dashboard won't reach the engine (separate containers)
- `/chart` route on dashboard needs Supabase credentials

### Phase 2 File Structure

```
src/
  position_manager.py        # Shared Position/TradeRecord + PositionManager
  live/
    __init__.py
    data_feed.py             # OANDA candle polling + indicator computation
    executor.py              # Paper/live order execution
    engine.py                # LiveEngine orchestrator (5 strategy slots)
    run.py                   # Entry point (--once or continuous)
    health.py                # Threaded HTTP health server for Railway
  run_regime_analysis.py     # Per-year performance breakdown
  run_correlation_analysis.py# Signal overlap + portfolio metrics
  run_kelly_sizing.py        # Kelly criterion + Monte Carlo
config/
  system.yaml                # phase2: section with strategy slot config
results/
  phase2/
    regime_analysis.json
    correlation_analysis.json
    kelly_sizing.json
```

---

## Strategies (Legacy Reference)

### SMA Cross (original)

Long-only strategy. Goes long when short SMA > long SMA, flat otherwise.

```yaml
strategy:
  rule: sma_cross
  params:
    short: 50
    long: 100
```

### Pivot Retest + Engulfing

Long/short strategy with dual take-profit and ATR-based stop loss.

**Entry conditions (all must be true):**
- Price retests a pivot level (broke through, then returned within ATR tolerance)
- SMA 50 aligns with trade direction relative to the pivot level
- Engulfing candle pattern confirmed
- Strong close (in top/bottom 30% of candle range)

**Position management:**
- SL: 1.5x ATR from entry
- TP1: next pivot level in trade direction (close 50%, move SL to breakeven)
- TP2: pivot level after TP1 (close remaining 50%)

```yaml
strategy:
  rule: pivot_retest_engulfing
  params:
    sma_period: 50
    lookback_bars: 20
    retest_tolerance_atr: 0.5
    strong_close_pct: 0.30
    sl_atr_multiplier: 1.5
```

To switch strategies, edit `config/system.yaml` and change `strategy.rule`.

## Configuration Reference

All settings live in `config/system.yaml`:

| Section | Key settings |
|---------|-------------|
| `brokers[0].instruments` | Currency pairs to trade (e.g. `EUR_USD`) |
| `data.candle_granularities` | Timeframes (`M5`, `M15`, `H1`, etc.) |
| `features.*` | Indicator windows (SMA, EMA, RSI, ATR, VWAP, volatility) |
| `strategy.*` | Active strategy rule + parameters |
| `ai.*` | ML ensemble config, confidence threshold, sanity checks |
| `execution.paper_mode` | `true` for paper trading, `false` for live |
| `execution.interval_seconds` | Bot loop interval |
| `execution.max_positions` | Max concurrent open trades |

## Web Dashboard

Flask-based UI for remote management.

```bash
# Railway (already deployed)
# https://dashboard-production-73e7.up.railway.app

# Docker (local)
docker-compose up -d

# Or run directly
python src/dashboard.py
```

Pages: Status (`/`), Backtest (`/backtest`), Chart (`/chart`), Config (`/config`), Logs (`/logs`), Health (`/health`)

## Kill Switch

Create `STOP_ALL_TRADING` in the project root to halt all trading immediately. The bot checks for this file every loop iteration. In live mode, it also closes all open trades. Toggle via the dashboard or manually:

```bash
touch STOP_ALL_TRADING   # activate
rm STOP_ALL_TRADING      # deactivate
```

## Economic Calendar

Filters trade entries near high-impact economic events (NFP, CPI, rate decisions, etc.) to reduce slippage and false signals. Uses the [Trading Economics API](https://tradingeconomics.com/api/).

**Setup:**
1. Subscribe to a Trading Economics API plan that includes Calendar data ([pricing](https://tradingeconomics.com/api/pricing.aspx))
2. Add your API key to `config/.env`: `TRADING_ECONOMICS_API_KEY=your-key-here`
3. Create the Supabase table: run `sql/create_economic_calendar.sql` in the SQL Editor
4. Fetch events: `python src/economic_calendar.py` (full load) or `python src/economic_calendar.py --incremental`

**Config** (`config/system.yaml` → `economic_calendar`):

| Key | Default | Description |
|-----|---------|-------------|
| `enabled` | `true` | Toggle calendar filter on/off |
| `impact_threshold` | `High` | Minimum impact level to block entries (`Low`, `Medium`, `High`) |
| `event_buffer_minutes` | `30` | Block entries within this many minutes of an event |
| `days_back` | `400` | How far back to fetch on full load |

**Status:** Table created in Supabase. Awaiting Trading Economics API key (subscription request pending). Once the key is obtained, run the loader and the backtester will automatically filter entries near high-impact events.

## Running the Bot

```bash
# Railway (24/7, already deployed)
# Live engine runs automatically on push to main

# Local: single cycle
python src/live/run.py --once

# Local: continuous loop (60s interval)
python src/live/run.py

# Docker (local)
docker-compose up -d
```