mt5-ai-xauusd-trader/MONITORING_ALERTING.md

Monitoring & Alerting Strategy: MT5 AI/ML XAUUSD Trading Bot

Executive Summary

Comprehensive real-time monitoring and intelligent alerting system providing 24/7 visibility into trading bot health, performance, and profitability. Enables proactive issue detection and rapid incident response.

1.1 Health Check Endpoints (Enterprise)

The system provides specialized HTTP endpoints for automated health monitoring and container orchestration:

• /health/liveness: Lightweight probe for process heartbeat. Returns 200 OK if the FastAPI server is responsive.
• /health/readiness: Deep dependency check. Verifies connectivity to MT5, Database, and ensures AI Models are correctly loaded. Returns 503 Service Unavailable if critical dependencies are unreachable.
• /health/full: Comprehensive diagnostic report including hardware stats, configuration validity, and individual component health.
• /metrics: Standard Prometheus metrics scrape target.

1. Monitoring Architecture

1.1 Monitoring Stack

• Metrics Collection: Prometheus for time-series metrics
• Log Aggregation: ELK Stack (Elasticsearch, Logstash, Kibana)
• Tracing: Jaeger for distributed tracing and internal end-to-end trace correlation using UUIDs
• Visualization: Grafana for dashboards
• Alerting: Prometheus AlertManager + PagerDuty
• Data Retention: 30 days hot, 1 year cold storage

1.2 Instrumentation

• Application Metrics: Custom metrics per module
• System Metrics: CPU, memory, disk, network
• Business Metrics: Trades, P&L, Sharpe ratio
• Infrastructure Metrics: Container, Kubernetes, database
• Custom Metrics: Domain-specific trading metrics

1.3 Data Pipeline

• Collection: 30-second scrape interval
• Aggregation: 5-minute aggregates
• Storage: Time-series database (InfluxDB/Prometheus)
• Visualization: Real-time dashboards
• Archival: Automatic cold storage after 30 days

2. Key Metrics

2.1 Trading Performance Metrics

• Total Trades: Cumulative trade count
• Win Rate: Percentage of profitable trades
• Sharpe Ratio: Risk-adjusted returns (target >2.0)
• Maximum Drawdown: Peak-to-trough decline (target <25%)
• Daily P&L: Profit/loss per day
• Monthly Return: Monthly cumulative return
• Average Trade Duration: Mean trade holding time

2.2 Execution Metrics

• Order Execution Latency: Time from signal to execution (target <100ms)
• Slippage: Difference between expected and actual price
• Fill Rate: Percentage of orders filled at intended price
• Partial Fills: Count and percentage of partial fills
• Rejected Orders: Count and reason for rejections
• Order Errors: Error count by type

2.3 System Health Metrics

• CPU Utilization: Percentage (target <70%)
• Memory Usage: MB and percentage (target <80%)
• Disk Usage: GB and percentage (alert <10% free)
• Network I/O: Bytes in/out
• Process Uptime: Seconds since startup
• Error Rate: Errors per minute

2.4 Model Metrics

• Inference Latency: Time to generate prediction (target <10ms)
• Prediction Accuracy: Percentage of correct predictions
• Model Drift: Statistical drift from baseline
• Feature Freshness: Age of latest features
• Retraining Status: Last training time
• Model Version: Current model deployed

2.5 API Metrics

• Request Rate: Requests per second
• Error Rate: Errors per 1000 requests (target <1)
• Latency p50/p95/p99: Response time percentiles
• Active Connections: Current connection count
• Throughput: MB/s processed
• Rate Limit Hits: Rate limiting violations

2.6 Data Quality Metrics

• Data Freshness: Age of latest data point
• Data Completeness: Percentage of non-null values
• Data Validation Failures: Count of validation errors
• Missing Data Points: Count of missing candles
• Data Latency: End-to-end data pipeline delay
• Reconciliation Mismatches: Broker vs local balance differences
• Trace Correlation: Consistency of UUID propagation across logs and database records

3. Dashboards

3.1 Executive Dashboard

• YTD Performance: Year-to-date returns
• Monthly Return: Current month P&L
• Sharpe Ratio: Risk-adjusted performance
• Maximum Drawdown: Current drawdown status
• Win Rate: Trading success percentage
• Trade Count: Total trades to date
• System Status: Green/yellow/red status
• Latest Trades: Recent trading activity

3.2 Operations Dashboard

• System Health: CPU, memory, disk status
• API Health: Request rate, error rate, latency
• Trading Metrics: Win rate, P&L, trade count
• Execution Quality: Slippage, fill rate, latency
• Data Quality: Freshness, completeness, latency
• Error Logs: Recent errors and warnings
• Alerts: Active and recent alerts
• Model Status: Current model, last training, drift

3.3 Performance Dashboard

• Request Latency: p50, p95, p99 over time
• Throughput: Requests per second trending
• Error Rate: Percentage of errors trending
• Resource Usage: CPU, memory, disk trending
• Model Inference: Prediction latency trending
• Database Performance: Query latency, connection count
• Network I/O: Bytes in/out trending
• Garbage Collection: GC pause times

3.4 Trading Dashboard

• Equity Curve: Daily equity value over time
• Drawdown Chart: Drawdown percentage over time
• Win/Loss Ratio: Win rate trending
• Daily P&L: Daily profit/loss bars
• Trade Distribution: Trade size distribution
• Holding Time: Average trade duration trending
• Win/Loss per Trade: Average winning/losing trade
• Risk Metrics: Volatility, correlation, beta

4. Alerting Strategy

4.1 Alert Severity Levels

• Critical (P1): Immediate action required, trading impacted

  • Response time: 5 minutes
  • Escalation: Auto-escalate after 15 minutes
  • Page: Immediate page to on-call engineer

• High (P2): Action needed within 1 hour, trading degraded

  • Response time: 30 minutes
  • Escalation: Auto-escalate after 1 hour
  • Page: During business hours, ticket during off-hours

• Medium (P3): Action needed within 24 hours, monitoring issue

  • Response time: 2 hours
  • Ticket: Create ticket automatically
  • Page: No immediate page

• Low (P4): Information only, no immediate action needed

  • Response time: Next business day
  • Ticket: Optional ticket creation
  • Page: No paging

4.2 Trading-Related Alerts (Critical)

• Trading Bot Offline: Process not running >5 minutes
• Broker Connection Lost: MT5 connection down >2 minutes
• Account Balance Mismatch: Broker balance != local balance >5%
• Margin Call Alert: Account margin ratio <120%
• Liquidity Crisis: Bid-ask spread >10x normal
• Max Drawdown Exceeded: Current DD > 30% (emergency stop)
• Order Failure Rate: >10% of orders rejected
• Execution Latency: >1 second latency

4.3 System Alerts (Critical)

• High Memory: >90% memory usage
• Disk Full: <5% free disk space
• CPU Throttled: CPU >95% sustained >5 minutes
• Database Connection Pool Exhausted: All connections in use
• Network Connectivity: Latency >100ms to broker
• Service Unhealthy: Health check failed >3 times
• Crash Loop: Process restarting >3x in 10 minutes

4.4 Model Alerts (High)

• Model Drift Detected: Statistical drift > threshold
• Prediction Accuracy Drop: Accuracy <baseline-10%
• Model Inference Timeout: Prediction >100ms
• Feature Missing: Feature not available
• Model Load Failed: Cannot load model at startup
• Stale Model: Model older than 7 days
• Training Failed: Model retraining failed

4.5 Data Quality Alerts (Medium)

• Data Freshness: Data older than 5 minutes
• Missing Data Points: >10% missing candles
• Data Validation Failures: >5% validation errors
• Late Data Arrival: Data arriving >10 seconds late
• Duplicate Records: Duplicate trades detected
• Data Inconsistency: Inconsistent data patterns

4.6 Performance Alerts (Medium)

• API Latency High: p95 latency >500ms
• Error Rate Elevated: Error rate >0.5%
• Throughput Low: <50% of expected throughput
• Database Slow: Query latency >1 second
• GC Pause Long: GC pause >500ms

5. Alert Configuration Examples

5.1 Prometheus Alert Rules

groups:
- name: trading-bot
  rules:
  # Critical: Trading bot offline
  - alert: TradingBotOffline
    expr: up{job="trading-bot"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Trading bot is offline"

  # Critical: High error rate
  - alert: HighErrorRate
    expr: rate(errors_total[5m]) > 0.01
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Error rate is {{ $value }} per second"

  # High: Model drift detected
  - alert: ModelDriftDetected
    expr: model_drift_score > 0.3
    for: 10m
    labels:
      severity: high
    annotations:
      summary: "Model drift score: {{ $value }}"

  # Medium: High memory usage
  - alert: HighMemoryUsage
    expr: memory_usage_percent > 85
    for: 5m
    labels:
      severity: medium
    annotations:
      summary: "Memory usage is {{ $value }}%"

6. Notification Channels

6.1 Alert Routing

• Critical (P1):

  • PagerDuty (immediate)
  • SMS (immediate)
  • Slack #alerts-critical
  • Email

• High (P2):

  • PagerDuty (business hours)
  • Slack #alerts-high
  • Email

• Medium (P3):

  • Slack #alerts-medium
  • Email (daily digest)
  • Jira ticket

• Low (P4):

  • Slack #alerts-low
  • Email (weekly digest)

6.2 On-Call Rotation

• Primary: Engineer on-call
• Secondary: Backup engineer
• Escalation: Team lead after 30 minutes
• Rotation: Weekly rotation schedule

7. Incident Response

7.1 Incident Triage

1. Acknowledge alert within 5 minutes
2. Verify issue is real (not false positive)
3. Classify severity level
4. Notify stakeholders
5. Begin investigation

7.2 Incident Timeline

• T+0: Alert fires
• T+5m: On-call acknowledges
• T+10m: Incident classification
• T+15m: Escalation if critical
• T+1h: Root cause analysis
• T+2h: Remediation actions
• T+24h: Post-mortem review

7.3 Runbooks

Maintain runbooks for common incidents:

• Trading bot offline recovery
• Broker connection loss recovery
• Database connection issues
• Model inference timeout
• High error rate troubleshooting
• Drawdown excessive recovery

8. Metrics Collection Code Examples

8.1 Python Prometheus Client

from prometheus_client import Counter, Histogram, Gauge

# Metrics definition
trade_counter = Counter('trades_total', 'Total trades executed', ['side'])
error_counter = Counter('errors_total', 'Total errors', ['error_type'])
loss_gauge = Gauge('account_pnl', 'Account P&L')
latency_histogram = Histogram('execution_latency_ms', 'Execution latency')

# Usage
trade_counter.labels(side='buy').inc()
error_counter.labels(error_type='connection').inc()
loss_gauge.set(1500.50)
latency_histogram.observe(45.2)

9. Dashboard Access

9.1 User Roles

• Admin: Full access to all dashboards and alerts
• Trader: Access to trading and performance dashboards
• Engineer: Access to operations and system dashboards
• Public: Read-only access to selected metrics

9.2 Dashboard URLs

• Executive: https://grafana.example.com/dashboards/executive
• Operations: https://grafana.example.com/dashboards/operations
• Performance: https://grafana.example.com/dashboards/performance
• Trading: https://grafana.example.com/dashboards/trading

10. SLA and Response Times

10.1 Service Level Objectives

• Availability: 99.5% uptime (measured during market hours)
• Latency: p95 <250ms (inference), p95 <1.5s (end-to-end)
• Error Rate: <0.1% trade execution failure
• Data Freshness: <5 minutes
• Alert Response: <5 minutes for P0/Critical

10.2 Response Time SLAs

• Critical (P0): 5-minute response time
• High (P1): 15-minute response time
• Medium (P2): 2-hour response time
• Low (P3): 24-hour response time

Conclusion

Proactive monitoring and intelligent alerting are critical for maintaining a high-performance trading bot. Regular review of alerts, tuning of thresholds, and continuous improvement of observability ensure reliable operations 24/7.

11. Exposed Metrics (Prometheus)

The following metrics are exposed by the Monitor module:

11.1 Trading Performance

• trading_equity: Current account equity (Gauge)
• trading_pnl_daily: Realized P&L for the current day (Gauge)
• trading_trades_total: Total number of trades executed (Counter)
• trading_drawdown_percent: Current account drawdown percentage (Gauge)
• trading_sharpe_ratio: Annualized Sharpe Ratio (Gauge)
• trading_win_rate: Trading win rate percentage (Gauge)

11.2 Execution Quality

• trading_execution_latency_seconds: Time from signal to execution (Histogram)
• trading_slippage_pips: Difference between expected and actual price (Histogram)
• trading_fill_rate: Percentage of orders filled at intended price (Gauge)
• trading_orders_rejected_total: Total number of rejected orders (Counter)
• trading_market_stability: Real-time market regime confidence/stability score (Gauge)

11.3 System Health

• system_cpu_usage_percent: System CPU utilization percentage (Gauge)
• system_memory_usage_percent: System memory usage percentage (Gauge)
• system_disk_usage_percent: System disk usage percentage (Gauge)
• trading_system_errors: Total count of system errors labeled by component (Counter)
• trading_block_duration_seconds: Duration of trading code blocks labeled by label (Histogram)
• trading_iteration_heartbeat_timestamp: Unix timestamp of the last iteration heartbeat (Gauge)
• trading_iteration_duration_seconds: Total duration of a single trading iteration cycle (Histogram)

11.4 Model & Data

• trading_model_confidence: Latest model prediction confidence (Gauge)
• trading_model_accuracy: Model prediction accuracy (Gauge)
• trading_model_drift_score: Statistical drift from baseline (Gauge)
• trading_data_freshness_seconds: Age of latest data point in seconds (Gauge)