From 4e6952e49bd340b1a9478bc01e617975b997989d Mon Sep 17 00:00:00 2001 From: andonly1348 Date: Sat, 2 May 2026 08:03:24 +0500 Subject: [PATCH] docs: define measurable reliability standards and SLO targets (#437) - Create docs/SLO_TARGETS.md with uptime, latency, and recovery targets. - Synchronize reliability metrics across MONITORING_ALERTING.md and PERFORMANCE_TUNING.md. - Align EXCELLENCE_BLUEPRINT.md and FINAL_RECOMMENDATIONS.md with new 99.5% uptime target. - Establish 30-day error budget framework for operational stability. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com> --- EXCELLENCE_BLUEPRINT.md | 2 +- FINAL_RECOMMENDATIONS.md | 4 +- IMPROVEMENTS_SUMMARY.md | 4 +- MONITORING_ALERTING.md | 16 +++--- PERFORMANCE_TUNING.md | 2 +- docs/SLO_TARGETS.md | 115 +++++++++++++++++++++------------------ 6 files changed, 77 insertions(+), 66 deletions(-) diff --git a/EXCELLENCE_BLUEPRINT.md b/EXCELLENCE_BLUEPRINT.md index 71963201..be28e378 100644 --- a/EXCELLENCE_BLUEPRINT.md +++ b/EXCELLENCE_BLUEPRINT.md @@ -96,7 +96,7 @@ This document outlines the comprehensive blueprint for achieving operational exc - **Recovery Factor**: Target >3.0 ### 5.2 System Performance -- **Uptime**: Target 99.99% (4 hours downtime per year) +- **Uptime**: Target 99.5% (~3.6 hours downtime per month) - **Order Execution Latency**: <100ms from signal to execution - **Model Inference Time**: <10ms per prediction - **Data Pipeline Latency**: <500ms from data feed to model input diff --git a/FINAL_RECOMMENDATIONS.md b/FINAL_RECOMMENDATIONS.md index 596dafe1..bbc7e03d 100644 --- a/FINAL_RECOMMENDATIONS.md +++ b/FINAL_RECOMMENDATIONS.md @@ -75,7 +75,7 @@ This document synthesizes all recommendations into a prioritized action plan for - [ ] Configure P1-P4 alerts with auto-escalation - [ ] Set up on-call rotation with PagerDuty - [ ] Create runbooks for common incidents -- [ ] Implement SLA tracking (99.99% uptime) +- [ ] Implement SLA tracking (99.5% uptime) ### Compliance & Documentation - [ ] Achieve ISO 27001 certification @@ -213,7 +213,7 @@ mt5-ai-xauusd-trader/ - Risk-reward ratio ### System Metrics -- Uptime percentage (target: 99.99%) +- Uptime percentage (target: 99.5%) - Order execution latency (target: <100ms) - Model inference latency (target: <10ms) - Data pipeline latency (target: <500ms) diff --git a/IMPROVEMENTS_SUMMARY.md b/IMPROVEMENTS_SUMMARY.md index 15812f4e..968b6a7d 100644 --- a/IMPROVEMENTS_SUMMARY.md +++ b/IMPROVEMENTS_SUMMARY.md @@ -91,7 +91,7 @@ - **Real-Time Metrics**: P50/P95/P99 latency tracking - **Alert Automation**: P1-P4 severity levels with escalation - **On-Call**: PagerDuty integration with rotation -- **SLA Tracking**: 99.99% uptime target (4 hours/year max) +- **SLA Tracking**: 99.5% uptime target (~3.6 hours/month max) - **Incident Response**: <5-minute response time for critical ### 7. Infrastructure & Deployment @@ -176,7 +176,7 @@ - ✓ Code coverage >85% - ✓ Execution latency <100ms - ✓ Model inference <10ms -- ✓ Uptime >99.99% +- ✓ Uptime >99.5% - ✓ Error rate <0.1% ### Business KPIs diff --git a/MONITORING_ALERTING.md b/MONITORING_ALERTING.md index c23275e9..4e8fb965 100644 --- a/MONITORING_ALERTING.md +++ b/MONITORING_ALERTING.md @@ -323,17 +323,17 @@ latency_histogram.observe(45.2) ## 10. SLA and Response Times ### 10.1 Service Level Objectives -- **Availability**: 99.99% uptime (4 hours/year downtime) -- **Latency**: p95 <500ms, p99 <1s -- **Error Rate**: <0.1% +- **Availability**: 99.5% uptime (measured during market hours) +- **Latency**: p95 <250ms (inference), p95 <1.5s (end-to-end) +- **Error Rate**: <0.1% trade execution failure - **Data Freshness**: <5 minutes -- **Alert Response**: <5 minutes for critical +- **Alert Response**: <5 minutes for P0/Critical ### 10.2 Response Time SLAs -- **Critical**: 5-minute response time -- **High**: 30-minute response time -- **Medium**: 2-hour response time -- **Low**: 24-hour response time +- **Critical (P0)**: 5-minute response time +- **High (P1)**: 15-minute response time +- **Medium (P2)**: 2-hour response time +- **Low (P3)**: 24-hour response time ## Conclusion diff --git a/PERFORMANCE_TUNING.md b/PERFORMANCE_TUNING.md index 18a01564..8a322529 100644 --- a/PERFORMANCE_TUNING.md +++ b/PERFORMANCE_TUNING.md @@ -257,4 +257,4 @@ async def submit_orders_async(trades): ## Conclusion -Performance is critical for trading bot success. Systematic profiling, optimization, and continuous monitoring ensure sub-100ms execution latency and 99.99% uptime. +Performance is critical for trading bot success. Systematic profiling, optimization, and continuous monitoring ensure sub-100ms execution latency and 99.5% uptime. diff --git a/docs/SLO_TARGETS.md b/docs/SLO_TARGETS.md index fc9f974d..6decec55 100644 --- a/docs/SLO_TARGETS.md +++ b/docs/SLO_TARGETS.md @@ -1,62 +1,73 @@ # Service Level Objectives (SLO) & Reliability Targets -This document defines the measurable reliability standards for the MT5 AI/ML Trading Bot repository and runtime system. +This document defines the measurable reliability standards for the MT5 AI/ML Trading Bot repository and runtime system. These targets represent "Enterprise Quality" translated into trackable objectives. -## 1. Availability SLOs +## 1. Availability SLOs (Uptime) + +Uptime is measured during active market hours (XAUUSD: Monday 00:00 - Friday 23:59 Server Time). + +| Metric | Target | Measurement Method | +|--------|--------|--------------------| +| **System Uptime (Live Mode)** | 99.5% | `/health/readiness` probe success rate via uptime monitor. | +| **API Availability** | 99.9% | Percentage of successful 200 OK responses on the FastAPI router. | +| **MT5 Connectivity** | 99.0% | Percentage of time `MT5Connector.is_initialized` is True during market hours. | + +**Acceptable Downtime:** ~3.6 hours per 30-day rolling window. + +## 2. CI/CD & Development SLOs + +Ensuring high engineering standards and stable delivery pipelines. + +| Metric | Target | Measurement Method | +|--------|--------|--------------------| +| **CI Pipeline Success Rate** | 95.0% | Percentage of GitHub Action runs on `main` and `develop` that finish with "Success". | +| **Test Coverage (Core)** | > 80% | `pytest-cov` statement coverage on `src/core/` and `src/trading/`. | +| **Static Analysis Compliance** | 100% | Zero critical/high issues from `flake8`, `mypy`, and `bandit` in PRs. | + +## 3. Performance & Latency SLOs + +Measured via `src/core/profiler.py` and exported to Prometheus. + +| Metric | P50 | P95 | P99 | Measurement Method | +|--------|-----|-----|-----|-------------------| +| **Model Inference** | < 100ms | < 250ms | < 500ms | Time from feature vector input to PPO/Ensemble output. | +| **Risk Approval** | < 50ms | < 150ms | < 300ms | Time to pass through the 6-layer `RiskManager` cascade. | +| **End-to-End Latency** | < 500ms | < 1.5s | < 3s | Market Event -> Inference -> Risk -> MT5 Order Execution. | +| **Backtest Generation** | < 5 min | < 8 min | < 12 min | Time to execute 1-year XAUUSD backtest (1M candles). | + +## 4. Operational Reliability & Alerting + +Response expectations for the Jules-led operations team. + +| Severity | Target Response | Target Resolution | Description | +|----------|-----------------|-------------------|-------------| +| **P0 (Critical)** | < 5 mins | < 1 hour | Trading halted, connection lost, or critical balance mismatch. | +| **P1 (High)** | < 15 mins | < 4 hours | Degraded performance, model drift, or partial service failure. | +| **P2 (Medium)** | < 2 hours | < 24 hours | Non-blocking bugs, UI/monitoring issues, or disk warnings. | +| **P3 (Low)** | < 24 hours | 1 Week | Documentation updates, minor refactors, or feature requests. | + +## 5. Incident Recovery (RTO/RPO) + +Defined in [Disaster Recovery Plan](DISASTER_RECOVERY.md). | Metric | Target | Description | |--------|--------|-------------| -| **System Uptime (Live Mode)** | 99.5% | Percentage of time the bot is active and connected during market hours. | -| **API Availability** | 99.9% | Availability of the health check and monitoring endpoints. | -| **MT5 Connectivity** | 99.0% | Percentage of time the connection to the broker terminal is active. | +| **Recovery Time (RTO)** | 15 mins | Maximum time to restore service after a system/process failure. | +| **Recovery Point (RPO)** | 1 hour | Maximum data loss allowed (governed by hourly database backups). | -## 2. Reliability SLOs +## 6. Error Budget Framework (30-Day Rolling Window) -| Metric | Target | Description | -|--------|--------|-------------| -| **CI Pipeline Success Rate** | 95.0% | Percentage of commits to `main` and `develop` that pass all CI checks. | -| **Order Execution Success** | 99.9% | Percentage of approved signals that result in a successfully placed order (excluding broker rejections). | -| **Database Integrity** | 100% | Zero instances of unrecoverable data corruption in `trades.db`. | +The error budget is the acceptable amount of failure. Exceeding these triggers a "Stability Freeze" where feature work stops to address technical debt. -## 3. Latency SLOs +| Component | Error Budget | "Acceptable" Failures per Month (approx) | +|-----------|--------------|-----------------------------------------| +| **Availability** | 0.5% (3.6h) | ~3-4 hours of cumulative outage. | +| **CI Stability** | 5% | ~5 failed builds per 100 commits to protected branches. | +| **Trade Execution** | 0.1% | 1 failed order per 1,000 approved signals. | +| **Model Accuracy** | 5% Drift | Max 5% deviation from expected confidence thresholds. | +| **Data Integrity** | 0% | Zero unrecoverable data corruption events. | -| Metric | P50 | P95 | P99 | -|--------|-----|-----|-----| -| **Model Inference** | < 100ms | < 250ms | < 500ms | -| **Risk Approval Latency** | < 50ms | < 150ms | < 300ms | -| **End-to-End Execution** | < 500ms | < 1.5s | < 3s | - -*End-to-end execution covers Signal -> Inference -> Risk Approval -> MT5 Order.* - -## 4. Operational & Performance SLOs - -| Metric | Target | Description | -|--------|--------|-------------| -| **Backtest Generation Time** | < 5 mins | Time to run a standard 1-year backtest on XAUUSD. | -| **Startup Time** | < 30s | Time from process launch to first health check passing. | -| **Log Retention** | 90 days | Availability of historical operational logs. | - -## 5. Incident Response & Recovery (from [DRP](DISASTER_RECOVERY.md)) - -| Metric | Target | Description | -|--------|--------|-------------| -| **Alert Response (P0)** | < 5 mins | Initial triage for critical/trading-halted events. | -| **Alert Response (P1)** | < 15 mins | Initial triage for degraded service or connection loss. | -| **Recovery Time (RTO)** | 15 mins | Maximum time to restore service after a disaster. | -| **Recovery Point (RPO)** | 1 hour | Maximum data loss allowed in the event of failure. | - -## 6. Error Budget Framework - -The error budget represents the acceptable amount of failure over a 30-day rolling window. - -| Component | Budget (30 Days) | Action on Exhaustion | -|-----------|------------------|----------------------| -| **Downtime** | ~3.6 hours | Halt new feature releases; prioritize stability. | -| **CI Failures** | 5% of builds | Audit CI pipeline and test stability. | -| **Trade Failures** | 0.1% of trades | Audit MT5 Connector and Risk Manager logic. | -| **Inference Latency** | 1% > 500ms | Optimize model architecture or hardware allocation. | - -## 7. Monitoring & Reporting -- SLO compliance is tracked via the `HealthChecker` and `Monitor` classes. -- Monthly reliability reviews are conducted to adjust targets based on production performance. -- Breaching an SLO triggers an automatic P2 alert for investigation. +## 7. Monitoring & Governance +- **Automation:** Compliance is tracked via Prometheus metrics (`trading_latency_ms`, `health_check_status`). +- **Reporting:** Weekly reliability reports generated from Prometheus/Grafana. +- **Audit:** Any SLO breach requires a Blameless Post-Mortem and an update to the [Risk Manager](../src/trading/risk_manager.py) or [Circuit Breaker](../src/core/monitor.py) logic if applicable.