gavindiaz/mt5-ai-xauusd-trader
1
0
Fork 0
mirror of https://github.com/triqbit/mt5-ai-xauusd-trader.git synced 2026-11-02 06:56:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: define measurable reliability standards and SLO targets (#437)

Browse Source 
- Create docs/SLO_TARGETS.md with uptime, latency, and recovery targets.
- Synchronize reliability metrics across MONITORING_ALERTING.md and PERFORMANCE_TUNING.md.
- Align EXCELLENCE_BLUEPRINT.md and FINAL_RECOMMENDATIONS.md with new 99.5% uptime target.
- Establish 30-day error budget framework for operational stability.

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
This commit is contained in:
andonly1348
2026-05-02 03:03:24 +00:00
committed by GitHub
co-authored by google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> andonly1348
parent 76f2fe50be
commit 4e6952e49b
6 changed files with 77 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
EXCELLENCE_BLUEPRINT.md
+1 -1
View File
@@ -96,7 +96,7 @@ This document outlines the comprehensive blueprint for achieving operational exc
- **Recovery Factor**: Target >3.0
### 5.2 System Performance
- **Uptime**: Target 99.99% (4 hours downtime per year)
- **Uptime**: Target 99.5% (~3.6 hours downtime per month)
- **Order Execution Latency**: <100ms from signal to execution
- **Model Inference Time**: <10ms per prediction
- **Data Pipeline Latency**: <500ms from data feed to model input
FINAL_RECOMMENDATIONS.md
+2 -2
View File
@@ -75,7 +75,7 @@ This document synthesizes all recommendations into a prioritized action plan for
- [ ] Configure P1-P4 alerts with auto-escalation
- [ ] Set up on-call rotation with PagerDuty
- [ ] Create runbooks for common incidents
- [ ] Implement SLA tracking (99.99% uptime)
- [ ] Implement SLA tracking (99.5% uptime)
### Compliance & Documentation
- [ ] Achieve ISO 27001 certification
@@ -213,7 +213,7 @@ mt5-ai-xauusd-trader/
- Risk-reward ratio
### System Metrics
- Uptime percentage (target: 99.99%)
- Uptime percentage (target: 99.5%)
- Order execution latency (target: <100ms)
- Model inference latency (target: <10ms)
- Data pipeline latency (target: <500ms)
IMPROVEMENTS_SUMMARY.md
+2 -2
View File
@@ -91,7 +91,7 @@
- **Real-Time Metrics**: P50/P95/P99 latency tracking
- **Alert Automation**: P1-P4 severity levels with escalation
- **On-Call**: PagerDuty integration with rotation
- **SLA Tracking**: 99.99% uptime target (4 hours/year max)
- **SLA Tracking**: 99.5% uptime target (~3.6 hours/month max)
- **Incident Response**: <5-minute response time for critical
### 7. Infrastructure & Deployment
@@ -176,7 +176,7 @@
- ✓ Code coverage >85%
- ✓ Execution latency <100ms
- ✓ Model inference <10ms
- ✓ Uptime >99.99%
- ✓ Uptime >99.5%
- ✓ Error rate <0.1%
### Business KPIs
MONITORING_ALERTING.md
+8 -8
View File
@@ -323,17 +323,17 @@ latency_histogram.observe(45.2)
## 10. SLA and Response Times
### 10.1 Service Level Objectives
- **Availability**: 99.99% uptime (4 hours/year downtime)
- **Latency**: p95 <500ms, p99 <1s
- **Error Rate**: <0.1%
- **Availability**: 99.5% uptime (measured during market hours)
- **Latency**: p95 <250ms (inference), p95 <1.5s (end-to-end)
- **Error Rate**: <0.1% trade execution failure
- **Data Freshness**: <5 minutes
- **Alert Response**: <5 minutes for critical
- **Alert Response**: <5 minutes for P0/Critical
### 10.2 Response Time SLAs
- **Critical**: 5-minute response time
- **High**: 30-minute response time
- **Medium**: 2-hour response time
- **Low**: 24-hour response time
- **Critical (P0)**: 5-minute response time
- **High (P1)**: 15-minute response time
- **Medium (P2)**: 2-hour response time
- **Low (P3)**: 24-hour response time
## Conclusion
PERFORMANCE_TUNING.md
+1 -1
View File
@@ -257,4 +257,4 @@ async def submit_orders_async(trades):
## Conclusion
Performance is critical for trading bot success. Systematic profiling, optimization, and continuous monitoring ensure sub-100ms execution latency and 99.99% uptime.
Performance is critical for trading bot success. Systematic profiling, optimization, and continuous monitoring ensure sub-100ms execution latency and 99.5% uptime.
docs/SLO_TARGETS.md
+63 -52
View File
@@ -1,62 +1,73 @@
# Service Level Objectives (SLO) & Reliability Targets
This document defines the measurable reliability standards for the MT5 AI/ML Trading Bot repository and runtime system.
This document defines the measurable reliability standards for the MT5 AI/ML Trading Bot repository and runtime system. These targets represent "Enterprise Quality" translated into trackable objectives.
## 1. Availability SLOs
## 1. Availability SLOs (Uptime)
Uptime is measured during active market hours (XAUUSD: Monday 00:00 - Friday 23:59 Server Time).
| Metric | Target | Measurement Method |
|--------|--------|--------------------|
| **System Uptime (Live Mode)** | 99.5% | `/health/readiness` probe success rate via uptime monitor. |
| **API Availability** | 99.9% | Percentage of successful 200 OK responses on the FastAPI router. |
| **MT5 Connectivity** | 99.0% | Percentage of time `MT5Connector.is_initialized` is True during market hours. |
**Acceptable Downtime:** ~3.6 hours per 30-day rolling window.
## 2. CI/CD & Development SLOs
Ensuring high engineering standards and stable delivery pipelines.
| Metric | Target | Measurement Method |
|--------|--------|--------------------|
| **CI Pipeline Success Rate** | 95.0% | Percentage of GitHub Action runs on `main` and `develop` that finish with "Success". |
| **Test Coverage (Core)** | > 80% | `pytest-cov` statement coverage on `src/core/` and `src/trading/`. |
| **Static Analysis Compliance** | 100% | Zero critical/high issues from `flake8`, `mypy`, and `bandit` in PRs. |
## 3. Performance & Latency SLOs
Measured via `src/core/profiler.py` and exported to Prometheus.
| Metric | P50 | P95 | P99 | Measurement Method |
|--------|-----|-----|-----|-------------------|
| **Model Inference** | < 100ms | < 250ms | < 500ms | Time from feature vector input to PPO/Ensemble output. |
| **Risk Approval** | < 50ms | < 150ms | < 300ms | Time to pass through the 6-layer `RiskManager` cascade. |
| **End-to-End Latency** | < 500ms | < 1.5s | < 3s | Market Event -> Inference -> Risk -> MT5 Order Execution. |
| **Backtest Generation** | < 5 min | < 8 min | < 12 min | Time to execute 1-year XAUUSD backtest (1M candles). |
## 4. Operational Reliability & Alerting
Response expectations for the Jules-led operations team.
| Severity | Target Response | Target Resolution | Description |
|----------|-----------------|-------------------|-------------|
| **P0 (Critical)** | < 5 mins | < 1 hour | Trading halted, connection lost, or critical balance mismatch. |
| **P1 (High)** | < 15 mins | < 4 hours | Degraded performance, model drift, or partial service failure. |
| **P2 (Medium)** | < 2 hours | < 24 hours | Non-blocking bugs, UI/monitoring issues, or disk warnings. |
| **P3 (Low)** | < 24 hours | 1 Week | Documentation updates, minor refactors, or feature requests. |
## 5. Incident Recovery (RTO/RPO)
Defined in [Disaster Recovery Plan](DISASTER_RECOVERY.md).
| Metric | Target | Description |
|--------|--------|-------------|
| **System Uptime (Live Mode)** | 99.5% | Percentage of time the bot is active and connected during market hours. |
| **API Availability** | 99.9% | Availability of the health check and monitoring endpoints. |
| **MT5 Connectivity** | 99.0% | Percentage of time the connection to the broker terminal is active. |
| **Recovery Time (RTO)** | 15 mins | Maximum time to restore service after a system/process failure. |
| **Recovery Point (RPO)** | 1 hour | Maximum data loss allowed (governed by hourly database backups). |
## 2. Reliability SLOs
## 6. Error Budget Framework (30-Day Rolling Window)
| Metric | Target | Description |
|--------|--------|-------------|
| **CI Pipeline Success Rate** | 95.0% | Percentage of commits to `main` and `develop` that pass all CI checks. |
| **Order Execution Success** | 99.9% | Percentage of approved signals that result in a successfully placed order (excluding broker rejections). |
| **Database Integrity** | 100% | Zero instances of unrecoverable data corruption in `trades.db`. |
The error budget is the acceptable amount of failure. Exceeding these triggers a "Stability Freeze" where feature work stops to address technical debt.
## 3. Latency SLOs
| Component | Error Budget | "Acceptable" Failures per Month (approx) |
|-----------|--------------|-----------------------------------------|
| **Availability** | 0.5% (3.6h) | ~3-4 hours of cumulative outage. |
| **CI Stability** | 5% | ~5 failed builds per 100 commits to protected branches. |
| **Trade Execution** | 0.1% | 1 failed order per 1,000 approved signals. |
| **Model Accuracy** | 5% Drift | Max 5% deviation from expected confidence thresholds. |
| **Data Integrity** | 0% | Zero unrecoverable data corruption events. |
| Metric | P50 | P95 | P99 |
|--------|-----|-----|-----|
| **Model Inference** | < 100ms | < 250ms | < 500ms |
| **Risk Approval Latency** | < 50ms | < 150ms | < 300ms |
| **End-to-End Execution** | < 500ms | < 1.5s | < 3s |
*End-to-end execution covers Signal -> Inference -> Risk Approval -> MT5 Order.*
## 4. Operational & Performance SLOs
| Metric | Target | Description |
|--------|--------|-------------|
| **Backtest Generation Time** | < 5 mins | Time to run a standard 1-year backtest on XAUUSD. |
| **Startup Time** | < 30s | Time from process launch to first health check passing. |
| **Log Retention** | 90 days | Availability of historical operational logs. |
## 5. Incident Response & Recovery (from [DRP](DISASTER_RECOVERY.md))
| Metric | Target | Description |
|--------|--------|-------------|
| **Alert Response (P0)** | < 5 mins | Initial triage for critical/trading-halted events. |
| **Alert Response (P1)** | < 15 mins | Initial triage for degraded service or connection loss. |
| **Recovery Time (RTO)** | 15 mins | Maximum time to restore service after a disaster. |
| **Recovery Point (RPO)** | 1 hour | Maximum data loss allowed in the event of failure. |
## 6. Error Budget Framework
The error budget represents the acceptable amount of failure over a 30-day rolling window.
| Component | Budget (30 Days) | Action on Exhaustion |
|-----------|------------------|----------------------|
| **Downtime** | ~3.6 hours | Halt new feature releases; prioritize stability. |
| **CI Failures** | 5% of builds | Audit CI pipeline and test stability. |
| **Trade Failures** | 0.1% of trades | Audit MT5 Connector and Risk Manager logic. |
| **Inference Latency** | 1% > 500ms | Optimize model architecture or hardware allocation. |
## 7. Monitoring & Reporting
- SLO compliance is tracked via the `HealthChecker` and `Monitor` classes.
- Monthly reliability reviews are conducted to adjust targets based on production performance.
- Breaching an SLO triggers an automatic P2 alert for investigation.
## 7. Monitoring & Governance
- **Automation:** Compliance is tracked via Prometheus metrics (`trading_latency_ms`, `health_check_status`).
- **Reporting:** Weekly reliability reports generated from Prometheus/Grafana.
- **Audit:** Any SLO breach requires a Blameless Post-Mortem and an update to the [Risk Manager](../src/trading/risk_manager.py) or [Circuit Breaker](../src/core/monitor.py) logic if applicable.