mt5-ai-xauusd-trader/docs/DATA_RETENTION_POLICY.md

Data Retention Policy

1. Overview

This policy defines the retention periods and disposal procedures for operational data generated by the MT5 AI/ML Trading Bot. It ensures compliance with financial regulations (e.g., SEC/FINRA-like standards) while maintaining system performance and storage efficiency.

2. Retention Schedule

Data Category	Data Type	Retention Period	Disposal Method	Reason
Compliance	Trade Records (trades table)	7 Years	Archival + Purge	Regulatory Requirement
Compliance	Execution Quality (execution_qualities)	7 Years	Archival + Purge	Execution Analytics
Audit	Audit Log (audit_log table)	7 Years	Archival + Purge	Enterprise Traceability
Audit	Risk Events (risk_events table)	2 Years	Archival + Purge	Operational Auditing
Audit	Linked Model Signals	7 Years	Archival + Purge	Trade Traceability
Operational	Unlinked Model Signals	90 Days	Purge	Performance Tuning
Operational	Blocked Signal Analysis	90 Days	Purge	Opportunity Cost
Operational	Performance Metrics	2 Years	Archival + Purge	Trend Analysis
Diagnostics	Application Logs (logs/*.log)	90 Days	Deletion	Troubleshooting
Research	Backtest Results	1 Year	Archival + Deletion	Optimization

3. Data Classification

3.1 Audit-Critical Data (Preserve)

The following data must be preserved for the full 7-year compliance window:

• Trade Records: All fields in the trades table, including tickets, symbols, prices, and PnL.
• Execution Quality: High-precision execution analytics linked to each trade record.
• Audit Log: System actions, configuration changes, and operator events in the audit_log table.
• Linked Signals: Any model_signals record referenced by a trade record via signal_id OR a risk_event record via signal_id.

3.2 Retention for Linked Data

To maintain referential integrity and auditability:

• A model_signals record is considered "Linked" if it is associated with at least one trade or risk_event.
• Linked signals inherit the maximum retention period of their associated records (e.g., 7 years if linked to a trade).
• ExecutionQuality records are linked to Trade and inherit its 7-year retention.
• BlockedSignalAnalysis records are linked to ModelSignal and inherit the signal's retention (typically 90 days unless linked to a risk event).
• When a parent record (Trade/RiskEvent) is purged, the cleanup script must verify if the associated signal is still required by another active record before deletion.

3.3 Ephemeral Data (Rotate/Purge)

The following data can be purged more frequently to manage storage:

• Unlinked Signals: AI predictions that did not result in an executed trade.
• Application Logs: Standard output and error logs from the trading engine.
• Temporary Backtest Artifacts: Large CSVs or parquet files generated during model validation.

4. Archival and Purging Rules

4.1 Archival Procedures

Prior to purging, data in the Compliance, Audit, Operational (Performance Metrics), and Research (Backtest Results) categories must be archived:

• Format:
  • Database records are exported to compressed CSV format.
  • Filesystem-based research data (Backtest Results) is bundled into compressed .tar.gz archives.
• Destination: Archives are stored in type-specific subdirectories within the archives/ directory:
  • archives/compliance/: Trade records and execution quality analytics.
  • archives/audit/: Audit logs and risk events.
  • archives/performance/: Performance metrics and snapshots.
  • archives/research/: Backtest results and research artifacts.
• Integrity: SHA256 checksums are generated for each archive file to ensure immutability.

4.2 Archive Lifecycle Management

Archives themselves are subject to a retention policy to prevent indefinite storage growth:

• Compliance Archives: Kept for 7 years from the date of archival.
• Audit/Performance Archives: Kept for 2 years from the date of archival.
• Research Archives: Kept for 1 year from the date of archival.
• Cleanup: The scripts/data_cleanup.py script is responsible for purging expired archives.

4.3 Automated Purging Schedule

The automated cleanup script (scripts/data_cleanup.py) is executed on a regular schedule to ensure continuous compliance and storage hygiene.

• Schedule: Weekly (Sundays at 00:00 UTC).
• Automation: Managed via the .github/workflows/data-cleanup.yml GitHub Action.

4.4 Safe Deletion Logic

• Foreign Key Integrity: The cleanup script ensures that model_signals linked to trades or risk_events are NOT deleted if the parent record is still within its retention window.
• Dry-Run Mode: All cleanup operations must support a --dry-run flag to verify deletions before execution.
• Logging: Every cleanup operation is logged to the system audit trail.

5. Implementation

The primary mechanism for enforcement is the scripts/data_cleanup.py tool. It uses the database_url and logs_dir from the system configuration to target the correct data stores.

---

Version: 1.4.1 Effective Date: 2024-06-10 Owner: Release Reliability & Governance (Jules03)