* docs: improve developer onboarding and contribution experience
- Harden DX domain: format and fix 35+ files in scripts/ and migrations/.
- Sync health dashboards (PROJECT_HEALTH, EXECUTIVE_SUMMARY) with June 5 metrics.
- Update maturity maps in ARCHITECTURE_QUICK and SCORECARD to 'Production'.
- Reduce total repo lint debt from 146 to 33.
Co-authored-by: triqbit <242833443+triqbit@users.noreply.github.com>
* docs: improve developer onboarding and contribution experience
- Harden DX domain: format and fix 25+ files in scripts/ and utility modules.
- Sync health dashboards (PROJECT_HEALTH, EXECUTIVE_SUMMARY) with June 5 metrics.
- Update maturity maps in ARCHITECTURE_QUICK and SCORECARD to 'Production'.
- Ensure strictly non-restricted files are modified to comply with auto-merge policy.
Co-authored-by: triqbit <242833443+triqbit@users.noreply.github.com>
---------
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: triqbit <242833443+triqbit@users.noreply.github.com>
* 🧬 Jules02: Synthetic test scenarios — Operational lifecycle and execution quality
This change enhances the synthetic data generation capabilities to support
broader and safer testing of operational recovery and execution quality.
Key improvements:
- Enhanced ScenarioGenerator with deterministic spread_pips and movement-correlated volume.
- Added LifecycleScenarioBuilder for testing multi-stage system transitions (Failure -> Recovery).
- Added ExecutionQualityScenarioBuilder for validating performance guards and slippage logic.
- Robust test coverage for all new generators.
Co-authored-by: xnessom <255288032+xnessom@users.noreply.github.com>
* feat: 🧬 Jules02: Synthetic test scenarios — Operational lifecycle and execution quality
This change enhances the synthetic data generation capabilities to support
broader and safer testing of operational recovery and execution quality.
Key improvements:
- Enhanced ScenarioGenerator with deterministic spread_pips and movement-correlated volume.
- Added LifecycleScenarioBuilder for testing multi-stage system transitions (Failure -> Recovery).
- Added ExecutionQualityScenarioBuilder for validating performance guards and slippage logic.
- Updated documentation in docs/testing/SYNTHETIC_DATA.md.
- Upgraded python-socketio to 5.14.0 to resolve security vulnerability GHSA-g8c6-8fjj-2r4m.
- Full test coverage for new features (tests/test_lifecycle_scenarios.py).
Co-authored-by: xnessom <255288032+xnessom@users.noreply.github.com>
---------
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: xnessom <255288032+xnessom@users.noreply.github.com>
- Establish formal data retention windows in docs/DATA_RETENTION_POLICY.md.
- Define 7-year compliance window for trades and audit logs.
- Enhance scripts/data_cleanup.py with categorized archiving (compliance, audit, performance, research).
- Implement automated purging of expired archives.
- Add SQLite VACUUM support for proactive storage management.
- Update test suite to cover new archive lifecycle logic.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Created `docs/DATA_RETENTION_POLICY.md` specifying retention windows:
- Logs: 90 days
- Trade records: 7 years
- Risk events/Performance metrics: 2 years
- Implemented `scripts/data_cleanup.py` with safe deletion logic:
- Automates purging of old operational data.
- Archives audit-critical data to compressed CSV/tar.gz with SHA256 checksums.
- Enforces 500MB proactive disk space check before archival.
- Maintains referential integrity for linked model signals.
- Integrates with enterprise `AuditLogger`.
- Verified via `tests/test_data_cleanup.py` and `tests/test_data_cleanup_system.py`.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Update docs/DATA_RETENTION_POLICY.md with 7-year compliance windows for trades and execution analytics.
- Implement automated archival and purging logic in scripts/data_cleanup.py.
- Add support for ExecutionQuality and BlockedSignalAnalysis tables.
- Ensure referential integrity for linked model signals.
- Update tests/test_data_cleanup.py to verify new table purging.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Create docs/DATA_RETENTION_POLICY.md defining retention windows for compliance and operations.
- Update scripts/data_cleanup.py to automate purging and archival.
- Ensure referential integrity by preserving linked signals until parent records are purged.
- Implement archival for trades, risk events, and performance metrics (CSV + SHA256).
- Implement archival for backtest results (tar.gz + SHA256).
- Use bulk deletes for database performance.
- Ensure accurate dry-run reporting via refined subqueries.
- Integrate 500MB disk space safety check for archival.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Create docs/DATA_RETENTION_POLICY.md (v1.3.0) defining retention windows for compliance and operational data.
- Refactor scripts/data_cleanup.py with UTC consistency, disk space checks, and AuditLogger integration.
- Implement safe deletion logic to preserve signals linked to trades or risk events.
- Add .github/workflows/data-cleanup.yml for automated weekly cleanup and archival.
- Update tests/test_data_cleanup_system.py to verify new referential integrity and disk space logic.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Defined operational data retention windows in docs/DATA_RETENTION_POLICY.md.
- Enhanced scripts/data_cleanup.py with archival logic for trades, audit logs, performance metrics, and backtest results.
- Implemented safe deletion logic to preserve model signals linked to active trades.
- Added SHA256 checksum generation for all archived artifacts.
- Introduced comprehensive verification tests in tests/test_data_cleanup_system.py.
- Updated .gitignore to exclude ephemeral archive outputs.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Define data retention windows for trades, signals, logs, and audit trails in docs/DATA_RETENTION_POLICY.md.
- Implement scripts/data_cleanup.py for automated purging with CSV archival and SHA256 checksums.
- Ensure safe deletion logic preserves signals linked to compliance-critical trades.
- Improve configuration resilience of cleanup tools to run without MT5 credentials.
- Fix pre-existing integration test failure in backtest initialization.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
* Establish data retention policy and automated cleanup logic
- Create docs/DATA_RETENTION_POLICY.md defining retention windows (90d logs, 7y trades) and archival rules.
- Implement scripts/data_cleanup.py with safe deletion logic (preserving linked signals) and dry-run support.
- Ensure correct handling of SecretStr for database URLs.
- Verified with unit tests and full test suite.
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
* Establish data retention policy and automated cleanup logic
- Create docs/DATA_RETENTION_POLICY.md defining retention windows (90d logs, 7y trades) and archival rules.
- Implement scripts/data_cleanup.py with safe deletion logic (preserving linked signals) and dry-run support.
- Correct SecretStr handling for database URLs in cleanup script.
- Update torch/torchvision in requirements.txt to 2.3.1/0.18.1 to resolve dependency conflict with stable-baselines3 2.8.0.
- Verified with unit tests and full test suite.
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
---------
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
This change defines the formal operational data retention policy and provides
an automated cleanup script to enforce it.
Key improvements:
- Created docs/DATA_RETENTION_POLICY.md defining retention windows:
- Trade records, Linked Signals, Audit Logs: 7 years (Compliance)
- Risk Events, Performance Metrics: 2 years (Audit)
- Unlinked Signals, App Logs: 90 days (Operational)
- Backtest Results: 1 year (Research)
- Enhanced scripts/data_cleanup.py with:
- Support for audit_log purging.
- Multi-database URL support (handling separate audit DBs).
- Safe deletion logic for linked records.
- Mandatory dry-run and safety warnings for production DBs.
- Improved tests/test_data_cleanup.py with SQLAlchemy StaticPool for
reliable in-memory SQLite testing.
Signed-off-by: Jules03 (andonly1348)
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
- Established formal Data Retention Policy in docs/DATA_RETENTION_POLICY.md.
- Enhanced scripts/data_cleanup.py to support backtest artifact purging.
- Implemented recursive deletion and empty directory removal for backtests.
- Added comprehensive unit tests in tests/test_data_cleanup.py.
- Ensured compliance with archival rules and foreign key integrity.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: andonly1348 <279141437+andonly1348@users.noreply.github.com>
