- Introduced `pipeline_certification.py` for Phase 13 checks on the observability layer, including stage order, monotonicity, unique IDs, and stale reports.
- Implemented `pipeline_profiler.py` for event-based profiling, ensuring integrity across pipeline stages and providing methods for recording events and generating reports.
- Created `prediction.py` to standardize prediction results and ensure compatibility across different components.
- Added a smoke test script to validate module imports and function signatures before running notebooks.
- Implemented survival analysis report generation with CSV output.
- Added probability report generation with summary statistics.
- Introduced feature drift report with metrics for monitoring shifts.
- Created HMM regime report for per-timeframe analysis.
- Developed self-verification table to ensure required artifacts are present and contain data.
- Added new CSV files: probability_report_M5.csv and survival_analysis.csv.
This commit introduces a new smoke test script that exercises every phase of the pipeline observability process. The test generates synthetic data for two timeframes (M15 and M5), simulating various stages of the pipeline and verifying that all artifacts are produced and non-empty. The script includes assertions to ensure the integrity of the generated outputs and checks for expected conditions in the decision log, survival matrix, and audit JSON.
- Introduced `pipeline_verification.py` containing multiple verifier classes for dataset integrity, feature leakage, evaluation integrity, prediction alignment, and more.
- Added `SessionFilter` class in `session_filter.py` for unified session/time-of-day filtering.
- Implemented `VerificationResult` data class for structured verification results.
- Created utility functions for hashing and manifest building.
- Established a test suite for the session filter to ensure correct functionality.
- Updated `__init__.py` to include the new shared module.
Luke Mosoti