gavindiazandClaude Code
3a005ddcc6
Docs / Validate documentation (push) Canceled after 0s
Docs / Deploy GitHub Pages (push) Canceled after 0s
CI Final Summary (AI) / AI Final Summary (push) Canceled after 0s
CI Final Summary (AI) / Push/PR event feed (push) Canceled after 0s
CI / CI Integrity and Change Classification (push) Canceled after 0s
CI / Code Quality & Tests (push) Canceled after 0s
Dependency Lock & Migration Safety / Dependency drift (lock vs pyproject) (push) Canceled after 0s
Dependency Lock & Migration Safety / Migration safety (fail-loud + version postconditions) (push) Canceled after 0s
JS Tests / Frontend JS Unit Tests (push) Canceled after 0s
Security / CodeQL Analysis (push) Canceled after 0s
Security / Trivy Vulnerability Scan (push) Canceled after 0s
Tests (OS Matrix) / Py Tests (macos-latest) (push) Canceled after 0s
Tests (OS Matrix) / Py Tests (windows-latest) (push) Canceled after 0s
CI / Extended Validation (Tier 2) (push) Canceled after 0s
CI / Heavy CI (database-provider) (push) Canceled after 0s
CI / Heavy CI (e2e) (push) Canceled after 0s
CI / Heavy CI (integration) (push) Canceled after 0s
CI / Heavy CI (model-validation) (push) Canceled after 0s
CI / Heavy CI (research-backtest) (push) Canceled after 0s
CI / Aggregate CI Results (push) Canceled after 0s
Full project import for the self-hosted code hosting (https://code.aifunny.ltd/gavindiaz/my-NexusTradingForexBot.git). Includes the live-freshness + scaler-hash fixes for the permanent STALE dashboard (tick_pipeline now_utc stamps; infer_probabilities computes model_input_hash on the actual scaled 70D tensor), the data-fetch default-path fix, the g29 web-auth test fixes, and a Chinese user manual (使用手册.md) for the web dashboard. Scratch diagnostic scripts (check*.py, contain a live web-auth token) and the .kilo/ agent-worktree directory are gitignored, not committed. Co-Authored-By: Claude Code <noreply@anthropic.com>
202 lines
8.0 KiB
Markdown
202 lines
8.0 KiB
Markdown
# 70D DATA CONTRACT — scalp_v3 (TASK-03-70D-PARITY)
|
|
|
|
> Agent: Hermes-Parity · 2026-08-19 · TASK-03-70D-PARITY
|
|
> Canonical schema id: `scalp_v3` · Dimension: **70**
|
|
> Single source of truth: `src/nexus_scalp/features/schema_contract.py`
|
|
> Executable verification: `tests/unit/test_70d_contract_parity_task3.py` (+ dataset / replay / validator / runtime suites)
|
|
|
|
---
|
|
|
|
## 1. Canonical vector layout (INVARIANT)
|
|
|
|
```text
|
|
index 0..49 Base 50D = scalp_v1 protected contract (UNCHANGED)
|
|
index 50..59 News 10D = news_context_v1 fields 0..8 + news_state (index 10)
|
|
index 60..69 Liquidity 10D = liquidity_engine.compute_liquidity_features as_vector() order
|
|
```
|
|
|
|
The vector is ALWAYS generated by index from the canonical name tuple.
|
|
Never `list(features.values())` — the registry controls ordering.
|
|
|
|
### 1.1 News 10D selection (documented decision)
|
|
|
|
The repo's canonical news context is the 12-field `news_context_v1`
|
|
(`model_generation/news_bridge.py`). The 70D news block keeps:
|
|
|
|
```text
|
|
0 active_high_impact_events 5 conflict_score
|
|
1 xauusd_relevance 6 novelty
|
|
2 usd_relevance 7 freshness
|
|
3 bullish_pressure 8 confidence
|
|
4 bearish_pressure 9 news_state (index 10 of the 12-field vector)
|
|
```
|
|
|
|
`source_consensus` (index 9 of the 12-field vector) and
|
|
`time_since_event_sec` (index 11) stay OUTSIDE the 70D block by contract
|
|
(they remain part of the 12-field context for the 60D path). The selection is
|
|
NOT a blind first-10 slice; it is hashed so any drift is detected.
|
|
|
|
### 1.2 Liquidity 10D (canonical order)
|
|
|
|
```text
|
|
60 bsl_distance_atr 65 internal_liquidity_distance
|
|
61 ssl_distance_atr 66 external_liquidity_distance
|
|
62 eqh_strength 67 liquidity_confluence
|
|
63 eql_strength 68 liquidity_sweep_state
|
|
64 htf_liquidity_score 69 post_sweep_displacement
|
|
```
|
|
|
|
Producer: `features/liquidity_engine.compute_liquidity_features` →
|
|
`LiquidityFeatures.as_vector()`. Causally confirmed pools only; HTF buckets
|
|
fully closed at decision time only.
|
|
|
|
---
|
|
|
|
## 2. Schema identity & hash
|
|
|
|
`feature_schema_hash()` = SHA-256 (first 16 hex chars) of the canonical
|
|
registry JSON:
|
|
|
|
```json
|
|
{
|
|
"schema_id": "scalp_v3",
|
|
"schema_version": "1.0.0",
|
|
"dimension": 70,
|
|
"base": {"count": 50, "end": 50, "start": 0},
|
|
"news": {"count": 10, "end": 60, "start": 50},
|
|
"liquidity": {"count": 10, "end": 70, "start": 60},
|
|
"features": [{"index": 0, "name": "...", "family": "base"}, ... 70 entries]
|
|
}
|
|
```
|
|
|
|
- The hash covers index, name AND family for every dimension.
|
|
- Two schemas with the same dimension but different ordering → different hash.
|
|
- Training, replay, inference and live compare the SAME canonical value.
|
|
- Current value (2026-08-19): `235b8fccc96b7e0e`
|
|
|
|
---
|
|
|
|
## 3. Snapshot (Feature70Snapshot)
|
|
|
|
Immutable after construction (`features/features70.py`). Carries:
|
|
|
|
```text
|
|
schema_id, schema_version, symbol, timeframe, timestamp_utc,
|
|
feature_vector[70], feature_names, source_state, news_status,
|
|
liquidity_status, news_available, liquidity_available,
|
|
news_context_hash, calculation_version, model_id, model_version,
|
|
feature_schema_hash
|
|
```
|
|
|
|
No downstream subsystem may reorder or reinterpret dimensions. `validate()`
|
|
re-checks dimension, hash, finiteness and bounds; violation → explicit error.
|
|
|
|
## 4. Family states (never silently guessed)
|
|
|
|
| State | Meaning |
|
|
| :--- | :--- |
|
|
| FEATURE_AVAILABLE | family computed from real data |
|
|
| FEATURE_DISABLED | family explicitly off (neutral block, e.g. news off → 10 zeros) |
|
|
| FEATURE_UNAVAILABLE | family declared on but no data → SNAPSHOT/BLOCK (never fake) |
|
|
| FEATURE_INVALID | data present but violates contract → BLOCK |
|
|
|
|
Missing values follow the documented conventions (NaN/Inf → 0.0 then clip
|
|
[-3,+3]; liquidity "far" → 3.0; no pool → documented neutral). NEVER a
|
|
random sentinel.
|
|
|
|
## 5. Dataset builder (`compute_70d_frame`)
|
|
|
|
- Same canonical 55-bar causal window for the 50D engine as the 50D/60D paths;
|
|
FULL causal history for the liquidity engine (matches live governor).
|
|
- Per-row: `news_context_at(t)` (causally correct) → news 10D; liquidity
|
|
engine `as_vector()`.
|
|
- News absent → `FEATURE_DISABLED` + zero block (explicit).
|
|
- `verify_70d_artifact`: dimension==70, all finite, all in [-3,+3],
|
|
schema_id==scalp_v3, schema hash correct, no duplicate timestamps/sample
|
|
ids, exact per-reason rejection counts (brief 41).
|
|
- Reproducible: same source + same schema + same code → identical frame.
|
|
|
|
## 6. Replay (`replay_70d_vector`)
|
|
|
|
Rebuilds the canonical vector at a historical timestamp from raw bars +
|
|
news only (no dataset artifact readback). Future bars/news/liquidity events
|
|
invisible (causal filter `t <= decision`). Bit-exact parity with the dataset
|
|
row for the same timestamp is asserted by tests (TEST-70D-PARITY-08).
|
|
|
|
## 7. Inference validation (`InferenceValidator`)
|
|
|
|
Full chain before any model call:
|
|
|
|
```text
|
|
EXPECTED_SCHEMA -> ACTUAL_SCHEMA -> DIMENSION -> ORDER -> HASH
|
|
-> FINITE -> BOUNDS -> NEWS/LIQUIDITY -> FRESHNESS -> SCALER -> INFERENCE
|
|
```
|
|
|
|
Explicit rejection codes (brief 29):
|
|
|
|
```text
|
|
SCHEMA_MISMATCH / DIMENSION_MISMATCH / FEATURE_ORDER_MISMATCH /
|
|
SCHEMA_HASH_MISMATCH / SCALER_MISMATCH / NONFINITE_FEATURE /
|
|
OUT_OF_RANGE_FEATURE / NEWS_UNAVAILABLE / LIQUIDITY_UNAVAILABLE /
|
|
STALE_FEATURES
|
|
```
|
|
|
|
NEVER silently repairs, pads, truncates or substitutes. Schema metadata
|
|
cached at construction; no DB / file I/O on the per-tick path (INV-001).
|
|
|
|
## 8. Model compatibility (legacy protection)
|
|
|
|
| Model | Runtime | Result |
|
|
| :--- | :--- | :--- |
|
|
| scalp_v2 / 60D | 60D vector | PASS |
|
|
| scalp_v2 / 60D | 70D vector | BLOCK (SCHEMA_MISMATCH) |
|
|
| scalp_v3 / 70D | 70D vector | PASS |
|
|
| scalp_v3 / 70D | 60D vector | BLOCK (SCHEMA_MISMATCH) |
|
|
|
|
Scaler: dimension must equal feature dimension; NEVER padded/truncated to
|
|
match (SCHEMA_INCOMPATIBLE stops). Legacy 50D/60D models remain loadable and
|
|
receive exactly their contracted vectors.
|
|
|
|
## 9. Runtime hook (`Runtime70Hook`)
|
|
|
|
- Observability-only: never touches orders/SL/TP/RiskEngine/execution.
|
|
- Thread-safe toggles (news/liquidity) — toggle never corrupts the vector;
|
|
dimension stays 70; family status reflects the toggle.
|
|
- `[FEATURE_CONTRACT]` structured trace: VALID (schema/dimension/hash/news/
|
|
liquidity/status) or REJECTED (reason, expected/actual, correlation_id).
|
|
- Per-stage timing: base/news/liquidity/assembly/validation ms.
|
|
- Model gate: 70D model + liquidity disabled → MODEL_INPUT_UNAVAILABLE
|
|
(no inference for that snapshot), never fake values.
|
|
- UI/API state: `to_state_dict()` exposes the same schema/dimension/toggles
|
|
the engine actually uses (no frontend-only state).
|
|
|
|
## 10. Performance (brief 49, 200-bar M1 window, measured 2026-08-19)
|
|
|
|
| Stage | p50 | p95 | max |
|
|
| :--- | ---: | ---: | ---: |
|
|
| 50D base construction | 3.1 ms | 9.0 ms | 19.2 ms |
|
|
| 70D assembly + validate | 4.1 ms | 9.3 ms | 11.2 ms |
|
|
| validator per tick | 0.15 ms | 0.90 ms | 1.84 ms |
|
|
| runtime70 live snapshot | 27.3 ms | 33.9 ms | 39.9 ms |
|
|
| replay_70d full rebuild | 19.4 ms | 108.6 ms | 108.6 ms |
|
|
|
|
All within the 100 ms hot-path budget. No DB access on the feature path
|
|
(INV-001, asserted by test).
|
|
|
|
## 11. Tests
|
|
|
|
| Suite | File | Covers |
|
|
| :--- | :--- | :--- |
|
|
| contract | test_70d_contract_parity_task3.py | dimension, schema id, hash, family layout, vector validation (PARITY-01..05) |
|
|
| dataset | test_70d_dataset_parity_task3.py | builder 70D, news disabled explicit, reproducibility, snapshot (PARITY-06..10/19/41) |
|
|
| replay | test_70d_replay_parity_task3.py | dataset==replay, anti-leakage, golden corpus (PARITY-05/08/18/30) |
|
|
| validator | test_70d_inference_validator_task3.py | rejection codes, scaler, 60D/70D matrix (PARITY-09..13/21..24) |
|
|
| runtime | test_70d_runtime_hook_task3.py | four toggle combos, toggle safety, model gate (PARITY-14..17/25/26) |
|
|
| perf | test_70d_perf_task3.py | hot-path budget + no-DB proof |
|
|
| golden | tests/helpers/golden70d.py | 11 deterministic scenarios |
|
|
|
|
## 12. Error codes (runtime, structured)
|
|
|
|
[FEATURE_CONTRACT] event=VALID|REJECTED reason=<code> expected=<schema>
|
|
actual=<schema> correlation_id=<ctx> model_id=<id> model_version=<ver>
|