my-NexusTradingForexBot/docs/70D_DATA_CONTRACT.md

70D DATA CONTRACT — scalp_v3 (TASK-03-70D-PARITY)

Agent: Hermes-Parity · 2026-08-19 · TASK-03-70D-PARITY Canonical schema id: scalp_v3 · Dimension: 70 Single source of truth: src/nexus_scalp/features/schema_contract.py Executable verification: tests/unit/test_70d_contract_parity_task3.py (+ dataset / replay / validator / runtime suites)

---

1. Canonical vector layout (INVARIANT)

index  0..49   Base 50D     = scalp_v1 protected contract (UNCHANGED)
index 50..59   News 10D     = news_context_v1 fields 0..8 + news_state (index 10)
index 60..69   Liquidity 10D = liquidity_engine.compute_liquidity_features as_vector() order

The vector is ALWAYS generated by index from the canonical name tuple. Never list(features.values()) — the registry controls ordering.

1.1 News 10D selection (documented decision)

The repo's canonical news context is the 12-field news_context_v1 (model_generation/news_bridge.py). The 70D news block keeps:

0 active_high_impact_events   5 conflict_score
1 xauusd_relevance            6 novelty
2 usd_relevance               7 freshness
3 bullish_pressure            8 confidence
4 bearish_pressure            9 news_state (index 10 of the 12-field vector)

source_consensus (index 9 of the 12-field vector) and time_since_event_sec (index 11) stay OUTSIDE the 70D block by contract (they remain part of the 12-field context for the 60D path). The selection is NOT a blind first-10 slice; it is hashed so any drift is detected.

1.2 Liquidity 10D (canonical order)

60 bsl_distance_atr           65 internal_liquidity_distance
61 ssl_distance_atr           66 external_liquidity_distance
62 eqh_strength               67 liquidity_confluence
63 eql_strength               68 liquidity_sweep_state
64 htf_liquidity_score        69 post_sweep_displacement

Producer: features/liquidity_engine.compute_liquidity_features → LiquidityFeatures.as_vector(). Causally confirmed pools only; HTF buckets fully closed at decision time only.

---

2. Schema identity & hash

feature_schema_hash() = SHA-256 (first 16 hex chars) of the canonical registry JSON:

{
  "schema_id": "scalp_v3",
  "schema_version": "1.0.0",
  "dimension": 70,
  "base":      {"count": 50, "end": 50, "start": 0},
  "news":      {"count": 10, "end": 60, "start": 50},
  "liquidity": {"count": 10, "end": 70, "start": 60},
  "features":  [{"index": 0, "name": "...", "family": "base"}, ... 70 entries]
}

• The hash covers index, name AND family for every dimension.
• Two schemas with the same dimension but different ordering → different hash.
• Training, replay, inference and live compare the SAME canonical value.
• Current value (2026-08-19): 235b8fccc96b7e0e

---

3. Snapshot (Feature70Snapshot)

Immutable after construction (features/features70.py). Carries:

schema_id, schema_version, symbol, timeframe, timestamp_utc,
feature_vector[70], feature_names, source_state, news_status,
liquidity_status, news_available, liquidity_available,
news_context_hash, calculation_version, model_id, model_version,
feature_schema_hash

No downstream subsystem may reorder or reinterpret dimensions. validate() re-checks dimension, hash, finiteness and bounds; violation → explicit error.

4. Family states (never silently guessed)

State	Meaning
FEATURE_AVAILABLE	family computed from real data
FEATURE_DISABLED	family explicitly off (neutral block, e.g. news off → 10 zeros)
FEATURE_UNAVAILABLE	family declared on but no data → SNAPSHOT/BLOCK (never fake)
FEATURE_INVALID	data present but violates contract → BLOCK

Missing values follow the documented conventions (NaN/Inf → 0.0 then clip [-3,+3]; liquidity "far" → 3.0; no pool → documented neutral). NEVER a random sentinel.

5. Dataset builder (compute_70d_frame)

• Same canonical 55-bar causal window for the 50D engine as the 50D/60D paths; FULL causal history for the liquidity engine (matches live governor).
• Per-row: news_context_at(t) (causally correct) → news 10D; liquidity engine as_vector().
• News absent → FEATURE_DISABLED + zero block (explicit).
• verify_70d_artifact: dimension==70, all finite, all in [-3,+3], schema_id==scalp_v3, schema hash correct, no duplicate timestamps/sample ids, exact per-reason rejection counts (brief 41).
• Reproducible: same source + same schema + same code → identical frame.

6. Replay (replay_70d_vector)

Rebuilds the canonical vector at a historical timestamp from raw bars + news only (no dataset artifact readback). Future bars/news/liquidity events invisible (causal filter t <= decision). Bit-exact parity with the dataset row for the same timestamp is asserted by tests (TEST-70D-PARITY-08).

7. Inference validation (InferenceValidator)

Full chain before any model call:

EXPECTED_SCHEMA -> ACTUAL_SCHEMA -> DIMENSION -> ORDER -> HASH
-> FINITE -> BOUNDS -> NEWS/LIQUIDITY -> FRESHNESS -> SCALER -> INFERENCE

Explicit rejection codes (brief 29):

SCHEMA_MISMATCH / DIMENSION_MISMATCH / FEATURE_ORDER_MISMATCH /
SCHEMA_HASH_MISMATCH / SCALER_MISMATCH / NONFINITE_FEATURE /
OUT_OF_RANGE_FEATURE / NEWS_UNAVAILABLE / LIQUIDITY_UNAVAILABLE /
STALE_FEATURES

NEVER silently repairs, pads, truncates or substitutes. Schema metadata cached at construction; no DB / file I/O on the per-tick path (INV-001).

8. Model compatibility (legacy protection)

Model	Runtime	Result
scalp_v2 / 60D	60D vector	PASS
scalp_v2 / 60D	70D vector	BLOCK (SCHEMA_MISMATCH)
scalp_v3 / 70D	70D vector	PASS
scalp_v3 / 70D	60D vector	BLOCK (SCHEMA_MISMATCH)

Scaler: dimension must equal feature dimension; NEVER padded/truncated to match (SCHEMA_INCOMPATIBLE stops). Legacy 50D/60D models remain loadable and receive exactly their contracted vectors.

9. Runtime hook (Runtime70Hook)

• Observability-only: never touches orders/SL/TP/RiskEngine/execution.
• Thread-safe toggles (news/liquidity) — toggle never corrupts the vector; dimension stays 70; family status reflects the toggle.
• [FEATURE_CONTRACT] structured trace: VALID (schema/dimension/hash/news/ liquidity/status) or REJECTED (reason, expected/actual, correlation_id).
• Per-stage timing: base/news/liquidity/assembly/validation ms.
• Model gate: 70D model + liquidity disabled → MODEL_INPUT_UNAVAILABLE (no inference for that snapshot), never fake values.
• UI/API state: to_state_dict() exposes the same schema/dimension/toggles the engine actually uses (no frontend-only state).

10. Performance (brief 49, 200-bar M1 window, measured 2026-08-19)

Stage	p50	p95	max
50D base construction	3.1 ms	9.0 ms	19.2 ms
70D assembly + validate	4.1 ms	9.3 ms	11.2 ms
validator per tick	0.15 ms	0.90 ms	1.84 ms
runtime70 live snapshot	27.3 ms	33.9 ms	39.9 ms
replay_70d full rebuild	19.4 ms	108.6 ms	108.6 ms

All within the 100 ms hot-path budget. No DB access on the feature path (INV-001, asserted by test).

11. Tests

Suite	File	Covers
contract	test_70d_contract_parity_task3.py	dimension, schema id, hash, family layout, vector validation (PARITY-01..05)
dataset	test_70d_dataset_parity_task3.py	builder 70D, news disabled explicit, reproducibility, snapshot (PARITY-06..10/19/41)
replay	test_70d_replay_parity_task3.py	dataset==replay, anti-leakage, golden corpus (PARITY-05/08/18/30)
validator	test_70d_inference_validator_task3.py	rejection codes, scaler, 60D/70D matrix (PARITY-09..13/21..24)
runtime	test_70d_runtime_hook_task3.py	four toggle combos, toggle safety, model gate (PARITY-14..17/25/26)
perf	test_70d_perf_task3.py	hot-path budget + no-DB proof
golden	tests/helpers/golden70d.py	11 deterministic scenarios

12. Error codes (runtime, structured)

[FEATURE_CONTRACT] event=VALID|REJECTED reason= expected= actual= correlation_id= model_id= model_version=