The nightly Mac-coupled bulk ingest was the fragile link (today: Mac slept
through 08:00, the pipeline started 09:31, and the sftp bulk pull wedged in
timeout cascades holding the db lock — digest blocked behind it). Stage 0
moves the fold onto the box and reduces the Mac to an incremental mirror:
- recorder/fold.py (sidecar, capture stays PID 1): closed gz segments ->
zstd parquet /data/parquet/<fam>/date=*/segment.parquet, row-parity
verified, manifest-logged, THEN gz deleted. Deletion invariant STRONGER:
raw needs a verified parquet; parquet needs a Mac ACK before the disk
guard may prune it. duckdb capped 384MB; VM 256MB -> 1GB.
- recorder/sync_tape.py (com.jaxperro.tape-sync every 15 min + daily.sh):
manifest-driven incremental sftp pull, per-file row verify, append into
rtds.duckdb NATIVE tables (views-over-parquet rejected: sim's per-asset
point queries would crawl), segment-keyed idempotence shared with the
legacy ingest.py (kept as fallback), ack back to the box.
- recorder/bootstrap_parquet.py: pre-fold history exported to the mirror,
parity OK (13.84M trades + 3.07M aux). live/parquet/ is now the complete
durable layer Stage 1 (MotherDuck/ClickHouse) would consume.
- research/tape.py connect(): brief retry — the 15-min sync holds the
write lock for seconds.
First run: backlog folded in <60s on the box, 36/36 files mirrored+
verified, rtds.duckdb 13.8M -> 18.2M trades, tape age 24h+ -> ~15-45 min.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the 07-20 CASH≠CHAIN -$1.75: some markets stream RTDS
trades with empty title/outcome/conditionId; the seed stored them
verbatim, on_wallet_activity's 'tok not in self.conds' guard made the
empty value sticky, and settle_resolved's 'if not cond' skip left the
position silently unsettleable until the venue auto-redeemed it.
Three-layer fix:
- RTDS seed: when the payload lacks conditionId/title, enrich from the
token id via _token_market (gamma, cached) before dispatch; the copy
still goes through on a miss.
- conds write: falsy-overwrite instead of key-presence, so a later,
richer source repairs an empty entry.
- repair_market_meta (called at the top of every settle pass, holding
the lock): re-resolves empty conds for my_pos AND open missed entries,
backfills position/bet/missed records, 300s per-token lookup backoff,
and a once-per-token alarm when a position has sat unsettleable >1h.
A repaired token settles in the SAME pass — the Odyssey bet would have
self-healed the moment the market resolved.
The paper book currently carries 4 empty conds and both books have
cond-less open missed entries — the repair pass fixes those on the first
cycle after deploy. tests/test_meta_repair.py covers backfill, backoff,
the single alarm, recovery, and the end-to-end repair->settle un-stick;
full suite 8/8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CASH≠CHAIN -$1.75 root-caused: the 07-16 BikesAreTheBikes copy booked with
empty title/outcome/cond, so settle_resolved could never resolve it; the
venue auto-redeemed $1.694914 on-chain (tx 0x4f303e0e…) while the book
held it open. Hand-settled exactly as settle_resolved would (won @ 1.0,
pnl +$0.68, my_pos/conds cleaned, metadata backfilled from the venue
record) and folded the +$0.050314 fill-rounding remainder as an
adjustments entry (non-trading income, zeroes CASH≠CHAIN, cash credited).
Post: cash == chain $49.324270 (3 RPCs agree), raw drift +0.0003 (0.00).
Machine stopped + heartbeats watched to cessation per the HANDOFF ops
procedure; root cause filed as #18.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FINDINGS: 'The tape era opens' section (sharp screen 742/742 proxy
validation, Study A identity-null, Study B winner's curse, crater refill
clock) + scorecard rows for the two forward-window studies + research/ in
repo layout. HANDOFF: #16/#17 in the open queue, research silo + nightly
launchd in snapshot/ops. README: research/ directory row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The flat 10s retry was calibrated to nothing; the tape now says craters
refill at very different speeds (research requote_timing, 775k crater
prints): crypto 94% within 4s, esports 83% within 10s, sports only 70% at
10s but 76% by 25s, geo/politics tails run minutes (price guard is the
protection there, not the retry). Waits move to FAK_RETRY_NICHE_DEFAULT
{crypto 4, esports 10, sports/geo/politics/other 25}, classified by the
validated research niche patterns (esports before sports so 'LoL: A vs B'
doesn't fall through on ' vs '). cfg fak_retry_niche_s overrides per key;
fak_retry_s stays the fallback (tennis: no measurement yet) and 0 still
disables the feature. Paper-parity: same map both bots. test t8 covers the
map, first-match classing, fallback, and config override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'no orders found to match with FAK order' is the #1 miss class (13 of the
last 48h's misses) — the copy lands in the crater the sharp just swept,
before makers requote. Instead of recording the miss at the first
rejection, hand the OPEN to Copybot.fak_requote_retry: sleep fak_retry_s
(default 10, 0 disables) OUTSIDE the bot lock, then re-run the whole gated
buy path (fresh quote, price guard, depth gate) exactly once, mirroring
the webhook call site's fill drain so a retry fill books lag + bet rows.
A second rejection records the miss tagged 'twice (re-quote retry)'.
Paper-parity: the hook is installed in both modes, and PaperExecutor's
FAK model re-decides on the fresh book. tests/test_fak_retry.py covers
hook scheduling, no-double-count sizing (their_size=0 re-entry), ADD and
non-FAK bypasses, and the end-to-end retry thread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- failed mirror-exits queue for up to 10 heartbeat retries (recovered
exits ping Discord; exhaustion pages ⚠ EXIT STUCK and the position
rides knowingly; in-play holds hand to the pending registry). 3 stub
paths pass incl. the 1.6 spec's fail-fail-fill.
- every RTDS Set E detection appends a durable shadow-ledger row
(lat_s + which trigger won) that rides the publish commit — the 24h
go/no-go data for flipping RTDS on the live app.
- start.sh verifies the boot clone's HEAD against the GitHub API and
re-clones stale replicas (bit twice today).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chain-sweep backfill (784k conds ≈ ~1k calls, not 8 days of per-cond
queries) → shadow audit → consumer flips shallowest-to-deepest →
trust.py simplification last. Ground rules codify the 2026-07-08
alignment-audit discipline: overlay never rewrite, one flip per day,
never coupled with follow-set changes, calibration continuity notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Old app (leaked key ...OdWgOi) deleted, new app wwf-rpc-2026-07
(...w0BxV5) wired into config.json + ALCHEMY_RPC_URL secret. Old key
verified dead (401), new key live, bot copying, Notify webhook intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Railway project magnificent-kindness DELETED (soft-delete, purges
2026-07-10) — retired-infra fully gone; requirements.txt loses its
Railway-detection rationale (stdlib core, optional extras noted).
- tests/test_check_book.py: the Option A orphan-fix regression harness
moves into the repo (was session-local); 6 cases, no network/config.
- daily.sh: calibration step — one row/day of live-book vs backtest
equity to live/history/calibration.csv, the series that sizes real
money (day-zero row seeded: live $1,000 vs model $22,027).
- config.json.bak (untracked, stale copy of the leaked Alchemy key)
deleted locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README: Set E + fresh-book framing in 'system today', /live page in the
dashboard row, --bank in the ops table, book-reset procedure row, gotchas
12-15 (three cache cursors, res_t is endDate metadata / price is the sold
discriminator / redeemable is resolution truth, refund-harvester
archetype, single-writer state-surgery rule). FINDINGS: three new
sections — aligning the three books (7/9 cross-check, honest 29.1k->17.4k,
Set E selection), the refund harvesters, the calibration experiment
(in-sample warning + bank-size ceiling effect). HANDOFF rev 3: fresh
pickup with watch-list, rejected-with-evidence list, Phase 2 framing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-contained handoff for a fresh session: current state (Set D live, honest
P&L, ledger reconciled), the orphan/cash-debit root-cause with the proper fix
(Option A surgical: record every drained buy fill + invariant assert; Option B
architectural: derive cash from the book), Phase 1 status (6/8 done; 1.3 worker
deps + 1.8 live-money dashboard remain), and standing to-dos (rotate Alchemy
key, delete Railway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jaxperro