6f6bd81cb1
fix tape ingest data loss + daily digest send
...
ingest: sftp needs stdin=DEVNULL under launchd (no tty) — without it, empty/partial files decoded to 0 lines and the missing integrity guard MARKED+DELETED them (36 segments lost). Guard: a >200B gz that yields 0 lines is a failed fetch, left on box to retry.
digest: retry+backoff moved INTO _post so the digest send (not just the start ping) survives the post-wake DNS gap; ingest ping gets the same.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-20 13:19:59 -04:00
7086569024
ingest: sftp timeout 900s, fallback timeout contained, both-transports-failed skips instead of crashing the backlog ( #11 hardening)
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 17:12:45 -04:00
ab56b2aa4b
reconcile_entries cadence 5m→30m ( closes #12 ); tape transport sftp-first with base64 fallback ( closes #11 )
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 16:46:28 -04:00
2744b67587
chain-balance gates in reconcile_exits + retry queue — ghosts dropped, sells clamped to chain ( closes #7 ); tape ingest pings the daily channel at start + finish
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 16:34:52 -04:00
ea69aec127
docs: recorder README (full-firehose design, preservation policy, ops) + README/HANDOFF sync
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 16:07:39 -04:00
b8c0734c5a
recorder: never delete un-ingested tape — warn from 80%, single-oldest drop only at 95% as loud last resort (user directive); volume 10→25GB
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 16:00:20 -04:00
77b2457523
recorder: FULL firehose — activity/* (trades + orders_matched), comments, crypto_prices; aux segment family + ingest branch (probed: rfq/prices dead)
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 15:55:10 -04:00
5e375f6d88
recorder: dual-socket capture + cross-socket dedupe + 15s stale guard — single socket measured 92.9% minute coverage; the twin covers per-connection silences
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 15:37:52 -04:00
baa4182343
audit hardening: book fully serialized, chain-gated sweep, boot-id single-writer guard, TLS on user-ws, atomic valuebot state, hot-path throttle, newest-kept seen_tx, daily lock, ingest txn, AIcAIc pin
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 14:07:43 -04:00
006d79ac18
ingest: 900s per-segment timeout — busy-hour tape outgrew 300s over ssh
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 13:43:06 -04:00
38c2d51366
ingest: resolve flyctl absolutely — launchd's minimal PATH broke the first nightly run
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-19 12:18:56 -04:00
315990cd90
recorder: reset freshness clock on reconnect (insta-close loop bug) + 45s stale guard — cuts per-flap tape loss
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-18 09:06:45 -04:00
ddf1e97a54
recorder: nightly tape ingest into live/rtds.duckdb (own DB — never contends with the bet cache lock)
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-17 13:02:41 -04:00
3a3e366de5
recorder: RTDS tape silo — hour-rotated gzip segments on a Fly volume
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-17 12:53:54 -04:00
jaxperro