Files

T

History

jaxperro 6f6bd81cb1 fix tape ingest data loss + daily digest send

ingest: sftp needs stdin=DEVNULL under launchd (no tty) — without it, empty/partial files decoded to 0 lines and the missing integrity guard MARKED+DELETED them (36 segments lost). Guard: a >200B gz that yields 0 lines is a failed fetch, left on box to retry.
digest: retry+backoff moved INTO _post so the digest send (not just the start ping) survives the post-wake DNS gap; ingest ping gets the same.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-20 13:19:59 -04:00

fly.Dockerfile

recorder: RTDS tape silo — hour-rotated gzip segments on a Fly volume

2026-07-17 12:53:54 -04:00

ingest.py

fix tape ingest data loss + daily digest send

2026-07-20 13:19:59 -04:00

README.md

docs: recorder README (full-firehose design, preservation policy, ops) + README/HANDOFF sync

2026-07-19 16:07:39 -04:00

recorder.py

recorder: never delete un-ingested tape — warn from 80%, single-oldest drop only at 95% as loud last resort (user directive); volume 10→25GB

2026-07-19 16:00:20 -04:00

README.md

RECORDER — the full RTDS firehose tape

Third silo: records everything Polymarket's real-time socket emits — every trade, every order match (the maker side of each fill), every market comment, every crypto price tick — into a durable, downloadable archive. ~8M events/day, ~$5-6/month all-in. No keys, no repo clone (code baked into the image), no shared anything with the trading bots: a recorder crash can never touch trading, a bot deploy can never gap the tape.

Capture design (why it's ~complete)

Dual sockets to wss://ws-live-data.polymarket.com, both subscribed to activity/* + comments/* + crypto_prices/* (rfq/prices probed dead 2026-07-19). The stream silences per-CONNECTION every ~10 min; a single socket measured 92.9% minute-coverage — the twin covers each gap. Cross-socket dedupe on (tx, asset, side, size, price) for trades, payload-hash for aux; a 15s per-connection stale guard (at 3k+ msg/min, 5s of silence is pathological; false trips cost nothing with a twin).
Segments: hour-rotated to the volume — rtds_YYYYMMDD_HH.jsonl.gz (trades, stable schema) + aux_… (everything else, raw payloads).
Preservation (user directive 2026-07-19): nothing is deleted until the Mac has verifiably downloaded it. Only ingest.py deletes, after a committed transaction. The 25GB volume holds ~5 weeks un-pulled; the recorder warns from 80% full and only at 95% drops the single oldest hour per rotation, loudly (⚠⚠ TAPE LOSS) — a full disk would otherwise kill the CURRENT tape too.

The nightly ingest (Mac, daily.sh)

recorder/ingest.py: lists closed segments over flyctl ssh, pulls each (base64 — see issue #8 for the sftp upgrade), inserts into live/rtds.duckdb (trades + aux tables; own DB file so it can never fight cache.duckdb's writer lock), marks it in ingested, then deletes on the box. Idempotent and resumable; a crash mid-segment re-ingests cleanly (single transaction per segment).

Ops

flyctl logs -a wwf-recorder --no-tail          # "tape: N trades/min · aux …"
python3 recorder/ingest.py                     # manual pull any time
flyctl deploy --remote-only -c fly.recorder.toml -a wwf-recorder --ha=false
                                               # ANY code change = image rebuild
flyctl ssh console -a wwf-recorder -C "df -h /data"   # volume headroom

Storage: rtds.duckdb grows ~0.5-0.8GB/day on the Mac (~20GB/mo; 1.4TB free as of 2026-07-19 — revisit around the ~100GB mark: Parquet-spool old months).