mirror of
https://github.com/jaxperro/winning-wallet-finder.git
synced 2026-07-27 15:57:47 +00:00
jaxperro
790ef42c40
FINDINGS: full round-3 section (mechanism, the 81%-vs-26% audit, how the paper harness caught it, corrected verdicts, three standing rules) + scorecard updates (surge dead, sub5c dead 0/38, oracle revived at E>=0.07) + correction banner on the 07-20 tape-era section. HANDOFF: queue + snapshot reflect the kill, surgebot's instrument role, scorer law, Friday's combined agenda. research/README: SCORER LAW + independent-instrument rule + full current layout. recorder/README: sync transport fallback. README: /surge dashboard + research row truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
80 lines
4.5 KiB
Markdown
80 lines
4.5 KiB
Markdown
# RECORDER — the full RTDS firehose tape
|
|
|
|
Third silo: records **everything Polymarket's real-time socket emits** —
|
|
every trade, every order match (the maker side of each fill), every market
|
|
comment, every crypto price tick — into a durable, queryable archive.
|
|
~8M events/day, ~$10/month all-in (1GB VM + volume). No keys, no repo
|
|
clone (code baked into the image), no shared anything with the trading
|
|
bots: a recorder crash can never touch trading, a bot deploy can never gap
|
|
the tape.
|
|
|
|
## Capture design (why it's ~complete)
|
|
|
|
- **Dual sockets** to `wss://ws-live-data.polymarket.com`, both subscribed
|
|
to `activity/*` + `comments/*` + `crypto_prices/*` (`rfq`/`prices` probed
|
|
dead 2026-07-19). The stream silences per-CONNECTION every ~10 min; a
|
|
single socket measured **92.9%** minute-coverage — the twin covers each
|
|
gap. Cross-socket dedupe on (tx, asset, side, size, price) for trades,
|
|
payload-hash for aux; a 15s per-connection stale guard (at 3k+ msg/min,
|
|
5s of silence is pathological; false trips cost nothing with a twin).
|
|
- **Segments**: hour-rotated to the volume — `rtds_YYYYMMDD_HH.jsonl.gz`
|
|
(trades, stable schema) + `aux_…` (everything else, raw payloads).
|
|
- **Preservation (user directive 2026-07-19, STRENGTHENED by Stage 0
|
|
2026-07-21): nothing is deleted without a verified second copy.** The
|
|
fold sidecar deletes a raw gz only after its Parquet re-reads with a
|
|
matching row count; the disk guard deletes Parquet only oldest-first and
|
|
only files the Mac mirror has ACKED. The recorder's own last-resort
|
|
guard (95% -> drop oldest, `⚠⚠ TAPE LOSS`) still protects the live tape
|
|
from a full disk.
|
|
|
|
## Stage-0 warehouse (2026-07-21): fold on the box, mirror on the Mac
|
|
|
|
- **`fold.py`** (sidecar in the same machine, capture is PID 1): every
|
|
2 min, each closed gz segment becomes
|
|
`/data/parquet/<family>/date=YYYY-MM-DD/<segment>.parquet` (zstd),
|
|
row-parity verified, appended to `/data/parquet/manifest.jsonl`, then the
|
|
gz is deleted. The volume IS the warehouse: immutable Parquet any client
|
|
mirrors incrementally. duckdb capped at 384MB (1GB VM) so a busy-hour
|
|
fold can never starve capture. ~250-400MB/day of parquet -> months of
|
|
headroom, months more once mirrored+acked files get pruned.
|
|
- **`sync_tape.py`** (Mac, every 15 min via `com.jaxperro.tape-sync` +
|
|
from daily.sh): pulls new manifest entries over `flyctl sftp` (240s cap,
|
|
then the base64-over-console fallback that carried ingest.py — sftp
|
|
flaked twice in two days, 2026-07-22, leaving magic-byte-less partials),
|
|
row-verifies each file, appends its rows into `live/rtds.duckdb`'s
|
|
native tables (research keeps native speed — views-over-parquet was
|
|
rejected: per-asset point queries would crawl), records it in
|
|
`ingested`, ACKs the box (`/data/parquet/acks/<f>.ok`). Tape freshness
|
|
went from "nightly, if the Mac was awake" to ~15 min whenever awake, and
|
|
the box no longer depends on the Mac to stay healthy for weeks.
|
|
- **`bootstrap_parquet.py`**: one-shot export of the pre-fold history
|
|
(2026-07-17..21, existed only in rtds.duckdb) into the mirror — ran
|
|
2026-07-21, parity OK (13.84M trades + 3.07M aux). `live/parquet/` is
|
|
the complete durable layer Stage 1 (MotherDuck/ClickHouse) would consume.
|
|
|
|
## The legacy bulk ingest (fallback)
|
|
|
|
`recorder/ingest.py` (sftp-first, base64 fallback, integrity guards) stays
|
|
as the fallback path for a fold-less recorder; with fold running it finds
|
|
no gz segments and no-ops. Same idempotence keys (`ingested` by segment
|
|
name) as sync_tape, so the two can never double-insert.
|
|
|
|
## Ops
|
|
|
|
```bash
|
|
flyctl logs -a wwf-recorder --no-tail # capture: "tape: N trades/min"
|
|
flyctl logs -a wwf-recorder --no-tail | grep '\[fold\]' # fold health
|
|
python3 recorder/sync_tape.py # manual mirror pull any time
|
|
# (launchd does it every 15 min)
|
|
flyctl deploy --remote-only -c fly.recorder.toml -a wwf-recorder --ha=false
|
|
# ANY code change = image rebuild
|
|
# (~30-60s capture gap — deploy rarely)
|
|
flyctl ssh console -a wwf-recorder -C "df -h /data" # volume headroom
|
|
```
|
|
|
|
Storage: parquet ~250-400MB/day on both the volume and the Mac mirror
|
|
(`live/parquet/`); rtds.duckdb adds ~0.5-0.8GB/day on the Mac. The mirror
|
|
IS the Stage-1 warehouse feedstock (MotherDuck/ClickHouse = a load job,
|
|
not a migration); if local disk ever matters, old mirror partitions can be
|
|
cold-stored — they are immutable and manifest-indexed.
|