docs: stage-0 coherence pass across README/HANDOFF/recorder/research

- README: recorder row + architecture diagram reflect fold->parquet-> 15-min mirror (and research/ joins the diagram); cheat-sheet gains a 'sync the tape now' row - HANDOFF: snapshot dated 2026-07-21, launchd roster (tape-sync + research-nightly wait behavior), #15 back in the open queue, tape ops line in quick-reference - recorder/README: ops block (sync_tape + fold health, deploy gap note), storage economics (/mo, mirror as Stage-1 feedstock) - research/README: nightly.sh wait-for-fresh-tape semantics - live/tape_sharps.json: refreshed 4-day chain-validated screen (the smoke run had briefly left a chain-less version) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 15:57:47 +00:00 · 2026-07-21 13:07:31 -04:00
parent e3e05d62d1
commit 8edccda58a
5 changed files with 684 additions and 674 deletions
@@ -23,10 +23,11 @@ REOPENED #4: the redeemer's phantom-cash BOOBY TRAP is fixed + proven
 Builder API Key (SDK gasless relay) the runtime bot deliberately lacks —
 LOW urgency, the platform auto-redeems winners itself. auto_redeem OFF.
 Open now: #1 Signal A, #2 tape research, #4 redeem builder-key, #13 Fri
-bench, #14 edge verdict, #16 surge momentum (forward window OPEN), #17
-oracle fair value (accumulating, nothing frozen). #18 (empty-cond copies
-unsettleable) closed same day: RTDS seed enrichment + falsy-cond repair
-pass + 1h alarm.
+bench, #14 edge verdict, #15 tape-pull batching (Stage-0 fold+mirror
+superseded the bulk path; per-file sftp batching still worth it), #16
+surge momentum (forward window OPEN), #17 oracle fair value (accumulating,
+nothing frozen). #18 (empty-cond copies unsettleable) closed same day:
+RTDS seed enrichment + falsy-cond repair pass + 1h alarm.

 ## Operating boundary (user, 2026-07-13 — standing)
 **Full autonomy on the bots**; the real-money bot **stays ARMED**. Never
@@ -37,7 +38,7 @@ min_order_usd $1 = venue reality). If something looks genuinely dangerous,
 DISARM (`flyctl secrets unset LIVE_CONFIRM -a wwf-copybot-live`) rather than
 push through.

-## Snapshot (2026-07-19)
+## Snapshot (2026-07-21)
 - **wwf-copybot-live** (REAL, ARMED): ~$62 equity ($66.42 contributed,
  realized −$12.17 lifetime — day-one incident + honest recognitions),
  6-wallet Set E rev 4, 4% of working equity/bet, alarm-free after the
@@ -75,13 +76,17 @@ push through.
  #16/#17 forward thresholds (in their issues).
 - Dashboards: jaxperro.com/{trading,live,value} · daily pipeline on the Mac
  at 08:00 (launchd, lockfile) — floors, bench forward table, edge row,
-  tape ingest, Discord digest · research nightly at 09:15
-  (com.jaxperro.research-nightly → research/nightly.sh, self-commits the
-  forward ledger; remove with `launchctl unload`).
+  tape sync, Discord digest · tape mirror every 15 min
+  (com.jaxperro.tape-sync → recorder/sync_tape.py) · research nightly
+  fires 09:15 then WAITS for fresh tape (com.jaxperro.research-nightly →
+  research/nightly.sh, self-commits the forward ledger). All launchd agents
+  removable with `launchctl unload ~/Library/LaunchAgents/<label>.plist`.

 ## Ops quick-reference
 - Follow-set change: edit live/copybot.paper.json → `./live/deploy_bot.sh`;
  mirror config.live.example.json (nothing auto-writes it) + backtest.json.
+- Tape now: `python3 recorder/sync_tape.py` (launchd does it every 15 min);
+  box-side fold health: `flyctl logs -a wwf-recorder` (grep `[fold]`).
 - State surgery: stop the machine, **watch heartbeats actually CEASE**
  (gotcha 15c — and the boot-id guard now makes a zombie writer yield),
  pull → edit → push → start → verify the first heartbeat.
@@ -27,7 +27,7 @@ Three deployed pieces + one static dashboard:
 | **copybot worker** (`copybot.py` via `host/start.sh`) | **Fly.io app `wwf-copybot`, region `arn` (Stockholm), 24/7** — migrated off Railway 2026-07-06: Railway ran it in a US region, which Polymarket's IP geoblock would 403 the moment orders got real; Stockholm is unrestricted AND ~25ms from the CLOB's eu-west-2 primaries. Every boot self-checks the geo-gate (`host/geocheck.py`) and verifies the clone against `git ls-remote` (stale-replica guard) | **T0 = the RTDS trade stream** (gotcha 18): every platform trade, wallet-attributed, **~1s detection** (measured p50 0.8s over 22k msgs), silent-stale guard force-reconnects a quiet socket. Backstops: the Alchemy address-activity webhook (~3s, `POST /alchemy`, signature-verified — **T0b chain seed** 2026-07-14: its tx hashes decode straight from the receipt's OrderFilled logs when the data-api indexer lags, so RTDS-silent markets still copy at push speed), a 5-min backstop poll, and the reconcile janitor. Per-wallet **trade cursor** paginates the activity feed so clip bursts can't scroll past a fetch (H3). 60s heartbeat settles/publishes; paper-copies with real fees/lag/slippage AND live-FAK thin-book reality (**paper FAK parity** 2026-07-16: a quote with no ask inside the protected band books a MISS, not a fill — keeps the live-vs-paper per-signal ratio honest; `live/edge.py` turns it into the daily bankroll-decision verdict in `history/edge.csv` + the Discord digest footer); commits its book back to the repo |
 | **REAL-MONEY worker** (same code, `COPYBOT_ROLE=live`) | **Fly.io app `wwf-copybot-live`, `arn`, 24/7, ARMED 2026-07-10** — separate app so the two books never share a process, state file, or failure mode. Trades the signer's **Deposit Wallet** `0x455e…45a1` (pUSD collateral) via the unified SDK (`polymarket-client`) | Same T0 RTDS detection (`RTDS_DETECT=1`) + Alchemy/poll backstops; **own-fill push** via the CLOB user channel (`userws` in the heartbeat) makes in-play pending holds adopt the moment they match — ws events only *trigger* the resolver, `get_order` stays the arbiter (gotcha 17). Paper-parity sizing (4% of equity, $1 venue floor, no price floor, hard caps retired — the **depth gate** is the scaling rail: skip dust books (ask5c<$50), stake ≤10% of 5c depth — the spread>0.08 SKIP was retired 2026-07-19 as a lag-era relic (guard + protected band bound overpay)). Every placement/exit/settle pings Discord. Disarm: `flyctl secrets unset LIVE_CONFIRM -a wwf-copybot-live` |
 | **VALUE bot** (CLOSED 2026-07-19 — [`value/README.md`](value/README.md)) | ~~Fly app `wwf-valuebot`~~ destroyed after the paper verdict: 994 tickets, 1W/993L, 0.075x — today's sub-2¢ asks are fully informed; the 2025-era calibration edge is gone — a hard SILO: own code (`value/valuebot.py`, zero copybot imports), own state/feed, own image, no shared wallet ever | Systematically buys sub-2¢ contracts (the calibration study's one underpriced bucket — [`value/PLAN.md`](value/PLAN.md)); honest FAK fill model, chain-truth settles, $1 tickets, event cap 1. Verdict at ~2k resolved tickets vs a ~1.05% break-even hit rate |
-| **RTDS tape recorder** ([`recorder/README.md`](recorder/README.md)) | **Fly.io app `wwf-recorder`, `arn`, 24/7** — third silo: no repo clone (code in image), no keys, 25GB volume | Records the FULL firehose (~8M events/day): every trade, order MATCH (maker side), comment, crypto tick — dual-socket capture (one socket alone measured 92.9% coverage; the twin covers per-conn silences). **Nothing deletes until the Mac's nightly ingest verifiably commits it** (~5 weeks offline headroom) → `live/rtds.duckdb` (trades + aux) |
+| **RTDS tape recorder** ([`recorder/README.md`](recorder/README.md)) | **Fly.io app `wwf-recorder`, `arn`, 24/7** — third silo: no repo clone (code in image), no keys, 25GB volume | Records the FULL firehose (~8M events/day): every trade, order MATCH (maker side), comment, crypto tick — dual-socket capture (one socket alone measured 92.9% coverage; the twin covers per-conn silences). **Stage-0 warehouse (2026-07-21): the box folds its own segments into row-verified Parquet on the volume; the Mac mirrors + appends into `live/rtds.duckdb` every 15 min** (`com.jaxperro.tape-sync`). Nothing deletes without a verified second copy (gz needs its parquet; parquet needs the Mac's ack) |
 | **Discord digest** (`live/discord_daily.py`) | end of the daily pipeline | one message/day: the sharp list with profile links + 30-day conviction stats (per-trade pings retired 2026-07-04 for PAPER; the LIVE book pings every real placement/exit/settle) |
 | **dashboard** | [jaxperro.com/trading](https://jaxperro.com/trading) + [jaxperro.com/live](https://jaxperro.com/live) (static, in the `jaxperro` repo) | `/trading` renders the paper book, backtest book and sharp table; `/live` is the REAL MONEY page (reads `live/copybot_live_real.json` — live since 2026-07-10). Both pages share `trading/copybot-section.js` — one renderer, no drift |

@@ -78,10 +78,12 @@ for the phased path; Phase 1 code prep is complete, Phase 2 is funding).
 daily.sh: enumerate → collect → skill (5-gate)          wwf-recorder: FULL firehose:
 → conviction+timing gates → sync_floors (pins)          trades + order matches +
 → portfolio.py (bench forward table) → edge.py          comments + ticks (~8M ev/day,
- (bankroll verdict → digest footer) → tape ingest        twin-socket ~99.9% capture) →
- → dashboard → publish → Discord digest                  25GB volume, download-gated
- cache.duckdb: 26M resolved bets, single-writer;         deletes → nightly ingest →
- TRUSTED rows only may score (trust.py)                  rtds.duckdb (trades + aux)
+ (bankroll verdict → digest footer) → tape sync          twin-socket ~99.9% capture)
+ → dashboard → publish → Discord digest                  → fold.py sidecar → verified
+ cache.duckdb: 26M resolved bets, single-writer;         zstd parquet on the volume →
+ TRUSTED rows only may score (trust.py)                  15-min Mac mirror (sync_tape)
+ research/: edge factory over the tape (SILO;            → rtds.duckdb + live/parquet/
+ studies #16/#17, forward_ledger = the verdicts)         (Stage-1 warehouse feedstock)
 ```

 ---
@@ -99,6 +101,7 @@ for the phased path; Phase 1 code prep is complete, Phase 2 is funding).
 | **reset the paper book** | **stop the machine FIRST** (`flyctl machine stop <id> --app wwf-copybot`), then write a fresh `new_state()` into `copybot_state.json`, archive `copybot_fills.jsonl`, commit+push, then `flyctl apps restart`. Order matters: the running bot's memory is the book's single writer and its publish flow will re-commit its own book over yours (by design — see gotcha 15). Verify the first heartbeat reads `free $1,000/$1,000 · realized $+0` |
 | **watchdog** | two layers, both automatic: fly.toml's http `/health` check (Fly restarts a dark machine) + `.github/workflows/watchdog.yml` (GitHub-hosted probe every ~30-150 min → Discord alert via the `DISCORD_WEBHOOK_URL` repo secret when 3 probes fail). Test: `gh workflow run watchdog` |
 | **run the daily pipeline manually** | `cd live && bash daily.sh` (launchd runs it 08:00; ~40 min, mostly collect). Never run two at once — the cache is single-writer |
+| **sync the tape now** | `python3 recorder/sync_tape.py` — incremental + idempotent (launchd `com.jaxperro.tape-sync` does it every 15 min anyway); box-side fold health: `flyctl logs -a wwf-recorder --no-tail \| grep '\[fold\]'` |
 | **refresh just the sharp list** | `cd live && python3 conviction_scan.py && python3 validate_timing.py` |
 | **daily Discord digest** | sent by `live/discord_daily.py` at the end of `daily.sh`; webhook = `daily_webhook` in gitignored `config.json` |
 | **go real-money** | **DONE 2026-07-10** — [`LIVE_ROLLOUT.md`](LIVE_ROLLOUT.md) executed through Phase 4 (first fills verified end-to-end); the armed bot runs paper-parity rules (caps retired by user decision, superseding rule 0.6). [`HANDOFF.md`](HANDOFF.md) is the current-state entry point |
@@ -2,10 +2,11 @@

 Third silo: records **everything Polymarket's real-time socket emits** —
 every trade, every order match (the maker side of each fill), every market
-comment, every crypto price tick — into a durable, downloadable archive.
-~8M events/day, ~$5-6/month all-in. No keys, no repo clone (code baked into
-the image), no shared anything with the trading bots: a recorder crash can
-never touch trading, a bot deploy can never gap the tape.
+comment, every crypto price tick — into a durable, queryable archive.
+~8M events/day, ~$10/month all-in (1GB VM + volume). No keys, no repo
+clone (code baked into the image), no shared anything with the trading
+bots: a recorder crash can never touch trading, a bot deploy can never gap
+the tape.

 ## Capture design (why it's ~complete)

@@ -59,12 +60,18 @@ name) as sync_tape, so the two can never double-insert.
 ## Ops

 ```bash
-flyctl logs -a wwf-recorder --no-tail          # "tape: N trades/min · aux …"
-python3 recorder/ingest.py                     # manual pull any time
+flyctl logs -a wwf-recorder --no-tail          # capture: "tape: N trades/min"
+flyctl logs -a wwf-recorder --no-tail | grep '\[fold\]'   # fold health
+python3 recorder/sync_tape.py                  # manual mirror pull any time
+                                               # (launchd does it every 15 min)
 flyctl deploy --remote-only -c fly.recorder.toml -a wwf-recorder --ha=false
                                               # ANY code change = image rebuild
+                                               # (~30-60s capture gap — deploy rarely)
 flyctl ssh console -a wwf-recorder -C "df -h /data"   # volume headroom
 ```

-Storage: rtds.duckdb grows ~0.5-0.8GB/day on the Mac (~20GB/mo; 1.4TB free
-as of 2026-07-19 — revisit around the ~100GB mark: Parquet-spool old months).
+Storage: parquet ~250-400MB/day on both the volume and the Mac mirror
+(`live/parquet/`); rtds.duckdb adds ~0.5-0.8GB/day on the Mac. The mirror
+IS the Stage-1 warehouse feedstock (MotherDuck/ClickHouse = a load job,
+not a migration); if local disk ever matters, old mirror partitions can be
+cold-stored — they are immutable and manifest-indexed.
@@ -28,4 +28,6 @@ Layout:
  requote.py       crater→requote timing measurement (retry tuning)
  forward.py       scores frozen studies on new tape days → forward_ledger.jsonl
  params/          frozen study parameters (committed = frozen)
-  nightly.sh       manual/launchd runner (separate from daily.sh)
+  nightly.sh       launchd runner (fires 09:15, then WAITS for fresh tape —
+                   Stage 0 keeps the tape ~15-min fresh whenever the Mac is
+                   awake; deadline 8h, then scores anyway and logs staleness)