Per-file 'flyctl ssh console' acks ride WireGuard; travel networks throttle
it past the 120s timeout, and the raised TimeoutExpired killed whole
backlog runs mid-flight (52/64 files, twice). The delete invariant is
one-directional — the box may delete ONLY what the Mac acked — so a
MISSING ack is always safe. Batch at end, best-effort, log and continue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A TimeoutExpired get leaves a size>0 partial that the exists-check treated
as success, so the console fallback never ran for timeouts; the caller's
parquet read then failed (no magic bytes), deleted the file, and the next
run repeated the identical loop — 8 hours of tape-head stall overnight
2026-07-23. Partials from timeouts are now removed and the fallback runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FINDINGS: full round-3 section (mechanism, the 81%-vs-26% audit, how the
paper harness caught it, corrected verdicts, three standing rules) +
scorecard updates (surge dead, sub5c dead 0/38, oracle revived at E>=0.07)
+ correction banner on the 07-20 tape-era section. HANDOFF: queue +
snapshot reflect the kill, surgebot's instrument role, scorer law, Friday's
combined agenda. research/README: SCORER LAW + independent-instrument rule
+ full current layout. recorder/README: sync transport fallback. README:
/surge dashboard + research row truth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
flyctl sftp flaked again (900s timeouts, magic-byte-less partial parquets)
and 14h of tape queued behind it — the research nightly waits on tape
freshness by design, so everything downstream stalled too. Port ingest.py's
proven console fallback: sftp gets 240s, then base64 over ssh console
(600s); the manifest row-count verify remains the integrity gate and
partials are cleaned on both paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nightly Mac-coupled bulk ingest was the fragile link (today: Mac slept
through 08:00, the pipeline started 09:31, and the sftp bulk pull wedged in
timeout cascades holding the db lock — digest blocked behind it). Stage 0
moves the fold onto the box and reduces the Mac to an incremental mirror:
- recorder/fold.py (sidecar, capture stays PID 1): closed gz segments ->
zstd parquet /data/parquet/<fam>/date=*/segment.parquet, row-parity
verified, manifest-logged, THEN gz deleted. Deletion invariant STRONGER:
raw needs a verified parquet; parquet needs a Mac ACK before the disk
guard may prune it. duckdb capped 384MB; VM 256MB -> 1GB.
- recorder/sync_tape.py (com.jaxperro.tape-sync every 15 min + daily.sh):
manifest-driven incremental sftp pull, per-file row verify, append into
rtds.duckdb NATIVE tables (views-over-parquet rejected: sim's per-asset
point queries would crawl), segment-keyed idempotence shared with the
legacy ingest.py (kept as fallback), ack back to the box.
- recorder/bootstrap_parquet.py: pre-fold history exported to the mirror,
parity OK (13.84M trades + 3.07M aux). live/parquet/ is now the complete
durable layer Stage 1 (MotherDuck/ClickHouse) would consume.
- research/tape.py connect(): brief retry — the 15-min sync holds the
write lock for seconds.
First run: backlog folded in <60s on the box, 36/36 files mirrored+
verified, rtds.duckdb 13.8M -> 18.2M trades, tape age 24h+ -> ~15-45 min.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ingest: sftp needs stdin=DEVNULL under launchd (no tty) — without it, empty/partial files decoded to 0 lines and the missing integrity guard MARKED+DELETED them (36 segments lost). Guard: a >200B gz that yields 0 lines is a failed fetch, left on box to retry.
digest: retry+backoff moved INTO _post so the digest send (not just the start ping) survives the post-wake DNS gap; ingest ping gets the same.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jaxperro