The nightly Mac-coupled bulk ingest was the fragile link (today: Mac slept
through 08:00, the pipeline started 09:31, and the sftp bulk pull wedged in
timeout cascades holding the db lock — digest blocked behind it). Stage 0
moves the fold onto the box and reduces the Mac to an incremental mirror:
- recorder/fold.py (sidecar, capture stays PID 1): closed gz segments ->
zstd parquet /data/parquet/<fam>/date=*/segment.parquet, row-parity
verified, manifest-logged, THEN gz deleted. Deletion invariant STRONGER:
raw needs a verified parquet; parquet needs a Mac ACK before the disk
guard may prune it. duckdb capped 384MB; VM 256MB -> 1GB.
- recorder/sync_tape.py (com.jaxperro.tape-sync every 15 min + daily.sh):
manifest-driven incremental sftp pull, per-file row verify, append into
rtds.duckdb NATIVE tables (views-over-parquet rejected: sim's per-asset
point queries would crawl), segment-keyed idempotence shared with the
legacy ingest.py (kept as fallback), ack back to the box.
- recorder/bootstrap_parquet.py: pre-fold history exported to the mirror,
parity OK (13.84M trades + 3.07M aux). live/parquet/ is now the complete
durable layer Stage 1 (MotherDuck/ClickHouse) would consume.
- research/tape.py connect(): brief retry — the 15-min sync holds the
write lock for seconds.
First run: backlog folded in <60s on the box, 36/36 files mirrored+
verified, rtds.duckdb 13.8M -> 18.2M trades, tape age 24h+ -> ~15-45 min.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From the automated-writer clobber audit (3-agent workflow):
- sync_floors now ACTUALLY mirrors floor_pin into backtest.json (the comment
claimed it; the code never did — so a paper pin diverged from the backtest's
dynamic p80, defeating calibration). Kruto 80 now consistent across all 3.
- sync_floors warns loudly when about to revert an UNPINNED floor that looks
hand-edited (>5 and >15%) — floor_pin is the only protected key, so a direct
edit used to vanish silently at 08:00.
- validate_timing logs the sharps add/drop delta and documents watch_sharps.json
as generated-only (format kept a plain list — dashboard/discord read it so).
- daily.sh warns if copybot.paper.json vs backtest.json class_pct desync.
Audit verdict otherwise CLEAN: sync_floors is the only pipeline step that
mutates a manual-knob file, and it round-trips the whole JSON touching only
floor — all other knobs (caps, guard, depth_gate, follow set, band) survive.
config.live.example.json is untouched by every automated writer (safe by
neglect). No second floor_pin-class bug exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Railway project magnificent-kindness DELETED (soft-delete, purges
2026-07-10) — retired-infra fully gone; requirements.txt loses its
Railway-detection rationale (stdlib core, optional extras noted).
- tests/test_check_book.py: the Option A orphan-fix regression harness
moves into the repo (was session-local); 6 cases, no network/config.
- daily.sh: calibration step — one row/day of live-book vs backtest
equity to live/history/calibration.csv, the series that sizes real
money (day-zero row seeded: live $1,000 vs model $22,027).
- config.json.bak (untracked, stale copy of the leaked Alchemy key)
deleted locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_handle_their_buy referenced their_prev in the fresh-OPEN branch but only
assigned it in the is_add branch — every new position crashed with
UnboundLocalError the moment the their-bet ceiling landed 2026-07-06. The
webhook/heartbeat try/except swallowed it as 'handler error', so the bot
logged FOLLOW then silently placed NOTHING (last real fill 07-05 23:17;
live Fly logs show FOLLOW ArbTrader ... -> heartbeat error: their_prev).
This is the primary cause of 'live bot missing bets'. Fix: hoist their_prev
above the if/else so both branches see the signal's prior position.
Also (the user's two asks):
- copybot on_wallet_activity: a QUALIFYING bet we were too slow to catch
(webhook missed / bot down / fast market resolved before we polled) is now
recorded as a missed bet 'too slow to follow (Nm late)' instead of silently
dropped; below-floor dust stays console-only. Filter runs before the stale
gate so we know if it would have qualified.
- sync_floors.py rewritten: PIN each paper-bot wallet's floor to the trusted
cache p80 (identical to the backtest's conv_thr), written to
copybot.paper.json, wired into daily.sh + committed. Kills the boot-time
data-api floor drift (fortuneking $1,498 boot vs $892 backtest) that made
the live bot filter out bets the backtest kept.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run from a mktemp snapshot so committing daily.sh while a run is in flight
can't shift bash's read offset (the 2026-07-06 phantom-syntax-error crash),
and let the publish-step rebase autostash so a dirty tree can't wedge the
auto-push behind the copybot's feed commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One plain line when the cache refresh starts, so the end-of-run digest
isn't the only sign of life. Non-fatal if the webhook is missing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* per-trade Discord pings removed everywhere: copybot placement/settle/startup
alerts silenced (engine.webhook forced off), webhook_receiver.py archived,
Procfile removed. The ONLY Discord output is now live/discord_daily.py at the
end of the daily pipeline: one embed listing every watch_sharps wallet with a
Polymarket profile link + 30D conviction win% / record / P&L.
* trust.py fixes found on the first scheduled run (sharp list collapsed 25->7):
- Polymarket rewrites endDate after resolution, so freshly re-pulled wallets'
corrected res_t stopped matching the stale cross-wallet consensus
(374/454 of iohihoo's rows wrongly distrusted). v2 rows with resolved=TRUE
are now self-certifying (resolved=TRUE implies endDate-based res_t observed
post-end) — consensus only gates legacy rows.
- daily.sh invalidates watchlist wallets by DELETING their pulled row; a
transiently failed re-pull then dropped the wallet's entire history from
the trusted set (the 0x73afc816 whale vanished). Legacy rows of wallets
missing from pulled now fall back to trusting markets resolved >=14d ago.
Re-run: 85 conviction wallets, 34 copy-positive sharps — whale/iohihoo/
Stavenson/LSB1/Kruto all recovered.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New portfolio.py replays the followed wallets' conviction bets ($1k/$50,
hold-to-resolution) from cache.duckdb — which has res_t, so capital
RECYCLES at the true resolution time (no client-side clob storms, no
phantom-locked capital) and covers the full bet history (~40 pages, not
the browser's 4). Writes portfolio.json (equity, splits, current/
resolved/missed tables, per-wallet); wired into daily.sh + publish. The
dashboard now renders the top page from this feed (1 request) with the
client-side replay kept as a fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The daily run regenerates dashboard.html but the publish step never
committed it, leaving it perpetually modified. Add it to the publish git
add list and commit the current snapshot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
daily.sh now POSTs a summary (finish time, sharp count, push status) to a
Discord webhook after the publish step. The webhook URL is read from the
gitignored config.json (daily_webhook key) so it never lands in the public
repo; the ping is skipped if the key is absent. Sends a browser User-Agent
(Discord 403s requests without one).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the sharps pipeline (conviction_scan.py + validate_timing.py) to the
daily run so watch_sharps.json self-updates, and a publish step that
commits + pushes the refreshed outputs (watch_skilled / watch_sharps /
conviction_wallets) so the live dashboard at jaxperro.com/trading picks
up the new set. Also force-refresh the sharps watchlist (not just
skilled) before the cache top-up for accurate forward stats. Publish
does pull --rebase before push so a diverged remote can't wedge it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
live/: operationalizes the LBS/Yale "skilled ~3%" result against the live
data-api. Enumerate recent liquid markets -> top traders -> candidate pool;
cache every wallet's resolved bets once in DuckDB (~26k wallets / 12.5M bets,
keyed by per-bet resolution time so any cutoff re-scores in seconds); 5-gate
skill funnel (n>=15, z>0, BH-FDR, split-half OOS, MM/bot cap); dashboard +
daily refresh.
Key finding: copying the high-win-rate "favorite-rider" cohort looks +23.6%
in-sample but loses -7.4% once selected on pre-June-1 data only (99% -> 68%
win rate) — selection bias, reproducing the paper's "lucky winners revert"
result on live data. Win rate != edge, again.
wide/: bulk subgraph->DuckDB scanner (survivorship-bias-free over all wallets),
but the public subgraph is frozen at Jan 2026 -> historical tool only.
Large local data (*.duckdb, candidates.json, *_scored.json, history/) gitignored.
README + FINDINGS updated with the current logic and the clean result.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jaxperro