The flat 10s retry was calibrated to nothing; the tape now says craters
refill at very different speeds (research requote_timing, 775k crater
prints): crypto 94% within 4s, esports 83% within 10s, sports only 70% at
10s but 76% by 25s, geo/politics tails run minutes (price guard is the
protection there, not the retry). Waits move to FAK_RETRY_NICHE_DEFAULT
{crypto 4, esports 10, sports/geo/politics/other 25}, classified by the
validated research niche patterns (esports before sports so 'LoL: A vs B'
doesn't fall through on ' vs '). cfg fak_retry_niche_s overrides per key;
fak_retry_s stays the fallback (tennis: no measurement yet) and 0 still
disables the feature. Paper-parity: same map both bots. test t8 covers the
map, first-match classing, fallback, and config override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'no orders found to match with FAK order' is the #1 miss class (13 of the
last 48h's misses) — the copy lands in the crater the sharp just swept,
before makers requote. Instead of recording the miss at the first
rejection, hand the OPEN to Copybot.fak_requote_retry: sleep fak_retry_s
(default 10, 0 disables) OUTSIDE the bot lock, then re-run the whole gated
buy path (fresh quote, price guard, depth gate) exactly once, mirroring
the webhook call site's fill drain so a retry fill books lag + bet rows.
A second rejection records the miss tagged 'twice (re-quote retry)'.
Paper-parity: the hook is installed in both modes, and PaperExecutor's
FAK model re-decides on the fresh book. tests/test_fak_retry.py covers
hook scheduling, no-double-count sizing (their_size=0 re-entry), ADD and
non-FAK bypasses, and the end-to-end retry thread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Diagnosis of the 122s paper avg (user report): RTDS SAW every slow fill at
~0.3s (shadow ledger proof) but the copy landed 130-305s later — because
on_wallet_activity threw the RTDS payload away and re-fetched from the
data-api, whose indexer lagged that long on badaf/1kto1m crypto+index
markets, so the push was wasted and the 300s backstop poll did the copy.
Fix (option B as augmentation): the RTDS message carries every field
handle_trade needs, so seed it into on_wallet_activity's candidate set
(deduped by tx). The re-fetch + fill-split merge still run — the seed just
guarantees the trade is present at ~1s. detect_lag now reflects true RTDS
delivery (~1s) instead of indexer lag. 3 stub paths pass (seed-copies-on-
empty-data-api, dedup, no-seed-backstop-unaffected).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- _fetch_since_cursor cold start (no cursor) now seeds at now-600s like the
pre-cursor fetch — the 02:10 boot walked 500 rows/wallet of history through
the funnel and spammed ~230 phantom 'too slow (20,000m late)' missed rows
per book (user saw them on the dashboard). Purged from both books
(116 paper + 114 live; genuine misses remain).
- feed lag tile falls back to the LIFETIME average when the 24h window is
empty (the live page showed '—' after a copy-less day).
- bet records now carry lag_s so the dashboard shows per-copy exec lag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 4dp bound (sub-penny fix) violated tick conformity on 1c books —
'max_price must conform to tick size 0.01' failed a winning SPX copy at
14:36Z. The quoted price is always a tick multiple, so its own decimal
count is the finest safe precision: bound rounds to quote precision
(coarser is always valid on finer ticks, never the reverse). Verified
across 2dp/3dp/sub-penny quotes both sides.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
System-today table: T0 RTDS detection (~1s) on both workers, own-fill push
+ depth gate on the real-money row, clone-guard note. Calibration section:
Set E rev 2 (5 wallets, drops explained, bench of 4 in backtest). Model
section: guard validation evidence + the depth-gate bullet. Data sources:
RTDS + CLOB user-channel rows. Gotcha 17 gains the 2026-07-12 corollary
(balance-diff is a fallback, one pending per token); gotcha 18 documents
the RTDS stream's two quirks (broken filters, silent-stale) and the
heartbeat-fields-over-boot-logs lesson. copybot.py docstring reflects the
layered trigger stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DEPTH GATE (copytrade.book_depth + gate in _handle_their_buy, fitted on
131 book-annotated fills): pre-order, skip if spread>0.08 (market mid-move,
median |slip| ~14%) or ask5c<$50 (dust books mispriced every fill), else
cap the stake at 10% of 5c ask depth (keeps impact <~2%; >20% of depth made
>+2% slip 33-50% likely). Fail-open: a failed book fetch declines to bind.
This is what lets stakes scale — binds mainly on slow-market wallets like
0xbadaf319 (median 23% of depth at $40-100 stakes). config.depth_gate;
paper+live inherit DEFAULT_CONFIG.
GUARD: counterfactuals VALIDATED the 0.05 absolute guard — it sits at the
EV knee (0.05-0.10 moves ≈ breakeven, >0.10 = -20% ROI). No change, comment
records the evidence. (The +9-14% lag-drift tail in deep esports books is a
detection-lag problem the depth gate can't see; RTDS ~1s already addresses
it — protected max_price is the backstop.)
CLOB_USER: UserFillsListener streams our order/trade lifecycle from
ws-subscriptions-clob/ws/user (auth = client.credentials, the SDK's derived
L2 creds; all-markets, no per-market mgmt). A matching order id triggers an
IMMEDIATE resolve_pendings() — in-play holds adopt on match, not on the 60s
tick. Events only TRIGGER; get_order stays the arbiter (2026-07-12
anti-phantom-cash invariant intact). resolve_pendings now re-entrancy-locked
(ws + heartbeat race). 60s poll stays as the fallback; ws failure degrades
to today. Live role only.
Also: sub-$1 BUY amount bumped to the $1 venue minimum (share-flooring
shaved gated $1.00 stakes to $0.99 — 7 of 13 live rejections).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- H3 (burst overflow): per-wallet trade cursor + paginated activity fetch
(up to 5×100, walks back to the last processed timestamp) — a gkmg-scale
clip burst can no longer scroll past a single newest-100 page. Cursor
only advances after a fetch; 600s overlap re-serves boundary rows into
the seen-tx dedupe. offset pagination verified against the data-api.
- clone-guard: Bearer auth (fine-grained PATs reject the 'token' scheme)
with unauthenticated fallback + 40-hex sanity on the answer — Fly's
shared egress IPs rate-limit the anonymous path.
- preflight_live.py rewritten for the unified SDK: deposit-wallet auth,
pUSD collateral, order-book reach, RTDS stream delivery, geo verdict.
- ledger-drift ALARM floor 1c → 5c (4dp fill rounding accumulates pennies
across 60+ copies; the check stays for real bugs — it caught $15.45 and
$7.24. The self-heal path keeps its own threshold.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User call 2026-07-13 after a big 0.001 longshot was floor-blocked (the
same band class that blocked the 16x Hive copy at 0.05). Executor's
protected prices now round to 4dp and scale with the price — the old 2dp
rounding zeroed the bound on sub-penny books (BUY never bounds below the
quoted cross; SELL never above the quoted bid; verified at 0.001/0.002).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 phantom-cash incident (+$7.86 book-only, alarms caught it):
overlapping pendings on one token each read the same balance drop via
max(matched, diff) — one real 1.48-share sell booked 3×. No real money
moved wrongly (the CLOB rejects oversells); the book just self-inflated.
Fixes: (1) order's own size caps every adoption fill; (2) the balance
diff is consulted ONLY when the exchange no longer answers for the order
(a definitive killed/0 is final); (3) SELL adoptions cap at book
holdings; (4) ONE outstanding pending per token — engine buy/sell and
exit-retries refuse while the resolver owns a token. Regression suite:
the incident scenario books exactly the real fill; phantoms zero.
Surgery (machine stopped): cash → chain $11.206, the two poisoned bets
corrected from real chain fills (residual adjustment −$0.08), drift 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 04:00Z: the socket sat 'up' delivering nothing for 35 min with
no close event, so the on-close reconnect never fired (backstops covered
detection). The ping thread now force-closes a 120s-silent stream to
trigger the clean reconnect path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- failed mirror-exits queue for up to 10 heartbeat retries (recovered
exits ping Discord; exhaustion pages ⚠ EXIT STUCK and the position
rides knowingly; in-play holds hand to the pending registry). 3 stub
paths pass incl. the 1.6 spec's fail-fail-fill.
- every RTDS Set E detection appends a durable shadow-ledger row
(lat_s + which trigger won) that rides the publish commit — the 24h
go/no-go data for flipping RTDS on the live app.
- start.sh verifies the boot clone's HEAD against the GitHub API and
re-clones stale replicas (bit twice today).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Polymarket's real-time data socket (wss://ws-live-data.polymarket.com,
topic activity/type trades — undocumented but official: powers the site's
live feed, spec'd in Polymarket/real-time-data-client) streams every
platform trade with proxyWallet attribution. Probe 2026-07-10: median
0.8s delivery over 22k msgs, ~45/s at peak, zero drops in 45 min.
RtdsListener: unfiltered subscribe (server-side filters broken, RTDC#34),
client-side Set E match → the same on_wallet_activity funnel as the
Alchemy push (locks internally). Capped-backoff reconnect forever; if the
lib or stream is missing, detection degrades to the existing backstops
(Alchemy ~3s, 300s poll, reconcile janitor) — never to zero. Heartbeat
shows 'rtds up Ns' / '⚠ rtds down'. Paper role: ON by default (24h shadow
run starts now). Live role: OFF until the shadow validates —
RTDS_DETECT=1 enables. 5 stub paths pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1. Caps retired: risk block mirrors the paper sentinels; sizing is the
paper bot's 4%-of-equity (class_pct 0.04/0.12), floored at the venue's
$1 min order (4% of a $22 book is $0.89 — sub-min stakes died at the
gate). bankroll rebased to the real $22.28 equity; spend tracker reset
(the confusing $35 is gone with the daily cap).
2. Price guard is now ABSOLUTE +0.05 (both books): 0.14→0.15 follows,
0.14→0.20 skips. The relative 5% blocked one-tick moves on cheap
in-play books.
3. Pending-order registry: in-play 'delayed' holds are no longer cancelled
at 20s — the executor hands them to state.pending_orders with full copy
context; the heartbeat resolver adopts the fill whenever it lands
(bets/my_pos/cash/ledger, TTL 600s → cancel + honest miss). Recovers
Rune-Eaters-class holds (+$7.50 forfeited by the old cancel).
5 stub-client paths pass incl. adopt + expire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 09:10-13:14Z: six $5 in-play esports copies returned AcceptedOrder
status 'delayed' with zero matched; the executor read matched==0 as a
rejection, recorded honest misses, and forgot the orders — which then filled
at the exchange minutes later. Book went blind (6 untracked fills, ~$30 real
spend vs $5 booked), caps stopped binding, exits never mirrored. Net damage
contained by the CLOB's own balance checks: −$2.7 incl fees, two legs won.
Executor invariant now: NO order outlives _order() untracked. Zero-matched
acceptances poll get_order → cancel remainder → measure truth via the
exchange's CONDITIONAL balance diff; exception paths sweep open orders on
the token the same way. 5 stub-client unit paths pass.
State surgery (machine disarmed first): cash reset to chain ($10.03), six
fills adopted as bets (4 settled −$4.38 w/ fees, Aurora G2 open+won), spend
tracker set to the real $35/day, false miss rows purged, one documented
-0.59 adjustment (bank-vs-deposit + probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
py-clob-client is archived; the CLOB rejects its orders globally. New
LedgerLiveExecutor: SecureClient.create(private_key) — deposit wallet
auto-resolves, no api_key at runtime; place_market_order FAK/FOK with
protected prices (quoted ±live.max_slippage_pct, default 5%, clamped to
[0.01,0.99]); fills parsed from AcceptedOrder (BUY: making=USD given,
taking=shares got; SELL reversed — semantics PROVEN by today's $5 round
trip: buy matched 7.35294 @ 0.68, sell matched 7.35 @ 0.67). Still never
raises into the trade loop. chain_cash_gap repointed at the deposit
wallet's pUSD (was the emptied legacy proxy → CASH≠CHAIN +24.73 alarm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of zero live placements: the FIRST two qualifying signals
(Kruto 21:21, badaf Epic 21:53) each CRASHED the bot inside the
unguarded live order path (machine events: exit_code=1 seconds after
each signal; tick-size rejection the likely thrower — price was rounded
to 3dp on 1c-tick books). Fly restarted it and boot baseline marked the
fresh trades seen-without-copying: crash -> restart -> trade eaten,
twice. Fixes: _order never raises (failures return ok:False and feed
the new missed-row instrumentation); tick-size-aware pricing (cached
get_tick_size, buys ceil / sells floor, 0.01 fallback valid on every
book); size floored to 2dp; per-wallet poll guard (parity with the push
handler); baseline exempts trades younger than the stale window so a
restart can never eat a copyable trade again. host/order_probe.py =
controlled end-to-end placement proof (~$2 round trip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fast in-play markets can resolve before the 5-min pass sees the sharp's
sell (and pre-2026-07-09 rows predate exit-aware settling) — won/lost
stamps hid real pre-resolution sells. A redeem prints exactly the
payout; an exit print >2c away revalues the counterfactual at the
sharp's exit. Idempotent via exit_checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The missed ledger valued everything hold-to-resolution — the wrong
counterfactual for a mirror strategy (Kruto's 3c Hive entry: he banked
16x selling at 48c; the paper book mirrored for +$722; the live book's
missed row would have shown the map result instead). Three affirmative
facts before settling (same as reconcile_exits), their reconstructed
exit price, entry+exit taker fees. Also: the reconcile miss reason is
now honest — 'not copied in the detection window (reconciled)'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bets{} is keyed by token; re-entering a settled market OVERWROTE the old
record and its pnl left 'realized' — the drift alarm caught it same-day
(WTI −9.97, Dota Game-1 +19.30 recovered from state history and restored
as archive keys). All three record-writers now archive settled records
to token#settled_ts keys first; aggregators iterate values() so archived
rows keep counting. NOTE: restoring truth WIDENS displayed drift — a
second, opposite-sign residual (~−11) remains under investigation
(morning forensics queued in HANDOFF); the alarm stays red on the paper
book intentionally. Live book unaffected (0 trades, drift 0).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A sweep or rapid clip entry splits ONE bet into several sub-floor fills:
gkmg's $612 MOUZ entry today read as 3×$204 same-second rows, every
clip below his $304 floor — the bot skipped a bet the backtest's
position-level view takes (he sold +60% five minutes later). Same-token
BUY rows within 120s now merge into the bet the sharp actually made
(weighted price, summed size, all component txs marked processed);
SELLs stay per-fill for the proportional mirror. reconcile_entries
applies the same clustering to its not-a-miss verdict so downtime
fill-splits count as real misses. Applies to BOTH books identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live path gates PER TRADE; reconcile_entries gated on POSITION size,
so 0xbadaf319's paired YES+NO arb clips ($6-37 each, all correctly
skipped below his $45 floor) accumulated into $40+ positions that read
as downtime misses while the bot was demonstrably UP. Now a miss needs
some SINGLE buy on the market to pass the same follow filter (market-
filtered /trades, full history); sub-floor accumulations log-and-skip
with a persisted verdict (state.no_miss_toks). Four phantom records
removed from the book; their hypothetical P&L never belonged in the
missed counterfactual.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New 'book' field in the fills ledger: {bb, ba, spread, bid5c, ask5c}
from CLOB /book at fill time. The calibration experiment's weakest
assumption is the flat SLIP haircut; weeks of these snapshots give an
empirical fill model for the backtest and a depth gate before real
money sizes up (today's probe: a stake vs $77 of bids within 5c).
Best-effort, never blocks a copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The execution engine was load-bearing from archive/ — the one thing in
there that wasn't retired. Now at repo root next to copybot.py; the two
archived scripts that imported it as a sibling get a parent-path shim.
fly.toml gains an http /health check so Fly restarts a dark machine
(self-heal); the notify half is the GH Actions watchdog (next commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Abort -> hard-reset to origin -> re-commit the running book. The
process is the book's single writer; remote state surgery must stop
the bot first (2026-07-08 BOOK RESET race: stale boot clone rebased
old state onto the reset and every publish died in UU until reboot).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every drained buy fill is now recorded: the tok-match keeps _record_lag
(real lag/slippage), leftovers get a bet record synthesized from the fill
(_record_untracked_buy). check_book() asserts the book invariant after
every trade and at boot: my_pos token => bet record + conds entry + cash
debited. Records/conds self-correct; the cash leg heals only at boot and
only when ledger_drift matches one un-vouched position's cost+fee exactly
— anything else stays a visible ⚠ drift. my_pos entries now carry cond
(engine) and fills-ledger BUY lines carry token, so the book can always
self-attribute. reconcile_exits' drain no longer discards leftover buys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The heartbeat/dashboard 'avg lag' was a lifetime accumulator that mixed the
retired 60s poll era into push mode, so a genuinely ~3s bot read as 48s.
lag_recent now keeps the last 24h of (ts, lag, slip); summary + feed report
the 24h average (n = lifetime copies, n24 = fills in window). Backfilled from
copybot_fills.jsonl on boot so it's populated immediately. Verified: 24h view
12s vs 48s lifetime — reflects real push execution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A position can land in my_pos without a bet record when its buy fill is
drained in a context where _record_lag doesn't fire (e.g. a second market on
the same event in one batch): cash-correct (open_count/deployed read my_pos)
but INVISIBLE in the feed table — the header showed '2 open / $75 deployed'
while only 1 bet rendered. write_feed now synthesizes a bet record from any
open my_pos position lacking one, so the count always equals what's shown.
Ledger was unaffected (drift -0.0) — display-only. my_pos entries now carry
'wallet' for proper name attribution on the synthesized record.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_handle_their_buy referenced their_prev in the fresh-OPEN branch but only
assigned it in the is_add branch — every new position crashed with
UnboundLocalError the moment the their-bet ceiling landed 2026-07-06. The
webhook/heartbeat try/except swallowed it as 'handler error', so the bot
logged FOLLOW then silently placed NOTHING (last real fill 07-05 23:17;
live Fly logs show FOLLOW ArbTrader ... -> heartbeat error: their_prev).
This is the primary cause of 'live bot missing bets'. Fix: hoist their_prev
above the if/else so both branches see the signal's prior position.
Also (the user's two asks):
- copybot on_wallet_activity: a QUALIFYING bet we were too slow to catch
(webhook missed / bot down / fast market resolved before we polled) is now
recorded as a missed bet 'too slow to follow (Nm late)' instead of silently
dropped; below-floor dust stays console-only. Filter runs before the stale
gate so we know if it would have qualified.
- sync_floors.py rewritten: PIN each paper-bot wallet's floor to the trusted
cache p80 (identical to the backtest's conv_thr), written to
copybot.paper.json, wired into daily.sh + committed. Kills the boot-time
data-api floor drift (fortuneking $1,498 boot vs $892 backtest) that made
the live bot filter out bets the backtest kept.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
copybot.py:
- 1.1 feed_path/fill_log are config-driven (live run won't clobber the
paper book's feed or fills ledger)
- 1.2 LIVE_PRIVATE_KEY / LIVE_FUNDER_ADDRESS / LIVE_SIGNATURE_TYPE env
overrides (Fly worker has no gitignored config.live.json)
- 1.4 chain_cash_gap(): live-only book-cash vs CLOB USDC balance, logged
as ⚠ CASH≠CHAIN (the real-money ledger_drift)
- 1.5 geo-gate is FATAL in live mode — unauth order probe; refuses to arm
from a 403-geo-blocked box before the confirm phrase (verified: refuses
from this US Mac)
- 1.6 mirror-exit retry: FAK sells that fill 0 on thin books retry each
backstop poll, ⚠ EXIT STUCK alert after 10 unfilled attempts
discord_daily.py: --ping retries 4x (5/10/15s) — launchd wakes the Mac to
run daily.sh and the heads-up ping was racing DNS/WiFi coming up after
sleep (getaddrinfo 'nodename nor servname'); the end-of-run digest always
landed, only the start ping lost the race.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The backtest reads positions (state); the bot reads trades (events). A trade
during downtime was silently baselined away at the next boot — not copied,
not missed, invisible — while the backtest still showed it (2026-07-07:
9 ArbTrader positions opened into the Fly trial-expiry crash-loop appeared
only in portfolio.json's current list). Now every boot + backstop poll
records any unseeded, unheld, unrecorded position that clears the follow
filter as a missed bet (reason 'bot offline'), settled hypothetically like
every other miss. Never copied — the entry is stale by definition. Dust
below risk.min_order_usd is ignored; API failure skips (silence is never
data). Bets that open AND close entirely within downtime stay the
backtest's domain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit 2026-07-06 found the live paper book stuck: 4 of 5 open positions were
50/50 refunds the bot could never settle, and the 5th was a position the
signal had exited a day earlier.
- resolution_price tier 2: CTF payoutNumerators/Denominator via RPC
(ALCHEMY_RPC_URL env or config alchemy_key). CLOB winner flags never
populate for operator-resolved in-play markets and never express 50/50
refunds; the chain records both. Token-id match now precedes outcome-label
match in tier 1. New bet status: 'refund' (settles at $0.50/share,
redeems in live mode).
- reconcile_exits(): at boot + every backstop poll, verify the signal still
holds each copied token (market-filtered data-api queries — no pagination
cap). Exit requires three AFFIRMATIVE facts (open-positions fetched+empty,
closed-positions fetched+contains token, market fetched+still trading) —
an API failure must never read as an exit.
- ledger_drift(): cash-vs-ledger invariant checked every heartbeat, warned
in the log, published in the feed (ledger_drift / adjustments fields).
Accumulators no longer round per fill (that drifted cents per trim);
the feed rounds at render. SELL fills now logged to copybot_fills.jsonl.
- their_positions seed cap 500 -> 2000 (whale books exceed 500).
Dry-run vs a copy of the real stuck state: 4 refunds settled at 0.50,
McCormick mirror-exited at 0.69, drift stable at the known +15.45 residue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Setting ALCHEMY_SIGNING_KEY on the worker switches start.sh to webhook
mode: instant copies on POST /alchemy (signature-verified), a 60s
housekeeping heartbeat (settle, feed, summary), and a full backstop poll
every 5 min so a dropped push costs minutes, not a silent miss. Webhook
mode now baselines at boot and publishes an initial feed like poll mode.
No signing key -> classic 60s poll, unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jaxperro