jaxperro
08f87c589e
- copybot: model Polymarket taker fees (V2, since 2026-03-30: shares*rate*p*(1-p), sports 0.03) on every paper/live fill; track fees_paid in state + feed; settle P&L nets the entry fee. publish_feed now commits state + fills ledger with the feed (autostash rebase) so the local poller is the sole state writer. - validate_timing: copy_pnl/held_pnl are fee-aware -> sharp selection now requires clearing real copy costs; conv stats + lead profile use resolved bets only. - conviction_scan/skill: exclude res_t>now rows (early-sold positions in unresolved markets scored at curPrice - a mark, not an outcome; was ~5% of the June test window at a 72% pseudo-win rate). - portfolio: skip unresolved rows (stake used to vanish from equity); missed bets of kind=open no longer KeyError - mark-to-market hypothetical P&L. - collect: cap stale refreshes at STALE_CAP=2500/run (bulk-aged pool turned the daily refresh into a ~40h pull holding the DuckDB lock). - Actions cron disabled: GitHub ran */5 every ~1.5-2.5h and the 10-min stale window skipped the rest -> 1 of ~104 qualifying buys copied. launchd --poll 60 is now the runner; workflow stays as manual backstop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
60 lines
2.5 KiB
Python
60 lines
2.5 KiB
Python
#!/usr/bin/env python3
|
|
"""Collect EVERY candidate wallet's resolved bets into the cache, up to present.
|
|
|
|
One-time (per refresh window) comprehensive pull so the whole candidate pool is
|
|
local. Resumable: cache.get_bets skips wallets pulled within MAX_AGE_DAYS, so
|
|
killing and re-running continues where it left off. Most-active wallets first,
|
|
so a partial cache already covers the wallets most likely to be skilled.
|
|
|
|
python3 collect.py
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import sys
|
|
import time
|
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
|
|
|
import cache
|
|
|
|
HERE = os.path.dirname(__file__)
|
|
WORKERS = 16
|
|
# Bound each run: every never-pulled wallet is collected, but at most STALE_CAP of
|
|
# the already-cached ones are refreshed (stalest first). Without the cap, the day
|
|
# the bulk-ingested pool crosses MAX_AGE_DAYS together, a "daily" run balloons into
|
|
# a ~40h re-pull that blocks every scoring step behind it in daily.sh and holds the
|
|
# DuckDB write lock (even read_only connections fail) all day. At 2,500/day the
|
|
# whole pool still turns over well inside the 14-day freshness window.
|
|
STALE_CAP = int(os.environ.get("STALE_CAP", 2500))
|
|
|
|
|
|
def main():
|
|
cands = json.load(open(os.path.join(HERE, "candidates.json")))
|
|
cands.sort(key=lambda c: c.get("markets_seen", 0), reverse=True)
|
|
ages = cache.pulled_ages()
|
|
fresh_cut = time.time() - cache.MAX_AGE_DAYS * 86400
|
|
new = [c["wallet"] for c in cands if c["wallet"] not in ages]
|
|
stale = sorted((c["wallet"] for c in cands
|
|
if 0 < ages.get(c["wallet"], 0) < fresh_cut), key=ages.get)
|
|
wallets = new + stale[:STALE_CAP]
|
|
print(f"collecting {len(wallets):,} wallets ({len(new):,} new + "
|
|
f"{len(stale[:STALE_CAP]):,} of {len(stale):,} stale, cap {STALE_CAP}) · "
|
|
f"{WORKERS} workers", flush=True)
|
|
done, t0 = 0, time.time()
|
|
with ThreadPoolExecutor(max_workers=WORKERS) as ex:
|
|
futs = [ex.submit(cache.get_bets, w) for w in wallets]
|
|
for _ in as_completed(futs):
|
|
done += 1
|
|
if done % 200 == 0:
|
|
w, b = cache.stats()
|
|
rate = done / max(1e-9, time.time() - t0)
|
|
eta = (len(wallets) - done) / max(1e-9, rate) / 3600
|
|
print(f" {done:,}/{len(wallets):,} · cache {w:,}w/{b:,}bets · "
|
|
f"{rate:.1f}/s · ETA {eta:.1f}h", flush=True)
|
|
w, b = cache.stats()
|
|
print(f"DONE {time.strftime('%F %T')} — cache: {w:,} wallets, {b:,} bets", flush=True)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|