cleanup: archive everything the live system no longer touches
- retired-infra/: Railway config, Mac launchd runner, GH-Actions cron (worker is Fly.io arn now; Actions cron last fired 2026-07-02) - live-research/: the June selection experiments (strategy/followability/ pnl_basket/pnl_focused/backtest_june/clean_test + outputs) - us-venue/: the scrapped Polymarket-US listability probe (+ its env-gated ONLY_CONDS replay filter stays in portfolio.py — generally useful for subset replays) - root sweeps hunt/huntwide/oos/copyback/watch.json -> archive/ - untracked logs/CSVs of dead experiments -> archive/local/ (gitignored) - READMEs updated: Fly migration, geoblock gotcha, new file map Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jaxperro
@@ -41,3 +41,4 @@ live/*_scored.json
|
||||
live/watch_prejune*.json
|
||||
live/history/
|
||||
live/slug_cache.json
|
||||
archive/local/
|
||||
|
||||
@@ -24,12 +24,13 @@ Three deployed pieces + one static dashboard:
|
||||
| piece | where it runs | what it does |
|
||||
|-------|--------------|--------------|
|
||||
| **daily pipeline** (`live/daily.sh`) | this Mac, launchd **08:00** (runs on wake if the Mac was asleep) | refresh the bet cache → 5-gate skill scan → fee-aware sharp selection → conviction floors → backtest book → publish JSON feeds to GitHub |
|
||||
| **copybot worker** (`copybot.py` via `host/start.sh`) | Railway, 24/7 | **push mode**: an Alchemy address-activity webhook (`copybot follow set`, Polygon) pings `POST /alchemy` the moment a followed wallet trades (~2–5s detection), signature-verified; a 60s heartbeat settles/publishes and a 5-min backstop poll catches dropped pushes. Paper-copies with real fees/lag/slippage, settles at CLOB resolution, commits its book back to the repo |
|
||||
| **copybot worker** (`copybot.py` via `host/start.sh`) | **Fly.io app `wwf-copybot`, region `arn` (Stockholm), 24/7** — migrated off Railway 2026-07-06: Railway ran it in a US region, which Polymarket's IP geoblock would 403 the moment orders got real; Stockholm is unrestricted AND ~25ms from the CLOB's eu-west-2 primaries. Every boot self-checks the geo-gate (`host/geocheck.py`) | **push mode**: an Alchemy address-activity webhook (`copybot follow set`, Polygon) pings `POST /alchemy` the moment a followed wallet trades (~2–5s detection), signature-verified; a 60s heartbeat settles/publishes and a 5-min backstop poll catches dropped pushes. Paper-copies with real fees/lag/slippage, settles at CLOB resolution, commits its book back to the repo |
|
||||
| **Discord digest** (`live/discord_daily.py`) | end of the daily pipeline | one message/day: the sharp list with profile links + 30-day conviction stats (per-trade pings retired 2026-07-04; the old Alchemy watcher lives in `archive/webhook_receiver.py`) |
|
||||
| **dashboard** | [jaxperro.com/trading](https://jaxperro.com/trading) (static, in the `jaxperro` repo) | renders the three JSON feeds: live bot book, backtest book, sharp table |
|
||||
|
||||
**The July 2026 live test:** a fresh $1,000 paper book (started 2026-07-02, on
|
||||
Railway) following the wallet set in **`live/copybot.paper.json`** — the single
|
||||
**The July 2026 live test:** a fresh $1,000 paper book (started 2026-07-02 on
|
||||
Railway; moved to Fly.io Stockholm 2026-07-06 with the book intact) following
|
||||
the wallet set in **`live/copybot.paper.json`** — the single
|
||||
source of truth (currently 5 volume + 3 whale wallets; the dashboard hero lists
|
||||
them live). Two stake classes: **volume** wallets (4% of equity/bet) are copied
|
||||
on their **conviction bets only** (auto p80 floor derived at boot); **whale**
|
||||
@@ -43,7 +44,7 @@ up, real money follows (see [`LIVE_TEST.md`](LIVE_TEST.md)).
|
||||
```
|
||||
data layer selection execution display
|
||||
────────── ───────── ───────── ───────
|
||||
live/cache.duckdb ──▶ trust.py (trusted-row filter) copybot.py (Railway) ─▶ jaxperro.com/trading
|
||||
live/cache.duckdb ──▶ trust.py (trusted-row filter) copybot.py (Fly arn) ─▶ jaxperro.com/trading
|
||||
(schema v2: skill.py (5-gate funnel) · class % of equity, · copybot_live.json
|
||||
35k wallets, conviction_scan.py (p80 + z_all) capped at the (live bot book)
|
||||
20M+ resolved bets, validate_timing.py (fee-aware signal's own bet · portfolio.json
|
||||
@@ -59,11 +60,11 @@ up, real money follows (see [`LIVE_TEST.md`](LIVE_TEST.md)).
|
||||
|
||||
| task | how |
|
||||
|------|-----|
|
||||
| **add / remove / reclass a LIVE wallet** | edit the `wallets` list in `live/copybot.paper.json` (`{"wallet","name","class":"volume"\|"whale","floor":123?}` — floor optional, auto p80 at boot; whales ignore floors) then run **`./live/deploy_bot.sh`** — it validates, previews, **syncs the Alchemy webhook's address list** (`live/sync_webhook.py`, Notify API), commits, pushes, redeploys Railway, and confirms the boot banner. Fully self-contained — nothing to click in any dashboard |
|
||||
| **add / remove / reclass a LIVE wallet** | edit the `wallets` list in `live/copybot.paper.json` (`{"wallet","name","class":"volume"\|"whale","floor":123?}` — floor optional, auto p80 at boot; whales ignore floors) then run **`./live/deploy_bot.sh`** — it validates, previews, **syncs the Alchemy webhook's address list** (`live/sync_webhook.py`, Notify API), commits, pushes, restarts the Fly machine, and confirms the boot banner. Fully self-contained — nothing to click in any dashboard |
|
||||
| **backtest any wallet set** | edit `live/backtest.json` (same entry shape) → `python3 live/portfolio.py`; ad-hoc without touching the dashboard: `python3 portfolio.py --wallets 0xabc,0xdef:whale --days 14 --out /tmp/t.json` |
|
||||
| **promote a wallet to live** | prove it in `backtest.json` first, copy the same entry into `copybot.paper.json`, run `deploy_bot.sh` |
|
||||
| **watch the live bot** | `railway logs --service copybot` (one summary line per 60s poll); the book is also committed as `live/copybot_live.json` and rendered on the dashboard |
|
||||
| **restart / redeploy the bot** | `railway redeploy --service copybot --yes` (config/code changes: the worker clones the repo fresh at boot). **Changes to `host/start.sh`, `railway.json`, or env vars need a full rebuild: `railway up --service copybot --detach`** — redeploy reuses the old image and env snapshot (this bit us: the image's launcher was frozen for days) |
|
||||
| **watch the live bot** | `flyctl logs --app wwf-copybot` (one summary line per 60s heartbeat); the book is also committed as `live/copybot_live.json` and rendered on the dashboard |
|
||||
| **restart / redeploy the bot** | `flyctl apps restart wwf-copybot` (config/code changes: the worker clones the repo fresh at boot, so a restart IS the deploy). **Changes to `host/` or `fly.toml` need an image rebuild: `flyctl deploy --remote-only`.** Secrets: `flyctl secrets set K=V` (applies with an automatic restart). Keep it ONE machine — Fly loves creating a second for "high availability", and two bots = two writers on one book (`flyctl scale count 1`) |
|
||||
| **run the daily pipeline manually** | `cd live && bash daily.sh` (launchd runs it 08:00; ~40 min, mostly collect). Never run two at once — the cache is single-writer |
|
||||
| **refresh just the sharp list** | `cd live && python3 conviction_scan.py && python3 validate_timing.py` |
|
||||
| **daily Discord digest** | sent by `live/discord_daily.py` at the end of `daily.sh`; webhook = `daily_webhook` in gitignored `config.json` |
|
||||
@@ -71,10 +72,11 @@ up, real money follows (see [`LIVE_TEST.md`](LIVE_TEST.md)).
|
||||
|
||||
Three moving parts talk to each other: this repo (research + bot + feeds), the
|
||||
`jaxperro` repo (static dashboard reading this repo's raw JSON feeds), and the
|
||||
Railway project `magnificent-kindness` (service `copybot`; the old `web`
|
||||
watcher service is retired). The bot **commits its own state/feed back to this
|
||||
repo** every few minutes — always `git pull --rebase --autostash` before you
|
||||
push (every script here already does).
|
||||
Fly.io app `wwf-copybot` (Stockholm; the old Railway project
|
||||
`magnificent-kindness` is stopped and can be deleted). The bot **commits its
|
||||
own state/feed back to this repo** every few minutes — always
|
||||
`git pull --rebase --autostash` before you push (every script here already
|
||||
does).
|
||||
|
||||
---
|
||||
|
||||
@@ -85,20 +87,20 @@ push (every script here already does).
|
||||
| `live/` | **the current system**: cache, scanners, sharp selection, backtest, daily pipeline ([live/README](live/README.md)) |
|
||||
| `live/trust.py` | the trusted-row filter every selector must read through (see gotchas 8–9) |
|
||||
| `live/backtest.json` · `live/copybot.paper.json` | the two wallet-set configs: backtest experiments vs. the live paper bot (same entry shape) |
|
||||
| `live/deploy_bot.sh` | one-command live-bot deploy: validate → preview → Alchemy address sync → commit → push → Railway redeploy → confirm boot |
|
||||
| `live/deploy_bot.sh` | one-command live-bot deploy: validate → preview → Alchemy address sync → commit → push → Fly restart → confirm boot |
|
||||
| `live/sync_webhook.py` | diffs the follow set against the push-mode Alchemy webhook and patches add/remove (token in gitignored `config.json`) |
|
||||
| `railway.json` · `nixpacks.toml` · `.railwayignore` | Railway build config: NIXPACKS builder + `bash host/start.sh` start command pinned as code; secrets/cache excluded from build uploads |
|
||||
| `fly.toml` · `fly.Dockerfile` | Fly.io app config (wwf-copybot, region `arn`, single machine, `:8080` webhook ingress) + the worker image (python + git; the bot code itself is cloned fresh at boot) |
|
||||
| `host/geocheck.py` | 3-probe geo-gate verdict (ipinfo → Polymarket's `/api/geoblock` → unauth CLOB order POST); runs at every boot, `GEOCHECK_ONLY=1` = probe-and-idle for testing a new host/region without a second book-writer |
|
||||
| `live/discord_daily.py` | the daily Discord digest (the only Discord output) |
|
||||
| `copybot.py` | the copy-trading bot: push/poll trigger → follow filter → execution engine (paper + live) |
|
||||
| `archive/copytrade.py` | the execution engine the bot reuses: sizing, risk gates, price guard, paper/live executors |
|
||||
| `host/start.sh` | 24/7 worker bootstrap for Railway/Fly/VPS (clones repo, resumes committed state) |
|
||||
| `host/start.sh` | 24/7 worker bootstrap for Fly/any VPS (geo-gate check, clones repo, resumes committed state) |
|
||||
| `LIVE_TEST.md` · `preflight_live.py` · `redeem.py` | real-money runbook, read-only credential preflight, on-chain redemption |
|
||||
| `insider.py` | the original detector: z-score, pre-resolution timing, fresh-wallet flags, funding-cluster rings |
|
||||
| `smart_money.py` | shared HTTP helper + survivorship-corrected win-rate dashboard (`:8899`) |
|
||||
| `archive/webhook_receiver.py` | retired 2026-07-04: Alchemy webhook → per-trade Discord pings (replaced by `live/discord_daily.py`'s daily digest) |
|
||||
| `wide/` | frozen-subgraph bulk scanner (1.76M wallets, historical only — subgraph froze Jan 2026) |
|
||||
| `archive/` | the six strategies that didn't work, kept honest ([archive/README](archive/README.md)) |
|
||||
| `hunt.py` · `huntwide.py` · `oos.py` · `copyback.py` | earlier research sweeps/backtests (superseded by `live/`) |
|
||||
| `archive/` | everything retired, kept honest ([archive/README](archive/README.md)): the six failed strategies, earlier research sweeps (`hunt/huntwide/oos/copyback`), the superseded live selection layer (`live-research/`), the scrapped Polymarket-US venue probe (`us-venue/`), and retired infra (`retired-infra/`: Railway config, Mac launchd runner, GH-Actions cron) |
|
||||
|
||||
---
|
||||
|
||||
@@ -166,7 +168,7 @@ in [`live/README.md`](live/README.md).
|
||||
| `live/copybot.paper.json` | **committed** (no secrets): the live paper bot's wallet set + classes + follow/risk params — deploy with `live/deploy_bot.sh` |
|
||||
| `config.json` | `daily_webhook` (Discord digest) · `alchemy_notify_token` + `alchemy_webhook_id` (webhook address sync) · `alchemy_signing_key` (local push-mode runs) · Alchemy RPC key · a legacy curated watchlist + floors (`sync_floors.py`) |
|
||||
| `config.live.json` | live-trading credentials (`private_key`, `funder_address`) + tiny test caps — see `LIVE_TEST.md` |
|
||||
| Railway `copybot` service | `GITHUB_TOKEN` (fine-grained PAT, contents-RW on this repo — the bot commits its state/feed back) · `ALCHEMY_SIGNING_KEY` (presence = push mode; delete it to fall back to 60s polling) · `PORT=8080` (the public domain routes here) |
|
||||
| Fly `wwf-copybot` secrets | `GITHUB_TOKEN` (fine-grained PAT, contents-RW on this repo — the bot commits its state/feed back) · `ALCHEMY_SIGNING_KEY` (presence = push mode; remove it to fall back to 60s polling) · `DISCORD_WEBHOOK`. The webhook ingress is `https://wwf-copybot.fly.dev/alchemy` (Alchemy webhook `wh_blf4qjjvfdbqs9mc`, address list auto-synced by `deploy_bot.sh`) |
|
||||
|
||||
---
|
||||
|
||||
@@ -215,7 +217,7 @@ backtest and bot share:
|
||||
Safety: paper is the default; live requires `mode:"live"` **and** `--live`
|
||||
**and** a typed confirmation phrase, under hard caps. The GH-Actions cron
|
||||
runner is retired (GitHub throttled `*/5` to ~2h in practice — it copied 1 of
|
||||
~104 qualifying trades in June; the always-on Railway poller replaced it).
|
||||
~104 qualifying trades in June; the always-on worker replaced it).
|
||||
|
||||
---
|
||||
|
||||
@@ -278,7 +280,17 @@ runner is retired (GitHub throttled `*/5` to ~2h in practice — it copied 1 of
|
||||
a transiently failed re-pull then hides the wallet's whole history from
|
||||
exact `pulled_at` checks (trust.py has a 14-day fallback for this).
|
||||
|
||||
10. **Hold-to-resolution P&L is a copy ceiling, not the wallet's bank
|
||||
10. **Polymarket geoblocks ORDER PLACEMENT by IP** (reads are open everywhere —
|
||||
which is why the paper bot never noticed Railway was in a US region).
|
||||
Restricted list has surprises: US, UK, France, Germany, Italy, Netherlands,
|
||||
Poland, Singapore, Australia, Ontario, **Brazil**
|
||||
(docs.polymarket.com/developers/CLOB/geoblock). `host/geocheck.py` gives a
|
||||
3-probe verdict from any box and runs at every worker boot; **never go live
|
||||
from a box that doesn't print `VERDICT: TRADABLE`** — and the machine's
|
||||
location doesn't relocate *you* (trade live from Colombia months, paper
|
||||
from US months).
|
||||
|
||||
11. **Hold-to-resolution P&L is a copy ceiling, not the wallet's bank
|
||||
statement.** The dashboard's Conv/All-Time P&L columns price every entry
|
||||
held to resolution at the wallet's own stakes — the right yardstick for a
|
||||
copier that holds, and the wrong one for judging the wallet itself.
|
||||
|
||||
@@ -19,3 +19,17 @@ run one you'd adjust the import path.
|
||||
| `xarb.py` | Cross-venue scanner: match the same event on Polymarket vs Kalshi, flag price gaps. | Venues priced efficiently (~1¢); both legs cost >$1 after fees. |
|
||||
|
||||
The keeper that came out of all this lives at the repo root: `insider.py`.
|
||||
|
||||
## Later additions (2026-07-06 cleanup)
|
||||
|
||||
| Path | What it was | Why it's here |
|
||||
|------|-------------|---------------|
|
||||
| `hunt.py` / `huntwide.py` / `oos.py` / `copyback.py` / `watch.json` | pre-`live/` research sweeps: candidate hunts, out-of-sample replays, copy-backtests | superseded by the `live/` selection layer (skill → conviction → validate_timing) |
|
||||
| `live-research/` | the June 2026 selection experiments: `strategy.py`, `followability.py`, `pnl_basket/focused.py`, `backtest_june.py`, `clean_test.sh` + their outputs | the research is settled (see FINDINGS); the surviving ideas live on in `live/` |
|
||||
| `us-venue/` | `us_listable.py` — mapped the replay stream against Polymarket US listings (95/794 matched; US settles off-chain via FCM/clearinghouse, so no wallet tracking) | the US move was scrapped 2026-07-06: no on-chain wallets, no copyable signal |
|
||||
| `retired-infra/` | Railway build config (`railway.json`, `nixpacks.toml`, `.railwayignore`, `runtime.txt`), the Mac launchd runner (`com.jaxperro.copybot.plist`, `run_copybot.sh`), the GH-Actions cron (`copybot.workflow.yml`) | the worker now runs on Fly.io `arn` (see root README); the Actions cron was throttled to uselessness; the Mac poller was replaced by the cloud worker |
|
||||
| `local/` | untracked logs/CSVs/state from the above experiments | kept out of git; safe to delete wholesale |
|
||||
|
||||
**Still-live code in this directory:** `copytrade.py` is imported by the root
|
||||
`copybot.py` as its execution engine (sizing, risk gates, executors) — archived
|
||||
as a *strategy* (raw copy-trading is −EV), kept as a *library*.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Which of the replayed bets' markets were ALSO listed on Polymarket US?
|
||||
|
||||
Polymarket US (gateway.polymarket.us, public/no-auth) settles off-chain — no
|
||||
wallets to copy — but it lists the same game markets, and it RETAINS closed
|
||||
historical listings (back to Oct 2025). So we can check, for every bet in the
|
||||
portfolio.py replay stream, whether a real US listing existed for that market:
|
||||
|
||||
tier 1 deterministic slug: US slug = "aec-"/"tec-" + intl slug
|
||||
(holds for esports/team sports; batched via the repeatable ?slug= param)
|
||||
tier 2 name+date+line match: tennis abbreviates differently per venue
|
||||
(US trimcc/pedsak vs intl mccormi/sakamo; itfme-/itfwo- vs itf-), so
|
||||
match full names from the intl question against a US window of the
|
||||
bet date's moneyline/totals/spreads listings.
|
||||
|
||||
Sub-market winners ("Set 1 Winner: …") name-match the WRONG (full-match) US
|
||||
market, so anything with a set/map/half qualifier is tier-1 only.
|
||||
|
||||
Output -> us_listable.json {cond: {listed, us_slug, how, title}}. Replay just
|
||||
the listable subset with:
|
||||
ONLY_CONDS=us_listable.json PORTFOLIO_OUT=/tmp/pf_us.json python3 portfolio.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import ssl
|
||||
import time
|
||||
import unicodedata
|
||||
import urllib.request
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import portfolio as pf
|
||||
|
||||
_SSL = ssl._create_unverified_context()
|
||||
GATEWAY = "https://gateway.polymarket.us/v1/markets"
|
||||
HERE = os.path.dirname(__file__)
|
||||
OUT = os.path.join(HERE, "us_listable.json")
|
||||
|
||||
|
||||
def get(url):
|
||||
for attempt in range(3):
|
||||
try:
|
||||
r = urllib.request.urlopen(urllib.request.Request(
|
||||
url, headers={"User-Agent": "Mozilla/5.0"}), timeout=30, context=_SSL)
|
||||
return json.loads(r.read())
|
||||
except Exception:
|
||||
time.sleep(1 + attempt)
|
||||
return {}
|
||||
|
||||
|
||||
def norm(s):
|
||||
s = unicodedata.normalize("NFD", s or "").encode("ascii", "ignore").decode().lower()
|
||||
return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", " ", s)).strip()
|
||||
|
||||
|
||||
def name_in(name, hay):
|
||||
ws = [w for w in name.split() if len(w) > 1]
|
||||
return bool(ws) and all(w in hay for w in ws)
|
||||
|
||||
|
||||
# sub-market qualifiers: a "Set 1 Winner"/"First Map" market would name-match
|
||||
# the full-match US moneyline — a false positive — so these skip tier 2
|
||||
QUAL = re.compile(r"\b(set \d|1st|first|2nd|second|3rd|third|map \d|game \d|handicap|"
|
||||
r"to score|half|quarter|period|race to|双|correct score|doubles|"
|
||||
r"corners|cards|btts|total sets)\b", re.I)
|
||||
|
||||
|
||||
def parse_title(t):
|
||||
"""intl question -> {a, b, kind, line} or None (None = tier-1 only)."""
|
||||
s = str(t or "")
|
||||
ou = re.search(r"O/U\s+(\d+(?:\.\d+)?)", s, re.I)
|
||||
hc = re.search(r"\([+-](\d+(?:\.\d+)?)\)", s)
|
||||
kind = "totals" if ou else "spreads" if hc else "moneyline"
|
||||
line = ou.group(1) if ou else (hc.group(1) if hc else None)
|
||||
if QUAL.search(s):
|
||||
return None
|
||||
s = re.sub(r"\(BO\d+\)", " ", s, flags=re.I)
|
||||
s = re.sub(r"\([+-]?\d+(?:\.\d+)?\)", " ", s)
|
||||
parts = re.split(r"\svs\.?\s", s, flags=re.I)
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
a = parts[0].split(":", 1)[1] if ":" in parts[0] else parts[0]
|
||||
# esports format "Counter-Strike: A vs B (BO3) - IEM Cologne Playoffs":
|
||||
# drop the " - Tournament" tail and any ": qualifier" tail from side b
|
||||
b = parts[1].split(":")[0].split(" - ")[0]
|
||||
a, b = norm(a), norm(b)
|
||||
if not a or not b:
|
||||
return None
|
||||
return {"a": a, "b": b, "kind": kind, "line": line}
|
||||
|
||||
|
||||
_WINDOWS = {} # (kind, date) -> [{slug, hay}]
|
||||
|
||||
|
||||
def us_window(kind, date):
|
||||
"""All US <kind> markets listed around <date> (startDate ≈ listing time,
|
||||
day-of/day-before for game markets), closed ones included."""
|
||||
key = (kind, date)
|
||||
if key in _WINDOWS:
|
||||
return _WINDOWS[key]
|
||||
t = time.mktime(time.strptime(date, "%Y-%m-%d"))
|
||||
iso = lambda x: time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(x))
|
||||
lo, hi = iso(t - 2 * 86400), iso(t + 2 * 86400)
|
||||
out = []
|
||||
# Wimbledon-week windows run past 2k moneylines — never truncate silently
|
||||
for off in range(0, 20000, 100):
|
||||
d = get(f"{GATEWAY}?marketTypes={kind}&startDateMin={lo}&startDateMax={hi}"
|
||||
f"&limit=100&offset={off}")
|
||||
ms = d.get("markets", [])
|
||||
for m in ms:
|
||||
out.append({"slug": m["slug"],
|
||||
"hay": norm(m.get("question", "")) + " " + norm(m["slug"])})
|
||||
if len(ms) < 100:
|
||||
break
|
||||
_WINDOWS[key] = out
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
# the exact bet stream portfolio.py replays (same dedupe: first entry per market)
|
||||
resolved = pf.window_bets()
|
||||
opened = pf.open_bets()
|
||||
by = {}
|
||||
for b in resolved + opened:
|
||||
if b.get("cond") and (b["cond"] not in by or b["entry_t"] < by[b["cond"]]["entry_t"]):
|
||||
by[b["cond"]] = b
|
||||
conds = list(by)
|
||||
print(f"[us] {len(conds)} unique markets in the {pf.DAYS}d replay stream", flush=True)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=8) as ex:
|
||||
list(ex.map(pf.market_meta, conds))
|
||||
meta = {c: pf.market_meta(c) for c in conds}
|
||||
|
||||
result = {}
|
||||
|
||||
# ---- tier 1: deterministic prefixed slugs, batched --------------------
|
||||
cand = {} # us_slug -> cond
|
||||
for c in conds:
|
||||
slug = meta[c]["slug"]
|
||||
if slug:
|
||||
cand["aec-" + slug] = c
|
||||
cand["tec-" + slug] = c
|
||||
slugs = list(cand)
|
||||
for i in range(0, len(slugs), 25):
|
||||
qs = "&".join("slug=" + s for s in slugs[i:i + 25])
|
||||
for m in get(f"{GATEWAY}?{qs}&limit=100").get("markets", []):
|
||||
c = cand.get(m["slug"])
|
||||
if c and c not in result:
|
||||
result[c] = {"listed": True, "us_slug": m["slug"], "how": "slug",
|
||||
"title": meta[c]["title"]}
|
||||
print(f"[us] tier 1 (direct slug): {len(result)} listed", flush=True)
|
||||
|
||||
# ---- tier 2: name + date + line ---------------------------------------
|
||||
t2 = 0
|
||||
for c in conds:
|
||||
if c in result:
|
||||
continue
|
||||
title, slug = meta[c]["title"], meta[c]["slug"]
|
||||
p = parse_title(title)
|
||||
dm = re.search(r"(\d{4}-\d{2}-\d{2})", slug or "")
|
||||
if not p or not dm:
|
||||
result[c] = {"listed": False, "how": "no-parse", "title": title}
|
||||
continue
|
||||
hit = None
|
||||
for m in us_window(p["kind"], dm.group(1)):
|
||||
if not (name_in(p["a"], m["hay"]) and name_in(p["b"], m["hay"])):
|
||||
continue
|
||||
if p["line"] and norm(p["line"]) not in m["hay"]:
|
||||
continue
|
||||
hit = m
|
||||
break
|
||||
if hit:
|
||||
t2 += 1
|
||||
result[c] = {"listed": True, "us_slug": hit["slug"], "how": "name",
|
||||
"title": title}
|
||||
else:
|
||||
result[c] = {"listed": False, "how": "unmatched", "title": title}
|
||||
print(f"[us] tier 2 (name+date): {t2} more listed "
|
||||
f"({len(_WINDOWS)} US windows fetched)", flush=True)
|
||||
|
||||
json.dump(result, open(OUT, "w"), indent=1)
|
||||
n_listed = sum(1 for v in result.values() if v["listed"])
|
||||
print(f"[us] TOTAL: {n_listed}/{len(conds)} replayed markets had a US listing "
|
||||
f"-> {os.path.basename(OUT)}", flush=True)
|
||||
|
||||
# breakdown by intl slug prefix so we can see WHERE coverage holds
|
||||
pre = lambda s: (s or "?").split("-")[0]
|
||||
for label, keep in [("listed", True), ("unlisted", False)]:
|
||||
ctr = Counter(pre(meta[c]["slug"]) for c, v in result.items() if v["listed"] == keep)
|
||||
print(f"[us] {label:9s} " + " ".join(f"{k}:{n}" for k, n in ctr.most_common(14)),
|
||||
flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+3
-3
@@ -40,8 +40,8 @@ underdog/longshot prices — the copyable alpha), `balanced`, or `favorite`
|
||||
| sharps | `conviction_scan.py` + `validate_timing.py` | conviction-profile scan → copy-positive-holder selection → `watch_sharps.json` (see "The repeatable find") |
|
||||
| portfolio | `portfolio.py` | $1k paper book off the cache → `portfolio.json` (see "Paper portfolio") |
|
||||
| dashboard | `dashboard.py` | self-contained `dashboard.html` — sortable, archetype-tagged, live recent-trade lookup |
|
||||
| backtest | `backtest_june.py [arch]` | copy an archetype's June-1+ entries, $1000, no lag → P&L |
|
||||
| clean test | `clean_test.sh` | **the honest test**: re-select on pre-June-1 data only, then backtest June-1+ forward |
|
||||
| backtest | `../archive/live-research/backtest_june.py [arch]` | copy an archetype's June-1+ entries, $1000, no lag → P&L (archived 2026-07-06 with the rest of the June research layer) |
|
||||
| clean test | `../archive/live-research/clean_test.sh` | **the honest test**: re-select on pre-June-1 data only, then backtest June-1+ forward |
|
||||
|
||||
## The cache is the point
|
||||
|
||||
@@ -84,7 +84,7 @@ turn into losers out-of-sample. **Don't copy favorite-riders.** The `value`
|
||||
archetype (beats underdog prices) is where real alpha may live — test it with
|
||||
`backtest_june.py value`.
|
||||
|
||||
## Strategy backtests
|
||||
## Strategy backtests (June research layer — archived to `../archive/live-research/` 2026-07-06)
|
||||
|
||||
- `strategy.py` — train (pre-May-30) / test (June1+) wallet selection on copy-ROI
|
||||
+ z + monthly consistency + diversification. → `selection.json`.
|
||||
|
||||
@@ -251,6 +251,19 @@ def main():
|
||||
b["kind"] = "open"; by_mkt[b["cond"]] = b
|
||||
stream = sorted(by_mkt.values(), key=lambda b: b["entry_t"])
|
||||
|
||||
# ONLY_CONDS=<json path>: replay only these markets — {cond: bool} or
|
||||
# us_listable.py's {cond: {"listed": bool, ...}}. Models "same signal, but
|
||||
# I can only execute the subset" (e.g. bets also listed on Polymarket US);
|
||||
# thresholds/sizing still come from the full signal, capital only chases
|
||||
# the executable bets.
|
||||
_only = os.environ.get("ONLY_CONDS")
|
||||
if _only:
|
||||
_allow = {c for c, v in json.load(open(_only)).items()
|
||||
if (v.get("listed") if isinstance(v, dict) else v)}
|
||||
_pre = len(stream)
|
||||
stream = [b for b in stream if b["cond"] in _allow]
|
||||
print(f"portfolio: ONLY_CONDS filter kept {len(stream)}/{_pre} bets", flush=True)
|
||||
|
||||
# prefetch every replayed market's slug (threaded; disk-cached) so the
|
||||
# event-correlation cap can group markets by real-world event
|
||||
with ThreadPoolExecutor(max_workers=8) as ex:
|
||||
|
||||
Reference in New Issue
Block a user