jaxperro
3d0bc7f001
live/: operationalizes the LBS/Yale "skilled ~3%" result against the live data-api. Enumerate recent liquid markets -> top traders -> candidate pool; cache every wallet's resolved bets once in DuckDB (~26k wallets / 12.5M bets, keyed by per-bet resolution time so any cutoff re-scores in seconds); 5-gate skill funnel (n>=15, z>0, BH-FDR, split-half OOS, MM/bot cap); dashboard + daily refresh. Key finding: copying the high-win-rate "favorite-rider" cohort looks +23.6% in-sample but loses -7.4% once selected on pre-June-1 data only (99% -> 68% win rate) — selection bias, reproducing the paper's "lucky winners revert" result on live data. Win rate != edge, again. wide/: bulk subgraph->DuckDB scanner (survivorship-bias-free over all wallets), but the public subgraph is frozen at Jan 2026 -> historical tool only. Large local data (*.duckdb, candidates.json, *_scored.json, history/) gitignored. README + FINDINGS updated with the current logic and the clean result. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
# wide/ — bulk subgraph edge scanner
|
||
|
||
Find wallets with a real edge across **all** of Polymarket (~1.76M traders,
|
||
268k conditions) by bulk-ingesting the on-chain subgraph into a local DuckDB
|
||
and ranking with SQL — instead of per-wallet API calls (which cap out at a few
|
||
hundred wallets and rate-limit).
|
||
|
||
## Why a local DB, not the API
|
||
|
||
The Goldsky orderbook subgraph **times out on any `orderBy`** over a
|
||
non-indexed field (`numTrades`, `scaledProfit`, …). You cannot ask it for "top
|
||
wallets." The only scalable pattern is: cursor-paginate every row by `id` (the
|
||
indexed key), land it locally, and rank in DuckDB. That constraint *is* the
|
||
architecture.
|
||
|
||
## Why the win rate here is honest
|
||
|
||
The data-api hides losers: Polymarket only redeems winning shares, so losing
|
||
positions sit unredeemed and never enter `/closed-positions`. Measuring win
|
||
rate there reads ~90% when the truth is ~50% (see ../FINDINGS.md). The subgraph
|
||
records a `marketPosition` for **every** buy regardless of redemption, so the
|
||
survivorship bias structurally does not exist in this data.
|
||
|
||
## What "edge" means
|
||
|
||
A high win rate is not edge — you can hit 90% by only buying 95¢ favorites.
|
||
Edge is **beating the prices you paid**:
|
||
|
||
```
|
||
p = valueBought / quantityBought (entry price, 0..1)
|
||
won = the outcome you held paid out
|
||
z = (wins − Σp) / sqrt(Σ p(1−p)) standard deviations above odds-implied
|
||
```
|
||
|
||
`z` high over enough bets, on a wallet that isn't a market-maker, is the real
|
||
signal (the same one ../insider.py computes, here over the entire market).
|
||
|
||
## Pipeline
|
||
|
||
| step | script | source | notes |
|
||
|------|--------|--------|-------|
|
||
| 1 | `ingest.py conditions` | subgraph | resolution + payoutNumerators → winning outcome |
|
||
| 2 | `gamma_tokens.py` | Gamma | token_id → (condition, outcome_index); subgraph's `outcomeIndex` is null |
|
||
| 3 | `ingest.py accounts` | subgraph | `numTrades` (market-maker filter), `creationTimestamp` (freshness) |
|
||
| 4 | `ingest.py market_positions` | subgraph | the heavy table: entry price + win/loss per bet |
|
||
| 5 | `score.py` | DuckDB | z, true win rate, profit + FDR + out-of-sample |
|
||
|
||
All ingests are **resumable** (per-table `id` cursor in `_cursor`), so a long
|
||
run can be stopped and restarted. Tables join in `edge.sql`.
|
||
|
||
## Guardrails (so a 1.76M-wallet scan doesn't just surface luck)
|
||
|
||
- **min-n + market-maker cap.** A 300k-trade grinder posts huge z with no
|
||
information. Require ≥30 resolved bets and cap `numTrades`.
|
||
- **Benjamini–Hochberg FDR.** Scan 100k wallets and thousands clear z>3 by
|
||
chance (look-elsewhere effect). `score.py` reports how many survive 5% FDR.
|
||
- **Out-of-sample.** `score.py --cutoff YYYY-MM-DD` selects wallets on bets
|
||
resolved before the date, then measures the *same* wallets forward. Edge that
|
||
is real persists; edge that is curve-fit reverts to z≈0 — which is what every
|
||
strategy in ../FINDINGS.md did. **Do not size up on in-sample z.**
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
pip install duckdb
|
||
python3 ingest.py conditions accounts # small tables
|
||
python3 gamma_tokens.py # token→outcome map
|
||
python3 ingest.py -p market_positions # heavy (~53M rows); parallel + resumable
|
||
python3 score.py --min-n 15 --max-trades 5000 --top 40
|
||
python3 score.py --cutoff 2026-04-30 --min-n 15 # in-sample vs forward
|
||
```
|