V2 support
This commit is contained in:
Daniel Sapkota
@@ -1,11 +1,8 @@
|
||||
# poly-maker (v2)
|
||||
# poly-maker
|
||||
|
||||
A maker-only market-making bot for **Polymarket CLOB V2**, focused on political
|
||||
markets. Single async process, local-file config (no Google Sheets), typed and
|
||||
tested. This is a ground-up rewrite of v1 — see [`docs/scoping/`](docs/scoping/00-overview.md)
|
||||
for the design and [`docs/scoping/06-migration-plan.md`](docs/scoping/06-migration-plan.md)
|
||||
for the plan. The original v1 code has been removed; see the git history (branch
|
||||
`main`) if you need to reference it.
|
||||
tested.
|
||||
|
||||
> [!WARNING]
|
||||
> Market making on Polymarket is competitive and can lose money. This is a
|
||||
@@ -14,13 +11,13 @@ for the plan. The original v1 code has been removed; see the git history (branch
|
||||
|
||||
## What it does
|
||||
|
||||
- Discovers political markets via the **Gamma API** (seconds, not the v1 hour-long
|
||||
crawl) and ranks them by reward + rebate income vs. volatility/spread risk.
|
||||
- Discovers political markets via the **Gamma API** (seconds) and ranks them by
|
||||
reward + rebate income vs. volatility/spread risk.
|
||||
- Maintains a live order book per token from the **market WebSocket**.
|
||||
- Quotes **maker-only** (every order is post-only): a fair-value + inventory-skew
|
||||
strategy that posts BUY-YES and BUY-NO as the canonical two-sided quote, with
|
||||
live volatility/toxicity estimation and a regime machine that pulls quotes
|
||||
during news events. See [`docs/scoping/04-strategy.md`](docs/scoping/04-strategy.md).
|
||||
- Quotes **maker-only** — every order is post-only. Fair-value + inventory-skew
|
||||
strategy that posts BUY-YES and BUY-NO as a two-sided quote, with live
|
||||
volatility/toxicity estimation and a regime machine that pulls quotes during
|
||||
news events (see [Strategy](#strategy)).
|
||||
- Reconciles a target quote set against live orders with churn tolerances; runs
|
||||
the exchange **heartbeat** dead-man switch; enforces risk caps and a daily-loss
|
||||
kill switch.
|
||||
@@ -39,13 +36,13 @@ uv run polymaker --help
|
||||
## Configure
|
||||
|
||||
```bash
|
||||
cp .env.example .env # then edit: PK + BROWSER_ADDRESS (same wallet as the UI)
|
||||
cp .env.example .env # then edit: PK + BROWSER_ADDRESS
|
||||
```
|
||||
|
||||
### Which wallet address?
|
||||
|
||||
Polymarket's current **deposit-wallet** architecture (rolled out ~June 2026)
|
||||
shows several addresses in the UI, and the labels are inconsistent. What matters:
|
||||
Polymarket's current **deposit-wallet** architecture shows several addresses in
|
||||
the UI, and the labels are inconsistent. What matters:
|
||||
|
||||
- **`BROWSER_ADDRESS` = the funder** — the *smart-contract wallet that actually
|
||||
holds your pUSD and positions*. Depending on the account it may be shown as the
|
||||
@@ -61,8 +58,7 @@ Set `signature_type` in `config/config.toml` to match how the account was made:
|
||||
`3` = POLY_1271 deposit wallet (current default), `2` = older browser-wallet
|
||||
Gnosis Safe, `1` = email/magic proxy, `0` = plain EOA. A wrong type fails loudly
|
||||
(`polymaker doctor` / `livetest` report it). Approvals must have been granted
|
||||
from the funding wallet — trading once in the UI does this. See
|
||||
[`docs/scoping/03-api-layer.md`](docs/scoping/03-api-layer.md) §1/§9.
|
||||
from the funding wallet — trading once in the UI does this.
|
||||
|
||||
Everything else is TOML under [`config/`](config/):
|
||||
|
||||
@@ -105,10 +101,41 @@ user WS ─▶ StateStore RiskManager
|
||||
Gamma ─▶ Catalog/scanner ─▶ SQLite periodic REST reconcile ┘
|
||||
```
|
||||
|
||||
The strategy layer is a pure function `(book, inventory, params, clock) →
|
||||
TargetQuotes` — deterministic and unit-tested. The engine owns all I/O and
|
||||
state around it. Full component map in
|
||||
[`docs/scoping/02-architecture.md`](docs/scoping/02-architecture.md).
|
||||
One async event loop. The strategy layer is a pure function `(book, inventory,
|
||||
params, clock) → TargetQuotes` — deterministic and unit-tested. The engine owns
|
||||
all I/O and state around it; the `ExecutionGateway` wraps `py-clob-client-v2`
|
||||
(which handles the V2 EIP-712 signing) and offloads its blocking calls to a
|
||||
thread pool so the hot path never stalls. State (positions, orders, PnL, catalog)
|
||||
lives in one SQLite file; raw WS/order events are journaled to `journal/` for
|
||||
replay.
|
||||
|
||||
## Strategy
|
||||
|
||||
Maker-only, quoting both sides of each market as USDC-collateralized bids:
|
||||
|
||||
- **Fair value** — depth-weighted microprice off the live book, nudged by an
|
||||
EWMA of signed trade flow.
|
||||
- **Quote construction** — reservation price `r = FV − skew(inventory)`;
|
||||
half-spread `δ = base + c_vol·σ + c_tox·toxicity`. Post **BUY-YES at `r − δ`**
|
||||
and **BUY-NO at `(1 − r) − δ`**. Because both legs are bids that sum below 1,
|
||||
a filled pair merges back to USDC at locked edge `1 − p − q` — a maker-only
|
||||
exit that never crosses the spread.
|
||||
- **Inventory skew** — net position leans both quotes: long YES → bid YES lower,
|
||||
bid NO higher (acquire the offsetting leg). Size tapers as inventory approaches
|
||||
a soft cap, then the adding side is pulled entirely.
|
||||
- **Volatility / toxicity** — realized-vol and per-fill markout (adverse
|
||||
selection) EWMAs widen the spread and shrink size in markets that pick us off.
|
||||
- **Regime machine** — per market: `QUIET` (farm rewards in-band), `TRENDING`
|
||||
(lean + widen + half size), `EVENT` (sweep/jump detected → pull quotes, cool
|
||||
off), `REDUCE_ONLY` (inventory cap / near end date → exits only), `HALTED`
|
||||
(stale data / resolved / kill switch → cancel all).
|
||||
- **Rewards + rebates** — quotes stay inside the liquidity-rewards band in QUIET;
|
||||
the market selector also scores the new maker-rebate program (a share of taker
|
||||
fees rebated to makers).
|
||||
- **Risk** — per-market notional cap, neg-risk event-group worst-case cap, total
|
||||
exposure cap, daily-loss kill switch, WS-staleness halt.
|
||||
|
||||
Tune it all via profiles in `config/strategy.toml`.
|
||||
|
||||
## Develop
|
||||
|
||||
@@ -119,20 +146,18 @@ uv run ruff check src tests # lint
|
||||
uv run mypy src # types (strict)
|
||||
```
|
||||
|
||||
## Status / roadmap
|
||||
## Status
|
||||
|
||||
Implemented: config, catalog/scanner, order book + analytics, strategy (FV,
|
||||
Implemented and live-verified end to end (auth → book → strategy → sign → post →
|
||||
cancel): config, catalog/scanner, order book + analytics, strategy (FV,
|
||||
vol/toxicity, regime, quoting), state store + lifecycle, execution gateway +
|
||||
reconciler + heartbeat, market/user websockets, risk manager, merger (EOA path),
|
||||
engine, CLI, paper mode, journal capture. 83 tests; the market-data path is
|
||||
verified live.
|
||||
engine, CLI, paper mode, journal capture. 83 tests; ruff + mypy strict clean.
|
||||
|
||||
Spike-gated (need the live wallet — see [`docs/scoping/03-api-layer.md`](docs/scoping/03-api-layer.md) §9):
|
||||
V2 order signing with a Safe/proxy wallet (`livetest` is the probe), pUSD
|
||||
wrap/allowance mechanics, exact user-WS field names, native Safe merge execution.
|
||||
|
||||
Deferred (scoped, not built): replay backtester over captured journals, external
|
||||
data feeds (polls/news/cross-venue — [`docs/scoping/07-future-external-data.md`](docs/scoping/07-future-external-data.md)).
|
||||
Not yet built: a replay backtester over the captured journals, and external data
|
||||
feeds (polls / news / cross-venue). Merging through a Safe/proxy wallet routes a
|
||||
tx via the relayer and isn't wired yet — until then inventory exits via limit
|
||||
sells rather than merging.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
# polymaker global config. Secrets live in .env, never here.
|
||||
# See docs/scoping/05-config-and-ops.md.
|
||||
# See the README.
|
||||
|
||||
[wallet]
|
||||
# chain: Polygon mainnet
|
||||
@@ -12,7 +12,7 @@ chain_id = 137
|
||||
# If your UI shows a "deposit address", you almost certainly have a deposit
|
||||
# wallet -> use 3. In every proxy case (1/2/3), BROWSER_ADDRESS is the DEPOSIT
|
||||
# address (the funder that holds your pUSD), NOT your signing EOA and NOT the
|
||||
# deployer/factory. Known SDK auth bugs live here (docs 03 §9) — `polymaker
|
||||
# deployer/factory. Known SDK auth bugs live here (see the README) — `polymaker
|
||||
# doctor` / `livetest` will tell you if you picked the wrong type.
|
||||
signature_type = 3
|
||||
clob_host = "https://clob.polymarket.com"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Named strategy parameter profiles (replaces the v1 Hyperparameters sheet).
|
||||
# markets.toml maps each market to one of these profiles.
|
||||
# See docs/scoping/04-strategy.md §8.
|
||||
# See the README.
|
||||
|
||||
[profiles.political-longdated]
|
||||
# --- fair value ---
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# Poly-Maker v2 — Scoping Overview
|
||||
|
||||
**Date: 2026-07-05.** Scope for a ground-up v2 of this Polymarket market-making
|
||||
bot, targeting political markets.
|
||||
|
||||
## Why now (the forcing function)
|
||||
|
||||
Polymarket cut over to **CLOB V2 on April 28, 2026** — new order signing, new
|
||||
exchange contracts, pUSD collateral — and archived the `py-clob-client`
|
||||
library this repo is built on. **v1 of this bot can no longer place an
|
||||
order.** We are not refactoring a working system; we're rebuilding a dead one,
|
||||
which frees us to fix everything at once.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **CLOB V2 migration** — new client (`py-clob-client-v2`), V2 order signing,
|
||||
pUSD collateral handling, post-only orders, heartbeat dead-man switch,
|
||||
batch endpoints. → [03-api-layer.md](03-api-layer.md)
|
||||
2. **Kill Google Sheets** — config, market selection, and dashboards become
|
||||
local TOML files + SQLite + a `polymaker` CLI. A laptop with the repo and a
|
||||
funded wallet is a complete deployment. → [05-config-and-ops.md](05-config-and-ops.md)
|
||||
3. **Fast market discovery** — Gamma API keyset queries with server-side
|
||||
politics/liquidity/reward filters replace the hour-long
|
||||
crawl-every-orderbook scanner: seconds, one process. → [03](03-api-layer.md) §4, [05](05-config-and-ops.md) §2
|
||||
4. **Much better strategy** — fair-value + inventory-skew quoting, live
|
||||
volatility/toxicity estimation, regime detection (knowing when *not* to
|
||||
quote), rewards **and** the new maker-rebates program as explicit objective
|
||||
terms, and a replay backtester + paper mode to prove it before capital.
|
||||
→ [04-strategy.md](04-strategy.md)
|
||||
5. **Maker-only, always** — every order is post-only; exits are repriced
|
||||
limits, the YES+NO merge loop, and size discipline. No taker orders,
|
||||
period. → [04](04-strategy.md) §3, §6
|
||||
6. **Faster & simpler** — one async process (no threads, no Node.js
|
||||
subprocess, no gc.collect() folklore), uvloop, event-to-order latency
|
||||
target <50ms internal, Python 3.12+, uv end-to-end, typed and tested.
|
||||
→ [02-architecture.md](02-architecture.md)
|
||||
7. **Political markets first, external data later** — v2 trades on
|
||||
microstructure alone; the architecture reserves an explicit seat (SignalBus)
|
||||
for polls/news/cross-venue feeds in a later phase.
|
||||
→ [07-future-external-data.md](07-future-external-data.md)
|
||||
|
||||
## Reading order
|
||||
|
||||
| Doc | Contents |
|
||||
|---|---|
|
||||
| [01-current-state.md](01-current-state.md) | Audit of v1: what exists, what's broken, what dies, what's worth keeping |
|
||||
| [02-architecture.md](02-architecture.md) | v2 design: components, state machine, package layout, dependencies |
|
||||
| [03-api-layer.md](03-api-layer.md) | CLOB V2 migration facts, client choice, endpoints, websockets, rate limits, on-chain/merge |
|
||||
| [04-strategy.md](04-strategy.md) | The quoting engine: FV, vol/toxicity, regimes, rewards+rebates utility, risk, simulation |
|
||||
| [05-config-and-ops.md](05-config-and-ops.md) | Local config schemas, scanner/CLI, observability, deployment |
|
||||
| [06-migration-plan.md](06-migration-plan.md) | Phases 0–5, exit criteria, deletion ledger, risk register |
|
||||
| [07-future-external-data.md](07-future-external-data.md) | Deferred feed layer, scoped so it's additive later |
|
||||
|
||||
## Headline decisions (details in the docs)
|
||||
|
||||
- **Build in place, delete v1 as we go** — v1 can't trade anyway; no parallel
|
||||
run. (The v1 code was parked under `legacy/` during the port and has since
|
||||
been deleted — it lives on in git history on `main`.)
|
||||
- **Two de-risking spikes before building** (Phase 2): wallet auth on V2 with
|
||||
our proxy-wallet setup (known SDK bugs around signature types), and live WS
|
||||
payload capture (schemas changed post-2025 and docs are young).
|
||||
- **BUY-YES + BUY-NO is the canonical two-sided quote** — USDC(pUSD)-
|
||||
collateralized on both sides, pairs merge back to cash with locked edge,
|
||||
and it satisfies the rewards program's two-sidedness rules.
|
||||
- **Paper-trading gate before live capital**: ≥2 weeks, ≥10 markets, positive
|
||||
PnL under a conservative fill model, markouts > −0.5 tick at +5m.
|
||||
|
||||
## Open questions (tracked in 03 §9)
|
||||
|
||||
WS ping/pong contract; `py-sdk` (beta unified SDK) maturity; pUSD
|
||||
wrap/allowance/merge-collateral mechanics; current `price_change` payload
|
||||
schema; rebate payout currency; signature-type-2 health in the v2 SDK.
|
||||
@@ -1,154 +0,0 @@
|
||||
# 01 — Current State Audit (v1)
|
||||
|
||||
Honest inventory of what exists today, what's broken, and what dies in v2.
|
||||
This is the baseline the rest of the scoping docs build on.
|
||||
|
||||
## What v1 is
|
||||
|
||||
A single-process Python market maker (~2,700 LOC) plus a separate market-scanner
|
||||
script, glued together through a **Google Sheet** that acts as config store,
|
||||
market database, and dashboard at the same time.
|
||||
|
||||
```
|
||||
main.py entry point: REST polling thread + 2 websockets
|
||||
trading.py the entire strategy (perform_trade, 470 lines)
|
||||
update_markets.py scanner: crawls all markets, writes to Google Sheets hourly
|
||||
update_stats.py writes account stats/earnings back to Google Sheets
|
||||
poly_data/ client wrapper, global mutable state, WS handlers, book
|
||||
poly_utils/ Google Sheets access (incl. a read-only CSV-export hack)
|
||||
data_updater/ scanner internals (reward math, volatility fetch)
|
||||
poly_stats/ earnings/positions summary -> Sheets
|
||||
poly_merger/ Node.js subprocess to merge YES+NO via Gnosis Safe
|
||||
positions/*.json ad-hoc risk-off state files written next to the code
|
||||
```
|
||||
|
||||
### Data flow today
|
||||
|
||||
1. `update_markets.py` (separate process, ideally separate IP) crawls
|
||||
`clob.polymarket.com/sampling-markets` page by page, then fetches **one order
|
||||
book per market** via REST plus one `prices-history` call per market to
|
||||
compute volatility — thousands of sequential REST calls per scan, ~hourly.
|
||||
Results are pasted into the "All Markets" / "Volatility Markets" sheets.
|
||||
2. A human picks rows into the "Selected Markets" sheet and tunes the
|
||||
"Hyperparameters" sheet.
|
||||
3. `main.py` re-reads the sheet **every 30 seconds** (network call to Google in
|
||||
the hot path), polls positions/orders REST every 5s in a background *thread*,
|
||||
and maintains order books from the market websocket in the asyncio loop.
|
||||
4. Every `book`/`price_change`/user event spawns `perform_trade(market)` as a
|
||||
new asyncio task, serialized per market by an `asyncio.Lock`.
|
||||
|
||||
## What's structurally wrong
|
||||
|
||||
### Concurrency model
|
||||
- **threading + asyncio mixed**: a daemon thread mutates `global_state` (dicts of
|
||||
positions/orders) while asyncio tasks read them. No coherent locking
|
||||
(`global_state.lock` exists but is essentially unused).
|
||||
- **Blocking I/O inside the event loop**: `py-clob-client` and `requests` are
|
||||
synchronous. Every order placement/cancel blocks the loop — while it's
|
||||
signing + POSTing, the bot is blind to every other market's book updates.
|
||||
- **`perform_trade` holds the per-market lock through a `gc.collect()` and a
|
||||
hard-coded `await asyncio.sleep(2)`** (trading.py:470). Every quote decision
|
||||
on a market therefore takes ≥2s and events pile up behind the lock. This
|
||||
alone caps reactivity at ~0.5 Hz per market in a game where queue position
|
||||
and reaction to sweeps decide profitability.
|
||||
- Event storm handling is "spawn a task per WS message" — no coalescing, no
|
||||
debounce; the lock queue grows unboundedly under load.
|
||||
|
||||
### State management
|
||||
- One module of **global mutable dicts** (`global_state.py`) shared across a
|
||||
thread and dozens of tasks. Position tracking is a merge of three sources
|
||||
(data-api REST, user WS fills, on-chain balance) reconciled with the
|
||||
`performing` set + 15-second staleness hack — a hand-rolled, race-prone
|
||||
answer to "did my fill land yet?".
|
||||
- `set_order()` in `poly_data/data_utils.py:136` **overwrites both sides** of a
|
||||
token's order record with a dict containing only the side that just updated —
|
||||
the other side's `{price, size}` silently vanishes until the next 5s REST poll.
|
||||
- Local order book is keyed by condition_id, stores only the YES token book,
|
||||
derives the NO view by `1 - price`. Fine in principle, but book resync after
|
||||
WS reconnect relies on the server re-sending a `book` snapshot; there's no
|
||||
sequence-gap detection or hash check (the WS actually provides a book `hash`).
|
||||
|
||||
### Google Sheets as infrastructure
|
||||
- Config store, market DB, ops dashboard, and inter-process message bus are all
|
||||
one spreadsheet. It's rate-limited (Sheets API quota), slow (hundreds of ms
|
||||
per read), needs service-account credentials, and pulls a network dependency
|
||||
into the trading loop. The "read-only mode" fallback in
|
||||
`poly_utils/google_utils.py` literally guesses gid numbers 0–4 against a CSV
|
||||
export URL.
|
||||
- The scanner and the bot communicate *through* the sheet, so market metadata
|
||||
in the hot path can be an hour stale (`row['best_bid']` used as a sanity
|
||||
reference in trading.py:356 comes from the sheet).
|
||||
|
||||
### Code quality
|
||||
- Bare `except:` everywhere (~30 sites), `print()` as the only logging, no
|
||||
tests at all, no types, dead code left as comments, `gc.collect()` sprinkled
|
||||
as a superstition (7 call sites), `deets` reused as both the outcome list and
|
||||
the book snapshot inside the same loop (trading.py:158/197).
|
||||
- `data_updater/` duplicates `poly_utils` (two `google_utils.py`, two
|
||||
`trading_utils.py` with different contents).
|
||||
- Pinned to **Python 3.9** (.python-version = 3.9.18, EOL Oct 2025) and
|
||||
`py-clob-client==0.28.0`. That client line is **dead**: Polymarket cut over
|
||||
to CLOB V2 on 2026-04-28 (new order signing, new contracts) and archived
|
||||
py-clob-client — **v1 of this bot cannot place an order at all anymore**
|
||||
(see 03). This isn't a refactor of a working bot; it's a rebuild of a
|
||||
non-functional one.
|
||||
- A **Node.js subprocess** (`poly_merger/`) exists solely to call
|
||||
`mergePositions` through a Gnosis Safe — a whole second runtime, package.json
|
||||
and ethers v5 dependency for one contract call that web3.py can do.
|
||||
|
||||
### Strategy weaknesses (why it loses money — detailed fix in 04-strategy.md)
|
||||
- **Quote logic is penny-jumping**: `bid = best_bid + tick`, `ask = best_ask -
|
||||
tick`, with a couple of size-based exceptions. No fair-value estimate, no
|
||||
microprice, no inventory skew — position size only gates *whether* to quote,
|
||||
never *where*.
|
||||
- **Adverse selection is handled after the fact**: the only defenses are an
|
||||
hourly 3-hour volatility number from the spreadsheet and a stop-loss that
|
||||
**crosses the spread** (sells at best bid — a taker exit, exactly what we
|
||||
want to ban) then sleeps the market for hours via a JSON file on disk.
|
||||
- **Rewards are a filter, not an objective**: the scanner computes
|
||||
reward-per-100 carefully, but the live quoter only checks "is my bid inside
|
||||
the incentive band" as a boolean. Placement within the band (where the
|
||||
reward-vs-fill-risk tradeoff actually lives) is never optimized.
|
||||
- Hard-coded magic everywhere: 0.1–0.9 price band, 250 share absolute cap,
|
||||
0.005 reprice threshold, 15s staleness, 0.95/0.97/1.01 ratios, `max_size*2`
|
||||
total-exposure allowance.
|
||||
- Take-profit reprices only when 2% away from target; sell-side quote sits at
|
||||
`avgPrice` floor (trading.py / trading_utils.py:136) — i.e. the bot refuses
|
||||
to exit below cost, which turns losers into resolution lottery tickets.
|
||||
|
||||
## What's worth keeping (conceptually)
|
||||
|
||||
- **The two-token + merge insight**: buying YES and NO such that both fill and
|
||||
merging the pair back to USDC is a maker-only exit that never crosses the
|
||||
spread. v2 promotes this from an afterthought to a core mechanism.
|
||||
- Book maintenance via `SortedDict` and the YES/NO mirror trick — cheap and
|
||||
correct.
|
||||
- The desired-vs-existing order diffing idea in `send_buy_order`/`send_sell_order`
|
||||
(only cancel/replace on meaningful change) — right instinct, generalized into
|
||||
a proper reconciler in v2.
|
||||
- The scanner's reward-per-$100 estimator (`add_formula_params` implements the
|
||||
official S(v,s) = ((v−s)/v)² scoring) — the math moves into v2's selector.
|
||||
- The `performing` concept (in-flight fills gate REST reconciliation) — becomes
|
||||
a real state machine instead of dicts of sets.
|
||||
|
||||
## What dies in v2
|
||||
|
||||
| Component | Fate |
|
||||
|---|---|
|
||||
| Google Sheets (gspread, gspread-dataframe, google-auth, service accounts) | **Deleted.** Local files + SQLite (02/05) |
|
||||
| `poly_merger/` Node.js + ethers + package.json | **Deleted.** Native Python Safe execution (03) |
|
||||
| `update_stats.py`, `poly_stats/` (Sheets dashboards) | **Deleted.** Replaced by local status CLI/log (05) |
|
||||
| `data_updater/` sampling-markets crawl + per-market book fetch | **Deleted.** Gamma API scanner, minutes → seconds (03/05) |
|
||||
| `global_state.py` module-level dicts | **Deleted.** Typed state store owned by the engine (02) |
|
||||
| Background REST polling thread | **Deleted.** Single asyncio loop; REST only for reconciliation (02) |
|
||||
| `positions/*.json` risk files | **Deleted.** SQLite state (02) |
|
||||
| Python 3.9 pin, py-clob-client 0.28 (archived, V1-only) | Upgraded: 3.12+, py-clob-client-v2 + own async HTTP (03) |
|
||||
| `print()` logging, bare excepts, gc.collect() | Replaced: structured logging, real error policy (02/05) |
|
||||
|
||||
## Repo facts for reference
|
||||
|
||||
- uv is **already adopted** (uv.lock, pyproject.toml, hatchling build) — v2
|
||||
keeps uv and finishes the job: bump `requires-python` to ≥3.12, refresh all
|
||||
pins, add ruff + mypy + pytest as dev group, `src/` layout.
|
||||
- Git history is shallow and healthy; no CI exists; MIT licensed; README
|
||||
already warns the v1 strategy is unprofitable.
|
||||
@@ -1,214 +0,0 @@
|
||||
# 02 — v2 Architecture
|
||||
|
||||
Single async Python process, local-file config, typed state, no threads, no
|
||||
Google, no Node. Everything the bot needs to run lives in the repo directory.
|
||||
|
||||
## Design principles
|
||||
|
||||
1. **One event loop.** All I/O is async. Anything blocking (order signing is
|
||||
CPU-trivial; EIP-712 signing is ~1ms) stays inline; anything slow-blocking
|
||||
that can't be made async is banished to `asyncio.to_thread` — but the goal
|
||||
is zero such call sites.
|
||||
2. **Websocket-first, REST-reconcile.** The WS feeds are the source of truth
|
||||
for books, orders, and fills in real time. REST is used for (a) startup
|
||||
snapshots, (b) periodic drift reconciliation, (c) actions. Never poll REST
|
||||
for something the WS already pushes.
|
||||
3. **Desired-state reconciliation.** The strategy never "places an order"; it
|
||||
emits a *target quote set* per market. The execution layer diffs targets
|
||||
against live open orders and issues the minimal cancel/place set, applying
|
||||
tolerances (don't churn for sub-tick / sub-10% size changes) and rate-limit
|
||||
budgets. This is the v1 `should_cancel` instinct, made a first-class layer.
|
||||
4. **Local-first ops.** Config = files in the repo. State = SQLite. Dashboards
|
||||
= CLI. An operator with a laptop and the repo can run, inspect, and tune
|
||||
everything with no external accounts beyond the Polymarket wallet.
|
||||
5. **Deterministic and testable.** Strategy is a pure function
|
||||
`(BookState, Inventory, Params, Clock) -> TargetQuotes`. That makes replay
|
||||
backtesting and unit tests trivial and is the enabler for 04-strategy.
|
||||
|
||||
## Process layout
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────┐
|
||||
│ polymaker run │
|
||||
│ │
|
||||
Gamma REST ──────▶│ MarketCatalog (refresh ~60s, metadata) │
|
||||
│ │
|
||||
CLOB market WS ──▶│ MarketDataService │
|
||||
│ └─ per-market OrderBook (seq/hash check) │
|
||||
CLOB user WS ────▶│ UserStream │
|
||||
│ └─ fills / order lifecycle events │
|
||||
│ │
|
||||
│ StateStore (positions, open orders, PnL) │
|
||||
│ ▲ WS events ▲ REST reconcile │
|
||||
│ │
|
||||
│ StrategyEngine │
|
||||
│ └─ per-market Quoter task (debounced) │
|
||||
│ emits TargetQuotes │
|
||||
│ │
|
||||
│ RiskManager (pre-trade gates, kill switch) │
|
||||
│ │
|
||||
│ ExecutionGateway │
|
||||
│ ├─ order signer (EIP-712, local) │
|
||||
│ ├─ REST client (async, rate budgeted) │
|
||||
│ └─ reconciler (target vs live diff) │
|
||||
│ │
|
||||
│ Merger (YES+NO -> USDC, web3.py native) │
|
||||
│ │
|
||||
│ Persistence: SQLite + JSONL event journal │
|
||||
│ Telemetry: structlog -> console/file │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Component responsibilities
|
||||
|
||||
**MarketCatalog** — market metadata (tokens, tick size, neg-risk flag, reward
|
||||
params, fees, end date) fetched from Gamma/CLOB REST (see 03). Refreshed on a
|
||||
slow timer, cached in SQLite. Nothing in the hot path ever awaits this.
|
||||
|
||||
**MarketDataService** — owns the market WS connection(s). Maintains one
|
||||
`OrderBook` per subscribed market: `SortedDict` bids/asks (keep v1's approach),
|
||||
YES-canonical with NO-side views derived. Handles: snapshot on subscribe,
|
||||
`price_change` deltas, `tick_size_change`, book-hash verification when
|
||||
available, and reconnect-with-resnapshot. Exposes sync accessors (microprice,
|
||||
depth-within-band, imbalance) — pure reads, no awaits — plus an
|
||||
`asyncio.Event`-style dirty flag per market that wakes the Quoter.
|
||||
|
||||
**UserStream** — user WS channel: order placements/cancels/matches and trade
|
||||
status transitions (MATCHED → MINED → CONFIRMED / FAILED). Feeds StateStore.
|
||||
Replaces v1's `performing` dicts with an explicit per-order/per-trade state
|
||||
machine (see below).
|
||||
|
||||
**StateStore** — single owner of positions and open orders. In-memory, typed
|
||||
(dataclasses), persisted to SQLite on change (WAL mode; writes are µs-cheap).
|
||||
Three inputs, one arbitration rule:
|
||||
- WS fill events apply immediately (optimistic),
|
||||
- REST reconciliation (every ~30s, and on demand after anomalies) corrects
|
||||
drift *only* for tokens with no in-flight trades,
|
||||
- on-chain balance is consulted only by the Merger before merging.
|
||||
This is v1's `performing`/`last_trade_update`/`avgOnly` logic, but as one
|
||||
explicit component with tests instead of three dicts and a timeout.
|
||||
|
||||
**StrategyEngine / Quoter** — one lightweight task per market. Loop:
|
||||
```
|
||||
wait for (book dirty | fill | timer tick | param change), coalesced
|
||||
debounce: recompute at most once per N ms (default 200ms), immediately on fill
|
||||
compute TargetQuotes = strategy(book, inventory, params, clock) # pure
|
||||
hand to RiskManager -> ExecutionGateway
|
||||
```
|
||||
No lock held during I/O; the Quoter never awaits network calls itself.
|
||||
|
||||
**RiskManager** — pre-trade checks (per-market notional cap, global exposure
|
||||
cap, neg-risk group cap, max open order value, price sanity band vs recent
|
||||
mid), plus global circuit breakers (drawdown kill switch, WS staleness halt,
|
||||
API error-rate halt). Owns the "reduce-only" and "halt" market modes. Detailed
|
||||
policy in 04-strategy.md §Risk.
|
||||
|
||||
**ExecutionGateway** — the only component that talks to the CLOB REST API for
|
||||
actions. Async HTTP (httpx), local EIP-712 signing (V2 order format), a
|
||||
token-bucket rate budgeter tuned to documented API limits (03), batch
|
||||
place/cancel endpoints, **post-only on every quote**, and the target-vs-live
|
||||
reconciler with churn tolerances. Also owns the **exchange heartbeat** (03):
|
||||
a ~5s keepalive; if the process dies or partitions, the exchange cancels all
|
||||
our orders within ~10s — the dead-man switch a maker-only bot must have.
|
||||
Every request/response is journaled.
|
||||
|
||||
**Merger** — pure-Python replacement for poly_merger (03 §Merge). Triggered by
|
||||
StateStore when min(YES,NO) ≥ merge threshold; runs as a low-priority task,
|
||||
never blocks quoting.
|
||||
|
||||
**Persistence** — `state.db` (SQLite): market catalog cache, positions, order
|
||||
history, fills, PnL marks, risk events. `journal/*.jsonl`: append-only raw
|
||||
event log (WS messages in, orders out) — this is also the capture format the
|
||||
backtester replays (04 §Simulation).
|
||||
|
||||
**Telemetry** — structlog JSON to file + human console; `polymaker status`
|
||||
CLI reads SQLite for positions/PnL/quotes (05). Optional later: Prometheus
|
||||
exporter. No Sheets dashboards.
|
||||
|
||||
## Order/trade state machine
|
||||
|
||||
Replaces `performing` + timestamps + 15s cleanup:
|
||||
|
||||
```
|
||||
Order: DRAFT -> SIGNED -> POSTED -> LIVE -> {PARTIALLY_FILLED} -> DONE
|
||||
└─ REJECTED └─ CANCELED
|
||||
Trade: MATCHED -> MINED -> CONFIRMED
|
||||
└────────────-> FAILED (position rolled back, reconcile forced)
|
||||
```
|
||||
Timeouts are per-transition (e.g. MATCHED without MINED after T → force REST
|
||||
reconcile for that token) instead of one global 15s sweep.
|
||||
|
||||
## Package layout
|
||||
|
||||
```
|
||||
poly-maker/
|
||||
├── pyproject.toml # uv-managed; python >=3.12
|
||||
├── uv.lock
|
||||
├── config/
|
||||
│ ├── config.toml # bot settings: wallet env refs, global risk, engine knobs
|
||||
│ ├── strategy.toml # named parameter profiles (replaces Hyperparameters sheet)
|
||||
│ └── markets.toml # selected markets + per-market overrides (replaces Selected Markets)
|
||||
├── src/polymaker/
|
||||
│ ├── cli.py # `polymaker` entrypoint: run/scan/markets/status/flatten/...
|
||||
│ ├── config.py # pydantic-settings models, TOML load, hot-reload (watchfiles)
|
||||
│ ├── catalog/ # Gamma/CLOB market discovery + scoring (05)
|
||||
│ ├── marketdata/ # WS client, order book, book analytics
|
||||
│ ├── userstream/ # user WS, event parsing
|
||||
│ ├── state/ # StateStore, models, SQLite persistence
|
||||
│ ├── strategy/ # pure quoting logic + parameter models (04)
|
||||
│ ├── risk/ # RiskManager, circuit breakers
|
||||
│ ├── execution/ # signer, REST client, rate budgeter, reconciler
|
||||
│ ├── merge.py # native merge (03)
|
||||
│ ├── journal.py # JSONL event journal
|
||||
│ └── sim/ # replay backtester + paper-trading mode (04)
|
||||
├── tests/
|
||||
└── docs/scoping/
|
||||
```
|
||||
|
||||
`poly_data`, `poly_utils`, `poly_stats`, `data_updater`, `poly_merger`,
|
||||
`trading.py`, `main.py`, `update_markets.py`, `update_stats.py` are all
|
||||
deleted once their replacements land (06 has the phasing).
|
||||
|
||||
## Concurrency & performance notes
|
||||
|
||||
- Python 3.12+ (faster asyncio, per-interpreter GIL groundwork irrelevant here
|
||||
— one loop is enough; this workload is I/O bound with tiny CPU bursts).
|
||||
- `uvloop` as the event loop (free ~2-4x loop throughput on macOS/Linux).
|
||||
- WS message → book update → quoter wake is all sync code on the loop; the
|
||||
only awaits in the hot path are the WS `recv` and the outbound HTTP.
|
||||
- Signing: `py_order_utils`/eth-account key ops are ~1ms — sign inline, no
|
||||
executor.
|
||||
- Target: event-to-order-POST latency < 50ms internally (network to Polymarket
|
||||
will dominate at ~50–150ms; colocating the box geographically near the CLOB
|
||||
(AWS us-east) is an ops choice, noted in 05).
|
||||
- Memory: no gc.collect() calls; books for ~50 markets are a few MB.
|
||||
|
||||
## Error policy
|
||||
|
||||
- No bare excepts. Each component defines recoverable (log + retry with
|
||||
backoff) vs fatal (halt quoting, keep cancel capability, alert) errors.
|
||||
- On any uncertainty about our own open-order/position state → freeze new
|
||||
quotes for that market, force reconcile, resume. "When confused, cancel and
|
||||
resync" is the universal recovery.
|
||||
- On process crash: the exchange heartbeat has already cancelled our orders
|
||||
within ~10s; startup sequence is cancel-all (belt) → REST snapshot → resume.
|
||||
|
||||
## Dependency set (target)
|
||||
|
||||
| Purpose | Package |
|
||||
|---|---|
|
||||
| HTTP | httpx (async) |
|
||||
| WS | websockets (current) |
|
||||
| Signing/order building | **py-clob-client-v2** (V1 client is archived/dead); direct py_order_utils + eth-account if we outgrow it (03) |
|
||||
| Chain (merge, balances) | web3.py 7.x |
|
||||
| Config/validation | pydantic v2 + pydantic-settings, tomllib |
|
||||
| Hot reload | watchfiles |
|
||||
| State | sqlite3 stdlib (+ tiny DAO layer; no ORM) |
|
||||
| Logging | structlog |
|
||||
| CLI | typer + rich |
|
||||
| Loop | uvloop |
|
||||
| Dev | ruff, mypy, pytest, pytest-asyncio |
|
||||
|
||||
pandas/numpy leave the runtime hot path (they remain fine inside the scanner
|
||||
and sim analysis). gspread, gspread-dataframe, google-auth, py-clob-client
|
||||
(v1), and the entire Node/ethers runtime go; sortedcontainers stays (book).
|
||||
@@ -1,203 +0,0 @@
|
||||
# 03 — API Layer: V1 → V2 Migration
|
||||
|
||||
Polymarket executed a **hard, non-backward-compatible cutover to CLOB V2 on
|
||||
April 28, 2026**. V1-signed orders stopped working that day; the legacy
|
||||
`py-clob-client` this repo pins (0.28.0; latest 0.34.6) was **archived May 11,
|
||||
2026** and cannot trade against V2. There is no incremental path — the API
|
||||
layer is a rewrite, which is exactly what this scoping assumes.
|
||||
|
||||
Primary sources: [V2 migration guide](https://docs.polymarket.com/v2-migration),
|
||||
[changelog](https://docs.polymarket.com/changelog),
|
||||
[llms.txt doc index](https://docs.polymarket.com/llms.txt).
|
||||
|
||||
## 1. What changed in CLOB V2
|
||||
|
||||
| Area | V1 (this repo) | V2 (target) |
|
||||
|---|---|---|
|
||||
| Base URL | `https://clob.polymarket.com` | same URL, new backend |
|
||||
| Order struct | `nonce`, `feeRateBps`, `taker` fields | removed; adds `timestamp` (ms), `metadata`, `builder`; fees set by protocol at match time |
|
||||
| EIP-712 domain | Exchange version "1" | version **"2"**, new verifying contracts |
|
||||
| Exchange contracts | `0x4bFb41d5...` / neg-risk `0xC5d563A3...` | CTF Exchange V2 `0xE111180000d2663C0091e4f400237545B87B996B`; Neg Risk CTF Exchange V2 `0xe2222d279d744050d28e00520010520000310F59` |
|
||||
| Collateral | USDC.e (`0x2791Bca1...`) | **pUSD** (ERC-20, 1:1 USDC-backed); API traders wrap via the Collateral Onramp contract and set pUSD allowances |
|
||||
| Auth | L1 EIP-712 + L2 HMAC headers | **unchanged**; existing API keys carried over |
|
||||
| Fees | fields existed, mostly zero | **makers pay zero**; taker-only fees `fee = C · feeRate · p·(1−p)` per category (Politics 1.00%, geopolitics/world-events free); live rate from each market's `feeSchedule` — never hardcode |
|
||||
| Neg-risk | PartialCreateOrderOptions flag | fully supported; neg-risk orders sign against the neg-risk V2 contract (SDK handles) |
|
||||
|
||||
### Consequences for us
|
||||
- **Wallet/allowance setup is a new ops step**: wrap USDC→pUSD via the onramp,
|
||||
approve pUSD to the V2 exchange (buys) and conditional tokens (sells).
|
||||
Balance checks read pUSD, not USDC.e (v1's hardcoded USDC contract in
|
||||
`polymarket_client.py:78` dies).
|
||||
- **Signature-type risk**: v1 runs `signature_type=2` (Polymarket proxy/Safe
|
||||
funder). `py-clob-client-v2` has a cluster of open auth bugs around
|
||||
signature types 2/3 and Safe wallets (issues #70/#85/#87/#90/#91) and
|
||||
"invalid order version" migration confusion (#92). **Phase 2 starts with a
|
||||
wallet-auth spike**: place/cancel one order with our exact wallet setup
|
||||
before building anything on top. Fallback: EOA wallet (signature_type=0).
|
||||
- Fine ticks now exist (0.001, even 0.0025); `tick_size_change` events (fires
|
||||
crossing 0.96/0.04) must be handled live or orders start rejecting.
|
||||
|
||||
## 2. Client library choice
|
||||
|
||||
| Option | Status | Call |
|
||||
|---|---|---|
|
||||
| `py-clob-client` 0.34.x | archived, V1-only | **dead — remove** |
|
||||
| **`py-clob-client-v2`** (1.0.2, Jul 2026) | official successor, active | **use for Phase 2.** Sync-only → wrap calls we need in a thin async adapter, or use it only for signing/building and POST via httpx ourselves |
|
||||
| `polymarket-client` (py-sdk) | official unified SDK, **beta**, sync+async | watch; adopt when stable if its async client is solid — re-evaluate at Phase 2 start |
|
||||
| Direct: `py_order_utils` + eth-account + httpx | max control | fallback if the v2 client's sync-ness or Safe bugs block us; order signing is a well-documented EIP-712 struct |
|
||||
|
||||
Decision: **start with `py-clob-client-v2` for order building/signing, own the
|
||||
HTTP layer with httpx** (async, connection-pooled, our rate budgeter, our
|
||||
retries). The signing code is pure CPU and composes fine with our loop. This
|
||||
also keeps us one small step from going fully direct if the SDK disappoints.
|
||||
|
||||
## 3. Endpoints v2 uses (and the v1 habits they replace)
|
||||
|
||||
### Trading (CLOB, L2 auth)
|
||||
- `POST /order` — single post-only/GTC/GTD order.
|
||||
- **`POST /orders` (batch, ≤15)** — quote both tokens × layers in one request;
|
||||
v1 placed orders one blocking call at a time.
|
||||
- `DELETE /orders` (≤1,000 ids) — targeted mass cancel; replaces v1's
|
||||
cancel-all-per-asset churn.
|
||||
- `DELETE /cancel-market-orders`, `DELETE /cancel-all` — recovery paths
|
||||
(startup, kill switch).
|
||||
- **`POST /heartbeats`** (Jan 2026): opt-in dead-man switch — miss a ~10s
|
||||
heartbeat window and the exchange cancels all our orders. **Mandatory for
|
||||
v2**: a maker-only book of stale quotes after a crash/network partition is
|
||||
the biggest tail risk we have. ExecutionGateway sends one every ~5s.
|
||||
- **Order types**: GTC, GTD, FOK, FAK, and **Post-Only (Jan 2026, GTC/GTD
|
||||
only)** — the enforcement mechanism for our maker-only mandate (04 §3.2);
|
||||
every quote goes out post-only.
|
||||
|
||||
### Market data (REST, no auth)
|
||||
- `GET /book`, **`POST /books` (batch)**, `/midpoint(s)`, `/price(s)` —
|
||||
snapshots at startup/resync. The scanner uses batch `/books` instead of v1's
|
||||
one-book-per-market crawl.
|
||||
- `GET /prices-history` — still exists; only used by the scanner for coarse
|
||||
historical vol (live vol comes from our own WS-derived estimates).
|
||||
- `GET /sampling-markets` — replaced by Gamma for discovery (below); CLOB
|
||||
market metadata (`min_incentive_size`, `max_incentive_spread`, tick size,
|
||||
neg_risk) still fetched per selected market as authoritative.
|
||||
|
||||
### Rewards & earnings
|
||||
- Liquidity-rewards program is live and mechanically the same family as v1:
|
||||
score `S = ((v−s)/v)²`, adjusted midpoint (dust-filtered), two-sided
|
||||
`Q_min = max(min(Q_one,Q_two), max(Q_one/c, Q_two/c))` with c=3.0,
|
||||
per-minute sampling, daily payout, $1/market minimum, **double-sided
|
||||
quoting required to score at all when mid < 0.10 or > 0.90**.
|
||||
[docs](https://docs.polymarket.com/developers/market-makers/liquidity-rewards)
|
||||
- **Maker Rebates Program (new, Jan 2026)**: ~20–25% of taker fees collected
|
||||
in fee-enabled categories (politics 25%-ish) rebated daily to that market's
|
||||
makers pro-rata. A second income stream v1 never knew about — enters the
|
||||
strategy utility function (04 §4) and the market selector's scoring (05).
|
||||
- Earnings actuals: rewards user endpoints (v1's `poly_stats` hit a
|
||||
polymarket.com internal API with L2 headers; v2 uses the documented rewards
|
||||
endpoints) — pulled daily into SQLite for calibration.
|
||||
|
||||
### Relayer (only if we keep Safe-style wallets)
|
||||
- Relayer `POST /submit` now returns `{transactionID, state}` — poll
|
||||
`GET /transaction` for the hash. Affects the merge flow if executed via
|
||||
proxy wallet (§6).
|
||||
|
||||
## 4. Market discovery: Gamma API (the "better, faster market list")
|
||||
|
||||
v1's scanner crawled `sampling-markets` page by page then fetched **one order
|
||||
book + one price history per market** — thousands of sequential calls, ~hourly
|
||||
cadence, results pasted into Google Sheets. v2 replaces the whole pipeline
|
||||
with Gamma (`https://gamma-api.polymarket.com`, no auth):
|
||||
|
||||
- **`GET /markets/keyset`** (Apr 2026, cursor-paginated, limit ≤100; the
|
||||
offset endpoints are slated for deprecation — build keyset-first) with
|
||||
server-side filters: `active`, `closed` (defaults false now), `tag_id` (+
|
||||
`related_tags`) for **politics**, `liquidity_num_min`, `volume_num_min`,
|
||||
`end_date_min/max`, `rewards_min_size`.
|
||||
- One response row already carries what v1 burned two extra calls per market
|
||||
to compute: `bestBid`, `bestAsk`, `spread`, `liquidityNum`, `volumeNum` (+
|
||||
24h/1wk/1mo), `rewardsMinSize`, `rewardsMaxSpread`, `feeSchedule`/`feesEnabled`,
|
||||
`orderPriceMinTickSize`, `orderMinSize`, `negRisk`, `clobTokenIds`,
|
||||
`conditionId`, `endDate`.
|
||||
- Tag IDs are numeric — resolve once via `GET /tags`, cache in SQLite.
|
||||
- A politics-filtered scan is **a handful of paginated requests (seconds)**
|
||||
vs. v1's ~hour; run it every few minutes if we like. Rate limits are ample
|
||||
(Gamma 4,000 req/10s global, `/markets` 300/10s).
|
||||
- Depth/reward-optimality checks for the shortlist only: batch `POST /books`
|
||||
on candidate tokens.
|
||||
- Event grouping for neg-risk risk caps comes from `GET /events/keyset`
|
||||
(markets nest their event; events give us the sibling-outcome structure).
|
||||
|
||||
## 5. WebSockets
|
||||
|
||||
Same hosts as v1 (`wss://ws-subscriptions-clob.polymarket.com/ws/{market,user}`)
|
||||
— **URLs unchanged by the V2 cutover** — but the payloads moved:
|
||||
|
||||
- **Market channel**: subscribe with
|
||||
`{"assets_ids": [...], "type": "market", "custom_feature_enabled": true}`.
|
||||
Events: `book` (snapshot), `price_change` (**schema changed Sept 2025 —
|
||||
now includes best_bid/best_ask; re-derive the parser from current docs**),
|
||||
`tick_size_change`, `last_trade_price` (includes `fee_rate_bps`), and with
|
||||
the custom flag: **`best_bid_ask`, `new_market`, `market_resolved`**.
|
||||
`market_resolved` is a free early-resolution alarm for the risk ladder
|
||||
(04 §6). The old ~100-token subscription cap was removed (May 2025).
|
||||
- **User channel**: auth payload as v1, but scope subscriptions with
|
||||
`"markets": [condition_ids]`. `order` + `trade` events, trade status ladder
|
||||
`MATCHED → MINED → CONFIRMED / RETRYING / FAILED` (maps directly onto the
|
||||
02 state machine).
|
||||
- **Ping/pong**: reportedly server pings every 5s with a 10s pong deadline —
|
||||
not confirmed on the official channel pages; **verify empirically** and make
|
||||
keepalive parameters config, not constants.
|
||||
- **RTDS** (`wss://ws-live-data.polymarket.com`, Sept 2025): separate
|
||||
real-time data service — `activity` (all trades/matches platform-wide),
|
||||
`comments`, crypto/equity prices, `clob_user` topic. Not needed for the
|
||||
core loop; the `activity` topic is a cheap future enhancement for
|
||||
cross-market flow signals (still Polymarket-internal data, so it doesn't
|
||||
violate the no-external-feeds rule; noted in 07).
|
||||
|
||||
## 6. On-chain layer (merge + balances) — Node.js dies
|
||||
|
||||
- **Merger rewrite in Python** (web3.py 7.x, which the repo already ships):
|
||||
`mergePositions` on the ConditionalTokens contract (plain markets) or the
|
||||
NegRiskAdapter (neg-risk), exactly what `poly_merger/merge.js` does through
|
||||
ethers v5. For proxy/Safe wallets, execute through the Safe with
|
||||
`safe-eth-py`, or via Polymarket's relayer client — decide in the Phase 2
|
||||
wallet spike (same spike as §1 signature-type risk; if we land on an EOA,
|
||||
the Safe code path disappears entirely).
|
||||
- **pUSD questions to resolve in the spike** (docs are young): pUSD token
|
||||
address; whether V2-era markets' CTF collateral is pUSD (merge returns pUSD)
|
||||
and whether pre-V2 political markets still merge to USDC.e; onramp
|
||||
wrap/unwrap mechanics and gas. The Merger and the balance sheet code treat
|
||||
collateral as config-driven per market, not a constant.
|
||||
- Balances: pUSD ERC-20 for cash, CTF `balanceOf` for positions (unchanged),
|
||||
data-api `/positions` for cross-checks (rate limit 150/10s — reconcile use
|
||||
only).
|
||||
|
||||
## 7. Rate budget (per official limits, Jun 2026)
|
||||
|
||||
Relevant ceilings: `POST /order` 5,000/10s burst; batch `POST /orders`
|
||||
2,000/10s; `DELETE /orders` 2,000/10s; `cancel-market-orders` 1,500/10s;
|
||||
books/prices 500–1,500/10s; Cloudflare throttles (queues) rather than 429s —
|
||||
so the real risk is **silent latency injection**, not errors.
|
||||
|
||||
For ~50 markets × 2 tokens × 2 layers ≈ 200 live orders, a full re-quote is
|
||||
~14 batch calls — nothing. The ExecutionGateway still gets a token-bucket
|
||||
budgeter (config: fraction of documented limits, default 25%) because
|
||||
throttle-queueing would silently add latency exactly when the market is
|
||||
moving; the budgeter surfaces pressure as a metric + shed-load policy
|
||||
(skip reprices whose edge < threshold first) instead.
|
||||
|
||||
## 8. Out of scope
|
||||
|
||||
- **Polymarket US** (docs.polymarket.us): separate CFTC-regulated venue,
|
||||
separate API/auth/SDKs. Not targeted; noted for the future in 07.
|
||||
- Builder attribution (`builderCode`): not needed unless we monetize order
|
||||
flow; field exists in the order struct if ever wanted.
|
||||
|
||||
## 9. Open items to verify at implementation time
|
||||
|
||||
1. WS ping/pong contract (empirical test).
|
||||
2. `py-sdk` maturity re-check at Phase 2 start (async client + WS coverage).
|
||||
3. pUSD address / onramp mechanics / merge collateral (wallet spike).
|
||||
4. Exact current `price_change` + `best_bid_ask` payload schemas.
|
||||
5. Maker-rebate payout currency (USDC vs pUSD) — accounting only.
|
||||
6. Whether heartbeats interact with batch cancels/cancel-all on reconnect.
|
||||
7. Signature type 2 (proxy/Safe) health in `py-clob-client-v2` (issue tracker
|
||||
before the spike).
|
||||
@@ -1,292 +0,0 @@
|
||||
# 04 — Strategy v2: Maker-Only Quoting for Political Markets
|
||||
|
||||
The v1 strategy is penny-jumping with an after-the-fact stop loss. v2 replaces
|
||||
it with a quoting engine built around four ideas:
|
||||
|
||||
1. a **fair value estimate** derived purely from market microstructure (no
|
||||
external data in this phase — see 07 for the future feed layer),
|
||||
2. **inventory-skewed, volatility-scaled quotes** that never cross the spread,
|
||||
3. **liquidity rewards as an explicit term in the objective**, not a boolean
|
||||
filter,
|
||||
4. **regime detection** that decides when *not* to quote — which is where most
|
||||
of the money is saved.
|
||||
|
||||
Strategy code is a pure function `(BookState, Inventory, Params, Clock) ->
|
||||
TargetQuotes`, unit-testable and replayable (see §9).
|
||||
|
||||
## 0. The market we're fitting
|
||||
|
||||
Political markets on Polymarket have a specific microstructure profile:
|
||||
|
||||
- **Binary, bounded [0,1]**, usually inside **neg-risk event groups** (multi-
|
||||
candidate events where Σ YES = 1 by construction).
|
||||
- **Long-dated with punctuated equilibrium**: weeks of low-vol mean-reversion
|
||||
interrupted by violent news repricing (debates, indictments, dropouts,
|
||||
election nights). The flow between events is heavily retail/directional
|
||||
(favorable for a maker); the flow *during* events is fast and toxic.
|
||||
- **Early-resolution risk**: a candidate dropping out gates a market to 0
|
||||
instantly. There is no external feed to warn us in v2 — microstructure
|
||||
(sweeps, book evaporation) is the alarm, and size caps are the survival
|
||||
mechanism.
|
||||
- Depth is deep at the touch on flagship markets, thin on down-ballot ones —
|
||||
parameters must scale with observed book depth, not fixed constants.
|
||||
|
||||
Everything below is designed against this profile.
|
||||
|
||||
## 1. Fair value (FV) engine
|
||||
|
||||
All inputs are local: the book we maintain from the WS and the trade tape.
|
||||
|
||||
- **Microprice**: depth-weighted mid over the top K levels (K≈3), i.e. mid
|
||||
pulled toward the thin side. Robust to size-1 dust orders via per-level
|
||||
minimum-size filtering (keep v1's `find_best_price_with_size` idea, tuned by
|
||||
observed depth instead of hardcoded 100/20 shares).
|
||||
- **Flow adjustment**: EWMA of signed aggressor volume (from `last_trade_price`
|
||||
/trade events) over ~1–5 min nudges FV in the direction of persistent flow.
|
||||
Political markets trend on news; leaning with flow reduces adverse fills.
|
||||
- **Cross-outcome consistency (neg-risk)**: Σ of YES microprices across the
|
||||
event should ≈ 1. Persistent deviation → books dislocated (news repricing in
|
||||
progress) → contributes to regime score (§5); large deviation is also the
|
||||
trigger for the optional arb module (§7).
|
||||
- **Clamping**: FV is smoothed (short EWMA) and clamped against jumps unless
|
||||
the jump is confirmed by trades, not just quote flicker.
|
||||
|
||||
Explicitly *not* an input in v2: polls, news, other venues (see 07).
|
||||
|
||||
## 2. Volatility & toxicity engine
|
||||
|
||||
Replaces the spreadsheet's hour-stale "3_hour volatility" with live estimates:
|
||||
|
||||
- **Realized vol**: EWMA of squared FV returns at three horizons (~10s, ~1m,
|
||||
~15m), computed on the WS stream. σ_short drives spread width; the
|
||||
short/long ratio drives regime detection.
|
||||
- **Sweep detector**: single aggression that consumes ≥N levels or moves FV by
|
||||
≥X ticks → immediate event flag (§5).
|
||||
- **Markout tracker** (self-tuning core): for every one of our fills, record
|
||||
FV at +30s and +5m. Per-market EWMA of markout = realized adverse-selection
|
||||
cost. Consistently negative markouts automatically widen that market's
|
||||
spread / reduce its size via a toxicity multiplier. This is how the bot
|
||||
learns "this market eats me" without any human or external data.
|
||||
|
||||
## 3. Quote construction (maker-only)
|
||||
|
||||
### 3.1 Reservation price and half-spread
|
||||
|
||||
```
|
||||
skew = γ · σ_short · (q / q_max) # in price units
|
||||
r = FV − skew # reservation price
|
||||
δ = max(δ_min, c_vol·σ_short + c_tox·toxicity)
|
||||
bid_raw = r − δ ; ask_raw = r + δ
|
||||
```
|
||||
- `q` = net inventory in the YES token (NO inventory counts negative),
|
||||
`q_max` = per-market cap. Long inventory pushes both quotes down: buy
|
||||
cheaper, sell more eagerly. This single mechanism replaces v1's pile of
|
||||
size-gating special cases.
|
||||
- Fees: on CLOB V2 **makers pay zero** — no maker fee term in δ. Taker fees
|
||||
(politics 1.00%, per-market `feeSchedule` from the catalog, 03) matter to
|
||||
us twice: they fund the **maker rebates** income term (§4), and they enter
|
||||
the cost model of the optional set-arb module (§7) only.
|
||||
- All prices then snap to tick and go through **placement logic**:
|
||||
|
||||
### 3.2 Placement: join, don't jump
|
||||
|
||||
- Default: place at the better of (raw price, one tick behind the touch) —
|
||||
i.e. **join the level or sit behind it**; jump the queue (improve the touch)
|
||||
only when `FV − best_ask` / `best_bid − FV` edge exceeds tick + costs. v1
|
||||
penny-jumps unconditionally, which wins queue position precisely when the
|
||||
price is about to be wrong.
|
||||
- **Never cross**: candidate bid ≥ best_ask ⇒ reprice to best_ask − tick (and
|
||||
symmetric for asks), validated against the live book at send time. Enforced
|
||||
at the exchange too: **every quote is a Post-Only order** (added to CLOB
|
||||
Jan 2026, GTC/GTD — 03), which rejects rather than takes. Maker-only is a
|
||||
property of the system, not a convention.
|
||||
- **Min edge vs FV**: bid ≤ FV − δ_edge_min always; we never pay through fair
|
||||
value for queue position or rewards.
|
||||
- **Layering**: optionally split size across 2–3 levels per side (touch-join +
|
||||
deeper levels). Sweeps fill the deep levels at prices that already include
|
||||
the move; layered in-band size also raises the reward score (§4).
|
||||
- **Churn control**: reprice only if |target − live| > reprice_ticks or size
|
||||
drift > 15%; batch cancel+place; respect the rate budget (03). Queue
|
||||
position is an asset — don't burn it for sub-tick prettiness.
|
||||
|
||||
### 3.3 Two-token quoting and the merge loop
|
||||
|
||||
On Polymarket, BUY NO @ q is economically SELL YES @ 1−q, but it is
|
||||
collateralized by USDC rather than by YES inventory. v2 makes the **BUY-YES +
|
||||
BUY-NO pair the canonical two-sided quote**:
|
||||
|
||||
- Both quotes are bids (USDC-collateralized) → we can quote two-sided with
|
||||
zero inventory, from day one, on both sides of the reward band.
|
||||
- When both fill: hold YES @ p and NO @ q with p + q < 1 → **merge** pair back
|
||||
to 1 USDC. Realized edge = 1 − p − q, captured without ever selling. The
|
||||
merge (native Python, 03) runs opportunistically in the background.
|
||||
- When inventory accumulates on one leg, exits are dual-routed, still maker:
|
||||
(a) keep/raise the opposite-token bid (merge exit), or
|
||||
(b) place a SELL limit on the held token above FV.
|
||||
The quoter maintains whichever route has the better expected time-to-fill ×
|
||||
price; often both, sized to not double-exit.
|
||||
|
||||
### 3.4 Sizing
|
||||
|
||||
- Base size per market from config; scaled down by toxicity, by inventory
|
||||
utilization (q/q_max), and near end-date (§6).
|
||||
- Respect `orderMinSize`; low-price multiplier logic from v1 (more shares when
|
||||
price < 0.10) is replaced by sizing in **USDC notional**, constant per quote
|
||||
regardless of price level.
|
||||
- No hardcoded 0.1–0.9 price band: quote anywhere the reward band and risk
|
||||
checks allow, but size shrinks near the boundaries where payoff asymmetry is
|
||||
extreme (long-shot bias zone; near-certain markets pay ~nothing to make).
|
||||
|
||||
## 4. Rewards and rebates as an objective
|
||||
|
||||
Two live income programs (03 §3), both maker-side, both enter the objective:
|
||||
|
||||
- **Liquidity rewards** (same family v1 targeted, confirmed live 2026):
|
||||
per-market daily rate, sampled ~per minute; score `S = ((v − s)/v)²` per
|
||||
order (s = distance from the dust-filtered *adjusted midpoint*, v =
|
||||
max_spread), min-size gate, two-sided rule
|
||||
`Q_min = max(min(Q_one, Q_two), max(Q_one/c, Q_two/c))` with c = 3.0, and —
|
||||
important — **when mid < 0.10 or > 0.90 only double-sided liquidity scores
|
||||
at all**. The formula already lives in v1's scanner (`add_formula_params`);
|
||||
v2 moves it into the strategy as a live utility term.
|
||||
- **Maker rebates** (new Jan 2026): ~25% of taker fees collected in politics
|
||||
markets is redistributed daily to that market's makers pro-rata. Unlike
|
||||
rewards (paid for *resting* in-band), rebates are paid for *being filled* —
|
||||
they directly subsidize adverse selection and shift the optimizer toward
|
||||
tighter quoting in fee-enabled markets with heavy taker flow.
|
||||
|
||||
```
|
||||
utility(placement) = E[spread capture] + E[reward $/day at this placement]
|
||||
+ E[rebate | fill] − E[adverse selection | regime]
|
||||
− inventory penalty
|
||||
```
|
||||
|
||||
Consequences the optimizer discovers on its own:
|
||||
- **Quiet regime** → sit deeper inside the band with more size (reward-dense,
|
||||
low fill probability, low AS cost) — "farming posture".
|
||||
- **Hot regime** → rewards can't compensate expected AS loss → widen beyond
|
||||
the band or pull entirely. v1 could never do this; its reward check was a
|
||||
one-way "must be inside band" constraint that *forced* it to quote tight
|
||||
exactly when it shouldn't.
|
||||
- **Two-sidedness**: the Qmin rule (and the hard double-sided requirement at
|
||||
extreme mids) pays for balanced quoting — another reason the
|
||||
BUY-YES/BUY-NO pair posture (§3.3) is the default; it keeps both sides
|
||||
in-band even while carrying inventory.
|
||||
- Income accounting: actual reward + rebate payouts pulled daily (03) and
|
||||
attributed per market, so the utility model's estimates are calibrated
|
||||
against reality instead of drifting.
|
||||
|
||||
## 5. Regime detection (when not to quote)
|
||||
|
||||
Per-market state machine, evaluated on every quoter wake:
|
||||
|
||||
| Regime | Trigger (any) | Behavior |
|
||||
|---|---|---|
|
||||
| QUIET | default | farming posture: in-band, layered, full size |
|
||||
| TRENDING | flow EWMA persistent one-sided; σ_short/σ_long elevated | lean FV with flow, widen δ, half size |
|
||||
| EVENT | sweep detector; FV jump > J ticks; neg-risk Σ dislocation; book depth evaporates | **pull all quotes**, cooloff T_c (~30–120s), re-enter at 2–3× δ decaying back to normal |
|
||||
| REDUCE_ONLY | inventory ≥ hard cap; end-date proximity; risk manager order | exit quotes only |
|
||||
| HALTED | WS stale > T; market flagged closed/`acceptingOrders=false`; kill switch | cancel all, no quoting |
|
||||
|
||||
The EVENT → re-enter-wide → decay pattern replaces v1's "stop-loss then sleep
|
||||
4 hours via JSON file". Cooldowns are seconds-to-minutes and price-aware, not
|
||||
hours and blind.
|
||||
|
||||
## 6. Inventory, exits, and lifecycle risk — without taker orders
|
||||
|
||||
Maker-only exit means **we cannot rely on being able to get out**. The
|
||||
controls compensate:
|
||||
|
||||
- **Soft cap `q_soft`**: skew (§3.1) already leans quotes; beyond q_soft the
|
||||
adding side is pulled entirely, exit side gains urgency: exit price decays
|
||||
from FV + δ toward one tick above best bid over τ_urgency (still maker,
|
||||
never crossing). Urgency accelerates if FV drifts against the position.
|
||||
- **Hard cap `q_max`**: reduce-only. Sizing (§3.4) makes approaching q_max
|
||||
progressively harder, so the hard stop is rarely the active constraint.
|
||||
- **No cost-basis anchoring**: v1 refused to ask below avgPrice, converting
|
||||
losses into resolution lotteries. v2 prices exits off FV and urgency only;
|
||||
cost basis is reporting, not strategy.
|
||||
- **Merge always-on**: min(YES, NO) ≥ threshold → merge to USDC (frees
|
||||
capital, realizes locked edge, zero market impact).
|
||||
- **End-date ladder** (from catalog metadata): T−D days → size taper begins;
|
||||
T−H hours → REDUCE_ONLY; T−S → HALTED + cancel. Political markets also
|
||||
resolve early: the market WS now pushes **`market_resolved` / `new_market`
|
||||
events** (03 §5) — instant halt on resolution — with catalog refresh
|
||||
(closed/`acceptingOrders` flags) and the EVENT regime as backstops.
|
||||
- **Per-market and global risk** (RiskManager, 02): market notional cap,
|
||||
neg-risk **event-group worst-case-loss cap** (multiple candidates in one
|
||||
event are one bet, not N), total exposure cap, daily realized-loss kill
|
||||
switch (halts quoting, keeps cancel/exit capability), stale-data halt.
|
||||
|
||||
## 7. Neg-risk structural opportunities (optional module, off by default)
|
||||
|
||||
Political events are neg-risk groups; the structure occasionally gifts money:
|
||||
|
||||
- **Set arbitrage**: if Σ best-asks of all outcomes < 1 − ε, buy the set
|
||||
(maker where possible, but this is the one place taker math is ever
|
||||
justified — behind a config flag, default off, consistent with the
|
||||
maker-only policy).
|
||||
- **NegRiskAdapter conversions**: NO(i) sets convert to YES(others) + USDC —
|
||||
enables inventory rebalancing across outcomes without trading. Scoped in 03
|
||||
(§contracts); implementation phase 4+.
|
||||
- Both are opportunistic add-ons; core PnL is quoting + rewards. Scoped now so
|
||||
the state model (event-group awareness) is built in from day one.
|
||||
|
||||
## 8. Parameters
|
||||
|
||||
`config/strategy.toml` holds named profiles (replaces the Hyperparameters
|
||||
sheet); `markets.toml` maps markets → profile + overrides (05 has schemas).
|
||||
Illustrative:
|
||||
|
||||
```toml
|
||||
[profiles.political-longdated]
|
||||
gamma = 0.5 # inventory risk aversion (skew strength)
|
||||
delta_min_ticks = 2
|
||||
c_vol = 1.2 # spread per unit short-vol
|
||||
c_tox = 2.0 # spread per unit toxicity score
|
||||
q_max_usdc = 500 # hard inventory cap, notional
|
||||
q_soft_frac = 0.6
|
||||
layers = 2
|
||||
reprice_ticks = 2
|
||||
debounce_ms = 200
|
||||
event_cooloff_s = 60
|
||||
end_date_taper_d = 7
|
||||
reduce_only_h = 24
|
||||
```
|
||||
|
||||
Every v1 magic number (0.005 reprice, 250 cap, 0.95/0.97/1.01, 15s, 0.1–0.9
|
||||
band, `max_size*2`) either becomes a named documented parameter or is deleted
|
||||
by design. Hot-reload via watchfiles; changes apply on next quoter wake, no
|
||||
restart.
|
||||
|
||||
## 9. Simulation, paper trading, and measurement
|
||||
|
||||
The strategy is only "much better" if we can prove it. Three rungs:
|
||||
|
||||
1. **Replay backtest** (`polymaker sim replay`): the journal (02) captures raw
|
||||
WS streams; the sim reconstructs books and runs the strategy with a queue-
|
||||
position fill model — conservative mode fills our order only when the tape
|
||||
prints *through* our price; optimistic mode fills pro-rata at the touch.
|
||||
Truth lies between; both are reported. Reward income is computed from the
|
||||
scoring formula on our simulated quotes. Start capturing journals from the
|
||||
moment Phase 2 lands, so weeks of political-market data exist before the
|
||||
strategy ships.
|
||||
2. **Paper mode** (`polymaker run --paper`): full live pipeline, orders
|
||||
journaled but not posted; fills simulated from the live tape. The gate for
|
||||
live capital.
|
||||
3. **Live KPIs** (05 §status): per-market markout curves (+30s/+5m), fill
|
||||
ratio, two-sided in-band uptime (the reward KPI), reward $ vs estimate,
|
||||
realized/unrealized PnL, inventory duration, EVENT-regime frequency.
|
||||
|
||||
Acceptance criteria to go live (Phase 4 gate, see 06): paper-mode ≥ 2 weeks on
|
||||
≥ 10 political markets with (a) positive net PnL including simulated rewards
|
||||
under the **conservative** fill model, (b) average +5m markout > −0.5 tick,
|
||||
(c) zero risk-limit breaches.
|
||||
|
||||
## 10. Explicit non-goals in v2 (deferred to 07)
|
||||
|
||||
- External data: polls, news, social monitors, other venues (Kalshi et al).
|
||||
- Taker execution of any kind (except the flagged neg-risk set-arb, default off).
|
||||
- Cross-event statistical arbitrage.
|
||||
- Sports/crypto-specific logic — the engine is generic, but tuning and the
|
||||
selector target political markets first.
|
||||
@@ -1,132 +0,0 @@
|
||||
# 05 — Local Config, Market Selection, and Operations
|
||||
|
||||
Google Sheets currently plays four roles: config store, market database,
|
||||
selection UI, and dashboard. v2 replaces each with a local equivalent. The
|
||||
test: **a laptop with the repo, a `.env`, and a funded wallet is a complete
|
||||
deployment.** No Google account, no service-account JSON, no second IP running
|
||||
a scanner.
|
||||
|
||||
## 1. Configuration files (replaces Hyperparameters + Selected Markets sheets)
|
||||
|
||||
All TOML, all in `config/`, all validated by pydantic at load, all
|
||||
hot-reloadable via watchfiles (applied at the next quoter wake; invalid edits
|
||||
are rejected with a logged diff, keeping the last good config — an edit can
|
||||
never crash the bot).
|
||||
|
||||
### `config/config.toml` — engine & account
|
||||
```toml
|
||||
[wallet]
|
||||
# secrets stay in .env: PK, BROWSER_ADDRESS; this file only references them
|
||||
signature_type = 2 # revisit after Phase 2 wallet spike (03 §1)
|
||||
|
||||
[engine]
|
||||
debounce_ms = 200
|
||||
reconcile_interval_s = 30
|
||||
heartbeat = true # exchange dead-man switch (03 §3)
|
||||
journal = true # raw WS/order journal for the backtester
|
||||
|
||||
[risk] # global — per-market caps live in markets.toml
|
||||
max_total_exposure_usdc = 5000
|
||||
max_event_group_loss_usdc = 1000 # neg-risk group worst-case
|
||||
daily_loss_kill_usdc = 250
|
||||
ws_stale_halt_s = 10
|
||||
```
|
||||
|
||||
### `config/strategy.toml` — named parameter profiles
|
||||
Schema in 04 §8. Profiles (`political-longdated`, `political-hot`, …) replace
|
||||
the Hyperparameters sheet's `param_type` groups.
|
||||
|
||||
### `config/markets.toml` — the trade list (replaces Selected Markets sheet)
|
||||
```toml
|
||||
[[markets]]
|
||||
slug = "will-x-win-the-2028-democratic-nomination"
|
||||
profile = "political-longdated"
|
||||
q_max_usdc = 800 # optional per-market overrides of the profile
|
||||
|
||||
[[markets]]
|
||||
condition_id = "0x37a6de..." # slug or condition_id both accepted
|
||||
profile = "political-hot"
|
||||
enabled = false # keep the entry, stop quoting
|
||||
```
|
||||
The bot resolves slugs → tokens/metadata via the catalog at startup and on
|
||||
reload. Adding a market while running = edit file, save; the engine
|
||||
subscribes, seeds the book, starts quoting. Removing/disabling = cancel that
|
||||
market's orders, unsubscribe. This is the entire "market management UI".
|
||||
|
||||
`.env` keeps only secrets: `PK`, `BROWSER_ADDRESS`, optional RPC URL override.
|
||||
`SPREADSHEET_URL` and `credentials.json` are deleted.
|
||||
|
||||
## 2. Market catalog & scanner (replaces data_updater + All/Volatility Markets sheets)
|
||||
|
||||
`state.db` (SQLite) holds the market catalog; the scanner is a subcommand, not
|
||||
a second deployment:
|
||||
|
||||
- `polymaker scan` — Gamma keyset sweep with server-side filters (03 §4):
|
||||
politics tag(s), active, min liquidity/volume, end-date window, rewards
|
||||
present. Enriches the shortlist with batch `/books` depth and reward-density
|
||||
estimates (the v1 `add_formula_params` math, kept) **plus the maker-rebate
|
||||
estimate** (taker-fee volume × rebate share — 03 §3). Writes to SQLite.
|
||||
Seconds per run; scheduled inside the bot process (e.g. every 15 min) —
|
||||
the "run on a different IP" advice dies with the crawl that motivated it.
|
||||
- `polymaker markets` — rank/browse the catalog in the terminal (rich table):
|
||||
`polymaker markets --tag politics --min-reward 20 --max-vol-sum 20 --sort score`.
|
||||
Score = expected daily income (rewards + rebates) vs. volatility & spread
|
||||
risk — v1's `gm_reward_per_100` / `volatility_sum` composite, recalibrated.
|
||||
- `polymaker markets add <slug> [--profile P]` — appends to `markets.toml`
|
||||
(file remains the source of truth; the command is a convenience editor).
|
||||
- Volatility for ranking: `prices-history` coarse estimate at scan time;
|
||||
markets we quote get live WS-derived vol which supersedes it.
|
||||
|
||||
## 3. Runtime CLI (replaces update_stats.py + Summary sheet + eyeballing prints)
|
||||
|
||||
Single `polymaker` entrypoint (typer):
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `polymaker run [--paper]` | start the engine (paper mode: full pipeline, no order POSTs) |
|
||||
| `polymaker status` | positions, open quotes, inventory, PnL, regime per market (reads SQLite; works while the bot runs) |
|
||||
| `polymaker pnl [--daily]` | realized/unrealized PnL, reward + rebate income vs. estimates |
|
||||
| `polymaker flatten [market]` | reduce-only mode (maker exits) for one/all markets |
|
||||
| `polymaker cancel-all` | panic button (also runs automatically at startup) |
|
||||
| `polymaker scan` / `markets` | catalog (above) |
|
||||
| `polymaker sim replay ...` | backtester (04 §9) |
|
||||
| `polymaker doctor` | preflight: wallet auth spike checks, pUSD balance/allowances, WS reachability, clock skew |
|
||||
|
||||
## 4. Observability
|
||||
|
||||
- **structlog** JSON logs to `logs/` + human-readable console. Every order
|
||||
decision logs its inputs (FV, σ, inventory, regime) — a quote is always
|
||||
explainable after the fact.
|
||||
- **Journal** (`journal/*.jsonl`): raw WS in / orders out. Feeds the
|
||||
backtester and post-mortems.
|
||||
- **SQLite** is the queryable operational record: fills, orders, PnL marks,
|
||||
reward payouts, risk events. Any dashboard we want later (Grafana, a small
|
||||
web page) reads this — no scoping dependency on it now.
|
||||
- **Alerting (minimal, phase 4)**: kill-switch trips, WS-stale halts,
|
||||
reconcile divergence, heartbeat failures → log at CRITICAL + optional
|
||||
webhook URL in config (generic; Discord/Telegram/ntfy all take a POST).
|
||||
- KPI set defined in 04 §9 (markouts, two-sided in-band uptime, fill ratio,
|
||||
income vs. estimate).
|
||||
|
||||
## 5. Deployment & runtime ops
|
||||
|
||||
- **uv end-to-end** (already adopted; finish the job): `requires-python =
|
||||
">=3.12"`, refreshed lock, `uv run polymaker ...`, dev group with ruff,
|
||||
mypy, pytest, pytest-asyncio. Delete the 3.9 pin.
|
||||
- **Placement**: any always-on Linux box; latency to the CLOB (AWS us-east
|
||||
region) is the one infra knob that matters for queue position — a small VPS
|
||||
in us-east-1 beats a laptop on wifi. Not required for correctness.
|
||||
- **Supervision**: systemd unit example in the README (`Restart=always`);
|
||||
startup sequence is cancel-all → snapshot → resume, so a restart is always
|
||||
safe. The exchange heartbeat covers the gap between death and restart.
|
||||
- **CI (GitHub Actions)**: ruff + mypy + pytest on PR. No deploy pipeline
|
||||
needed for a single-operator bot.
|
||||
- **State backup**: `state.db` + `config/` are the whole world; journal grows
|
||||
~MBs/day per market — rotate + optionally compress with age.
|
||||
|
||||
## 6. Security notes
|
||||
|
||||
- `PK` only ever in `.env` (gitignored) / environment; never in TOML, logs,
|
||||
or SQLite. Log wallet addresses truncated.
|
||||
- The CLI never prints secrets; `doctor` verifies without echoing.
|
||||
- Dependencies pinned via uv.lock; renovate/dependabot optional later.
|
||||
@@ -1,109 +0,0 @@
|
||||
# 06 — Migration Plan
|
||||
|
||||
Phased so that every phase leaves the repo in a working, testable state, and
|
||||
the risky unknowns (wallet auth on V2, pUSD) are attacked first. v1 cannot
|
||||
trade anyway (CLOB V1 is gone — 03), so there is no parallel-run constraint:
|
||||
we are building v2 in place, deleting v1 as replacements land.
|
||||
|
||||
## Phase 0 — Repo hygiene & toolchain (small)
|
||||
|
||||
- `src/polymaker/` package skeleton (02 layout), typer CLI stub.
|
||||
- pyproject: `requires-python = ">=3.12"`, `.python-version` → 3.12.x, fresh
|
||||
`uv lock`; dev group: ruff, mypy, pytest, pytest-asyncio; ruff replaces
|
||||
black; GitHub Actions running lint+types+tests.
|
||||
- Delete immediately (nothing depends on them working): `poly_stats/`,
|
||||
`update_stats.py`, the read-only Sheets CSV hack, dead commented code.
|
||||
The rest of the v1 code (`trading.py`/`main.py`/`data_updater/` …) was parked
|
||||
under `legacy/` during the port and has now been deleted — it survives only in
|
||||
git history on `main`.
|
||||
- Remove deps: gspread, gspread-dataframe, google-auth. Add: httpx, pydantic,
|
||||
structlog, typer, rich, watchfiles, uvloop, py-clob-client-v2.
|
||||
|
||||
**Exit:** `uv run polymaker --help` works; CI green; no Google imports left.
|
||||
|
||||
## Phase 1 — Catalog, config, scanner (replaces Sheets as data source)
|
||||
|
||||
- pydantic config models + TOML loading + hot reload (05 §1).
|
||||
- Gamma keyset client, tag resolution, SQLite catalog, market scoring
|
||||
(rewards + rebates + vol), `polymaker scan` / `markets` / `markets add`
|
||||
(05 §2).
|
||||
- CLOB public data client (books batch, midpoints, prices-history) for
|
||||
enrichment.
|
||||
|
||||
**Exit:** politics markets scanned, ranked, and selected into `markets.toml`
|
||||
in seconds, with no Google anywhere. (v1 bot is now un-runnable — fine, see
|
||||
above.)
|
||||
|
||||
## Phase 2 — Execution core (the V2 unknowns live here — do the spikes first)
|
||||
|
||||
- **Spike A (days, throwaway code): wallet auth on V2.** Wrap USDC→pUSD,
|
||||
set allowances, place + cancel one post-only order with our exact wallet
|
||||
(signature_type 2), read it back on the user WS. Resolves 03 §9 items 3/7.
|
||||
Decision output: keep proxy wallet vs. move to EOA; py-clob-client-v2 vs.
|
||||
direct signing.
|
||||
- **Spike B: WS contract.** Capture live `book`/`price_change`/`best_bid_ask`
|
||||
/`tick_size_change` payloads (custom features on), confirm ping/pong,
|
||||
write the parsers against reality.
|
||||
- Then build: MarketDataService (books + analytics + resync), UserStream,
|
||||
StateStore + order/trade state machine, ExecutionGateway (signer, httpx,
|
||||
rate budgeter, heartbeats, batch ops, reconciler), Merger (native Python,
|
||||
Safe-or-EOA per Spike A), journal.
|
||||
- Integration test: on 1–2 quiet live markets with $10-size manual targets,
|
||||
prove place/cancel/fill/merge/reconcile round-trips, then heartbeat-driven
|
||||
auto-cancel on kill.
|
||||
|
||||
**Exit:** engine can hold a hand-specified quote set live, survive WS drops,
|
||||
crashes, and restarts safely. Journal capture running 24/7 from here on (the
|
||||
backtester's data supply).
|
||||
|
||||
## Phase 3 — Strategy engine (04) + simulation
|
||||
|
||||
- Pure strategy module: FV, vol/toxicity, regime machine, quote construction,
|
||||
reward/rebate utility, inventory/exit logic — with unit tests per component.
|
||||
- Replay backtester over the journals accumulated since Phase 2; parameter
|
||||
study for the political profiles.
|
||||
- Paper mode (`run --paper`) end-to-end.
|
||||
- Delete the parked v1 code — the port is done. (Done: `legacy/` removed.)
|
||||
|
||||
**Exit:** paper trading ≥2 weeks, ≥10 political markets, meeting the
|
||||
acceptance gates from 04 §9 (positive PnL under conservative fills, markouts
|
||||
> −0.5 tick at +5m, zero risk breaches).
|
||||
|
||||
## Phase 4 — Live rollout + ops hardening
|
||||
|
||||
- RiskManager fully wired (caps, kill switches, event-group limits), alert
|
||||
webhook, `status`/`pnl`/`flatten`/`doctor` polished.
|
||||
- Go live: 2–3 markets at minimum size → widen coverage/size on weekly review
|
||||
of the KPI set. us-east VPS placement.
|
||||
- Reward/rebate reconciliation against actual daily payouts.
|
||||
|
||||
**Exit:** stable unattended operation; income attribution (spread vs rewards
|
||||
vs rebates) reported per market.
|
||||
|
||||
## Phase 5 — Deferred (separately scoped)
|
||||
|
||||
- External data feeds (07): polls, news, cross-venue.
|
||||
- Neg-risk set-arb + NegRiskAdapter conversions (04 §7) if the structural
|
||||
edge shows up in captured data.
|
||||
- Optional: RTDS `activity` flow features, Prometheus/Grafana, Polymarket US
|
||||
venue.
|
||||
|
||||
## Deletion ledger (what's gone by the end of Phase 3)
|
||||
|
||||
`trading.py`, `main.py`, `update_markets.py`, `update_stats.py`, `poly_data/`,
|
||||
`poly_utils/`, `poly_stats/`, `data_updater/`, `poly_merger/` (entire Node
|
||||
runtime), `positions/` runtime dir, `credentials.json` expectations,
|
||||
`SPREADSHEET_URL`, Python 3.9 pin, gspread/gspread-dataframe/google-auth,
|
||||
py-clob-client (v1), pandas-in-hot-path, all `print()` logging, all bare
|
||||
excepts, all `gc.collect()` calls.
|
||||
|
||||
## Risk register
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| V2 SDK auth bugs with proxy/Safe wallets | Spike A first; EOA fallback decided in days, not after building |
|
||||
| pUSD mechanics undocumented corners | Spike A covers wrap/allowance/merge collateral; treat collateral as per-market config |
|
||||
| WS payload drift (young V2 docs) | Spike B parses from captured reality; parsers tolerate unknown fields |
|
||||
| Maker-only exits too slow in a news jump | Position caps sized for survivable worst case (04 §6); EVENT regime pulls quotes early; accept this as the strategy's core tradeoff |
|
||||
| Rewards program parameters shift | All reward math data-driven from per-market API fields; utility recalibrated from actual payouts |
|
||||
| Strategy underperforms | Paper-gate before capital; journal-driven iteration loop is the product as much as the bot |
|
||||
@@ -1,63 +0,0 @@
|
||||
# 07 — Future: External Data Feeds (deferred, scoped now)
|
||||
|
||||
v2 trades political markets on **microstructure alone** (04 §1). That is a
|
||||
deliberate constraint, not a belief that external data has no value — the
|
||||
architecture reserves a seat for it so adding feeds later is additive, not a
|
||||
refactor. This doc scopes the seat and the candidate feeds; **none of this is
|
||||
built in v2.**
|
||||
|
||||
## Architectural seat (built in v2, empty)
|
||||
|
||||
The strategy signature already is `(BookState, Inventory, Params, Clock) ->
|
||||
TargetQuotes`. The extension is one optional input:
|
||||
|
||||
```
|
||||
SignalBus: name -> {value, confidence, as_of, ttl}
|
||||
```
|
||||
|
||||
- Strategy reads signals as **FV adjusters and regime inputs** with explicit
|
||||
staleness: a signal past its TTL contributes nothing (fail-safe to pure
|
||||
microstructure — the bot must never be worse than v2 because a feed died).
|
||||
- Feed adapters are separate async tasks (or even separate processes writing
|
||||
to the bus via SQLite/socket) — never in the quote path; the hot loop only
|
||||
reads a dict.
|
||||
- Every signal is journaled like WS data, so the replay backtester (04 §9)
|
||||
can grade any feed's marginal value before it touches live quotes. **A feed
|
||||
earns its way in through replay evidence, or it doesn't ship.**
|
||||
|
||||
## Candidate feeds for political markets (rough priority)
|
||||
|
||||
1. **Event calendar (highest value / lowest effort)**: debates, primaries,
|
||||
election nights, court dates, scheduled rulings. Even a hand-maintained
|
||||
`calendar.toml` (date, market tags, severity) lets the regime machine
|
||||
pre-widen/pull *before* scheduled volatility instead of reacting to the
|
||||
first sweep. No API needed to start.
|
||||
2. **Cross-venue prices**: Kalshi (public API), and Polymarket US as it
|
||||
grows. Same-event price divergence is both an FV prior and a toxicity
|
||||
warning (fast flow often arbs venues; if Kalshi moved and we didn't, our
|
||||
quote is stale). Clean REST/WS, well-bounded work.
|
||||
3. **Poll aggregates / forecast models**: RCP/538-style averages, Silver
|
||||
Bulletin, model outputs. Slow-moving FV anchor for long-dated markets —
|
||||
mostly useful as a *sanity band* (flag when market FV drifts far from
|
||||
model FV) rather than a quoting signal.
|
||||
4. **News/headline triggers**: wire headlines, X/Twitter firehose, Google
|
||||
Trends spikes. Highest alpha, highest noise and engineering cost;
|
||||
realistically an "instant EVENT-regime trigger" (pull quotes on keyword
|
||||
burst), not a pricing input. Last in line.
|
||||
5. **Polymarket-internal extras** (not really "external", could land earlier):
|
||||
RTDS `activity` topic for platform-wide flow/toxicity features, comment
|
||||
velocity per market as an attention proxy (03 §5).
|
||||
|
||||
## Explicitly out of scope until then
|
||||
|
||||
- Any taker execution driven by signals (feeds inform *maker* quoting only —
|
||||
the maker-only policy survives the feed era).
|
||||
- LLM/news-summarization pipelines, sentiment models.
|
||||
- Trading the Polymarket US venue (separate API/regulatory surface — 03 §8).
|
||||
|
||||
## Preconditions before starting this phase
|
||||
|
||||
- v2 live and stable through at least one high-volatility political event.
|
||||
- Journal + replay infrastructure proven (it is the evaluation harness here).
|
||||
- KPI baseline established, so each feed's marginal contribution is measured
|
||||
against a known control.
|
||||
@@ -62,7 +62,7 @@ class GammaClient:
|
||||
|
||||
Uses the offset `/markets` endpoint because it reliably supports
|
||||
`tag_id` filtering today. (Keyset is the go-forward per docs; switch when
|
||||
it supports tag filtering. See docs/scoping/03-api-layer.md §4.)
|
||||
it supports tag filtering. See the README.)
|
||||
"""
|
||||
offset = 0
|
||||
for _ in range(max_pages):
|
||||
|
||||
@@ -4,7 +4,7 @@ Verifies the environment is ready to trade WITHOUT posting any order:
|
||||
config + secrets, CLOB/Gamma reachable, wallet auth (L1->L2 creds), collateral
|
||||
balance + positions ON THE FUNDER (deposit/developer wallet, where funds live),
|
||||
a live market-WS book frame, and an authenticated user-WS connection. This is
|
||||
the gate before the live $5 round-trip (docs/scoping/06 Phase 2).
|
||||
the gate before the live $5 round-trip (the README).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -84,7 +84,7 @@ async def run_doctor(cfg: Config, console: Console) -> bool:
|
||||
except Exception as e: # noqa: BLE001
|
||||
check("wallet auth (L2 creds derived)", False, str(e))
|
||||
console.print(" [yellow]! signature-type mismatch? deposit wallets use sig_type=3 "
|
||||
"(config.toml). See docs 03 §9.[/yellow]")
|
||||
"(config.toml). See the README.[/yellow]")
|
||||
else:
|
||||
console.print(" [yellow]! skipping wallet checks (no secrets)[/yellow]")
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ class Side(str, Enum):
|
||||
|
||||
|
||||
class Regime(str, Enum):
|
||||
"""Per-market quoting regime (see docs/scoping/04-strategy.md §5)."""
|
||||
"""Per-market quoting regime (see the README)."""
|
||||
|
||||
QUIET = "QUIET" # farming posture: in-band, layered, full size
|
||||
TRENDING = "TRENDING" # persistent one-sided flow: lean + widen + half size
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Append-only JSONL event journal.
|
||||
|
||||
Captures raw WS-in and orders-out so the replay backtester (docs 04 §9) can
|
||||
Captures raw WS-in and orders-out so the replay backtester (see the README) can
|
||||
reconstruct books and re-run the strategy. Also the substrate for post-mortems.
|
||||
Cheap: one line per event, flushed, rotated by day.
|
||||
"""
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Live wallet round-trip test — the Phase-2 wallet spike (docs 06).
|
||||
"""Live wallet round-trip test — the Phase-2 wallet spike (see the README).
|
||||
|
||||
Proves the full V2 order path against the real exchange with minimal risk:
|
||||
places ONE post-only BUY well below the touch (so it rests and cannot fill),
|
||||
@@ -50,7 +50,7 @@ async def run_livetest(cfg: Config, console: Console, notional_usdc: float = 5.0
|
||||
console.print(f" [green]✓[/green] wallet auth — address {gw.address[:12]}…")
|
||||
except Exception as e: # noqa: BLE001
|
||||
console.print(f" [red]✗ wallet auth failed:[/red] {e}")
|
||||
console.print(" [yellow]Auth/signature-type mismatch (docs 03 §9). If your account has a "
|
||||
console.print(" [yellow]Auth/signature-type mismatch (see the README). If your account has a "
|
||||
"'deposit address', set signature_type=3 (POLY_1271) in config.toml and use the "
|
||||
"deposit address as BROWSER_ADDRESS. Errors like 'maker address not allowed, use "
|
||||
"the deposit wallet flow' or 'signer must be the API key address' mean the type is "
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Pure parsers for market-WS wire messages -> structured updates.
|
||||
|
||||
Kept separate from the socket so they're unit-testable against captured frames.
|
||||
Verified against live frames on 2026-07-05 (docs/scoping/03-api-layer.md §5):
|
||||
Verified against live frames on 2026-07-05 (the README):
|
||||
|
||||
book: {market, asset_id, bids:[{price,size}], asks:[...], timestamp, hash, tick_size}
|
||||
price_change:{market, timestamp, price_changes:[{asset_id, price, size, side, hash}]}
|
||||
|
||||
@@ -6,7 +6,7 @@ collateral (1 USDC/pUSD per pair) — a maker-only exit with zero market impact.
|
||||
Two execution paths:
|
||||
* EOA wallet (signature_type=0): direct contract call, fully implemented here.
|
||||
* Proxy/Safe wallet (signature_type 1/2): the merge tx must be routed through
|
||||
the Safe. That path is gated on the Phase-2 wallet spike (docs 03 §6) — until
|
||||
the Safe. That path is gated on the Phase-2 wallet spike (see the README) — until
|
||||
then merging is skipped (logged), and inventory is exited via limit sells
|
||||
instead. The bot is fully functional without it; merging just frees capital
|
||||
sooner.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""RiskManager: pre-trade gates and circuit breakers (docs 04 §6, 02).
|
||||
"""RiskManager: pre-trade gates and circuit breakers (see the README).
|
||||
|
||||
Consulted by the engine before every quote set. Returns a per-market decision
|
||||
(size scale / reduce-only / halt) and owns the global kill switches. Position
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""StateStore: the single owner of positions and open orders.
|
||||
|
||||
Replaces v1's module-level global dicts + the `performing`/`last_trade_update`
|
||||
races. Three inputs, one arbitration rule (docs/scoping/02-architecture.md):
|
||||
races. Three inputs, one arbitration rule (the README):
|
||||
|
||||
* WS fill events apply immediately (optimistic),
|
||||
* REST reconciliation corrects drift ONLY for tokens with no in-flight trades,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Consumes *normalized* user-stream events (the wire-format extraction lives in
|
||||
userstream/, so this is unit-testable with synthetic events) and drives the
|
||||
state machine from docs/scoping/02-architecture.md:
|
||||
state machine from the README:
|
||||
|
||||
Trade: MATCHED -> MINED -> CONFIRMED
|
||||
└──────────-> FAILED (roll back the optimistic fill, reconcile)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
This is the deterministic core of the strategy. No I/O, no wall-clock reads
|
||||
except values passed in. Everything here is exercised directly by unit tests.
|
||||
|
||||
Model (see docs/scoping/04-strategy.md):
|
||||
Model (see the README):
|
||||
reservation r = FV - skew(inventory)
|
||||
half-spread δ = base + c_vol·σ + c_tox·toxicity (clamped to reward band in QUIET)
|
||||
YES entry bid = r - δ (BUY YES, USDC-collateralized)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Per-market regime decision (see docs/scoping/04-strategy.md §5).
|
||||
"""Per-market regime decision (see the README).
|
||||
|
||||
Priority order, highest first:
|
||||
HALTED kill switch / stale data / resolved / past halt-before window
|
||||
|
||||
@@ -8,7 +8,7 @@ always the maker):
|
||||
* maker & taker on DIFFERENT outcomes -> a mint: we BUY the opposite token
|
||||
|
||||
NOTE: exact field names must be reconfirmed in the Phase-2 wallet spike
|
||||
(docs/scoping/03-api-layer.md §9); this is coded to the v1-observed shape.
|
||||
(the README); this is coded to the v1-observed shape.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
Reference in New Issue
Block a user